odelia_v0.6.0

Release odelia_v0.6.0 - Swarm learning with weak supervision enables automatic breast cancer detection in magnetic resonance imaging

We are excited to announce the release of odelia_v0.6.0 for the swarm-learning-hpe project, focusing on significant enhancements in utilizing swarm learning for radiology image analysis. This release incorporates findings from our latest research, demonstrating the power of weakly supervised learning combined with swarm learning (SL) for effective and privacy-preserving medical AI development.

Highlights of This Release

• Integration of Weakly Supervised Learning: Leveraging case labels for tumor detection in MRI images without the need for detailed annotations, as detailed in our latest research paper.
• Advanced Swarm Learning Framework: Enhanced with the latest HPE Swarm Learning version 2.2.0, facilitating decentralized, privacy-preserving machine learning across multiple nodes without the need for raw data exchange.
• Comprehensive Medical Imaging Analysis: Support for both histopathology and radiology image analysis, including implementations for attention MIL-based models and 3D-CNN models for breast MRI examinations.
• Preprocessing Workflow Enhancement: A unified preprocessing pipeline for all datasets, ensuring optimal model training and evaluation conditions.

What's New

• Preprocessed Dataset Support: Introduction of an optional step to download preprocessed datasets for streamlined setup and testing. Preprocessing repo
• Real-World Validation: Incorporation of real-world multicentric study results, validating the 3D-ResNet101 model architecture across diverse datasets, demonstrating superior performance and generalizability
• Explainability Features: New explainability analyses, including Gradient-weighted Class Activation Mapping (GradCAM++) and Occlusion Sensitivity Analysis, provide insights into model decision-making processes.

Installation & Usage

Please refer to the Installation Guide for detailed steps on setting up the swarm learning environment. Ensure you meet the hardware and software prerequisites for optimal performance.

Quick Start

1. Clone the repository and navigate to the project directory:

   git clone https://github.com/KatherLab/swarm-learning-hpe.git
   cd swarm-learning-hpe

2. Follow the installation instructions to set up your swarm learning environment.
3. Begin training your models using our preprocessed datasets or your data following our comprehensive Usage Guide.

Contributing

We welcome contributions from the community! If you're interested in enhancing the project or have ideas, please open an issue or submit a pull request. Check our contributing guidelines for more details.

Acknowledgments

Special thanks to our collaborators and contributors who have made this release possible. This project benefits from the findings of our recent research on integrating weakly supervised learning with swarm learning for breast MRI analysis.

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

We hope this release will empower more researchers and practitioners in the medical imaging field to leverage the potential of swarm learning. For any questions or support, please reach out through our Issues section.

swag_v0.1.0

SWAG Consortium Release Notes

Key Features

• Connection via Tailscale: Easy setup instructions for connecting to the SWAG network using Tailscale. Tailscale Installation Guide.

• Latent Diffusion Model (LDM) Workspace: Detailed guide for applying the Latent Diffusion Model including environment setup, dataset preparation, training the autoencoder, training the LDM, and sampling from the trained LDM. LDM Workspace.

  • Dataset: Utilizes the NIH Chest X-rays dataset available at Kaggle.
  • Training and sampling commands provided with detailed arguments explanation.

• Score Based Model Workspace: Instructions on preparing and using a sample X-ray dataset for score-based modeling. Score-Based Model Workspace.

  • Dataset Acquisition: Direct link for unconditional dataset from the vinDr dataset provided. Instructions for creating your own dataset from DICOM files.
  • Dataset Specification: Guidelines for dataset structure and recommendations for image properties.

Additional Resources

• Dataset Links: Direct access to required datasets for both Latent Diffusion Model and Score Based Model implementations.
• Configuration Tips: Advice on setting user-specific path settings for successful model training and evaluation.

User Specific Path Settings

• Instructions for configuring the swop_profile.yaml file to specify the path of the dataset directory on your local machine.

Getting Started

1. Install Tailscale and connect to the SWAG network as described in the Connection via Tailscale section.
2. Choose a diffusion model to apply (LDM or Score Based Model) and follow the respective workspace instructions for setup and use.

Contact

For authentication key and further assistance, please contact the SWAG administrator: Fabian Laqua ([email protected]).

This is a pre-release version and will undergo real-world testing soon. Feedback and contributions are welcome.

Full Changelog: v0.4.0...swag_v0.1.0

v0.4.0

We are happy to announce the latest working version with Swarm 2.2.0 community release integrated.

This release brings significant enhancements and powerful features to the Swarm manageability framework, improving the management of user ML workloads. Notable updates include:

• Features:

  • Targeted SWOP Command: Allows targeting tasks on a specific SWOP node, dynamic addition of peers to ongoing task execution, and retrying failed tasks.
  • WITH ALL PEERS Command: Triggers task execution on all available peers.
  • Swarm Support for SPIRE as Certificate Manager: Includes a CLI-based SPIRE example (spire/cifar10).
  • Real World NIH Example: Showcases a Swarm use case with a real-world NIH dataset.
  • Documentation Enhancements: Provides updated and comprehensive documentation for users.
  • **Aliases added. please see readme for convenient command excecution.
  • **Working version with 3-site joint training with diverse breast MRI data classification.

• Defect Fixes:

  • Resolved an issue where the stale SL Admin node was stuck waiting for quorum while a new Admin was selected.
  • Enabled support for non-default APLS ports from SLM-UI.
  • Fixed issues during the restart of SLM-UI container while running a training.

v0.3.0

🚀 Announcement: Release of Swarm Learning 0.3 🚀

We're excited to unveil the latest upgrades to HPE Swarm Learning, encompassing both versions 2.0.0 and 2.1.0. These new releases come packed with significant updates, enhancements, and fixes tailored to optimize user experience and offer advanced features.

💡 Key Updates:

Upgrading from Older Versions:

For those who have the older Swarm Learning versions installed (1.x.x), please refer to the Upgrade Guide to seamlessly transition to 2.x.x.

Note: During the SL setup process, you'll require specific credentials. Please send me a private message once your SL machine is ready, and I will distribute the necessary credentials securely.

SLM-UI Enhancements:

• Display of model training metrics.
• Easier navigation through ML logs.
• Centralized swarm log collector for streamlined diagnostics.
• Swarm product installation and management via SLM-UI.

Merge Algorithm Options:

• Introduction of merge methods like Co-ordinate Median & Geometric Median.
• Customizable merges, allowing configurations either through I/O or Memory-optimized modes.

Persistent Blockchain Data:

• Facilitating offline analysis and rapid Swarm network restarts.

Podman Support:

• A Docker alternative with features like rootless privileges and SELinux compatibility on RHEL.

High Availability Enhancements:

• Resilient handling of SN node failures, enhanced mesh topology, and streamlined management of SL leader failures.

Swarm Client Library:

• Extension possibilities for new ML platforms.

Miscellaneous:

• Augmented diagnostics for SWOP & SN.
• Integration of a containerized License Server (APLS).
• Refreshed documentation with new examples.

🪲 Defect Fixes:

• Refinements in SN restart pathway.
• 'LIST NODES' functionality now only showcases active nodes.
• Updated Swarm Learning topology and reverse proxy considerations.
• Diagnostics in place for expired certificate scenarios.

Your collaboration is invaluable to us, and we're dedicated to ensuring a seamless experience for you. Let's continue evolving together! 🌟

Odelia WP1

Release Description v0.2.0:

Swarm-learning-hpe v0.2.0 is the second release of the decentralized, privacy-preserving Machine Learning (ML) framework, based on HPE platform. This release provides several improvements, new features and bug fixes. The repository now provides two working models: marugoto_mri and odelia-breast-mri for Attention MIL based model and 3D-CNN model respectively. Furthermore, there is a new automated workflow for keeping track of the ongoing processes in the repository. The issue section is updated for ease of use, so people can dump their ideas and raise any questions encountered when using this repo.

This release supports Ubuntu 20.04 LTS, which is the recommended operating system. There are specific hardware recommendations for the Swarm Learning Environment, such as 64 GB of RAM, 16 CPU cores, an NVIDIA GPU with 48 GB of RAM, and 8 TB of storage. We have tested the Swarm Learning Environment on Ubuntu 20.04 LTS, Ubuntu 22.04.2 LTS, and Ubuntu 20.04.5 LTS.

The release provides detailed instructions on how to install and set up the Swarm Learning Environment. The Install section of the readme file provides information on the prerequisites and how to set up the user and repository. The Usage section provides instructions on how to run Swarm Learning Nodes, check logs, and view results. The Troubleshooting section provides solutions to some of the most common issues that users might encounter while setting up the environment.

The repository also includes a Milestone section for keeping track of the project's progress, a NotionPage section for collaboration, and a Contributing section for guidelines on how to contribute to the project.

Swarm-learning-hpe v0.2.0 is maintained by the TUD Swarm learning team, with Jeff as the primary maintainer. This project exists thanks to all the people who contribute. The project uses the platform from the HewlettPackard/swarm-learning repository, created by HewlettPackard.

Swarm-learning-hpe v0.2.0 is released under the MIT license.