Skip to content

Commit d618e6d

Browse files
mocsharpMMelQin
andauthored
MONAI Pipeline Generator (#550)
* Initial check-in for pipeline generator Signed-off-by: Victor Chang <vicchang@nvidia.com> * Remove duplicated MONAI models Signed-off-by: Victor Chang <vicchang@nvidia.com> * Add ImageOverlayWriter and update ImageDirectoryLoader - Introduced ImageOverlayWriter for blending segmentation masks onto RGB images and saving as PNG. - Updated ImageDirectoryLoader to support channel-first output configuration and improved file searching logic. - Adjusted documentation and examples to reflect the transition from Poetry to uv for command execution. - Updated various configuration files and templates to accommodate new features and dependencies. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Add support for Llama3-VILA-M3 models with new operators - Introduced PromptsLoaderOperator to read prompts from a YAML file and emit them sequentially. - Added Llama3VILAInferenceOperator for running inference with the Llama3-VILA-M3-3B model, supporting multiple output types. - Created VLMResultsWriterOperator to write results to disk based on the specified output type. - Updated configuration files and templates to integrate new operators and support custom input/output types. - Added unit tests for the new operators to ensure functionality and correctness. Signed-off-by: [Your Name] <your.email@example.com> Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update README and design documentation for pipeline generator - Enhanced README with links to MONAI Deploy and MONAI Bundles for better clarity. - Clarified usage instructions and examples for generating applications from models. - Updated design documentation to reflect the correct Python version requirement (changed from 3.12 to 3.10). - Improved descriptions in the generated application structure and additional models section. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Bump version from 0.1.0 to 1.0.0 in pyproject.toml for pipeline-generator Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor operator imports and improve code formatting - Updated import statements in various operator files to use parentheses for better readability. - Cleaned up whitespace and formatting inconsistencies across multiple files, enhancing overall code clarity. - Ensured consistent handling of newlines and indentation in operator implementations and test files. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Add test pipeline generator workflow and bump version to 1.0.0 - Introduced a new GitHub Actions workflow for testing the pipeline generator, including steps for setting up Python 3.10, installing dependencies, and running tests. - Updated the version of the pipeline-generator package from 0.1.0 to 1.0.0 in the uv.lock file. - Refactored import statements and improved code formatting in various files for better readability and consistency. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor ImageOverlayWriter documentation and enhance model_id validation - Moved the documentation for ImageOverlayWriter into the class docstring for better organization and clarity. - Improved the model_id validation logic in AppGenerator to prevent code injection and path traversal, ensuring stricter input checks. - Updated the generated application template to reflect changes in the channel_first logic. - Added unit tests to verify the correctness of the refactored channel_first logic. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance application documentation and refine requirements for pipeline generator - Updated the application class documentation to provide clearer descriptions of the vision-language model (VLM) functionality, including details on prompt processing and output generation. - Refined the version constraints for the pydicom dependency in the requirements template to ensure compatibility with future updates while maintaining support for existing features. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor image loading operators and enhance directory scanning functionality - Replaced ImageDirectoryLoader with a new GenericDirectoryScanner and ImageFileLoader for improved flexibility in file handling. - Updated operator imports in the application template to reflect the new structure. - Added unit tests for the GenericDirectoryScanner to ensure correct functionality, including edge cases for file detection and filtering. - Removed deprecated NiftiDirectoryLoader and ImageDirectoryLoader to streamline the codebase. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor operator imports and enhance pipeline generator functionality - Removed deprecated operators from the MONAI Deploy SDK and updated the import paths in the application template to reflect the new structure. - Introduced new operators such as GenericDirectoryScanner and ImageFileLoader for improved file handling. - Enhanced the NiftiDataLoader to handle various dimensionalities correctly and added logging for unexpected shapes. - Updated the pipeline generator to include new operators and refined the requirements for dependencies in the configuration files. - Added comprehensive tests for the new operators and updated existing tests to ensure functionality and correctness. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor operator imports and enhance code clarity - Updated operator imports in the MONAI Deploy SDK to streamline the structure and improve readability. - Refined the NiftiDataLoader to ensure proper handling of various dimensionalities and added logging for unexpected shapes. - Enhanced the pipeline generator to include new operators and improved the handling of output types. - Cleaned up whitespace and formatting inconsistencies across multiple files for better code clarity. - Removed deprecated test files related to GenericDirectoryScanner and VLM operators to maintain a clean codebase. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Fix formatting inconsistencies and improve error message clarity - Adjusted whitespace in print statements for better readability in the DICOM data loader operator. - Enhanced the error message in the AppGenerator class to improve clarity regarding valid model_id formats. - Corrected spacing in the file processing output of the generic directory scanner operator. - Removed unused import from the test_run_command module to clean up the code. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Remove deprecated test file for the pipeline generator - Deleted the __init__.py test file from the pipeline-generator tests directory to clean up the codebase and remove unused files. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance type hinting and improve code clarity across operators - Added type hints for lists and dictionaries in various operator classes to improve code readability and maintainability. - Updated import statements to include type ignoring for YAML to prevent type checking issues. - Refined the initialization of file lists in the GenericDirectoryScanner operator for better type safety. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance pipeline generator functionality and improve bundle organization - Updated the CLI output to reflect "Verified" models instead of "Tested" for better clarity. - Added new model configurations for pancreas segmentation and spleen segmentation in the config file. - Implemented a method to organize downloaded bundle structures into the standard MONAI format, improving file management. - Enhanced dependency handling in the AppGenerator to resolve conflicts between configuration and metadata. - Added unit tests to verify the new bundle organization functionality and ensure correct behavior under various scenarios. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance bundle organization and improve model handling in pipeline generator - Added functionality to ensure the bundle root is included in sys.path for script imports. - Introduced a new model configuration for pediatric abdominal CT segmentation in the config file. - Improved the organization of model files by preferring PyTorch models over TensorRT models and handling subdirectory structures. - Enhanced unit tests to verify the new model organization logic and ensure correct behavior under various scenarios. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor whitespace and improve code clarity in pipeline generator - Removed unnecessary whitespace in the app_generator and bundle_downloader files to enhance readability. - Streamlined import statements in test files for better organization. - Ensured consistent formatting across various sections of the codebase. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance model handling and CLI functionality in pipeline generator - Added support for listing all models regardless of file extension with the `--all` flag in the CLI. - Updated the model listing logic to differentiate between verified and non-verified models. - Enhanced the model information structure to include detected file extensions and primary extension for better clarity. - Improved documentation to reflect the new model handling capabilities and supported file formats. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Fix formatting Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor model retrieval logic in HuggingFaceClient - Updated variable name from 'model' to 'model_info' for clarity in the model retrieval process. - Ensured consistent naming conventions to enhance code readability. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance model loading and configuration handling in MonaiBundleInferenceOperator - Introduced helper functions to streamline model loading from directory-based bundles and ensure proper configuration parsing. - Improved error handling for missing model files and configuration metadata. - Refactored the get_bundle_config function to utilize the new helper for better code organization and readability. - Updated the NiftiDataLoader to handle image dimensionalities more effectively using SimpleITK metadata. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor whitespace and improve code clarity in MonaiBundleInferenceOperator and NiftiDataLoader - Removed unnecessary whitespace in the MonaiBundleInferenceOperator to enhance readability. - Streamlined function definitions and logging statements in NiftiDataLoader for better organization and clarity. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Improve error handling in model loading for MonaiBundleInferenceOperator - Enhanced exception handling by capturing the original exception in model loading functions to provide clearer error messages. - Updated the handling of network instantiation errors to include the original exception context for better debugging. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance type hinting in MonaiBundleInferenceOperator - Added TYPE_CHECKING imports for torch to improve type hinting and static analysis. - Utilized type casting for model loading return values to ensure proper type safety and clarity. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor type hinting in MonaiBundleInferenceOperator - Updated type hints for device and return values in model loading functions to use the alias `torch_typing` for improved clarity and consistency. - Enhanced type safety by ensuring all model-related return types are correctly annotated. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Refactor MonaiBundleInferenceOperator for improved path handling and image conversion - Updated bundle path handling to use `Path` objects for consistency and clarity. - Simplified image conversion logic by consolidating conditional checks into a single line for better readability. - Ensured that directory checks and model loading operations utilize the updated path handling. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Enhance image conversion logic in MonaiBundleInferenceOperator - Improved handling of different dimensional outputs during image conversion to maintain DICOM pixel data arrangement. - Added specific cases for 2D and 3D images to ensure correct axis handling and data type conversion. Signed-off-by: Victor Chang <vicchang@nvidia.com> * Pin monai<=1.5.0 since newly released 1.5.1 introduced breaking changes Signed-off-by: M Q <mingmelvinq@nvidia.com> * Was wondering but did not expect the test is asseting on the value in actual code Signed-off-by: M Q <mingmelvinq@nvidia.com> * Use bundle_path property for consistency, and added warning note on nii loader Signed-off-by: M Q <mingmelvinq@nvidia.com> --------- Signed-off-by: Victor Chang <vicchang@nvidia.com> Signed-off-by: [Your Name] <your.email@example.com> Signed-off-by: M Q <mingmelvinq@nvidia.com> Co-authored-by: M Q <mingmelvinq@nvidia.com>
1 parent 4d3e1ab commit d618e6d

45 files changed

Lines changed: 10162 additions & 13 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/pr.yml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,22 @@ jobs:
5555
with:
5656
fail_ci_if_error: false
5757
files: ./coverage.xml
58+
59+
test-pipeline-generator:
60+
runs-on: ubuntu-latest
61+
steps:
62+
- uses: actions/checkout@v2
63+
- name: Set up Python 3.10
64+
uses: actions/setup-python@v2
65+
with:
66+
python-version: "3.10"
67+
- name: Install uv
68+
uses: astral-sh/setup-uv@v6
69+
- name: Install dependencies
70+
working-directory: tools/pipeline-generator
71+
run: |
72+
uv sync
73+
- name: Run tests
74+
working-directory: tools/pipeline-generator
75+
run: |
76+
uv run pytest

monai/deploy/operators/monai_bundle_inference_operator.py

Lines changed: 320 additions & 12 deletions
Large diffs are not rendered by default.

monai/deploy/operators/nii_data_loader_operator.py

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,13 @@
2323

2424
# @md.env(pip_packages=["SimpleITK>=2.0.2"])
2525
class NiftiDataLoader(Operator):
26+
# NOTE:
27+
# This loader has very limited use because it does not capture or embed the metadata of a NIfTI file.
28+
# Future improvements are needed to retrieve the metadata and return an Image object with the metadata,
29+
# or use monai image transform to load the image and return a meta tensor.
30+
# Also SITK GetArrayFromImage() returns an NDArray, and the newly added transpose here is to change the
31+
# index ordering to match the index ordering of the DCOM pixel array. Existing client of this operator,
32+
# if any, will likely have issues.
2633
"""
2734
This operator reads a nifti image, extracts the numpy array and forwards it to the next operator
2835
@@ -80,7 +87,40 @@ def convert_and_save(self, nii_path):
8087
image_reader = SimpleITK.ImageFileReader()
8188
image_reader.SetFileName(str(nii_path))
8289
image = image_reader.Execute()
83-
image_np = np.transpose(SimpleITK.GetArrayFromImage(image), [2, 1, 0])
90+
image_np = SimpleITK.GetArrayFromImage(image)
91+
92+
# Get image metadata to properly distinguish between different image types
93+
spatial_dims = image.GetDimension() # Actual spatial dimensions (2D, 3D, etc.)
94+
num_components = image.GetNumberOfComponentsPerPixel() # Components/channels per pixel
95+
96+
self._logger.debug(
97+
f"Image spatial dimensions: {spatial_dims}, components per pixel: {num_components}, array shape: {image_np.shape}"
98+
)
99+
100+
# Handle different dimensionalities properly using SimpleITK metadata
101+
if spatial_dims == 2:
102+
if num_components == 1:
103+
# 2D grayscale: transpose from (y, x) to (x, y)
104+
image_np = np.transpose(image_np, [1, 0])
105+
else:
106+
# 2D with multiple components/channels: transpose from (y, x, c) to (x, y, c)
107+
# SimpleITK stores multi-component 2D images as (y, x, c)
108+
image_np = np.transpose(image_np, [1, 0, 2])
109+
elif spatial_dims == 3:
110+
if num_components == 1:
111+
# 3D grayscale volume: transpose from (z, y, x) to (x, y, z)
112+
image_np = np.transpose(image_np, [2, 1, 0])
113+
else:
114+
# 3D volume with multiple components: transpose from (z, y, x, c) to (x, y, z, c)
115+
# SimpleITK stores multi-component 3D images as (z, y, x, c)
116+
image_np = np.transpose(image_np, [2, 1, 0, 3])
117+
else:
118+
# For other spatial dimensions, log a warning and return as-is
119+
self._logger.warning(
120+
f"Unexpected {spatial_dims}D spatial image with {num_components} components per pixel, "
121+
f"array shape {image_np.shape} from {nii_path}, returning without transpose"
122+
)
123+
84124
return image_np
85125

86126

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
results*/
2+
test_*/

tools/pipeline-generator/README.md

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# Pipeline Generator
2+
3+
A CLI tool for generating [MONAI Deploy](https://github.com/Project-MONAI/monai-deploy-app-sdk) application pipelines from [MONAI Bundles](https://docs.monai.io/en/stable/bundle_intro.html).
4+
5+
## Features
6+
7+
- List available MONAI models from HuggingFace
8+
- Generate complete MONAI Deploy applications from HuggingFace models
9+
- Support for multiple model sources through configuration
10+
- Automatic bundle download and analysis
11+
- Template-based code generation with Jinja2
12+
- Beautiful output formatting with Rich (Python library for rich text and beautiful formatting)
13+
14+
## Platform Requirements
15+
16+
- **Linux/Unix operating systems only**
17+
- Compatible with MONAI Deploy App SDK platform support
18+
- Ubuntu 22.04+ recommended (aligned with main SDK requirements)
19+
20+
## Installation
21+
22+
```bash
23+
# Clone the repository
24+
cd tools/pipeline-generator/
25+
26+
# Install with uv (no virtualenv needed - uv manages it per command)
27+
uv pip install -e ".[dev]"
28+
```
29+
30+
### Running Commands
31+
32+
With uv, you can run commands directly without a prior "install" (pg is the Pipeline Generator command):
33+
34+
```bash
35+
uv run pg --help
36+
uv run pg list
37+
uv run pg gen MONAI/spleen_ct_segmentation --output ./app
38+
```
39+
40+
## Usage
41+
42+
### Complete Workflow Example
43+
44+
```bash
45+
# 1. List available models
46+
uv run pg list
47+
48+
# 2. Generate an application from a model
49+
uv run pg gen MONAI/spleen_ct_segmentation --output my_app
50+
51+
# 3. Run the application
52+
uv run pg run my_app --input /path/to/test/data --output ./results
53+
```
54+
55+
> [!NOTE]
56+
> Input can be NIfTI files (.nii, .nii.gz) or DICOM series directories Refer to [config.yaml](tools/pipeline-generator/pipeline_generator/config/config.yaml) for details.
57+
58+
> [!NOTE]
59+
> For DICOM input types, refer to the model documentation for DICOM series processing limitations.
60+
61+
### List Available Models
62+
63+
List all models from configured endpoints:
64+
65+
```bash
66+
uv run pg list
67+
```
68+
69+
Show only MONAI Bundles:
70+
71+
```bash
72+
uv run pg list --bundles-only
73+
```
74+
75+
Show only tested models:
76+
77+
```bash
78+
uv run pg list --tested-only
79+
```
80+
81+
Combine filters:
82+
83+
```bash
84+
uv run pg list --bundles-only --tested-only # Show only tested MONAI Bundles
85+
```
86+
87+
Use different output formats:
88+
89+
```bash
90+
uv run pg list --format simple # Simple list format
91+
uv run pg list --format json # JSON output
92+
uv run pg list --format table # Default table format
93+
```
94+
95+
Use a custom configuration file:
96+
97+
```bash
98+
uv run pg --config /path/to/config.yaml list
99+
```
100+
101+
### Generate MONAI Deploy Application
102+
103+
Generate an application from a HuggingFace model. Models are specified using the format `organization/model_name` (e.g., `MONAI/spleen_ct_segmentation`):
104+
105+
```bash
106+
uv run pg gen MONAI/spleen_ct_segmentation --output my_app
107+
```
108+
109+
Options:
110+
111+
- `--output, -o`: Output directory for generated app (default: ./output)
112+
- `--app-name, -n`: Custom application class name (default: derived from model name)
113+
- `--format`: Input/output data format (optional): auto, dicom, or nifti (default: auto)
114+
- For tested models, format is automatically detected from configuration
115+
- For untested models, attempts detection from model metadata
116+
- **DICOM Limitation**: Refer to the model documentation for multi-series support.
117+
- `--force, -f`: Overwrite existing output directory
118+
119+
Generate with custom application class name:
120+
121+
```bash
122+
uv run pg gen MONAI/lung_nodule_ct_detection --output lung_app --app-name LungDetectorApp
123+
```
124+
125+
Force overwrite existing directory:
126+
127+
```bash
128+
uv run pg gen MONAI/spleen_ct_segmentation --output test_app --force
129+
```
130+
131+
Override data format (optional - auto-detected for tested models):
132+
133+
```bash
134+
# Force DICOM format instead of auto-detection
135+
uv run pg gen MONAI/some_model --output my_app --format dicom
136+
```
137+
138+
### Run Generated Application
139+
140+
Run a generated application with automatic environment setup:
141+
142+
```bash
143+
uv run pg run my_app --input /path/to/input --output /path/to/output
144+
```
145+
146+
The `run` command will:
147+
148+
1. Create a virtual environment if it doesn't exist
149+
1. Install dependencies from requirements.txt
150+
1. Run the application with the specified input/output
151+
152+
Options:
153+
154+
- `--input, -i`: Input data directory (required)
155+
- `--output, -o`: Output directory for results (default: ./output)
156+
- `--model, -m`: Override model/bundle path
157+
- `--venv-name`: Virtual environment directory name (default: .venv)
158+
- `--skip-install`: Skip dependency installation
159+
- `--gpu/--no-gpu`: Enable/disable GPU support (default: enabled)
160+
161+
Examples:
162+
163+
```bash
164+
# Skip dependency installation (if already installed)
165+
uv run pg run my_app --input test_data --output results --skip-install
166+
167+
# Run without GPU
168+
uv run pg run my_app --input test_data --output results --no-gpu
169+
170+
# Use custom model path
171+
uv run pg run my_app --input test_data --output results --model ./custom_model
172+
```
173+
174+
## Configuration
175+
176+
The tool uses a YAML configuration file to define model sources. By default, it looks for `config.yaml` in the package directory.
177+
178+
Example configuration:
179+
180+
```yaml
181+
# HuggingFace endpoints to scan for MONAI models
182+
endpoints:
183+
- organization: "MONAI"
184+
base_url: "https://huggingface.co"
185+
description: "Official MONAI organization models"
186+
187+
# Additional specific models not under the main organization
188+
additional_models:
189+
- model_id: "Project-MONAI/exaonepath"
190+
base_url: "https://huggingface.co"
191+
description: "ExaOnePath model for digital pathology"
192+
```
193+
194+
## Generated Application Structure
195+
196+
When you run `pg gen`, it creates:
197+
198+
```
199+
output/
200+
├── app.py # Main application code
201+
├── app.yaml # Configuration for MONAI Deploy packaging
202+
├── requirements.txt # Python dependencies
203+
├── README.md # Documentation
204+
├── operators/ # Custom operators (if needed)
205+
│ └── nifti_operators.py
206+
└── model/ # Downloaded model files
207+
├── configs/
208+
├── models/
209+
└── docs/
210+
```
211+
212+
## Development
213+
214+
### Running Tests
215+
216+
```bash
217+
# Run all tests
218+
uv run pytest
219+
220+
# Run with coverage
221+
uv run pytest --cov=pipeline_generator
222+
223+
# Run specific test file
224+
uv run pytest tests/test_cli.py
225+
```
226+
227+
### Code Quality
228+
229+
```bash
230+
# Format code
231+
uv run black pipeline_generator tests
232+
233+
# Lint code
234+
uv run flake8 pipeline_generator tests
235+
236+
# Type checking
237+
uv run mypy pipeline_generator
238+
```
239+
240+
## Future Commands
241+
242+
The CLI is designed to be extensible. Planned commands include:
243+
244+
- `pg package <app>` - Package an application using the Holoscan CLI packaging tool
245+
246+
## License
247+
248+
This project is part of the MONAI Deploy App SDK and is licensed under the Apache License 2.0. See the main repository's LICENSE file for details.

0 commit comments

Comments
 (0)