feat: Add multi-provider instance support for LLM configurations

- Support multiple instances of same provider type (e.g., ollama_qwen3, ollama_deepseek)
- Add provider type extraction from config or name prefix
- Enhance provider validation with validate_provider_name() method
- Add comprehensive multi-provider-guide.md documentation
- Update configuration guide with multi-provider examples
- Improve LLM provider manager initialization logic
- Add quality score format option for automation
- Add test sets and software lists for LLM comparison
- Update schemas and CLI commands for better provider handling

Author: alvagante

.gitignore

@@ -198,4 +198,5 @@ logs/
 .coverage
 htmlcov/
 .pytest_cache/
-test-results/
+test-results/
+llm-comparison*

CHANGELOG.md

@@ -8,6 +8,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- **Multi-Provider Instance Support**: Configure multiple instances of the same LLM provider type
+  - Support for multiple Ollama models with unique names (e.g., `ollama_qwen3`, `ollama_deepseek`)
+  - Support for multiple OpenAI endpoints (official, Azure, local)
+  - New `provider` field in configuration to explicitly specify provider type
+  - Provider type extraction from name prefix as fallback (e.g., `ollama_qwen3` → `ollama`)
+  - Enhanced provider validation with `validate_provider_name()` method
+  - New `extract_provider_type()` method for flexible provider type detection
+  - Comprehensive multi-provider guide documentation
+  - Support for model comparison workflows and A/B testing
+- **Quality Command Score Format**: New `--format score` option for `saigen quality` command
+  - Returns just the numeric quality score (0.000-1.000) without additional text
+  - Useful for automation, scripting, and CI/CD pipelines
+  - Suppresses progress messages when using score format
+  - Works with both overall score and specific metric scores
 - **API-Based Repository Support**: Complete implementation of API-based repository downloaders
   - New `ApiDownloader` class for fetching package data from REST APIs
   - Support for API-based repositories (Docker Hub, Hashicorp, etc.)
@@ -186,6 +200,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Security Enhancements**: File size limits for provider YAML files to prevent DoS attacks
 
 ### Changed
+- **LLM Provider Manager**: Enhanced to support multiple instances of the same provider type
+  - Provider initialization now extracts base type from configuration or name
+  - Improved error messages showing both provider name and type
+  - Better handling of provider-specific configurations
+  - Support for both dict and Pydantic model configs
+- **Configuration Documentation**: Enhanced configuration guide with multi-provider examples
+  - Added comprehensive examples for multiple Ollama models
+  - Added examples for multiple OpenAI endpoints
+  - Documented naming conventions and best practices
+  - Added troubleshooting section for common issues
 - **Repository Configuration Architecture**: Major restructuring of repository configuration system
   - Migrated from monolithic YAML files to individual provider configs
   - Enhanced schema with API endpoint and authentication support

README.md

@@ -481,6 +481,17 @@ llm_providers:
     provider: anthropic
     model: claude-3-sonnet-20240229
     enabled: false
+  # Multiple instances of the same provider type are supported
+  ollama_qwen3:
+    provider: ollama
+    api_base: http://localhost:11434
+    model: qwen3-coder:30b
+    enabled: true
+  ollama_deepseek:
+    provider: ollama
+    api_base: http://localhost:11434
+    model: deepseek-r1:8b
+    enabled: true
 
 repositories:
   apt:

docs/summaries/multi-provider-support.md

@@ -0,0 +1,129 @@
+# Multi-Provider Instance Support Implementation
+
+## Summary
+
+Implemented support for multiple instances of the same LLM provider type (e.g., multiple Ollama models) with different configurations. Users can now configure multiple providers of the same type by using unique names in their configuration.
+
+## Problem
+
+Previously, the system only supported one instance per provider type (openai, anthropic, ollama, vllm). Users couldn't configure multiple Ollama models or multiple OpenAI endpoints with different settings. The code tried to convert provider names directly to the `LLMProvider` enum, which failed for names like `ollama_qwen3`.
+
+## Solution
+
+### 1. Provider Type Extraction
+
+Added logic to extract the base provider type from provider names or configuration:
+- Provider names like `ollama_qwen3` are split to extract `ollama` as the type
+- The `provider` field in configuration explicitly specifies the type
+- New helper methods in `LLMProviderManager`:
+  - `extract_provider_type()`: Extracts base provider type from name/config
+  - `validate_provider_name()`: Validates provider names against configuration
+
+### 2. Provider Name Handling
+
+Updated all CLI commands to use provider names as strings instead of converting to enum:
+- `batch.py`: Validates provider names against config keys
+- `generate.py`: Uses provider names directly
+- `update.py`: Uses provider names directly
+- Better error messages showing available configured providers
+
+### 3. Model Updates
+
+Changed `GenerationRequest` and `BatchGenerationRequest` models:
+- `llm_provider` field changed from `LLMProvider` enum to `str`
+- Maintains backward compatibility with existing code
+- Generation engine already handled both string and enum values
+
+### 4. Validation Improvements
+
+Updated configuration validation:
+- Ollama and vLLM providers no longer require API keys
+- Validation checks provider existence in configuration
+- Clear error messages with list of available providers
+
+### 5. Documentation Updates
+
+Updated documentation to explain multi-provider support:
+- `README.md`: Added example with multiple Ollama instances
+- `saigen/docs/configuration-guide.md`: Added dedicated section on multi-provider instances
+- `saigen/docs/examples/saigen-config-sample.yaml`: Updated with multiple Ollama examples
+- CLI help text updated to reflect provider name usage
+
+## Configuration Example
+
+```yaml
+llm_providers:
+  openai:
+    provider: openai
+    model: gpt-4o-mini
+    enabled: true
+  
+  # Multiple Ollama instances with different models
+  ollama_qwen3:
+    provider: ollama
+    api_base: http://localhost:11434
+    model: qwen3-coder:30b
+    enabled: true
+  
+  ollama_deepseek:
+    provider: ollama
+    api_base: http://localhost:11434
+    model: deepseek-r1:8b
+    enabled: true
+  
+  ollama_phi3:
+    provider: ollama
+    api_base: http://localhost:11434
+    model: phi3:latest
+    enabled: true
+```
+
+## Usage
+
+```bash
+# Use specific provider
+saigen generate nginx --llm-provider ollama_qwen3
+
+# Use different provider
+saigen batch software-list.txt --llm-provider ollama_deepseek
+
+# Use default (first enabled provider)
+saigen generate nginx
+```
+
+## Files Modified
+
+### Core Implementation
+- `saigen/llm/provider_manager.py`: Added provider type extraction logic
+- `saigen/models/generation.py`: Changed llm_provider field to string
+- `saigen/core/update_engine.py`: Updated method signature
+
+### CLI Commands
+- `saigen/cli/main.py`: Updated help text
+- `saigen/cli/commands/batch.py`: Provider name validation
+- `saigen/cli/commands/generate.py`: Provider name validation
+- `saigen/cli/commands/update.py`: Provider name validation
+
+### Configuration & Validation
+- `saigen/utils/config.py`: Skip API key validation for ollama/vllm
+
+### Documentation
+- `README.md`: Added multi-provider example
+- `saigen/docs/configuration-guide.md`: Added multi-provider section
+- `saigen/docs/examples/saigen-config-sample.yaml`: Updated examples
+
+## Testing
+
+All existing tests pass. The implementation:
+- Maintains backward compatibility with existing configurations
+- Handles both string and enum provider values in generation engine
+- Validates provider names against actual configuration
+- Provides clear error messages when invalid providers are specified
+
+## Benefits
+
+1. **Flexibility**: Users can configure multiple models of the same provider type
+2. **Model Comparison**: Easy to compare different models by switching providers
+3. **Resource Management**: Different models can use different endpoints/servers
+4. **Clear Naming**: Descriptive names like `ollama_qwen3` make it obvious which model is being used
+5. **Backward Compatible**: Existing single-provider configurations continue to work

sai/core/saidata_loader.py

@@ -947,6 +947,7 @@ def _validate_services(
                 "windows_service",
                 "docker",
                 "kubernetes",
+                "none"
             ]:
                 errors.append(f"Service {i} has invalid type: {service_type}")
 

saigen/cli/commands/batch.py

@@ -106,10 +106,14 @@ def batch(
     # Use global LLM provider if specified, otherwise use config default
     llm_provider_name = ctx.obj["llm_provider"]
     if llm_provider_name:
-        try:
-            llm_provider = LLMProvider(llm_provider_name)
-        except ValueError:
-            raise click.BadParameter(f"Invalid LLM provider: {llm_provider_name}")
+        # Validate that the provider exists in config
+        if not config.llm_providers or llm_provider_name not in config.llm_providers:
+            available = list(config.llm_providers.keys()) if config.llm_providers else []
+            raise click.BadParameter(
+                f"Invalid LLM provider: {llm_provider_name}. "
+                f"Available providers: {', '.join(available) if available else 'none configured'}"
+            )
+        llm_provider = llm_provider_name
     else:
         # Use default from config or fallback
         if hasattr(config, "llm_providers") and config.llm_providers:
@@ -124,12 +128,9 @@ def batch(
                 # No enabled providers, use first one anyway
                 first_provider = next(iter(config.llm_providers.keys()), "openai")
 
-            try:
-                llm_provider = LLMProvider(first_provider)
-            except ValueError:
-                llm_provider = LLMProvider.OPENAI  # Fallback
+            llm_provider = first_provider
         else:
-            llm_provider = LLMProvider.OPENAI  # Fallback
+            llm_provider = "openai"  # Fallback
 
     # Validate concurrency
     if max_concurrent < 1 or max_concurrent > 20:

saigen/cli/commands/generate.py

@@ -200,43 +200,42 @@ def generate(
 
         # Determine LLM provider
         if llm_provider:
-            try:
-                provider_enum = LLMProvider(llm_provider)
-            except ValueError:
-                error_msg = f"Invalid LLM provider '{llm_provider}'. Available: {
-                    [
-                        p.value for p in LLMProvider]}"
+            # Validate that the provider exists in config
+            if not config.llm_providers or llm_provider not in config.llm_providers:
+                available = list(config.llm_providers.keys()) if config.llm_providers else []
+                error_msg = (
+                    f"Invalid LLM provider '{llm_provider}'. "
+                    f"Available providers: {', '.join(available) if available else 'none configured'}"
+                )
                 if generation_logger:
                     generation_logger.log_error(error_msg)
                 click.echo(f"Error: {error_msg}", err=True)
                 ctx.exit(1)
+            provider_name = llm_provider
         else:
             # Use default from config or fallback
             if hasattr(config, "llm_providers") and config.llm_providers:
                 # Get first enabled provider from config
                 first_provider = None
-                for provider_name, provider_config in config.llm_providers.items():
+                for prov_name, provider_config in config.llm_providers.items():
                     if provider_config.enabled:
-                        first_provider = provider_name
+                        first_provider = prov_name
                         break
 
                 if not first_provider:
                     # No enabled providers, use first one anyway
                     first_provider = next(iter(config.llm_providers.keys()), "openai")
 
-                try:
-                    provider_enum = LLMProvider(first_provider)
-                except ValueError:
-                    provider_enum = LLMProvider.OPENAI  # Fallback
+                provider_name = first_provider
             else:
-                provider_enum = LLMProvider.OPENAI  # Fallback
+                provider_name = "openai"  # Fallback
 
         # Create generation request
         target_providers_list = list(providers) if providers else []
         request = GenerationRequest(
             software_name=software_name,
             target_providers=target_providers_list,
-            llm_provider=provider_enum,
+            llm_provider=provider_name,
             use_rag=not no_rag,
             user_hints=None,
             existing_saidata=None,
@@ -249,7 +248,7 @@ def generate(
         if verbose:
             click.echo(f"Generating saidata for: {software_name}")
             click.echo(f"Schema version: 0.3")
-            click.echo(f"LLM Provider: {provider_enum.value}")
+            click.echo(f"LLM Provider: {provider_name}")
             click.echo(f"Target providers: {request.target_providers}")
             click.echo(f"RAG enabled: {request.use_rag}")
             click.echo(f"Output file: {output}")

saigen/cli/commands/quality.py

@@ -31,9 +31,9 @@
 @click.option(
     "--format",
     "output_format",
-    type=click.Choice(["text", "json", "csv"]),
+    type=click.Choice(["text", "json", "csv", "score"]),
     default="text",
-    help="Output format",
+    help="Output format (score: just the numeric value)",
 )
 @click.option("--export", type=click.Path(path_type=Path), help="Export detailed report to file")
 def quality(
@@ -121,17 +121,20 @@ async def _run_quality_assessment(
             repository_manager = RepositoryManager(cache_dir, config_dir)
             await repository_manager.initialize()
 
-            click.echo("🔍 Initializing repository data...", err=True)
+            if output_format != "score":
+                click.echo("🔍 Initializing repository data...", err=True)
         except Exception as e:
-            click.echo(f"⚠️  Repository checking disabled: {e}", err=True)
+            if output_format != "score":
+                click.echo(f"⚠️  Repository checking disabled: {e}", err=True)
             check_repository_accuracy = False
 
     # Create advanced validator
     base_validator = SaidataValidator()
     advanced_validator = AdvancedSaidataValidator(repository_manager, base_validator)
 
     # Run quality assessment
-    click.echo("📊 Assessing quality metrics...", err=True)
+    if output_format != "score":
+        click.echo("📊 Assessing quality metrics...", err=True)
     quality_report = await advanced_validator.validate_comprehensive(
         saidata, check_repository_accuracy
     )
@@ -152,6 +155,9 @@ async def _run_quality_assessment(
     elif output_format == "csv":
         output = _format_csv_output(quality_report, file_path)
         result_text = output
+    elif output_format == "score":
+        output = _format_score_output(quality_report, metric_filter)
+        result_text = output
     else:
         output = _format_text_output(quality_report, file_path, threshold, metric_filter)
         result_text = output
@@ -168,7 +174,8 @@ async def _run_quality_assessment(
                 json.dump(json.loads(result_text), f, indent=2)
             else:
                 f.write(result_text)
-        click.echo(f"📄 Report exported to {export_path}", err=True)
+        if output_format != "score":
+            click.echo(f"📄 Report exported to {export_path}", err=True)
 
     # Clean up
     if repository_manager:
@@ -232,6 +239,18 @@ def _format_csv_output(quality_report, file_path: Path) -> str:
     return "\n".join(lines)
 
 
+def _format_score_output(quality_report, metric_filter: Optional[str]) -> str:
+    """Format quality report as just the score value."""
+    if metric_filter:
+        # If filtering by specific metric, show that metric's score
+        metric_enum = QualityMetric(metric_filter)
+        score = quality_report.metric_scores[metric_enum]
+        return f"{score.score:.3f}"
+    else:
+        # Show overall score
+        return f"{quality_report.overall_score:.3f}"
+
+
 def _format_text_output(
     quality_report, file_path: Path, threshold: float, metric_filter: Optional[str]
 ) -> str: