feat(build): per-app python_version with cross-resource validation (#322)

* feat(build): per-app python_version with cross-resource validation (AE-2827)

Expand GPU and CPU worker images to support Python 3.10, 3.11, and 3.12.
Flash apps ship as one tarball so every resource must share a Python
version; the build step now reconciles per-resource python_version
declarations into a single app-level value or accepts an explicit
--python-version override.

- constants.py: expand GPU_PYTHON_VERSIONS / CPU_PYTHON_VERSIONS tuples
- manifest.py: add _reconcile_python_version; stamp target_python_version
  on every resource; raise on conflicting declarations
- build.py / deploy.py: add --python-version CLI flag, thread through
  run_build and ManifestBuilder
- docs: document per-app python_version, cold-start tradeoff for non-3.12
  GPU images, and the 3.10 EOL window
- tests/unit/test_dotenv_loading.py: add preserve_runpod_flash_modules
  fixture so module-deletion tests don't leak stale module references
  into sibling test files (unblocks deterministic test ordering)

* refactor(manifest): drop dead python_version assignment in __init__

The effective app-level Python version is assigned by build() via
_reconcile_python_version(). Pre-build access always saw a value that
build() would overwrite — misleading and never read. Initialize to None
so readers fail fast if they touch the attribute before build().

* docs(flash-build): drop stale --keep-build and --preview refs

Neither flag is exposed by flash/cli/commands/build.py::build_command;
--preview now lives on flash deploy. Aligns the docs with actual CLI
behavior — build directory is always retained on success, so the
--keep-build toggle is gone.

Caught by Copilot review on PR #322.

Author: deanq

docs/Deployment_Architecture.md

@@ -31,7 +31,7 @@ flash build
     │
     ├── 4. Dependency Installation
     │   ├── Install Python packages for linux/x86_64
-    │   ├── Target Python 3.12 for wheel ABI selection
+    │   ├── Target the app's python_version for wheel ABI selection (3.10 / 3.11 / 3.12)
     │   └── Binary wheels only (no compilation)
     │
     └── 5. Packaging

docs/Flash_Deploy_Guide.md

@@ -6,10 +6,23 @@ This guide walks through deploying a Flash application from local development to
 
 ## Prerequisites
 
-- Python 3.10+
+- Python 3.10, 3.11, or 3.12
 - `pip install runpod-flash`
 - A Runpod account with API key ([get one here](https://docs.runpod.io/get-started/api-keys))
 
+### Python version selection
+
+Flash apps ship as a single tarball, so every resource in an app shares one Python version. The worker runtime defaults to 3.12 (the version torch is pre-installed for in the GPU base image). Select a different version in two ways:
+
+- **Per-resource declaration**: set `python_version="3.11"` on any resource config — all resources in the same app must agree or leave it unset.
+- **App-level override**: pass `--python-version 3.11` to `flash build` or `flash deploy`. The override wins over per-resource values that are unset and must match any that are set.
+
+| Version | Status | GPU cold-start | Notes |
+|---------|--------|----------------|-------|
+| 3.10 | Supported (EOL 2026-10-31) | +~7 GB alt-Python install | Consider migrating to 3.11 before EOL |
+| 3.11 | Supported | +~7 GB alt-Python install | |
+| 3.12 | Supported (default) | No overhead | Torch pre-installed in base image |
+
 ## Quick Start
 
 ```bash

src/runpod_flash/cli/commands/build.py

@@ -22,10 +22,7 @@
     import tomli as tomllib  # Python 3.10
 
 from runpod_flash.cli.utils.formatting import print_error, print_warning
-from runpod_flash.core.resources.constants import (
-    DEFAULT_PYTHON_VERSION,
-    MAX_TARBALL_SIZE_MB,
-)
+from runpod_flash.core.resources.constants import MAX_TARBALL_SIZE_MB
 
 from ..utils.ignore import get_file_tree, load_ignore_patterns
 from .build_utils.handler_generator import HandlerGenerator
@@ -273,6 +270,7 @@ def run_build(
     output_name: str | None = None,
     exclude: str | None = None,
     verbose: bool = False,
+    python_version: str | None = None,
 ) -> Path:
     """Run the build process and return the artifact path.
 
@@ -287,6 +285,10 @@ def run_build(
         output_name: Custom archive name (default: artifact.tar.gz)
         exclude: Comma-separated packages to exclude
         verbose: Show archive and build directory paths in summary
+        python_version: Optional app-level Python version override. When None,
+            inferred from resource configs (defaulting to DEFAULT_PYTHON_VERSION
+            if none declare one). One tarball serves every resource in an app,
+            so all resources must agree on one version.
 
     Returns:
         Path to the created artifact archive
@@ -311,10 +313,9 @@ def run_build(
     spec = load_ignore_patterns(project_dir)
     files = get_file_tree(project_dir, spec)
 
-    # all packaging and image selection targets 3.12 regardless of local python.
-    # pip downloads wheels for 3.12 via --python-version, and all worker images
-    # run 3.12, so the local interpreter version does not affect the build output.
-    python_version = DEFAULT_PYTHON_VERSION
+    # Resolved later by ManifestBuilder from resource configs (or the override
+    # above). Pip wheel selection re-reads this via _resolve_pip_python_version.
+    manifest_python_version_override = python_version
 
     try:
         copy_project_files(files, project_dir, build_dir)
@@ -335,7 +336,7 @@ def run_build(
                 remote_functions,
                 scanner,
                 build_dir=build_dir,
-                python_version=python_version,
+                python_version=manifest_python_version_override,
             )
             manifest = manifest_builder.build()
             manifest["source_fingerprint"] = compute_source_fingerprint(
@@ -513,6 +514,15 @@ def build_command(
         "--exclude",
         help="Comma-separated additional packages to exclude (torch packages are auto-excluded)",
     ),
+    python_version: str | None = typer.Option(
+        None,
+        "--python-version",
+        help=(
+            "Target Python version for worker images (3.10, 3.11, or 3.12). "
+            "Overrides per-resource python_version declarations. "
+            "Defaults to the version declared on resource configs, or 3.12 if none set."
+        ),
+    ),
 ):
     """
     Build Flash application for debugging (build only, no deploy).
@@ -525,6 +535,7 @@ def build_command(
       flash build --no-deps                    # Skip transitive dependencies
       flash build -o my-app.tar.gz             # Custom archive name
       flash build --exclude transformers       # Exclude additional large packages
+      flash build --python-version 3.11        # Target Python 3.11 workers
     """
     try:
         project_dir, app_name = discover_flash_project()
@@ -536,6 +547,7 @@ def build_command(
             output_name=output_name,
             exclude=exclude,
             verbose=True,
+            python_version=python_version,
         )
 
     except KeyboardInterrupt:

src/runpod_flash/cli/commands/build_utils/manifest.py

@@ -11,7 +11,7 @@
 
 from runpod_flash.core.resources.constants import (
     DEFAULT_PYTHON_VERSION,
-    GPU_BASE_IMAGE_PYTHON_VERSION,
+    validate_python_version,
 )
 
 from .scanner import (
@@ -93,7 +93,10 @@ def __init__(
         self.remote_functions = remote_functions
         self.scanner = scanner  # Optional: RuntimeScanner with resource config info
         self.build_dir = build_dir
-        self.python_version = python_version or DEFAULT_PYTHON_VERSION
+        # User-supplied app-level override; None means "infer from resources".
+        self._python_version_override = python_version
+        # Effective app-level version; set by build() via _reconcile_python_version.
+        self.python_version: Optional[str] = None
 
     def _import_module(self, file_path: Path):
         """Import a module from file path, returning (module, cleanup_fn).
@@ -216,6 +219,12 @@ def _extract_config_properties(config: Dict[str, Any], resource_config) -> None:
         if hasattr(resource_config, "imageName") and resource_config.imageName:
             config["imageName"] = resource_config.imageName
 
+        if (
+            hasattr(resource_config, "python_version")
+            and resource_config.python_version
+        ):
+            config["python_version"] = resource_config.python_version
+
         if hasattr(resource_config, "templateId") and resource_config.templateId:
             config["templateId"] = resource_config.templateId
 
@@ -309,6 +318,63 @@ def _extract_config_properties(config: Dict[str, Any], resource_config) -> None:
 
         return config
 
+    def _reconcile_python_version(
+        self, resources_dict: Dict[str, Dict[str, Any]]
+    ) -> str:
+        """Pick one Python version for the app from per-resource declarations.
+
+        Flash apps ship as a single tarball, so every resource must target the
+        same Python ABI. Resolution order:
+          1. Explicit override passed to ManifestBuilder (validated)
+          2. Exactly one distinct ``python_version`` declared across resources
+          3. ``DEFAULT_PYTHON_VERSION`` when no resource declares one
+
+        Raises:
+            ValueError: When resources declare conflicting ``python_version``
+                values, or when the override conflicts with a resource's
+                explicit declaration.
+        """
+        per_resource: Dict[str, str] = {
+            name: r["python_version"]
+            for name, r in resources_dict.items()
+            if r.get("python_version")
+        }
+        distinct = set(per_resource.values())
+
+        if self._python_version_override:
+            chosen = validate_python_version(self._python_version_override)
+            conflicting = {
+                name: version
+                for name, version in per_resource.items()
+                if version != chosen
+            }
+            if conflicting:
+                details = ", ".join(
+                    f"{name}={version}" for name, version in sorted(conflicting.items())
+                )
+                raise ValueError(
+                    f"python_version override '{chosen}' conflicts with resource "
+                    f"declarations: {details}. Either remove the override or "
+                    f"align all resources to '{chosen}'."
+                )
+            return chosen
+
+        if len(distinct) > 1:
+            details = ", ".join(
+                f"{name}={version}" for name, version in sorted(per_resource.items())
+            )
+            raise ValueError(
+                "Flash apps require one python_version across all resources "
+                f"(found {sorted(distinct)}): {details}. Set python_version to the "
+                "same value on every resource, or omit it to use the default "
+                f"({DEFAULT_PYTHON_VERSION})."
+            )
+
+        if distinct:
+            return validate_python_version(next(iter(distinct)))
+
+        return DEFAULT_PYTHON_VERSION
+
     def build(self) -> Dict[str, Any]:
         """Build the manifest dictionary.
 
@@ -436,20 +502,6 @@ def build(self) -> Dict[str, Any]:
             # Determine if this resource makes remote calls
             makes_remote_calls = any(func.calls_remote_functions for func in functions)
 
-            # One tarball serves all resources, so target_python_version must agree.
-            # GPU resources are pinned to the base image's Python; CPU resources
-            # use DEFAULT_PYTHON_VERSION (aligned to GPU to avoid ABI mismatch).
-            _GPU_RESOURCE_TYPES = {
-                "LiveServerless",
-                "LiveLoadBalancer",
-                "LoadBalancerSlsResource",
-                "ServerlessEndpoint",
-            }
-            if resource_type in _GPU_RESOURCE_TYPES:
-                target_python_version = GPU_BASE_IMAGE_PYTHON_VERSION
-            else:
-                target_python_version = DEFAULT_PYTHON_VERSION
-
             resources_dict[resource_name] = {
                 "resource_type": resource_type,
                 "file_path": file_path_str,
@@ -460,8 +512,7 @@ def build(self) -> Dict[str, Any]:
                 "is_live_resource": is_live_resource,
                 "config_variable": config_variable,
                 "makes_remote_calls": makes_remote_calls,
-                "target_python_version": target_python_version,
-                **deployment_config,  # Include imageName, templateId, gpuIds, workers config
+                **deployment_config,  # Include imageName, templateId, gpuIds, workers config, python_version
             }
 
             # max_concurrency is QB-only; warn and remove for LB endpoints
@@ -495,6 +546,15 @@ def build(self) -> Dict[str, Any]:
                     )
                 function_registry[f.function_name] = resource_name
 
+        # Reconcile app-level python_version across resources. One tarball serves
+        # every resource in an app, so all resources must agree on one version.
+        self.python_version = self._reconcile_python_version(resources_dict)
+
+        # Stamp every resource's target_python_version with the reconciled
+        # app-level value so the runtime and pip-wheel step see a consistent ABI.
+        for resource in resources_dict.values():
+            resource["target_python_version"] = self.python_version
+
         manifest = {
             "version": "1.0",
             "python_version": self.python_version,

src/runpod_flash/cli/commands/deploy.py

@@ -43,6 +43,14 @@ def deploy_command(
         "--preview",
         help="Build and launch local preview environment instead of deploying",
     ),
+    python_version: str | None = typer.Option(
+        None,
+        "--python-version",
+        help=(
+            "Target Python version for worker images (3.10, 3.11, or 3.12). "
+            "Overrides per-resource python_version declarations."
+        ),
+    ),
 ):
     """
     Build and deploy Flash application.
@@ -56,6 +64,7 @@ def deploy_command(
       flash deploy --app my-app --env prod      # deploy a different app
       flash deploy --preview                    # build + launch local preview
       flash deploy --exclude transformers        # exclude additional packages from build
+      flash deploy --python-version 3.11        # target Python 3.11 workers
     """
     try:
         project_dir, discovered_app_name = discover_flash_project()
@@ -68,6 +77,7 @@ def deploy_command(
             no_deps=no_deps,
             output_name=output_name,
             exclude=exclude,
+            python_version=python_version,
         )
 
         if preview: