Merge upstream/develop into aicomnet_dev_slurm_multi

Merges 4 upstream commits (ef87d05..3a68be8) into the multi-node SLURM branch:
- feat: canonicalize launcher aliases; extract MAD_MULTI_NODE_RUNNER resolver
  into _resolve_local_multi_node_runner_env() (upstream ROCm#132)
- fix(run): --skip-model-run now starts container before skipping model script (ROCm#145)
- chore: consolidate optional deps into default dependencies (ROCm#146)
- Guyen/develop build-context folder change 2 (ROCm#144)
Conflict in src/madengine/execution/container_runner.py resolved:
- Upstream's if/else skip_model_run wrapper is the outer structure
- Our try/except RuntimeError (container diagnostics on failure) is placed
  inside the else: branch around model_docker.sh()
- Our elif skip_perf_collection: status branch and fallback or-clause are
  preserved in the else: branch status logic

Author: mkuznet1

.gitignore

@@ -144,4 +144,4 @@ rocm_trace_lite_output/
 slurm_results/
 MagicMock/
 .madengine_session_start
-run_directory/
+run_directory/

CHANGELOG.md

@@ -15,9 +15,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **`pytest` lower bound pinned to `>=7.0`**: Aligns the dependency pin with `minversion = "7.0"` already declared in `[tool.pytest.ini_options]`, preventing accidental resolution of older pytest versions that cannot run this project's tests.
 
+### Changed
+
+- **`--skip-model-run` now matches v1 semantics**: The flag previously short-circuited the entire run before any container started, and only took effect when a build ran in the same invocation (otherwise it was ignored with a warning). It now starts the container and runs `pre_scripts` and `post_scripts` as normal, skipping **only** the model script invocation — regardless of whether a build ran. The skip decision was moved out of `RunOrchestrator` and into `ContainerRunner`, so it applies uniformly to build+run and manifest-only (`--manifest-file`) invocations.
+
+- **`--skip-model-run` runs report `SKIPPED`, not `FAILURE`**: A skipped model is now aggregated as a successful run with status `SKIPPED`, and the overall workflow exits `0`. Previously a skipped run could surface as a failure.
+
+### Added
+
+- **`--skip-model-run --keep-alive` for live container debugging**: Combining the two flags leaves a fully-set-up container alive after the skipped run, ready for manual exec (`docker exec -it <container> bash`). When `--keep-alive` is set, the run prints the exact `cd <model_dir> && <script> <args>` command to invoke the model by hand; otherwise it hints to re-run with `--keep-alive`.
+
+- **Warning for local-only flags on distributed targets**: Passing `--skip-model-run`, `--keep-alive`, or `--keep-model-dir` with a SLURM or Kubernetes target now prints a yellow warning that these local Docker-only flags are ignored.
+
 ### Fixed
 
-- **`tools/` build context path corrected**: `docker build` now resolves the shared tools directory as `./tools` (project root) instead of `./scripts/common/tools`. The previous path was stale — `scripts/common/tools` is a temporary directory populated at runtime by `madengine run`, so it was absent during standalone `madengine build` invocations, silently omitting the `--build-context tools=…` flag and breaking Dockerfiles that rely on it via `COPY --from=tools`.
+- **`tools/` build context path corrected**: `docker build` now resolves the shared tools directory as `./docker/common` (project root) instead of `./scripts/common/tools`. The previous path was stale — `scripts/common/tools` is a temporary directory populated at runtime by `madengine run`, so it was absent during standalone `madengine build` invocations, silently omitting the `--build-context tools=…` flag and breaking Dockerfiles that rely on it via `COPY --from=tools`.
 
 - **Hatch package artifacts include `scripts/`**: `pyproject.toml` now uses `[tool.hatch.build.artifacts]` to include the `scripts/` directory in the built wheel. The previous `force-include` directive caused `duplicate file` errors with newer hatchling versions (which are stricter about files already covered by the default source inclusion). Switching to `artifacts` bypasses `.gitignore` exclusion without risk of duplication. The `deployment/templates` force-include was also removed as it is already captured by the default wheel source scan.
 
@@ -621,8 +633,8 @@ black src/ tests/ && isort src/ tests/
 # Run pre-commit hooks manually
 pre-commit run --all-files
 
-# Build without running (CI/CD)
-madengine run --tags model --skip-model-run
+# Skip model script (container starts, pre_scripts run); leave live container for debugging
+madengine run --tags model --skip-model-run --keep-alive
 
 # Debug with verbose output
 madengine run --tags model --verbose --live-output

CLAUDE.md

@@ -2,6 +2,72 @@
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
+## Behavioral Guidelines
+
+Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.
+
+**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.
+
+## 1. Think Before Coding
+
+**Don't assume. Don't hide confusion. Surface tradeoffs.**
+
+Before implementing:
+- State your assumptions explicitly. If uncertain, ask.
+- If multiple interpretations exist, present them - don't pick silently.
+- If a simpler approach exists, say so. Push back when warranted.
+- If something is unclear, stop. Name what's confusing. Ask.
+
+## 2. Simplicity First
+
+**Minimum code that solves the problem. Nothing speculative.**
+
+- No features beyond what was asked.
+- No abstractions for single-use code.
+- No "flexibility" or "configurability" that wasn't requested.
+- No error handling for impossible scenarios.
+- If you write 200 lines and it could be 50, rewrite it.
+
+Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
+
+## 3. Surgical Changes
+
+**Touch only what you must. Clean up only your own mess.**
+
+When editing existing code:
+- Don't "improve" adjacent code, comments, or formatting.
+- Don't refactor things that aren't broken.
+- Match existing style, even if you'd do it differently.
+- If you notice unrelated dead code, mention it - don't delete it.
+
+When your changes create orphans:
+- Remove imports/variables/functions that YOUR changes made unused.
+- Don't remove pre-existing dead code unless asked.
+
+The test: Every changed line should trace directly to the user's request.
+
+## 4. Goal-Driven Execution
+
+**Define success criteria. Loop until verified.**
+
+Transform tasks into verifiable goals:
+- "Add validation" → "Write tests for invalid inputs, then make them pass"
+- "Fix the bug" → "Write a test that reproduces it, then make it pass"
+- "Refactor X" → "Ensure tests pass before and after"
+
+For multi-step tasks, state a brief plan:
+```
+1. [Step] → verify: [check]
+2. [Step] → verify: [check]
+3. [Step] → verify: [check]
+```
+
+Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.
+
+---
+
+**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.
+
 ## Development Setup
 
 ```bash

README.md

@@ -587,7 +587,7 @@ After a local Docker run, madengine can scan the captured **run log** for common
 ### Build & Deployment
 
 - **Separate build and run phases** for distributed deployments
-- **Build without executing:** `madengine run --tags … --skip-model-run` skips container execution **after a build in that same invocation** (ignored when using an existing `--manifest-file`). See [Usage — Skip model run after build](docs/usage.md#skip-model-run-after-build).
+- **Skip model script:** `madengine run --tags … --skip-model-run` starts the container and runs `pre_scripts`, but skips the model script. Combine with `--keep-alive` for a live container ready for manual exec. Ignored with a warning on SLURM/K8s. See [Usage — Skip model run after build](docs/usage.md#skip-model-run-after-build).
 - **Use registries** for multi-node execution (K8s/SLURM)
 - **Use batch build mode** for CI/CD to optimize build times
 - **Specify `--target-archs`** when building for multiple GPU architectures

docs/cli-reference.md

@@ -97,7 +97,7 @@ madengine build [OPTIONS]
 | `--tags` | `-t` | TEXT | `[]` | Model tags to build (can specify multiple) |
 | `--target-archs` | `-a` | TEXT | `[]` | Target GPU architectures (e.g., gfx908,gfx90a,gfx942) |
 | `--registry` | `-r` | TEXT | `None` | Docker registry to push images to |
-| `--use-image` | | TEXT | `None` | Skip Docker build and use a pre-built image. Omit value or pass `auto` to resolve from model card's `DOCKER_IMAGE_NAME`. Mutually exclusive with `--registry` and `--build-on-compute` |
+| `--use-image` | | TEXT | `None` | Skip Docker build and use a pre-built image. Pass an image name, or `auto` to resolve from model card's `DOCKER_IMAGE_NAME`. Mutually exclusive with `--registry` and `--build-on-compute` |
 | `--build-on-compute` | | FLAG | `False` | Build Docker images on a SLURM compute node and push to registry. Requires `--registry` |
 | `--batch-manifest` | | TEXT | `None` | Input batch.json file for batch build mode |
 | `--additional-context` | `-c` | TEXT | `"{}"` | Additional context as JSON string |
@@ -149,7 +149,7 @@ madengine build --tags model --live-output --verbose
 madengine build --tags model --use-image lmsysorg/sglang:v0.5.2rc1-rocm700-mi30x
 
 # Auto-detect image from model card's DOCKER_IMAGE_NAME
-madengine build --tags model --use-image
+madengine build --tags model --use-image auto
 
 # Build on SLURM compute node and push to registry
 madengine build --tags model --build-on-compute --registry docker.io/myorg
@@ -226,10 +226,10 @@ madengine run [OPTIONS]
 | `--timeout` | | INT | `-1` | Timeout in seconds (-1=default 7200s, 0=no timeout) |
 | `--additional-context` | `-c` | TEXT | `"{}"` | Additional context as JSON string |
 | `--additional-context-file` | `-f` | TEXT | `None` | File containing additional context JSON |
-| `--keep-alive` | | FLAG | `False` | Keep Docker containers alive after run |
-| `--keep-model-dir` | | FLAG | `False` | Keep model directory after run |
+| `--keep-alive` | | FLAG | `False` | Keep Docker containers alive after run (local Docker only; ignored with a warning on SLURM/K8s) |
+| `--keep-model-dir` | | FLAG | `False` | Keep model directory after run (local Docker only; ignored with a warning on SLURM/K8s) |
 | `--clean-docker-cache` | | FLAG | `False` | Rebuild images without using cache (full workflow) |
-| `--skip-model-run` | | FLAG | `False` | After a **build in this invocation**, skip executing models (manifest/images still produced). **Ignored** when using `--manifest-file` with an existing manifest (run-only), or when no build ran in this invocation. See [Usage — Skip model run](usage.md#skip-model-run-after-build). |
+| `--skip-model-run` | | FLAG | `False` | Skip the model script inside each container. The container still starts and `pre_scripts` still run; only the model script invocation is skipped (status reported as `SKIPPED`, exit code `0`). Combine with `--keep-alive` to leave a live container for manual exec. Ignored with a warning on SLURM/K8s targets. See [Usage — Skip model run](usage.md#skip-model-run-after-build). |
 | `--manifest-output` | | TEXT | `build_manifest.json` | Output file for build manifest (full workflow) |
 | `--summary-output` | `-s` | TEXT | `None` | Output file for summary JSON |
 | `--live-output` | `-l` | FLAG | `False` | Print output in real-time |

docs/deployment.md

@@ -303,7 +303,7 @@ For workloads that use externally maintained Docker images (e.g. SGLang, vLLM re
 madengine build --tags model --use-image lmsysorg/sglang:latest
 
 # Auto-detect image from model card's DOCKER_IMAGE_NAME
-madengine build --tags model --use-image
+madengine build --tags model --use-image auto
 
 # Build on a SLURM compute node and push to registry
 madengine build --tags model --build-on-compute --registry docker.io/myorg

docs/launchers.md

@@ -629,7 +629,7 @@ slurm_multi models typically use pre-built images. The build phase has a **regis
 madengine build --tags my_model --use-image lmsysorg/sglang:latest
 
 # Auto-detect image from model card's DOCKER_IMAGE_NAME
-madengine build --tags my_model --use-image
+madengine build --tags my_model --use-image auto
 
 # Build on compute node and push to registry
 madengine build --tags my_model --build-on-compute --registry docker.io/myorg

docs/usage.md

@@ -283,15 +283,25 @@ madengine build --batch-manifest batch.json \
 
 ### Skip model run after build
 
-When `madengine run` **builds** in the same invocation (no pre-existing `--manifest-file`), you can pass **`--skip-model-run`** to produce images and `build_manifest.json` **without** running model containers.
+Pass **`--skip-model-run`** to start containers and run `pre_scripts`, but skip executing the model script itself.
 
-- **Ignored** when `--manifest-file` points at an existing manifest (execution-only mode): use plain `madengine run --manifest-file ...` to run later.
-- **Ignored** with a warning if this invocation did not perform a build (for example a manifest was already present and no rebuild occurred).
+- The Docker container **is started** and `pre_scripts` run normally.
+- Only the model script invocation is skipped; `post_scripts` and container cleanup still run.
+- Exit status is `SKIPPED` (not `FAILURE`) — the overall run exits `0`.
+- Combine with **`--keep-alive`** to leave a fully-set-up, live container for manual inspection or debugging.
+- Has **no effect** on distributed (SLURM/K8s) targets — a warning is printed if used with them.
 
 ```bash
+# Skip the model script; container starts and pre_scripts run
 madengine run --tags model \
   --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' \
   --skip-model-run
+
+# Leave a live container ready for manual exec
+madengine run --tags model \
+  --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' \
+  --skip-model-run --keep-alive
+# Then: docker exec -it <container> bash
 ```
 
 See [CLI Reference — `run`](cli-reference.md#run---execute-models) and `madengine run --help`.
@@ -425,9 +435,13 @@ madengine run --tags model --timeout 0
 ### Debugging
 
 ```bash
-# Keep containers alive
+# Keep container alive after run (local Docker only)
 madengine run --tags model --keep-alive
 
+# Skip model script and leave a live container ready for manual exec
+madengine run --tags model --skip-model-run --keep-alive
+# Then inspect: docker exec -it <container> bash
+
 # Verbose output
 madengine run --tags model --verbose --live-output
 
@@ -714,9 +728,12 @@ madengine run --tags model1 --tags model2,model3
 # Full verbose output with real-time logs
 madengine run --tags model --verbose --live-output
 
-# Keep container alive for inspection
+# Keep container alive for inspection (local Docker only)
 madengine run --tags model --keep-alive
 
+# Skip model script and leave live container for manual exec
+madengine run --tags model --skip-model-run --keep-alive
+
 # Check what will be discovered
 madengine discover --tags model --verbose
 ```

docs/wiki/index.html

@@ -295,7 +295,7 @@ <h2>Quick start</h2>
 madengine build --tags pyt_sglang_disagg --use-image registry.io/sglang:latest
 
 # Option B — auto-resolve DOCKER_IMAGE_NAME from model card
-madengine build --tags pyt_sglang_disagg --use-image
+madengine build --tags pyt_sglang_disagg --use-image auto
 
 # Option C — build on compute node, push, then run pulls in parallel
 madengine build --tags pyt_sglang_disagg \
@@ -621,7 +621,7 @@ <h4><code>--use-image</code> modes</h4>
 <table>
 <thead><tr><th>Invocation</th><th>Behavior</th></tr></thead>
 <tbody>
-<tr><td><code>--use-image</code> (bare flag)</td><td>Resolves to <code>"auto"</code> — reads <code>DOCKER_IMAGE_NAME</code> from model card <code>env_vars</code></td></tr>
+<tr><td><code>--use-image auto</code></td><td>Reads <code>DOCKER_IMAGE_NAME</code> from model card <code>env_vars</code></td></tr>
 <tr><td><code>--use-image registry.io/img:tag</code></td><td>Uses the explicit image name; skips all Docker build steps</td></tr>
 </tbody>
 </table>
@@ -1115,7 +1115,7 @@ <h3>Build modes</h3>
 <thead><tr><th>Mode</th><th>Flag</th><th>Behavior</th></tr></thead>
 <tbody>
 <tr><td>Use prebuilt image</td><td><code>--use-image registry.io/img:tag</code></td><td>Skip local build. Uses explicit image.</td></tr>
-<tr><td>Auto-resolve from model card</td><td><code>--use-image</code> (bare)</td><td>Reads <code>env_vars.DOCKER_IMAGE_NAME</code> from model card.</td></tr>
+<tr><td>Auto-resolve from model card</td><td><code>--use-image auto</code></td><td>Reads <code>env_vars.DOCKER_IMAGE_NAME</code> from model card.</td></tr>
 <tr><td>Build on compute</td><td><code>--build-on-compute --registry reg.io/ml</code></td><td>Builds on SLURM compute node, pushes to registry. Manifest sets <code>built_on_compute: true</code>. Run phase pulls in parallel on all nodes.</td></tr>
 <tr><td>Implicit fallback</td><td>no flags</td><td>If model card has <code>DOCKER_IMAGE_NAME</code>, auto-uses it. Otherwise raises <code>ConfigurationError</code> listing options.</td></tr>
 </tbody>

src/madengine/cli/commands/build.py

@@ -55,19 +55,13 @@ def build(
             "--batch-manifest", help="Input batch.json file for batch build mode"
         ),
     ] = None,
-    # NOTE: `is_flag=False, flag_value="auto"` lets `--use-image` (no value)
-    # mean "auto-detect from the model card's DOCKER_IMAGE_NAME", matching
-    # MAD-private PR #186's documented UX. Typer is deprecating this pattern
-    # for a future release; when removed, switch to requiring an explicit
-    # value (e.g. `--use-image auto` as the documented sentinel) and update
-    # MAD-private's docs in lockstep.
+    # Pass `--use-image auto` to auto-detect the image from the model card's
+    # DOCKER_IMAGE_NAME, or `--use-image NAME` to use an explicit image.
     use_image: Annotated[
         Optional[str],
         typer.Option(
             "--use-image",
-            is_flag=False,
-            flag_value="auto",
-            help="Skip Docker build and use pre-built image. Optionally specify image name, or omit to auto-detect from model card's DOCKER_IMAGE_NAME"
+            help="Skip Docker build and use a pre-built image. Pass an image name, or 'auto' to auto-detect from the model card's DOCKER_IMAGE_NAME",
         ),
     ] = None,
     build_on_compute: Annotated[