Add AI agent guidance docs (AGENTS.md, CLAUDE.md, skill.md) (#1013)

Author: anandhu-eng

.claude/skill.md

@@ -0,0 +1,252 @@
+# skill.md — Claude working guide for mlperf-automations
+
+Compact task-oriented reference. Read AGENTS.md for full technical detail.
+
+---
+
+## Mental model (read this first)
+
+```
+mlcflow CLI  →  finds script by tags  →  resolves variations & deps  →
+  customise.py:preprocess()  →  run.sh  →  customise.py:postprocess()  →
+  caches new_env to ~/MLC/repos/local/cache/{uid}/
+```
+
+**Two repos, two roles:**
+- `mlperf-automations` (this repo) = content: 377 script directories + `automation/` engine
+- `mlcflow` = driver CLI: installs via pip, dynamically loads the engine above
+
+**Three key files per script:**
+- `meta.yaml` — identity, tags, deps, variations, env mapping (schema in `automation/script/meta_schema.py`)
+- `customize.py` — Python hooks: `preprocess()` builds the shell command; `postprocess()` reads results
+- `run.sh` — executes `eval "${MLC_MLPERF_ENDPOINT_CMD}"` and exits non-zero on failure
+
+---
+
+## Task playbooks
+
+### "I need to understand what a script does"
+
+```bash
+cat script/<alias>/meta.yaml        # tags, deps, variations, input_mapping
+cat script/<alias>/customize.py     # what it validates and what command it builds
+cat script/<alias>/run.sh           # what actually executes
+cat script/<alias>/README.md        # human summary
+```
+
+Key fields to read in meta.yaml: `tags`, `deps`, `variations`, `input_mapping`,
+`new_env_keys`, `default_env`.
+
+### "I need to add a new script"
+
+1. Scaffold with `mlc add script`:
+   ```bash
+   # Basic skeleton (copies template,generic)
+   mlc add script mlcommons@mlperf-automations:<alias> --tags=<tags>
+
+   # Copy nearest existing script as the template instead
+   mlc add script mlcommons@mlperf-automations:<alias> --tags=<tags> \
+       --template_tags=app,mlperf,inference,reference
+   ```
+   This creates `script/<alias>/` with `meta.yaml`, `customize.py`, and `run.sh`.
+   If multiple scripts match `--template_tags`, it prompts you to pick one.
+
+2. Edit `meta.yaml` — update at minimum:
+   - `alias`, `uid` (already generated), `tags`, `category`
+   - `input_mapping` (CLI args → env vars)
+   - `new_env_keys` (what this script promises to export)
+   - `deps` chain
+3. Edit `customize.py` — implement `preprocess(i)`:
+   - Guard required env vars (return `{'return':1,'error':'...'}` if missing)
+   - Build the shell command string in `env['MY_CMD']`
+4. Edit `run.sh` — ensure it evals the command and exits non-zero on failure.
+5. `mlc lint script --tags=<alias>` — fix key order
+6. `mlct <alias>` — run built-in tests
+7. PR → `main` branch
+
+### "I need to add a variation"
+
+Add to `meta.yaml variations:`:
+```yaml
+variations:
+  my-variant:
+    group: my-group        # omit group if stackable (free-standing)
+    default: true          # omit if not the default
+    env:
+      MY_ENV_VAR: value
+    deps:
+      - tags: get,extra,dep   # extra dep only when this variation is active
+```
+
+Invoke: `mlcr <tags>,_my-variant` (underscore prefix on CLI).
+
+### "I need to add a conditional dependency"
+
+```yaml
+deps:
+  - tags: get,cuda
+    enable_if_env:
+      MLC_MLPERF_DEVICE: [gpu, cuda]   # only run if device is gpu or cuda
+  - tags: get,rocm
+    skip_if_env:
+      MLC_HOST_OS_TYPE: [windows]      # skip on Windows
+```
+
+### "I need to debug a script that fails silently"
+
+```bash
+mlcr <tags> --verbose                    # full debug output
+mlc find cache --tags=<failing-dep>      # check if dep is cached
+mlc rm cache --tags=<failing-dep>        # clear dep cache and re-run
+mlc show cache --tags=<script-tags>      # inspect cached new_env
+cat ~/MLC/repos/local/cache/*/mlc-cached-state.json | python3 -m json.tool
+```
+
+Common root causes:
+- Dep cached but stale → `mlc rm cache --tags=<dep-tags>`
+- `new_env_keys` missing from meta.yaml → key is silently dropped at the boundary
+- `preprocess` returned success but didn't set the expected env var → add assertion
+- `MLC_TMP_*` var expected downstream but not in `new_env_keys` → rename it or declare it
+
+### "I need to understand why an env var is not reaching a script"
+
+Env propagation rules:
+1. CLI `--key=val` → `input_mapping` → `env[MAPPED_VAR]`
+2. `env` passes to deps unless dep has `clean_env_keys` that matches
+3. Dep's `new_env_keys` is the **only** way values come back from a dep
+4. `MLC_TMP_*` never cached, never passed to deps automatically
+5. If a parent caches, only declared `new_env_keys` are replayed on cache hit
+
+### "I need to run the inference benchmark locally for testing"
+
+```bash
+# Quickest smoke test — resnet50, onnxruntime, CPU, 500 samples
+mlcr run-mlperf,inference,_submission,_short,_r6.0-dev \
+     --model=resnet50 --implementation=mlcommons-python \
+     --backend=onnxruntime --device=cpu \
+     --scenario=Offline --test_query_count=500 --target_qps=1 \
+     --hw_name=my_machine --quiet
+
+# Find performance (tunes target QPS before a real run)
+mlcr run-mlperf,inference,_find-performance,_short,_r6.0-dev \
+     --model=resnet50 --implementation=mlcommons-python \
+     --backend=onnxruntime --device=cpu \
+     --scenario=Offline --hw_name=my_machine --quiet
+
+# Accuracy-only run
+mlcr run-mlperf,inference,_accuracy-only,_short,_r6.0-dev \
+     --model=resnet50 --implementation=mlcommons-python \
+     --backend=onnxruntime --device=cpu \
+     --scenario=Offline --hw_name=my_machine --quiet
+
+# Full submission run (both modes + compliance + checker + tar)
+mlcr run-mlperf,inference,_submission,_full,_r6.0-dev \
+     --model=resnet50 --implementation=mlcommons-python \
+     --backend=onnxruntime --device=cpu \
+     --scenario=Offline --execution_mode=valid \
+     --submitter=MLCommons --hw_name=my_machine --quiet
+```
+
+### "I need to inspect or clear cache"
+
+```bash
+mlc find cache --tags=get,mlperf,endpoints  # find cache folder
+mlc show cache --tags=get,mlperf,endpoints  # print new_env snapshot
+mlc rm cache --tags=get,mlperf,endpoints    # delete specific cache
+mlc rm cache -f                             # delete ALL caches
+mlc prune cache                             # delete expired (past cache_expiration)
+```
+
+### "I need to run in Docker"
+
+```bash
+mlcd app,mlperf,inference,endpoints,_echo-server --num_samples=50
+# Rebuilds image if meta.yaml docker: section changed:
+mlcd app,mlperf,inference,endpoints --docker_rebuild
+```
+
+---
+
+## Quick reference: meta.yaml field cheat-sheet
+
+| Field | What it does |
+|---|---|
+| `alias` | Script directory name and lookup key |
+| `uid` | 16-hex unique ID; never change after first commit |
+| `automation_uid` | UID of the `script` automation type (`5b4e0237da074764`); the only automation type currently in this repo |
+| `tags` | Comma-separated discovery tags; must be a superset of what callers request |
+| `category` | Grouping label in docs |
+| `default_env` | Env vars set before variation env (lowest priority) |
+| `new_env_keys` | **Only** these keys leave the script; use `*` and `?` wildcards |
+| `input_mapping` | CLI `--key` → `ENV_VAR` translation |
+| `deps` | Scripts to run before `preprocess()` |
+| `prehook_deps` | Scripts to run after `preprocess()`, before `run.sh` |
+| `posthook_deps` | Scripts to run after `run.sh`, before `postprocess()` |
+| `post_deps` | Scripts to run after `postprocess()` |
+| `variations` | Named parameter sets; `group:` makes them mutually exclusive |
+| `add_deps_recursive` / `adr` | Override tags/versions on named deps deep in subtree |
+| `versions` | Per-version dep overrides and env |
+| `cache` | Enable caching (default: false) |
+| `cache_expiration` | Auto-invalidate after `1h` / `1d` / `1w` |
+| `tests` | Inline test cases run by `mlct` |
+
+---
+
+## customize.py hook signatures
+
+```python
+def preprocess(i):  # before run.sh
+def postprocess(i): # after run.sh
+def predeps(i):     # before dep execution
+def postdeps(i):    # after dep execution
+```
+
+All receive the same `i` dict:
+```python
+i['env']         # mutable env dict → set vars here
+i['automation']  # ScriptAutomation instance → i['automation'].logger
+i['os_info']     # dict from detect-os
+i['meta']        # parsed meta.yaml
+```
+
+All must return `{'return': 0}` or `{'return': 1, 'error': 'reason'}`.
+
+---
+
+## Env variable naming conventions
+
+No rigid rules — just two hard constraints and a naming guideline:
+
+- **`MLC_` prefix** — use on all env vars set by MLC scripts so they are clearly distinguishable from the surrounding environment.
+- **`MLC_TMP_*`** — reserved for transient vars: never cached, never passed to deps. Use this when a value is only needed within the current script's run.
+- **`+VAR` in `new_env_keys`** — prepend to an existing env var (e.g. `+PATH`); the engine handles concatenation.
+
+Everything else: name vars to reflect the script they come from and what they hold. Keep names reasonably short and meaningful — no other convention is enforced.
+
+---
+
+## Files to read first by task type
+
+| Task | Read these first |
+|---|---|
+| Understand the inference benchmark | `script/run-mlperf-inference-app/meta.yaml`, `customize.py` |
+| Reference for a new MLPerf benchmark (dep chain, variations, dispatch pattern) | `script/app-mlperf-inference-mlcommons-python/meta.yaml`, `script/run-mlperf-inference-app/meta.yaml` |
+| Debug env propagation | `automation/script/module.py` (search: `new_env_keys`, `clean_env_keys`) |
+| Debug caching | `automation/script/cache_utils.py` |
+| Add a script | Nearest similar script + `automation/script/meta_schema.py` |
+| Fix meta.yaml format | `automation/script/lint.py` |
+| Run CI locally | `.github/workflows/test-mlc-script-features.yml` |
+| Understand Docker path | `automation/script/docker.py` |
+
+---
+
+## Don'ts
+
+- Don't use `print()` in `customize.py` — use `i['automation'].logger`
+- Don't raise exceptions for recoverable errors — return `{'return': 1, 'error': '...'}`
+- Don't edit `mlc-cached-state.json` or `tmp-env.sh` by hand
+- Don't push directly to `main` — always open a PR; use `dev` only for urgent merges without approval
+- Don't hard-code paths in `run.sh` — use env vars set by `preprocess`
+- Don't change `automation_uid: 5b4e0237da074764` — it's the UID of the `script` automation type, not a script-specific value
+- Don't commit API keys — pass via `--api_key=...` only
+- Don't set `MLC_TMP_*` in `new_env_keys` (it's a transient namespace)