diff --git a/.github/workflows/README.md b/.github/workflows/README.md index e330a9484c..1f011eff24 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -3,16 +3,21 @@ In order to test configurations described in `configs`, the primary workflow file used is `.github/workflows/e2e-tests.yml`. As input, this workflow takes in the CLI arguments for the `utils/matrix_logic/generate_sweep_configs.py` script. The usage for this script is shown below: ``` -usage: generate_sweep_configs.py [-h] {full-sweep,test-config} ... +usage: generate_sweep_configs.py [-h] {full-sweep,curated-full-sweep,test-config} ... Generate benchmark configurations from YAML config files positional arguments: - {full-sweep,test-config} + {full-sweep,curated-full-sweep,test-config} Available commands full-sweep Generate full sweep configurations with optional filtering by model, precision, framework, runner type, and sequence lengths + curated-full-sweep Generate the canonical production full sweep matching + the ClusterMAX dashboard charts (kimik2.5 single-node + vLLM + dsv4 multi-node dynamo-vllm/llmd-vllm), excluding + TensorRT, dynamo-sglang, dsr1 single-node (deprecated) + and all qwen3.5 configs. test-config Generate full sweep for specific config keys. Supports wildcard patterns (* and ?) for matching multiple keys at once. @@ -90,6 +95,42 @@ full-sweep --multi-node --config-files configs/nvidia-master.yaml full-sweep --scenario-type agentic-coding --config-files configs/nvidia-master.yaml configs/amd-master.yaml ``` +## `curated-full-sweep` Command + +`curated-full-sweep` is the **canonical production full sweep**. It exists because a full sweep must run EXACTLY the model×engine×node-type combinations that the ClusterMAX dashboard charts render — the charts are the source of truth — and the `full-sweep` filters combine with AND within a single invocation, so these chart scenarios cannot be expressed in one `full-sweep` call (and the bare `full-sweep --config-files configs/nvidia-master.yaml` default sweeps *everything*, including qwen3.5 and TensorRT). The dashboard keeps only two active tabs, so `curated-full-sweep` composes the union of exactly two passes internally: + +- **kimik2.5 single-node** (`--single-node --model-prefix kimik2.5 --framework vllm`). +- **dsv4 multi-node** (`--multi-node --model-prefix dsv4 --framework dynamo-vllm llmd-vllm`) — the chart's dsv4 multi-node is vLLM-only; `llmd-vllm` emits vllm-prefixed metrics so it renders as vLLM. `dynamo-sglang` and `dynamo-trt` are excluded. +- **Excluded:** TensorRT (`dynamo-trt` / `trt`), `dynamo-sglang`, and **every** `qwen3.5-*` config. +- **Deprecated / deselected:** `dsr1` single-node (SGLang) and `gptoss120b` single-node are deprecated tabs and are no longer part of the curated sweep. `dsr1` configs remain in `nvidia-master.yaml` (deselected, not deleted — other one-off dispatches may still use them). `gptoss120b` has **no active master config** (only `configs/deprecated/`), so the harness cannot sweep it from the current `nvidia-master.yaml` anyway; its chart is fed from other/legacy data, which is out of scope for the harness. + +Model prefix, framework, and node-type are fixed by the curated target set and are not overridable. Only trimming filters are exposed (`--precision`, `--runner-type`, `--seq-lens`, `--step-size`, `--min-conc`, `--max-conc`, `--max-tp`, `--max-ep`) so a smoke run can shrink the sweep without ever widening the target set. Eval flags (`--no-evals`, `--evals-only`, `--all-evals`) behave as in `full-sweep`. + +``` +usage: generate_sweep_configs.py curated-full-sweep + --config-files CONFIG_FILES [CONFIG_FILES ...] + [--runner-config RUNNER_CONFIG] + [--no-evals | --evals-only] [--all-evals] + [--precision PRECISION [PRECISION ...]] + [--runner-type RUNNER_TYPE [RUNNER_TYPE ...]] + [--seq-lens {1k1k,8k1k} [{1k1k,8k1k} ...]] + [--step-size STEP_SIZE] + [--min-conc MIN_CONC] [--max-conc MAX_CONC] + [--max-tp MAX_TP] [--max-ep MAX_EP] +``` + +### Examples + +**Dispatch the canonical production full sweep** (pass verbatim as `inputs[generate-cli-command]` to `e2e-tests.yml`): +``` +curated-full-sweep --config-files configs/nvidia-master.yaml +``` + +**Trim to an 8k1k smoke slice for faster validation:** +``` +curated-full-sweep --config-files configs/nvidia-master.yaml --seq-lens 8k1k --max-conc 64 +``` + ## `test-config` Command The `test-config` command generates the full sweep for one or more specific config keys. This is useful for testing individual configurations without filtering by model prefix, framework, etc. diff --git a/AGENTS.md b/AGENTS.md index fdf2765b27..154fbcfa04 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -64,6 +64,27 @@ python utils/matrix_logic/generate_sweep_configs.py full-sweep \ [--runner-type b200|h100|h200|gb200|...] ``` +The canonical **production full sweep** is `curated-full-sweep` — it matches the +ClusterMAX dashboard charts (the source of truth), which keep only two active +tabs, and runs EXACTLY the union of two passes: (1) `kimik2.5` single-node on +`vllm`, and (2) `dsv4` multi-node on `dynamo-vllm` and `llmd-vllm` (the chart's +dsv4 multi-node is vLLM-only; `llmd-vllm` emits vllm-prefixed metrics). It +excludes `dynamo-sglang`, `dynamo-trt`/`trt`, and every `qwen3.5-*` config. +`dsr1` single-node (SGLang) and `gptoss120b` single-node are **deprecated** tabs +and are no longer part of the curated sweep — `dsr1` configs remain in +`nvidia-master.yaml` (deselected, not deleted; other one-off dispatches may still +use them), and `gptoss120b` has no active master config (only +`configs/deprecated/`) so it cannot be swept from `nvidia-master.yaml` anyway. +Prefer this over hand-rolled `full-sweep` filters for full sweeps (a bare +`full-sweep --config-files configs/nvidia-master.yaml` sweeps everything, +including qwen and TensorRT). Only trimming filters are overridable: + +```bash +python utils/matrix_logic/generate_sweep_configs.py curated-full-sweep \ + --config-files configs/nvidia-master.yaml \ + [--seq-lens 1k1k|8k1k] [--max-conc N] [--precision fp4|fp8] [--runner-type ...] +``` + Process results: `python utils/process_result.py`. ## Supported Configuration Values diff --git a/utils/matrix_logic/generate_sweep_configs.py b/utils/matrix_logic/generate_sweep_configs.py index e14c50c66b..ff9e16a010 100644 --- a/utils/matrix_logic/generate_sweep_configs.py +++ b/utils/matrix_logic/generate_sweep_configs.py @@ -1,3 +1,4 @@ +import copy import fnmatch import json import argparse @@ -776,6 +777,74 @@ def generate_full_sweep(args, all_config_data, runner_data): return matrix_values +# Curated "full sweep" target set. The ClusterMAX dashboard charts are the +# source of truth: the production full sweep must run EXACTLY the +# model x engine x node-type combinations the charts render, and nothing else. +# The dashboard keeps only two active tabs, so the curated sweep is the union of +# two passes, mapped to the concrete framework values in nvidia-master.yaml: +# - kimik2.5 single-node -> framework `vllm` +# - dsv4 multi-node -> frameworks `dynamo-vllm` + `llmd-vllm` (the chart's +# dsv4 multinode is vLLM-only; llmd-vllm emits vllm-prefixed metrics so it +# renders as vLLM). `dynamo-sglang` and `dynamo-trt` are EXCLUDED. +# Every `qwen3.5-*` config and all TensorRT (`trt`/`dynamo-trt`) are excluded. +# +# Deprecated / deselected chart scenarios (their configs stay in the master +# config for one-off dispatches, but the curated sweep no longer selects them): +# - dsr1 single-node (SGLang): deprecated tab, removed from the harness sweep. +# - gptoss120b single-node: has NO active master config (lives only under +# configs/deprecated/), so it cannot be swept from nvidia-master.yaml anyway. +# The charts keep these from other/legacy data; that is out of scope here. +# +# The generator's --model-prefix/--framework/--single-node/--multi-node filters +# combine with AND within a single invocation, so each chart scenario needs its +# own pass; the curated sweep composes them here so the harness has a single +# entrypoint. +CURATED_SWEEP_PASSES = [ + {"model-prefix": ["kimik2.5"], "framework": ["vllm"], "single-node": True}, + { + "model-prefix": ["dsv4"], + "framework": ["dynamo-vllm", "llmd-vllm"], + "single-node": False, + }, +] + + +def _curated_pass_args(base_args, *, model_prefix, framework, single_node): + """Clone base args, pinning the dimensions the curated sweep fixes.""" + passed = copy.copy(base_args) + passed.model_prefix = model_prefix + passed.framework = framework + passed.single_node = single_node + passed.multi_node = not single_node + return passed + + +def generate_curated_full_sweep(args, all_config_data, runner_data): + """Generate the canonical production full sweep. + + Composes the `generate_full_sweep` passes in CURATED_SWEEP_PASSES so the + result is EXACTLY the two active dashboard chart scenarios: kimi single-node + on vLLM, and dsv4 multi-node on dynamo-vllm/llmd-vllm. qwen3.5, TensorRT, + dynamo-sglang, dsr1 single-node (deprecated), and gptoss (no active master + config) are all excluded. Optional filters on `args` (--seq-lens, + --min-conc/--max-conc, --step-size, --runner-type, etc.) apply to every pass, + so trimming for a smoke run stays possible without widening the target set. + """ + matrix_values = [] + for sweep_pass in CURATED_SWEEP_PASSES: + matrix_values += generate_full_sweep( + _curated_pass_args( + args, + model_prefix=sweep_pass["model-prefix"], + framework=sweep_pass["framework"], + single_node=sweep_pass["single-node"], + ), + all_config_data, + runner_data, + ) + return matrix_values + + def _runner_values_for_filter(runner: str, runner_data: dict, runner_node_filter: str | None) -> list[str]: if not runner_node_filter: return [runner] @@ -1256,6 +1325,75 @@ def main(): help='Show this help message and exit' ) + # Subcommand: curated-full-sweep + curated_parser = subparsers.add_parser( + 'curated-full-sweep', + parents=[parent_parser], + add_help=False, + help=( + 'Generate the canonical production full sweep matching the ' + 'ClusterMAX dashboard charts: kimik2.5 single-node vLLM + dsv4 ' + 'multi-node dynamo-vllm/llmd-vllm, excluding TensorRT, ' + 'dynamo-sglang, dsr1 single-node (deprecated) and all qwen3.5 ' + 'configs. Model/framework/node-type are fixed by the curated target ' + 'set; only trimming filters are exposed.' + ) + ) + curated_parser.add_argument( + '--precision', + nargs='+', + required=False, + help='Precision(s) to filter by (e.g., fp4, fp8) (optional, can specify multiple)' + ) + curated_parser.add_argument( + '--runner-type', + nargs='+', + required=False, + help='Runner type(s) to filter by (e.g., h200, gb200) (optional, can specify multiple)' + ) + curated_parser.add_argument( + '--seq-lens', + nargs='+', + choices=list(seq_len_stoi.keys()), + required=False, + help=f"Sequence length configurations to include: {', '.join(seq_len_stoi.keys())}. If not specified, all sequence lengths are included." + ) + curated_parser.add_argument( + '--step-size', + type=int, + default=2, + help='Step size for concurrency values (default: 2)' + ) + curated_parser.add_argument( + '--min-conc', + type=int, + required=False, + help='Minimum concurrency value to include (filters out lower concurrency values)' + ) + curated_parser.add_argument( + '--max-conc', + type=int, + required=False, + help='Maximum concurrency value to include (filters out higher concurrency values)' + ) + curated_parser.add_argument( + '--max-tp', + type=int, + required=False, + help='Maximum tensor parallelism value to include (single-node only)' + ) + curated_parser.add_argument( + '--max-ep', + type=int, + required=False, + help='Maximum expert parallelism value to include (single-node only)' + ) + curated_parser.add_argument( + '-h', '--help', + action='help', + help='Show this help message and exit' + ) + # Subcommand: test-config test_config_keys_parser = subparsers.add_parser( 'test-config', @@ -1291,10 +1429,10 @@ def main(): args = parser.parse_args() apply_node_type_defaults(args) - if args.command == 'full-sweep' and args.step_size <= 1: + if args.command in ('full-sweep', 'curated-full-sweep') and args.step_size <= 1: parser.error("--step-size must be greater than 1") if ( - args.command == 'full-sweep' + args.command in ('full-sweep', 'curated-full-sweep') and args.min_conc is not None and args.max_conc is not None and args.min_conc > args.max_conc @@ -1310,6 +1448,8 @@ def main(): # Route to appropriate function based on subcommand if args.command == 'full-sweep': matrix_values = generate_full_sweep(args, all_config_data, runner_data) + elif args.command == 'curated-full-sweep': + matrix_values = generate_curated_full_sweep(args, all_config_data, runner_data) elif args.command == 'test-config': matrix_values = generate_test_config_sweep(args, all_config_data, runner_data) else: diff --git a/utils/matrix_logic/test_generate_sweep_configs.py b/utils/matrix_logic/test_generate_sweep_configs.py index 9a1c67d86e..8483c006ca 100644 --- a/utils/matrix_logic/test_generate_sweep_configs.py +++ b/utils/matrix_logic/test_generate_sweep_configs.py @@ -2695,3 +2695,168 @@ def test_prefill_entries_never_in_single_or_evals(self, mixed_entries): assert all('prefill' in x for x in multi) assert all('prefill' not in x for x in single) assert all('prefill' not in x for x in evals) + + +# ============================================================================= +# Guard test: curated production full sweep target set +# ============================================================================= + +import argparse as _argparse +from pathlib import Path as _Path + +from generate_sweep_configs import ( + CURATED_SWEEP_PASSES, + generate_curated_full_sweep, +) +from validation import load_config_files, load_runner_file + +_REPO_ROOT = _Path(__file__).resolve().parents[2] +_NVIDIA_MASTER = _REPO_ROOT / "configs" / "nvidia-master.yaml" +_RUNNERS = _REPO_ROOT / "configs" / "runners.yaml" + +# Independently-encoded expectation (NOT imported from the generator) so the +# guard cross-checks intent rather than echoing the implementation constants. +# The ClusterMAX dashboard charts are the source of truth; the dashboard keeps +# only two active tabs, so the curated sweep must render exactly: +# - kimik2.5 single-node -> vllm +# - dsv4 multi-node -> dynamo-vllm / llmd-vllm +# dsr1 single-node (deprecated) and gptoss120b (no active master config) are +# absent from the generated set; their configs stay in the master config. +_ALLOWED_MODEL_PREFIXES = {"kimik2.5", "dsv4"} +_ALLOWED_FRAMEWORKS = {"vllm", "dynamo-vllm", "llmd-vllm"} +_FORBIDDEN_FRAMEWORKS = {"dynamo-trt", "trt", "dynamo-sglang"} + + +def _curated_base_args(): + """Args mirroring the `curated-full-sweep` subcommand with no trimming.""" + args = _argparse.Namespace() + args.precision = None + args.runner_type = None + args.seq_lens = None + args.step_size = 2 + args.min_conc = None + args.max_conc = None + args.max_tp = None + args.max_ep = None + args.runner_node_filter = None + args.scenario_type = None + return args + + +@pytest.fixture(scope="module") +def curated_entries(): + all_config_data = load_config_files([str(_NVIDIA_MASTER)]) + runner_data = load_runner_file(str(_RUNNERS)) + return generate_curated_full_sweep( + _curated_base_args(), all_config_data, runner_data + ) + + +class TestCuratedFullSweepTargetSet: + """Guard the canonical production full sweep against the real master config. + + Runs `curated-full-sweep` over configs/nvidia-master.yaml and asserts the + generated set is EXACTLY the two active dashboard chart scenarios: kimi + single-node on vLLM, and dsv4 multi-node on dynamo-vllm / llmd-vllm. qwen3.5, + TensorRT, dynamo-sglang, dsr1 single-node (deprecated), and gptoss are all + excluded. + """ + + def test_generates_a_non_empty_sweep(self, curated_entries): + assert curated_entries, "curated full sweep unexpectedly empty" + + def test_every_model_prefix_is_in_target_set(self, curated_entries): + prefixes = {e["model-prefix"] for e in curated_entries} + assert prefixes <= _ALLOWED_MODEL_PREFIXES, ( + f"unexpected model-prefix(es): {prefixes - _ALLOWED_MODEL_PREFIXES}" + ) + + def test_no_qwen_entries(self, curated_entries): + assert not any( + str(e["model-prefix"]).startswith("qwen") for e in curated_entries + ), "qwen configs must be fully excluded from the curated full sweep" + + def test_no_gptoss_entries(self, curated_entries): + # gptoss120b is a chart scenario but has no active master config + # (deprecated-only), so it cannot appear in the generated set. + assert not any( + str(e["model-prefix"]).startswith("gptoss") for e in curated_entries + ), "gptoss has no active master config and must not appear" + + def test_every_framework_is_in_target_set(self, curated_entries): + frameworks = {e["framework"] for e in curated_entries} + assert frameworks <= _ALLOWED_FRAMEWORKS, ( + f"unexpected framework(s): {frameworks - _ALLOWED_FRAMEWORKS}" + ) + + def test_no_forbidden_frameworks(self, curated_entries): + assert not any( + e["framework"] in _FORBIDDEN_FRAMEWORKS for e in curated_entries + ), "trt/dynamo-trt/dynamo-sglang must be excluded from the curated full sweep" + + def test_kimi_entries_are_single_node_vllm(self, curated_entries): + kimi = [e for e in curated_entries if e["model-prefix"] == "kimik2.5"] + assert kimi, "expected kimi single-node entries in the curated sweep" + for e in kimi: + assert "prefill" not in e, "kimi entries must be single-node" + assert e["framework"] == "vllm", "kimi must run on framework vllm" + + def test_no_dsr1_entries(self, curated_entries): + # dsr1 single-node is a deprecated tab; it is deselected from the curated + # sweep (its configs remain in the master config for one-off dispatch). + assert not any( + str(e["model-prefix"]).startswith("dsr1") for e in curated_entries + ), "dsr1 is deprecated and must not appear in the curated full sweep" + + def test_dsv4_entries_are_multi_node_vllm_engines(self, curated_entries): + dsv4 = [e for e in curated_entries if e["model-prefix"] == "dsv4"] + assert dsv4, "expected dsv4 multi-node entries in the curated sweep" + for e in dsv4: + assert "prefill" in e, "dsv4 entries must be multi-node" + assert e["framework"] in {"dynamo-vllm", "llmd-vllm"}, ( + "dsv4 multi-node must be vLLM-only (dynamo-vllm / llmd-vllm)" + ) + + def test_dsv4_multinode_vllm_engines_are_represented(self, curated_entries): + dsv4_frameworks = { + e["framework"] for e in curated_entries if e["model-prefix"] == "dsv4" + } + assert {"dynamo-vllm", "llmd-vllm"} <= dsv4_frameworks, ( + f"missing dsv4 engine(s): {{'dynamo-vllm', 'llmd-vllm'}} - {dsv4_frameworks}" + ) + + def test_master_config_actually_contains_excluded_configs(self): + """Sanity: the exclusions are meaningful because qwen, dynamo-trt, + dynamo-sglang, and dsr1 single-node configs really exist in the master + config being filtered. Their absence from the curated sweep is + deselection, NOT deletion — the entries are preserved in the master + config for one-off dispatches.""" + all_config_data = load_config_files([str(_NVIDIA_MASTER)]) + keys = list(all_config_data.keys()) + assert any(k.startswith("qwen3.5") for k in keys), "expected qwen3.5 configs to exist" + assert any("dynamo-trt" in k for k in keys), "expected dynamo-trt configs to exist" + assert any("dynamo-sglang" in k for k in keys), "expected dynamo-sglang configs to exist" + # dsr1 single-node (sglang, multinode=False) must still exist in the + # master config even though the curated sweep no longer selects it. + dsr1_single_node = [ + k + for k, v in all_config_data.items() + if k.startswith("dsr1") + and v.get("framework") == "sglang" + and not v.get("multinode", False) + ] + assert dsr1_single_node, ( + "expected dsr1 single-node sglang configs to still exist in master " + "(deselected from curated sweep, not deleted)" + ) + + def test_curated_constants_match_intent(self): + """The generator passes must encode the documented chart target set.""" + assert CURATED_SWEEP_PASSES == [ + {"model-prefix": ["kimik2.5"], "framework": ["vllm"], "single-node": True}, + { + "model-prefix": ["dsv4"], + "framework": ["dynamo-vllm", "llmd-vllm"], + "single-node": False, + }, + ]