Add --test HF CLI path for 2-layer random model configs, olive run and ModelBuilder support, Qwen how-to/layer-types fix, and merge conflict resolution (microsoft#2459)

## Describe your changes

Adds a CLI test path for Hugging Face models so generated `config.json`
can carry a lightweight random-model definition instead of always using
pretrained weights. When `--test` is passed, Olive now preserves the
source architecture, instantiates a random model with 2 hidden layers,
and can persist that test model for reuse.

- **CLI/config support**
- Added `--test` to HF-backed CLI commands using shared input-model
options.
- `--test` now accepts an optional folder path where the generated test
model is saved for reuse.
- Emitted `input_model.test_model_config` into generated run configs,
and now also emits `input_model.test_model_path` when a save folder is
provided or derived.
    ```json
    {
      "input_model": {
        "type": "HfModel",
        "model_path": "model-id",
        "test_model_config": { "hidden_layers": 2 },
        "test_model_path": "path/to/test_model"
      }
    }
    ```
- When `--test` is used without an explicit folder, Olive uses
`<output_path>/test_model`.
- If `--test` is used in a context where no output path is available,
Olive now fails clearly instead of silently skipping persistence.

- **`olive run` support**
- Extended `olive run --test` so it can apply the same lightweight HF
test-model override to an existing Hugging Face `input_model` already
present in a workflow config.
- When `olive run --test` is used without an explicit folder, it derives
the saved test-model location from the effective workflow output path.
- `olive run --test` now fails clearly when the workflow config does not
contain a Hugging Face `input_model`.

- **HF model loading**
  - Extended HF config loading to accept `test_model_config`.
- Derived a lightweight config from the original model config by
overriding the architecture-specific hidden-layer field
(`num_hidden_layers`, `num_layers`, `n_layer`, `n_layers`).
- For Qwen-style configs that carry per-layer metadata, Olive now also
trims `layer_types` to match the reduced hidden-layer count so the saved
reduced config remains valid when reloaded.
- Switched test-model loading to instantiate from config (`from_config`)
so the model is random-initialized rather than loaded from pretrained
weights.
- Updated the test-model path to fail fast if the selected model class
cannot be instantiated from the reduced config, instead of falling back
to another candidate class that could produce a misleading larger model.
- Refactored the `from_config` loading path to avoid nested `try/except`
handling by only passing `trust_remote_code` when the model class
signature supports it.
- Added persistence/reuse support for test models: if `test_model_path`
already contains a saved HF model, Olive loads that model instead of
recreating it; otherwise it creates the reduced model once and saves it
there.

- **ModelBuilder support**
- Updated the ModelBuilder pass so `--test` workflows export from the
saved reduced Hugging Face test checkpoint instead of still using the
original full checkpoint.
- When `test_model_config` is present, ModelBuilder now materializes or
reuses `test_model_path` before export and passes that saved checkpoint
to the builder.
- This fixes the smoke-test flow so it avoids the original full-model
dtype path instead of only deferring the same failure.

- **IO config / dummy input propagation**
- Threaded `test_model_config` through HF IO-config and dummy-input
generation so the reduced-layer model shape metadata stays consistent
with the generated test model.

- **Documentation**
- Added a new how-to page showing how to convert a Qwen LLM with a quick
`--test` smoke check first, then rerun the full conversion.
  - Linked the new how-to page from the docs How Tos index.
- Updated the how-to flow to use `olive optimize --dry_run` followed by
`olive run --test`, matching the new CLI support.
- Clarified the smoke-test commands so the generated ONNX artifacts are
written to a dedicated output folder and are easy to find.
- Updated the example model from Phi to `Qwen/Qwen3-0.6B` and renamed
the how-to page and index entry to match.

- **Merge conflict resolution**
- Merged `origin/main` into this PR branch and resolved the conflict in
`test/passes/onnx/test_model_builder.py`.
- Preserved both the upstream `ModelBuilder` fallback/multi-file output
test coverage and this PR's saved test-model-path coverage.
- Updated the upstream mock-based `ModelBuilder` tests to define the new
Hugging Face test-model attributes used by this PR (`test_model_config`
and `test_model_path`).

- **Targeted coverage**
  - Added focused tests for:
    - CLI config generation with `--test`
    - input-model config serialization of `test_model_config`
    - input-model config serialization of `test_model_path`
    - validation when `--test` needs an explicit folder
- `olive run --test` overriding an existing HF `input_model` from a
workflow config
- validation when `olive run --test` is used on a non-HF workflow config
- HF random-model instantiation for multiple config naming conventions
- fail-fast behavior when test-model instantiation cannot use the
expected model class
- conditional `trust_remote_code` handling for supported, omitted, and
unsupported `from_config` signatures
    - saving and reusing a persisted HF test model
- ModelBuilder exporting from the saved reduced test-model checkpoint
when `test_model_config` is active
- a CLI smoke-flow test that follows the documented `olive optimize
--dry_run` then `olive run --test` commands with
`hf-internal-testing/tiny-random-LlamaForCausalLM` and verifies an ONNX
artifact is produced
- a Qwen3 regression test that verifies reduced test-model configs keep
`layer_types` aligned with the reduced hidden-layer count and can be
reloaded successfully
- conflict-resolved `ModelBuilder` tests covering saved test-model
reuse, single-file annotation fallback, and multi-file output component
naming

Examples:

```bash
olive optimize \
  -m Qwen/Qwen3-0.6B \
  --test out/test_model \
  --dry_run \
  -o out
```

```bash
olive run \
  --config out/config.json \
  --test out/test_model \
  --output_path out/qwen-smoke-run
```

## Checklist before requesting a review
- [x] Add unit tests for this change.
- [x] Make sure all tests can pass.
- [x] Update documents if necessary.
- [x] Lint and apply fixes to your code by running `lintrunner -a`
- [x] Is this a user-facing change? If yes, give a description of this
change to be included in the release notes.

Release notes: Added a `--test` option for Hugging Face CLI workflows
that can take a folder path, writes a lightweight 2-layer random-model
config, saves and reuses the generated HF test model from that folder,
fails fast if the reduced test model cannot be instantiated from the
expected model class, only passes `trust_remote_code` when the target
`from_config` supports it, and now keeps Qwen-style `layer_types`
metadata aligned with reduced test-model layer counts so saved reduced
configs reload cleanly. Also added `olive run --test` support for
workflow configs with Hugging Face input models, updated ModelBuilder to
export from the saved reduced test checkpoint for `--test` flows, and
added a Qwen3 0.6B how-to page for running a quick smoke test before the
full conversion with an explicit output path for the generated ONNX
files.

## (Optional) Issue link

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: xadupre <22452781+xadupre@users.noreply.github.com>
Co-authored-by: Xavier Dupré <xadupre@microsoft.com>
Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Author: 4 people

.github/workflows/test-model-fast.yml

@@ -0,0 +1,39 @@
+# This job is run as a github action.
+# It checks Olive works on random and small models.
+name: Test model fast
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  ubuntu-test-model-fast:
+    name: Ubuntu test model fast
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install -r requirements.txt
+          python -m pip install -r test/requirements-test-cpu.txt
+
+      - name: pip freeze
+        run: |
+          python -m pip freeze
+
+      - name: Run fast test
+        run: |
+          python -m pytest -v -s -p no:warnings --disable-warnings --log-cli-level=WARNING test/cli/test_cli_test_model_smoke.py

docs/source/how-to/cli/cli-fast-test.md

@@ -0,0 +1,77 @@
+# How to convert a Qwen model with a quick `--test` smoke check
+
+If you are converting a large language model, it is often useful to validate the Olive command, environment, and conversion recipe on a much smaller model before spending time on the full checkpoint.
+
+The `--test` option does that for Hugging Face models. Olive keeps the same model architecture, reduces it to a random 2-layer test model, saves it to the folder you provide, and reuses that folder on later runs.
+
+This example uses [`Qwen/Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B), but the same pattern works for other supported Hugging Face LLMs.
+
+## Step 1: generate the workflow config
+
+Start by generating the config that Olive will run for the Qwen conversion.
+
+```bash
+olive optimize \
+    --model_name_or_path Qwen/Qwen3-0.6B \
+    --device cpu \
+    --provider CPUExecutionProvider \
+    --precision int4 \
+    --output_path out/qwen \
+    --dry_run
+```
+
+This creates `out/qwen/config.json` without launching the full conversion yet.
+
+## Step 2: run a fast smoke test with `olive run --test`
+
+Use the generated config with `olive run` and pass `--test` so Olive swaps in a reduced random Qwen model.
+
+```bash
+olive run \
+    --config out/qwen/config.json \
+    --test out/qwen-test-model \
+    --output_path out/qwen-test-run
+```
+
+What this does:
+
+- `--test out/qwen-test-model` creates a reduced random Qwen model and saves it in `out/qwen-test-model`
+- later runs reuse the same saved test model instead of recreating it
+- `--output_path out/qwen-test-run` gives the smoke test its own output folder, so the generated ONNX artifacts are easy to find
+- Olive marks that output folder as a test-only run and refuses to reuse a non-test conversion folder for `--test`
+
+After the smoke test finishes, look under `out/qwen-test-run` for the exported ONNX model and related files.
+
+This is a quick way to confirm that:
+
+- Olive can load the source model
+- the selected optimization recipe is valid for your setup
+- the conversion path completes before you run the full model
+
+If you omit the folder and just pass `--test`, `olive run` will save the reduced model under `<output_path>/test_model`.
+
+## Step 3: run the full conversion
+
+Once the smoke test succeeds, rerun the conversion on the full Qwen checkpoint by removing `--test`.
+
+```bash
+olive run \
+    --config out/qwen/config.json \
+    --output_path out/qwen-full
+```
+
+At this point you know the Olive command and the conversion recipe already worked on the lightweight test model, so you can focus on the full-model run instead of debugging both at once.
+
+## Why keep the test model folder?
+
+The saved test model is useful beyond the first smoke test:
+
+- you can rerun the reduced conversion quickly while iterating on options
+- you can reuse the same HF test model later when comparing the Hugging Face model against the exported ONNX model
+- you avoid recreating a new random test checkpoint every time
+
+## Related docs
+
+- [How to use the `olive optimize` command to optimize a Pytorch model](cli-optimize)
+- [How to write a new workflow from scratch](../configure-workflows/build-workflow)
+- [CLI reference](../../reference/cli)

docs/source/how-to/index.md

@@ -12,6 +12,7 @@ The Olive CLI provides a set of primitives such as `quantize`, `finetune`, `onnx
 - [how to use the `olive finetune` command to create (Q)LoRA adapters](cli/cli-finetune)
 - [How to use the `olive quantize` command to quantize your model with different precisions and techniques such as AWQ](cli/cli-quantize)
 - [How to use the `olive run` command to execute an Olive workflow.](cli/cli-run)
+- [How to convert a Qwen model with a quick `--test` fast check](cli/cli-fast-test)
 
 # Olive Python API
 
@@ -43,6 +44,7 @@ The Olive CLI provides a set of primitives such as `quantize`, `finetune`, `onnx
 
 installation
 cli/cli-optimize
+cli/cli-fast-test
 cli/cli-auto-opt
 cli/cli-finetune
 cli/cli-quantize

olive/cli/base.py

@@ -17,6 +17,52 @@
 from olive.hardware.constants import DEVICE_TO_EXECUTION_PROVIDERS
 from olive.resource_path import OLIVE_RESOURCE_ANNOTATIONS
 
+TEST_OUTPUT_MARKER_FILE = "olive_test_output.json"
+
+
+def _get_test_output_marker_path(output_path: str) -> Path:
+    return Path(output_path) / TEST_OUTPUT_MARKER_FILE
+
+
+def is_test_output_dir(output_path: str) -> bool:
+    marker_path = _get_test_output_marker_path(output_path)
+    if not marker_path.is_file():
+        return False
+
+    try:
+        marker = json.loads(marker_path.read_text())
+    except (OSError, TypeError, ValueError):
+        return False
+
+    return marker.get("type") == "olive_hf_test_output"
+
+
+def validate_test_output_path(output_path: Optional[str], test_value) -> None:
+    if test_value in (None, False) or not output_path:
+        return
+
+    output_dir = Path(output_path)
+    if not output_dir.exists():
+        return
+    if not output_dir.is_dir():
+        raise ValueError(f"--output_path {output_path} must be a directory.")
+    if any(output_dir.iterdir()) and not is_test_output_dir(output_path):
+        raise ValueError(
+            f"--output_path {output_path} already exists and is not marked as an Olive test output directory. "
+            "Use a dedicated output folder for --test runs."
+        )
+
+
+def mark_test_output_path(output_path: Optional[str]) -> None:
+    if not output_path:
+        return
+
+    output_dir = Path(output_path)
+    if not output_dir.is_dir():
+        return
+
+    _get_test_output_marker_path(output_path).write_text(json.dumps({"type": "olive_hf_test_output"}, indent=2))
+
 
 class BaseOliveCLICommand(ABC):
     allow_unknown_args: ClassVar[bool] = False
@@ -33,16 +79,21 @@ def _run_workflow(self):
 
         from olive.workflows import run as olive_run
 
+        validate_test_output_path(self.args.output_path, getattr(self.args, "test", None))
         Path(self.args.output_path).mkdir(parents=True, exist_ok=True)
 
         with tempfile.TemporaryDirectory(prefix="olive-cli-tmp-", dir=self.args.output_path) as tempdir:
             run_config = self._get_run_config(tempdir)
             if self.args.save_config_file or self.args.dry_run:
                 self._save_config_file(run_config)
             if self.args.dry_run:
+                if getattr(self.args, "test", None) not in (None, False):
+                    mark_test_output_path(self.args.output_path)
                 print("Dry run mode enabled. Configuration file is generated but no optimization is performed.")
                 return None
             workflow_output = olive_run(run_config)
+            if getattr(self.args, "test", None) not in (None, False):
+                mark_test_output_path(self.args.output_path)
             if not workflow_output.has_output_model():
                 print("No output model produced. Please check the log for details.")
             else:
@@ -82,6 +133,21 @@ def run(self):
         raise NotImplementedError
 
 
+def add_hf_test_model_config(input_model: dict, test_value, output_path: Optional[str] = None) -> dict:
+    if test_value in (None, False):
+        return input_model
+
+    test_model_output_path = test_value
+    # Use 2 layers to keep the test model fast and lightweight while preserving the original architecture family.
+    input_model["test_model_config"] = {"hidden_layers": 2}
+    if test_model_output_path is True:
+        if not output_path:
+            raise ValueError("--test requires an explicit folder when output_path is not available.")
+        test_model_output_path = str(Path(output_path) / "test_model")
+    input_model["test_model_path"] = test_model_output_path
+    return input_model
+
+
 def _get_hf_input_model(args: Namespace, model_path: OLIVE_RESOURCE_ANNOTATIONS) -> dict:
     """Get the input model config for HuggingFace model.
 
@@ -105,7 +171,7 @@ def _get_hf_input_model(args: Namespace, model_path: OLIVE_RESOURCE_ANNOTATIONS)
         input_model["adapter_path"] = args.adapter_path
     if getattr(args, "trust_remote_code", None) is not None:
         input_model["load_kwargs"]["trust_remote_code"] = args.trust_remote_code
-    return input_model
+    return add_hf_test_model_config(input_model, getattr(args, "test", None), getattr(args, "output_path", None))
 
 
 def _get_onnx_input_model(args: Namespace, model_path: str) -> dict:
@@ -371,6 +437,16 @@ def add_input_model_options(
         model_group.add_argument(
             "--trust_remote_code", action="store_true", help="Trust remote code when loading a huggingface model."
         )
+        model_group.add_argument(
+            "--test",
+            type=str,
+            nargs="?",
+            const=True,
+            help=(
+                "Use a randomly initialized test model with the same Hugging Face architecture and 2 hidden layers. "
+                "Optionally provide a folder where the generated test model should be saved and reused."
+            ),
+        )
 
     if enable_hf_adapter:
         assert enable_hf, "enable_hf must be True when enable_hf_adapter is True."

olive/cli/run.py

@@ -6,10 +6,13 @@
 
 from olive.cli.base import (
     BaseOliveCLICommand,
+    add_hf_test_model_config,
     add_input_model_options,
     add_logging_options,
     add_telemetry_options,
     get_input_model_config,
+    mark_test_output_path,
+    validate_test_output_path,
 )
 from olive.telemetry import action
 
@@ -59,6 +62,14 @@ def run(self):
         if input_model_config := get_input_model_config(self.args, required=False):
             print("Replacing input model config in run config")
             run_config["input_model"] = input_model_config
+        elif self.args.test not in (None, False):
+            input_model = run_config.get("input_model")
+            if not isinstance(input_model, dict) or input_model.get("type", "").lower() != "hfmodel":
+                raise ValueError("--test for olive run requires a Hugging Face input_model in the run config.")
+            output_path = (
+                self.args.output_path or run_config.get("output_dir") or run_config.get("engine", {}).get("output_dir")
+            )
+            run_config["input_model"] = add_hf_test_model_config(input_model, self.args.test, output_path)
 
         for arg_key, rc_key in [("output_path", "output_dir"), ("log_level", "log_severity_level")]:
             if (arg_value := getattr(self.args, arg_key)) is not None:
@@ -68,12 +79,16 @@ def run(self):
                 # add value to run config directly
                 run_config[rc_key] = arg_value
 
+        output_path = run_config.get("output_dir") or run_config.get("engine", {}).get("output_dir")
+        validate_test_output_path(output_path, self.args.test)
         workflow_output = olive_run(
             run_config,
             list_required_packages=self.args.list_required_packages,
             tempdir=self.args.tempdir,
             package_config=self.args.package_config,
         )
+        if self.args.test not in (None, False):
+            mark_test_output_path(output_path)
 
         if self.args.list_required_packages is True:
             print("Required packages listed!")

olive/common/hf/model_io.py

@@ -27,6 +27,7 @@ def get_model_io_config(
     model_name: str,
     task: str,
     model: Optional["PreTrainedModel"] = None,
+    test_model_config: Optional[dict[str, Any]] = None,
     **kwargs,
 ) -> Optional[dict[str, Any]]:
     """Get the input/output config for the model and task.
@@ -35,6 +36,7 @@ def get_model_io_config(
         model_name: The model name or path.
         task: The task type (e.g., "text-generation", "text-classification").
         model: Optional loaded model for input signature inspection.
+        test_model_config: Optional overrides for creating a lightweight random test model from the same config.
         **kwargs: Additional arguments including use_cache.
 
     Returns:
@@ -68,7 +70,7 @@ def get_model_io_config(
         return None
 
     # Get model config
-    model_config = get_model_config(model_name, **kwargs)
+    model_config = get_model_config(model_name, test_model_config=test_model_config, **kwargs)
 
     # Handle PEFT models
     actual_model = model
@@ -92,6 +94,7 @@ def get_model_dummy_input(
     model_name: str,
     task: str,
     model: Optional["PreTrainedModel"] = None,
+    test_model_config: Optional[dict[str, Any]] = None,
     **kwargs,
 ) -> Optional[dict[str, Any]]:
     """Get dummy inputs for the model and task.
@@ -100,6 +103,7 @@ def get_model_dummy_input(
         model_name: The model name or path.
         task: The task type.
         model: Optional loaded model for input signature inspection.
+        test_model_config: Optional overrides for creating a lightweight random test model from the same config.
         **kwargs: Additional arguments including use_cache, batch_size, sequence_length.
 
     Returns:
@@ -133,7 +137,7 @@ def get_model_dummy_input(
         return None
 
     # Get model config (handles MLflow paths)
-    model_config = get_model_config(model_name, **kwargs)
+    model_config = get_model_config(model_name, test_model_config=test_model_config, **kwargs)
 
     # Handle PEFT models
     actual_model = model