Merge 3 existing PR related to OnnxDiscrepancyCheck + llama.cpp integration (microsoft#2546)

## Describe your changes

Merges microsoft#2536, microsoft#2535, microsoft#2534.

Additionally adds llama.cpp integration and other improvements to
`OnnxDiscrepancyCheck`:

- **New `llama_cpp` flag** (`bool`, default `False`) on
`OnnxDiscrepancyCheck` — when enabled, converts the reference
HuggingFace model to GGUF format using `convert_hf_to_gguf.py` from
llama.cpp and compares inference with llama.cpp.
- **New `llama_cpp_env_path` parameter** (`Optional[str]`) — path to the
`llama_env` virtual environment where `llama-cpp-python` and
`convert_hf_to_gguf.py` are installed (defaults to `"llama_env"`
relative to cwd). The virtual environment also isolates potentially
conflicting versions of `torch`, `transformers`, etc.
- **`compare_llama_cpp()` method** — saves the reference model and
tokenizer to `output_dir/hf_model` (alongside the saved test
model/report output) using `save_pretrained` (standard HuggingFace
format), then:
1. Calls `convert_hf_to_gguf.py` (the official llama.cpp conversion CLI)
inside `llama_env` via subprocess to produce a GGUF F32 file at
`output_dir/model.gguf`.
2. Runs a self-contained inference helper script inside `llama_env` via
`subprocess.run` to measure first-token latency with `llama_cpp.Llama`.
Results include first-token match vs PyTorch, `llama_cpp_ttft_s`,
`llama_cpp_ttfn_s`, `llama_cpp_total_time_s`,
`llama_cpp_speedup_vs_pytorch`, and `llama_cpp_speedup_vs_onnx`. All
`llama_cpp`/`gguf` imports are strictly isolated to the subprocess — the
main Olive process never imports them.
- **New `--test_llama_path` CLI option** — specifies the path to the
`llama_env` virtual environment when running `olive optimize --test`.
When provided, the injected `OnnxDiscrepancyCheck` pass automatically
enables `llama_cpp=True` and forwards the path as `llama_cpp_env_path`.
Using `--test_llama_path` without `--test` emits a warning.
- **Improved `--test_metrics` parsing** — now accepts both
space-separated (`--test_metrics mae speedup`) and comma-separated
(`--test_metrics mae,speedup`) forms. A `_parse_test_metrics` type
function handles splitting and validation per token, and a
`_flatten_test_metrics` helper normalises the result before it is
forwarded to the pass.
- **Fixed `add_discrepancy_check_pass` update-in-place** — when a config
was previously generated by `olive optimize --dry_run --test`, the
`OnnxDiscrepancyCheck` pass was already present and was silently skipped
on subsequent `olive run --test --test_metrics …` calls. The function
now updates the existing pass in-place, refreshing
`reference_model_path` (resolved to absolute), `report_output_dir`,
metric settings (`max_mae` / `timing_iterations`), and llama.cpp
settings — so `--test_metrics`, `--output_path`, and `--test_llama_path`
from `olive run` always take effect.
- **Fixed test model persistence across engine cache hits** —
`ModelBuilder` now copies the test model directory to
`reference_hf_model/` alongside the generated ONNX output in the engine
cache. `OnnxDiscrepancyCheck` falls back to this cached copy when
`reference_model_path` (e.g. `out/tiny-test`) no longer exists on disk,
fixing the `OSError: Repo id must be in the form 'repo_name' or
'namespace/repo_name'` error that occurred on subsequent `olive run`
calls that hit the model-builder cache.
- **New `SaveTestModelConfig` pass**
(`olive/passes/pytorch/save_test_model_config.py`) — a new Olive pass
injected at the beginning of the passes list when `--test` is active. It
takes an `HfModelHandler`, writes `config.json` (with the reduced
hidden-layer count) and the Olive test-model marker file to
`test_model_path`, and returns the model unchanged. This ensures
`test_model_path` always exists as a config-only directory before
`ModelBuilder` or any downstream pass needs it, replacing the previous
`_save_test_model_config_for_dry_run` standalone function.
- **CI workflow** (`test-model-fast.yml`) — new step that creates a
`llama_env` virtual environment, installs `gguf`, `safetensors`,
`llama-cpp-python` (from pre-built CPU wheels at
`https://abetlen.github.io/llama-cpp-python/whl/cpu`), `transformers`,
`sentencepiece`, and `protobuf`, and downloads `convert_hf_to_gguf.py`
from the llama.cpp GitHub repository.
- **Updated documentation** (`cli-fast-test.md`) — clarifies WHERE the
2-layer reduction happens (`_apply_test_model_config` in
`olive/common/hf/utils.py`, called during the model-builder pass of
`olive run`) and WHEN `out/tiny-test` is created (by the
`SaveTestModelConfig` pass on first `olive run`, completed with weights
by `ModelBuilder`); documents the cache fallback behaviour; explains the
`--test_llama_path` option and `llama_env` setup; and clarifies that
`--test_metrics` is always respected even when the config was generated
by `olive optimize --dry_run --test`.

## Checklist before requesting a review
- [ ] Add unit tests for this change.
- [ ] Make sure all tests can pass.
- [ ] Update documents if necessary.
- [ ] Lint and apply fixes to your code by running `lintrunner -a`
- [ ] Is this a user-facing change? If yes, give a description of this
change to be included in the release notes.

## (Optional) Issue link

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Xiaoyu <85524621+xiaoyu-work@users.noreply.github.com>

Author: 4 people

.github/workflows/test-model-fast.yml

@@ -30,6 +30,18 @@ jobs:
           python -m pip install -r requirements.txt
           python -m pip install -r test/requirements-test-cpu.txt
 
+      - name: Create llama_env and install llama-cpp-python
+        run: |
+          LLAMA_ENV="$(pwd)/llama_env"
+          python -m venv "$LLAMA_ENV"
+          "$LLAMA_ENV/bin/pip" install --upgrade pip
+          "$LLAMA_ENV/bin/pip" install gguf safetensors llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
+          "$LLAMA_ENV/bin/pip" install transformers sentencepiece protobuf tabulate gguf
+          git clone --depth=1 --filter=blob:none --sparse https://github.com/ggerganov/llama.cpp.git /tmp/llama_cpp_repo
+          git -C /tmp/llama_cpp_repo sparse-checkout set convert_hf_to_gguf.py conversion --skip-checks
+          cp /tmp/llama_cpp_repo/convert_hf_to_gguf.py "$LLAMA_ENV/"
+          cp -r /tmp/llama_cpp_repo/conversion "$LLAMA_ENV/"
+
       - name: pip freeze
         run: |
           python -m pip freeze

docs/source/how-to/cli/cli-fast-test.md

@@ -2,12 +2,10 @@
 
 If you are converting a large language model, it is often useful to validate the Olive command, environment, and conversion recipe on a much smaller model before spending time on the full checkpoint.
 
-The `--test` option does that for Hugging Face models. Olive keeps the same model architecture, reduces it to a random 2-layer test model, saves it to the folder you provide, and reuses that folder on later runs.
+The `--test` option does that for Hugging Face models. Olive keeps the same model architecture, reduces it to a random **2-layer** test model, saves it under `<output_path>/reference_hf_model`, and reuses that folder on later runs.
 
 This example uses [`Qwen/Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B), but the same pattern works for other supported Hugging Face LLMs.
 
-## Step 1: generate the workflow config
-
 Start by generating the config that Olive will run for the Qwen conversion.
 
 ```bash
@@ -17,61 +15,28 @@ olive optimize \
     --provider CPUExecutionProvider \
     --precision int4 \
     --output_path out/qwen \
-    --dry_run
-```
-
-This creates `out/qwen/config.json` without launching the full conversion yet.
-
-## Step 2: run a fast smoke test with `olive run --test`
-
-Use the generated config with `olive run` and pass `--test` so Olive swaps in a reduced random Qwen model.
-
-```bash
-olive run \
-    --config out/qwen/config.json \
-    --test out/qwen-test-model \
-    --output_path out/qwen-test-run
+    --test
 ```
 
-What this does:
-
-- `--test out/qwen-test-model` creates a reduced random Qwen model and saves it in `out/qwen-test-model`
-- later runs reuse the same saved test model instead of recreating it
-- `--output_path out/qwen-test-run` gives the smoke test its own output folder, so the generated ONNX artifacts are easy to find
-- Olive marks that output folder as a test-only run and refuses to reuse a non-test conversion folder for `--test`
+Because this example runs without `--dry_run`, it produces:
 
-After the smoke test finishes, look under `out/qwen-test-run` for the exported ONNX model and related files.
-
-This is a quick way to confirm that:
-
-- Olive can load the source model
-- the selected optimization recipe is valid for your setup
-- the conversion path completes before you run the full model
-
-If you omit the folder and just pass `--test`, `olive run` will save the reduced model under `<output_path>/test_model`.
-
-## Step 3: run the full conversion
-
-Once the smoke test succeeds, rerun the conversion on the full Qwen checkpoint by removing `--test`.
-
-```bash
-olive run \
-    --config out/qwen/config.json \
-    --output_path out/qwen-full
-```
+- `out/qwen/olive_config.json` — the Olive configuration used for the run (named `olive_config.json` so it is never confused with the model's own `config.json`).
+- `out/qwen/reference_hf_model/` — the randomly initialized 2-layer reference model (weights, tokenizer, and `config.json`) that the ONNX model is compared against. It is created on the first run and reused afterwards.
+- `out/qwen/model/` — the optimized ONNX model.
+- `out/qwen/discrepancy_check_results.json` — the discrepancy report.
 
-At this point you know the Olive command and the conversion recipe already worked on the lightweight test model, so you can focus on the full-model run instead of debugging both at once.
+It also inserts an `OnnxDiscrepancyCheck` pass (if one is not already present) that compares the generated ONNX model against the 2-layer reference model. This pass only **loads** the reference model; it never saves a model or config itself (the reference model is materialized earlier in the workflow).
 
-## Why keep the test model folder?
+Additional metrics can be requested via `--test_metrics` (space- or comma-separated):
 
-The saved test model is useful beyond the first smoke test:
+- `mae`: enforces the max absolute error between the ONNX and reference logits (clean prefill with an empty KV cache, so both models run the identical forward pass). This is the default when `--test_metrics` is omitted.
+- `speedup`: ONNX-vs-PyTorch inference latency.
+- `first_token_20`: generates 20 tokens and compares the outputs of ONNX Runtime GenAI and transformers. It reports `first_token_matches` and `second_token_matches` (whether the first and second generated tokens are identical, along with the token ids for each backend) and `matching_leading_tokens` — the number of leading generated tokens that match. `matching_leading_tokens` is measured over the **generated** tokens only (the shared prompt is excluded) and is therefore capped at the 20-token generation length.
+- `tft`: time to the first generated token (reported for both ONNX Runtime GenAI and transformers).
+- `tf5t`: time to the first 5 generated tokens (reported for both ONNX Runtime GenAI and transformers).
 
-- you can rerun the reduced conversion quickly while iterating on options
-- you can reuse the same HF test model later when comparing the Hugging Face model against the exported ONNX model
-- you avoid recreating a new random test checkpoint every time
+For example, `--test_metrics mae,speedup,first_token_20,tft,tf5t`. The generation metrics (`first_token_20`, `tft`, `tf5t`) use the optimized ONNX model directory as the ONNX Runtime GenAI model when it contains a `genai_config.json` (as produced by the model builder). To keep tokenization consistent, ONNX Runtime GenAI is fed the exact token ids produced by the transformers tokenizer (including any BOS/special tokens) rather than re-encoding the prompt itself.
 
-## Related docs
+You can additionally compare against llama.cpp by passing `--test_llama_path <llama_env>` (a virtual environment with llama.cpp installed). When provided, `first_token_20` also reports `llama_cpp_first_token_matches`, `llama_cpp_second_token_matches`, and `llama_cpp_matching_leading_tokens`, again measured over generated tokens only and capped at the generation length.
 
-- [How to use the `olive optimize` command to optimize a Pytorch model](cli-optimize)
-- [How to write a new workflow from scratch](../configure-workflows/build-workflow)
-- [CLI reference](../../reference/cli)
+> **Note:** `--test_metrics` is always respected even when the config was generated by `olive optimize --test`, because Olive updates the existing `OnnxDiscrepancyCheck` settings each time `olive run --test` is invoked.