runtime : add --run-time-repack auto mode for swap-bound MoE safety

[AndrewMoryakov]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

What

Adds auto to --run-time-repack / -rtr and documents the -rtra / --run-time-repack-auto alias.

Mode	Behaviour
0 / off	Disable run-time repack
1 / on / bare -rtr	Force legacy run-time repack
auto	Repack only when the loader can conservatively prove that the non-mmap load fits; uncertainty disables repack and preserves requested mmap

The default remains 0.

Safety contract

This revision turns RTR auto into a fail-closed loader policy rather than a parser-only heuristic:

• forced RTR is coupled to effective loader mmap (repack_tensors=true forces loader mmap off), so the repack pass actually executes;
• auto uses metadata-only placement analysis and checked peak-memory arithmetic before constructing the real loader;
• every unsupported/ambiguous placement, overflow, memory-accounting failure, incompatible prefetch/defer mode, or platform uncertainty produces AUTO_UNKNOWN, disables repack, and preserves the requested mmap setting;
• peak accounting includes CPU-resident tensor bytes, maximum repack workspace, loader worker read buffers, and active CUDA staging buffers;
• tensor buffer overrides use the loader's first-match regex semantics;
• simple single-device GPU and -ncmoe placement is modeled; multi-device/merged/fit and other ambiguous paths remain UNKNOWN rather than optimistic;
• Linux memory headroom resolves /proc/self/cgroup and /proc/self/mountinfo, cgroup v1/v2 namespaces, bind/full mounts, ancestors, and v1 memsw, intersecting all applicable limits;
• cgroup errors remain fail-closed. For v2 only, a missing memory.max + memory.current pair is skipped as unbounded exactly at the resolved mountpoint; a partial pair or missing pair below it is UNKNOWN;
• Windows Job Object membership is conservatively UNKNOWN instead of using unsafe host-wide headroom; macOS/unsupported platform fallbacks are also conservative.

Effective state and tooling

The public model API exposes requested/effective mmap and RTR state separately, including whether mmap-backed buffers exist, whether the repack pass ran, the number of repacked tensors, and the final RTR status.

llama-bench writes an immutable test_v3 schema with requested/effective mmap and RTR fields. compare-llama-bench.py reads compatible test, test_v2, and test_v3 sources without cross-schema joins/averaging, uses effective mmap for v3 comparison, and renders unknown legacy RTR metadata explicitly.

ABI/API note

This PR adds trailing fields to the public llama_model_params structure (repack_tensors_auto and prefetch_experts). Applications linking the library must rebuild against matching headers; mixing old binaries/aggregate initializers with the new library is unsupported.

Validation for head 843de95f

Executed locally:

• Windows MSVC/Ninja: llama-bench, test-rtr-params, test-cgroup-resolver, test-rtr-auto-peak built successfully;
• CTest RTR/cgroup suite: 3/3 passed;
• compare-script Python suite: 14/14 passed;
• py_compile and git diff --check: passed;
• Linux/WSL g++ 13.3 with -std=c++17 -Wall -Wextra -Werror: cgroup resolver and peak-accounting tests passed;
• live cgroup-v2 smoke reproduced the root-mount missing-pair topology and resolved it correctly;
• real forced-RTR GGUF smoke verified: requested mmap=true, effective loader mmap=false, no mmap-backed model buffers, repack pass executed, n_repacked > 0, status=enabled;
• Windows auto smoke verified conservative auto_unknown under a Job Object while preserving mmap;
• real 60-column test_v3 SQL output was imported into SQLite and compared successfully.

A second independent review was run with Claude Code through Maestro Duet across five phases. Its final whole-PR verdict was GO after the cgroup-v2 root case and carry-forward cleanups were fixed.

These are evidence-backed guarantees for the exercised CPU/MSVC, Linux resolver, and Python/SQLite paths—not a universal platform guarantee. CUDA, Metal, MinGW, and the complete upstream matrix remain for upstream CI.

Scope deliberately left out

• changing the default RTR mode;
• making forced legacy -rtr 1 memory-safe (use auto for that policy);
• selective per-tensor mmap/repack decoupling;
• exact Windows Job Object limit accounting;
• changing malformed-present benchmark schemas from fail-hard to skip-with-warning.

Design notes

The safety contract, implementation stages, and regression matrix are recorded in:

• docs/RTR_AUTO_PR_FOLLOWUP_SPEC.md
• docs/RTR_AUTO_PR_FOLLOWUP_PLAN.md

[Ph0rk0z]

I guess only part is annoyingly having to add 1 after -rtr

[AndrewMoryakov]

@Ph0rk0z thanks for the early flag — to make sure I'm reading you right:

The bare -rtr form (no value after it) is preserved in this PR — it still means -rtr 1 exactly like before, so existing scripts and READMEs do not need to change. The optional value is only consumed when the next argv token is one of 0/1/off/on/auto; anything else (or end-of-args) leaves params.repack_tensors = true, params.repack_tensors_auto = false, which is the legacy fast path.

You can verify in the diff: common/common.cpp — the parser only does ++i to consume the value when is_mode_token is true.

So in practice:

Command	Behaviour
-rtr	legacy on (= 1)
-rtr 0 / -rtr off	off
-rtr 1 / -rtr on	on
-rtr auto	new safety-net mode
-rtra / --run-time-repack-auto	shortcut for -rtr auto

If you meant something different — e.g. you'd like the default of -rtr to be auto rather than 0, or you'd like a different short-form spelling — happy to adjust. The default (when -rtr is absent entirely) is unchanged on purpose in this PR; making auto the new default would be a worthwhile follow-up once the policy has been in the field for a bit.

[AndrewMoryakov]

Self-review found a real use_mmap regression — force-pushed a fix.

The bug: the parser was unconditionally setting params.use_mmap = false before parsing the optional value, which leaks two ways:

1. -rtr 0 (explicit disable) was also disabling mmap as a side effect — surprising because the user said "no repack".
2. -rtr auto was setting use_mmap = false at parse time, but if the auto policy then disabled repack at load (the whole point of the safety net on swap-bound models!), the load would still proceed with mmap = false — which on a model that already overflows RAM is exactly the wrong thing.

The fix (commit 0115ace21):

• common/common.cpp: rework the -rtr parser to determine the final state first, and only force use_mmap = false when repack is actually going to run unconditionally (legacy -rtr / -rtr 1). For -rtr 0 and -rtr auto we now leave params.use_mmap alone.
• src/llama.cpp: in llama_model_load, after the auto policy decides:
  • If auto keeps repack enabled → set use_mmap = false to match the legacy -rtr 1 coupling (repack pass writes back into tensor buffers).
  • If auto disables repack → leave use_mmap as the user configured it, so swap-bound MoE actually loads via mmap as intended.

End result is the four-way matrix that I think people would expect:

Command	repack	use_mmap (after load)
(no -rtr)	off	default true
-rtr 0	off	default true (was: false 🐛)
-rtr 1	on	false (legacy)
-rtr auto (in-RAM)	on (auto kept it)	false (matches -rtr 1)
-rtr auto (swap-bound)	off (auto disabled it)	default true (was: false 🐛)

Diff is +24 / -11 on top of the original commit; rebuilt cleanly on MSVC 2022 (Zen4); smoke-tested both the in-RAM and swap-bound paths via llama-bench/llama-cli.

Sorry for the noise — no changes to the API surface, only the internal coupling between repack and mmap got tightened.

[dmaivel]

Given that -rtr requires mmap to be disabled, I’m not sure users running swap-bounded MoE models are likely to specify it in the first place.

I tested -rtr auto with a model that does not fit in RAM, and it behaved the same as -rtr on, which seems problematic. This was on a Linux system with 64 GB of RAM and a ~120 GB model:

GGML_CUDA_NO_PINNED=1 ./llama-server \
    --model ~/.../Qwen3.5-397B-A17B-IQ2_XS-00001-of-00004.gguf \
    -c 4096 -ngl 99 -ot exps=CPU \
    -b 4096 -ub 4096 \
    -rtr auto

EDIT: For the above, I read your note about GPU+CPU split inference; if the path can't determine whether repacking should be enabled or disabled, isn't the wiser choice to disable? Otherwise, the user is left with the same failure message they would have gotten if they just used -rtr in its current form.

Also, checking total system memory is not sufficient because the allocation can still fail if other processes are consuming a significant amount of RAM. In that case, checking available memory would be more relevant.

I'm not sure it's necessary, but, if we do want to guard -rtr, would it make more sense to check whether there is enough available RAM and disable it automatically when there is not? That would avoid changing the argument to require on/off/auto; instead, we could simply print a message saying that rtr was disabled due to insufficient memory.

[AndrewMoryakov]

Thanks for catching this on a real workload. The reproducer is correct, and the current policy is wrong on it.

There are two concrete bugs in the code at 0115ace.

First, the auto path skips the check whenever n_gpu_layers > 0. The intent was to avoid false positives when GPU offload moves most weights out of CPU RAM, but the check only looks at n_gpu_layers and ignores tensor_buft_overrides. In your command, -ngl 99 offloads the non-expert parts, while -ot exps=CPU moves the expert weights back to CPU. That is still a swap-bound CPU-side load. The early skip returns false, the auto path keeps repack enabled, and llama_model_load then forces use_mmap=false, so it behaves like plain -rtr.

Second, the policy uses total physical RAM. That is not the right value for an allocation-safety decision. A busy system, page cache pressure, or container limits can make the effective available memory much lower than MEMORYSTATUSEX::ullTotalPhys or sysconf(_SC_PHYS_PAGES). The check should use available/effective memory instead.

I also agree with your edited point about uncertainty. If the auto path cannot determine that repack is safe, keeping repack enabled leaves the user with the same failure mode as -rtr. For an explicit safety mode, unknown should disable repack, preserve mmap, and log a warning.

If we keep the explicit -rtr auto design, the focused fix is:

1. use available/effective memory instead of total RAM;
2. remove the n_gpu_layers > 0 skip;
3. distinguish keep / disable / unknown instead of returning a bool;
4. make unknown safety-first for -rtr auto;
5. update the log/help text to say available memory.

There is still a conservative false-positive risk because the current probe compares full GGUF tensor bytes, not exact CPU-resident bytes after placement. That is safer than the current false negative, but a fully precise solution would need a later placement-aware estimator.

On the larger question: making -rtr itself self-protective is reasonable, but it changes legacy behavior and is not just a small patch on top of the current PR. The current legacy parser sets use_mmap=false for -rtr at parse time. If we add a load-time safety check without moving that coupling, we can still end up with repack disabled but mmap also disabled. A self-protective -rtr design should leave mmap untouched in the parser and resolve mmap only after the safety decision.

@ikawrakow, which direction would you prefer?

(a) keep the explicit -rtr auto mode and apply the focused fix above;
(b) rewrite this so existing -rtr becomes self-protective;
(c) defer this to a larger placement-aware design based on estimated CPU-resident bytes.

If there is no strong preference, I will go with (a) as the smallest reviewable change.

[ikawrakow]

In order this to be useful, it needs to consider actually available RAM and take into account tensor overrides and quantization types (i.e., use only the tensors that will get repacked when computing required memory).

Also, if I was a user with a low-RAM system where the model that I want to use does not fit in RAM, but I want to use the repacked quantization types, I would just offline repack the model using

./bin/llama-quantize --repack $model_in $model_out type

This would allow me to use the repacked model with mmap, which would be much better than swapping in and out.

[AndrewMoryakov]

Thanks for the direction. Plan is to walk probe.weights, filter by iqk_repacked_type() and resolve placement against tensor_buft_overrides / n_gpu_layers / -ncmoe, then compare CPU-resident repackable bytes against available memory.

One thing I want to flag. The loader forces use_mmap = false whenever repack_tensors=true (src/llama-model-loader.cpp:585), so a KEEP based on repackable bytes alone can still trigger swap when total CPU-resident bytes exceed available memory. I plan to AND-check total CPU-resident bytes as a secondary gate against the same threshold; the repackable-bytes metric stays primary and matches your guidance. Say so if you'd rather drop the total check or decouple repack from mmap separately.

For context: I run a CPU-only Zen4 box with 96 GiB RAM and no GPU. I cannot runtime-validate the -ngl 99 -ot exps=CPU configuration dmaivel reported, since exercising that path needs an actual GPU. The placement resolver handles it in code based on tensor_buft_overrides regex matching, but I have no way to run the case locally. Smoke covers in-RAM, swap-bound, and multi-shard cases on CPU only.

[AndrewMoryakov]

Pushed an update based on the maintainer feedback and dmaivel's reproducer. Current head is 1441ee9a9.

What changed since 0115ace21:

• the policy now uses available/effective memory instead of installed RAM;
• the decision is tri-state-ish: unsafe disables repack, unknown also disables repack with a WARN;
• the check walks probe.weights and filters by iqk_repacked_type() so the primary gate is CPU-resident repackable bytes, not full GGUF size;
• manual tensor_buft_overrides are applied with loader-compatible regex first-match semantics, so -ngl 99 -ot exps=CPU is no longer skipped just because n_gpu_layers > 0;
• simple layer placement is modeled for LAYER / NONE split modes; complex placement modes (--fit, merge modes, graph/attention split, multi-device -ncmoe distribution) are treated as unknown and safety-disable repack;
• there is also a secondary total CPU-resident bytes gate because the current loader still forces use_mmap=false whenever repack_tensors=true.

I also updated the PR description because the original body still described the old total-RAM / GPU-skip heuristic.

Local validation: MSVC 2022 nmake llama-cli succeeds, and llama-cli --help shows the updated -rtr [0|1|auto] syntax. I still cannot runtime-test the exact GPU reproducer locally; that path is covered by the override placement logic, not by hardware validation on my side.

[AndrewMoryakov]

Pushed a small follow-up cleanup at 15e04decd.

Three changes addressing reviewer feedback on 1441ee9a9:

1. Dropped the dense if (!is_moe) return NOT_APPLICABLE fast-exit. Dense models now go through the same byte-count walk as MoE. If the dense quant has repackable types and fits, the policy keeps repack; if it has none, the policy disables repack instead of leaving repack=true and silently forcing mmap=false in the loader. Closes the swap-thrash edge case on large dense models that lack repackable types in the current build.

2. Moved the unsupported-mode UNKNOWN exits (--fit, --merge-qkv, --merge-up-gate-experts, split_mode ATTN/GRAPH, available-memory query, and regex compile) before the metadata probe. The probe is real I/O on multi-shard models; bailing out early avoids paying it just to return UNKNOWN.

3. Removed a redundant if (cpu_repackable_bytes <= threshold) wrapper around the KEEP return and the unreachable trailing UNKNOWN. After the two upstream > checks the condition is always true.

Also dropped NOT_APPLICABLE from the decision enum and its caller-side case (now unreachable). The decision is a clean tri-state: KEEP / DISABLE / UNKNOWN.

Smoke matrix on CPU-only Zen4 96 GiB, all green:

Setup	Decision
Qwen3-30B Q4_K_M (17 GiB in-RAM)	KEEP
Qwen3.5-27B Q8_0 dense (29 GiB)	KEEP (was NOT_APPLICABLE before)
MiniMax M2.5 (90 GiB)	DISABLE via total-bytes gate
Qwen3-30B + --merge-qkv	UNKNOWN, early exit (no probe)
Qwen3-30B + -ot 'attn.*=CPU'	KEEP, override placement applied
gpt-oss-20b MXFP4 (11 GiB)	KEEP, repackable 1.2 GiB only

The -ngl 99 -ot exps=CPU reproducer still cannot be runtime-validated on my CPU-only box; the override placement logic was exercised in the -ot 'attn.*=CPU' smoke above.

[AndrewMoryakov]

Final pre-PR validation — 843de95f

I completed one more full review pass before updating this PR, including an independent five-phase Claude Code review through Maestro Duet. The final whole-PR verdict is GO: no known blocker remains in the reviewed RTR paths.

What the executed checks guarantee

For the configurations actually exercised:

• forced -rtr reaches the real repack pass: requested mmap stays observable, effective loader mmap is disabled, model buffers are not mmap-backed, and n_repacked > 0;
• -rtr auto remains fail-closed: every unresolved placement/memory/platform case disables repack and preserves requested mmap;
• cgroup v1/v2 parsing, mount/namespace resolution, ancestor intersection, and missing-v2-root-pair handling pass their regression tests;
• checked peak-memory arithmetic rejects overflow;
• requested/effective mmap and RTR getters are null-safe and distinguish loader decision, mmap-backed buffers, executed repack, count, and final status;
• test_v3 writer output imports into SQLite, and the compare script keeps schema generations isolated while handling legacy NULL RTR metadata.

Evidence

• MSVC/Ninja targets built: llama-bench, test-rtr-params, test-cgroup-resolver, test-rtr-auto-peak;
• CTest RTR/cgroup: 3/3 passed;
• Python compare suite: 14/14 passed;
• Linux/WSL g++ 13.3 -Wall -Wextra -Werror header tests: passed;
• live cgroup-v2 smoke: child memory pair present, root pair absent at mountpoint, correctly treated as an unbounded root level;
• real forced-RTR GGUF smoke and Windows Job Object auto smoke: passed with the expected effective states;
• real 60-column test_v3 SQL round-trip: passed;
• py_compile and git diff --check: passed.

Explicit boundary

This is not a claim that every backend/platform combination has been executed locally. CUDA, Metal, MinGW, and the full upstream matrix remain CI scope. The policy intentionally returns AUTO_UNKNOWN instead of making an optimistic memory decision when it cannot prove safety.

ABI reminder: llama_model_params gained trailing repack_tensors_auto and prefetch_experts fields; linked clients must rebuild against matching headers.