INT8 quantized runtime, CPU perf work, and a big code/docs cleanup

Quantized runtime: the INT8 PW->DW / DW->PW pairs and the residual/fork
tails now run as single streaming kernels (on-the-fly requant, no
intermediate boundary tensor), alongside NCHWc weight prepack, native
NCHWc pointwise GEMM, VNNI RHS prepack at load, and a SIMD requant
epilogue. Output stays bit-identical to the unfused per-op path.

fp32 CPU (Siamese tracker vs ONNX Runtime): fixed parallel GEMM being
short-circuited by the BLAS path, reworked blocked_gemm parallelism, and
added a 12x32 AVX-512 microkernel, pointwise_nx16/16x16 direct kernels,
streaming FusedPwDw/FusedDwPw, an AVX-512 c=16 depthwise path, and
FusedTransposeMatMul.

Cleanup: dropped session-specific notes from comments and deleted 12 dead
probe examples; split the eight biggest files (matmul, conv,
fused_pw_dw_3x3, backend, tensor ops, the ONNX runner, loader) into
directory modules — pure moves, tracker output and latency unchanged.
Refreshed the docs to the new layout, re-measured the README tracker
numbers, and made cargo doc build clean. Also repointed the doc-count
script and two CI microbench thresholds at the new file layout.

Author: enthropy7

AGENTS.md

@@ -73,7 +73,7 @@ reverted on sight, regardless of benchmark numbers.
    together, or stay scalar. Runtime feature detection via
    `std::arch::is_*_feature_detected!`. The dispatch pattern is
    consistent across the codebase — copy from
-   `crates/yscv-kernels/src/ops/matmul.rs` if in doubt.
+   `crates/yscv-kernels/src/ops/matmul/` if in doubt.
 
 3. Minimal code. Smallest correct change. Iterators where the
    compiler vectorises them, no `unwrap`, no `#[allow(dead_code)]`,

CHANGELOG.md

@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- Internal refactor: split the largest perf-arc source files (`matmul`,
+  `conv`, `fused_pw_dw_3x3`, `backend`, tensor `ops`, the ONNX `runner`,
+  and `loader`) into focused directory submodules. No public API or
+  behavior change — tracker output stays bitwise-identical and 1T/6T
+  latency is unchanged within noise.
+- Documentation and code comments cleaned of session-specific
+  optimization-arc notes and refreshed to the current module layout.
+
 ## [0.1.9] — 2026-05-02
 
 ### Changed

CONTRIBUTING.md

@@ -14,7 +14,7 @@ The workspace has 18 library crates, 3 application binaries (`apps/bench`, `apps
 
 Key crates and what they do:
 
-- **yscv-tensor** — the foundation. 115 tensor ops in `ops.rs`, f32/f16/bf16 dtype, SIMD (AVX/SSE/NEON).
+- **yscv-tensor** — the foundation. 115 tensor ops in `ops/`, f32/f16/bf16 dtype, SIMD (AVX/SSE/NEON).
 - **yscv-kernels** — CPU and GPU compute backends. SIMD dispatch (AVX + SSE + NEON with scalar fallback), 61 WGSL + 4 Metal compute shaders, rayon threading.
 - **yscv-autograd** — dynamic computation graph with 61 `Op` variants and gradient checkpointing.
 - **yscv-optim** — 8 optimizers (SGD/Adam/AdamW/RAdam/RmsProp/Adagrad/Lamb/Lars) all with NEON+AVX+SSE SIMD, Lookahead meta-optimizer, 11 LR schedulers.

README.md

@@ -13,7 +13,7 @@
 
 A complete computer vision and deep learning framework in pure Rust. One `cargo add yscv` gives you image processing (160 ops, faster than OpenCV), neural network training (39 layer types, 8 optimizers), ONNX inference (122 operators, INT4/INT8 quantization), LLM generation (KV-cache, RoPE, GQA), real-time detection + tracking + recognition (67µs per frame), H.264/HEVC/AV1 video decoding (4.5× faster than ffmpeg), hardware decode (VideoToolbox/VAAPI/NVDEC/MediaFoundation), and GPU compute via Vulkan/Metal/DX12 — all in a single statically-linked binary with zero Python or C++ dependencies.
 
-> Project focus. YSCV is built for CPU inference on edge devices — Raspberry Pi, Rockchip / Allwinner SBCs, drone boards, factory PCs, anything ARM Cortex-A or low-power x86. Hot paths are hand-tuned SIMD (NEON / AVX / SSE / scalar) with rayon multi-thread fork-join, profiled and CI-gated against a baseline. On bare-metal ARM SBCs we currently beat ONNX Runtime CPU on a public Siamese tracker; the rest of the perf arc is documented in [`docs/perf-arc-2026-04.md`](docs/perf-arc-2026-04.md). Other backends — wgpu cross-platform GPU, Apple MPSGraph, Rockchip RKNN NPU, Intel/AMD BLAS — exist as opt-in features and keep getting wider, but they're not the headline target. PRs are welcome; see [`CONTRIBUTING.md`](CONTRIBUTING.md).
+> Project focus. YSCV is built for CPU inference on edge devices — Raspberry Pi, Rockchip / Allwinner SBCs, drone boards, factory PCs, anything ARM Cortex-A or low-power x86. Hot paths are hand-tuned SIMD (NEON / AVX / SSE / scalar) with rayon multi-thread fork-join, profiled and CI-gated against a baseline. On bare-metal ARM SBCs we currently beat ONNX Runtime CPU on a public Siamese tracker; benchmarks across threads and hardware are in [`docs/performance-benchmarks.md`](docs/performance-benchmarks.md). Other backends — wgpu cross-platform GPU, Apple MPSGraph, Rockchip RKNN NPU, Intel/AMD BLAS — exist as opt-in features and keep getting wider, but they're not the headline target. PRs are welcome; see [`CONTRIBUTING.md`](CONTRIBUTING.md).
 >
 > Agent-friendly documentation. YSCV is structured so that an AI coding agent can wire it into a downstream project end-to-end without prior context: every crate has a focused `README.md` describing its surface, [`docs/cookbook.md`](docs/cookbook.md) has recipes per task, [`docs/feature-flags.md`](docs/feature-flags.md) is exhaustive on Cargo features and runtime env knobs, [`AGENTS.md`](AGENTS.md) has the workflow + style rules verbatim, and per-op profile labels (`YSCV_RUNNER_PROFILE=path` dumps fused-path JSON) make hot-path issues self-diagnosing. The benefit is downstream: agents can build working code on top of yscv quickly, not the other way around. Responsibility for any PR — including patches drafted by an agent — rests with the human author submitting it.
 
@@ -134,8 +134,8 @@ Every operation has hand-tuned SIMD on all platforms — NEON on ARM, AVX/SSE on
 | YOLO11n MPSGraph | **5.9ms** | all competitors FAIL | **WIN** |
 | Siamese tracker 1T (Orange Pi Zero 3, 2026-04-21) | **461.6ms** | onnxruntime 499.3ms | **1.08× faster** |
 | Siamese tracker 4T (Orange Pi Zero 3, 2026-04-21) | **150.2ms** | onnxruntime 164.6ms | **1.10× faster** |
-| Siamese tracker 1T (Zen 4 7500F, 2026-04-25) | 11.22ms | onnxruntime 8.07ms | 1.39× slower |
-| Siamese tracker 6T (Zen 4 7500F, 2026-04-25) | 3.17ms | onnxruntime 1.74ms | 1.82× slower |
+| Siamese tracker 1T (Zen 4 7500F, 2026-05-30) | 8.66ms | onnxruntime 1.24.4 8.10ms | 1.07× slower |
+| Siamese tracker 6T (Zen 4 7500F, 2026-05-30) | 2.48ms | onnxruntime 1.24.4 1.75ms | 1.42× slower |
 
 Full benchmark results in [docs/performance-benchmarks.md](docs/performance-benchmarks.md).
 
@@ -156,7 +156,7 @@ The framework is split into 18 crates, each doing one thing well:
 | `yscv-track` | DeepSORT, ByteTrack, Kalman filter, Hungarian assignment, Re-ID |
 | `yscv-recognize` | Cosine matching, VP-Tree ANN indexing, Recognizer with enroll/match |
 | `yscv-eval` | Classification/detection/tracking/regression/image-quality metrics, 8 dataset adapters |
-| `yscv-onnx` | 122-op ONNX CPU runtime, INT4/INT8 quantization, LLM generation (KV-cache, RoPE, GQA), graph optimizer, Metal/MPSGraph GPU |
+| `yscv-onnx` | 122 op ONNX CPU runtime, INT4/INT8 quantization, LLM generation (KV-cache, RoPE, GQA), graph optimizer, Metal/MPSGraph GPU |
 | `yscv-pipeline` | TOML-driven multi-accelerator dispatch (CPU / RKNN / MPSGraph / GPU), RT wiring, recovery, hot-reload |
 | `yscv-video-mpp` | Rockchip MPP hardware encoder integration (H.264, H.265) |
 | `yscv-cli` | Inference + evaluation CLI: camera diagnostics, dataset eval, pipeline runner |

apps/llm-bench/COMPARISON.md

@@ -55,6 +55,24 @@ Fresh desktop recheck on 2026-04-29, median of 3 runs:
 | yscv +`--int4-weights 32 --kv-dtype i8` |       1929 |        4568 |         3.50 |       6.40× |
 | ORT 1.25, fp32, 12 thr                |           69 |         714 |        22.40 |          1× |
 
+Nix ORT recheck on 2026-05-05 used `onnxruntime` 1.24.4 and 8 decode
+tokens. The yscv row includes the load-time constant-Transpose fold for
+`Transpose(initializer) -> MatMul RHS` and the prepacked-B slice GEMM route
+for rank-3 MatMul:
+
+| run | threads | prefill (ms) | decode tok/s |
+|---|---:|---:|---:|
+| yscv fp32 | 1 | 1223.6 | 8.44 |
+| yscv fp32 | 6 | 1176.8 | 8.46 |
+| yscv +`--int4-weights 32` | 1 | 1982.3 | 10.68 |
+| yscv +`--int4-weights 32` | 6 | 1987.0 | 10.84 |
+| ORT 1.24.4 fp32 | 1 | 319.3 | 18.60 |
+| ORT 1.24.4 fp32 | 6 | 64.9 | 25.99 |
+
+Before the fold/prepack fix, the same 8-token yscv fp32 harness was ~2.06
+tok/s and spent hundreds of milliseconds per forward re-transposing
+`model.embed_tokens.weight` for `/lm_head/MatMul`.
+
 The `--int4-weights GS` flag in-process packs every eligible MatMul/Gemm
 weight to symmetric INT4 with the supplied group size and re-runs the
 exact same harness; runtime dispatch routes through

apps/llm-bench/Cargo.toml

@@ -34,6 +34,10 @@ path = "src/bin/quantize_tracker.rs"
 name = "bench_tracker"
 path = "src/bin/bench_tracker.rs"
 
+[[bin]]
+name = "nchwc_coverage"
+path = "src/bin/nchwc_coverage.rs"
+
 [dependencies]
 yscv-onnx = { version = "0.1", path = "../../crates/yscv-onnx", default-features = false }
 yscv-tensor = { version = "0.1", path = "../../crates/yscv-tensor" }

apps/llm-bench/README.md

@@ -89,6 +89,8 @@ the existing `yscv_onnx::generate` helper.
 - `--max-tokens N` — number of new tokens to generate (default 64).
 - `--warmup K` — discarded warm-up forward passes before the timed
   run (default 2). Catches first-touch allocs / page faults.
+- `--threads N` — run yscv through `OnnxRunner::with_threads(N)`;
+  `0` keeps the default runner policy.
 - `--input-name NAME` / `--output-name NAME` — override the input
   tensor name (default `input_ids`) / logits output name (default
   `logits`) for non-standard exports.
@@ -178,10 +180,12 @@ Single human-readable line on stderr plus a JSON line on stdout:
 
 ```
 prompt=32t  prefill=215.4ms  decode=64t / 4823.2ms = 13.27 tok/s  total=5039.2ms
-{"model":"…/model.onnx","prompt_tokens":32,"decode_tokens":64,"prefill_ms":215.4,"decode_total_ms":4823.2,"decode_tokens_per_sec":13.27,"total_wall_ms":5039.2}
+{"model":"…/model.onnx","threads":6,"prompt_tokens":32,"decode_tokens":64,"prefill_ms":215.4,"decode_total_ms":4823.2,"decode_tokens_per_sec":13.27,"total_wall_ms":5039.2}
 ```
 
 Pipe stdout through `jq` to aggregate across runs.
+Set `YSCV_RUNNER_PROFILE=/tmp/llm-profile.json` to dump the same per-node
+runner profile schema used by `bench_tracker`.
 
 ## Tracker Multi-Input Quantization
 
@@ -260,14 +264,23 @@ QLinearConv and QuantizeLinear outputs are stored internally as real i8 tensors;
 `bench_tracker` reports `quant_i8_stores` and `quant_i8_materializations` so a
 run can prove whether it stayed quantized between Conv-like nodes. Entry
 `QuantizeLinear` nodes that feed QLinear activations quantize directly into i8
-storage through the scalar/AVX2/AVX-512F/NEON dispatch in `yscv-kernels` with packed x86 i8 stores; the
-old scalar iterator path is gone from that hot route. For explicit QLinear
-pointwise/MatMul shapes, yscv also builds the AVX-512 VNNI 4x16 RHS layout at
-model load so repeated tracker runs do not repack the same INT8 weights in the
-hot path. The benchmark JSON includes `quant_chain_candidates`,
-`quant_chain_executed`, and `quant_chain_fallback`; until fused INT8 chain
-actions land, the private tracker QLinear export should show candidates but
-zero executed chains.
+storage through the scalar/AVX2/AVX-512F/NEON dispatch in `yscv-kernels` with
+packed x86 i8 stores; large activation tensors split into per-thread chunks on
+multi-threaded runners, and the old scalar iterator path is gone from that hot
+route. Residual/fork QLinear chains also parallelize INT8 depthwise
+accumulation, NCHW/NHWC layout conversion, and requant/dequant glue. For
+explicit QLinear pointwise/MatMul shapes, yscv also builds the AVX-512 VNNI
+4x16 RHS layout at model load so repeated tracker runs do not repack the same
+INT8 weights in the hot path. The benchmark JSON includes `quant_chain_candidates`,
+`quant_chain_executed`, and `quant_chain_fallback`; the candidate count follows
+the same greedy non-overlapping ownership as the runtime plan, including
+PW->DW / DW->PW kernels, residual/fork suffix actions, and split
+`QLinearConv -> DQ` boundaries. On the private tracker QLinear export the
+expected Chunk 4 steady state is `quant_chain_candidates=50`,
+`quant_chain_executed=50`, `quant_chain_fallback=0`, and
+`qlinear_conv_fast=0`, meaning no `QLinearConv` is left to run as an
+individual per-node op. The current 3x200 QLinear gate on the private tracker
+is 27.63 ms at 1T, 11.67 ms at 4T, and 10.87 ms at 6T.
 
 To reproduce the tracker comparison matrix while tuning quant kernels:
 

apps/llm-bench/src/bin/bench_tracker.rs

@@ -143,7 +143,11 @@ fn qlinear_boundary_after(nodes: &[OnnxNode], start: usize, input: &str) -> Opti
 fn quant_chain_candidates(model: &OnnxModel) -> usize {
     let nodes = &model.nodes;
     let mut count = 0usize;
+    let mut skip = vec![false; nodes.len()];
     for i in 0..nodes.len() {
+        if skip[i] {
+            continue;
+        }
         let Some(kind) = qlinear_conv_kind(model, &nodes[i]) else {
             continue;
         };
@@ -157,22 +161,40 @@ fn quant_chain_candidates(model: &OnnxModel) -> usize {
             && ((kind == "pw" && next_kind == "dw") || (kind == "dw" && next_kind == "pw"))
         {
             count += 1;
+            for idx in i + 1..=q_idx + 1 {
+                skip[idx] = true;
+            }
             continue;
         }
-        if let Some((dq_idx, q_idx)) = qlinear_boundary_after(nodes, i + 1, qconv_out) {
-            let relu_offset = if nodes.get(dq_idx + 1).is_some_and(|n| n.op_type == "Relu") {
-                1
-            } else {
-                0
-            };
-            let add_idx = dq_idx + 1 + relu_offset;
-            if nodes.get(add_idx).is_some_and(|n| n.op_type == "Add")
-                && nodes
-                    .get(q_idx)
-                    .is_some_and(|n| n.op_type == "QuantizeLinear")
-            {
-                count += 1;
+
+        let dq_idx = i + 1;
+        let relu_idx = i + 2;
+        let conv_idx = i + 3;
+        let add_idx = i + 4;
+        let q_idx = i + 5;
+        if nodes
+            .get(dq_idx)
+            .is_some_and(|n| n.op_type == "DequantizeLinear" && n.inputs.first() == Some(qconv_out))
+            && nodes.get(relu_idx).is_some_and(|n| n.op_type == "Relu")
+            && nodes.get(conv_idx).is_some_and(|n| n.op_type == "Conv")
+            && nodes.get(add_idx).is_some_and(|n| n.op_type == "Add")
+            && nodes
+                .get(q_idx)
+                .is_some_and(|n| n.op_type == "QuantizeLinear")
+        {
+            count += 1;
+            for idx in dq_idx..=q_idx {
+                skip[idx] = true;
             }
+            continue;
+        }
+
+        if nodes
+            .get(dq_idx)
+            .is_some_and(|n| n.op_type == "DequantizeLinear" && n.inputs.first() == Some(qconv_out))
+        {
+            count += 1;
+            skip[dq_idx] = true;
         }
     }
     count

apps/llm-bench/src/bin/inspect.rs

@@ -46,5 +46,16 @@ fn main() {
                 println!("  [graph input or external]");
             }
         }
+        println!("outputs:");
+        for out in &node.outputs {
+            println!("  {out}");
+            for consumer in m
+                .nodes
+                .iter()
+                .filter(|n| n.inputs.iter().any(|inp| inp == out))
+            {
+                println!("    -> {} op={}", consumer.name, consumer.op_type);
+            }
+        }
     }
 }