ane: Phase 0 decision gate — all three unknowns resolved favorably

Phase 0 of the ANE offload strategy scoped in docs/2026-04-11-ane-offload-scoping.md.
Adds two measurement harnesses under ane_bench/ and a README documenting
the methodology and results.

Harnesses:
- ane_dispatch_bench.m: loads one pre-compiled LUT4 super-block from
  anemll-qwen35 and runs 200 back-to-back predictionFromFeatures: calls,
  reporting min/p50/p90/p99/max latency.
- ane_transfer_bench.m: 10k trials each of four MLMultiArray/MTLBuffer
  interaction patterns (zero-copy wrap, alloc+memcpy, readback, alloc only).

Results on M4 Pro mini-01 (2026-04-11):

Unknown danveloper#1 — Swift CoreML per-prediction dispatch overhead:
  p50: 6.642 ms per super-block (4 layers bundled: DDDA)
  p99: 7.444 ms, max: 7.586 ms across 200 calls
  This is FASTER than the anemll-qwen35 reference of 9.28 ms (their
  measurement included Python coremltools overhead; raw Obj-C is tighter).
  No thermal throttling visible.

Unknown danveloper#2 — MTLBuffer ↔ MLMultiArray transfer cost:
  Zero-copy wrap (initWithDataPointer): 0.5 us mean, p99 1.07 us
  Alloc + memcpy (naive fallback): 1.05 us mean, p99 4.05 us
  All paths sub-5-microseconds. Per-token overhead across 45 layer
  transitions = ~22 us total = completely negligible.

Unknown danveloper#3 — GPU + ANE simultaneous load contention:
  Ran ane_dispatch_bench concurrent with TQ_KV=1 ./infer --tokens 128.
  ANE under concurrent GPU load: p50 +4%, p90 +5%, p99 +14%.
  GPU inference: 6.05 tok/s vs historical 5.65-5.91 tok/s baseline at
  TQ_KV=1 128tok — within noise.
  Unified memory bandwidth is not a bottleneck for the two engines.

Verdict: ALL THREE DECISION GATES PASSED. Proceed to Phase 1.

Revised per-token budget for the full ANE offload port:
  ANE path: 15 super-blocks × ~6.97 ms (with concurrent penalty) = ~104 ms
  GPU path: 60 MoE dispatches in parallel = ~78 ms
  Wall clock: max(104, 78) = ~104 ms/token
  vs current warm-cache baseline: ~150 ms/token
  Expected speedup: ~31% wall-clock

This is materially better than the scoping doc's 10-15% estimate. The
scoping doc was written with conservative assumptions about CoreML
dispatch overhead; the actual measurement is 2.6 ms *below* the reference,
not above it.

superblock0.mlmodelc/ bundle (413 MB) is gitignored; reproduce via:
  rsync -a carl@192.168.0.62:/Users/carl/models/anemll-qwen3.5-9b/\
  qwen3_5_superblock0_lut4.mlmodelc/ ane_bench/superblock0.mlmodelc/

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: Herbie

.gitignore

@@ -7,6 +7,11 @@ metal_infer/bench_pread
 *.air
 *.metallib
 
+# ANE bench artifacts (mlmodelc bundles are 400+ MB and rebuildable from anemll-qwen35)
+ane_bench/ane_*_bench
+ane_bench/*.mlmodelc/
+ane_bench/*.mlpackage/
+
 # Generated model files (recreate with scripts — see CLAUDE.md Quick Start)
 metal_infer/model_weights.bin
 metal_infer/model_weights.json

ane_bench/README.md

@@ -0,0 +1,158 @@
+# ANE bench — Phase 0 decision gate for the linear-attention offload
+
+This directory holds the measurement harnesses for the **Phase 0** decision
+gate of the ANE offload strategy scoped in
+`docs/2026-04-11-ane-offload-scoping.md`.
+
+## Goal
+
+Before committing to the full ANE port (9-15 focused days across 5 phases),
+measure the three critical unknowns that could kill the strategy:
+
+1. **Swift / Obj-C CoreML per-prediction dispatch overhead** — if `MLModel
+   predictionFromFeatures:` costs several ms of overhead above the raw
+   kernel time, the 45-per-token call count makes the strategy infeasible.
+2. **MTLBuffer ↔ MLMultiArray transfer cost** — hidden state moves between
+   Metal and CoreML every layer transition; if this isn't zero-copy, we pay
+   45× per token.
+3. **GPU + ANE simultaneous workload contention** — flash_moe keeps the GPU
+   busy with MoE expert dispatch; if ANE and GPU contend on shared DRAM
+   bandwidth, the parallel gain evaporates.
+
+## Harnesses
+
+### `ane_dispatch_bench.m`
+
+Loads one pre-compiled LUT4 super-block (from `anemll-qwen35`) and runs 200
+back-to-back predictions with warmup, computing min / p50 / p90 / p99 / max
+latency. Compares against the `anemll-qwen35` reference of 9.28 ms pure-ANE
+time for super-block 0.
+
+### `ane_transfer_bench.m`
+
+Measures four MLMultiArray/MTLBuffer interaction patterns 10,000 times each:
+- Zero-copy wrap via `initWithDataPointer:` (the intended production path)
+- Alloc fresh + memcpy (naive fallback)
+- Readback memcpy (MLMultiArray → MTLBuffer)
+- Alloc only (isolates the alloc cost)
+
+## Build
+
+```bash
+clang -O2 -fobjc-arc -framework Foundation -framework CoreML \
+      ane_dispatch_bench.m -o ane_dispatch_bench
+
+clang -O2 -fobjc-arc -framework Foundation -framework CoreML -framework Metal \
+      ane_transfer_bench.m -o ane_transfer_bench
+```
+
+## Model bundle
+
+`superblock0.mlmodelc/` is copied from
+`carl@192.168.0.62:~/models/anemll-qwen3.5-9b/qwen3_5_superblock0_lut4.mlmodelc/`.
+It is gitignored (413 MB). To reproduce:
+
+```bash
+rsync -a carl@192.168.0.62:/Users/carl/models/anemll-qwen3.5-9b/qwen3_5_superblock0_lut4.mlmodelc/ \
+      ./superblock0.mlmodelc/
+```
+
+## Results — 2026-04-11, M4 Pro mini-01
+
+### Dispatch overhead (ane_dispatch_bench, solo)
+
+```
+Model load: 5273.1 ms (one-shot)
+Warmup converges at call 3 to ~6.7 ms.
+
+200 back-to-back predictions:
+  min:  6.567 ms
+  p50:  6.642 ms
+  p90:  6.861 ms
+  p99:  7.444 ms
+  max:  7.586 ms
+  mean: 6.706 ms
+  throughput: 149.1 predictions/sec
+```
+
+**Verdict: VIABLE.** p50 is *faster* than the `anemll-qwen35` reference of
+9.28 ms (their measurement included Python `coremltools` overhead; raw Obj-C
+is tighter). No thermal throttling visible across 200 tight calls.
+
+### Transfer cost (ane_transfer_bench)
+
+```
+Strategy A — zero-copy wrap (initWithDataPointer):
+  min=0.000 us  p50=0.000 us  p99=1.073 us  mean=0.491 us
+
+Strategy B — alloc + memcpy:
+  min=0.000 us  p50=0.954 us  p99=4.053 us  mean=1.050 us
+
+Strategy C — readback memcpy:
+  min=0.000 us  p50=0.000 us  p99=1.073 us  mean=0.131 us
+
+Strategy D — alloc only:
+  min=0.000 us  p50=0.000 us  p99=1.073 us  mean=0.283 us
+```
+
+**Verdict: free.** All paths are sub-5-microseconds. Per-token overhead
+across 45 layer transitions = ~22 us (~0.02 ms), negligible versus the
+~33 ms/token expected win.
+
+### GPU + ANE concurrent load
+
+Ran `ane_dispatch_bench` concurrently with `TQ_KV=1 ./infer --tokens 128`
+(flash_moe normal warm-cache inference).
+
+```
+ANE under concurrent GPU load:
+  p50: 6.902 ms  (+4% vs solo)
+  p90: 7.207 ms  (+5%)
+  p99: 8.516 ms  (+14%)
+  max: 10.002 ms (+32% tail)
+
+GPU inference alongside ANE bench:
+  Generation: 6.05 tok/s (TQ_KV=1, 128 tok)
+  Historical baseline at same config: 5.65-5.91 tok/s
+  → GPU is unchanged / slightly faster
+```
+
+**Verdict: no meaningful contention** on Apple M4 Pro unified memory. ANE and
+GPU can run in parallel at ~95% of their solo rates for the hot path. Tail
+latency is worse but still well within budget (10 ms worst case = 1.8 ms
+above reference).
+
+## Combined Phase 0 conclusion
+
+**All three decision gates passed. ANE offload proceeds to Phase 1 (weight
+conversion).**
+
+Revised per-token budget estimate for the full port:
+```
+ANE path (15 super-blocks × 6.64 ms p50 warm + ~5% concurrent penalty):
+  = 15 × ~6.97 ms  = ~104 ms/token
+GPU path (60 MoE dispatches in parallel):
+  = ~78 ms/token
+Transitions (45 × ~0.5 us):
+  = ~0.02 ms/token negligible
+Wall clock:
+  = max(104, 78) + 0.02  = ~104 ms/token
+Vs current warm-cache baseline (~150 ms/token):
+  = ~31% wall-clock speedup
+```
+
+Conservative estimate because:
+- Super-block 0 is the first super-block and may be faster/slower than
+  average; need to measure the full 15-super-block sum once we have them
+- flash_moe's 2× value-head count may increase per-super-block time (needs
+  actual shape measurement during Phase 1)
+- Sustained thermal throttling under minutes-long inference not measured
+  (Phase 2 validation)
+
+Non-conservative (optimistic) side:
+- Mini-01 is slightly faster than mini-02 where the reference was measured
+- We use Obj-C directly (no Python overhead)
+- super-block 0 measurements assume full 4-layer bundling; any simpler
+  per-layer .mlpackage split adds overhead only if we need to split
+
+See `../docs/2026-04-11-ane-offload-scoping.md` for the rest of the port plan.

ane_bench/ane_dispatch_bench.m

@@ -0,0 +1,165 @@
+// ane_dispatch_bench.m — ANE Phase 0 decision gate.
+//
+// Measures the wall-clock overhead of calling [MLModel predictionFromFeatures:]
+// at per-layer granularity on a compiled ANE super-block, to decide whether
+// the flash_moe linear-attention offload strategy is viable.
+//
+// Success criterion: p90 per-prediction latency is within ~1 ms of the
+// anemll-qwen35 reported pure-ANE kernel time (9.28 ms for super-block 0).
+// If overhead is >2 ms consistently, the ANE offload strategy is blocked.
+//
+// Build:  clang -O2 -fobjc-arc -framework Foundation -framework CoreML
+//               ane_dispatch_bench.m -o ane_dispatch_bench
+// Run:    ./ane_dispatch_bench [path_to_mlmodelc]
+
+#import <Foundation/Foundation.h>
+#import <CoreML/CoreML.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+static MLMultiArray *makeArray(NSArray<NSNumber *> *shape, NSError **err) {
+    MLMultiArray *a = [[MLMultiArray alloc] initWithShape:shape
+                                                 dataType:MLMultiArrayDataTypeFloat16
+                                                    error:err];
+    if (!a) return nil;
+    // Zero-fill is safe — super-block 0 handles zero input without NaN.
+    memset(a.dataPointer, 0, a.count * sizeof(uint16_t));
+    return a;
+}
+
+static int cmp_double(const void *a, const void *b) {
+    double da = *(const double *)a, db = *(const double *)b;
+    return (da > db) - (da < db);
+}
+
+int main(int argc, const char *argv[]) {
+    @autoreleasepool {
+        const char *path = (argc >= 2) ? argv[1]
+            : "/Users/carl/projects/turbomoe/flash_moe/ane_bench/superblock0.mlmodelc";
+        NSURL *url = [NSURL fileURLWithPath:@(path)];
+
+        NSError *err = nil;
+        MLModelConfiguration *cfg_ane = [[MLModelConfiguration alloc] init];
+        cfg_ane.computeUnits = MLComputeUnitsCPUAndNeuralEngine;
+
+        printf("=== ANE Phase 0 dispatch benchmark ===\n");
+        printf("Model: %s\n", path);
+        printf("Compute units: CPUAndNeuralEngine\n\n");
+
+        // --- 1. Load the model ---
+        double t_load0 = CFAbsoluteTimeGetCurrent();
+        MLModel *model = [MLModel modelWithContentsOfURL:url
+                                          configuration:cfg_ane
+                                                  error:&err];
+        double t_load1 = CFAbsoluteTimeGetCurrent();
+        if (!model) {
+            NSLog(@"load error: %@", err);
+            return 1;
+        }
+        printf("Model load: %.1f ms (one-shot)\n", (t_load1 - t_load0) * 1000.0);
+
+        // --- 2. Build zero-filled inputs matching the super-block signature ---
+        NSMutableDictionary<NSString *, MLMultiArray *> *inputs = [NSMutableDictionary new];
+        #define MK(name, ...) do { \
+            MLMultiArray *__a = makeArray(@[__VA_ARGS__], &err); \
+            if (!__a) { NSLog(@"makeArray %s: %@", name, err); return 1; } \
+            inputs[@name] = __a; \
+        } while(0)
+        MK("hidden_states",   @1, @1, @4096);
+        MK("cos",             @1, @1, @1, @64);
+        MK("sin",             @1, @1, @1, @64);
+        MK("gated_state_0",   @1, @32, @128, @128);
+        MK("conv_state_0",    @1, @3, @8192);
+        MK("gated_state_1",   @1, @32, @128, @128);
+        MK("conv_state_1",    @1, @3, @8192);
+        MK("gated_state_2",   @1, @32, @128, @128);
+        MK("conv_state_2",    @1, @3, @8192);
+        #undef MK
+
+        MLDictionaryFeatureProvider *features =
+            [[MLDictionaryFeatureProvider alloc] initWithDictionary:inputs error:&err];
+        if (!features) {
+            NSLog(@"features error: %@", err);
+            return 1;
+        }
+
+        // --- 3. Warmup (ANE compilation/placement happens on first call) ---
+        printf("\nWarmup (5 calls, first call may cold-compile)...\n");
+        for (int i = 0; i < 5; i++) {
+            double t0 = CFAbsoluteTimeGetCurrent();
+            id<MLFeatureProvider> out = [model predictionFromFeatures:features error:&err];
+            double t1 = CFAbsoluteTimeGetCurrent();
+            if (!out) {
+                NSLog(@"warmup pred %d error: %@", i, err);
+                return 1;
+            }
+            printf("  warmup[%d]: %.3f ms\n", i, (t1 - t0) * 1000.0);
+        }
+
+        // --- 4. Benchmark loop: 200 tight back-to-back predictions ---
+        const int N = 200;
+        double *times = malloc(N * sizeof(double));
+
+        printf("\nBenchmark: %d back-to-back predictions...\n", N);
+        double t_bench0 = CFAbsoluteTimeGetCurrent();
+        for (int i = 0; i < N; i++) {
+            double t0 = CFAbsoluteTimeGetCurrent();
+            id<MLFeatureProvider> out = [model predictionFromFeatures:features error:&err];
+            double t1 = CFAbsoluteTimeGetCurrent();
+            if (!out) {
+                NSLog(@"pred %d error: %@", i, err);
+                return 1;
+            }
+            times[i] = (t1 - t0) * 1000.0;  // ms
+            (void)out;  // discard; ARC releases
+        }
+        double t_bench1 = CFAbsoluteTimeGetCurrent();
+
+        // --- 5. Stats ---
+        double sum = 0;
+        double raw_min = times[0], raw_max = times[0];
+        for (int i = 0; i < N; i++) {
+            sum += times[i];
+            if (times[i] < raw_min) raw_min = times[i];
+            if (times[i] > raw_max) raw_max = times[i];
+        }
+        double mean = sum / N;
+
+        double *sorted = malloc(N * sizeof(double));
+        memcpy(sorted, times, N * sizeof(double));
+        qsort(sorted, N, sizeof(double), cmp_double);
+
+        printf("\n=== Results ===\n");
+        printf("  Wall clock over loop: %.1f ms\n", (t_bench1 - t_bench0) * 1000.0);
+        printf("  Throughput:           %.1f predictions/sec\n",
+               (double)N / (t_bench1 - t_bench0));
+        printf("  Per-prediction:\n");
+        printf("    min:  %.3f ms\n", sorted[0]);
+        printf("    p50:  %.3f ms\n", sorted[N / 2]);
+        printf("    p90:  %.3f ms\n", sorted[(int)(N * 0.90)]);
+        printf("    p99:  %.3f ms\n", sorted[(int)(N * 0.99)]);
+        printf("    max:  %.3f ms\n", sorted[N - 1]);
+        printf("    mean: %.3f ms\n", mean);
+
+        printf("\n=== Decision gate ===\n");
+        printf("  anemll-qwen35 reference pure-ANE time (super-block 0): 9.28 ms\n");
+        printf("  p50 overhead above reference: %+.3f ms\n", sorted[N / 2] - 9.28);
+        printf("  p90 overhead above reference: %+.3f ms\n", sorted[(int)(N * 0.90)] - 9.28);
+        printf("\n");
+        double overhead_p90 = sorted[(int)(N * 0.90)] - 9.28;
+        if (overhead_p90 < 1.0) {
+            printf("  VERDICT: p90 overhead < 1 ms — ANE offload is VIABLE.\n");
+        } else if (overhead_p90 < 2.0) {
+            printf("  VERDICT: p90 overhead 1-2 ms — ANE offload is MARGINAL.\n");
+            printf("  Consider the data-transfer cost on top of this.\n");
+        } else {
+            printf("  VERDICT: p90 overhead > 2 ms — ANE offload is BLOCKED.\n");
+            printf("  Pivot to mixed-bit per-expert quantization (priority #2).\n");
+        }
+
+        free(sorted);
+        free(times);
+    }
+    return 0;
+}