Skip to content

Commit a680856

Browse files
ruvnetruvnet
committed
feat(aether-arena): ADR-149 spatial-intelligence benchmark — scorer + CI harness gate (M1-M4)
AetherArena ("AA") — the official, project-agnostic Spatial-Intelligence Benchmark (ADR-149, Accepted). Iteration 1 of the long-horizon build: - ADR-149 accepted: name locked (ruvnet/aether-arena), v0 metrics locked (pose/presence/latency/determinism), dataset legality resolved (MM-Fi CC BY-NC only; Wi-Pose excluded). Adds four-part framing, threat model, arena_score formula, submission state machine, neutrality/governance, and the §7 acceptance test. - aa_score_runner: deterministic scorer bin reusing the real ruview_metrics pose harness on a fixed seed=42 fixture → RuViewTier-style verdict + cross-platform SHA-256 proof hash. Builds --no-default-features (no torch/GPU). VERDICT: PASS. - CI harness gate: .github/workflows/aether-arena-harness.yml runs the scorer on every PR — the "PR that runs the harness as part of the build" requirement. - Scaffold: aether-arena/{README,VERIFY,STATUS}.md + schema/aa-submission.toml. - Horizon record persisted (.claude-flow/horizons/aether-arena-aa.json). Infra = the deliverable; model SOTA (MM-Fi PCK@20) is a separate effort blocked on ADR-079 data collection, tracked as a stretch goal, not an infra exit. Co-Authored-By: claude-flow <ruv@ruv.net>
1 parent 9ad550d commit a680856

10 files changed

Lines changed: 824 additions & 0 deletions

File tree

‎.claude-flow/horizons/aether-arena-aa.json‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
{
2+
"id": "aether-arena-aa",
3+
"name": "AetherArena (AA) — Official Spatial-Intelligence Benchmark",
4+
"adr": "ADR-149",
5+
"adrPath": "docs/adr/ADR-149-public-community-leaderboard-huggingface.md",
6+
"status": "Accepted",
7+
"initializedDate": "2026-05-30",
8+
"targetDate": "2026-08-31",
9+
"exitCriteria": "Benchmark INFRASTRUCTURE done, tested, CI-gated, deploy-ready: aa_score_runner.rs passes deterministic fixture test; CI harness-gate green on every PR; aether-arena repo scaffold committed (README four-part framing + aa-submission.toml schema + VERIFY.md); public smoke split committed; HF Space lifecycle skeleton deployed; signed Parquet ledger functional; RuView baseline PCK@20 ~2.5% entered; ADR-149 §7 acceptance test (five-step stranger test) passes. NOTE: ML SOTA (MM-Fi PCK@20 ~72%) is a separate long-running stretch goal blocked on ADR-079 camera-ground-truth — it is NOT an infra exit criterion.",
10+
"baselineState": {
11+
"adrStatus": "Accepted, committed 2026-05-30",
12+
"scorerCode": "ruview_metrics.rs + ablation.rs + proof.rs exist in wifi-densepose-train; aa_score_runner.rs not yet created",
13+
"aetherArenaRepo": "does not exist yet — needs user authorization to create ruvnet/aether-arena public repo",
14+
"hfSpace": "does not exist yet — needs HF_TOKEN and user authorization to deploy ruvnet/aether-arena HF Space",
15+
"smokeDataset": "not committed",
16+
"resultsLedger": "not created",
17+
"ruviewBaseline": "PCK@20 ~2.5% self-reported, not formally entered",
18+
"ciGate": "not added to workflow"
19+
},
20+
"milestones": {
21+
"m1": {
22+
"name": "ADR-149 Accepted + committed",
23+
"status": "DONE",
24+
"completedDate": "2026-05-30",
25+
"completionCriteria": "ADR-149 file committed to docs/adr/ with status Accepted",
26+
"notes": "Done this session. File at docs/adr/ADR-149-public-community-leaderboard-huggingface.md"
27+
},
28+
"m2": {
29+
"name": "Deterministic scorer runner bin (aa_score_runner.rs)",
30+
"status": "NOT_STARTED",
31+
"completionCriteria": "aa_score_runner.rs compiles, runs ruview_metrics on a committed fixture, emits RuViewTier + SHA-256 proof hash, mirrors existing *_proof_runner.rs pattern; cargo test passes",
32+
"estimatedEffort": "3-5 days",
33+
"owner": "wifi-densepose-train crate or new aa-scorer crate"
34+
},
35+
"m3": {
36+
"name": "CI harness-gate: GitHub Actions workflow",
37+
"status": "NOT_STARTED",
38+
"completionCriteria": "A GitHub Actions workflow runs aa_score_runner on every PR as a build gate; PR fails if scorer fails determinism check; workflow committed and green",
39+
"estimatedEffort": "2-3 days",
40+
"dependency": "M2 must be done first"
41+
},
42+
"m4": {
43+
"name": "aether-arena repo scaffold",
44+
"status": "NOT_STARTED",
45+
"completionCriteria": "ruvnet/aether-arena repo created with: README (four-part framing: Public leaderboard / Private eval split / Open scorer / Signed results); aa-submission.toml manifest schema; VERIFY.md (ADR-149 §7 stranger acceptance test); neutrality/governance section (§2.8); contribution guide",
46+
"estimatedEffort": "3-5 days",
47+
"blockers": ["Needs user authorization to create public ruvnet/aether-arena repo on GitHub"]
48+
},
49+
"m5": {
50+
"name": "Public smoke split committed + private MM-Fi held-out split prep",
51+
"status": "NOT_STARTED",
52+
"completionCriteria": "Public smoke split committed to aether-arena repo (stranger can score locally); private MM-Fi held-out split prepared under non-public path with CC BY-NC 4.0 attribution; Wi-Pose explicitly excluded from v0",
53+
"estimatedEffort": "5-7 days",
54+
"riskNotes": "MM-Fi CC BY-NC 4.0: AA must remain non-commercial and carry MM-Fi attribution; raw frames stay in private split; only derived CSI features + scores may be exposed"
55+
},
56+
"m6": {
57+
"name": "HF Space (Gradio) skeleton",
58+
"status": "BLOCKED",
59+
"completionCriteria": "HF Space deployed at ruvnet/aether-arena with submission lifecycle (submitted->validated->quarantined->smoke_scored->full_scored->published/rejected); sandboxed scorer container wired; basic leaderboard table rendered",
60+
"estimatedEffort": "7-10 days",
61+
"blockers": [
62+
"Needs HF_TOKEN — check .env for HF_TOKEN or HUGGINGFACE_TOKEN",
63+
"Needs user authorization to create/deploy ruvnet/aether-arena HF Space (outward-facing public deployment)"
64+
]
65+
},
66+
"m7": {
67+
"name": "Signed append-only Parquet results ledger",
68+
"status": "NOT_STARTED",
69+
"completionCriteria": "HF dataset ruvnet/aether-arena-results created; append-only Parquet ledger with signed rows; determinism_gate enforced; no row can be silently edited",
70+
"estimatedEffort": "3-5 days",
71+
"ledgerSchema": "submitter, model_ref, category, feature_set, tier, pck20, oks, mota, vitals_bpm_err, latency_p50, latency_p95, privacy_leakage, cross_room_deg, proof_sha256, scored_at, harness_version",
72+
"dependency": "M6 must be scaffolded first"
73+
},
74+
"m8": {
75+
"name": "RuView baseline entry + public launch",
76+
"status": "NOT_STARTED",
77+
"completionCriteria": "RuView wifi-densepose-pretrained baseline entered (honest PCK@20 ~2.5%); ADR-149 §7 five-step stranger acceptance test passes; v0 live with Presence + Pose + Edge-latency + Determinism categories active; Privacy and Cross-room shown as gated/coming-soon",
78+
"estimatedEffort": "3-5 days",
79+
"dependency": "M4+M5+M6+M7 complete",
80+
"notes": "ML SOTA improvement (PCK@20 ~72%) is a SEPARATE stretch goal blocked on ADR-079 P7-P9 camera ground truth. NOT a blocker for infra launch."
81+
}
82+
},
83+
"activeMilestone": "m2",
84+
"completedMilestones": ["m1"],
85+
"knownRisks": [
86+
"HF_TOKEN not confirmed present in .env — check before M6 work begins",
87+
"ruvnet/aether-arena public repo creation is outward-facing — needs explicit user authorization",
88+
"MM-Fi CC BY-NC 4.0: AA must stay legally non-commercial and brand-distinct from commercial RuView product; or seek MM-Fi commercial grant before any paid tier",
89+
"Wi-Pose has research-use-only terms (no redistribution grant) — excluded from v0; revisit only if terms are clarified with authors",
90+
"HF Space free CPU tier may be too slow for Candle/tch inference pipeline — may need ZeroGPU or self-hosted scorer on cognitum-20260110 GCloud A100/L4",
91+
"ADR-079 camera-ground-truth (PCK@20 SOTA) is P7-P9 pending — NOT an infra blocker; must not be conflated with AA infra completion",
92+
"Neutrality/governance risk: RuView seeded the scorer — must be demonstrably scored through the same public pipeline as any other entrant (§2.8 controls)"
93+
],
94+
"driftSignals": {
95+
"timeline": "GREEN — just initialized, no timeline pressure yet",
96+
"scope": "GREEN — scope locked at four-part structure per ADR-149 §2 decision",
97+
"approach": "GREEN — reuse pattern (existing ruview_metrics + proof.rs) confirmed in ADR-149",
98+
"dependency": "YELLOW — HF_TOKEN and ruvnet/aether-arena repo authorization are external blockers with unknown ETA",
99+
"priority": "GREEN — active feature branch feat/adr-136-146-streaming-engine in progress; AA infra can proceed in parallel on its own branch"
100+
},
101+
"stretchGoals": {
102+
"sotaML": "MM-Fi PCK@20 SOTA ~72% — separate ML effort blocked on ADR-079 P7-P9 camera-ground-truth data collection; NOT an infra exit criterion",
103+
"privacyAxis": "ADR-145 §10 membership-inference attacker — activate Privacy leaderboard axis once attacker is implemented and published",
104+
"crossRoom": "Multi-room held-out split — activate Cross-room generalization axis",
105+
"multiOrgSteering": "Invite co-maintainers from other projects once >=N external entries land"
106+
},
107+
"sessionHistory": [
108+
{
109+
"date": "2026-05-30",
110+
"type": "initialization",
111+
"accomplished": [
112+
"ADR-149 Accepted and committed to docs/adr/",
113+
"Horizon record initialized in .claude-flow/horizons/aether-arena-aa.json",
114+
"Memory stored in horizons namespace under key horizon-aether-arena-aa",
115+
"Session check-in record stored in horizon-sessions namespace"
116+
]
117+
}
118+
]
119+
}

‎.github/workflows/aether-arena-harness.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: AetherArena harness gate (ADR-149)
2+
3+
# Runs the AetherArena scoring harness as a PR build gate. Every PR that touches
4+
# the scorer, the metrics, or the benchmark scaffold must keep the deterministic
5+
# score hash stable (ADR-149 §2.5 determinism_gate). If the scoring maths changes,
6+
# the hash moves and this gate fails until `expected_score.sha256` is regenerated
7+
# and reviewed — so scorer drift can never land silently.
8+
#
9+
# This is the "a PR that runs the harness as part of the build process" requirement.
10+
11+
on:
12+
pull_request:
13+
paths:
14+
- 'v2/crates/wifi-densepose-train/src/ruview_metrics.rs'
15+
- 'v2/crates/wifi-densepose-train/src/ablation.rs'
16+
- 'v2/crates/wifi-densepose-train/src/bin/aa_score_runner.rs'
17+
- 'aether-arena/**'
18+
- '.github/workflows/aether-arena-harness.yml'
19+
push:
20+
branches: ['feat/adr-149-aether-arena']
21+
workflow_dispatch:
22+
23+
permissions:
24+
contents: read
25+
pull-requests: write
26+
27+
jobs:
28+
harness-gate:
29+
name: Run AA scorer harness (determinism gate)
30+
runs-on: ubuntu-latest
31+
defaults:
32+
run:
33+
working-directory: v2
34+
steps:
35+
- uses: actions/checkout@v4
36+
37+
- name: Install Rust toolchain
38+
run: rustup show && rustc --version
39+
40+
- name: Cache cargo
41+
uses: actions/cache@v4
42+
with:
43+
path: |
44+
~/.cargo/registry
45+
~/.cargo/git
46+
v2/target
47+
key: aa-harness-${{ runner.os }}-${{ hashFiles('v2/Cargo.lock') }}
48+
49+
# 1. Build the pure-Rust scorer (no torch / no GPU → fast PR gate).
50+
- name: Build AA score runner
51+
run: cargo build -p wifi-densepose-train --bin aa_score_runner --no-default-features
52+
53+
# 2. Determinism gate: the committed expected hash must still match. A
54+
# non-zero exit here fails the PR.
55+
- name: Run determinism gate
56+
run: cargo run -q -p wifi-densepose-train --bin aa_score_runner --no-default-features
57+
58+
# 3. Emit the score row into the PR run summary (leaderboard-ledger shape).
59+
- name: Score row → job summary
60+
if: always()
61+
run: |
62+
ROW=$(cargo run -q -p wifi-densepose-train --bin aa_score_runner --no-default-features -- --json)
63+
{
64+
echo "## AetherArena harness gate"
65+
echo ""
66+
echo "Deterministic score row (ADR-149 §2.2):"
67+
echo '```json'
68+
echo "$ROW"
69+
echo '```'
70+
echo ""
71+
echo "If the determinism gate failed, the scoring maths changed: regenerate with"
72+
echo '`cargo run -p wifi-densepose-train --bin aa_score_runner --no-default-features -- --generate-hash > aether-arena/fixtures/expected_score.sha256` and review the diff.'
73+
} >> "$GITHUB_STEP_SUMMARY"

‎aether-arena/README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# AetherArena ("AA") — The Official Spatial-Intelligence Benchmark
2+
3+
> **Public leaderboard. Private evaluation split. Open scorer. Signed results.**
4+
5+
AetherArena is a **standalone, project-agnostic benchmark** for camera-free **spatial intelligence** — pose, presence, occupancy, tracking, and vitals from RF/WiFi (and, over time, mmWave / UWB / radar / lidar / multimodal). It is **not** a single-vendor leaderboard: any team, framework, or sensing modality can enter, and every entrant — including the RuView baseline that donated the seed scorer — is scored by the identical, open, pinned harness.
6+
7+
Specified in [ADR-149](../docs/adr/ADR-149-public-community-leaderboard-huggingface.md) (Accepted).
8+
9+
Canonical home: **`ruvnet/aether-arena`** + a Hugging Face Space (deploy pending — see `STATUS`).
10+
11+
---
12+
13+
## Why
14+
15+
WiFi/RF spatial sensing has no shared yardstick — papers self-report against inconsistent splits and metrics, with **no accounting for latency, reproducibility, or privacy leakage**. AA fixes the *measurement*, not just the models: a single deterministic scorer, a private held-out split nobody can train on, and a signed result ledger that can't be silently edited.
16+
17+
## What gets measured (v0)
18+
19+
| Category | Metric | Status |
20+
|----------|--------|--------|
21+
| **Pose** | PCK@0.2 (all / torso), OKS | Ranked |
22+
| **Presence** | accuracy, FP/FN | Ranked |
23+
| **Edge latency** | p50 / p95 / p99 ms | Ranked |
24+
| **Determinism** | proof-hash pass/fail | Ranked (gate) |
25+
| Tracking (MOTA) | — | activates when multi-person clips land |
26+
| Vitals (BPM err) | — | activates when paired vitals ground truth lands |
27+
| **Privacy leakage** | membership-inference ∈ [0,1] | **gated — not ranked** until the attacker ships |
28+
| Cross-room | degradation ratio | coming soon |
29+
30+
The headline rank is the **category metric**; an optional `arena_score = quality × latency_factor × privacy_factor × determinism_gate` is exposed alongside (never instead) so accuracy can't win at any cost. See ADR-149 §2.5.
31+
32+
## How scoring works
33+
34+
The scorer is RuView's **already-published** `wifi-densepose-train` acceptance harness (`ruview_metrics` + ADR-145 `ablation`), run in a pinned sandbox. **You submit a model, not predictions** — predictions on data you hold prove nothing. Your model is scored against a **private** MM-Fi held-out split (CC BY-NC 4.0; Wi-Pose excluded for redistribution reasons), and one **signed, append-only** row is written to the results ledger with a determinism proof hash.
35+
36+
Submission lifecycle: `submitted → validated → quarantined → smoke_scored → full_scored → published` (or `rejected` with a reason). The model only ever runs inside a no-network, read-only-FS sandbox.
37+
38+
## Submit (when the Space is live)
39+
40+
1. Write a manifest: [`schema/aa-submission.toml`](schema/aa-submission.toml).
41+
2. Push your model artifact (`.safetensors` / `.rvf` / LoRA adapter) + manifest to the Space.
42+
3. Watch it move through the lifecycle; your signed row appears on the board.
43+
44+
## Verify it's fair (you don't have to trust us)
45+
46+
See [`VERIFY.md`](VERIFY.md) — run the **open scorer** locally on the **public smoke split**, reproduce the determinism hash, and confirm RuView's own entries were scored by the identical path. That five-step check is the launch gate (ADR-149 §7).
47+
48+
## Neutrality
49+
50+
AA is a neutral commons. The scorer is open and versioned; any metric change is a public `harness_version` bump that **re-scores all entries**. RuView donated the seed harness and enters as one baseline — it gets no special treatment (ADR-149 §2.8).

‎aether-arena/STATUS.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# AetherArena — Build Status
2+
3+
Tracks ADR-149 implementation milestones. "Complete" = benchmark **infrastructure** done,
4+
tested, CI-gated, deploy-ready, RuView baseline entered, §7 acceptance test passing.
5+
Model **SOTA** (e.g. MM-Fi PCK@20 ~72%) is a separate long-running ML effort, blocked on
6+
ADR-079 camera-ground-truth collection — *not* an infra-completion blocker.
7+
8+
| # | Milestone | Status |
9+
|---|-----------|--------|
10+
| M1 | ADR-149 Accepted + committed | ✅ done |
11+
| M2 | Deterministic scorer runner (`aa_score_runner`) → tier + proof hash | ✅ done — builds `--no-default-features`, hash stable, VERDICT: PASS |
12+
| M3 | CI harness-gate workflow (PR runs the scorer) | ✅ done — `.github/workflows/aether-arena-harness.yml` |
13+
| M4 | Scaffold: README + submission schema + VERIFY (acceptance test) | ✅ done |
14+
| M5 | Public smoke split (committed) + private MM-Fi held-out split prep | ⏳ next |
15+
| M6 | HF Space (Gradio) submission flow + sandboxed scorer container | ⛔ blocked — needs HF token / maintainer authorization to deploy |
16+
| M7 | Signed append-only Parquet results ledger | ⏳ |
17+
| M8 | RuView baseline entry (honest PCK@20) + public launch | ⏳ |
18+
19+
## Blockers / decisions needed
20+
- **HF deploy (M6)** needs an HF token and authorization to create the public `ruvnet/aether-arena` Space.
21+
- **MM-Fi is CC BY-NC** → AA must stay non-commercial / legally distinct from the commercial RuView product.
22+
- **Realism of M2 fixture**: current fixture is a *determinism* fixture (stable hash), not a realistic baseline; M5 swaps in real MM-Fi held-out scoring.

‎aether-arena/VERIFY.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Verifying AetherArena (you don't have to trust us)
2+
3+
AA's credibility rests on a stranger being able to reproduce a score and see that the rules are fair. This is the **launch gate** (ADR-149 §7): v0 does not ship until all five checks below pass for someone with no insider access.
4+
5+
## The open scorer
6+
7+
The scoring engine is a pure-Rust, GPU-free binary: `aa_score_runner` in `wifi-densepose-train`. It runs the real `ruview_metrics` pose-acceptance harness on a fixed fixture and emits a cross-platform-stable SHA-256 **determinism proof**.
8+
9+
### Reproduce the determinism hash locally
10+
11+
```bash
12+
cd v2
13+
# Verify the committed expected hash still matches (this is the CI gate):
14+
cargo run -q -p wifi-densepose-train --bin aa_score_runner --no-default-features
15+
# → prints the score, the proof sha256, and "VERDICT: PASS"
16+
17+
# See the leaderboard-ledger row as JSON:
18+
cargo run -q -p wifi-densepose-train --bin aa_score_runner --no-default-features -- --json
19+
```
20+
21+
The expected hash is committed at [`fixtures/expected_score.sha256`](fixtures/expected_score.sha256). Same harness version + same fixture → same hash on glibc / MSVC / Apple. If your local run prints `VERDICT: PASS`, you have reproduced the scorer.
22+
23+
### What happens if the scoring maths changes
24+
25+
Any edit to `ruview_metrics.rs`, `ablation.rs`, or `aa_score_runner.rs` moves the hash and **fails the CI gate** (`.github/workflows/aether-arena-harness.yml`) until the maintainer regenerates and reviews:
26+
27+
```bash
28+
cargo run -p wifi-densepose-train --bin aa_score_runner --no-default-features -- --generate-hash \
29+
> aether-arena/fixtures/expected_score.sha256
30+
```
31+
32+
So a scorer change is always a reviewed, public diff — never silent. That's `harness_version` pinning + `determinism_gate` in action (ADR-149 §2.4–§2.5).
33+
34+
## The five-step acceptance test (v0 launch gate)
35+
36+
A stranger must be able to:
37+
38+
1. **Submit** a model (artifact + `schema/aa-submission.toml`) with no insider help.
39+
2. **Get a deterministic score** — same model + same `harness_version` → same numbers.
40+
3. **See the signed row** appended to the public results ledger.
41+
4. **Rerun the scorer locally** on the public smoke split and reproduce the logic (the command above).
42+
5. **Understand why the rank is fair** — private split, open scorer, pinned version, proof hash — from these docs alone.
43+
44+
If any step fails, v0 is not ready.
45+
46+
## Current status
47+
48+
- ✅ Step 4 (rerun the open scorer locally, reproduce the hash) — **works today** via `aa_score_runner`.
49+
- ✅ CI harness gate runs the scorer on every PR.
50+
- ⏳ Steps 1–3, 5 (HF Space submission flow + signed ledger) — in progress; require the HF Space deploy (needs an HF token / maintainer authorization).

‎aether-arena/fixtures/expected_score.sha256‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
dee374bb4ba22bc4583e8280f9d03567d0e174c7e7aac8664a804bb34a428b0e

0 commit comments

Comments
 (0)