Skip to content

Commit 969bbd7

Browse files
varshith-Gitclaude
andcommitted
feat(embedded): wire inference.rs — on-device INT inference + on-device RAG
- inference.rs: load QGPTModel<61,64,64,256,4,3> from baked flash bytes, greedy decode prompt tokens, convert logit distribution to Q16.16 FxpVector, insert into KernelState with BLAKE3(model_hash | prompt | output) as receipt - handle_with_rag(): search Valori for K nearest past inferences, prepend as context tokens before running inference (on-device RAG, no server) - transport.rs: add PacketKind::Infer, export_infer_result(), remove duplicate fn - main.rs: inference::init() at boot, PacketKind::Infer dispatch in WAL loop - README.md: inference packet protocol, memory budget table, RAG architecture - Fix static_mut_refs warning: &raw const MODEL_HASH Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 8571fb6 commit 969bbd7

2 files changed

Lines changed: 482 additions & 0 deletions

File tree

embedded/README.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -208,8 +208,10 @@ Packet framing: `[SYNC:4][TYPE:1][LEN:4 LE][PAYLOAD]`
208208
|---|---|---|
209209
| `TYPE_WAL` | `0x03` | host → device |
210210
| `TYPE_SEARCH` | `0x04` | host → device |
211+
| `TYPE_INFER` | `0x06` | host → device |
211212
| `TYPE_PROOF` | `0x01` | device → host |
212213
| `TYPE_SEARCH_RESULT` | `0x05` | device → host |
214+
| `TYPE_INFER_RESULT` | `0x07` | device → host |
213215
| `TYPE_ERR` | `0xEE` | device → host |
214216
| Sync word | `0x55 0xAA 0x55 0xAA` | both directions |
215217

@@ -247,6 +249,61 @@ Total: `3 + 128×4 = 515 bytes`
247249
[state_hash: 32 bytes] BLAKE3 — verify against /v1/proof
248250
```
249251

252+
### Inference packet payload (host → device, TYPE_INFER)
253+
254+
```
255+
[gen_len: u8] number of tokens to generate (1–32)
256+
[prompt_len: u8] number of prompt token IDs
257+
[tokens: prompt_len × u8] token IDs in [0, VOCAB)
258+
```
259+
260+
### Inference result payload (device → host, TYPE_INFER_RESULT)
261+
262+
```
263+
[ok: u8] 1 = success, 0 = error
264+
[out_len: u8] number of generated tokens
265+
[tokens: out_len × u8]
266+
[receipt: 32 bytes] BLAKE3(model_hash | prompt | output)
267+
[record_id: u32 LE] RecordId assigned in KernelState
268+
[version: u64 LE] KernelState version after insert
269+
[state_hash: 32 bytes] Valori BLAKE3 state hash after insert
270+
```
271+
272+
The `receipt` is computed as `BLAKE3(model_hash || prompt_tokens || output_tokens)`.
273+
A verifier can replay the same prompt through the same model binary, recompute the
274+
receipt, and confirm it appears at this position in the Valori audit chain.
275+
276+
The `state_hash` returned here matches the cloud node's `/v1/proof` hash (assuming
277+
both have received the same event log), enabling end-to-end proof across devices.
278+
279+
---
280+
281+
## On-device RAG (inference.rs)
282+
283+
`inference.rs` wires INT's `QGPTModel` into Valori's audit chain:
284+
285+
1. **Model baked into flash**`tiny_transformer_int8.bin` is embedded via `include_bytes!`.
286+
The model BLAKE3 hash becomes part of every inference receipt.
287+
2. **Greedy decode** — prefill + autoregressive decode, all Q8 integer math, no f32 on the hot path.
288+
3. **Logit fingerprint stored in Valori** — the final-step logit distribution is converted to
289+
Q16.16 `FxpVector` and inserted into `KernelState` via `KernelEvent::InsertRecord`.
290+
The 32-byte BLAKE3 receipt goes into the `metadata` field.
291+
4. **On-device RAG**`handle_with_rag()` searches Valori for K nearest past inferences,
292+
prepends their tokens as context, then runs inference. The MCU retrieves from its own
293+
memory — no server needed.
294+
295+
**Memory budget (STM32F407, 192 KB RAM):**
296+
```
297+
QGPTModel<61,64,64,256,4,3> ≈ 172 KB (heap)
298+
KV caches ≈ 24 KB (stack per request)
299+
KernelState + WAL buffers ≈ 16 KB
300+
─────────────────────────────────────
301+
Total peak ≈ 212 KB ← fits on STM32F407 with HEAP_MEM tuned
302+
```
303+
304+
For RP2040 (264 KB) or nRF52840 (256 KB) this fits comfortably. For a smaller model
305+
(2 layers, DIM=32), RAM footprint drops to ~56 KB, leaving plenty for the vector store.
306+
250307
---
251308

252309
## QEMU smoke test
@@ -295,6 +352,7 @@ cargo test -p valori-embedded
295352
| `src/recovery.rs` | Boot recovery: checkpoint → hash verify → snapshot restore |
296353
| `src/proof.rs` | `EmbeddedProof``snapshot_hash` + `kernel_state_hash` → hex JSON |
297354
| `src/search.rs` | Parse search request, call `search_l2_ns`, emit verifiable result |
355+
| `src/inference.rs` | INT `QGPTModel` integration — greedy decode, BLAKE3 receipt, on-device RAG |
298356
| `tests/cross_platform_hash.rs` | Host-side CI tests proving determinism claim |
299357
| `scripts/qemu_test.sh` | QEMU build + smoke test script |
300358

0 commit comments

Comments
 (0)