|
1 | 1 | # Atome LM |
2 | 2 |
|
3 | | -> **The smallest language model that talks.** |
4 | | -> **No GPU. No cloud. No RAM.** Runs on a $2 microcontroller. |
5 | | -
|
6 | | -A tiny ternary language model designed for the Atome C99 inference engine. Three-pathway router architecture, byte-level tokenizer, integer-arithmetic forward pass, ships in under 100 KB of flash at default config — small enough for a Cortex-M0, an RP2040, an ESP32-C3. |
7 | | - |
8 | | -**Try it now:** [live browser demo at atomelm.com](https://atomelm.com/demo.html) (no install, no signup). |
9 | | - |
10 | | -**Get the kit:** training code, C engine, MCU firmware, benchmarks, paper — all in this repository, released under the [Apache 2.0 License](LICENSE). Train your own checkpoint with `scripts/train_demo.py` in ~30 min on a CPU. |
11 | | - |
12 | | -Trained model weights are not part of this open-source release. Commercial integration support — MCU bring-up, the Atome Secure Boot Pack, per-platform hardening — is available separately at [atomelm.com](https://atomelm.com). |
| 3 | +> A reference implementation of a routed-ternary tiny language model with a |
| 4 | +> bit-exact Python ↔ C99 inference engine, sized for microcontroller-class RAM |
| 5 | +> budgets. |
| 6 | +
|
| 7 | +60K-default-parameter LM combining three known ideas into one open kit: |
| 8 | +ternary weights (after [BitNet b1.58](https://arxiv.org/abs/2402.17764)), |
| 9 | +a hybrid SSM + sparse-attention + local-conv block routed per token |
| 10 | +(after [Hymba](https://arxiv.org/abs/2411.13676) and |
| 11 | +[MossNet](https://arxiv.org/abs/2510.26182)), |
| 12 | +and a byte tokenizer at super-tiny scale |
| 13 | +(after [Guertler 2024](https://arxiv.org/abs/2405.14159)). |
| 14 | +**The contribution is integration, not architecture**: a complete train → |
| 15 | +ternary export → base-3 packing → C99 inference path, with bit-exact |
| 16 | +Python ↔ C parity enforced by tests. |
| 17 | + |
| 18 | +**Get the kit:** training code, C engine, benchmarks, paper — all in this |
| 19 | +repository, released under the [Apache 2.0 License](LICENSE). Train your |
| 20 | +own checkpoint with `scripts/train_demo.py` in ~30 min on a CPU. |
| 21 | + |
| 22 | +**MCU status:** QEMU ARM (Cortex-M3, MPS2-AN385) parity passes to FP32 |
| 23 | +epsilon. **Silicon bring-up is not done in this repository.** For real |
| 24 | +device deployment we sell integration — silicon bring-up, the Atome Secure |
| 25 | +Boot Pack, per-platform hardening — at [atomelm.com](https://atomelm.com). |
| 26 | +Trained model weights are not part of this open-source release. |
13 | 27 |
|
14 | 28 | --- |
15 | 29 |
|
16 | | -## Frontier finding |
| 30 | +## Reproducible result, narrow regime |
17 | 31 |
|
18 | | -At fixed parameter count, Atome's 3-pathway ternary architecture beats a |
19 | | -vanilla GPT-FP32 baseline by **22 % on perplexity** (6.31 vs 8.12 ppl) |
20 | | -while using **16× less disk** (15.1 KB vs 237.5 KB). At fixed flash |
21 | | -budget, Atome wins by **52 %** (6.31 vs 13.10 ppl). 3,000 steps on |
22 | | -TinyStories, single seed; full reproduction in [FRONTIER.md](FRONTIER.md). |
| 32 | +On TinyStories, 3000 steps, single seed: at fixed parameter count Atome's |
| 33 | +routed-ternary block reaches **6.31 ppl vs 8.12** for a vanilla GPT-FP32 |
| 34 | +baseline (−22 %); at fixed flash budget **6.31 vs 13.10** (−52 %). Disk |
| 35 | +footprint is 16× smaller at param-match (15.1 KB vs 237.5 KB). |
23 | 36 |
|
24 | | -**This win holds at the ~60 K-parameter MCU regime and only there.** In a |
25 | | -944 K-parameter scale-up A/B the vanilla FP32 baseline reverses it (wins by |
26 | | -~11 %) — Atome's bet is deliberately the sub-1M, MCU-class regime. The full |
27 | | -honest reading, reversal included, is in [`FRONTIER.md`](FRONTIER.md) and |
28 | | -[`HONEST_RESULTS.md`](HONEST_RESULTS.md). |
| 37 | +**The result reverses at 944 K parameters**, where the vanilla FP32 baseline |
| 38 | +wins by ~11 %. Atome's bet is deliberately the sub-1M, MCU-class regime; |
| 39 | +above it ternary's capacity ceiling closes the gap and overruns it. Full |
| 40 | +reproduction in [`FRONTIER.md`](FRONTIER.md), full honest reading including |
| 41 | +the reversal in [`HONEST_RESULTS.md`](HONEST_RESULTS.md). |
29 | 42 |
|
30 | 43 | ## Why |
31 | 44 |
|
|
0 commit comments