Skip to content

Commit dfb04dd

Browse files
author
Atome LM Team
committed
Atome LM v0.3.0 — initial public source-available release
Tiny ternary language model for microcontroller deployment: 3-pathway routed-ternary architecture, byte tokenizer, and a pure-C99 inference engine with bit-exact single-forward Python parity. Source-available under the Atome LM Proprietary License; trained weights are not included (train-from-scratch).
0 parents  commit dfb04dd

119 files changed

Lines changed: 14764 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
__pycache__/
2+
*.pyc
3+
*.pyo
4+
*.egg-info/
5+
.pytest_cache/
6+
build/
7+
8+
# Trained checkpoints — too big and binary
9+
checkpoints/*.pt
10+
checkpoints/*.atome
11+
checkpoints/*.atome2
12+
checkpoints/ab_sweep/
13+
14+
# Generated corpus + on-disk fetch cache (sample.txt is intentionally tracked)
15+
data/*.txt
16+
!data/sample.txt
17+
.corpus_cache/
18+
19+
# Generated firmware artifacts
20+
c_engine/targets/*/model_data.h
21+
c_engine/targets/*/prompt_data.h
22+
c_engine/targets/*/firmware.elf
23+
c_engine/targets/*/firmware.map
24+
c_engine/targets/*/*.o
25+
26+
# Distribution kit zips and source tarballs (ship via Gumroad, not git)
27+
atome-research-kit-v*.zip
28+
atomelm-site.zip
29+
dist/*.tar.gz
30+
dist/*.zip
31+
32+
# Drafts and editor cruft
33+
*.draft.txt
34+
.DS_Store
35+
.idea/
36+
.vscode/
37+
38+
# Local venv
39+
.venv/
40+
venv/
41+

‎CITATION.cff‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
cff-version: 1.2.0
2+
title: "Atome LM: a tiny ternary language model for microcontroller deployment"
3+
message: "If you use this software, please cite it as below."
4+
type: software
5+
authors:
6+
- name: "Atome LM contributors"
7+
url: "https://atomelm.com"
8+
version: "0.3.0"
9+
date-released: 2026-05-21
10+
license: LicenseRef-Atome-LM-Proprietary
11+
keywords:
12+
- "language model"
13+
- "ternary"
14+
- "bitnet"
15+
- "microcontroller"
16+
- "edge-ai"
17+
- "tinyml"

‎FRONTIER.md‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Atome LM — Frontier Findings
2+
3+
> **Update 2026-05-11 — scale-up A/B at 944K reverses the headline.**
4+
> Same recipe, same val slice, same fairness audit, a 944K-param vanilla
5+
> GPT-FP32 baseline (950,608 params, +0.63 % vs Atome's 944,640) reaches
6+
> val loss 0.9337 / ppl 2.54, beating Atome ternary at 944K by 11.4 %
7+
> in loss and 11.5 % in perplexity. The +22 % param-fair / +52 %
8+
> flash-fair gains below hold at the **60K-param MCU regime** and only
9+
> that regime. Above ~1M params the inductive bias of the 3-pathway
10+
> block stops substituting for capacity and starts constraining it.
11+
> The honest framing is: *Atome's bet is the small-model regime —
12+
> sub-1M params, MCU-class deployment, no network.* See
13+
> [`HONEST_RESULTS.md`](HONEST_RESULTS.md) for the full 944K reading.
14+
> Multi-seed pending.
15+
16+
**Date.** 2026-05-09. CPU only, no GPU.
17+
**Hardware.** 4-thread CPU box. PyTorch 2.x, FP32 reference path.
18+
**Corpus.** TinyStories validation slice, 500 KB UTF-8 (~99.9 % ASCII).
19+
Train/eval split 90/10 over 64-byte chunks → 7,030 train chunks /
20+
782 held-out chunks.
21+
**Optimizer.** AdamW, lr 3e-4, batch 16, seq 64, 3,000 steps.
22+
**Single seed** (seed 0). Results have not been replicated across seeds.
23+
24+
This document reports the first apples-to-apples A/B between Atome's
25+
3-pathway ternary architecture and vanilla decoder-only Transformers
26+
(FP32) at fixed parameter count and at fixed flash budget. The closest
27+
published peer is Andrej Karpathy's `Stories260K` — a 260 K-parameter
28+
FP32 plain transformer trained on TinyStories. Atome's frontier claim
29+
is "smaller flash, better quality, smaller bits-per-weight, *and*
30+
deployable on a $2 microcontroller." This page tests the first three
31+
of those claims directly; MCU deployment is verified separately via
32+
bit-exact Python ↔ C ↔ Cortex-M3 (QEMU) parity (see `tests/test_qemu_parity.py`).
33+
34+
## TL;DR
35+
36+
| Model | Params | Bits/wt | Disk | bpb ↓ | Perplexity ↓ |
37+
|---|---:|---:|---:|---:|---:|
38+
| **Atome 3-pathway, ternary** | **60,800** | **1.58** | **15.1 KB**¹ / **17.2 KB**² | **2.66** | **6.31** |
39+
| Vanilla GPT, FP32 (param-fair) | 60,808 | 32 | 237.5 KB | 3.02 | 8.12 |
40+
| Vanilla GPT, FP32 (flash-fair) | 5,968 | 32 | 23.3 KB | 3.71 | 13.10 |
41+
42+
¹ ATOME01, 4 trits/byte (current C engine reads this).
43+
² ATOME02, 5 trits/byte base-3 packing — 14.4 % smaller, near
44+
information-theoretic floor of `log2(3) ≈ 1.585` bits/trit. Python
45+
encoder + decoder shipped today; C decoder is a future change.
46+
47+
## What this proves
48+
49+
1. **At the same parameter count, the 3-pathway ternary architecture
50+
beats a plain transformer by 22 % on perplexity (6.31 vs 8.12)
51+
while using 16× less disk.**
52+
53+
The vanilla baseline is *not* over-parameterized — it's matched at
54+
60.8 K params (`d_model=44, n_layers=3, n_heads=4, d_ff=44`,
55+
selected by brute-force search to land within 8 params of the
56+
target). This is the same architecture every public tiny-LM paper
57+
(`Stories260K`, the TinyStories paper, BitNet at small scale) uses,
58+
modulo trivia.
59+
60+
2. **At the same flash budget, the 3-pathway ternary architecture beats
61+
a plain transformer by 52 % on perplexity (6.31 vs 13.10).**
62+
63+
The flash-fair vanilla baseline is `d_model=8, n_layers=2,
64+
n_heads=4, d_ff=24`. It sits in the same 20–25 KB on-disk budget as
65+
the Atome ATOME01 binary (15.1 KB) and ATOME02 binary (17.2 KB).
66+
67+
3. **The 1.58-bit weights cost ~22 % perplexity vs FP32 at the same
68+
architecture parameters** — but the FP32 version costs 16× more
69+
flash. On any device where flash is the bottleneck (every MCU we
70+
target), ternary wins. On any device where compute is the
71+
bottleneck and flash is free (server CPUs), FP32 wins on quality.
72+
73+
4. **ATOME02 base-3 packing reaches 1.6 bits/trit — within 1 % of the
74+
information-theoretic floor of 1.585 bits/trit** — and reduces the
75+
on-disk binary from 20.1 KB to 17.2 KB on the same trained
76+
60.8 K-param model. C decoder still pending.
77+
78+
## What this does NOT prove
79+
80+
- **Single seed only.** All three numbers are seed 0. We have not run
81+
multi-seed yet to estimate variance. The 22 % / 52 % gaps are very
82+
large compared to typical seed-noise at this scale, but the variance
83+
is unmeasured.
84+
- **Single corpus.** TinyStories is a forgiving target — short stories
85+
with restricted vocabulary. Wider-domain or code corpora may favor
86+
vanilla attention. We have not measured.
87+
- **Single training horizon.** 3,000 steps is well short of
88+
convergence. The relative ranking might swap or amplify with more
89+
training. A 10 K-step run is in flight; we'll update this page if it
90+
changes the headline.
91+
- **No real silicon.** All MCU claims are verified on QEMU
92+
Cortex-M3, not on physical RP2040 / STM32 hardware. Tokens/sec and
93+
Joules/token on real silicon are still pending.
94+
- **Stories260K direct comparison still pending.** Karpathy's exact
95+
setup is `Stories260K` at 260 K params + a 32 K-token SentencePiece
96+
vocab. Our byte-tokenizer + 60 K config is ~4× smaller. A direct
97+
apples-to-apples vs `Stories260K` would need either (a) us to scale
98+
up to 260 K params and a SentencePiece tokenizer, or (b) Karpathy's
99+
setup retrained at 60 K params with a byte tokenizer. Neither is
100+
done.
101+
102+
## Comparison with the published frontier
103+
104+
| System | Smallest target | Params | Bits/wt | Real MCU? | Architecture beats vanilla? |
105+
|---|---|---:|---:|---|---|
106+
| Microsoft BitNet b1.58 | server CPU | 700 M – 3 B | 1.58 | no | (matches at scale) |
107+
| Meta MobileLLM | smartphone | 125 M – 1 B | 4–8 | no | yes (vs same-size vanilla) |
108+
| Karpathy `Stories260K` | laptop / browser | 260 K | 32 | no firmware | n/a (is the vanilla baseline) |
109+
| llama.cpp on RP2040 (hobby) | RP2040 + SD | ~1 B (swapped) | 4 | yes (slow, requires SD) | not measured |
110+
| TFLite Micro / Edge Impulse | Cortex-M0+ | – | 8 | yes | no language tasks |
111+
| **Atome LM (this work)** | **Cortex-M0+, 16 KB SRAM** | **60 K** | **1.58** | **QEMU yes, silicon pending** | **+22 % at param-fair, +52 % at flash-fair** |
112+
113+
Smaller, more bit-efficient, *and* architecturally beats vanilla at
114+
the budgets we target. To our knowledge, the smallest published LM
115+
where the routed-architecture win has been measured directly against
116+
a vanilla baseline at the same flash budget.
117+
118+
## Reproduce
119+
120+
```bash
121+
# from the repository root
122+
PYTHONPATH=. python3 scripts/build_corpus.py --source tinystories \
123+
--max-bytes 500000 --output data/tinystories.txt
124+
125+
PYTHONPATH=. python3 scripts/run_ab_sweep.py \
126+
--train data/tinystories.txt --steps 3000 \
127+
--output ab_results.json
128+
```
129+
130+
`ab_results.json` will contain the same numbers as in the table above
131+
(modulo platform-dependent rounding in PyTorch's matmul kernels).
132+
133+
## Open questions / next pushes
134+
135+
- **A1.** Multi-seed (3 seeds × 3 configs) to estimate variance on the
136+
22 % / 52 % gaps.
137+
- **A2.** Train all three to ≥ 10 K steps. Does the gap close, hold,
138+
or widen?
139+
- **A3.** Ablate: which of the three pathways (local conv, diagonal
140+
SSM, top-k sparse attention) carries most of the architecture win?
141+
Drop each, measure.
142+
- **A4.** Ship a C decoder for ATOME02. Cuts the demo binary from
143+
20.1 KB to 17.2 KB without code changes elsewhere.
144+
- **A5.** Real silicon. Flash an RP2040 with the engine + this 60.8 K
145+
ckpt. Measure tokens/sec, Joules/token. **The headline number that
146+
turns the claim from "frontier" into a fact.**
147+
- **A6.** Distillation from a strong teacher (GPT-4 / Claude on
148+
10 MB of curated narrow-domain text) into the same 60 K Atome.
149+
Open question: does the architecture advantage compound under
150+
distillation?
151+
- **A7.** Bug A fix (Python `generate` ↔ C `atome_generate`
152+
short-prompt SSM divergence). Touches the bit-exact-parity
153+
contract — needs explicit user sign-off.
154+
155+
## Files of record
156+
157+
- `ab_results.json` — exact numbers and config of the run reported here.
158+
- Trained A/B checkpoints (`atome_60k_ternary`, `vanilla_60k_fp32`,
159+
`vanilla_6k_fp32`) are *not* shipped — regenerate them with the harness
160+
below (this kit is train-from-scratch).
161+
- `atome_llm/baselines/vanilla_transformer.py` — the baseline.
162+
- `scripts/run_ab_sweep.py` — the harness.
163+
- `tests/test_vanilla_baseline.py` — 10 sanity tests on the baseline.
164+
- `tests/test_export_packed.py` — 5 tests on ATOME02 round-trip.
165+
- `tests/test_trit_packing.py` — 11 tests on the base-3 packer.

‎HONEST_RESULTS.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Atome LM — Honest Results Dossier
2+
3+
> One page, no marketing. What we measured, on what hardware, with what
4+
> seed. Where we beat vanilla, where we don't, where we don't yet know.
5+
6+
**Last updated.** 2026-05-13. Compiled from `checkpoints/*.train.json`
7+
and `ab_results.json` (which are the actual run artifacts — open them).
8+
9+
---
10+
11+
## Table 1 — The numbers, as measured
12+
13+
| Config | Params | Bits/wt | Loss ↓ | PPL ↓ | Disk | Status |
14+
|---|---:|---:|---:|---:|---:|---|
15+
| **60K regime (MCU target)** | | | | | | |
16+
| Atome 3-pathway ternary | 60,800 | 1.58 | 1.84 | 6.31 | 15.1 KB¹ | ✅ measured |
17+
| Vanilla GPT FP32 (param-fair) | 60,808 | 32 | 2.09 | 8.12 | 237.5 KB | ✅ measured |
18+
| Vanilla GPT FP32 (flash-fair) | 5,968 | 32 | 2.57 | 13.10 | 23.3 KB | ✅ measured |
19+
| **944K regime (scale-up A/B)** | | | | | | |
20+
| Atome 3-pathway ternary | 944,640 | 1.58 | **1.0545** | 2.87 | 184 KB¹ | ✅ measured |
21+
| Vanilla GPT FP32 (param-fair) | 950,608 | 32 | **0.9337** | 2.54 | 3.7 MB | ✅ measured |
22+
| Atome 3-pathway, power3 (per-tensor) | 944,640 | 2.81 | TBD | TBD | ~325 KB est | ⏳ launcher ready |
23+
| Atome 3-pathway, power3 (per-row α) | 944,640 | 2.81² | TBD | TBD | ~330 KB est | ⏳ launcher ready |
24+
25+
¹ ATOME01, 4 trits/byte packing.
26+
² Per-tensor portion is 2.81 bits/wt; per-row α adds one FP16 per output
27+
row (negligible % overhead at 944K).
28+
29+
**Headlines, unsanitized:**
30+
31+
- At the 60K MCU target, the 3-pathway ternary architecture beats vanilla
32+
FP32 by **22% in perplexity at the same param count** and **52% at the
33+
same flash budget**.
34+
- At 944K, plain ternary **loses to vanilla FP32 by 11.4% in val loss /
35+
11.5% in perplexity**. Same recipe, same val slice, same seed.
36+
- The 944K reversal is the most important honest finding in this kit.
37+
It says: the 3-pathway inductive bias substitutes for capacity at
38+
small scale and constrains it at larger scale. Atome's bet is the
39+
small-model / MCU regime — not "tiny ternary beats everything."
40+
41+
## Table 2 — What the 944K result is conditioned on
42+
43+
| Variable | Value |
44+
|---|---|
45+
| Corpus | TinyStories full (`train.txt + valid.txt` concatenated, ~1.7 GB raw) |
46+
| Steps | 30,000 |
47+
| Sequence length | 256 |
48+
| Batch × accum | 64 × 4 |
49+
| Optimizer | AdamW, lr=3e-4 → 3e-5 cosine, warmup=1000, weight_decay=0.1 |
50+
| Precision | BF16 autocast |
51+
| Seed | 0 (single seed; multi-seed pending) |
52+
| Hardware | RunPod A100/A6000 (atome) — vast A100 (vanilla, 2026-05-11) |
53+
54+
## Table 3 — What we have NOT measured
55+
56+
| Question | Why it matters | Cost to resolve |
57+
|---|---|---|
58+
| Multi-seed variance at 944K | Single seed isn't a finding | ~$2 vast (3 seeds × atome + vanilla) |
59+
| Crossover point | Where exactly does 3-pathway start losing? | ~$8 vast (sweep 100K / 300K / 600K / 1.5M) |
60+
| Power-of-3 closes the 944K gap | If yes: the loss-reversal headline flips | ~$6 vast (this kit's launcher) |
61+
| Q15 fixed-point inference RAM | RP2040 RAM target was missed at 944K (411 KB peak) | ~3 days engineering |
62+
| Real silicon throughput | All MCU claims are QEMU; turns "frontier" into "fact" | $0 (RP2040 sitting on desk) + ~1 day |
63+
| Distillation from vanilla teacher | Ternary students often close 80%+ of float-teacher gap | ~$1–2 vast |
64+
| Wider-domain corpora | TinyStories favors local-pattern models | ~$4 vast |
65+
66+
## Table 4 — What's solid vs what's load-bearing-but-thin
67+
68+
**Solid (don't change without strong reason):**
69+
70+
- 146/146 tests green at HEAD (16 of those are power3-specific).
71+
- Bit-exact Python ↔ C ↔ Cortex-M3 (QEMU) parity for single forward
72+
(`tests/test_parity_with_c.py`, `tests/c_parity/parity_main.c`).
73+
- Trained atome_1m_v1.pt + vanilla_1m_v1.pt artifacts on disk, both
74+
with full training logs in `checkpoints/*.train.json` (open them — every step's
75+
loss is recorded).
76+
- 60K param-fair / flash-fair A/B reproducible in ~30 min CPU
77+
(`scripts/run_ab_sweep.py`).
78+
79+
**Load-bearing but thin:**
80+
81+
- All headline numbers are single-seed.
82+
- Multi-token C generation has a known SSM-state divergence bug
83+
(Bug A) — fixed in Python; the C engine path needs a re-derive of
84+
`atome_generate` to match. Single-forward Python↔C parity is
85+
unaffected and remains bit-exact (`tests/test_parity_with_c.py`).
86+
- RP2040 demo currently exceeds 264 KB SRAM at 944K — the MCU claim
87+
is regime-dependent and the launcher in this kit is testing whether
88+
power3 narrows the param budget enough to bring 944K back into scope
89+
(it doesn't on its own; needs Q15 or smaller hidden state).
90+
91+
## Table 5 — Cost of every measurement done to date
92+
93+
| Work | Date | Cost | Result on disk |
94+
|---|---|---:|---|
95+
| 60K A/B sweep | 2026-05-09 | $0 (CPU) | `ab_results.json` |
96+
| 944K Atome | 2026-05-10 | ~$0.40 (RunPod A40) | `atome_1m_v1.pt` |
97+
| 944K Vanilla | 2026-05-11 | ~$0.55 (Vast A100) | `vanilla_1m_v1.pt` |
98+
| Power-3 wiring + tests + CPU smoke | 2026-05-12/13 | $0 (CPU) | `atome_llm/core/power3.py` + 6 new tests |
99+
| **Total spent so far** | | **< $1.00** | — |
100+
| Pending: 944K A/B with power3 + power3_pr | — | ~$3.60–$6.40 cap $8 | launcher in `scripts/` |
101+
102+
## Files of record
103+
104+
Trained checkpoints (`*.pt`, `checkpoints/ab_sweep/`) are **not shipped** in
105+
this public repository — Atome LM is train-from-scratch (see README). The
106+
training *logs* below ship, so every reported number stays auditable
107+
step-by-step:
108+
109+
- `checkpoints/atome_1m_v1.train.json` — 944K Atome, every-1000-step log.
110+
- `checkpoints/vanilla_1m_v1.train.json` — 944K vanilla FP32, same.
111+
- `ab_results.json` — exact 60K A/B result row.
112+
- `FRONTIER.md` — frontier writeup with full 944K disclosure.
113+
- `PAPER.md` — architecture writeup.
114+
- `tests/` — 146 green tests.

0 commit comments

Comments
 (0)