Atome LM v0.3.0 — initial public release

Author: Atome LM Team

.github/workflows/test.yml

@@ -0,0 +1,48 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  pytest:
+    name: pytest (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install build essentials + qemu-system-arm
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            build-essential gcc-arm-none-eabi qemu-system-arm
+
+      - name: Install CPU-only PyTorch
+        run: |
+          python -m pip install --upgrade pip
+          pip install --index-url https://download.pytorch.org/whl/cpu \
+            "torch>=2.0"
+
+      - name: Install atome-llm + test deps
+        run: pip install -e ".[dev]"
+
+      - name: Environment check
+        run: python check_env.py
+
+      - name: pytest
+        run: pytest -q

.gitignore

@@ -0,0 +1,47 @@
+__pycache__/
+*.pyc
+*.pyo
+*.egg-info/
+.pytest_cache/
+build/
+
+# Trained checkpoints — too big and binary, EXCEPT the three released artifacts
+# (944K demo blob + its PyTorch source + the vanilla FP32 baseline used for
+# the HONEST_RESULTS reversal A/B). These are tracked.
+checkpoints/*.pt
+checkpoints/*.atome
+checkpoints/*.atome2
+checkpoints/*.bin
+checkpoints/ab_sweep/
+!checkpoints/atome_944k.bin
+!checkpoints/atome_1m_v1.pt
+!checkpoints/vanilla_1m_v1.pt
+
+# Generated corpus + on-disk fetch cache (sample.txt is intentionally tracked)
+data/*.txt
+!data/sample.txt
+.corpus_cache/
+
+# Generated firmware artifacts
+c_engine/targets/*/model_data.h
+c_engine/targets/*/prompt_data.h
+c_engine/targets/*/firmware.elf
+c_engine/targets/*/firmware.map
+c_engine/targets/*/*.o
+
+# Distribution kit zips and source tarballs
+atome-research-kit-*.zip
+atomelm-site.zip
+dist/*.tar.gz
+dist/*.zip
+
+# Drafts and editor cruft
+*.draft.txt
+.DS_Store
+.idea/
+.vscode/
+
+# Local venv
+.venv/
+venv/
+

CITATION.cff

@@ -0,0 +1,17 @@
+cff-version: 1.2.0
+title: "Atome LM: a tiny ternary language model for microcontroller deployment"
+message: "If you use this software, please cite it as below."
+type: software
+authors:
+  - name: "Atome LM contributors"
+url: "https://atomelm.com"
+version: "0.3.0"
+date-released: 2026-05-21
+license: Apache-2.0
+keywords:
+  - "language model"
+  - "ternary"
+  - "bitnet"
+  - "microcontroller"
+  - "edge-ai"
+  - "tinyml"

CODE_OF_CONDUCT.md

@@ -0,0 +1,7 @@
+# Code of Conduct
+
+This project follows the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
+
+In short: be respectful, assume good faith, and focus discussion on the work. Everyone is welcome to contribute regardless of background or identity. Reports of unacceptable behavior can be sent privately to **hello@atomelm.com** and will be reviewed promptly and confidentially.
+
+The full text of the Contributor Covenant — including the complete enforcement guidelines — is at the link above and applies to this project in full.

CONTRIBUTING.md

@@ -0,0 +1,67 @@
+# Contributing to Atome LM
+
+Thanks for considering a contribution. This is a small, focused project — a tiny ternary language model + a C99 inference engine that talks to it bit-exactly. Read `PROJECT_CONTENT.md` first; it covers what you must not break.
+
+## Quick start
+
+```bash
+git clone https://github.com/atome-lm/atome-llm-kit
+cd atome-llm-kit
+./install.sh
+. .venv/bin/activate
+pytest -q        # expect: 146 passed (or 145 + 1 skipped without qemu-system-arm)
+```
+
+## Reporting bugs
+
+Open an issue on GitHub with:
+
+- what you ran (exact command)
+- what you expected
+- what happened (full error, not paraphrased)
+- your platform: OS, Python version, and `python -c "import torch; print(torch.__version__)"`
+
+If you hit a parity failure (Python forward ≠ C forward), please attach the failing seed and any checkpoint you trained — these are the highest-priority bugs.
+
+## Submitting a pull request
+
+1. Fork the repo and create a branch off `main`.
+2. Make your change.
+3. Run the full test suite — every PR must keep `pytest -q` green.
+4. If your change touches `atome_llm/core/`, `c_engine/upstream/`, or the export format, **specifically confirm** these tests still pass:
+   - `tests/test_parity_with_c.py` — single-forward Python ↔ C parity
+   - `tests/test_parity_multitoken.py` — multi-token Python ↔ C parity
+   - `tests/test_export_format.py` — binary format + header generation
+5. Open the PR. CI will rerun the suite on Python 3.10 / 3.11 / 3.12.
+
+## Scope of acceptable changes
+
+Welcome:
+
+- Bug fixes
+- New test coverage (especially fuzz cases on the C parser and edge inputs to `atome_predict_next` / `atome_generate`)
+- Performance improvements that preserve bit-exact parity
+- Documentation fixes and clarifications
+- New MCU target boards under `c_engine/targets/`, *as long as they don't change the upstream engine*
+- New baselines under `atome_llm/baselines/` for honest A/B comparison
+
+Out of scope, please don't open PRs for these:
+
+- Adding heap allocation, dynamic memory, or libc dependencies to `c_engine/upstream/`
+- Adding "shouldn't happen" fallbacks to deterministic code paths
+- Bundling new tokenizers (BPE / sentencepiece) — the byte tokenizer is a load-bearing design choice for MCU flash budget
+- Changes that break Python ↔ C parity, even if they improve a benchmark
+- New features that promote code from `c_engine/experiments/` into `c_engine/upstream/` without full parity + bounds-check coverage
+
+## Coding standards
+
+- Python: keep it simple, no helper layers, no decorators-for-style. Match the existing voice — small functions, no premature abstraction, comments only when the *why* is non-obvious.
+- C: C99 only, no GNU extensions, no libc beyond `<string.h>` / `<math.h>` / `<stdint.h>`. Static buffers sized by compile-time `ATOME_*` macros. Bounds-check all public API inputs.
+
+## Security
+
+If you find a security issue (anything that lets a malicious checkpoint or `.atome` blob compromise a host running the engine), please email **hello@atomelm.com** instead of filing a public issue. We'll coordinate disclosure.
+
+## License
+
+By submitting a contribution you agree it will be released under the Apache License 2.0 (the project license — see `LICENSE`).

FRONTIER.md

@@ -0,0 +1,165 @@
+# Atome LM — Frontier Findings
+
+> **Update 2026-05-11 — scale-up A/B at 944K reverses the headline.**
+> Same recipe, same val slice, same fairness audit, a 944K-param vanilla
+> GPT-FP32 baseline (950,608 params, +0.63 % vs Atome's 944,640) reaches
+> val loss 0.9337 / ppl 2.54, beating Atome ternary at 944K by 11.4 %
+> in loss and 11.5 % in perplexity. The +22 % param-fair / +52 %
+> flash-fair gains below hold at the **60K-param MCU regime** and only
+> that regime. Above ~1M params the inductive bias of the 3-pathway
+> block stops substituting for capacity and starts constraining it.
+> The honest framing is: *Atome's bet is the small-model regime —
+> sub-1M params, MCU-class deployment, no network.* See
+> [`HONEST_RESULTS.md`](HONEST_RESULTS.md) for the full 944K reading.
+> Multi-seed pending.
+
+**Date.** 2026-05-09. CPU only, no GPU.
+**Hardware.** 4-thread CPU box. PyTorch 2.x, FP32 reference path.
+**Corpus.** TinyStories validation slice, 500 KB UTF-8 (~99.9 % ASCII).
+Train/eval split 90/10 over 64-byte chunks → 7,030 train chunks /
+782 held-out chunks.
+**Optimizer.** AdamW, lr 3e-4, batch 16, seq 64, 3,000 steps.
+**Single seed** (seed 0). Results have not been replicated across seeds.
+
+This document reports the first apples-to-apples A/B between Atome's
+3-pathway ternary architecture and vanilla decoder-only Transformers
+(FP32) at fixed parameter count and at fixed flash budget. The closest
+published peer is Andrej Karpathy's `Stories260K` — a 260 K-parameter
+FP32 plain transformer trained on TinyStories. Atome's frontier claim
+is "smaller flash, better quality, smaller bits-per-weight, *and*
+deployable on a $2 microcontroller." This page tests the first three
+of those claims directly; MCU deployment is verified separately via
+bit-exact Python ↔ C ↔ Cortex-M3 (QEMU) parity (see `tests/test_qemu_parity.py`).
+
+## TL;DR
+
+| Model | Params | Bits/wt | Disk | bpb ↓ | Perplexity ↓ |
+|---|---:|---:|---:|---:|---:|
+| **Atome 3-pathway, ternary** | **60,800** | **1.58** | **15.1 KB**¹ / **17.2 KB**² | **2.66** | **6.31** |
+| Vanilla GPT, FP32 (param-fair) | 60,808 | 32 | 237.5 KB | 3.02 | 8.12 |
+| Vanilla GPT, FP32 (flash-fair) | 5,968 | 32 | 23.3 KB | 3.71 | 13.10 |
+
+¹ ATOME01, 4 trits/byte (current C engine reads this).
+² ATOME02, 5 trits/byte base-3 packing — 14.4 % smaller, near
+information-theoretic floor of `log2(3) ≈ 1.585` bits/trit. Python
+encoder + decoder shipped today; C decoder is a future change.
+
+## What this proves
+
+1. **At the same parameter count, the 3-pathway ternary architecture
+   beats a plain transformer by 22 % on perplexity (6.31 vs 8.12)
+   while using 16× less disk.**
+
+   The vanilla baseline is *not* over-parameterized — it's matched at
+   60.8 K params (`d_model=44, n_layers=3, n_heads=4, d_ff=44`,
+   selected by brute-force search to land within 8 params of the
+   target). This is the same architecture every public tiny-LM paper
+   (`Stories260K`, the TinyStories paper, BitNet at small scale) uses,
+   modulo trivia.
+
+2. **At the same flash budget, the 3-pathway ternary architecture beats
+   a plain transformer by 52 % on perplexity (6.31 vs 13.10).**
+
+   The flash-fair vanilla baseline is `d_model=8, n_layers=2,
+   n_heads=4, d_ff=24`. It sits in the same 20–25 KB on-disk budget as
+   the Atome ATOME01 binary (15.1 KB) and ATOME02 binary (17.2 KB).
+
+3. **The 1.58-bit weights cost ~22 % perplexity vs FP32 at the same
+   architecture parameters** — but the FP32 version costs 16× more
+   flash. On any device where flash is the bottleneck (every MCU we
+   target), ternary wins. On any device where compute is the
+   bottleneck and flash is free (server CPUs), FP32 wins on quality.
+
+4. **ATOME02 base-3 packing reaches 1.6 bits/trit — within 1 % of the
+   information-theoretic floor of 1.585 bits/trit** — and reduces the
+   on-disk binary from 20.1 KB to 17.2 KB on the same trained
+   60.8 K-param model. C decoder still pending.
+
+## What this does NOT prove
+
+- **Single seed only.** All three numbers are seed 0. We have not run
+  multi-seed yet to estimate variance. The 22 % / 52 % gaps are very
+  large compared to typical seed-noise at this scale, but the variance
+  is unmeasured.
+- **Single corpus.** TinyStories is a forgiving target — short stories
+  with restricted vocabulary. Wider-domain or code corpora may favor
+  vanilla attention. We have not measured.
+- **Single training horizon.** 3,000 steps is well short of
+  convergence. The relative ranking might swap or amplify with more
+  training. A 10 K-step run is in flight; we'll update this page if it
+  changes the headline.
+- **No real silicon.** All MCU claims are verified on QEMU
+  Cortex-M3, not on physical RP2040 / STM32 hardware. Tokens/sec and
+  Joules/token on real silicon are still pending.
+- **Stories260K direct comparison still pending.** Karpathy's exact
+  setup is `Stories260K` at 260 K params + a 32 K-token SentencePiece
+  vocab. Our byte-tokenizer + 60 K config is ~4× smaller. A direct
+  apples-to-apples vs `Stories260K` would need either (a) us to scale
+  up to 260 K params and a SentencePiece tokenizer, or (b) Karpathy's
+  setup retrained at 60 K params with a byte tokenizer. Neither is
+  done.
+
+## Comparison with the published frontier
+
+| System | Smallest target | Params | Bits/wt | Real MCU? | Architecture beats vanilla? |
+|---|---|---:|---:|---|---|
+| Microsoft BitNet b1.58 | server CPU | 700 M – 3 B | 1.58 | no | (matches at scale) |
+| Meta MobileLLM | smartphone | 125 M – 1 B | 4–8 | no | yes (vs same-size vanilla) |
+| Karpathy `Stories260K` | laptop / browser | 260 K | 32 | no firmware | n/a (is the vanilla baseline) |
+| llama.cpp on RP2040 (hobby) | RP2040 + SD | ~1 B (swapped) | 4 | yes (slow, requires SD) | not measured |
+| TFLite Micro / Edge Impulse | Cortex-M0+ | – | 8 | yes | no language tasks |
+| **Atome LM (this work)** | **Cortex-M0+, 16 KB SRAM** | **60 K** | **1.58** | **QEMU yes, silicon pending** | **+22 % at param-fair, +52 % at flash-fair** |
+
+Smaller, more bit-efficient, *and* architecturally beats vanilla at
+the budgets we target. To our knowledge, the smallest published LM
+where the routed-architecture win has been measured directly against
+a vanilla baseline at the same flash budget.
+
+## Reproduce
+
+```bash
+# from the repository root
+PYTHONPATH=. python3 scripts/build_corpus.py --source tinystories \
+    --max-bytes 500000 --output data/tinystories.txt
+
+PYTHONPATH=. python3 scripts/run_ab_sweep.py \
+    --train data/tinystories.txt --steps 3000 \
+    --output ab_results.json
+```
+
+`ab_results.json` will contain the same numbers as in the table above
+(modulo platform-dependent rounding in PyTorch's matmul kernels).
+
+## Open questions / next pushes
+
+- **A1.** Multi-seed (3 seeds × 3 configs) to estimate variance on the
+  22 % / 52 % gaps.
+- **A2.** Train all three to ≥ 10 K steps. Does the gap close, hold,
+  or widen?
+- **A3.** Ablate: which of the three pathways (local conv, diagonal
+  SSM, top-k sparse attention) carries most of the architecture win?
+  Drop each, measure.
+- **A4.** Ship a C decoder for ATOME02. Cuts the demo binary from
+  20.1 KB to 17.2 KB without code changes elsewhere.
+- **A5.** Real silicon. Flash an RP2040 with the engine + this 60.8 K
+  ckpt. Measure tokens/sec, Joules/token. **The headline number that
+  turns the claim from "frontier" into a fact.**
+- **A6.** Distillation from a strong teacher LLM (10 MB of curated
+  narrow-domain text generated by a frontier model) into the same 60 K Atome.
+  Open question: does the architecture advantage compound under
+  distillation?
+- **A7.** Bug A fix (Python `generate` ↔ C `atome_generate`
+  short-prompt SSM divergence). Touches the bit-exact-parity
+  contract — needs explicit user sign-off.
+
+## Files of record
+
+- `ab_results.json` — exact numbers and config of the run reported here.
+- Trained A/B checkpoints (`atome_60k_ternary`, `vanilla_60k_fp32`,
+  `vanilla_6k_fp32`) are *not* shipped — regenerate them with the harness
+  below (this kit is train-from-scratch).
+- `atome_llm/baselines/vanilla_transformer.py` — the baseline.
+- `scripts/run_ab_sweep.py` — the harness.
+- `tests/test_vanilla_baseline.py` — 10 sanity tests on the baseline.
+- `tests/test_export_packed.py` — 5 tests on ATOME02 round-trip.
+- `tests/test_trit_packing.py` — 11 tests on the base-3 packer.