An open research wiki mapping the frontier of latent-space reasoning, continuous thought, and inter-agent latent communication in large language models.
Explore the Wiki | View the Spectrum | Read the Overview
Large language models are internally continuous systems — dense vectors in R^d at every layer — yet they're forced to interface with the world through discrete token sampling. This bottleneck:
- Discards distributional uncertainty (a full probability distribution collapses to one token)
- Prevents superposition of hypotheses (one token = one path)
- Wastes compute on fluency tokens that carry no reasoning content
What happens when you remove it — both within a single model and between collaborating models?
This wiki systematically tracks 27 sources (26 papers + 1 open-source project, Dec 2022 – Apr 2026) exploring that question, organized into a deeply cross-referenced knowledge graph with 1400+ internal links.
Feed hidden states back as input embeddings instead of decoding to tokens. Models reason silently in continuous vector space, enabling superposition — maintaining multiple reasoning paths simultaneously in a single vector.
Key result: Coconut achieves 97.0% on planning tasks via emergent BFS, vs. 77.5% for chain-of-thought.
Key papers: Coconut, SoftCoT, Thinking States, iCoT, Pause Tokens, Superposition Theory
Replace text-based multi-agent debate with continuous representations. A 10-level depth spectrum from natural language (~15 bits/position) to full hidden-state sequences (~40K bits/position).
Key result: LatentMAS achieves 471x theoretical compression over text with zero training.
Key papers: CIPHER, AC, KVComm, C2C, Interlat, SDE, ThoughtComm, Vision Wormhole, LatentMAS
One of the wiki's central frameworks — a 10-level taxonomy of how agents can share information, ordered from shallowest to deepest:
Compatible Restrictive
Interpretable Opaque
Low bandwidth High bandwidth
L0 L1 L2 L3 L4 L5 L6 L7 L8 L9
NL Emb Delta Struct Vision KV-sel KV-x AC Full-HS KV+Lat
|---------|---------|---------|---------|---------|---------|---------|---------|
~15 ~40K
bits/pos bits/pos
The research frontier is about bending this curve — achieving high information density without requiring tight architectural coupling.
See communication-depth-spectrum for the full guided walkthrough.
| Page | What You'll Find | |
|---|---|---|
| Overview | overview-state-of-field |
Single narrative synthesis of the entire research landscape |
| Reasoning | latent-reasoning |
Guided path: Pause Tokens → iCoT → Coconut → Superposition → SoftCoT |
| Communication | latent-communication |
Guided path across all communication methods |
| Depth Spectrum | communication-depth-spectrum |
10-level walkthrough from NL to full working memory |
| Unified | unified-frameworks |
Systems combining reasoning + communication |
| Cross-Architecture | cross-architecture |
Cross-architecture compatibility |
| Practical Systems | practical-systems |
Engineering lens: scaling, method selection, deployment |
| Theoretical | theoretical-foundations |
Mathematical underpinnings |
| Compression | compression-information-theory |
Compression & Information-Theoretic Bounds |
| Safety | safety-interpretability |
Safety, Interpretability & Auditability of Latent Systems |
| Comparison | method-comparison |
Side-by-side table: training, architecture, results |
| Frontiers | frontier-research-directions |
Where the field is heading next |
Note: See AGENTS.md for the authoritative vault schema.
beyond-the-token-bottleneck/
├── raw/ # Immutable source documents
│ ├── pdf/ # 26 source PDFs (arXiv, ACL, ICML, NeurIPS)
│ ├── latex/ # LaTeX source archives
│ ├── index.md # Per-paper provenance index
│ ├── checklist.md # Ingest tracking checklist
│ └── download_arxiv_papers.py # Bulk arXiv downloader
├── workflows/ # Maintainer workflow playbooks used by AGENTS.md
│ ├── create/ # ingest, batch-ingest, synthesize
│ ├── enrich/ # enrich, expand
│ ├── audit/ # gap-analysis, verification, review, lint,
│ │ # enrichment-audit, moc-gap-analysis,
│ │ # plugin-audit, schema-self-audit
│ ├── query/ # query
│ ├── meta/ # readme-github-maintenance
│ ├── _shared/ # Reusable fragments: procedures/, checklists/,
│ │ # rules/, glossary.md
│ ├── CONVENTIONS.md # Workflow authoring conventions
│ └── README.md # Workflow decision tree
├── wiki/ # Research wiki (120+ pages)
│ ├── sources/ # Paper summaries by theme
│ │ ├── reasoning/ # Coconut, SoftCoT, Pause Tokens, ...
│ │ ├── communication/ # CIPHER, AC, KVComm, C2C, ...
│ │ │ ├── embeddings/ # Output-layer methods
│ │ │ ├── activations/ # Hidden-state methods
│ │ │ ├── kv-cache/ # KV-cache methods
│ │ │ └── structured/ # Disentangled methods
│ │ ├── unified/ # LatentMAS, Vision Wormhole, ...
│ │ └── meta/ # Scaling, expressivity theory, latent reasoning analyses
│ ├── concepts/ # 10 cross-cutting concept pages (+ _partials/)
│ │ ├── challenges/ # catastrophic-forgetting
│ │ ├── communication/ # activation / embedding-space / kv-cache communication
│ │ ├── multi-agent/ # multi-agent-debate, temperature-diversity
│ │ ├── reasoning/ # latent-space-reasoning, continuous-vs-discrete,
│ │ │ # thought-structure
│ │ └── theory/ # latent-variable-model
│ ├── entities/ # 13 research group profiles (+ 26 reusable partials)
│ ├── analyses/ # 9 synthesis & comparison pages
│ ├── mocs/*.md # 9 Maps of Content
│ ├── index.md # Wiki index
│ ├── log.md # Change log
│ └── overview-state-of-field.md # Top-level narrative
├── assets/ # Repo-level images (e.g. Obsidian graph view)
├── AGENTS.md # Wiki schema & LLM instructions
├── CLAUDE.md # Pointer to AGENTS.md
├── LICENSE-APACHE / LICENSE-CC-BY # Split licensing (code / content)
└── README.md
| Deep Analysis | Cross-Referenced | LaTeX Math | Visual Diagrams | Guided Paths |
|---|---|---|---|---|
| Not summaries — detailed mechanisms, math, experimental numbers, ablation insights. 800–2000+ words per concept page. | 1400+ internal wiki-links. Every mention of a concept, entity, or paper links to its page. | All equations rendered with proper notation. Custom MathJax preamble with shared macros. | 21 Mermaid flowcharts for architectures, pipelines, and paper lineage trees. | Maps of Content with narrative reading order — not just link lists. |
The Obsidian graph view is configured with an Excel-style categorical palette so each wiki section is visually distinct: concepts (blue), entities (orange), sources (gray), analyses (gold), MOCs (green), top-level wiki notes (dark blue), raw/ (light orange), and workflows/ (dark green).
This vault is designed for Obsidian. Clone and open the repo as a vault.
| Plugin | Purpose | Install |
|---|---|---|
| LaTeX Suite | Fast math typesetting via text expansion snippets | GitHub |
| Extended MathJax | Custom preamble with \R, \Loss, \E macros |
GitHub |
| Pandoc | Export pages to PDF, DOCX, or ePub | GitHub |
| TikZJax | TikZ diagram rendering | GitHub |
| Diagrams (Draw.io) | Diagram editor (installed but not used for wiki content; AGENTS.md prohibits .drawio files in favor of Mermaid) |
GitHub |
git clone https://github.com/CompleteTech-LLC-AI-Research/beyond-the-token-bottleneck.git
cd beyond-the-token-bottleneck
# Open this folder as a vault in Obsidian
# Enable community plugins when prompted
# Start with: wiki/overview-state-of-field.mdThis vault implements the LLM Wiki pattern described by Andrej Karpathy — where an LLM doesn't just retrieve documents (RAG-style) but incrementally builds and maintains a persistent, structured wiki that compounds knowledge over time. The key insight: "the tedious part of maintaining a knowledge base is not the reading or the thinking — it's the bookkeeping," and LLMs excel at exactly that bookkeeping.
The workflow:
- Ingest — Source papers are collected into
raw/pdf/ - Summarize — An LLM reads each paper and generates a structured source summary following the schema in
AGENTS.md - Extract — Entities (research groups) and concepts (ideas, frameworks) are identified and given their own pages
- Cross-reference — Every mention of a known entity or concept is linked. Contradictions and connections between papers are surfaced.
- Synthesize — Maps of Content provide guided reading paths. Analysis pages compare methods and identify frontier directions.
Detailed maintainer playbooks for these operational workflows live under workflows/{create,enrich,audit,query,meta}/. Start at workflows/README.md for the decision tree.
The result is a deeply interlinked knowledge graph that's more useful than any individual paper summary — a second brain for latent-space reasoning research.
billion-token-one-task/latent-communication
An open collaboration initiative recruiting researchers to advance latent communication for LLM agents. Their team has demonstrated 91% accuracy on GSM8K using 512-byte compressed latent vectors — replacing MB-level token communication with near-lossless continuous channels. Three active research directions:
- Large-scale compressor training across task families
- Native latent communication pretraining
- Hybrid latent + tool-use communication
This wiki's latent-communication and method-comparison provide deep background on the methods underlying their work — particularly KV-cache compression (KVComm, C2C), embedding-space communication (CIPHER, SDE), and the theoretical foundations of why continuous channels outperform discrete ones.
The methodology behind this vault. Karpathy's gist describes a three-layer architecture — raw sources (immutable), the wiki (LLM-generated), and the schema (configuration) — where the LLM acts as an active maintainer rather than a passive retriever.
This repo is a production implementation of that pattern applied to a focused research domain. Our schema (AGENTS.md) defines page types, linking conventions, depth standards, and workflows. The LLM ingests papers, updates 10-15 pages per source, and maintains cross-references across 1400+ links — exactly the "bookkeeping" that makes knowledge bases collapse when maintained by hand.
Latent Reasoning (7 papers)
| Year | Paper | Authors | Venue |
|---|---|---|---|
| 2023 | Think Before You Speak: Pause Tokens | Goyal et al. | ICLR 2024 |
| 2024 | From Explicit CoT to Implicit CoT (iCoT) | Deng, Choi, Shieber | arXiv |
| 2024 | Coconut: Reasoning in Continuous Latent Space | Hao et al. | ICLR 2025 |
| 2025 | SoftCoT: Soft Chain-of-Thought | Xu et al. | ACL 2025 |
| 2025 | Reasoning by Superposition | Zhu et al. | NeurIPS 2025 |
| 2025 | Towards Inference-time Scaling for Continuous Space Reasoning | Wang et al. | arXiv |
| 2026 | Thinking States: Supervised Latent Reasoning | Amos et al. | arXiv |
Latent Communication (12 papers)
| Year | Paper | Authors | Venue |
|---|---|---|---|
| 2023 | CIPHER: Multiagent Debate through Embeddings | Pham et al. | ICLR 2024 |
| 2025 | Communicating Activations Between LM Agents | Ramesh, Li | ICML 2025 |
| 2025 | SDE: State Delta Trajectory | Tang et al. | EMNLP 2025 |
| 2025 | KVComm: Selective KV Sharing | Shi et al. | ICLR 2026 |
| 2025 | Cache-to-Cache (C2C) | Fu et al. | ICLR 2026 |
| 2025 | KVCOMM: Online Cross-context KV Communication | Ye et al. | NeurIPS 2025 |
| 2025 | ThoughtComm: Thought Communication in Multiagent | Zheng et al. | NeurIPS 2025 |
| 2025 | Interlat: Agents Communicating in Latent Space | Du et al. | OpenReview |
| 2025 | LatentMAS: Latent Collaboration in MAS | Zou et al. | arXiv |
| 2026 | KV-Cache Alignment (Shared Space) | Dery et al. | arXiv |
| 2026 | Vision Wormhole: Heterogeneous MAS | Liu et al. | arXiv |
| 2026 | Agent Primitives: Latent Building Blocks | Jin et al. | arXiv |
Theoretical Foundations & Meta (8 papers)
| Year | Paper | Authors | Venue |
|---|---|---|---|
| 2022 | Relative Representations: Zero-Shot Latent Space Communication | Moschella et al. | ICLR 2023 |
| 2023 | Towards Revealing the Mystery behind Chain of Thought | Feng et al. | NeurIPS 2023 |
| 2023 | Multiagent Debate for Factuality and Reasoning | Du et al. | arXiv |
| 2023 | Linearity of Relation Decoding in Transformers | Hernandez et al. | ICLR 2024 |
| 2024 | The Platonic Representation Hypothesis | Huh et al. | ICML 2024 |
| 2025 | Scaling Agent Systems | Kim et al. | arXiv |
| 2026 | How Do Latent Reasoning Methods Perform Under Weak and Strong Supervision? | Cui et al. | arXiv |
| — | LatentCompress Open Call | billion-token-one-task | GitHub |
latent-space-reasoning continuous-thought chain-of-thought multi-agent-systems LLM-communication KV-cache-sharing activation-communication embedding-space transformer-internals hidden-state-transfer vision-language-models self-distillation representation-learning platonic-representation-hypothesis multi-agent-debate training-free-inference latent-variable-models disentangled-representations cross-architecture practical-systems theoretical-foundations
This repository uses split licensing:
- Code and automation are licensed under Apache-2.0. See
LICENSE-APACHE. - Original wiki and documentation content are licensed under CC BY 4.0. See
LICENSE-CC-BY.
Third-party papers, PDFs, LaTeX sources, and other imported research artifacts are not covered by these licenses and remain under their original terms.
Built by CompleteTech LLC AI Research
Methodology based on the LLM Wiki pattern by Andrej Karpathy.
Research wiki content is provided for educational and research purposes.
Source PDFs in raw/pdf/ are subject to their respective publishers' copyright terms.