tok/CITATION.cff at main · GrayCodeAI/tok · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
cff-version: 1.2.0
title: "Tok: Token-Aware CLI Proxy with 31-Layer Compression Pipeline"
message: "If you use Tok in your research, please cite it using these metadata."
type: software
authors:
  - name: "GrayCode AI"
    website: "https://github.com/lakshmanpatel"
repository-code: "https://github.com/GrayCodeAI/tok"
url: "https://github.com/GrayCodeAI/tok"
license: MIT
version: "0.28.2"
date-released: "2026-04-03"
keywords:
  - token-reduction
  - llm
  - ai-coding-assistant
  - compression
  - cli-proxy
  - context-window
  - claude-code
  - cursor
  - copilot
abstract: >-
  Tok is a token-aware CLI proxy that intercepts CLI commands and applies
  a 31-layer compression pipeline to reduce token usage for AI coding assistants.
  Built on research from 120+ papers, it achieves 60-90% token reduction on
  common development operations. The pipeline includes entropy filtering,
  perplexity pruning, goal-driven selection, AST preservation, contrastive
  ranking, and 15+ additional research-backed compression layers.
references:
  - type: article
    title: "Selective Context for Language Models"
    authors:
      - family-names: "Li"
        given-names: "Yucheng"
    year: 2023
    notes: "Foundation for Layer 1: Entropy Filtering"
  - type: article
    title: "LLMLingua: Compressing Prompts for Accelerated Inference"
    authors:
      - family-names: "Jiang"
        given-names: "Huiqiang"
    year: 2023
    notes: "Foundation for Layer 2: Perplexity Pruning"
  - type: article
    title: "LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios"
    authors:
      - family-names: "Jiang"
        given-names: "Huiqiang"
    year: 2024
    notes: "Foundation for Layer 5: Contrastive Ranking"
  - type: article
    title: "H2O: Heavy-Hitter Oracle for Efficient Generative Inference"
    year: 2023
    notes: "Foundation for Layer 13: H2O Filter"
  - type: article
    title: "Efficient Streaming Language Models with Attention Sinks"
    year: 2023
    notes: "Foundation for Layer 14: Attention Sink"