Skip to content

Commit e4d65e2

Browse files
SimplyLizSimplyLiz
committed
docs: add comprehensive documentation directory
Split monolithic README into 9 focused docs pages: - api-reference, compression-pipeline, deduplication, token-budget, llm-integration, round-trip, provenance, preservation-rules, benchmarks All content verified against source code.
1 parent 8d70fb7 commit e4d65e2

9 files changed

Lines changed: 1637 additions & 0 deletions

‎docs/api-reference.md‎

Lines changed: 297 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
# API Reference
2+
3+
Complete reference for all exports from `context-compression-engine`.
4+
5+
## Exports
6+
7+
```ts
8+
// Primary
9+
export { compress, defaultTokenCounter } from './compress.js';
10+
export { uncompress } from './expand.js';
11+
export type { StoreLookup } from './expand.js';
12+
13+
// Helpers (LLM integration)
14+
export { createSummarizer, createEscalatingSummarizer } from './summarizer.js';
15+
16+
// Types
17+
export type {
18+
CompressOptions,
19+
CompressResult,
20+
CreateSummarizerOptions,
21+
Message,
22+
Summarizer,
23+
UncompressOptions,
24+
UncompressResult,
25+
VerbatimMap,
26+
} from './types.js';
27+
```
28+
29+
---
30+
31+
## `compress`
32+
33+
Deterministic compression by default. Returns a `Promise` when a `summarizer` is provided.
34+
35+
### Signatures
36+
37+
```ts
38+
function compress(messages: Message[], options?: CompressOptions): CompressResult;
39+
function compress(
40+
messages: Message[],
41+
options: CompressOptions & { summarizer: Summarizer },
42+
): Promise<CompressResult>;
43+
```
44+
45+
### Parameters
46+
47+
| Parameter | Type | Description |
48+
| ---------- | ----------------- | ------------------ |
49+
| `messages` | `Message[]` | Messages to compress |
50+
| `options` | `CompressOptions` | Compression options (see below) |
51+
52+
### CompressOptions
53+
54+
| Option | Type | Default | Description |
55+
| ------------------ | -------------------------- | --------------------- | ------------------------------------------------------------------------------------------- |
56+
| `preserve` | `string[]` | `['system']` | Roles to never compress |
57+
| `recencyWindow` | `number` | `4` | Protect the last N messages from compression |
58+
| `sourceVersion` | `number` | `0` | Version tag for [provenance tracking](provenance.md) |
59+
| `summarizer` | `Summarizer` | - | LLM-powered summarizer. When provided, `compress()` returns a `Promise`. See [LLM integration](llm-integration.md) |
60+
| `tokenBudget` | `number` | - | Target token count. Binary-searches `recencyWindow` to fit. See [Token budget](token-budget.md) |
61+
| `minRecencyWindow` | `number` | `0` | Floor for `recencyWindow` when using `tokenBudget` |
62+
| `dedup` | `boolean` | `true` | Replace earlier exact-duplicate messages with a compact reference. See [Deduplication](deduplication.md) |
63+
| `fuzzyDedup` | `boolean` | `false` | Detect near-duplicate messages using line-level similarity. See [Deduplication](deduplication.md) |
64+
| `fuzzyThreshold` | `number` | `0.85` | Similarity threshold for fuzzy dedup (0-1) |
65+
| `embedSummaryId` | `boolean` | `false` | Embed `summary_id` in compressed content for downstream reference. See [Provenance](provenance.md) |
66+
| `forceConverge` | `boolean` | `false` | Hard-truncate non-recency messages when binary search bottoms out. See [Token budget](token-budget.md) |
67+
| `tokenCounter` | `(msg: Message) => number` | `defaultTokenCounter` | Custom token counter per message. See [Token budget](token-budget.md) |
68+
69+
### CompressResult
70+
71+
| Field | Type | Description |
72+
| ------------------------------------ | ----------------------- | ------------------------------------------------------------------------ |
73+
| `messages` | `Message[]` | Compressed message array |
74+
| `verbatim` | `VerbatimMap` | Original messages keyed by ID. Must be persisted atomically with `messages` |
75+
| `compression.original_version` | `number` | Mirrors `sourceVersion` |
76+
| `compression.ratio` | `number` | Character-based compression ratio. >1 means savings |
77+
| `compression.token_ratio` | `number` | Token-based compression ratio. >1 means savings |
78+
| `compression.messages_compressed` | `number` | Messages that were compressed |
79+
| `compression.messages_preserved` | `number` | Messages kept as-is |
80+
| `compression.messages_deduped` | `number \| undefined` | Exact duplicates replaced (when `dedup: true`) |
81+
| `compression.messages_fuzzy_deduped` | `number \| undefined` | Near-duplicates replaced (when `fuzzyDedup: true`) |
82+
| `fits` | `boolean \| undefined` | Whether result fits within `tokenBudget`. Present when `tokenBudget` is set |
83+
| `tokenCount` | `number \| undefined` | Estimated token count. Present when `tokenBudget` is set |
84+
| `recencyWindow` | `number \| undefined` | The `recencyWindow` the binary search settled on. Present when `tokenBudget` is set |
85+
86+
### Example
87+
88+
```ts
89+
import { compress } from 'context-compression-engine';
90+
91+
// Sync
92+
const result = compress(messages, {
93+
preserve: ['system'],
94+
recencyWindow: 4,
95+
sourceVersion: 1,
96+
});
97+
98+
// Async (with LLM summarizer)
99+
const result = await compress(messages, {
100+
summarizer: async (text) => myLlm.summarize(text),
101+
});
102+
```
103+
104+
---
105+
106+
## `uncompress`
107+
108+
Restore originals from the verbatim store. Always synchronous. See [Round-trip](round-trip.md) for full details.
109+
110+
### Signature
111+
112+
```ts
113+
function uncompress(
114+
messages: Message[],
115+
store: StoreLookup,
116+
options?: UncompressOptions,
117+
): UncompressResult;
118+
```
119+
120+
### Parameters
121+
122+
| Parameter | Type | Description |
123+
| ---------- | ------------------ | ----------- |
124+
| `messages` | `Message[]` | Compressed messages to expand |
125+
| `store` | `StoreLookup` | `VerbatimMap` object or `(id: string) => Message \| undefined` function |
126+
| `options` | `UncompressOptions` | Expansion options (see below) |
127+
128+
### UncompressOptions
129+
130+
| Option | Type | Default | Description |
131+
| ----------- | --------- | ------- | ----------- |
132+
| `recursive` | `boolean` | `false` | Recursively expand messages whose originals are also compressed (up to 10 levels) |
133+
134+
### UncompressResult
135+
136+
| Field | Type | Description |
137+
| --------------------- | ---------- | ----------- |
138+
| `messages` | `Message[]` | Expanded messages |
139+
| `messages_expanded` | `number` | How many compressed messages were restored |
140+
| `messages_passthrough` | `number` | How many messages passed through unchanged |
141+
| `missing_ids` | `string[]` | IDs looked up but not found. Non-empty = data loss |
142+
143+
### Example
144+
145+
```ts
146+
import { uncompress } from 'context-compression-engine';
147+
148+
const { messages, missing_ids } = uncompress(compressed, verbatim);
149+
150+
// Recursive expansion
151+
const deep = uncompress(compressed, verbatim, { recursive: true });
152+
153+
// Function store (database-backed)
154+
const result = uncompress(compressed, (id) => db.getMessageById(id));
155+
```
156+
157+
---
158+
159+
## `defaultTokenCounter`
160+
161+
The built-in token estimator used when no custom `tokenCounter` is provided.
162+
163+
### Signature
164+
165+
```ts
166+
function defaultTokenCounter(msg: Message): number;
167+
```
168+
169+
### Formula
170+
171+
```ts
172+
Math.ceil(msg.content.length / 3.5)
173+
```
174+
175+
Approximates ~3.5 characters per token. Suitable for rough estimates. For accurate budgeting, replace with a real tokenizer. See [Token budget](token-budget.md).
176+
177+
---
178+
179+
## `createSummarizer`
180+
181+
Creates an LLM-powered summarizer with an optimized prompt template. See [LLM integration](llm-integration.md) for provider examples.
182+
183+
### Signature
184+
185+
```ts
186+
function createSummarizer(
187+
callLlm: (prompt: string) => string | Promise<string>,
188+
options?: CreateSummarizerOptions,
189+
): Summarizer;
190+
```
191+
192+
### CreateSummarizerOptions
193+
194+
| Option | Type | Default | Description |
195+
| ------------------- | -------------------------- | ---------- | -------------------------------------------------------------------- |
196+
| `maxResponseTokens` | `number` | `300` | Hint for maximum tokens in the LLM response |
197+
| `systemPrompt` | `string` | - | Domain-specific instructions prepended to the built-in rules |
198+
| `mode` | `'normal' \| 'aggressive'` | `'normal'` | `'aggressive'` produces terse bullet points at half the token budget |
199+
| `preserveTerms` | `string[]` | - | Domain-specific terms appended to the built-in preserve list |
200+
201+
### Built-in preserve list
202+
203+
The prompt always preserves: code references, file paths, function/variable names, URLs, API keys, error messages, numbers, and technical decisions. Add domain terms via `preserveTerms`.
204+
205+
### Example
206+
207+
```ts
208+
import { createSummarizer, compress } from 'context-compression-engine';
209+
210+
const summarizer = createSummarizer(
211+
async (prompt) => myLlm.complete(prompt),
212+
{
213+
maxResponseTokens: 300,
214+
systemPrompt: 'This is a legal contract. Preserve all clause numbers.',
215+
preserveTerms: ['clause numbers', 'party names'],
216+
},
217+
);
218+
219+
const result = await compress(messages, { summarizer });
220+
```
221+
222+
---
223+
224+
## `createEscalatingSummarizer`
225+
226+
Three-level escalation summarizer. See [LLM integration](llm-integration.md) and [Compression pipeline](compression-pipeline.md) for how the fallback chain works.
227+
228+
### Signature
229+
230+
```ts
231+
function createEscalatingSummarizer(
232+
callLlm: (prompt: string) => string | Promise<string>,
233+
options?: Omit<CreateSummarizerOptions, 'mode'>,
234+
): Summarizer;
235+
```
236+
237+
### Escalation levels
238+
239+
1. **Level 1: Normal** - concise prose summary via the LLM
240+
2. **Level 2: Aggressive** - terse bullet points at half the token budget (if Level 1 fails or returns longer text)
241+
3. **Level 3: Deterministic** - sentence extraction fallback via the compression pipeline (handled by `withFallback` in `compress`)
242+
243+
### Options
244+
245+
Same as `CreateSummarizerOptions` but without `mode` (managed internally).
246+
247+
| Option | Type | Default | Description |
248+
| ------------------- | ---------- | ------- | ------------------------------------------------------------ |
249+
| `maxResponseTokens` | `number` | `300` | Hint for maximum tokens in the LLM response |
250+
| `systemPrompt` | `string` | - | Domain-specific instructions prepended to the built-in rules |
251+
| `preserveTerms` | `string[]` | - | Domain-specific terms appended to the built-in preserve list |
252+
253+
---
254+
255+
## Types
256+
257+
### `Message`
258+
259+
```ts
260+
type Message = {
261+
id: string;
262+
index: number;
263+
role?: string;
264+
content?: string;
265+
metadata?: Record<string, unknown>;
266+
tool_calls?: unknown[];
267+
[key: string]: unknown;
268+
};
269+
```
270+
271+
### `Summarizer`
272+
273+
```ts
274+
type Summarizer = (text: string) => string | Promise<string>;
275+
```
276+
277+
### `VerbatimMap`
278+
279+
```ts
280+
type VerbatimMap = Record<string, Message>;
281+
```
282+
283+
### `StoreLookup`
284+
285+
```ts
286+
type StoreLookup = VerbatimMap | ((id: string) => Message | undefined);
287+
```
288+
289+
---
290+
291+
## See also
292+
293+
- [Compression pipeline](compression-pipeline.md) - how the engine processes messages
294+
- [Token budget](token-budget.md) - budget-driven compression
295+
- [LLM integration](llm-integration.md) - provider examples
296+
- [Round-trip](round-trip.md) - lossless compress/uncompress
297+
- [Provenance](provenance.md) - metadata tracking

0 commit comments

Comments
 (0)