Skip to content

Commit 594b463

Browse files
SimplyLizSimplyLiz
committed
chore: add CI/CD enhancements, linting, formatting, and code hardening
- Add ESLint (flat config) + Prettier with project conventions - Add @vitest/coverage-v8, upgrade vitest to v4 - Add audit, lint, and coverage jobs to CI workflow - Add version-tag validation and GitHub Release to publish workflow - Add Dependabot config for npm and GitHub Actions - Add input validation to uncompress(), defensive guard in forceConvergePass - Fix lint errors: unused imports/interfaces, useless regex escapes - Format entire codebase with Prettier - Remove stale pnpm-lock.yaml - Add headline benchmark numbers to README
1 parent 167ced1 commit 594b463

23 files changed

Lines changed: 4701 additions & 1829 deletions

‎.github/dependabot.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: npm
4+
directory: /
5+
schedule:
6+
interval: weekly
7+
day: monday
8+
target-branch: main
9+
open-pull-requests-limit: 10
10+
groups:
11+
production-deps:
12+
dependency-type: production
13+
update-types:
14+
- minor
15+
- patch
16+
dev-deps:
17+
dependency-type: development
18+
update-types:
19+
- minor
20+
- patch
21+
22+
- package-ecosystem: github-actions
23+
directory: /
24+
schedule:
25+
interval: weekly
26+
day: monday
27+
target-branch: main
28+
open-pull-requests-limit: 10

‎.github/workflows/ci.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
audit:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- uses: actions/setup-node@v4
15+
with:
16+
node-version: 22
17+
cache: npm
18+
- run: npm ci
19+
- run: npm audit --omit=dev --audit-level=high
20+
21+
lint:
22+
runs-on: ubuntu-latest
23+
steps:
24+
- uses: actions/checkout@v4
25+
- uses: actions/setup-node@v4
26+
with:
27+
node-version: 22
28+
cache: npm
29+
- run: npm ci
30+
- run: npx eslint .
31+
- run: npx prettier --check .
32+
33+
test:
34+
runs-on: ubuntu-latest
35+
strategy:
36+
matrix:
37+
node-version: [18, 20, 22]
38+
steps:
39+
- uses: actions/checkout@v4
40+
- uses: actions/setup-node@v4
41+
with:
42+
node-version: ${{ matrix.node-version }}
43+
cache: npm
44+
- run: npm ci
45+
- run: npm run test:coverage
46+
- run: npx tsc --noEmit

‎.github/workflows/publish.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: Publish
2+
3+
on:
4+
push:
5+
tags:
6+
- 'v*.*.*'
7+
8+
permissions:
9+
contents: write
10+
id-token: write
11+
12+
jobs:
13+
publish:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
- uses: actions/setup-node@v4
18+
with:
19+
node-version: 22
20+
registry-url: https://registry.npmjs.org
21+
cache: npm
22+
- run: npm ci
23+
24+
- name: Validate version tag
25+
run: |
26+
TAG_VERSION="${GITHUB_REF_NAME#v}"
27+
PKG_VERSION=$(node -p "require('./package.json').version")
28+
if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then
29+
echo "::error::Tag version ($TAG_VERSION) does not match package.json version ($PKG_VERSION)"
30+
exit 1
31+
fi
32+
33+
- run: npx eslint .
34+
- run: npx prettier --check .
35+
- run: npm test
36+
- run: npx tsc
37+
38+
- run: npm publish --provenance --access public
39+
env:
40+
NODE_AUTH_TOKEN: ${{ secrets.NODE_AUTH_TOKEN }}
41+
42+
- name: Extract release notes
43+
id: release_notes
44+
uses: ffurrer2/extract-release-notes@v3
45+
46+
- name: Create GitHub Release
47+
uses: softprops/action-gh-release@v2
48+
with:
49+
body: ${{ steps.release_notes.outputs.release_notes }}

‎.prettierignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
dist
2+
coverage
3+
node_modules
4+
package-lock.json

‎.prettierrc‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"semi": true,
3+
"singleQuote": true,
4+
"trailingComma": "all",
5+
"printWidth": 100,
6+
"tabWidth": 2
7+
}

‎README.md‎

Lines changed: 70 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
# context-compression-engine
22

3+
[![CI](https://github.com/SimplyLiz/ContextCompressionEngine/actions/workflows/ci.yml/badge.svg)](https://github.com/SimplyLiz/ContextCompressionEngine/actions/workflows/ci.yml)
34
[![npm version](https://img.shields.io/npm/v/context-compression-engine.svg)](https://www.npmjs.com/package/context-compression-engine)
45
[![license](https://img.shields.io/npm/l/context-compression-engine.svg)](LICENSE)
56

67
Lossless context compression for LLMs. Zero dependencies. Zero API calls. Works everywhere JavaScript runs.
78

9+
> **1.3-6.1x compression** on synthetic scenarios, **1.5x on real Claude Code sessions** (11.7M chars across 8,004 messages) — fully deterministic, no LLM needed. Largest session: 4,257 messages / 5.8M chars compressed in 651ms with zero negatives. Every compression is losslessly reversible.
10+
811
## The problem
912

1013
Context is the RAM of LLMs. As conversations grow, model attention spreads thin — a phenomenon known as **context rot**. Tokens spent on stale prose are tokens not spent on the task at hand.
@@ -35,7 +38,7 @@ The engine ships with a full benchmark suite that pits deterministic compression
3538

3639
**The deterministic engine achieves 1.3-6.1x compression with zero latency and zero cost.** It scores sentences, packs a budget, strips filler — and in most scenarios, it compresses tighter than an LLM.
3740

38-
Why? LLMs try to be *helpful*. They write fuller summaries that happen to be longer. The deterministic engine is optimized purely for compression — it doesn't care about readability, just signal density.
41+
Why? LLMs try to be _helpful_. They write fuller summaries that happen to be longer. The deterministic engine is optimized purely for compression — it doesn't care about readability, just signal density.
3942

4043
**LLM summarization is opt-in for cases where semantic understanding improves summary quality** — long, prose-heavy conversations where the LLM's ability to paraphrase and merge concepts across many messages genuinely helps. The engine supports this via a pluggable `summarizer` option with a built-in fallback chain that automatically rejects LLM output when it's longer than the deterministic result.
4144

@@ -80,9 +83,13 @@ Works in Node 18+, Deno, Bun, and edge runtimes. This is an ESM-only package —
8083
import { compress, uncompress } from 'context-compression-engine';
8184

8285
// compress — prose gets summarized, code stays verbatim
83-
const { messages: compressed, verbatim, compression } = compress(messages, {
84-
preserve: ['system'], // roles to never compress
85-
recencyWindow: 4, // protect the last N messages
86+
const {
87+
messages: compressed,
88+
verbatim,
89+
compression,
90+
} = compress(messages, {
91+
preserve: ['system'], // roles to never compress
92+
recencyWindow: 4, // protect the last N messages
8693
});
8794

8895
// uncompress — restore originals from the verbatim store
@@ -114,30 +121,30 @@ const result = await compress(messages, {
114121
},
115122
});
116123

117-
result.messages; // compressed message array
118-
result.verbatim; // original messages keyed by ID
119-
result.compression.ratio; // char compression ratio (>1 = savings)
120-
result.compression.token_ratio; // token compression ratio (>1 = savings)
121-
result.compression.messages_compressed; // how many were compressed
122-
result.compression.messages_preserved; // how many were kept as-is
123-
result.compression.messages_deduped; // exact duplicates replaced (when dedup: true)
124-
result.compression.messages_fuzzy_deduped; // near-duplicates replaced (when fuzzyDedup: true)
124+
result.messages; // compressed message array
125+
result.verbatim; // original messages keyed by ID
126+
result.compression.ratio; // char compression ratio (>1 = savings)
127+
result.compression.token_ratio; // token compression ratio (>1 = savings)
128+
result.compression.messages_compressed; // how many were compressed
129+
result.compression.messages_preserved; // how many were kept as-is
130+
result.compression.messages_deduped; // exact duplicates replaced (when dedup: true)
131+
result.compression.messages_fuzzy_deduped; // near-duplicates replaced (when fuzzyDedup: true)
125132
```
126133

127-
| Option | Type | Default | Description |
128-
|---|---|---|---|
129-
| `preserve` | `string[]` | `['system']` | Roles to never compress |
130-
| `recencyWindow` | `number` | `4` | Protect the last N messages from compression |
131-
| `sourceVersion` | `number` | `0` | Version tag for provenance tracking |
132-
| `summarizer` | `Summarizer` | — | LLM-powered summarizer. When provided, `compress()` returns a `Promise` |
133-
| `tokenBudget` | `number` | — | Target token count. When set, binary-searches `recencyWindow` to fit |
134-
| `minRecencyWindow` | `number` | `0` | Floor for `recencyWindow` when using `tokenBudget` |
135-
| `dedup` | `boolean` | `true` | Replace earlier exact-duplicate messages with a compact reference |
136-
| `fuzzyDedup` | `boolean` | `false` | Detect near-duplicate messages using line-level similarity |
137-
| `fuzzyThreshold` | `number` | `0.85` | Similarity threshold for fuzzy dedup (0-1) |
138-
| `embedSummaryId` | `boolean` | `false` | Embed `summary_id` in compressed content for downstream reference |
139-
| `forceConverge` | `boolean` | `false` | Hard-truncate non-recency messages when binary search bottoms out and budget still exceeded |
140-
| `tokenCounter` | `(msg: Message) => number` | `defaultTokenCounter` | Custom token counter per message. Default: `ceil(content.length / 3.5)` |
134+
| Option | Type | Default | Description |
135+
| ------------------ | -------------------------- | --------------------- | ------------------------------------------------------------------------------------------- |
136+
| `preserve` | `string[]` | `['system']` | Roles to never compress |
137+
| `recencyWindow` | `number` | `4` | Protect the last N messages from compression |
138+
| `sourceVersion` | `number` | `0` | Version tag for provenance tracking |
139+
| `summarizer` | `Summarizer` | — | LLM-powered summarizer. When provided, `compress()` returns a `Promise` |
140+
| `tokenBudget` | `number` | — | Target token count. When set, binary-searches `recencyWindow` to fit |
141+
| `minRecencyWindow` | `number` | `0` | Floor for `recencyWindow` when using `tokenBudget` |
142+
| `dedup` | `boolean` | `true` | Replace earlier exact-duplicate messages with a compact reference |
143+
| `fuzzyDedup` | `boolean` | `false` | Detect near-duplicate messages using line-level similarity |
144+
| `fuzzyThreshold` | `number` | `0.85` | Similarity threshold for fuzzy dedup (0-1) |
145+
| `embedSummaryId` | `boolean` | `false` | Embed `summary_id` in compressed content for downstream reference |
146+
| `forceConverge` | `boolean` | `false` | Hard-truncate non-recency messages when binary search bottoms out and budget still exceeded |
147+
| `tokenCounter` | `(msg: Message) => number` | `defaultTokenCounter` | Custom token counter per message. Default: `ceil(content.length / 3.5)` |
141148

142149
#### Summarizer fallback
143150

@@ -163,7 +170,7 @@ const result = compress(messages, {
163170
minRecencyWindow: 2,
164171
});
165172

166-
result.fits; // true if result fits within budget
173+
result.fits; // true if result fits within budget
167174
result.tokenCount; // token count (via tokenCounter)
168175

169176
// Plug in a real tokenizer
@@ -237,16 +244,17 @@ For domain-specific compression, use `systemPrompt` to inject context:
237244

238245
```ts
239246
const summarizer = createSummarizer(callLlm, {
240-
systemPrompt: 'This is a legal contract. Preserve all clause numbers, party names, and defined terms.',
247+
systemPrompt:
248+
'This is a legal contract. Preserve all clause numbers, party names, and defined terms.',
241249
});
242250
```
243251

244-
| Option | Type | Default | Description |
245-
|---|---|---|---|
246-
| `maxResponseTokens` | `number` | `300` | Hint for maximum tokens in the LLM response |
247-
| `systemPrompt` | `string` | — | Domain-specific instructions prepended to the built-in rules |
248-
| `mode` | `'normal' \| 'aggressive'` | `'normal'` | `'aggressive'` produces terse bullet points at half the token budget |
249-
| `preserveTerms` | `string[]` | — | Domain-specific terms appended to the built-in preserve list |
252+
| Option | Type | Default | Description |
253+
| ------------------- | -------------------------- | ---------- | -------------------------------------------------------------------- |
254+
| `maxResponseTokens` | `number` | `300` | Hint for maximum tokens in the LLM response |
255+
| `systemPrompt` | `string` | — | Domain-specific instructions prepended to the built-in rules |
256+
| `mode` | `'normal' \| 'aggressive'` | `'normal'` | `'aggressive'` produces terse bullet points at half the token budget |
257+
| `preserveTerms` | `string[]` | — | Domain-specific terms appended to the built-in preserve list |
250258

251259
### createEscalatingSummarizer
252260

@@ -259,23 +267,20 @@ Three-level escalation summarizer (normal → aggressive → deterministic fallb
259267
```ts
260268
import { createEscalatingSummarizer, compress } from 'context-compression-engine';
261269

262-
const summarizer = createEscalatingSummarizer(
263-
async (prompt) => myLlm.complete(prompt),
264-
{
265-
maxResponseTokens: 300,
266-
systemPrompt: 'This is a legal contract. Preserve all clause numbers.',
267-
preserveTerms: ['clause numbers', 'party names'],
268-
},
269-
);
270+
const summarizer = createEscalatingSummarizer(async (prompt) => myLlm.complete(prompt), {
271+
maxResponseTokens: 300,
272+
systemPrompt: 'This is a legal contract. Preserve all clause numbers.',
273+
preserveTerms: ['clause numbers', 'party names'],
274+
});
270275

271276
const result = await compress(messages, { summarizer });
272277
```
273278

274-
| Option | Type | Default | Description |
275-
|---|---|---|---|
276-
| `maxResponseTokens` | `number` | `300` | Hint for maximum tokens in the LLM response |
277-
| `systemPrompt` | `string` | — | Domain-specific instructions prepended to the built-in rules |
278-
| `preserveTerms` | `string[]` | — | Domain-specific terms appended to the built-in preserve list |
279+
| Option | Type | Default | Description |
280+
| ------------------- | ---------- | ------- | ------------------------------------------------------------ |
281+
| `maxResponseTokens` | `number` | `300` | Hint for maximum tokens in the LLM response |
282+
| `systemPrompt` | `string` | — | Domain-specific instructions prepended to the built-in rules |
283+
| `preserveTerms` | `string[]` | — | Domain-specific terms appended to the built-in preserve list |
279284

280285
Note: `mode` is not accepted — the escalating summarizer manages both modes internally.
281286

@@ -300,6 +305,7 @@ const result = compress(messages, { fuzzyDedup: true });
300305
Detects near-duplicates using line-level Jaccard similarity. Useful when the same file is read across edit cycles — the content evolves slightly but remains largely the same.
301306

302307
The algorithm:
308+
303309
1. **Fingerprint bucketing** — groups candidates by their first 5 non-empty normalized lines (requires 3+ shared)
304310
2. **Length-ratio pre-filter** — skips pairs where `min/max < 0.7`
305311
3. **Line-level Jaccard** — `|A ∩ B| / |A ∪ B|` using multiset frequency maps of normalized lines
@@ -445,19 +451,19 @@ If the summarizer throws or returns text longer than the input, the engine falls
445451

446452
The classifier automatically preserves content that should never be summarized:
447453

448-
| Content Type | Example | Preserved? |
449-
|---|---|---|
450-
| Code fences | `` ```ts const x = 1; ``` `` | Yes |
451-
| SQL | `SELECT * FROM users WHERE ...` | Yes |
452-
| JSON | `{"key": "value"}` | Yes |
453-
| API keys | `sk-proj-abc123...` | Yes |
454-
| URLs | `https://docs.example.com/api` | Yes |
455-
| File paths | `/etc/config.json` | Yes |
456-
| Short messages | `< 120 chars` | Yes |
457-
| Tool calls | Messages with `tool_calls` array | Yes |
458-
| System messages | `role: 'system'` (default) | Yes |
459-
| Duplicates | Repeated content (exact or fuzzy) | **Replaced with reference** |
460-
| Long prose | General discussion, explanations | **Compressed** |
454+
| Content Type | Example | Preserved? |
455+
| --------------- | --------------------------------- | --------------------------- |
456+
| Code fences | ` ```ts const x = 1; ``` ` | Yes |
457+
| SQL | `SELECT * FROM users WHERE ...` | Yes |
458+
| JSON | `{"key": "value"}` | Yes |
459+
| API keys | `sk-proj-abc123...` | Yes |
460+
| URLs | `https://docs.example.com/api` | Yes |
461+
| File paths | `/etc/config.json` | Yes |
462+
| Short messages | `< 120 chars` | Yes |
463+
| Tool calls | Messages with `tool_calls` array | Yes |
464+
| System messages | `role: 'system'` (default) | Yes |
465+
| Duplicates | Repeated content (exact or fuzzy) | **Replaced with reference** |
466+
| Long prose | General discussion, explanations | **Compressed** |
461467

462468
Code-mixed messages get split: prose is summarized, code fences stay verbatim.
463469

@@ -514,11 +520,11 @@ The benchmark covers seven conversation scenarios (coding assistant, long Q&A, t
514520

515521
The benchmark runner includes an opt-in LLM section that compares deterministic compression against real LLM-powered summarization head-to-head. Set one or more environment variables to enable it:
516522

517-
| Variable | Provider | Default model |
518-
|---|---|---|
519-
| `OPENAI_API_KEY` | OpenAI | `gpt-4.1-mini` (override: `OPENAI_MODEL`) |
523+
| Variable | Provider | Default model |
524+
| ------------------- | --------- | --------------------------------------------------------- |
525+
| `OPENAI_API_KEY` | OpenAI | `gpt-4.1-mini` (override: `OPENAI_MODEL`) |
520526
| `ANTHROPIC_API_KEY` | Anthropic | `claude-haiku-4-5-20251001` (override: `ANTHROPIC_MODEL`) |
521-
| `OLLAMA_MODEL` | Ollama | `llama3.2` (host override: `OLLAMA_HOST`) |
527+
| `OLLAMA_MODEL` | Ollama | `llama3.2` (host override: `OLLAMA_HOST`) |
522528

523529
```bash
524530
# Run with OpenAI

‎bench/llm.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,9 @@ export async function detectProviders(): Promise<LlmProvider[]> {
6969
},
7070
});
7171
} catch (err) {
72-
console.log(` OpenAI SDK not installed (needed for Ollama), skipping (${(err as Error).message})`);
72+
console.log(
73+
` OpenAI SDK not installed (needed for Ollama), skipping (${(err as Error).message})`,
74+
);
7375
}
7476
}
7577

0 commit comments

Comments
 (0)