Skip to content

Commit feb0d78

Browse files
committed
feat: Remove delta encoding and simplify example modes.
1 parent 7ba1c57 commit feb0d78

56 files changed

Lines changed: 167 additions & 826 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -367,7 +367,7 @@ ZON treats the format as a first-class language:
367367
- 🎯 **JSON Data Model**: Encodes the same objects, arrays, and primitives as JSON with deterministic, lossless round-trips
368368
- 📐 **Minimal Syntax**: Explicit headers (`@(N)` for count, column list) eliminate ambiguity for LLMs
369369
- 🌊 **Streaming Support** (New): Process gigabytes of data with `ZonStreamEncoder`/`Decoder`**Unique to ZON**
370-
- 📉 **Delta Encoding** (New): Sequential numbers are delta-encoded (`:delta`) for maximum compression – **Unique to ZON**
370+
371371
- 🧠 **LLM Optimization** (New): Context-aware encoding (`encodeLLM`) reorders fields for optimal tokenization – **Unique to ZON**
372372
- 🌳 **Deep Nesting**: Handles complex nested structures efficiently (91% compression on 50-level deep objects)
373373
- 🌐 **Browser & Edge Ready**: Verified support for Cloudflare Workers, Vercel Edge, and Browsers
@@ -688,7 +688,7 @@ true,carol@example.com,3,"Carol White",editor
688688

689689
### 2. Compact Mode
690690

691-
Maximum compression using tables, delta encoding, and abbreviations (`T`/`F` for booleans).
691+
Maximum compression using tables and abbreviations (`T`/`F` for booleans).
692692

693693
```typescript
694694
const result = encodeAdaptive(data, { mode: 'compact' });
@@ -850,7 +850,7 @@ company{departments{engineering{employees[{id:1,name:Alice}]},sales{employees[{i
850850
**Why flat is better:**
851851
1. **Tables work**: Uniform data → table format → massive token savings
852852
2. **Compression wins**: Repeated keys eliminated via column headers
853-
3. **Delta encoding**: Sequential IDs compressed (1, +1, +1 vs 1, 2, 3)
853+
3. **Dictionary compression**: Repeated strings stored once
854854
4. **LLM friendly**: Tabular data is easier for AI to parse and understand
855855

856856
### Mode Selection Guide
@@ -1199,12 +1199,12 @@ Comprehensive guides for every aspect of ZON:
11991199
- **[Syntax Cheatsheet](./docs/syntax-cheatsheet.md)** - Quick reference guide
12001200
- All ZON syntax at a glance
12011201
- Primitives, objects, arrays, tables
1202-
- Special features (delta, dictionary, metadata)
1202+
- Special features (dictionary, metadata)
12031203

12041204
### 🚀 Feature Guides
12051205

12061206
- **[Advanced Features](./docs/advanced-features.md)** - Optimization techniques
1207-
- Delta Encoding for sequential data
1207+
12081208
- Dictionary Compression for repeated values
12091209
- Type Coercion for LLM outputs
12101210
- LLM-Aware Field Ordering

SPEC.md

Lines changed: 3 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,7 @@ Pattern: `^[a-zA-Z0-9_\-\.]+$`
403403
**Examples:**
404404
```zonf
405405
name:Alice
406+
full_name:Alice Smith
406407
user_id:u123
407408
version:v1.1.0
408409
api-key:sk_test_key
@@ -415,7 +416,7 @@ Quote strings if they:
415416
1. **Contain structural chars:** `,`, `:`, `[`, `]`, `{`, `}`, `"`
416417
2. **Match literal keywords:** `T`, `F`, `true`, `false`, `null`, `none`, `nil`
417418
3. **Look like PURE numbers:** `123`, `3.14`, `1e6` (Complex patterns like `192.168.1.1` or `v1.1.0` do NOT need quoting)
418-
4. **Have whitespace:** Leading/trailing spaces, internal spaces (MUST quote to preserve)
419+
4. **Have whitespace:** Leading/trailing spaces (MUST quote to preserve). Internal spaces are allowed in unquoted strings.
419420
5. **Are empty:** `""` (MUST quote to distinguish from `null`)
420421
6. **Contain escapes:** Newlines, tabs, quotes (MUST quote to prevent structure breakage)
421422

@@ -586,28 +587,6 @@ users:@(3):id,name
586587
```json
587588
{"id": 2, "name": "Bob", "role": "admin", "score": 98}
588589
```
589-
590-
### 10.5 Delta Encoding (v1.1.0)
591-
592-
Sequential numeric columns can be delta-encoded to save tokens.
593-
594-
**Header Syntax:** `key:@(N):col1,col2:delta`
595-
596-
```zonf
597-
timeseries:@(3):ts:delta,val
598-
1000,10
599-
+60,12
600-
+60,15
601-
```
602-
603-
**Decodes to:**
604-
```json
605-
[
606-
{"ts": 1000, "val": 10},
607-
{"ts": 1060, "val": 12},
608-
{"ts": 1120, "val": 15}
609-
]
610-
```
611590

612591
### 10.6 Hierarchical Sparse Encoding (v1.1.0)
613592

@@ -1073,7 +1052,7 @@ users:@(1):id,name
10731052
### Appendix C: Changelog
10741053

10751054
**v1.1.0 (2025-12-01)**
1076-
- Delta Encoding (`:delta`)
1055+
- Sparse Tables (`key:value` in rows)
10771056
- Hierarchical Sparse Encoding (deep flattening)
10781057
- Metadata Optimization (grouped objects)
10791058
- Type Coercion (opt-in)

debug_decoder.ts

Lines changed: 0 additions & 37 deletions
This file was deleted.

docs/advanced-features.md

Lines changed: 2 additions & 96 deletions
Original file line numberDiff line numberDiff line change
@@ -9,108 +9,14 @@ ZON includes advanced compression and optimization features that dramatically re
99

1010
## Table of Contents
1111

12-
- [Delta Encoding](#delta-encoding)
12+
1313
- [Dictionary Compression](#dictionary-compression)
1414
- [Type Coercion](#type-coercion)
1515
- [LLM-Aware Field Ordering](#llm-aware-field-ordering)
1616
- [Hierarchical Sparse Encoding](#hierarchical-sparse-encoding)
1717

1818
---
1919

20-
## Delta Encoding
21-
22-
**Introduced:** v1.1.0
23-
**Purpose:** Compress sequential numeric columns
24-
25-
### How It Works
26-
27-
Instead of storing absolute values, delta encoding stores the difference from the previous value:
28-
29-
```
30-
# Without delta:
31-
ids:@(1000):id
32-
1,2,3,4,5,...,1000
33-
34-
# With delta (`:delta` marker):
35-
ids:@(1000):id:delta
36-
1,+1,+1,+1,+1,...,+1
37-
```
38-
39-
**Token Savings:** Up to 70% for sequential IDs or timestamps.
40-
41-
### When To Use
42-
43-
Delta encoding is automatically applied when ALL conditions are met:
44-
45-
1. Column contains **only numbers**
46-
2. Column has **≥5 values**
47-
3. Values are **sequential** (small deltas)
48-
49-
### Examples
50-
51-
```typescript
52-
import { encode } from 'zon-format';
53-
54-
// Sequential IDs
55-
const data = {
56-
records: Array.from({ length: 1000 }, (_, i) => ({
57-
id: i + 1,
58-
name: `Record ${i}`
59-
}))
60-
};
61-
62-
const zon = encode(data);
63-
console.log(zon);
64-
/*
65-
records:@(1000):id:delta,name
66-
1,Record 0
67-
+1,Record 1
68-
+1,Record 2
69-
...
70-
*/
71-
```
72-
73-
**Timestamps:**
74-
75-
```typescript
76-
const logs = [
77-
{ timestamp: 1609459200, message: 'Started' },
78-
{ timestamp: 1609459260, message: 'Processing' }, // +60
79-
{ timestamp: 1609459320, message: 'Done' } // +60
80-
];
81-
82-
// Encoded as:
83-
// logs:@(3):message,timestamp:delta
84-
// Started,1609459200
85-
// Processing,+60
86-
// Done,+60
87-
```
88-
89-
### Decoding
90-
91-
Delta encoding is automatically reversed during decoding:
92-
93-
```typescript
94-
import { decode } from 'zon-format';
95-
96-
const zon = `
97-
records:@(3):id:delta,name
98-
1,Alice
99-
+1,Bob
100-
+1,Carol
101-
`;
102-
103-
const data = decode(zon);
104-
console.log(data.records);
105-
// [
106-
// { id: 1, name: 'Alice' },
107-
// { id: 2, name: 'Bob' },
108-
// { id: 3, name: 'Carol' }
109-
// ]
110-
```
111-
112-
---
113-
11420
## Dictionary Compression
11521

11622
**Introduced:** v1.0.3
@@ -400,7 +306,7 @@ const data = {
400306

401307
## Performance Tips
402308

403-
1. **Delta encoding**: Best for time-series and sequential IDs
309+
1. **Dictionary compression**: Best for repetitive strings (enums, categories)
404310
2. **Dictionary compression**: Best for categorical data (status, roles, countries)
405311
3. **Type coercion**: Enable when dealing with LLM outputs
406312
4. **Field ordering**: Use for retrieval-heavy applications

docs/production-architecture.md

Lines changed: 11 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -93,20 +93,17 @@ type UserV2 = {
9393

9494
## Adaptive Encoding Patterns
9595

96-
### Automatic Mode Selection
97-
98-
```typescript
99-
import { encodeAdaptive, recommendMode } from 'zon-format';
100-
101-
// Get recommendation
102-
const rec = recommendMode(data);
103-
console.log(`Recommended: ${rec.mode} (${rec.confidence}% confident)`);
104-
console.log(`Reason: ${rec.reason}`);
105-
106-
// Encode with auto mode
107-
const result = encodeAdaptive(data, { mode: 'auto' });
108-
console.log(`Used ${result.selectedMode} mode`);
109-
```
96+
### Mode Selection
97+
98+
```typescript
99+
import { encodeAdaptive } from 'zon-format';
100+
101+
// Encode with compact mode (default)
102+
const compact = encodeAdaptive(data, { mode: 'compact' });
103+
104+
// Encode with readable mode for debugging
105+
const readable = encodeAdaptive(data, { mode: 'readable' });
106+
```
110107

111108
### Use Case-Specific Modes
112109

examples/encoding_modes_demo.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ console.log("\n");
4444
// ==========================================
4545
// 3. LLM-OPTIMIZED MODE - Balance clarity & tokens
4646
// ==========================================
47-
console.log("3. LLM-OPT IMIZED MODE (balance clarity & tokens):");
47+
console.log("3. LLM-OPTIMIZED MODE (balance clarity & tokens):");
4848
console.log("--------------------------------------------------");
4949
const llmResult = encodeAdaptive(sampleData, { mode: 'llm-optimized' });
5050
console.log(llmResult);

examples/modes/01_simple_key_value_auto.zonf

Lines changed: 0 additions & 5 deletions
This file was deleted.

examples/modes/01_simple_key_value_compact.zonf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
active:true
1+
active:T
22
description:null
33
name:ZON Format
44
score:98.5

examples/modes/01_simple_key_value_llm.zonf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
active:T
1+
active:true
22
description:null
33
name:ZON Format
44
score:98.5

examples/modes/02_array_of_primitives_auto.zonf

Lines changed: 0 additions & 1 deletion
This file was deleted.

0 commit comments

Comments
 (0)