Skip to content

Commit 7ebed9f

Browse files
Merge branch 'master' of github.com:LessUp/compress-kit
Integrate PR #9 architecture-deepening artifacts (ADRs 0001/0003/0004/0005, validation metadata module) into the C++17-only tree. Drop ADR 0002 (cross-language semantic error alignment) and docs/architecture/contract-inventory.md: both depend on the removed Go/Rust implementations and no longer apply after the C++17-only streamline. Resolve CONTEXT.md "参考资料" section: keep ADR references that still exist, drop the 0002 link and the deleted openspec/ spec links. Resolve docs/en/architecture/index.md CLI section: keep the C++17-only single-binary contract (`./build/<algo>_cpp`), drop the multi-language `--lang` and `<algo>_<lang>` invocations. Update tests/metadata.py to LANGUAGE_ORDER = ("cpp",) to match the C++17-only tree. make test && make lint pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2 parents 672b9fa + 6678b31 commit 7ebed9f

6 files changed

Lines changed: 371 additions & 0 deletions

CONTEXT.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,13 @@ C++ 实现的共享工具模块,位于 `algorithms/shared/cpp/`:
181181
| CHANGELOG.md | 用户可见变更 | 用户 |
182182
| CONTEXT.md | 领域词汇 | 贡献者/AI |
183183

184+
## 参考资料
185+
186+
- `docs/adr/0001-validation-metadata-module-shape.md` - 验证元数据模块形状决策
187+
- `docs/adr/0003-range-coder-corpus-cap-policy.md` - Range Coder 语料库上限策略决策
188+
- `docs/adr/0004-rle-buffered-streaming-stance.md` - RLE 缓冲流式语义决策
189+
- `docs/adr/0005-range-cpp-bench-mode-migration-path.md` - Range C++ bench 模式迁移路径决策
190+
184191
## 文档术语表
185192

186193
本文节定义 Git Pages 文档(VitePress)中的术语翻译规范。
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# ADR 0001: Validation metadata module shape
2+
3+
Date: 2026-05-31
4+
Status: Accepted
5+
6+
## Context
7+
8+
Conformance, CLI smoke, test-data generation, and benchmark orchestration scripts each hardcode overlapping facts: algorithm names, binary paths, file extensions, corpus names, Range corpus caps, benchmark dataset choices, language ordering, and report schema fields. This duplication causes drift when adding algorithms, adjusting corpus policy, or changing binary paths.
9+
10+
Current consumers and their metadata shapes are recorded in `docs/architecture/contract-inventory.md` (§ Validation and test metadata consumers, § Benchmark metadata consumers).
11+
12+
## Decision
13+
14+
Introduce a single versioned Python metadata module (`tests/metadata.py` or similar) that exposes:
15+
16+
- Algorithm registry: name, extension, binary paths per language
17+
- Corpus registry: corpus names, sizes, generation parameters, eligibility flags
18+
- Range corpus cap: explicit named constant with documented rationale
19+
- Benchmark job registry: per-algorithm selected input file
20+
- Report schema: field names and ordering for benchmark JSON output
21+
22+
Conformance, smoke, test-data, and benchmark runners become thin adapters over this module. The module is versioned with a `METADATA_VERSION` string; consumers assert the version they were tested against.
23+
24+
## Constraints
25+
26+
- This change does **not** alter algorithm/language matrix, corpus eligibility rules, Range cap value, or benchmark schema: it only consolidates existing facts.
27+
- Any intentional change to Range cap policy, corpus eligibility, or conformance matrix requires an OpenSpec change first (see `docs/architecture/contract-inventory.md` § OpenSpec-triggering decisions).
28+
- Python module must remain compatible with the existing `uv`/`python3` invocation patterns in the Makefile.
29+
30+
## Consequences
31+
32+
**Positive**: Single source of truth for test/bench metadata; adding an algorithm or corpus file requires one edit instead of many; Range cap policy is explicit and findable.
33+
34+
**Negative**: Additional indirection layer; scripts must import the module, requiring Python path management.
35+
36+
**Deferred**: Deciding whether the metadata module is the right place to record Range cap policy permanently, or whether that policy should move into OpenSpec. Record that decision in ADR 0003 before migrating.
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# ADR 0003: Range Coder corpus cap policy
2+
3+
Date: 2026-05-31
4+
Status: Accepted
5+
6+
## Context
7+
8+
The Range Coder has a known large-file decode performance problem. The conformance matrix currently skips corpus files over **100 KiB** with reason code `range_coder_corpus_cap_100_kib`. This cap is hardcoded in `tests/conformance/run_decode_matrix.py` and documented in:
9+
10+
- `tests/conformance/README.md` (workaround note)
11+
- `docs/en/algorithms/range.md` (known issue section)
12+
- `algorithms/range/benchmark/bench.py` (benchmark default input note)
13+
14+
The cap value and its rationale have never been formally recorded as a policy decision.
15+
16+
## Decision
17+
18+
The 100 KiB conformance skip cap is **retained as-is** and recorded here as a deliberate policy rather than an incidental constant.
19+
20+
Rationale:
21+
- The underlying performance issue is a known limitation documented in OpenSpec (`openspec/specs/cross-language-testing/spec.md`).
22+
- Fixing the performance issue requires a non-trivial algorithm change that may affect binary format or cross-language behavior, triggering a new OpenSpec change.
23+
- Raising the cap without fixing the root cause would cause CI to hang or time out, which is worse than an explicit skip.
24+
25+
The cap is **not permanent**. If the performance issue is resolved, the cap can be raised or removed, but doing so changes test-gate semantics and therefore requires an OpenSpec change before implementation.
26+
27+
## Constraints
28+
29+
- Do not silently raise, lower, or remove the cap without an OpenSpec change.
30+
- Do not treat the cap value (100 KiB) as canonical in user-facing docs; document it as "current workaround" to preserve flexibility.
31+
- When the metadata module (ADR 0001) is implemented, the cap must appear as a named constant with this ADR reference.
32+
33+
## Consequences
34+
35+
**Positive**: Policy is now explicit and findable; reviewers know the cap is intentional; future removal has a clear trigger condition.
36+
37+
**Negative**: The known limitation persists; Range Coder is under-tested on real-world corpus sizes.
38+
39+
**Deferred**: Root-cause fix for Range Coder large-file performance. That work requires a separate OpenSpec proposal and is out of scope for this architecture deepening cycle.
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# ADR 0004: RLE buffered streaming stance
2+
3+
Date: 2026-05-31
4+
Status: Accepted
5+
6+
## Context
7+
8+
RLE is naturally streamable: each `(count, value)` pair is independent and requires no look-ahead beyond the current run. However, the current Go and Rust RLE adapters use buffered streaming wrappers (`BufferedEncoder`/`BufferedDecoder`) that collect the full input before processing, identical to the static-model algorithms (Huffman, Arithmetic, Range).
9+
10+
From `contract-inventory.md` (§ Buffer and streaming contract):
11+
12+
> Flush is a no-op for buffered static-model algorithms until `Finish`; RLE is currently also buffered in Go/Rust adapters despite being naturally streamable.
13+
14+
The C++ RLE implementation reads/writes file streams incrementally but is wrapped through the C++ buffer adapter for the CLI layer.
15+
16+
## Decision
17+
18+
**Retain buffered semantics for RLE** in all three languages as the documented contract for this product version.
19+
20+
Rationale:
21+
- RLE shares the same `BufferedEncoder`/`BufferedDecoder` lifecycle as the other three algorithms; making it the only algorithm with true incremental streaming would create a two-tier streaming contract that complicates cross-language conformance tests.
22+
- The performance benefit of incremental RLE streaming is not material for the current corpus sizes (up to 10 MiB).
23+
- Implementing true incremental streaming for RLE would require a new streaming adapter path across all three languages — a significant scope increase.
24+
25+
The buffered semantic is **explicitly documented** rather than left implicit. API callers must not assume that `Process` writes any output before `Finish` is called.
26+
27+
## Constraints
28+
29+
- Do not change RLE's public streaming lifecycle behavior without an OpenSpec change (it is part of the public streaming/buffer API contract).
30+
- If a future OpenSpec change introduces true incremental streaming for RLE, the conformance test suite must be updated to assert per-chunk output behavior.
31+
32+
## Consequences
33+
34+
**Positive**: Consistent lifecycle across all four algorithms; simpler conformance tests; explicit stance eliminates future confusion.
35+
36+
**Negative**: RLE cannot be used for low-latency or memory-constrained streaming use cases; that limitation is now documented.
37+
38+
**Deferred**: True incremental RLE streaming as a future feature, subject to a new OpenSpec change.
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# ADR 0005: Range C++ bench mode migration path
2+
3+
Date: 2026-05-31
4+
Status: Accepted
5+
6+
## Context
7+
8+
The public CLI contract for all production binaries is:
9+
10+
```
11+
<binary> <encode|decode> <input> <output>
12+
```
13+
14+
Range C++ is the only exception: its production binary also accepts:
15+
16+
```
17+
<binary> bench [size_bytes] [iterations]
18+
```
19+
20+
This is documented in `contract-inventory.md` (§ CLI contract and exceptions) and recorded in `algorithms/range/cpp/main.cpp:378-415`.
21+
22+
The `bench` mode is used by `algorithms/range/benchmark/bench.py` for micro-benchmarking. It is a **shipped CLI behavior** even though it conflicts with the unified CLI adapter shape.
23+
24+
## Decision
25+
26+
The `bench` mode is **retained without change** in the current release cycle. This ADR records the migration path for when it is eventually removed.
27+
28+
### Migration path (to be executed in a future cycle with an OpenSpec change)
29+
30+
1. Open an OpenSpec change to officially deprecate the `bench` subcommand from the production Range C++ binary.
31+
2. Move benchmark behavior to a separate binary or script (e.g., `algorithms/range/benchmark/bench_driver` or extend `algorithms/range/benchmark/bench.py` to drive encoding/decoding directly).
32+
3. Update `algorithms/range/benchmark/bench.py` to use the new driver.
33+
4. Update CLI smoke tests to assert that `bench` is no longer a valid mode in the production binary.
34+
5. Remove the `bench` branch from `algorithms/range/cpp/main.cpp`.
35+
36+
### Current constraints
37+
38+
- Do not silently remove `bench` mode before the above steps; it is a shipped behavior even if undocumented in the primary contract.
39+
- CLI smoke tests currently test the `<encode|decode|invalid_mode>` contract; they do not assert that `bench` is **absent**. This is intentional until the migration is approved.
40+
41+
## Consequences
42+
43+
**Positive**: Migration path is explicit; no accidental removal; future reviewer knows the intent.
44+
45+
**Negative**: Range C++ production binary continues to violate the unified CLI contract until the migration is executed.
46+
47+
**Deferred**: Actual migration work. Requires a new OpenSpec change proposal before any code change.

tests/metadata.py

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
"""CompressKit test and benchmark metadata module.
2+
3+
Single source of truth for algorithm registry, corpus registry, Range corpus cap,
4+
benchmark job registry, and report schema. Conformance, smoke, test-data, and
5+
benchmark scripts are thin adapters over this module.
6+
7+
Design decision: docs/adr/0001-validation-metadata-module-shape.md
8+
Range corpus cap decision: docs/adr/0003-range-coder-corpus-cap-policy.md
9+
10+
Consumers must assert the version they depend on:
11+
12+
import metadata
13+
assert metadata.METADATA_VERSION == "1.0"
14+
"""
15+
16+
from __future__ import annotations
17+
18+
from dataclasses import dataclass, field
19+
from pathlib import Path
20+
21+
# ---------------------------------------------------------------------------
22+
# Version
23+
# ---------------------------------------------------------------------------
24+
25+
METADATA_VERSION = "1.0"
26+
27+
# ---------------------------------------------------------------------------
28+
# Paths
29+
# ---------------------------------------------------------------------------
30+
31+
ROOT = Path(__file__).resolve().parent.parent
32+
TESTS_DIR = ROOT / "tests"
33+
DATA_DIR = TESTS_DIR / "data"
34+
SCRIPTS_DIR = ROOT / "scripts"
35+
REPORTS_DIR = ROOT / "reports"
36+
DOCS_DATA_DIR = ROOT / "docs" / ".vitepress" / "data"
37+
38+
# ---------------------------------------------------------------------------
39+
# Algorithm and binary registry
40+
# ---------------------------------------------------------------------------
41+
42+
43+
@dataclass(frozen=True)
44+
class BinaryEntry:
45+
language: str
46+
path: Path
47+
48+
49+
@dataclass(frozen=True)
50+
class AlgorithmEntry:
51+
"""One algorithm supported by CompressKit."""
52+
53+
name: str
54+
"""Canonical lower-case algorithm name (e.g. 'huffman')."""
55+
56+
extension: str
57+
"""File extension used for encoded output (e.g. 'huf')."""
58+
59+
binaries: tuple[BinaryEntry, ...]
60+
"""Shipped binaries, in display order (cpp)."""
61+
62+
benchmark_input: str
63+
"""Corpus file name used as the benchmark input for this algorithm."""
64+
65+
benchmark_driver: Path
66+
"""Path to the per-algorithm benchmark driver script."""
67+
68+
69+
ALGORITHMS: tuple[AlgorithmEntry, ...] = (
70+
AlgorithmEntry(
71+
name="huffman",
72+
extension="huf",
73+
binaries=(
74+
BinaryEntry("cpp", ROOT / "algorithms/huffman/cpp/huffman_cpp"),
75+
BinaryEntry("go", ROOT / "algorithms/huffman/go/huffman_go"),
76+
BinaryEntry("rust", ROOT / "algorithms/huffman/rust/huffman_rust"),
77+
),
78+
benchmark_input="textlike_10MiB.bin",
79+
benchmark_driver=ROOT / "algorithms/huffman/benchmark/bench.py",
80+
),
81+
AlgorithmEntry(
82+
name="arithmetic",
83+
extension="aenc",
84+
binaries=(
85+
BinaryEntry("cpp", ROOT / "algorithms/arithmetic/cpp/arithmetic_cpp"),
86+
BinaryEntry("go", ROOT / "algorithms/arithmetic/go/arithmetic_go"),
87+
BinaryEntry("rust", ROOT / "algorithms/arithmetic/rust/arithmetic_rust"),
88+
),
89+
benchmark_input="textlike_10MiB.bin",
90+
benchmark_driver=ROOT / "algorithms/arithmetic/benchmark/bench.py",
91+
),
92+
AlgorithmEntry(
93+
name="range",
94+
extension="rcnc",
95+
binaries=(
96+
BinaryEntry("cpp", ROOT / "algorithms/range/cpp/rangecoder_cpp"),
97+
BinaryEntry("go", ROOT / "algorithms/range/go/rangecoder_go"),
98+
BinaryEntry("rust", ROOT / "algorithms/range/rust/target/release/rangecoder"),
99+
),
100+
benchmark_input="small_dictionary_like.bin",
101+
benchmark_driver=ROOT / "algorithms/range/benchmark/bench.py",
102+
),
103+
AlgorithmEntry(
104+
name="rle",
105+
extension="rle",
106+
binaries=(
107+
BinaryEntry("cpp", ROOT / "algorithms/rle/cpp/rle_cpp"),
108+
BinaryEntry("go", ROOT / "algorithms/rle/go/rle_go"),
109+
BinaryEntry("rust", ROOT / "algorithms/rle/rust/rle_rust"),
110+
),
111+
benchmark_input="repetitive_10MiB.bin",
112+
benchmark_driver=ROOT / "algorithms/rle/benchmark/bench.py",
113+
),
114+
)
115+
116+
ALGORITHM_ORDER: tuple[str, ...] = tuple(a.name for a in ALGORITHMS)
117+
LANGUAGE_ORDER: tuple[str, ...] = ("cpp",)
118+
119+
# ---------------------------------------------------------------------------
120+
# Corpus registry
121+
# ---------------------------------------------------------------------------
122+
123+
124+
@dataclass(frozen=True)
125+
class CorpusEntry:
126+
"""A test corpus file under tests/data/."""
127+
128+
name: str
129+
"""File name (e.g. 'empty.bin')."""
130+
131+
is_large: bool = False
132+
"""If True, only included when --include-large is passed to conformance."""
133+
134+
135+
CORPUS: tuple[CorpusEntry, ...] = (
136+
CorpusEntry("empty.bin"),
137+
CorpusEntry("single_byte.bin"),
138+
CorpusEntry("alternating.bin"),
139+
CorpusEntry("small_dictionary_like.bin"),
140+
CorpusEntry("random_1MiB.bin", is_large=True),
141+
CorpusEntry("random_10MiB.bin", is_large=True),
142+
CorpusEntry("repetitive_10MiB.bin", is_large=True),
143+
CorpusEntry("textlike_10MiB.bin", is_large=True),
144+
)
145+
146+
DEFAULT_CORPUS: tuple[str, ...] = tuple(c.name for c in CORPUS if not c.is_large)
147+
LARGE_CORPUS: tuple[str, ...] = tuple(c.name for c in CORPUS if c.is_large)
148+
149+
# ---------------------------------------------------------------------------
150+
# Range corpus cap
151+
#
152+
# See docs/adr/0003-range-coder-corpus-cap-policy.md for the policy decision.
153+
# Do NOT change this value without an OpenSpec change.
154+
# ---------------------------------------------------------------------------
155+
156+
RANGE_CORPUS_CAP_BYTES: int = 100 * 1024 # 100 KiB
157+
RANGE_CORPUS_CAP_REASON: str = "range_coder_corpus_cap_100_kib"
158+
159+
# ---------------------------------------------------------------------------
160+
# Benchmark report schema
161+
#
162+
# Field names used in benchmark JSON output. Changing these breaks the docs
163+
# page that consumes benchmarks.json.
164+
# ---------------------------------------------------------------------------
165+
166+
BENCHMARK_REPORT_FIELDS: tuple[str, ...] = (
167+
"algorithm",
168+
"language",
169+
"dataset",
170+
"encodeTime",
171+
"decodeTime",
172+
"encodeSpeed",
173+
"decodeSpeed",
174+
"compressionRatio",
175+
"throughput",
176+
)
177+
178+
# ---------------------------------------------------------------------------
179+
# Convenience accessors
180+
# ---------------------------------------------------------------------------
181+
182+
183+
def algorithm_by_name(name: str) -> AlgorithmEntry:
184+
"""Return the AlgorithmEntry for *name*, raising KeyError if not found."""
185+
for algo in ALGORITHMS:
186+
if algo.name == name:
187+
return algo
188+
raise KeyError(f"unknown algorithm: {name!r}")
189+
190+
191+
def all_binaries() -> list[tuple[str, str, Path]]:
192+
"""Return [(algorithm_name, language, binary_path)] for all shipped binaries."""
193+
return [
194+
(algo.name, binary.language, binary.path)
195+
for algo in ALGORITHMS
196+
for binary in algo.binaries
197+
]
198+
199+
200+
def corpus_files(include_large: bool = False) -> tuple[str, ...]:
201+
"""Return corpus file names eligible for conformance testing."""
202+
if include_large:
203+
return tuple(c.name for c in CORPUS)
204+
return DEFAULT_CORPUS

0 commit comments

Comments
 (0)