Skip to content

Commit 3272583

Browse files
gHashTagPerplexity Computer
gHashTag
and
Perplexity Computer
authored
feat(W7.3): grammar-directed fuzz baseline (E1 generator) (#46)
* feat(W7.3-E1): grammar-directed T27 fuzz generator (YARPGen-style) Заменяет W6.1 lexical fuzzer (tautological на shared front-end) на grammar-directed generator, эмитящий синтаксически валидные T27 модули. Coverage (initial subset): Module ::= UseDecl* ConstDecl* FnDecl+ FnDecl ::= fn ident (params?) -> Type { LetStmt* Return } Expr ::= Literal | Ident | BinOp | Cast | Cmp Types ::= u8 | u16 | u32 | u64 | usize | bool - Depth-bounded (max 6), seed-reproducible через ChaCha20Rng. - Standalone Cargo workspace в tests/fuzz/grammar_v2/ — не тянет зависимости в trios-mesh crate. - Smoke: 5 модулей на seed 0xC0FFEE — грамматически валидны, binops с correct type-matching, shifts clamped к <8u32. Caveat: E1 только генерирует. Round-trip harness (E2) и backend differential (E3, blocked на upstream Stmt::Let fix из t27#1401) — следующие коммиты в этой же ветке. Success criterion (post-E2): 100% parse, >=95% roundtrip-eq, 0 panics. См. docs/W7_3_FUZZ_BASELINE_PLAN.md. Refs: t27#1401 (E3 dependency), PR #44 (W7.1 root cause) phi^2 + phi^-2 = 3 * feat(W7.3-E2): parse-invariance harness + N=1000 baseline (100/100/0) Baseline result: 1000/1000 parse_ok, 1000/1000 whitespace-invariant, 0 panics, 9.9 sec. Все три success criteria \u0438\u0437 plan'a \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u044b. ## \u0427\u0442\u043e \u0432 \u044d\u0442\u043e\u043c \u043a\u043e\u043c\u043c\u0438\u0442\u0435 - tests/fuzz/grammar_v2/roundtrip.py: E2 harness. \u0422\u0440\u0438 invariant\u0430: 1. parse-success (t27c parse exit 0), 2. determinism (\u0434\u0432\u0430\u0436\u0434\u044b parse \u2192 identical AST), 3. whitespace-invariance (extra_spaces/newlines/trailing_ws mutations \u2192 identical normalized AST). Normalization strips 'line: N,' fields (source-position metadata), collapses whitespace. - tests/fuzz/grammar_v2/src/gen.rs: max_stmts_per_fn 8 \u2192 20. \u0421\u0432\u043e\u0434\u0438\u0442 doc-code drift (plan \u0433\u043e\u0432\u043e\u0440\u0438\u043b 20, code \u0431\u044b\u043b 8), \u0432\u044b\u044f\u0432\u043b\u0435\u043d\u043d\u044b\u0439 GLM peer-review @ 6c0c93d. - docs/W7_3_FUZZ_BASELINE.md: \u043e\u0442\u0447\u0451\u0442 N=1000 run \u0441 \u043c\u0435\u0442\u0440\u0438\u043a\u0430\u043c\u0438, \u0447\u0435\u0441\u0442\u043d\u044b\u043c \u043e\u043f\u0438\u0441\u0430\u043d\u0438\u0435\u043c \u043d\u043e\u0440\u043c\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438-wrinkle (\u043f\u0435\u0440\u0432\u044b\u0439 smoke \u0434\u0430\u043b 75% invariance-fail \u0438\u0437-\u0437\u0430 line-tracking; fix \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0438\u0440\u043e\u0432\u0430\u043d) \u0438 full reproducibility recipe. - docs/W7_3_FUZZ_BASELINE_PLAN.md: E2 status \u2192 LANDED, pretty-printer \u043a\u043e\u043d\u0441\u0442\u0440\u0435\u0439\u043d\u0442 \u043e\u0431\u044a\u044f\u0441\u043d\u0451\u043d (t27c \u043d\u0435 \u044d\u043a\u0441\u043f\u043e\u0437\u0438\u0442 public pretty- printer), + Tracked TODOs before E3 (params, Call, Index, If, dead-let reduction) \u0441 backstop 2026-07-19. ## \u041e\u0442\u0432\u0435\u0442 \u043d\u0430 GLM peer-review - \u2713 max_stmts drift \u0441\u0432\u0435\u0434\u0451\u043d (8 \u2192 20 \u0432 code). - \u2713 Coverage-gap zero-param \u0437\u0430\u0444\u0438\u043a\u0441\u0438\u0440\u043e\u0432\u0430\u043d \u043a\u0430\u043a Tracked TODO \u0432 plan. - \u2713 Dead-let \u043e\u0431\u0441\u0435\u0440\u0432\u0430\u0446\u0438\u044f \u2192 Tracked TODO. ## Discipline note E1 review @ 6c0c93d + E2 review \u2014 minimum reviewable unit \u043f\u043e GLM observation ('E1 without E2 \u043d\u0435\u0432\u0430\u043b\u0438\u0434\u0438\u0440\u0443\u0435\u043c'). \u0422\u0435\u043f\u0435\u0440\u044c \u0432\u043c\u0435\u0441\u0442\u0435 \u0433\u043e\u0442\u043e\u0432\u044b \u043a peer-review \u043a\u0430\u043a coherent \u0435\u0434\u0438\u043d\u0438\u0446\u0430. phi^2 + phi^-2 = 3 --------- Co-authored-by: Perplexity Computer <agent@perplexity.ai>
1 parent be06148 commit 3272583

6 files changed

Lines changed: 610 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
11
/target
22
**/*.rs.bk
33
*.pdb
4+
5+
# W7.3 fuzz workspace — build artifacts and generated modules stay local
6+
tests/fuzz/grammar_v2/target/
7+
tests/fuzz/grammar_v2/Cargo.lock
8+
tests/fuzz/grammar_v2/out/
9+

‎docs/W7_3_FUZZ_BASELINE.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# W7.3 E1+E2 baseline — parse-invariance run
2+
3+
Status: **BASELINE LANDED** (2026-07-05).
4+
Branch: `w7/testing/fuzz-baseline` (PR #46).
5+
Provenance: `tests/fuzz/grammar_v2/{Cargo.toml,src/gen.rs,roundtrip.py}` @ this commit.
6+
7+
## Setup
8+
9+
- **Generator**: `tests/fuzz/grammar_v2/src/gen.rs` (E1, W7.3-E1 commit + `max_stmts_per_fn` reconciliation).
10+
- **Harness**: `tests/fuzz/grammar_v2/roundtrip.py` (E2).
11+
- **t27c**: `/home/user/workspace/t27/target/release/t27c` (from t27 workspace, master post-#1348 era).
12+
- **Seed**: `0xC0FFEE` base + per-module offset `+i`.
13+
- **Corpus**: N=1000 modules generated to `/tmp/w73_baseline_1000/*.t27` (spec bodies not committed — reproducible from seed).
14+
15+
## Invariants tested
16+
17+
For each generated module, the harness runs three passes:
18+
19+
1. **Parse-success**: `t27c parse <path>` returns exit code 0 (no error, no panic).
20+
2. **Determinism**: parsing the same input twice yields identical AST (after normalization).
21+
3. **Whitespace-invariance**: three non-semantic mutations are applied and re-parsed:
22+
- `extra_spaces` — double every leading-indent space run.
23+
- `extra_newlines` — add blank line after every `}\n`.
24+
- `trailing_ws` — add trailing spaces to every non-empty line.
25+
Each mutated variant must parse to the same normalized AST as the original.
26+
27+
## Normalization
28+
29+
The harness strips `line: N,` fields from `t27c parse`'s Debug-formatted AST before comparison, because these are source-position metadata derived from layout, not structural content. Extra newlines shift them without changing meaning. Any remaining structural change after stripping is treated as a real invariance violation.
30+
31+
## Results (N=1000, seed range `0xC0FFEE..0xC0FFEE+999`)
32+
33+
| Metric | Value |
34+
|---|---|
35+
| Parse-success | **1000 / 1000 (100.0%)** |
36+
| Determinism | **1000 / 1000 (100.0%)** |
37+
| Whitespace-invariance | **1000 / 1000 (100.0%)** |
38+
| Parse errors | 0 |
39+
| Panics | 0 |
40+
| Non-determinism | 0 |
41+
| Elapsed | 9.9 sec (3000+ subprocess calls: baseline + determinism + 3 mutations per input) |
42+
43+
All three success criteria from `W7_3_FUZZ_BASELINE_PLAN.md` §Success-criterion met:
44+
45+
- ✓ 100% grammar-valid generations parse cleanly (target: 100%).
46+
- ✓ 100% whitespace-invariant (target: ≥95%).
47+
- ✓ 0 panics in t27c parser.
48+
49+
## Interpretation
50+
51+
**What this claim IS**: for the grammar subset E1 currently covers (Module, FnDecl with zero params, Let / Return stmts, Expr = Literal / Ident / BinOp / Cast / Cmp, six primitive types), t27c parses cleanly, deterministically, and is invariant to non-semantic whitespace changes across 1000 random seeds.
52+
53+
**What this claim IS NOT**:
54+
- Not a differential test — this is parser self-consistency only. Real backend-differential (E3) needs the upstream Stmt::Let fix from [t27#1401](https://github.com/gHashTag/t27/issues/1401) to land first.
55+
- Not full-grammar coverage. Missing from E1 today: function parameters, `Call`, `Index`, `If` statements, `UseDecl`, `ConstDecl`. See "Tracked TODOs before E3" below.
56+
- Not a full round-trip via pretty-printer. t27c doesn't expose a public pretty-printer in the current version. Parse-invariance is a strict subset of the intended round-trip and still catches parser non-determinism, whitespace-sensitivity, and panics. Full round-trip via pretty-printer is a TODO once t27c exposes one.
57+
58+
## Discipline honesty
59+
60+
One methodological wrinkle was caught during the smoke run (N=20 before N=1000):
61+
62+
The first version of `normalize_ast` collapsed whitespace but left `line: N` fields intact. This produced a 75% failure rate on the `extra_newlines` mutation, because adding blank lines shifts line numbers for every subsequent AST node. The failure was NOT a parser bug — it was over-strict normalization treating source-position metadata as structural content. Fix: strip `line: N,` fields before comparison. After the fix, N=20 smoke passed 100%, and the N=1000 full run followed. This wrinkle is documented so the normalization choice is auditable and the "100% invariance" claim is understood as "invariant modulo source-position metadata," not "byte-for-byte identical output."
63+
64+
## Tracked TODOs before E3 unblock
65+
66+
The E1 grammar subset does not exercise function parameters. The W6.2 audit found the `Vec<>` defect (E0107, Class 2) lives in param-position. E3's differential power depends on exercising the grammar regions where bugs hide. Therefore:
67+
68+
- **Before E3**: extend E1 to generate function parameters (including collection params like `Vec<u8>`) and `Call` / `Index` expressions.
69+
- **Backstop timer for E3**: 2026-07-19 12:24 UTC (14 days from t27#1401 publication), or terminal event on t27#1401 (won't-fix / closing PR / explicit reject), whichever comes first. See `W7_COLLAB_OPTIONS.md` §external-dep-timer rule.
70+
71+
Between now and that deadline, E1 grammar expansion is the primary open workstream on this branch.
72+
73+
## Reproducibility
74+
75+
```bash
76+
# From tri-net workspace root.
77+
cd tests/fuzz/grammar_v2
78+
cargo build --release
79+
W73_OUT=/tmp/w73_baseline_1000 ./target/release/gen 1000 0xC0FFEE
80+
cd ../../..
81+
python3 tests/fuzz/grammar_v2/roundtrip.py /tmp/w73_baseline_1000 --out /tmp/w73_baseline_1000_report.json
82+
```
83+
84+
Expected: `ok=1000 parse_err=0 mut_fail=0 non_det=0`, elapsed <15 sec on a modern x86_64 sandbox.
85+
86+
## Anchor
87+
88+
phi^2 + phi^-2 = 3

‎docs/W7_3_FUZZ_BASELINE_PLAN.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# W7.3 — Grammar-directed fuzz baseline
2+
3+
Status: **PLAN** (2026-07-05) — awaiting first generator commit.
4+
Branch: `w7/testing/fuzz-baseline`.
5+
Parent: W6.1 lexical fuzzer (`tests/fuzz/` — token-level, tautological на shared front-end).
6+
7+
## Задача
8+
9+
Заменить lexical fuzzer (W6.1, weak-point 1.5 из `W6_WEAK_POINTS_AND_W7_PLAN.md`) на **grammar-directed generator** в стиле YARPGen. W6.1 генерировал token streams и проверял, что parser не panic'ает — 100% agreement был тавтологией, потому что все три backend'а (Rust / C / Zig) шарят один front-end. Реальная differential мощь возможна только после того, как:
10+
11+
1. Генератор эмитит **валидные по grammar** программы (не token noise).
12+
2. Round-trip harness проверяет структурную инвариантность parser'а.
13+
3. Backend-differential применяется на well-typed inputs, где расхождение = семантический bug, а не parse-noise.
14+
15+
## Scope W7.3
16+
17+
Три этапа:
18+
19+
### E1 — Grammar-directed generator (первый коммит)
20+
21+
- `tests/fuzz/grammar_v2/generator.rs` — production rules с weighting.
22+
- Grammar покрытие minimum:
23+
- `Module { UseDecl* ConstDecl* FnDecl+ }`
24+
- `FnDecl { name, params, ret_type, body }`
25+
- `Stmt ::= Let | Return | If | ExprStmt`
26+
- `Expr ::= Literal | Ident | BinOp | Cast | Call | Index`
27+
- Types: `u8 | u16 | u32 | u64 | usize | bool`
28+
- Depth-bounded generation: max depth 6, max stmt count per fn 20.
29+
- Seed-reproducible через `StdRng::seed_from_u64`.
30+
- Output: N=1000 генераций в `target/fuzz/w7_3/*.t27`.
31+
32+
### E2 — Parse-invariance harness (второй коммит, **LANDED**)
33+
34+
Оригинальный план требовал full round-trip через pretty-printer, но t27c в current release не экспозит public pretty-printer. Parse-invariance — strict subset intended round-trip и ловит тот же класс багов (parser non-determinism, whitespace-sensitivity, panics):
35+
36+
- `tests/fuzz/grammar_v2/roundtrip.py`:
37+
1. Gen spec → `t27c parse` → baseline AST (Debug-format).
38+
2. Determinism: parse тот же input второй раз → identical normalized AST.
39+
3. Whitespace-invariance: 3 non-semantic мутации (extra_spaces / extra_newlines / trailing_ws) → parse → identical normalized AST.
40+
- Normalization strips `line: N,` метаданные (source-position, не structural). Остальное collapse whitespace.
41+
- Metric: `parse_ok_rate`, `invariance_ok_rate`, failure classes (parse_error / non_determinism / mutation_changed_ast:<mode>).
42+
- Full round-trip через pretty-printer — TODO когда t27c экспозит pretty-printer.
43+
44+
**Baseline result**: N=1000, 100.0% parse_ok / 100.0% invariance_ok / 0 panics / 9.9 sec. См. `docs/W7_3_FUZZ_BASELINE.md`.
45+
46+
### E3 — Backend differential (третий коммит, зависит от W7.1 upstream fix)
47+
48+
- Same input → `t27c gen rust|c|zig` → compile → runtime output.
49+
- Differential trigger: any two backends диверджируют на same seed.
50+
- **Caveat**: E3 валиден только когда upstream Stmt::Let fix у t27c приземлится. До этого gen/rust не имеет `let`, дифференциал структурно infeasible (см. W6.2 audit §3-5).
51+
- Пока E3 blocked, E1+E2 работают независимо на current tree.
52+
53+
## Baseline run
54+
55+
Первый full run после E1+E2:
56+
- N=1000 генераций.
57+
- Distribution по (depth × stmt-count) — гистограмма в `docs/W7_3_FUZZ_BASELINE.md` (post-E2 doc, отдельный PR).
58+
- Grammar coverage: доля production rules, exercised хотя бы одной генерацией.
59+
60+
## Success criterion
61+
62+
E1+E2 baseline считается **зелёным**, если:
63+
- 100% валидных по grammar генераций parse'ятся без ошибок.
64+
- ≥95% round-trip'ов структурно equal (allowed slack — pretty-printer whitespace normalization).
65+
- 0 panics в t27c parser.
66+
67+
Любое отклонение — bug в parser или в pretty-printer, файлится как t27c issue (не tri-net) с seed'ом воспроизведения.
68+
69+
## Caveats и honest scope
70+
71+
- **Codegen-only vs parser-side ambiguity**: baseline валиден пока upstream Stmt::Let fix остаётся codegen-only. Если maintainer t27 определит проблему как parser-side (маловероятно — spec содержит `let`, значит parser их видит), baseline придётся пересобрать: parser может уже сейчас терять information, которую мы предположительно проверяем round-trip'ом.
72+
- **Not a differential test** до E3. E1+E2 только проверяют parser self-consistency. Реальная differential мощь — E3, blocked на upstream.
73+
- **Grammar в этом плане — approximate**. Ground truth grammar сидит в `t27c/src/parser.rs` upstream. Первый E1 коммит перекроет subset, но не 100% grammar; расширение — итеративно.
74+
75+
## Tracked TODOs before E3 unblock
76+
77+
GLM-5.2 peer-review PR #46 @ 6c0c93d выявил coverage-gap: текущий E1 эмитит zero-param functions. Но W6.2 audit нашёл Vec<>-defect (E0107 Class 2) в **param-position** — E3 backend-differential будет слепым к этому дефекту, если grammar не расширить. Backstop таймер t27#1401 = 2026-07-19 12:24 UTC (14 дней). За это окно:
78+
79+
- [ ] Extend `gen_fn` to emit **function parameters** (от 0 до 4, mixed primitives + хотя бы один collection type).
80+
- [ ] Add `Call` expression to `gen_expr` с recursion на другие генерируемые функции.
81+
- [ ] Add `Index` expression если grammar поддерживает (верифицировать через `parse` на minimal specs).
82+
- [ ] Add `If` statement branching (grammar уже в plan’e, но в code нет).
83+
- [ ] Reduce dead-let частоту (вес ident-branch в `gen_expr` → 40%+).
84+
85+
Статус обновлять в этом файле по мере выполнения.
86+
87+
## Отношение к W6.1
88+
89+
W6.1 lexical fuzzer НЕ deprecated — token-level fuzzing ловит другой класс bugs (parser panic на malformed input). W7.3 grammar-directed — комплементарен, не замена. Оба живут в `tests/fuzz/`, разными namespace'ами.
90+
91+
## Anchor
92+
93+
phi^2 + phi^-2 = 3

‎tests/fuzz/grammar_v2/Cargo.toml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
[package]
2+
name = "w7_3_grammar_fuzz"
3+
version = "0.1.0"
4+
edition = "2021"
5+
license = "Apache-2.0"
6+
publish = false
7+
8+
[dependencies]
9+
rand = "0.8"
10+
rand_chacha = "0.3"
11+
12+
[[bin]]
13+
name = "gen"
14+
path = "src/gen.rs"
15+
16+
[profile.release]
17+
opt-level = 3
18+
lto = false

0 commit comments

Comments
 (0)