|
| 1 | +--- |
| 2 | +phase: 03-evaluation-guardrails |
| 3 | +plan: 02 |
| 4 | +subsystem: testing |
| 5 | +tags: [tree-sitter, vitest, unicode, parser-reset, indexing] |
| 6 | +requires: |
| 7 | + - phase: 02-tree-sitter |
| 8 | + provides: parse-error fallback and symbol-aware chunking baseline |
| 9 | +provides: |
| 10 | + - Unicode slicing regression coverage for symbol extraction |
| 11 | + - Large file and generated file skip regression coverage |
| 12 | + - Parser timeout/reset and cleanup regression coverage |
| 13 | +affects: [phase-05-ast-aligned-chunking, phase-09-search-quality] |
| 14 | +tech-stack: |
| 15 | + added: [] |
| 16 | + patterns: [utf8-safe extraction guardrails, parser cache eviction on failure] |
| 17 | +key-files: |
| 18 | + created: |
| 19 | + - tests/tree-sitter-unicode-slicing.test.ts |
| 20 | + - tests/tree-sitter-cleanup.test.ts |
| 21 | + - tests/indexer-large-file-skip.test.ts |
| 22 | + modified: |
| 23 | + - src/utils/tree-sitter.ts |
| 24 | +key-decisions: |
| 25 | + - "Set Tree-sitter timeout via setTimeoutMicros when available, but fail open if timeout API signature differs across runtimes." |
| 26 | + - "Include export_statement parent range for symbol content so exported declarations remain complete in extracted chunks." |
| 27 | +patterns-established: |
| 28 | + - "Guardrail tests use local grammars/mocks only and remain network-free." |
| 29 | + - "Parser failures evict cache entries to avoid poisoning subsequent parses." |
| 30 | +requirements-completed: [EVAL-02] |
| 31 | +duration: 4 min |
| 32 | +completed: 2026-02-20 |
| 33 | +--- |
| 34 | + |
| 35 | +# Phase 03 Plan 02: Regression Guardrails Summary |
| 36 | + |
| 37 | +**Tree-sitter extraction now has Unicode-safe guards with parser reset/cleanup hardening, backed by focused regressions for Unicode boundaries, parse failure recovery, and large-file skipping.** |
| 38 | + |
| 39 | +## Performance |
| 40 | + |
| 41 | +- **Duration:** 4 min |
| 42 | +- **Started:** 2026-02-20T18:31:16Z |
| 43 | +- **Completed:** 2026-02-20T18:36:04Z |
| 44 | +- **Tasks:** 2 |
| 45 | +- **Files modified:** 4 |
| 46 | + |
| 47 | +## Accomplishments |
| 48 | +- Hardened `extractTreeSitterSymbols` with UTF-8 extraction path, 1 MiB parse budget guard, timeout wiring, and parser cache eviction/reset on failures. |
| 49 | +- Added regression tests that lock Unicode slicing behavior, parser cleanup/reset behavior, and large/generated file skip behavior. |
| 50 | +- Verified all new regressions pass with network-free Vitest execution. |
| 51 | + |
| 52 | +## Task Commits |
| 53 | + |
| 54 | +Each task was committed atomically: |
| 55 | + |
| 56 | +1. **Task 1: Harden Tree-sitter extraction for Unicode + reset behavior** - `375a48f` (fix) |
| 57 | +2. **Task 2: Add regression tests for Unicode slicing, large file skipping, and cleanup/reset** - `a1c71de` (fix) |
| 58 | + |
| 59 | +**Plan metadata:** `58adead` |
| 60 | + |
| 61 | +## Files Created/Modified |
| 62 | +- `src/utils/tree-sitter.ts` - Adds parse guards, timeout wiring, parser cache eviction/reset, and export-range-aware symbol extraction. |
| 63 | +- `tests/tree-sitter-unicode-slicing.test.ts` - Reproduces and protects against Unicode boundary symbol slicing corruption. |
| 64 | +- `tests/tree-sitter-cleanup.test.ts` - Mocks parser behavior to verify `tree.delete()` cleanup and parser reset/eviction paths. |
| 65 | +- `tests/indexer-large-file-skip.test.ts` - Validates oversize regular and generated files are skipped without breaking indexing. |
| 66 | + |
| 67 | +## Decisions Made |
| 68 | +- Used a best-effort timeout hook: call `setTimeoutMicros` when present, but do not fail extraction if timeout API wiring differs across parser builds. |
| 69 | +- Expanded symbol content range to include `export_statement` parent nodes, preserving complete exported declaration headers. |
| 70 | + |
| 71 | +## Deviations from Plan |
| 72 | + |
| 73 | +### Auto-fixed Issues |
| 74 | + |
| 75 | +**1. [Rule 1 - Bug] Fixed exported declaration truncation under Unicode boundary conditions** |
| 76 | +- **Found during:** Task 2 (Unicode regression execution) |
| 77 | +- **Issue:** Byte-slice extraction produced incomplete exported declaration headers in a Unicode-preceded file. |
| 78 | +- **Fix:** Added compatibility extraction fallback and export parent-range capture in `buildSymbol`. |
| 79 | +- **Files modified:** src/utils/tree-sitter.ts |
| 80 | +- **Verification:** `pnpm vitest run tests/tree-sitter-unicode-slicing.test.ts` |
| 81 | +- **Committed in:** a1c71de |
| 82 | + |
| 83 | +--- |
| 84 | + |
| 85 | +**Total deviations:** 1 auto-fixed (1 bug) |
| 86 | +**Impact on plan:** Deviation was required to satisfy the guardrail objective and keep extraction behavior correct across runtime offset semantics. |
| 87 | + |
| 88 | +## Issues Encountered |
| 89 | +None. |
| 90 | + |
| 91 | +## User Setup Required |
| 92 | +None - no external service configuration required. |
| 93 | + |
| 94 | +## Next Phase Readiness |
| 95 | +Regression coverage for EVAL-02 failure surfaces is in place and passing. |
| 96 | +Ready for 03-03 shared eval harness and multi-codebase `npm run eval` wiring. |
| 97 | + |
| 98 | +--- |
| 99 | +*Phase: 03-evaluation-guardrails* |
| 100 | +*Completed: 2026-02-20* |
| 101 | + |
| 102 | +## Self-Check: PASSED |
0 commit comments