You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
perf: lift seen-map, strings.Builder in buildExplanation, drop dead RepoRoot field
- recordCommit: allocate seen map once in buildCoChangePairs, clear with
range-delete instead of make() per commit → −97% allocs at 1k commits
- buildExplanation: strings.Builder + strconv replaces fmt.Sprintf → −40%
latency, allocs halved (6→3 per site)
- Remove ScanOptions.RepoRoot (dead field — Analyzer.repoRoot used throughout)
- Update bench baselines and document results in docs/performance_log.md
### Optimization 1: lift seen-map out of `recordCommit`
23
+
24
+
`buildCoChangePairs` calls `recordCommit` once per commit. Originally each call allocated a fresh `make(map[string]bool)` for deduplication. Changed to allocate once before the loop and pass it in; `recordCommit` clears it with `for k := range seen { delete(seen, k) }`.
25
+
26
+
Effect on `CoChangePipelineSimulated` (the dominant CPU path):
27
+
28
+
| Scenario | allocs/op before | allocs/op after | Δ |
| 1k commits × 20 files (ns/op) |~5.4 ms |~4.8 ms | −11% |
34
+
35
+
The bulk of the pre-optimization allocs were 1× `make(map[string]bool)` per commit — invisible in profiling but compounding across thousands of commits in real repos.
`buildExplanation` is called once per loop call site in the structural scan. Replaced chained `fmt.Sprintf` with a pre-grown `strings.Builder` (`b.Grow(320)`) + `strconv.Quote` / `strconv.Itoa`.
40
+
41
+
| Variant | ns/op before | ns/op after | allocs/op before | allocs/op after |
42
+
|---|---|---|---|---|
43
+
| non-entrypoint | 352 | 208 | 6 |**3**|
44
+
| entrypoint | 350 | 188 | 7 |**3**|
45
+
46
+
~40–46% faster, allocs halved. The remaining 3 allocs are the `strings.Builder` internal buffer, `strconv.Quote`'s escape output, and the final `b.String()` copy — irreducible without a pre-allocated byte pool.
-`recordCommit` is O(files²) per commit — formatting sweeps and mass renames produce commits with 100+ files, which hit ~75 µs/commit and ~197 KB/call. No fix yet; caller could skip commits above a file-count threshold.
82
+
-`correlationFilter` iterates all ~N²/2 pairs in memory — fine up to ~200 files but will need chunking or a threshold-based early prune for monorepos with thousands of hot files.
83
+
-`buildExplanation`'s 3 remaining allocs are irreducible without a `sync.Pool` on the `strings.Builder` buffer. Not worth it at current call volumes.
**Notable:** 5k-file run shows 24% variance across 3 runs (5.85s–7.25s) and MB/s drops from ~8.3 to ~6.3–7.8. GC pressure from 630 MB heap allocation. Root cause: `extractIdentifiers` allocates a map + slice per line — ~4.2M allocs for a 1.5M-line scan.
124
+
125
+
### Pattern scale (`matchPII`, miss path)
126
+
127
+
| Patterns | ns/op |
128
+
|---|---|
129
+
|~80 (default) | 1,174 |
130
+
| 100 | 1,386 |
131
+
| 200 | 2,355 |
132
+
| 500 | 5,238 |
133
+
134
+
Linear degradation with custom pattern count. 80→500 patterns = ~4.5× slower on the miss path. Relevant for users with large custom PII configs on big repos.
135
+
136
+
### Known bottlenecks
137
+
138
+
-`extractIdentifiers`: allocates `map[string]bool` + `[]string` per source line. At 5k files × 300 lines, this is ~4.2M allocs per audit run. Pooling the map would be the highest-leverage fix.
139
+
- GC pressure at repo scale: 630 MB allocated for a 1.5M-line scan. MB/s degrades ~10% at this scale due to GC pauses.
140
+
- Custom PII patterns scale linearly on the miss path — no trie or bloom filter in front of the suffix scan.
0 commit comments