perf(scan): dual-path walker — parallel bulk, serial early-exit (#821)

## Summary

Adds a parallel async-`readdir` walker alongside the pre-existing serial
walker, selected per-walk by a new `concurrency` option. Bulk scans (DSN
detection, sourcemap discovery, uncapped grep) speed up by 22-37%;
early-exit paths (DSN `scanCodeForFirstDsn`) are unchanged.

## Why

Profiling against ripgrep on a 10k-file synthetic fixture revealed our
walker was 5× slower than a raw sync walk. The dominant cost: serial
async `readdir` (109ms for 3636 dirs vs 44ms sync) because each call
hops through a microtask. Overlapping `readdir` across directories
recovers the loss — 4 workers hit 28ms for the same traversal.

But we can't just make everything parallel. Two early-exit paths
(`scanCodeForFirstDsn`, `detectDsn.cold`) need to stop on the first hit.
The parallel walker's producer-consumer channel adds ~7 ms per emitted
file — fine for bulk scans, fatal for "find one and bail" consumers
where the serial walker is 2ms total. Benchmarking showed those paths
regressed 30-50× under a naïve uniform-parallel walker.

The fix is a dual-path dispatch based on `concurrency`: 1 → serial, > 1
→ parallel. The default is `bulkConcurrency()` (= `max(2,
availableParallelism())`). Early-exit callers pass `concurrency: 1`
explicitly.

## Design

`walkFiles` dispatches to one of two internal generators that share
per-entry helpers (`processEntry`, `maybeDescend`, `tryYieldFile`,
`classifyFile`):

- **`walkSerial`** — byte-for-byte the pre-refactor body: LIFO stack,
direct `yield`, sync `readdirSync`. Per-dir overhead ~8 µs. Optimal for
early-exit.
- **`walkParallel`** — N workers pull from the shared stack. Idle
workers park on a `workerAwake` Promise that uses **swap-then-resolve**
(new Promise installed before resolving the old) so late awaiters can't
latch onto a stale resolution. `signalWorkers()` fires on every descent
(`pushFrame`), last-active-worker-exit-with-empty-stack, and
cancellation. File entries flow through a `pending` buffer drained by
the generator.

A worker-side outer `try/catch` stores errors (including `AbortError`
from pre-fired signals) into `producerError` for the consumer to
re-throw. This fixes a hang where a pre-fired `AbortSignal` would park
the consumer on `consumerAwake` forever because `activeWorkers` was
never incremented.

## Caller updates

| Caller | Path | Concurrency |
|---|---|---|
| `scanCodeForDsns` | bulk scan | default (parallel) |
| `scanCodeForFirstDsn` | early-exit | explicit `1` (serial) |
| `collectGrep` (init wizard, DSN scan) | bulk | default (parallel) |
| `discoverFilePairs` (sourcemap inject) | bulk | default (parallel) |

## Tests

5 new parallel-walker tests in `test/lib/scan/walker.test.ts`:

- `yields the same set of files as the serial walker` — correctness
equivalence
- `honors gitignore rules across concurrent descents` — nested
`.gitignore` loads atomically per-dir
- `honors descentHook under concurrent traversal` — monorepo depth-reset
works with workers
- `aborts mid-walk when signal fires` — mid-walk abort propagates
- **`propagates a pre-fired abort through the parallel walker`** —
regression for the Blocker fix

All 36 walker tests + 232 scan tests + 5703 full-suite tests + 138
isolated tests pass.

## Performance (synthetic/large, 10k files, linux/x64, Bun 1.3.11)

| op | before | after | Δ |
|---|---:|---:|---:|
| `scanCodeForDsns` | 238ms | 181ms | **−24%** |
| `detectAllDsns.cold` | 253ms | 198ms | **−22%** |
| `scan.walk.noExt` | 451ms | 286ms | **−37%** |
| `scan.walk.dsnParity` | 146ms | 132ms | −10% |
| `scan.grepFiles` | 169ms | 163ms | −4% |
| `detectDsn.cold` | 5.57ms | 4.63ms | −17% |
| `scanCodeForFirstDsn` | 2.28ms | 1.92ms | −16% |

All other ops within ±10% of baseline.

## vs ripgrep

rg is still faster on bulk patterns (it uses 4 native threads with
work-stealing), but the gap narrowed considerably. Our uncapped
`NONEXISTENT_TOKEN_XYZ` went from 6.9× rg to ~4.5×. For capped grep with
`maxResults: 100` (init wizard's default) we're actually faster than rg
on dense patterns due to early-exit. This is probably the last major
architectural win available without native code.

## Review cycle

Two rounds of subagent review before opening the PR:
- **Round 1** (ses_249398303ffe0wYvzoF5RhiSBh): 4 Important findings + 4
Minor. All addressed.
- **Round 2** (ses_2491fea0cffeYtcfVYQRtKX9uk): 1 Blocker (pre-fired
abort hang), 5 Minor. All fixed; pre-fired abort now has a dedicated
regression test + negative verification.

## Test plan

- [x] `bunx tsc --noEmit` — clean
- [x] `bun run lint` — clean (1 pre-existing markdown.ts warning)
- [x] `bun test test/lib test/commands test/types` — 5703 pass, 0 fail
- [x] `bun test test/isolated` — 138 pass
- [x] `bun run bench --size large` — all ops equal-or-better
- [x] Negative-verified pre-fired abort test catches the bug it guards
against

Author: BYK

src/lib/dsn/code-scanner.ts

@@ -156,6 +156,12 @@ export function scanCodeForFirstDsn(cwd: string): Promise<DetectedDsn | null> {
         for await (const entry of walkFiles({
           cwd,
           ...dsnScanOptions(),
+          // Early-exit consumer: we `return` on the first DSN-bearing
+          // file. The parallel walker's channel adds ~7ms per-file
+          // overhead; the serial walker uses a direct `yield` so
+          // `break` cuts immediately. Measured 2ms (serial) vs ~75ms
+          // (parallel) on the large bench fixture.
+          concurrency: 1,
         })) {
           filesScanned += 1;
           let content: string;

src/lib/scan/index.ts

@@ -70,4 +70,4 @@ export type {
   WalkEntry,
   WalkOptions,
 } from "./types.js";
-export { walkFiles } from "./walker.js";
+export { bulkConcurrency, walkFiles } from "./walker.js";

src/lib/scan/types.ts

@@ -137,6 +137,25 @@ export type WalkOptions = {
    */
   signal?: AbortSignal;
 
+  // --- Parallelism ---
+  /**
+   * Max number of directories to `readdir` in parallel. Default:
+   * `bulkConcurrency()` (= `max(2, availableParallelism())`), chosen
+   * to overlap async `readdir` I/O across directories — the primary
+   * speedup (~25%) for bulk scans like DSN detection or sourcemap
+   * discovery.
+   *
+   * Set to `1` for **early-exit** traversal, which uses a direct
+   * `yield` serial path (no producer-consumer channel) and sync
+   * `readdirSync` — measurably faster when the consumer `break`s
+   * after the first few files (e.g. `scanCodeForFirstDsn`).
+   *
+   * This only controls directory-level concurrency. File content
+   * reads (done by callers like `grepFiles`) have their own
+   * concurrency knob.
+   */
+  concurrency?: number;
+
   // --- Time budget ---
   /**
    * Wall-clock budget in milliseconds. Every directory at depth