|
1 | 1 | # Changelog |
2 | 2 |
|
| 3 | +## [1.33.0.0] - 2026-05-11 |
| 4 | + |
| 5 | +## **`/sync-gbrain` memory stage no longer infinite-loops or silently throws away progress.** |
| 6 | +## **Per-file gitleaks scanning is opt-in, signal handling actually kills the gbrain child, and state writes are atomic.** |
| 7 | + |
| 8 | +`/sync-gbrain` memory ingest used to spawn `gitleaks detect` plus `gbrain put` once per file across 1,841+ transcripts and artifacts, then the orchestrator SIGTERM'd the whole pipeline at 35 minutes with no state flush. Every cold run started from zero and burned 35 minutes for nothing. v1.33 rewrites the memory stage around `gbrain import <dir>` (batch path that's been in gbrain since v0.20). The prepare phase walks sources, parses transcripts and artifacts, writes prepared markdown into a hierarchical staging directory mirroring slug structure, then invokes `gbrain import` once. Per-file failures get read back from `~/.gbrain/sync-failures.jsonl` via a byte-offset snapshot so the state file only records files that actually landed in PGLite. `--scan-secrets` is now an opt-in flag because `gstack-brain-sync` already runs a regex-based secret scanner at the actual cross-machine boundary (git push), making per-file ingest scans redundant defense-in-depth that cost ~470 seconds on every cold run. |
| 9 | + |
| 10 | +The signal handler now propagates `SIGTERM` and `SIGINT` to the gbrain child and synchronously cleans up the staging directory before `process.exit`, fixing the orphan-process bug that left gbrain holding the PGLite write lock and burning CPU for hours after the orchestrator gave up. State file writes use `tmp+rename` for atomicity so a crash mid-write can't truncate the ingest state. The full-file `sha256` change detection (was capped at 1MB) catches tail edits to long partial transcripts that the old algorithm silently missed. |
| 11 | + |
| 12 | +### The numbers that matter |
| 13 | + |
| 14 | +Source: live run on `~/.gstack/projects/` corpus (5,135 transcripts + artifacts), `bin/gstack-memory-ingest.ts --bulk` on a fresh PGLite at gbrain v0.31.2. |
| 15 | + |
| 16 | +| Metric | Before (v1.31.x) | After (v1.33) | Δ | |
| 17 | +|---|---|---|---| |
| 18 | +| Cold run completes | no, 35-min loop + null exit | yes | works | |
| 19 | +| Prepare phase time (5,135 files) | ~10-12 min | <10 sec | ~60x | |
| 20 | +| Per-file gitleaks scans | 1,841 mandatory | 0 by default, opt-in via `--scan-secrets` | gated | |
| 21 | +| State file flushed on SIGTERM | no, loss-on-kill | yes, sync cleanup before exit | fixed | |
| 22 | +| Orphan gbrain child after timeout | yes, observed 15hr CPU drain | no, signal forwarded | fixed | |
| 23 | +| FILE_TOO_LARGE blocks all advancement | yes | no, failed paths excluded via D7 | fixed | |
| 24 | +| Tests in `test/gstack-memory-ingest.test.ts` | 17 | 21 | +4 | |
| 25 | + |
| 26 | +| Decision | What landed | |
| 27 | +|---|---| |
| 28 | +| D1 hierarchical staging | `writeStaged` does `mkdir -p` per slug segment | |
| 29 | +| D2 cut over | `gbrainPutPage` deleted, no `--legacy-ingest` flag | |
| 30 | +| D3 source-first secret scan | Scan opt-in via `--scan-secrets`, default off | |
| 31 | +| D4 OK/ERR verdict | Per-file failures show in summary but only system errors mark ERR | |
| 32 | +| D5 unified state schema | No separate skip-list file | |
| 33 | +| D6 trust idempotency | gbrain's content_hash dedup makes reruns cheap | |
| 34 | +| D7 sync-failures byte-offset | `readNewFailures` reads only appended bytes since pre-import snapshot | |
| 35 | +| F6 atomic state writes | `tmp+rename` instead of direct overwrite | |
| 36 | +| F9 full-file sha256 | Removes 1MB cap that silently swallowed tail edits | |
| 37 | + |
| 38 | +Prepare phase dropped from ~10 minutes to <10 seconds because the dominant cost was `gitleaks detect` cold start (~256ms per file, 5,135 files = 22 minutes of subprocess startup). The cross-machine secret boundary is `git push`, and `gstack-brain-sync` already runs its own regex scanner there. Local PGLite ingest of files that already live on disk in plaintext doesn't change exposure. The opt-in flag survives for users who want per-file ingest scanning, but it's no longer the default tax on every cold run. |
| 39 | + |
| 40 | +### What this means for builders |
| 41 | + |
| 42 | +If you've been hitting the 35-minute hang on `/sync-gbrain`, it's gone. The architecture is correct on this side now. A separate `gbrain import` performance issue surfaced during testing where the gbrain CLI itself takes >10 minutes on 5,131-file staging dirs (10 seconds on 501 files), which is filed as a P2 TODO for gbrain proper. That's the next bottleneck to chase, but it lives in gbrain's import path, not in the gstack orchestrator. Run `/sync-gbrain` after upgrading. If you've been seeing the loop, this fixes it. |
| 43 | + |
| 44 | +### Itemized changes |
| 45 | + |
| 46 | +#### Added |
| 47 | +- `bin/gstack-memory-ingest.ts:1093` — `preparePages` pure function: walk sources, mtime-skip via state, optional gitleaks scan (`--scan-secrets`), parse transcripts and artifacts, render frontmatter with `title`/`type`/`tags` injected. |
| 48 | +- `bin/gstack-memory-ingest.ts:920` — `writeStaged` writes prepared markdown into a hierarchical staging directory mirroring slug structure. `mkdir -p` per slug segment. Slugs containing `/` (like `transcripts/claude-code/foo`) get the matching subdirectory tree so gbrain's path-authoritative `slugifyPath` round-trips exactly. |
| 49 | +- `bin/gstack-memory-ingest.ts:961` — `parseImportJson` reads gbrain's `--json` last-line payload. Returns `null` (treated as `system_error` by caller) instead of zero-padded silently when the line doesn't parse. |
| 50 | +- `bin/gstack-memory-ingest.ts:993` — `readNewFailures` snapshots `~/.gbrain/sync-failures.jsonl` byte offset before import, reads only appended bytes after, maps gbrain's staging-relative paths back to source paths via the `stagedPathToSource` map. |
| 51 | +- `bin/gstack-memory-ingest.ts:1009` — `runGbrainImport` async wrapper around `child_process.spawn` so the signal forwarder has a child reference to kill on parent `SIGTERM`/`SIGINT`. Pre-2026-05-11 `spawnSync` made signal forwarding impossible and gbrain orphaned every time the orchestrator timed out. |
| 52 | +- `bin/gstack-memory-ingest.ts:1218` — `installSignalForwarder` registers `SIGTERM`/`SIGINT` handlers that forward to the live child, synchronously clean up the active staging directory, then exit. Async `finally` blocks don't run after `process.exit` from inside a signal handler, so cleanup has to happen in the handler itself. |
| 53 | +- `bin/gstack-memory-ingest.ts:194` — `--scan-secrets` CLI flag and `GSTACK_MEMORY_INGEST_SCAN_SECRETS=1` env var to opt back into per-file gitleaks scanning during the prepare phase. Off by default. |
| 54 | +- `test/gstack-memory-ingest.test.ts:457` — 5 new tests covering hierarchical staging slug round-trip, frontmatter injection, D7 sync-failures exclusion, missing-`import`-subcommand error path, and `--scan-secrets` dirty-source skipping with a fake gitleaks shim. |
| 55 | +- `docs/designs/SYNC_GBRAIN_BATCH_INGEST.md` — full design doc with D1-D8 decisions, source-verified gbrain behaviors, performance measurements, F9 hash migration notes. |
| 56 | + |
| 57 | +#### Changed |
| 58 | +- `bin/gstack-memory-ingest.ts:288` — `saveState` now uses `tmp+rename` for atomicity (F6) so a crash mid-write can't truncate the state file. Matches the orchestrator's existing pattern at `gstack-gbrain-sync.ts:508`. |
| 59 | +- `bin/gstack-memory-ingest.ts:307` — `fileSha256` hashes the full file (F9). Pre-2026-05-11 it stopped at 1MB, so tail edits to long partial transcripts looked unchanged and never re-imported. One-time cliff on upgrade: files whose mtime hasn't moved keep their old 1MB-capped hash, files whose mtime moves get recomputed correctly. No data loss. |
| 60 | +- `bin/gstack-memory-ingest.ts:798` — `gbrainAvailable` probes for the `import` subcommand in `--help` output (was: `put` subcommand). Without `import`, the memory stage exits non-zero with a `system_error` instead of silently degrading. |
| 61 | +- `bin/gstack-gbrain-sync.ts:442` — memory-stage parser preferentially picks `[memory-ingest] ERR` lines over the latest `[memory-ingest]` line for the summary, strips the prefix, and surfaces `(killed by signal / timeout)` when the child exits with `status=null`. |
| 62 | + |
| 63 | +#### Fixed |
| 64 | +- Per-file gitleaks scan was running on every transcript and artifact during memory ingest as redundant defense-in-depth. The cross-machine secret boundary is `gstack-brain-sync` (git push), which already runs a Python regex scanner. Local PGLite ingest doesn't change exposure surface for content that already lives on disk in plaintext. |
| 65 | +- Signal handlers now kill the gbrain child and clean up the staging directory before exit. Pre-fix, every orchestrator timeout left a gbrain process holding the PGLite write lock and burning CPU until the user noticed and `kill -9`'d it manually (observed: a 15-hour-CPU-time orphan from yesterday's run was still alive today). |
| 66 | +- `parseImportJson` no longer silently returns `{imported: 0, errors: 0}` when gbrain's `--json` output doesn't parse. Returns `null`, caller surfaces as `system_error` so the orchestrator's verdict block shows ERR instead of misleading OK/0/0. |
| 67 | +- `bin/gstack-memory-ingest.ts` `require("fs")` calls replaced with top-level ESM `import`s for runtime portability. |
| 68 | + |
| 69 | +#### For contributors |
| 70 | +- Plan file at `/Users/garrytan/.claude/plans/purrfect-tumbling-quiche.md` captures the full review chain: `/investigate` → `/plan-eng-review` (5 architecture decisions D1-D5) → `/codex review` outside-voice plan challenge (9 findings, 3 reshaped the architecture into D6-D8). Plan also records the post-Codex user perf review that flipped D3 to opt-in. |
| 71 | +- `TODOS.md` filed P2: investigate `gbrain import` perf on large staging dirs (5,131 files takes >10 minutes when 501 takes 10 seconds — gbrain-side N+1 SQL or auto-link reconciliation suspected). P3: cache "no changes since last import" at the prepare-batch level for true no-op fast paths. |
| 72 | +- `Plan completion audit` ran via subagent on this branch: 17/21 DONE, 1 CHANGED (D3 made opt-in), 2 deferred (F8 benchmark harness as separate work, 24-path unit coverage went integration-only). |
| 73 | + |
3 | 74 | ## [1.32.0.0] - 2026-05-10 |
4 | 75 |
|
5 | 76 | ## **Seven contributor PRs land. Three are security or hardening.** |
|
0 commit comments