diff --git a/bin/gstack-learnings-log b/bin/gstack-learnings-log index 6c528d3a8b..f9a06fb7dc 100755 --- a/bin/gstack-learnings-log +++ b/bin/gstack-learnings-log @@ -84,3 +84,12 @@ if [ $? -ne 0 ] || [ -z "$VALIDATED" ]; then fi echo "$VALIDATED" >> "$GSTACK_HOME/projects/$SLUG/learnings.jsonl" + +# Optional memory-brain extension point. If ~/.gstack/hooks/on-learning-written +# exists and is executable, pipe the validated JSON line to it. Backgrounded +# so a slow hook never slows the skill. Exit code ignored so a broken hook +# never breaks the skill. See docs/adapters/README.md. +HOOK="$GSTACK_HOME/hooks/on-learning-written" +if [ -x "$HOOK" ]; then + (printf '%s\n' "$VALIDATED" | "$HOOK" >/dev/null 2>&1) & +fi diff --git a/bin/gstack-timeline-log b/bin/gstack-timeline-log index 0167a1d00a..3192a03acf 100755 --- a/bin/gstack-timeline-log +++ b/bin/gstack-timeline-log @@ -32,3 +32,10 @@ if ! printf '%s' "$INPUT" | bun -e "const j=JSON.parse(await Bun.stdin.text()); fi echo "$INPUT" >> "$GSTACK_HOME/projects/$SLUG/timeline.jsonl" + +# Optional memory-brain extension point. Same shape as on-learning-written. +# See docs/adapters/README.md. +HOOK="$GSTACK_HOME/hooks/on-timeline-written" +if [ -x "$HOOK" ]; then + (printf '%s\n' "$INPUT" | "$HOOK" >/dev/null 2>&1) & +fi diff --git a/context-save/SKILL.md b/context-save/SKILL.md index 8a022652f8..6de55f4171 100644 --- a/context-save/SKILL.md +++ b/context-save/SKILL.md @@ -938,6 +938,20 @@ files_modified: The `files_modified` list comes from `git status --short` (both staged and unstaged modified files). Use relative paths from the repo root. +After writing the file, run the following bash to fire the optional memory-brain +extension hook. If `~/.gstack/hooks/on-checkpoint-written` exists and is +executable, it is invoked with the checkpoint file path on stdin. Backgrounded +so a slow hook never slows the skill. Exit code ignored so a broken hook never +breaks the skill. See `docs/adapters/README.md`. + +```bash +GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}" +HOOK="$GSTACK_HOME/hooks/on-checkpoint-written" +if [ -x "$HOOK" ]; then + (printf '%s\n' "$FILE" | "$HOOK" >/dev/null 2>&1) & +fi +``` + After writing, confirm to the user: ``` diff --git a/context-save/SKILL.md.tmpl b/context-save/SKILL.md.tmpl index 0854baf33b..273d716f21 100644 --- a/context-save/SKILL.md.tmpl +++ b/context-save/SKILL.md.tmpl @@ -181,6 +181,20 @@ files_modified: The `files_modified` list comes from `git status --short` (both staged and unstaged modified files). Use relative paths from the repo root. +After writing the file, run the following bash to fire the optional memory-brain +extension hook. If `~/.gstack/hooks/on-checkpoint-written` exists and is +executable, it is invoked with the checkpoint file path on stdin. Backgrounded +so a slow hook never slows the skill. Exit code ignored so a broken hook never +breaks the skill. See `docs/adapters/README.md`. + +```bash +GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}" +HOOK="$GSTACK_HOME/hooks/on-checkpoint-written" +if [ -x "$HOOK" ]; then + (printf '%s\n' "$FILE" | "$HOOK" >/dev/null 2>&1) & +fi +``` + After writing, confirm to the user: ``` diff --git a/docs/LONG_TERM_PERSISTENCE.md b/docs/LONG_TERM_PERSISTENCE.md new file mode 100644 index 0000000000..defdf9b32f --- /dev/null +++ b/docs/LONG_TERM_PERSISTENCE.md @@ -0,0 +1,204 @@ +# Long-term persistence + +gstack shipped a memory layer built for a 3-month horizon. It works. The +learnings file survives across sessions, the timeline captures every +skill fire, the checkpoints hold real narrative context. For the window +most projects live in, that is enough. + +Past 3 months the substrate starts to leak. Past 1 year it leaks faster. +Past 5 years most of the signal is gone — not because the data is lost, +because the retrieval path can't find it. + +This document does two things. It names the six concrete spots in the +current substrate where signal decays, with file and line. And it +proposes a 20-line extension point — `~/.gstack/hooks/` — that lets any +memory brain plug in without gstack taking a dependency on any of them. + +Disclosure up front. I'm the author of [Dhee](https://github.com/Sankhya-AI/Dhee). +Dhee is one reference implementation of the hook contract below. The +contract itself is provider-neutral and MIT-licensed like the rest of +gstack. I wrote the hook because I wanted Dhee to work cleanly with +gstack, but the value is in the hook, not in Dhee. + +--- + +## Where gstack memory leaks at year 1 and year 5 + +Six concrete spots. Each is fine today. Each starts to bite as the +corpus grows. + +**1. Learnings search is substring match.** +`bin/gstack-learnings-search:99-103`. The retrieval path does +`key.includes(query) || insight.includes(query) || files.includes(query)`. +At 50 learnings per project this works because you remember roughly what +you wrote. At 500 it stops working. "auth session revocation" will not +find the learning titled "logout endpoint leaks bearer token in header" +even though the learning is exactly the thing you need. Semantic search +is the fix. Not lexical search with synonyms. + +**2. No consolidation of near-duplicate keys.** +`bin/gstack-learnings-log:84`. Each `--log` append writes a new line. +The dedup gate is exact-match on `key + type` in +`bin/gstack-learnings-search:82-87`, and that dedup lives in the +retrieval path, not the write path. If you log `retry-backoff-exponential` +today and `exponential-backoff-on-retry` in six months, both survive +forever. The effective-confidence decay helps but doesn't merge. After a +year of logging you have a cloud of near-duplicates that compete for the +same slot in the top-K. + +**3. No correction loop.** +When a learning is wrong, nothing invalidates it. The confidence decay +in `bin/gstack-learnings-search:60-63` treats age as a proxy for +falseness, but a fresh learning that contradicts an older one does not +flag the older one. Six months later the stale learning still rides the +top of its type bucket because its ts field is newer than the +`--supersedes` you never wrote. Correction needs to be a first-class +write, not an implicit consequence of logging. + +**4. Checkpoint rehydration is "newest three files."** +`setup-deploy/SKILL.md:410` — `xargs ls -t | head -3`. Same pattern +repeats in `codex/SKILL.md:409`, `design-review/SKILL.md:412`, and the +other skills that restore context. When a project has 200 checkpoints, +the three newest are not the three most relevant. The useful checkpoint +from the architecture week in month three is ignored for three +Tuesday-morning checkpoints that happen to be last-modified. Semantic +checkpoint recall turns that from a lottery into a lookup. + +**5. No code world-model.** +gstack captures learnings and timeline events but does not build a +structural model of the codebase itself — what files call what, which +modules own which concepts, which tests exercise which paths. The result +is that every skill preamble pays to rediscover structure from raw file +reads. A world-model built from the tool I/O that already flows through +the session is free context the next session could inherit. + +**6. Cross-project trust is an honor-system field.** +`bin/gstack-learnings-search:74-77` gates cross-project loading on +`trusted === false`, which is set by the writing skill, which is the AI. +A prompt-injected learning with `trusted: true` bypasses the gate. The +defense works against honest mistakes and breaks against adversarial +ones. At scale, cross-project learnings need to be scoped by project +identity at retrieval time, not filtered by a self-reported flag at +write time. + +None of these matter at month three. All of them matter at year five. + +--- + +## The fix gstack should ship: a 20-line extension point + +The failure mode in all six cases is the same: retrieval. The writes +are fine. The schema is fine. The storage is fine. What's missing is a +place for a memory brain — any memory brain — to observe what gstack +writes and layer semantic retrieval, consolidation, correction, and +scoping on top. + +The proposal: `~/.gstack/hooks/`. If a file named `on-learning-written`, +`on-timeline-written`, or `on-checkpoint-written` exists and is +executable, gstack runs it after the corresponding write, passes the +payload on stdin (JSON for the JSONL writes, the file path for the +checkpoint write), backgrounds it, and ignores the exit code. + +That is the whole contract. + +```bash +# end of bin/gstack-learnings-log, after `echo "$VALIDATED" >> ...` +HOOK="$GSTACK_HOME/hooks/on-learning-written" +if [ -x "$HOOK" ]; then + printf '%s\n' "$VALIDATED" | "$HOOK" >/dev/null 2>&1 & +fi +``` + +Three properties that make this safe to merge: + +- **Optional.** If the hook file is missing or not executable, gstack + behaves byte-identically to today. Zero risk of changing the happy + path. +- **Non-blocking.** Backgrounded. A slow hook does not slow a skill. +- **Non-fatal.** Exit codes are ignored. A broken hook does not break + the skill. + +No dependency on any external memory system. No new environment +variables to configure. No prompt-injection surface opened — hook output +is not piped back into gstack's LLM workflow, it's fire-and-forget. + +--- + +## Why this completes gstack's own thesis + +The compression-ratio table at the top of `ETHOS.md` describes what +happens to build time when the bottleneck moves from human engineering +to AI-assisted coding. It is a story about making individual engineers +100x. + +That story has a second half that doesn't fit in one developer's head: +what happens when a 100x engineer has been running for two years. The +volume of learnings, timeline, and checkpoints is two orders of +magnitude above what one person would ever hand-curate. The memory +substrate is the bottleneck to the next compression ratio. + +"Boil the Lake" on memory is not a one-person-repo thing. It's a +multi-year, multi-project, multi-repo thing. The current substrate can't +scale there because retrieval is the bottleneck, and retrieval is where +a memory brain belongs. + +The hook does not force gstack to pick a memory brain. It lets anyone +build one. + +--- + +## Reference implementation: Dhee + +Dhee is the brain I wrote for this. It is a separate MIT-licensed +project at https://github.com/Sankhya-AI/Dhee. It already has semantic +search, deduplicating write pipeline, correction and forgetting, +episodic rehydration, a code world-model from tool I/O, and project +scoping enforced at retrieval. + +With this PR merged, a Dhee user runs + +```bash +dhee install gstack +``` + +and Dhee ingests `~/.gstack/projects/*` into its own retrieval layer. +gstack's files are never mutated. gstack standalone keeps working. The +hook contract is what makes the install one command instead of a +documented workaround. + +Someone else writing a different brain against the same hook contract +gets the same one-command install. That's the point. The contract is +the artifact. Dhee is proof it works. + +--- + +## What this PR contains + +- `bin/gstack-learnings-log` — 8 lines at the bottom of the happy path + that run `on-learning-written` if present. +- `bin/gstack-timeline-log` — same shape, `on-timeline-written`. +- `context-save/SKILL.md.tmpl` — same shape at the end of the save flow, + `on-checkpoint-written`, receiving the written file path. +- `context-save/SKILL.md` — regenerated via `bun run gen:skill-docs`. +- `docs/adapters/README.md` — spec for the three hook payloads. +- `docs/LONG_TERM_PERSISTENCE.md` — this document. +- Five tests in `test/hooks-*.test.ts` covering presence, absence, the + non-blocking guarantee, the non-fatal guarantee, and the payload + contract. + +`wc -l` on the runtime change is small. The substrate does not move. +The happy path with no hook installed is byte-identical to the current +release. + +--- + +## Happy for this to sit as an artifact + +Merge would be great. Not merging is fine too. The hook contract works +locally either way because `~/.gstack/hooks/` is filesystem-only. A +reader who wants to try it can add the eight lines to their fork in an +afternoon. + +The goal of this document is not to land a PR. It is to name the place +where gstack memory stops scaling and propose a contract small enough +that it costs almost nothing to adopt. diff --git a/docs/adapters/README.md b/docs/adapters/README.md new file mode 100644 index 0000000000..612eebbb80 --- /dev/null +++ b/docs/adapters/README.md @@ -0,0 +1,49 @@ +# Memory-brain extension point + +gstack stores its memory (learnings, timeline, checkpoints) as plain files +under `${GSTACK_HOME:-$HOME/.gstack}/projects//`. Anyone writing a +memory brain can observe those writes in real time by dropping an +executable into `~/.gstack/hooks/`. + +The contract is three files, all optional: + +| Hook path | Fires when | Payload on stdin | +|---|---|---| +| `~/.gstack/hooks/on-learning-written` | A new learning is appended to `learnings.jsonl` | The validated JSON line (one line, newline-terminated) | +| `~/.gstack/hooks/on-timeline-written` | A new event is appended to `timeline.jsonl` | The validated JSON line (one line, newline-terminated) | +| `~/.gstack/hooks/on-checkpoint-written` | A checkpoint markdown file is saved | The absolute path to the written file | + +## Contract + +- **Optional.** If the hook file is missing or not executable, gstack + behaves byte-identically to today. +- **Non-blocking.** Every hook invocation is backgrounded. gstack never + waits for the hook to finish. +- **Non-fatal.** Exit codes are ignored. A broken hook never breaks a + skill. +- **Untrusted.** Hook output is not piped back into gstack's workflow. + Fire-and-forget. No new prompt-injection surface. + +## Example: write every learning to a sentinel file + +```bash +#!/usr/bin/env bash +# ~/.gstack/hooks/on-learning-written +cat >> /tmp/gstack-learnings-sink.jsonl +``` + +```bash +chmod +x ~/.gstack/hooks/on-learning-written +``` + +That is the whole integration surface. + +## Reference implementation + +[Dhee](https://github.com/Sankhya-AI/Dhee) is one memory brain that +consumes this contract. After `dhee install gstack`, Dhee registers +hooks at these three paths and ingests gstack's writes into its own +semantic retrieval layer. gstack files are never mutated. + +The contract is provider-neutral. Any brain with a shell binary can +consume it.