fix(cli): followups drift correctness — see the working tree (#229) + catch appended follow-ups (#231) (#235)

Two silent-data-loss bugs in `straymark followups drift`, both surfaced by
the reference adopter (Sentinel) and both prerequisites for wiring drift into
the Charter-close flow (RFC #135 Tier 3).

- #229: the default scan considered only the committed git range, so an
  uncommitted/untracked AILOG (the normal pre-commit state) was invisible and
  `drift --apply` reported "in sync" while real follow-up content went
  unextracted. The scan now unions the git range with the working tree
  (`git status --porcelain`), mirroring the v0 reference script.
- #231: `fully_extracted_ailogs` was a whole-AILOG idempotency gate — once an
  AILOG id was in it the file was never re-scanned, so follow-ups appended
  later (the multi-batch Charter pattern) were silently dropped. Drift now
  dedups per follow-up by a stable content hash (`Source-hash`, stored on each
  entry): already-extracted AILOGs are re-scanned and individual follow-ups
  deduped. The stored hash is captured at extraction time, immune to later
  triage rewording — preserving the zero-false-positive property; legacy
  entries fall back to recomputing the hash from Origin + description.

Updates FOLLOW-UPS-BACKLOG-PATTERN.md (EN/ES/zh-CN) to match; the old
"Per-AILOG vs per-bullet granularity" section documented the #231 bug as an
intentional trade-off. Bump fw-4.24.0 → 4.25.0 / cli-3.20.0 → 3.21.0.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: montfort

CHANGELOG.md

@@ -7,6 +7,21 @@ and this project uses [independent versioning](README.md#versioning) for Framewo
 
 ---
 
+## Framework 4.25.0 / CLI 3.21.0 — follow-ups drift correctness: see the working tree + catch appended follow-ups
+
+Two silent-data-loss bugs in `straymark followups drift`, both surfaced by the reference adopter (Sentinel) and both prerequisites for wiring drift into the Charter-close flow (RFC [#135](https://github.com/StrangeDaysTech/straymark/issues/135) Tier 3).
+
+### Fixed (CLI)
+
+- **`followups drift` default scan now sees the working tree** ([#229](https://github.com/StrangeDaysTech/straymark/issues/229)): the default scan considered only the committed git range (`git diff origin/main..HEAD`), so an uncommitted/untracked AILOG — the normal state at pre-commit time — was invisible, and the documented `drift --apply` flow reported "in sync" while real follow-up content went unextracted. The scan now unions the git range with the working tree (`git status --porcelain`), mirroring the v0 reference script. `--scan-all` semantics unchanged.
+- **`followups drift` catches follow-ups appended to an already-extracted AILOG** ([#231](https://github.com/StrangeDaysTech/straymark/issues/231)): `fully_extracted_ailogs` was a whole-AILOG idempotency gate — once an AILOG id was in it, the file was never re-scanned, so follow-ups added later (the multi-batch Charter pattern, where one AILOG's `§Follow-ups` grows across batches) were silently dropped. Drift now dedups **per follow-up by a stable content hash** (`Source-hash`, stored on each entry): already-extracted AILOGs are re-scanned and individual follow-ups deduped, so appended content is caught. The stored hash is captured at extraction time and immune to later triage rewording, preserving the zero-false-positive property that motivated the original per-AILOG choice; legacy entries (pre-cli-3.21.0, no `Source-hash`) fall back to recomputing the hash from `Origin` + `description`. `fully_extracted_ailogs` is retained as informational metadata, no longer the skip gate.
+
+### Changed (Framework)
+
+- **`FOLLOW-UPS-BACKLOG-PATTERN.md`** (EN/ES/zh-CN): documents the working-tree union, the per-follow-up content-hash dedup (replacing the "Per-AILOG vs per-bullet granularity" section that described the #231 bug as an intentional trade-off), the new auto-managed `Source-hash` entry field, and the revised role of `fully_extracted_ailogs`.
+
+---
+
 ## Framework 4.24.0 — portable installed skills: runtime-agnostic identity + current-work git context
 
 Five portability/correctness fixes to the installed skills, surfaced by a Codex (`gpt-5.5`) review of the StrayMark skills running under Codex CLI in the reference adopter Sentinel ([#232](https://github.com/StrangeDaysTech/straymark/issues/232)). The skills ship in four families under `dist/` (`.claude/`, `.gemini/`, `.agent/workflows/` hand-maintained; `.codex/` generated from `.claude/` by `gen_codex_skills`); the fixes were applied to the hand-maintained sources and the Codex tree regenerated.

README.md

@@ -277,8 +277,8 @@ StrayMark uses independent version tags for each component:
 
 | Component | Tag prefix | Example | Includes |
 | --- | --- | --- | --- |
-| Framework | `fw-` | `fw-4.24.0` | Templates (12 types), governance, directives, Charter template + schema |
-| CLI | `cli-` | `cli-3.20.0` | The `straymark` binary |
+| Framework | `fw-` | `fw-4.25.0` | Templates (12 types), governance, directives, Charter template + schema |
+| CLI | `cli-` | `cli-3.21.0` | The `straymark` binary |
 
 Check installed versions with `straymark status` or `straymark about`.
 
@@ -311,7 +311,7 @@ See [CLI Reference](https://github.com/StrangeDaysTech/straymark/blob/main/docs/
 ```bash
 # Download the latest framework release ZIP from GitHub
 # Go to https://github.com/StrangeDaysTech/straymark/releases
-# and download the latest fw-* release (e.g., fw-4.24.0)
+# and download the latest fw-* release (e.g., fw-4.25.0)
 
 # Extract and copy to your project
 unzip straymark-fw-*.zip -d your-project/

cli/Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "straymark-cli"
-version = "3.20.0"
+version = "3.21.0"
 edition = "2021"
 description = "CLI for StrayMark — the cognitive discipline your AI-assisted projects need"
 license = "MIT"

cli/Cargo.toml

@@ -4,9 +4,11 @@
 //! `check-followups-drift.sh`. Three modes, mirroring the v0 script:
 //!
 //! - **default** — scan AILOGs changed in `git diff origin/main..HEAD`
-//!   (fallback `origin/master..HEAD`, then `HEAD~1..HEAD` with a warning).
-//!   Warn + exit 1 when an AILOG with follow-up content is not in
-//!   `fully_extracted_ailogs`.
+//!   (fallback `origin/master..HEAD`, then `HEAD~1..HEAD` with a warning)
+//!   **unioned with the working tree** (`git status --porcelain`) so an
+//!   uncommitted/untracked AILOG is visible to the documented pre-commit flow
+//!   (#229). Warn + exit 1 when an AILOG carries follow-up content not yet in
+//!   the registry.
 //! - **`--apply`** — same scan + extract new entries into `## Bucket: ready`,
 //!   append the AILOG ids to `fully_extracted_ailogs`, **recompute the
 //!   `total_*` counters** (#214 Signal 2) and upgrade v0 → v1 in place.
@@ -15,9 +17,14 @@
 //!   refinement).
 //! - **`--scan-all`** — sweep every AILOG in the project.
 //!
-//! Granularity is **per-AILOG**, never per-bullet — the empirically validated
-//! design choice (0 false positives across 76 AILOGs in the reference
-//! adopter). See `FOLLOW-UPS-BACKLOG-PATTERN.md` §Drift detection.
+//! Drift is detected **per follow-up by content hash** (`fu_content_hash`),
+//! not by a whole-AILOG idempotency gate. Already-extracted AILOGs are
+//! re-scanned and individual follow-ups deduped against the registry, so
+//! content appended to a multi-batch AILOG after its first extraction is
+//! caught instead of silently skipped (#231). The hash dedup preserves the
+//! empirically validated zero-false-positive property (no AILOG re-flags once
+//! its content is in the registry). See `FOLLOW-UPS-BACKLOG-PATTERN.md`
+//! §Drift detection.
 //!
 //! ## Exit codes
 //! - `0` — no drift (or `--apply` extracted everything)
@@ -75,37 +82,50 @@ pub fn run(path: &str, apply: bool, scan_all: bool, range: Option<&str>) -> Resu
         utils::warn(w);
     }
 
-    // Candidate AILOG files.
+    // Candidate AILOG files. Default mode unions the committed git range with
+    // the working tree (#229): at pre-commit time the just-written AILOG is
+    // usually untracked, so a range-only scan is blind to exactly the file the
+    // documented `drift --apply` flow targets.
     let agent_logs = ailog::agent_logs_dir(project_root);
     let candidates: Vec<PathBuf> = if scan_all {
         walk_ailogs(&agent_logs)
     } else {
-        ailogs_in_git_range(project_root, &agent_logs, range)
+        let mut set = ailogs_in_git_range(project_root, &agent_logs, range);
+        for p in ailogs_in_working_tree(project_root, &agent_logs) {
+            if !set.contains(&p) {
+                set.push(p);
+            }
+        }
+        set
     };
 
-    // Drift = AILOG with follow-up content whose id is NOT in
-    // fully_extracted_ailogs.
-    let extracted_set: std::collections::HashSet<&str> = registry
-        .frontmatter
-        .fully_extracted_ailogs
-        .iter()
-        .map(|s| s.as_str())
-        .collect();
+    // Drift = an AILOG carrying follow-up content whose content hash is NOT yet
+    // in the registry (#231). Re-scanning already-extracted AILOGs and deduping
+    // per follow-up by content hash — instead of skipping the whole file once
+    // its id lands in `fully_extracted_ailogs` — is what catches follow-ups
+    // appended to a multi-batch AILOG after its first extraction.
+    let seen_hashes = followups::registry_extracted_hashes(&registry);
 
     let mut drifted: Vec<(String, PathBuf, Vec<ExtractedFu>)> = Vec::new();
     for path in candidates {
         let Some(id) = followups::ailog_id_from_path(&path) else {
             continue;
         };
-        if extracted_set.contains(id.as_str()) {
-            continue;
-        }
         let Ok(content) = std::fs::read_to_string(&path) else {
             continue;
         };
-        let found = followups::extract_followups_from_ailog(&content);
-        if !found.is_empty() {
-            drifted.push((id, path, found));
+        let new: Vec<ExtractedFu> = followups::extract_followups_from_ailog(&content)
+            .into_iter()
+            .filter(|fu| {
+                !seen_hashes.contains(&followups::fu_content_hash(
+                    &id,
+                    &fu.origin_section,
+                    &fu.description,
+                ))
+            })
+            .collect();
+        if !new.is_empty() {
+            drifted.push((id, path, new));
         }
     }
 
@@ -182,8 +202,20 @@ pub fn run(path: &str, apply: bool, scan_all: bool, range: Option<&str>) -> Resu
         }
     }
 
-    // Frontmatter: append AILOG ids, set last_scan, recompute counters + v1.
-    let new_ids: Vec<String> = drifted.iter().map(|(id, _, _)| id.clone()).collect();
+    // Frontmatter: append AILOG ids (deduped — a re-scanned AILOG already in
+    // the list must not be appended twice), set last_scan, recompute counters.
+    let already: std::collections::HashSet<&str> = registry
+        .frontmatter
+        .fully_extracted_ailogs
+        .iter()
+        .map(|s| s.as_str())
+        .collect();
+    let mut new_ids: Vec<String> = Vec::new();
+    for (id, _, _) in &drifted {
+        if !already.contains(id.as_str()) && !new_ids.contains(id) {
+            new_ids.push(id.clone());
+        }
+    }
     let mut fm = followups::fm_append_list_items(
         &registry.frontmatter_raw,
         "fully_extracted_ailogs",
@@ -303,6 +335,64 @@ fn ailogs_in_git_range(
         .collect()
 }
 
+/// AILOG files present in the working tree (staged, unstaged, or untracked),
+/// via `git status --porcelain`. Unioned into the default scan so the
+/// documented pre-commit `drift --apply` flow sees the just-written AILOG
+/// before it is committed (#229) — mirroring the v0 reference script, which
+/// folded `git status --porcelain` into its default scan set for exactly this
+/// in-progress agent workflow.
+fn ailogs_in_working_tree(project_root: &Path, agent_logs: &Path) -> Vec<PathBuf> {
+    let output = Command::new("git")
+        .args(["status", "--porcelain"])
+        .current_dir(project_root)
+        .output();
+    let Ok(output) = output else {
+        return Vec::new();
+    };
+    if !output.status.success() {
+        return Vec::new();
+    }
+    let agent_logs_rel = agent_logs
+        .strip_prefix(project_root)
+        .unwrap_or(agent_logs)
+        .to_path_buf();
+    parse_porcelain_ailogs(&String::from_utf8_lossy(&output.stdout), &agent_logs_rel)
+        .into_iter()
+        .map(|p| project_root.join(p))
+        .filter(|p| p.exists())
+        .collect()
+}
+
+/// Parse `git status --porcelain` (v1) output into AILOG file paths relative to
+/// the repo root, filtered to the agent-logs dir. Handles the `XY <path>`
+/// shape, untracked `?? <path>`, and rename/copy `R  <old> -> <new>` (takes the
+/// destination). Pure for testability.
+fn parse_porcelain_ailogs(stdout: &str, agent_logs_rel: &Path) -> Vec<PathBuf> {
+    let mut out = Vec::new();
+    for line in stdout.lines() {
+        // Porcelain v1: two status chars + a space, then the path (≥ 4 chars).
+        if line.len() < 4 {
+            continue;
+        }
+        let rest = &line[3..];
+        // Rename/copy entries render as `old -> new`; the destination is what
+        // exists on disk.
+        let path_str = rest.rsplit(" -> ").next().unwrap_or(rest).trim();
+        // Porcelain quotes paths containing special chars; strip the quotes.
+        let path_str = path_str.trim_matches('"');
+        let p = PathBuf::from(path_str);
+        if p.starts_with(agent_logs_rel)
+            && p.file_name()
+                .and_then(|n| n.to_str())
+                .map(|n| n.starts_with("AILOG-") && n.ends_with(".md"))
+                .unwrap_or(false)
+        {
+            out.push(p);
+        }
+    }
+    out
+}
+
 fn git_ref_exists(project_root: &Path, r: &str) -> bool {
     Command::new("git")
         .args(["rev-parse", "--verify", "--quiet", r])
@@ -311,3 +401,58 @@ fn git_ref_exists(project_root: &Path, r: &str) -> bool {
         .map(|o| o.status.success())
         .unwrap_or(false)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn parse_porcelain_picks_ailogs_across_status_codes() {
+        let rel = Path::new(".straymark/07-ai-audit/agent-logs");
+        // Built via join so leading status spaces (` M`) survive verbatim — a
+        // `\`-continued string literal would strip them.
+        let stdout = [
+            "?? .straymark/07-ai-audit/agent-logs/AILOG-2026-06-09-001-untracked.md",
+            " M .straymark/07-ai-audit/agent-logs/AILOG-2026-06-09-002-modified.md",
+            "A  .straymark/07-ai-audit/agent-logs/AILOG-2026-06-09-003-staged.md",
+            "R  .straymark/07-ai-audit/agent-logs/AILOG-old.md -> .straymark/07-ai-audit/agent-logs/AILOG-2026-06-09-004-renamed.md",
+            " M .straymark/07-ai-audit/agent-logs/notes.md",
+            "?? src/main.rs",
+            " M .straymark/07-ai-audit/decisions/AIDEC-2026-06-09-001.md",
+        ]
+        .join("\n");
+        let got = parse_porcelain_ailogs(&stdout, rel);
+        let names: Vec<String> = got
+            .iter()
+            .map(|p| p.file_name().unwrap().to_string_lossy().into_owned())
+            .collect();
+        assert_eq!(
+            names,
+            vec![
+                "AILOG-2026-06-09-001-untracked.md",
+                "AILOG-2026-06-09-002-modified.md",
+                "AILOG-2026-06-09-003-staged.md",
+                "AILOG-2026-06-09-004-renamed.md", // rename → destination path
+            ]
+        );
+        // notes.md (not AILOG-*), src/main.rs (outside dir), and the AIDEC
+        // (sibling dir) are all excluded.
+    }
+
+    #[test]
+    fn parse_porcelain_tolerates_quoted_paths_and_short_lines() {
+        let rel = Path::new(".straymark/07-ai-audit/agent-logs");
+        let stdout = [
+            "?? \".straymark/07-ai-audit/agent-logs/AILOG-2026-06-09-005-spaced name.md\"",
+            "",
+            "XX",
+        ]
+        .join("\n");
+        let got = parse_porcelain_ailogs(&stdout, rel);
+        assert_eq!(got.len(), 1);
+        assert_eq!(
+            got[0].file_name().unwrap().to_string_lossy(),
+            "AILOG-2026-06-09-005-spaced name.md"
+        );
+    }
+}