feat(orchestrator): startup worktree sweep for SIGKILL residue Refs #1570 (Gitea) (#875)

Layer 2 of the worktree lifecycle defence in depth from epic
terraphim/terraphim-ai#1567. Between Layer 1 (WorktreeGuard::Drop
on the happy/cancelled path) and Layer 3 (root-privileged
ExecStartPre), this sweep reconciles residue left behind when the
previous orchestrator process died via SIGKILL, OOM, or a
panic-across-runtime where Drop never ran.

Changes
-------
* scope.rs:
  - New `pub fn WorktreeManager::sweep_stale(&self, extra_roots:
    &[PathBuf]) -> SweepReport`. Synchronous on purpose so it can
    run inline in `AgentOrchestrator::new` before any tick thread
    is spawned.
  - Walks `worktree_base` for `WORKTREE_REVIEW_PREFIX`-prefixed
    children, walks every direct child of each `extra_roots` entry
    (no prefix filter -- the per-agent root convention has no
    prefix), tries `git worktree remove --force` and falls back to
    `std::fs::remove_dir_all`. EACCES/EPERM increments
    `root_owned_skipped` rather than `failed_count` (Layer 3
    territory). Then runs `git worktree prune --verbose` and emits
    a single structured `info!` line with every report field plus
    a synthetic `backlog_count = swept_count + root_owned_skipped`
    for Quickwit alerting.
  - New `pub struct SweepReport` carrying `swept_count`,
    `failed_count`, `root_owned_skipped`, `prune_succeeded`,
    `duration_ms`.
  - Seven new unit tests covering empty/missing base, review
    prefix removal, non-review preservation, prune execution,
    extra-roots no-prefix-filter, and the Linux-only root-owned
    contract test (runtime gated by `getuid() == 0`; documents the
    contract even when it cannot run).

* compound.rs:
  - New `pub fn CompoundReviewWorkflow::worktree_manager(&self)
    -> &WorktreeManager` trivial accessor for the field needed by
    the Layer 2 wiring.

* lib.rs:
  - `AgentOrchestrator::new` now calls
    `compound_workflow.worktree_manager().sweep_stale(&[
    PathBuf::from("/tmp/adf-worktrees") ])` immediately after the
    compound workflow is constructed, before `Ok(Self { ... })`.
    The /tmp/adf-worktrees literal mirrors lib.rs:5393 (the
    per-agent worktree root from `create_agent_worktree`).
  - Logs a `warn!` when the residue backlog at startup exceeds 10,
    which the research doc flagged as the prior-crash-storm
    threshold.

* tests/sweep_on_startup_test.rs (NEW):
  - Three integration tests around an isolated TempDir git repo
    via `scope::test_support::setup_git_repo`. Verifies that
    `AgentOrchestrator::new` does sweep `review-*` residue,
    preserves non-review siblings, and that extra_roots children
    are removed regardless of prefix.

* Cargo.toml:
  - Declares the new integration test with
    `required-features = ["test-helpers"]`, mirroring the Layer 1
    `compound_cancellation_test` entry.

Deviations from the brief
-------------------------
1. Struct name: the brief says `Orchestrator::new`; the actual
   symbol is `AgentOrchestrator::new` (lib.rs:661). Insertion site
   is between `compound_workflow` construction at :687 and the
   `Ok(Self { ... })` block at :786.

2. `#[cfg(not(test))]` gate on the sweep call. The in-lib
   `test_config()` (lib.rs:7926) points `repo_path` at the live
   terraphim-ai checkout. Without the gate, the sweep's
   `git worktree prune --verbose` ran against that real repo and
   raced against `test_orchestrator_compound_review_manual`'s
   concurrent `git worktree add` on the same shared
   `.git/worktrees/` admin registry. Production wiring is fully
   exercised by `tests/sweep_on_startup_test.rs` (a separate
   crate, so `cfg(test)` is not set on the lib's own code path
   when it links the integration test). The gate is documented
   inline with a pointer to the integration test.

3. SweepReport is placed at module scope in scope.rs (as the
   brief preferred), not wrapped in `WorktreeManager`.

4. The Linux/root-only contract test is included as the brief
   requested. It uses a raw `extern "C" getuid()` FFI to avoid
   pulling in `nix` or `libc` solely for a single gated test.

Verification
------------
* `cargo test -p terraphim_orchestrator --features test-helpers
   --test sweep_on_startup_test -- --nocapture`: 3 passed.
* `cargo test -p terraphim_orchestrator --features test-helpers`:
   691 lib tests + all integration tests pass, including Layer 1
   `compound_cancellation_test`.
* `cargo clippy -p terraphim_orchestrator --lib --tests
   --features test-helpers -- -D warnings`: clean.
* `cargo fmt`: clean.

Branched from task/1569-worktree-guard (Layer 1) which provides
the `WORKTREE_REVIEW_PREFIX` constant referenced by the sweep.

Co-authored-by: Alex <alex@example.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: 3 people

crates/terraphim_orchestrator/Cargo.toml

@@ -101,6 +101,15 @@ name = "compound_cancellation_test"
 path = "tests/compound_cancellation_test.rs"
 required-features = ["test-helpers"]
 
+# Layer 2 startup sweep integration test (epic #1567 / issue #1570).
+# Same test-helpers gate as Layer 1 -- the test re-uses the shared
+# `scope::test_support::setup_git_repo` fixture so the sweep never
+# runs against the live repository.
+[[test]]
+name = "sweep_on_startup_test"
+path = "tests/sweep_on_startup_test.rs"
+required-features = ["test-helpers"]
+
 
 [[bin]]
 name = "adf"

crates/terraphim_orchestrator/src/compound.rs

@@ -250,6 +250,18 @@ impl CompoundReviewWorkflow {
         Self::new(swarm_config)
     }
 
+    /// Borrow the inner [`WorktreeManager`].
+    ///
+    /// Layer 2 (epic #1567, issue #1570) calls
+    /// `worktree_manager().sweep_stale(...)` from
+    /// `AgentOrchestrator::new` to reconcile stale `review-*` residue
+    /// left behind by SIGKILL / OOM before any tick thread runs.
+    /// Production code outside the startup sweep should prefer the
+    /// higher-level workflow methods.
+    pub fn worktree_manager(&self) -> &WorktreeManager {
+        &self.worktree_manager
+    }
+
     /// Run a full compound review cycle.
     ///
     /// 1. Get changed files between git_ref and base_ref

crates/terraphim_orchestrator/src/lib.rs

@@ -693,6 +693,45 @@ impl AgentOrchestrator {
         let scheduler = TimeScheduler::new(&config.agents, Some(&config.compound_review.schedule))?;
         let compound_workflow =
             CompoundReviewWorkflow::from_compound_config(config.compound_review.clone());
+
+        // Layer 2 startup sweep (epic #1567, issue #1570).
+        //
+        // Reconcile any worktree residue left by a previous instance
+        // before we accept ticks. Synchronous: must finish before the
+        // tick thread is spawned in `run()` so a fresh review cycle
+        // never races against half-killed `review-*` directories.
+        //
+        // `extra_roots` mirrors the per-agent worktree convention
+        // from `lib.rs:5393`. If you change that literal there, change
+        // it here too.
+        //
+        // `cfg(not(test))` gate: the in-lib `test_config()` (see
+        // `lib.rs:7926`) points `repo_path` at the live terraphim-ai
+        // checkout. Without this gate, the sweep's
+        // `git worktree prune --verbose` races against
+        // `test_orchestrator_compound_review_manual`'s concurrent
+        // `git worktree add` on that shared real repo's
+        // `.git/worktrees/` admin registry. The production wiring is
+        // exercised end-to-end by `tests/sweep_on_startup_test.rs`,
+        // which builds an isolated `TempDir` repo and asserts the
+        // sweep DOES run from `AgentOrchestrator::new`. Do not remove
+        // this gate without first migrating the in-lib `test_config()`
+        // to a TempDir-based `repo_path`.
+        #[cfg(not(test))]
+        {
+            let sweep_report = compound_workflow
+                .worktree_manager()
+                .sweep_stale(&[PathBuf::from("/tmp/adf-worktrees")]);
+            if sweep_report.swept_count + sweep_report.root_owned_skipped > 10 {
+                warn!(
+                    swept_count = sweep_report.swept_count,
+                    root_owned_skipped = sweep_report.root_owned_skipped,
+                    failed_count = sweep_report.failed_count,
+                    "large worktree backlog at startup -- prior crash storm likely"
+                );
+            }
+        }
+
         let handoff_buffer = HandoffBuffer::new(config.handoff_buffer_ttl_secs.unwrap_or(86400));
         let handoff_ledger = HandoffLedger::new(config.working_dir.join("handoff-ledger.jsonl"));