Skip to content

Commit 4fa7452

Browse files
author
Terraphim CI
committed
docs: research + design for cursor-based mention polling Refs #186
Root cause analysis of the Apr 3 mention replay storm (load 301). Design follows Issue #186 with additions: - max_dispatches_per_tick rate limit - max_concurrent_mention_agents cap - Startup guard (cursor=now on first run) - Eliminates watch_issues config and processed HashSet Supersedes #249.
1 parent 2d109bc commit 4fa7452

2 files changed

Lines changed: 323 additions & 0 deletions

File tree

‎.docs/design-cursor-mention-polling.md‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
# Implementation Plan: Cursor-Based Mention Polling
2+
3+
**Research doc:** `.docs/research-mention-replay-storm.md`
4+
**Gitea issue:** #186
5+
**Closes:** #249 (superseded)
6+
7+
---
8+
9+
## 5/25 Elimination
10+
11+
**IN (top 5):**
12+
1. Cursor-based repo-wide polling (single API call)
13+
2. Persistent cursor via terraphim_persistence
14+
3. max_dispatches_per_tick rate limit
15+
4. Startup guard (cursor=now on first run)
16+
5. Backward compat deprecation of watch_issues
17+
18+
**AVOID:**
19+
- Per-issue depth tracking (remove, cursor makes it unnecessary)
20+
- Complex deduplication logic
21+
- Webhook-driven mentions (Issue #149, deferred)
22+
- Processed HashSet persistence (cursor replaces it)
23+
- Custom SQLite table for cursor (use existing KV store)
24+
25+
---
26+
27+
## File Changes
28+
29+
### Step 1: New `MentionCursor` type + persistence
30+
31+
**File:** `crates/terraphim_orchestrator/src/mention.rs`
32+
33+
```rust
34+
/// Persistent cursor for mention polling.
35+
/// Stored via terraphim_persistence as JSON at key "adf/mention_cursor".
36+
#[derive(Debug, Clone, Serialize, Deserialize)]
37+
pub struct MentionCursor {
38+
/// ISO 8601 timestamp of the last processed comment.
39+
pub last_seen_at: String,
40+
/// Counter of dispatches in current tick (reset each poll cycle).
41+
#[serde(skip)]
42+
pub dispatches_this_tick: u32,
43+
}
44+
45+
impl MentionCursor {
46+
/// Create a cursor set to "now" (skip all historical mentions).
47+
pub fn now() -> Self {
48+
Self {
49+
last_seen_at: chrono::Utc::now().to_rfc3339(),
50+
dispatches_this_tick: 0,
51+
}
52+
}
53+
54+
/// Load from persistence or create "now" cursor.
55+
pub async fn load_or_now() -> Self { ... }
56+
57+
/// Save to persistence.
58+
pub async fn save(&self) { ... }
59+
}
60+
```
61+
62+
**Remove:** `MentionTracker.processed` HashSet (no longer needed).
63+
**Keep:** `MentionTracker.depth_counters` renamed to `dispatches_per_issue: HashMap<u64, u32>` (flood protection).
64+
65+
### Step 2: New `fetch_repo_comments` in tracker
66+
67+
**File:** `crates/terraphim_tracker/src/gitea.rs`
68+
69+
```rust
70+
/// Fetch comments across all issues in a repo since a timestamp.
71+
pub async fn fetch_repo_comments(
72+
&self,
73+
since: &str,
74+
limit: u32,
75+
) -> Result<Vec<IssueComment>> {
76+
// GET /api/v1/repos/{owner}/{repo}/issues/comments?since={since}&limit={limit}
77+
}
78+
```
79+
80+
The `IssueComment` struct needs an `issue_number` field extracted from `issue_url`.
81+
82+
### Step 3: Update `MentionConfig`
83+
84+
**File:** `crates/terraphim_orchestrator/src/config.rs`
85+
86+
```rust
87+
pub struct MentionConfig {
88+
/// DEPRECATED: Ignored when cursor polling is active.
89+
#[serde(default)]
90+
pub watch_issues: Vec<u64>,
91+
92+
/// Max mentions to dispatch per poll tick.
93+
#[serde(default = "default_max_dispatches_per_tick")]
94+
pub max_dispatches_per_tick: u32, // default: 3
95+
96+
/// Max concurrent mention-spawned agents.
97+
#[serde(default = "default_max_concurrent_mention_agents")]
98+
pub max_concurrent_mention_agents: u32, // default: 5
99+
100+
/// Poll every N reconciliation ticks.
101+
#[serde(default = "default_poll_modulo")]
102+
pub poll_modulo: u64,
103+
104+
/// Persistence key prefix for cursor.
105+
#[serde(default)]
106+
pub cursor_key: String, // default: "adf/mention_cursor"
107+
}
108+
```
109+
110+
Remove `max_mention_depth` (replaced by `max_dispatches_per_tick`).
111+
112+
### Step 4: Rewrite `poll_mentions()`
113+
114+
**File:** `crates/terraphim_orchestrator/src/lib.rs` (lines 1055-1170)
115+
116+
```rust
117+
async fn poll_mentions(&mut self) {
118+
let cfg = match self.config.mentions.clone() { ... };
119+
let gitea_cfg = match self.config.gitea.clone() { ... };
120+
121+
if self.tick_count % cfg.poll_modulo != 0 { return; }
122+
123+
// Count currently active mention-spawned agents
124+
let active_mention_agents = self.active_agents.values()
125+
.filter(|a| a.spawned_by_mention)
126+
.count() as u32;
127+
if active_mention_agents >= cfg.max_concurrent_mention_agents { return; }
128+
129+
// Load cursor
130+
let mut cursor = MentionCursor::load_or_now().await;
131+
cursor.dispatches_this_tick = 0;
132+
133+
// Single API call: all comments since cursor
134+
let comments = tracker.fetch_repo_comments(&cursor.last_seen_at, 50).await?;
135+
136+
for comment in &comments {
137+
if cursor.dispatches_this_tick >= cfg.max_dispatches_per_tick { break; }
138+
139+
let mentions = parse_mentions(comment, comment.issue_number, &agents, &personas);
140+
for m in mentions {
141+
if cursor.dispatches_this_tick >= cfg.max_dispatches_per_tick { break; }
142+
143+
// Spawn
144+
self.spawn_agent(&mention_def).await?;
145+
cursor.dispatches_this_tick += 1;
146+
}
147+
148+
// Advance cursor past this comment
149+
cursor.last_seen_at = comment.created_at.clone();
150+
}
151+
152+
cursor.save().await;
153+
}
154+
```
155+
156+
### Step 5: Track mention-spawned agents
157+
158+
**File:** `crates/terraphim_orchestrator/src/lib.rs`
159+
160+
Add `spawned_by_mention: bool` to `ManagedAgent` struct. Set to `true` when spawned from `poll_mentions()`.
161+
162+
---
163+
164+
## Test Strategy
165+
166+
| Test | Type | What it verifies |
167+
|---|---|---|
168+
| `test_cursor_load_or_now` | Unit | Fresh cursor uses current time |
169+
| `test_cursor_persistence_roundtrip` | Unit | Save + load preserves last_seen_at |
170+
| `test_poll_mentions_respects_max_dispatches` | Unit | Stops after N dispatches per tick |
171+
| `test_poll_mentions_advances_cursor` | Unit | Cursor advances to latest comment |
172+
| `test_poll_mentions_concurrent_limit` | Unit | Skips poll when max concurrent reached |
173+
| `test_no_replay_on_restart` | Integration | Restart with persisted cursor doesn't re-dispatch |
174+
175+
---
176+
177+
## Step Sequence
178+
179+
| Step | Description | Files | Depends on |
180+
|---|---|---|---|
181+
| 1 | MentionCursor + persistence | mention.rs | — |
182+
| 2 | fetch_repo_comments API | terraphim_tracker/gitea.rs | — |
183+
| 3 | Update MentionConfig | config.rs | — |
184+
| 4 | Rewrite poll_mentions() | lib.rs | 1, 2, 3 |
185+
| 5 | spawned_by_mention tracking | lib.rs | 4 |
186+
| 6 | Tests | mention.rs, lib.rs tests | 1-5 |
187+
| 7 | Deploy + verify on bigbox | orchestrator.toml | 6 |
188+
189+
Steps 1-3 are independent and can be done in parallel.
190+
191+
---
192+
193+
## Rollback
194+
195+
If cursor polling fails, re-enable `[mentions]` with `watch_issues` — the old code path still works. The cursor is additive.
196+
197+
---
198+
199+
## Config migration
200+
201+
```toml
202+
# OLD (remove)
203+
[mentions]
204+
watch_issues = [107, 108, ...]
205+
max_mention_depth = 10
206+
poll_modulo = 2
207+
208+
# NEW
209+
[mentions]
210+
max_dispatches_per_tick = 3
211+
max_concurrent_mention_agents = 5
212+
poll_modulo = 2
213+
```

‎.docs/research-mention-replay-storm.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Research: ADF Mention Replay Storm
2+
3+
## Phase 1 — Disciplined Research
4+
5+
### 1. Problem Understanding
6+
7+
**What:** On ADF restart, `poll_mentions()` replays ALL unprocessed `@adf:agent-name` mentions from watched Gitea issues in a single tick. Each mention spawns a full agent process (opencode/claude). After extended downtime this creates a process storm.
8+
9+
**Impact:** On Apr 3 2026, 43 agents spawned simultaneously → 170+ processes → load average 301 → bigbox unresponsive for ~5 minutes.
10+
11+
**Who:** ADF operator (Alex). Any restart after >6h downtime triggers this.
12+
13+
**Success:** Mentions are processed incrementally with bounded concurrency. Restarts never cause storms.
14+
15+
### 2. Existing System Analysis
16+
17+
#### Current code (two versions in play)
18+
19+
| Component | Local (`feature/warp-drive-theme`) | Bigbox (`task/173-automata-orchestrator-integration`) |
20+
|---|---|---|
21+
| `MentionTracker` | In-memory `HashSet`, no persistence | Has `load_or_new()` + `Persistable` impl |
22+
| `poll_mentions()` | Same in both: per-issue loop, fetch ALL comments, spawn per unprocessed mention | Same |
23+
| Concurrency limit | **None** | **None** |
24+
| Cursor | **None** — refetches all comments every poll | **None** |
25+
26+
#### Data flow
27+
28+
```
29+
reconcile_tick() [every 30s]
30+
→ poll_mentions()
31+
→ for each issue in watch_issues (16 issues):
32+
fetch_comments(issue, None) ← ALL comments, no cursor
33+
for comment in comments:
34+
parse_mentions(comment)
35+
if not tracker.is_processed(mention):
36+
spawn_agent(mention_def) ← NO concurrency limit
37+
mark_processed(mention)
38+
```
39+
40+
#### Key files
41+
42+
| File | Lines | Role |
43+
|---|---|---|
44+
| `crates/terraphim_orchestrator/src/mention.rs` | ~400 | MentionTracker, parse_mentions, resolve_mention |
45+
| `crates/terraphim_orchestrator/src/lib.rs:1055-1170` | ~115 | poll_mentions() implementation |
46+
| `crates/terraphim_orchestrator/src/config.rs` | ~20 | MentionConfig struct |
47+
| `crates/terraphim_tracker/src/gitea.rs` | ? | fetch_comments() API call |
48+
49+
#### Existing issue
50+
51+
**Issue #186** already has a complete design for the fix:
52+
- Replace per-issue polling with repo-wide cursor-based polling
53+
- Single API call: `GET /repos/{owner}/{repo}/issues/comments?since={cursor}`
54+
- Persistent cursor → no replay on restart
55+
- Eliminates `watch_issues` config entirely
56+
57+
#### Why #186 wasn't implemented
58+
59+
The persistence code for `MentionTracker` was added in commit `1983aa0a` on branch `task/173-automata-orchestrator-integration`, but:
60+
1. It only persists the `processed` HashSet — not a cursor
61+
2. It still fetches ALL comments per issue (no `since` parameter)
62+
3. It has no concurrency limit on spawning
63+
4. The branch was never merged to main
64+
65+
### 3. Constraint Identification
66+
67+
| Constraint | Detail |
68+
|---|---|
69+
| **Gitea API** | `/repos/{owner}/{repo}/issues/comments?since=&limit=50` — confirmed working |
70+
| **terraphim_persistence** | Already available, SQLite profile works on bigbox |
71+
| **Concurrency** | bigbox has 125GB RAM but only 8 cores — 170 node processes will thrash |
72+
| **MentionTracker.processed** | Serialised as HashSet<(u64,u64,String)> — grows unbounded |
73+
| **Backward compat** | Old `watch_issues` config must degrade gracefully |
74+
| **Existing branches** | `task/173-automata-orchestrator-integration` has partial persistence, not merged |
75+
76+
### 4. Risk Assessment
77+
78+
| Risk | Likelihood | Impact | Mitigation |
79+
|---|---|---|---|
80+
| Cursor desync (missed mentions) | Low | Medium | Also keep processed set as safety net |
81+
| Gitea API rate limit | Low | Low | One call per tick (every 60s with poll_modulo=2) |
82+
| Large cursor gap (many new mentions) | Medium | High | Batch-process with max_dispatches_per_tick |
83+
| Orphaned agents from storm | Already happened | High | Add max_concurrent_mentions config |
84+
85+
### 5. Assumptions
86+
87+
1. Gitea API `since` parameter uses `created_at` timestamp (confirmed via curl)
88+
2. Comments are returned in chronological order
89+
3. A single repo-level call is cheaper than N per-issue calls
90+
4. `terraphim_persistence` DeviceStorage is initialised before `poll_mentions` runs
91+
92+
### 6. Unknowns
93+
94+
1. What happens if two mentions for the same agent arrive in one batch? (Should deduplicate by agent, only spawn one)
95+
2. Should Safety agents (security-sentinel, drift-detector) be exempt from mention concurrency limits?
96+
3. Is the `processed` HashSet in the old persistence unbounded? (Yes — needs TTL or size cap)
97+
98+
---
99+
100+
## Recommendation
101+
102+
Implement Issue #186 design with these additions:
103+
104+
1. **Cursor-based polling** (from #186) — single API call with `since=` parameter
105+
2. **Persistent cursor** via `terraphim_persistence` — survives restarts
106+
3. **`max_dispatches_per_tick`** (default: 3) — bounds concurrency per poll cycle
107+
4. **`max_concurrent_mention_agents`** (default: 5) — total concurrent mention-spawned agents
108+
5. **Startup guard** — on first run (no cursor), set cursor to `now` to skip backlog
109+
110+
This replaces Issue #249 (which I created prematurely — should be closed in favour of #186).

0 commit comments

Comments
 (0)