Skip to content

Commit 750230e

Browse files
docs: define the autonomous bug-reporting pipeline (cross-repo architecture, built-vs-proposed) (#43)
## What Adds `docs/AUTONOMOUS-BUG-PIPELINE.adoc` — the canonical definition of the **autonomous bug-reporting pipeline** spanning `feedback-o-tron`, `boj-server`, and `boj-server-cartridges`, transcribed from the 2026-06-28 architecture diagram and ground-truthed against the code on 2026-07-01. ## Why On 2026-06-28, boj-server #262/#263/#264 captured pieces of the *runtime* this pipeline would run in (containerisation, caching, tracing) but not the pipeline itself. This doc is the missing definition: the four wire contracts (C1 template-fetch → C2 synthesis → C3 verified-ABI → C4 dispatch), the per-repo role split, and an honest built-vs-proposed table with file:line evidence. Key findings recorded: - Only **C4 dispatch is real** (GitHub/GitLab/Bugzilla), and it bypasses the Idris2 ABI it's drawn behind. - C1/C2/C3 are **PROPOSED** — no template fetch, no Synthesis Engine, zero `.idr` files. - **D0 identity problem**: `feedback-o-tron` (engine) and the boj cartridge `feedback-mcp` (sentiment counter) share a name but no code. - Honesty debt: fictional "Idris-inside" badges, inert Deduplicator (`record/3` never called), unwired NetworkVerifier. ## Status **Draft for owner ratification** — contains the open D0 decision (Converge vs Separate), so per estate policy this PR is intentionally **not armed for auto-merge**. Companion tracker (umbrella epic in boj-server + clean re-file of #262/#263/#264) is prepared and held pending go-ahead. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 15beea0 commit 750230e

1 file changed

Lines changed: 252 additions & 0 deletions

File tree

docs/AUTONOMOUS-BUG-PIPELINE.adoc

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
// SPDX-License-Identifier: CC-BY-SA-4.0
2+
// Copyright (c) Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
3+
= Autonomous Bug-Reporting Pipeline — Cross-Repo Architecture
4+
Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
5+
:toc:
6+
:icons: font
7+
:source-highlighter: rouge
8+
9+
[IMPORTANT]
10+
====
11+
*Status: DRAFT / TARGET architecture.* This document describes a pipeline that spans
12+
three repositories (`feedback-o-tron`, `boj-server`, `boj-server-cartridges`). Most of it
13+
is *not yet built* — see <<reality>>. It exists to *define* the project that the
14+
2026-06-28 issue triplet (boj-server #262/#263/#264) failed to capture, and to make the
15+
built-vs-proposed boundary explicit and honest. Nothing here should be read as a
16+
description of shipping behaviour except where a row says `BUILT`.
17+
====
18+
19+
toc::[]
20+
21+
== Provenance
22+
23+
* *2026-06-28 15:35–15:39* — three issues filed in `boj-server` (#262 "[Deps]", #263 "[Perf]",
24+
#264 "[Feature]"). They are one set ("Issue 1 / Issue 2 / Issue 3") but each landed in the
25+
wrong issue template with a broken title. They describe *runtime plumbing* (containerisation,
26+
caching, tracing) for the pipeline below — not the pipeline itself.
27+
* *2026-06-28 16:44* — an architecture diagram was drawn (WhatsApp image). It is the real
28+
deliverable and the source of truth for this document.
29+
* *2026-07-01* — this document written to consolidate the diagram and to record the honest
30+
gap between it and the code, so the project can be tracked. Canonical location: this file.
31+
The tracking issue lives in `boj-server` (per owner decision: doc here, tracker there).
32+
33+
== 1. Purpose
34+
35+
An AI agent (or a system) hits a crash. Instead of emitting a cryptic stack trace, the
36+
pipeline turns that event into a *well-formed, deduplicated, schema-correct bug report* filed
37+
to the right forge — autonomously. The design intent is that the report is shaped by the
38+
*target repository's own issue template*, hydrated with real context, verified at an ABI
39+
boundary, and dispatched to GitHub / GitLab / Bugzilla.
40+
41+
The 2026-06-28 diagram, transcribed:
42+
43+
[source,text]
44+
----
45+
Agent / System Crash Trigger
46+
│ Initiate Bug Report
47+
48+
boj-server Gateway
49+
│ ① Fetch Template Schema
50+
51+
.github/ISSUE_TEMPLATE/bug.yml
52+
│ ② Pass Schema + System State
53+
54+
feedback-o-tron Synthesis Engine
55+
├── Hydrate Questions
56+
└── Environment Questions
57+
58+
┌─ Autonomous Fulfilment & Verification ────────────────┐
59+
│ LLM Context: Trajectory & Logs │
60+
│ NetworkVerifier / OS Inspection │
61+
│ └── Deduplicator: Fuzzy Match Existing Issues │
62+
└────────────────────────────────────────────────────────┘
63+
│ ③ Clean Validated Payload
64+
65+
Idris2 Verified ABI Layer
66+
│ ④ Secure Multi-Platform Dispatch
67+
68+
GitHub / GitLab / Bugzilla API
69+
----
70+
71+
== 2. Component roles (per-repo split)
72+
73+
[cols="1,1,3",options="header"]
74+
|===
75+
| Component | Repo | Intended responsibility
76+
77+
| *Gateway*
78+
| `boj-server`
79+
| Entry point. Fetch the target repo's issue-template schema (`.github/ISSUE_TEMPLATE/bug.yml`),
80+
gather system state, and hand both to the Synthesis Engine as a boj cartridge invocation.
81+
82+
| *Synthesis Engine*
83+
| `feedback-o-tron`
84+
| Hydrate the template's questions ("Hydrate" + "Environment") from LLM trajectory/logs and
85+
from `NetworkVerifier` / OS inspection; deduplicate against existing issues; emit a clean,
86+
schema-valid payload.
87+
88+
| *Verified ABI Layer*
89+
| `feedback-o-tron` (+ `proven` types)
90+
| Validate the payload's memory layout and shape at a formally-specified Idris2/Zig FFI
91+
boundary before it leaves the process.
92+
93+
| *Dispatch*
94+
| `feedback-o-tron` channels
95+
| File the validated payload to GitHub / GitLab / Bugzilla (and other forges).
96+
97+
| *Cartridge packaging*
98+
| `boj-server-cartridges` / `boj-server/cartridges`
99+
| Package the Synthesis Engine as a first-class boj cartridge so the Gateway can invoke it.
100+
|===
101+
102+
== 3. The four wire contracts
103+
104+
[cols="1,3",options="header"]
105+
|===
106+
| ID | Contract
107+
108+
| *C1* | `Fetch Template Schema` — Gateway → `.github/ISSUE_TEMPLATE/bug.yml`
109+
| *C2* | `Pass Schema + System State` — (schema + state) → Synthesis Engine
110+
| *C3* | `Clean Validated Payload` — Synthesis + Dedup → Idris2 Verified ABI Layer
111+
| *C4* | `Secure Multi-Platform Dispatch` — Idris2 ABI → GitHub / GitLab / Bugzilla API
112+
|===
113+
114+
[[reality]]
115+
== 4. Current reality vs target (verified 2026-07-01)
116+
117+
Ground-truthed against the code in both local repos. Statuses: *BUILT* (real working code),
118+
*PARTIAL* (real code, but unwired / bypassed / inert), *PROPOSED* (docs/diagram only, no code).
119+
120+
[cols="1,1,3",options="header"]
121+
|===
122+
| Contract / element | Status | Evidence
123+
124+
| *C1* Fetch Template Schema
125+
| PROPOSED
126+
| `boj-server` Gateway (`elixir/lib/boj_rest/router.ex`) exposes only 5 cartridge endpoints
127+
(`/health`, `/menu`, `/cartridges`, `/cartridge/:name`, `POST /cartridge/:name/invoke`).
128+
No code fetches an issue-template schema. There is *no* `bug.yml` anywhere — both repos ship
129+
plain Markdown templates (`bug_report.md`), not `.yml` issue forms.
130+
131+
| *C2* Pass Schema + System State
132+
| PROPOSED
133+
| There is no "Synthesis Engine" (grep `synthesis` = 0 hits). `feedback-o-tron` has *no HTTP
134+
intake* (no Bandit/Plug/Cowboy in the supervision tree); its only entry point is the MCP tool
135+
`submit_feedback`, whose `input_schema` requires `[title, body, repo]` — a plain issue, not a
136+
schema + system-state. The one cross-repo wire (`cartridges/feedback-mcp/mod.js` → POST
137+
`http://127.0.0.1:7722`) targets a backend `feedback-o-tron` does not serve.
138+
139+
| *C3* Clean Validated Payload → Idris2 ABI
140+
| PROPOSED
141+
| `feedback-o-tron` has *zero* `.idr`/`.ipkg` files — all `src/abi/*.idr` were removed
142+
(`PROOF-NEEDS.md` L27–31: "only RSR template scaffolding, no proofs"). `ffi/zig/src/main.zig`
143+
references phantom `src/abi/*.idr` and its `feedback_o_tron_process` is a stub
144+
(`_ = input; return .ok`). The Elixir core never calls any NIF/FFI. The `Deduplicator`
145+
algorithm is real (`deduplicator.ex`, normalized Levenshtein — comment mislabels it
146+
"Jaro-Winkler"), but `Deduplicator.record/3` is *never called*, so its indexes stay empty and
147+
`check/1` always returns `{:ok, :unique}` at runtime — the dedup is *inert as integrated*.
148+
149+
| *C4* Secure Multi-Platform Dispatch
150+
| PARTIAL
151+
| Dispatch is *real*: `channels/github.ex` shells `gh issue create`; `channels/gitlab.ex` shells
152+
`glab issue create`; `channels/bugzilla.ex` does a real REST v1 POST; 14+ channels registered;
153+
README documents 2 real Bugzilla bugs filed (#2437503/#2437504). *But* it runs entirely in
154+
Elixir and does *not* originate from any Idris2 ABI — the "via Idris2 ABI" source in the
155+
diagram is fictional.
156+
157+
| MCP server + `submit_feedback` tool
158+
| BUILT
159+
| Working JSON-RPC MCP server over stdio (`mcp/server.ex`), exposes `submit_feedback` +
160+
`migration_observe`. The real front door.
161+
162+
| Audit logging
163+
| BUILT
164+
| `audit_log.ex` — JSON-lines trail with rotation + secret-sanitisation, wired from `Submitter`.
165+
166+
| NetworkVerifier
167+
| PARTIAL
168+
| 607 lines of real probing (ping/traceroute/dig/openssl/crt.sh/RPKI), supervised
169+
(`application.ex:29`) — but *never called* by `Submitter` or any MCP tool. A disconnected
170+
island; does network probing, not OS/environment inspection for a payload.
171+
172+
| Migration-observation subsystem
173+
| PARTIAL
174+
| Real but off by default; observes ReScript *code*-migrations, not LLM trajectories. Likely the
175+
origin of the diagram's "gather logs" box, but it does not do what the diagram claims.
176+
|===
177+
178+
*Net:* the terminal dispatch (C4) is real; the template-fetch (C1), synthesis/schema-passing (C2),
179+
and verified-ABI (C3) stages that give the diagram its shape are unbuilt.
180+
181+
[[identity]]
182+
== 5. OPEN DECISION — the `feedback-o-tron` vs `feedback-mcp` identity problem
183+
184+
The single most under-defined thing in this project. Today these are *two unrelated programs
185+
that share a name*:
186+
187+
[cols="1,2,2",options="header"]
188+
|===
189+
| | `feedback-o-tron` (this repo) | `feedback-mcp` (boj cartridge)
190+
191+
| What it is | v1.0.0 Elixir bug-*filing* engine | ~448-line Zig FFI *sentiment counter*
192+
| Tools | `submit_feedback`, `migration_observe` | `register_channel`, `start_collecting`, `submit`, `get_stats`
193+
| Data model | issue → 15+ forge channels | 8-slot in-memory tally, fork-per-request (no persistence)
194+
| Runtime | MCP stdio / escript; shells `gh`/`glab`, REST | `mod.js` → HTTP `127.0.0.1:7722`
195+
| Shared code | — | *none* — separate reimplementation, name only
196+
|===
197+
198+
`EXPLAINME.adoc` says "feedback-o-tron is a cartridge itself" and the cartridge `README.adoc`
199+
claims it "implements feedback-o-tron / is the self-dogfooding loop." *Both are false in code.*
200+
The cartridge is even internally stale (README lists 10 tools; only 4 exist; `cartridge.json`
201+
declares an `so_path` that isn't built).
202+
203+
*Decision required (D0):* choose one —
204+
205+
. *Converge* — make `feedback-mcp` the cartridge packaging of the real `feedback-o-tron` engine
206+
(cartridge invokes the engine; retire the sentiment-counter or rename it). This is what the
207+
diagram assumes ("feedback-o-tron Synthesis Engine" reachable via the Gateway).
208+
. *Separate* — accept they are different products; rename one to end the collision
209+
(e.g. cartridge → `sentiment-mcp`), and define a *new* first-class cartridge that wraps the engine.
210+
211+
The rest of this architecture assumes *Converge*; if *Separate* is chosen, C2's cartridge wire
212+
must be redrawn to a new cartridge name.
213+
214+
== 6. Prerequisites & gaps to close (ordered)
215+
216+
. *Resolve D0* (<<identity>>) — nothing downstream is well-defined until the identity is fixed.
217+
. *Author `bug.yml` issue forms* — the diagram's C1 assumes structured `.yml` issue forms;
218+
today only Markdown templates exist. No schema = nothing to fetch or hydrate.
219+
. *Give `feedback-o-tron` a real intake* — either an HTTP backend on the port the cartridge
220+
already targets (`127.0.0.1:7722`) or a boj-cartridge invocation path, accepting
221+
`{template_schema, system_state}` — not just `{title, body, repo}`.
222+
. *Build the Synthesis/Hydration step* — a module that maps template fields → hydrated answers
223+
from LLM trajectory/logs + `NetworkVerifier`/OS inspection.
224+
. *Wire the Deduplicator* — call `Deduplicator.record/3` so dedup is not inert.
225+
. *Decide the ABI story* — either build a real Idris2/Zig verified boundary for the payload, or
226+
*drop the "Idris-inside" claim*. Do not ship a badge with no code.
227+
228+
== 7. Honesty debt to retire (independent of the build)
229+
230+
These are current false-impression items surfaced during verification; fixing them is cheap and
231+
should not wait for the pipeline:
232+
233+
* `README.adoc` badges "Idris-inside" / "Production Ready" — no `.idr` files exist in the repo.
234+
* `TOPOLOGY.md` asserts "100% / v1.0.0 Production Ready" for every box, while
235+
`elixir-mcp/ARCHITECTURE.md` admits much is "aspirational/future." Reconcile to reality.
236+
* `Deduplicator.record/3` never called → dedup inert. Either wire it or note it as disabled.
237+
* `NetworkVerifier` supervised but never invoked → note as a standalone tool, not a pipeline stage.
238+
* `ABI-FFI-README.md` is still an unfilled `{{PROJECT}}` RSR template.
239+
* `deduplicator.ex` comment says "Jaro-Winkler"; the code computes normalized Levenshtein.
240+
241+
== 8. Relationship to the issue tracker
242+
243+
* *Umbrella / tracking issue* (in `boj-server`) — captures this whole pipeline and links every
244+
sub-issue. This is the "fan-in" the triplet lacked.
245+
* *Re-filed sub-issues* — #262/#263/#264 re-filed cleanly as *feature* issues (they are all
246+
features, not a Deps advisory / a Perf report / an empty Feature):
247+
** OCI containerisation & immutable runtime isolation (was #262)
248+
** Offline pre-fetch & stale-while-revalidate cartridge caching (was #263)
249+
** OpenTelemetry tracing across the boj ↔ ABI ↔ cartridge boundary (was #264)
250+
* *Missing core-pipeline issues* — the actual "main effort," never filed: C1 template-fetch,
251+
C2 synthesis/hydration, C3 verified-ABI payload, plus D0 (the identity decision). These are
252+
the issues that make the pipeline exist; the triplet only hardens the box it would run in.

0 commit comments

Comments
 (0)