|
| 1 | +// SPDX-License-Identifier: CC-BY-SA-4.0 |
| 2 | +// Copyright (c) Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk> |
| 3 | += Autonomous Bug-Reporting Pipeline — Cross-Repo Architecture |
| 4 | +Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk> |
| 5 | +:toc: |
| 6 | +:icons: font |
| 7 | +:source-highlighter: rouge |
| 8 | + |
| 9 | +[IMPORTANT] |
| 10 | +==== |
| 11 | +*Status: DRAFT / TARGET architecture.* This document describes a pipeline that spans |
| 12 | +three repositories (`feedback-o-tron`, `boj-server`, `boj-server-cartridges`). Most of it |
| 13 | +is *not yet built* — see <<reality>>. It exists to *define* the project that the |
| 14 | +2026-06-28 issue triplet (boj-server #262/#263/#264) failed to capture, and to make the |
| 15 | +built-vs-proposed boundary explicit and honest. Nothing here should be read as a |
| 16 | +description of shipping behaviour except where a row says `BUILT`. |
| 17 | +==== |
| 18 | + |
| 19 | +toc::[] |
| 20 | + |
| 21 | +== Provenance |
| 22 | + |
| 23 | +* *2026-06-28 15:35–15:39* — three issues filed in `boj-server` (#262 "[Deps]", #263 "[Perf]", |
| 24 | + #264 "[Feature]"). They are one set ("Issue 1 / Issue 2 / Issue 3") but each landed in the |
| 25 | + wrong issue template with a broken title. They describe *runtime plumbing* (containerisation, |
| 26 | + caching, tracing) for the pipeline below — not the pipeline itself. |
| 27 | +* *2026-06-28 16:44* — an architecture diagram was drawn (WhatsApp image). It is the real |
| 28 | + deliverable and the source of truth for this document. |
| 29 | +* *2026-07-01* — this document written to consolidate the diagram and to record the honest |
| 30 | + gap between it and the code, so the project can be tracked. Canonical location: this file. |
| 31 | + The tracking issue lives in `boj-server` (per owner decision: doc here, tracker there). |
| 32 | + |
| 33 | +== 1. Purpose |
| 34 | + |
| 35 | +An AI agent (or a system) hits a crash. Instead of emitting a cryptic stack trace, the |
| 36 | +pipeline turns that event into a *well-formed, deduplicated, schema-correct bug report* filed |
| 37 | +to the right forge — autonomously. The design intent is that the report is shaped by the |
| 38 | +*target repository's own issue template*, hydrated with real context, verified at an ABI |
| 39 | +boundary, and dispatched to GitHub / GitLab / Bugzilla. |
| 40 | + |
| 41 | +The 2026-06-28 diagram, transcribed: |
| 42 | + |
| 43 | +[source,text] |
| 44 | +---- |
| 45 | +Agent / System Crash Trigger |
| 46 | + │ Initiate Bug Report |
| 47 | + ▼ |
| 48 | +boj-server Gateway |
| 49 | + │ ① Fetch Template Schema |
| 50 | + ▼ |
| 51 | +.github/ISSUE_TEMPLATE/bug.yml |
| 52 | + │ ② Pass Schema + System State |
| 53 | + ▼ |
| 54 | +feedback-o-tron Synthesis Engine |
| 55 | + ├── Hydrate Questions |
| 56 | + └── Environment Questions |
| 57 | + ▼ |
| 58 | +┌─ Autonomous Fulfilment & Verification ────────────────┐ |
| 59 | +│ LLM Context: Trajectory & Logs │ |
| 60 | +│ NetworkVerifier / OS Inspection │ |
| 61 | +│ └── Deduplicator: Fuzzy Match Existing Issues │ |
| 62 | +└────────────────────────────────────────────────────────┘ |
| 63 | + │ ③ Clean Validated Payload |
| 64 | + ▼ |
| 65 | +Idris2 Verified ABI Layer |
| 66 | + │ ④ Secure Multi-Platform Dispatch |
| 67 | + ▼ |
| 68 | +GitHub / GitLab / Bugzilla API |
| 69 | +---- |
| 70 | + |
| 71 | +== 2. Component roles (per-repo split) |
| 72 | + |
| 73 | +[cols="1,1,3",options="header"] |
| 74 | +|=== |
| 75 | +| Component | Repo | Intended responsibility |
| 76 | + |
| 77 | +| *Gateway* |
| 78 | +| `boj-server` |
| 79 | +| Entry point. Fetch the target repo's issue-template schema (`.github/ISSUE_TEMPLATE/bug.yml`), |
| 80 | + gather system state, and hand both to the Synthesis Engine as a boj cartridge invocation. |
| 81 | + |
| 82 | +| *Synthesis Engine* |
| 83 | +| `feedback-o-tron` |
| 84 | +| Hydrate the template's questions ("Hydrate" + "Environment") from LLM trajectory/logs and |
| 85 | + from `NetworkVerifier` / OS inspection; deduplicate against existing issues; emit a clean, |
| 86 | + schema-valid payload. |
| 87 | + |
| 88 | +| *Verified ABI Layer* |
| 89 | +| `feedback-o-tron` (+ `proven` types) |
| 90 | +| Validate the payload's memory layout and shape at a formally-specified Idris2/Zig FFI |
| 91 | + boundary before it leaves the process. |
| 92 | + |
| 93 | +| *Dispatch* |
| 94 | +| `feedback-o-tron` channels |
| 95 | +| File the validated payload to GitHub / GitLab / Bugzilla (and other forges). |
| 96 | + |
| 97 | +| *Cartridge packaging* |
| 98 | +| `boj-server-cartridges` / `boj-server/cartridges` |
| 99 | +| Package the Synthesis Engine as a first-class boj cartridge so the Gateway can invoke it. |
| 100 | +|=== |
| 101 | + |
| 102 | +== 3. The four wire contracts |
| 103 | + |
| 104 | +[cols="1,3",options="header"] |
| 105 | +|=== |
| 106 | +| ID | Contract |
| 107 | + |
| 108 | +| *C1* | `Fetch Template Schema` — Gateway → `.github/ISSUE_TEMPLATE/bug.yml` |
| 109 | +| *C2* | `Pass Schema + System State` — (schema + state) → Synthesis Engine |
| 110 | +| *C3* | `Clean Validated Payload` — Synthesis + Dedup → Idris2 Verified ABI Layer |
| 111 | +| *C4* | `Secure Multi-Platform Dispatch` — Idris2 ABI → GitHub / GitLab / Bugzilla API |
| 112 | +|=== |
| 113 | + |
| 114 | +[[reality]] |
| 115 | +== 4. Current reality vs target (verified 2026-07-01) |
| 116 | + |
| 117 | +Ground-truthed against the code in both local repos. Statuses: *BUILT* (real working code), |
| 118 | +*PARTIAL* (real code, but unwired / bypassed / inert), *PROPOSED* (docs/diagram only, no code). |
| 119 | + |
| 120 | +[cols="1,1,3",options="header"] |
| 121 | +|=== |
| 122 | +| Contract / element | Status | Evidence |
| 123 | + |
| 124 | +| *C1* Fetch Template Schema |
| 125 | +| PROPOSED |
| 126 | +| `boj-server` Gateway (`elixir/lib/boj_rest/router.ex`) exposes only 5 cartridge endpoints |
| 127 | + (`/health`, `/menu`, `/cartridges`, `/cartridge/:name`, `POST /cartridge/:name/invoke`). |
| 128 | + No code fetches an issue-template schema. There is *no* `bug.yml` anywhere — both repos ship |
| 129 | + plain Markdown templates (`bug_report.md`), not `.yml` issue forms. |
| 130 | + |
| 131 | +| *C2* Pass Schema + System State |
| 132 | +| PROPOSED |
| 133 | +| There is no "Synthesis Engine" (grep `synthesis` = 0 hits). `feedback-o-tron` has *no HTTP |
| 134 | + intake* (no Bandit/Plug/Cowboy in the supervision tree); its only entry point is the MCP tool |
| 135 | + `submit_feedback`, whose `input_schema` requires `[title, body, repo]` — a plain issue, not a |
| 136 | + schema + system-state. The one cross-repo wire (`cartridges/feedback-mcp/mod.js` → POST |
| 137 | + `http://127.0.0.1:7722`) targets a backend `feedback-o-tron` does not serve. |
| 138 | + |
| 139 | +| *C3* Clean Validated Payload → Idris2 ABI |
| 140 | +| PROPOSED |
| 141 | +| `feedback-o-tron` has *zero* `.idr`/`.ipkg` files — all `src/abi/*.idr` were removed |
| 142 | + (`PROOF-NEEDS.md` L27–31: "only RSR template scaffolding, no proofs"). `ffi/zig/src/main.zig` |
| 143 | + references phantom `src/abi/*.idr` and its `feedback_o_tron_process` is a stub |
| 144 | + (`_ = input; return .ok`). The Elixir core never calls any NIF/FFI. The `Deduplicator` |
| 145 | + algorithm is real (`deduplicator.ex`, normalized Levenshtein — comment mislabels it |
| 146 | + "Jaro-Winkler"), but `Deduplicator.record/3` is *never called*, so its indexes stay empty and |
| 147 | + `check/1` always returns `{:ok, :unique}` at runtime — the dedup is *inert as integrated*. |
| 148 | + |
| 149 | +| *C4* Secure Multi-Platform Dispatch |
| 150 | +| PARTIAL |
| 151 | +| Dispatch is *real*: `channels/github.ex` shells `gh issue create`; `channels/gitlab.ex` shells |
| 152 | + `glab issue create`; `channels/bugzilla.ex` does a real REST v1 POST; 14+ channels registered; |
| 153 | + README documents 2 real Bugzilla bugs filed (#2437503/#2437504). *But* it runs entirely in |
| 154 | + Elixir and does *not* originate from any Idris2 ABI — the "via Idris2 ABI" source in the |
| 155 | + diagram is fictional. |
| 156 | + |
| 157 | +| MCP server + `submit_feedback` tool |
| 158 | +| BUILT |
| 159 | +| Working JSON-RPC MCP server over stdio (`mcp/server.ex`), exposes `submit_feedback` + |
| 160 | + `migration_observe`. The real front door. |
| 161 | + |
| 162 | +| Audit logging |
| 163 | +| BUILT |
| 164 | +| `audit_log.ex` — JSON-lines trail with rotation + secret-sanitisation, wired from `Submitter`. |
| 165 | + |
| 166 | +| NetworkVerifier |
| 167 | +| PARTIAL |
| 168 | +| 607 lines of real probing (ping/traceroute/dig/openssl/crt.sh/RPKI), supervised |
| 169 | + (`application.ex:29`) — but *never called* by `Submitter` or any MCP tool. A disconnected |
| 170 | + island; does network probing, not OS/environment inspection for a payload. |
| 171 | + |
| 172 | +| Migration-observation subsystem |
| 173 | +| PARTIAL |
| 174 | +| Real but off by default; observes ReScript *code*-migrations, not LLM trajectories. Likely the |
| 175 | + origin of the diagram's "gather logs" box, but it does not do what the diagram claims. |
| 176 | +|=== |
| 177 | + |
| 178 | +*Net:* the terminal dispatch (C4) is real; the template-fetch (C1), synthesis/schema-passing (C2), |
| 179 | +and verified-ABI (C3) stages that give the diagram its shape are unbuilt. |
| 180 | + |
| 181 | +[[identity]] |
| 182 | +== 5. OPEN DECISION — the `feedback-o-tron` vs `feedback-mcp` identity problem |
| 183 | + |
| 184 | +The single most under-defined thing in this project. Today these are *two unrelated programs |
| 185 | +that share a name*: |
| 186 | + |
| 187 | +[cols="1,2,2",options="header"] |
| 188 | +|=== |
| 189 | +| | `feedback-o-tron` (this repo) | `feedback-mcp` (boj cartridge) |
| 190 | + |
| 191 | +| What it is | v1.0.0 Elixir bug-*filing* engine | ~448-line Zig FFI *sentiment counter* |
| 192 | +| Tools | `submit_feedback`, `migration_observe` | `register_channel`, `start_collecting`, `submit`, `get_stats` |
| 193 | +| Data model | issue → 15+ forge channels | 8-slot in-memory tally, fork-per-request (no persistence) |
| 194 | +| Runtime | MCP stdio / escript; shells `gh`/`glab`, REST | `mod.js` → HTTP `127.0.0.1:7722` |
| 195 | +| Shared code | — | *none* — separate reimplementation, name only |
| 196 | +|=== |
| 197 | + |
| 198 | +`EXPLAINME.adoc` says "feedback-o-tron is a cartridge itself" and the cartridge `README.adoc` |
| 199 | +claims it "implements feedback-o-tron / is the self-dogfooding loop." *Both are false in code.* |
| 200 | +The cartridge is even internally stale (README lists 10 tools; only 4 exist; `cartridge.json` |
| 201 | +declares an `so_path` that isn't built). |
| 202 | + |
| 203 | +*Decision required (D0):* choose one — |
| 204 | + |
| 205 | +. *Converge* — make `feedback-mcp` the cartridge packaging of the real `feedback-o-tron` engine |
| 206 | + (cartridge invokes the engine; retire the sentiment-counter or rename it). This is what the |
| 207 | + diagram assumes ("feedback-o-tron Synthesis Engine" reachable via the Gateway). |
| 208 | +. *Separate* — accept they are different products; rename one to end the collision |
| 209 | + (e.g. cartridge → `sentiment-mcp`), and define a *new* first-class cartridge that wraps the engine. |
| 210 | + |
| 211 | +The rest of this architecture assumes *Converge*; if *Separate* is chosen, C2's cartridge wire |
| 212 | +must be redrawn to a new cartridge name. |
| 213 | + |
| 214 | +== 6. Prerequisites & gaps to close (ordered) |
| 215 | + |
| 216 | +. *Resolve D0* (<<identity>>) — nothing downstream is well-defined until the identity is fixed. |
| 217 | +. *Author `bug.yml` issue forms* — the diagram's C1 assumes structured `.yml` issue forms; |
| 218 | + today only Markdown templates exist. No schema = nothing to fetch or hydrate. |
| 219 | +. *Give `feedback-o-tron` a real intake* — either an HTTP backend on the port the cartridge |
| 220 | + already targets (`127.0.0.1:7722`) or a boj-cartridge invocation path, accepting |
| 221 | + `{template_schema, system_state}` — not just `{title, body, repo}`. |
| 222 | +. *Build the Synthesis/Hydration step* — a module that maps template fields → hydrated answers |
| 223 | + from LLM trajectory/logs + `NetworkVerifier`/OS inspection. |
| 224 | +. *Wire the Deduplicator* — call `Deduplicator.record/3` so dedup is not inert. |
| 225 | +. *Decide the ABI story* — either build a real Idris2/Zig verified boundary for the payload, or |
| 226 | + *drop the "Idris-inside" claim*. Do not ship a badge with no code. |
| 227 | + |
| 228 | +== 7. Honesty debt to retire (independent of the build) |
| 229 | + |
| 230 | +These are current false-impression items surfaced during verification; fixing them is cheap and |
| 231 | +should not wait for the pipeline: |
| 232 | + |
| 233 | +* `README.adoc` badges "Idris-inside" / "Production Ready" — no `.idr` files exist in the repo. |
| 234 | +* `TOPOLOGY.md` asserts "100% / v1.0.0 Production Ready" for every box, while |
| 235 | + `elixir-mcp/ARCHITECTURE.md` admits much is "aspirational/future." Reconcile to reality. |
| 236 | +* `Deduplicator.record/3` never called → dedup inert. Either wire it or note it as disabled. |
| 237 | +* `NetworkVerifier` supervised but never invoked → note as a standalone tool, not a pipeline stage. |
| 238 | +* `ABI-FFI-README.md` is still an unfilled `{{PROJECT}}` RSR template. |
| 239 | +* `deduplicator.ex` comment says "Jaro-Winkler"; the code computes normalized Levenshtein. |
| 240 | + |
| 241 | +== 8. Relationship to the issue tracker |
| 242 | + |
| 243 | +* *Umbrella / tracking issue* (in `boj-server`) — captures this whole pipeline and links every |
| 244 | + sub-issue. This is the "fan-in" the triplet lacked. |
| 245 | +* *Re-filed sub-issues* — #262/#263/#264 re-filed cleanly as *feature* issues (they are all |
| 246 | + features, not a Deps advisory / a Perf report / an empty Feature): |
| 247 | + ** OCI containerisation & immutable runtime isolation (was #262) |
| 248 | + ** Offline pre-fetch & stale-while-revalidate cartridge caching (was #263) |
| 249 | + ** OpenTelemetry tracing across the boj ↔ ABI ↔ cartridge boundary (was #264) |
| 250 | +* *Missing core-pipeline issues* — the actual "main effort," never filed: C1 template-fetch, |
| 251 | + C2 synthesis/hydration, C3 verified-ABI payload, plus D0 (the identity decision). These are |
| 252 | + the issues that make the pipeline exist; the triplet only hardens the box it would run in. |
0 commit comments