chore(roadmap): truth-sync specs vs code, archive cold epics, browsable DAG, gardener automation#823
Merged
Merged
Conversation
Every specs/<id>/tasks.md was audited against the code it claims to
describe and re-ticked from the evidence (the symbol/route/test/import
must exist and do what the task says). 36 files, 773 flips: 747 tasks
ticked, 26 un-ticked. Repo-wide checkbox coverage 55% -> 81%.
Eight specs were showing a false `drafted`/`0%` badge while shipped:
001-code-execution (0/127 -> 74/127), 042-telemetry-tier2 (0 -> 60/91),
028-agent-tokens (0 -> 36/43), 009, 016, 017, 040, 044-*, 050, 055, 057,
073, 074.
The 26 un-ticks are the interesting half - tasks marked [x] whose
artifact never existed:
- 001-oas T049/T051/T053/T055 each carried an inline "COMPLETE: added
at line 64" annotation, yet docs/oas-coverage-report.md, the
verify-oas workflow, the CI step and the README section are all
absent. Annotated, not built.
- 001-oas T013-T015/T025-T027: no swag annotations on the secrets and
SSE handlers.
- 029 T020: the server-edition Linux matrix rows are commented out in
release.yml.
- 019 T046: ActivityWidget.vue exists but is never imported into
Dashboard.vue ("add X and wire it into Y" is not satisfied by X).
- 011 T033, 014 T046/T047/T054, 026 T104/T108/T109, 056 T019, 059 T025.
Unchecked tasks that are documented scope-outs were left alone. The diff
is checkbox-only: task text is byte-identical on both sides.
specs/README.md's "systematically stale" warning is rewritten to match:
badges are broadly trustworthy again, but remain a lagging signal.
044-diagnostics-taxonomy (59/106) drew the most auditor/verifier
disagreement and is worth a human pass.
roadmap.yaml grew monotonically because `done` epics never left it. It is
now the WORKING SET; cold shipped epics are swept into a new
roadmap.archive.yaml.
gen-roadmap.py --archive [--min-age-days N] [--dry-run]
An epic is swept when it is `done`, every child task is `done`, every PR
it references is MERGED, and the newest of those merges is at least
--min-age-days old (default 14, "shipped and two weeks cold"). `keep:
true` pins an epic in the working set forever.
Blocks are moved as raw YAML text rather than re-serialised, so notes,
inline comments and formatting survive byte-for-byte (a PyYAML
round-trip would have destroyed the header and the section banners). Two
fields are stamped on the way out: shipped_on, archived_on.
DAG integrity: depends_on now resolves against BOTH files. An edge into
the archive is satisfied by definition (archived == done) and is not
drawn; an id that exists in neither file is a new hard ERROR in
--check-github, which previously never validated depends_on at all.
First sweep: profiles-v2 and ts-code-exec-ga, both merged 2026-06-24.
The three v0.47.0 epics stay put - they are 7-11 days old, and "what
just shipped" is context people use.
ROADMAP.md rendering: the single `graph TD` held ~70 nodes (every epic
plus every task, in subgraphs) and rendered as an illegible postage
stamp on GitHub. Split by zoom level instead:
- "Roadmap at a glance": a `graph LR` of only the epics that actually
have a cross-epic edge. Drawing all of them produced an 896x2238
column of disconnected boxes - a list pretending to be a graph - so
dependency-free epics are listed as text beneath it.
- Per-epic <details> blocks, collapsed by default, each with the epic's
note, spec/PR links, a mini task DAG and a task table.
- A "Shipped (archived)" table for provenance.
Verified in a browser against a locally-served render: the overview is
896x606 with 16px labels, all mermaid blocks render with zero syntax
errors, no horizontal overflow, every <details> closed by default.
Also drops the Assignee column from the Epics table.
--archive is idempotent, --check stays byte-idempotent, and the swept
blocks are identical to their pre-sweep source apart from the two stamps.
Spec 080's two headline funnel numbers are measurement artifacts - the
same failure class the spec exists to fix. Both were re-measured against
the live telemetry D1 and corrected in the spec overview and in the
roadmap notes that quote them.
1. "17.7% return on a second day / 75.5% one-and-done" was computed on
raw anonymous_id, which churns per process and fakes one-day
installs (the +/-37% uncertainty the spec itself flags two
paragraphs later). Re-derived with the dashboard's machine-identity
dedup: returned >= once 52.0%, one-and-done 48.0%, day-1 retention
31.8%, day-7 16.7%, WAU-in-week-1 46.6% - matching the dashboard
retention page (52.2/47.9/31.0/16.6/46.6). The identical query on
raw anonymous_id reproduces the artifact (74.2% / 18.4%).
2. "41.7% retrieve_tools -> 16.4% real tool call" compares incomparable
metrics: the retrieve step uses the lifetime
activation.first_retrieve_tools_call_ever flag, while the real-call
step has no lifetime flag and was measured from windowed per-day v2
counters. Measured symmetrically (60d, n=1405 v2-capable installs):
retrieve ever 460, real call ever 464, both 412 - about 90% of
retrievers convert. There is no conversion cliff.
Retention is still the biggest leak - half of installs never return - but
it is roughly half as severe as stated, and no "retrieve-to-call
conversion" work is warranted.
New roadmap items:
- tpa-db epic (P1): versioned, offline-updatable signature database of
known Tool-Poisoning-Attack campaigns for the deterministic scanner,
guarded by the existing scan-eval recall/FP gate.
- action-log-tray-menu: surface recent activity in the tray menu.
- tray-menu-open-telemetry: menu opens are invisible today. Since Spec
048 menuWillOpen rebuilds from SSE state with zero REST calls, and
surface_requests['tray'] counts header-stamped REST calls, not opens.
- telemetry-v7-realcall-flag: add first_real_tool_call_ever, symmetric
to first_retrieve_tools_call_ever, so that funnel step becomes
measurable lifetime-vs-lifetime. This is the root cause of (2).
Epic reconciliation, driven by the tasks.md truth-sync:
- analytics-token-drain-graphs -> done, analytics-dashboard ->
in_progress: spec 069 is shipped (25/26; the one open task is a
Playwright sweep, not a feature).
- registries-search-ux and registries-search-add -> done: spec 070 is
21/24 and the three open tasks are pre-PR chores. Its stale
depends_on [ux-audit] is dropped - a done epic cannot depend on a
todo one.
- windows-tray: removed `spec: specs/002-windows-installer`. That is
the INSTALLER spec; this epic is the native TRAY, so the badge
described unrelated work.
- action-log-glance-view and action-log-retention-tie-in: spec links
removed. 019 and 073 are the shipped backends and painted these
unbuilt UX tasks ~99% green.
gen-roadmap.py --check-github reports 0 errors.
Deploying mcpproxy-docs with
|
| Latest commit: |
e0986de
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://0077857b.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://chore-roadmap-truth-sync.mcpproxy-docs.pages.dev |
Nothing consumes it. The Assignee column left the rendered Epics table with the DAG rework, and every remaining value was either `unassigned` or a stale role name (`BackendEngineer`) that never mapped to a person - exactly the sort of metadata that rots and then misleads. Removed from roadmap.yaml, roadmap.archive.yaml and the schema comment. Ownership, when it matters, lives on the linked PR or issue. Verified field-by-field: the only difference from the previous tree is the removed key; ROADMAP.md is byte-identical.
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 29075075400 --repo smart-mcp-proxy/mcpproxy-go
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Self-review of the raw-text block mover found two latent defects and one durability gap. None is currently triggered by roadmap.yaml, but each would corrupt or lose an epic the first time someone wrote ordinary YAML. 1. `- id: alpha # legacy` was invisible to EPIC_START_RE, which anchored on end-of-line. The epic would be silently absorbed into its predecessor's block and swept along with it. The regex now tolerates a trailing comment and a quoted id. 2. The sweep cuts text by line number while trusting a regex to find epic boundaries. Added `assert_blocks_match_yaml`: the ids the text scan finds must equal, in order, the ids PyYAML parses, or --archive refuses to touch the file. Cheap, total invariant. (It fires, for example, on an id containing '#', where the regex and the YAML scanner genuinely disagree about where the value ends.) 3. Writes were non-atomic and unordered: a crash between writing roadmap.yaml and roadmap.archive.yaml would LOSE the swept epics. Each file is now written via temp-file + rename, and the archive is written FIRST, so an interrupted sweep duplicates an epic (recoverable, and the next --archive is a no-op for it) rather than deleting it. Verified: trailing-comment and quoted ids parse; the guard rejects a regex/YAML disagreement and an oddly-spaced epic line; both real roadmap files pass it; --check stays byte-idempotent and --check-github is clean.
…l review
A Codex review of the archive sweep found eight defects; all eight reproduced
against the real script, and two were merge-blocking.
DATA LOSS (high). Duplicate epic ids silently dropped an epic. The sweep keys
raw text blocks by id, so two epics sharing an id collapse to one dict entry
while BOTH line ranges are cut from the source. The id-set safety check
compared sets, so it passed. YAML permits this because epics are list items,
not mapping keys, and nothing else enforced the schema's uniqueness rule.
--archive now refuses to run on duplicate ids, and --check-github reports them
as errors (across roadmap.yaml AND the archive).
UNVERIFIED MERGES (medium, and inverted). `--min-age-days 0` was documented as
waiving the cool-off, but it also skipped `gh` entirely — so without `gh` an
epic whose PR was still OPEN would be archived as shipped, while an epic with
no PR refs at all was skipped. Exactly backwards. `gh` is now required to prove
merge state; `--min-age-days 0` waives only the age threshold. The new
--force-undated opt-in sweeps on `done` status alone, warns loudly, and stamps
shipped_on: unknown.
Also fixed:
- archive_append_target(): the raw-text append assumed the file ends inside
the epics list. `epics: []`, `epics: null` or a trailing comment would have
produced a syntax error or swallowed the block. Validated up front.
- check_cycles(): depends_on was only checked for existence, so `a -> b -> a`
and self-loops rendered as a "DAG" describing a chain that can never start.
- node_id() collisions: `a-b` and `a_b` both sanitise to `a_b`, merging two
Mermaid nodes and their edges. Now an error.
- Injection: titles and notes went raw into <summary> and Markdown tables. A
title containing `</summary>` or a `|` broke the page. Added md_cell() and
html_text(); node_box() now neutralises quotes and brackets too.
- stamp_block() emitted LF stamps into a CRLF block, violating the
"raw text preserved" claim. It now matches the block's own line ending.
- SECTION_HDR_RE matched any indent-2 comment starting with a box glyph, so a
hand-written ` # ─ postmortem` could be treated as a section banner. It now
matches only the generated ` # ── TITLE` shape.
ROADMAP.md changes on exactly two lines: the two epic titles containing
a literal `&` now render as `&` inside their raw-HTML <summary>,
which is what that context requires. Nothing else needed escaping.
Dumbris
force-pushed
the
chore/roadmap-truth-sync
branch
from
July 10, 2026 06:37
80f7347 to
1a9b14b
Compare
A `- [x]` in tasks.md is an unverified assertion. The truth-sync found 26
tasks ticked whose artifact was never built — several carrying a
hand-written "**COMPLETE: added at line 64**" beside a file that does not
exist. This is the deterministic half of catching that: no model is
asked anything, it just extracts the paths and symbols a task cites and
checks whether they are in the tree.
The hard part is false positives, not detection. speckit tasks are
written before the code, so they cite a PLANNED path. Measured here: of
the 51 distinct paths cited by ticked tasks that do not exist, 50 NEVER
EXISTED in git history — `cmd/mcpproxy/doctor.go` was implemented as
`doctor_cmd.go`. A naive "does the path exist" gate is ~98% noise and
would be switched off within a week.
So a missing path is classified:
RELOCATED a same-stem sibling in the same directory (high) or a
repo-unique basename under the same top dir (medium).
Informational. Never satisfied by a `_test` file when the
task cited an implementation, and never by a loose stem
prefix (`serve.go` is not `serveredition_commands.go`).
REMOVED existed in git history, now deleted. Worth a look.
UNRESOLVED no path, no sibling, no history. The real signal.
A cited SYMBOL that exists outranks a path, because symbols survive file
moves; a task citing a findable symbol has its path complaints demoted.
Only distinctive symbols count (a camelCase hump or two underscores):
`error_code` proves nothing, `first_retrieve_tools_call_ever` does.
Current output: 62 unresolved, 1 removed, 25 relocated, 147
possibly-built. Spot-checked: `upgrade_funnel`, `ActivityLogView`,
`useEventStream` and `DiagnosticsDecoder` appear ONLY under specs/ and
nowhere in the source tree, yet their tasks are ticked. Residual noise
remains (a task citing `internal/contracts/config.go` whose types landed
under another name), which is why this reports and never edits.
Unticked tasks whose evidence all resolves are reported as
`possibly-built` — candidates for review, never auto-ticked.
Report-only by default (exit 0). `--strict` fails on unresolved,
`--json` feeds the roadmap-gardener routine. Pure stdlib, read-only.
hygiene-spec-evidence-check (done) and hygiene-spec-gardener (in progress) under planning-hygiene, so the automation that keeps the specs honest is itself on the roadmap it maintains. The gardener is a weekly Claude cloud routine, not a CI workflow: it runs on the maintainer's subscription, clones the default branch, and pushes to `claude/spec-gardener` (the `claude/` prefix is what the cloud git proxy permits without enabling unrestricted branch pushes). It no-ops until check-spec-evidence.py reaches main. Division of labour, which is the whole point: the script decides everything decidable and the model is spent only on the residue — does the task's prose intent actually hold. Un-ticks are never auto-applied.
Dumbris
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent pieces of roadmap/spec hygiene, one commit each.
1.
chore(specs)— tasks.md checkboxes vs real codeEvery
specs/<id>/tasks.mdwas audited against the code it claims to describe and re-ticked from the evidence. 773 flips across 36 files: 747 ticked, 26 un-ticked. Repo-wide checkbox coverage 55% → 81%.Eight specs were showing a false
drafted/0%badge while shipped —001-code-execution(0/127 → 74/127),042-telemetry-tier2(0 → 60/91),028-agent-tokens(0 → 36/43), and 009/016/017/040/044-*/050/055/057/073/074.The 26 un-ticks are the interesting half — tasks marked
[x]whose artifact never existed:001-oasT049/T051/T053/T055 each carried an inline**COMPLETE: Added at line 64**annotation, yetdocs/oas-coverage-report.md, the verify-oas workflow, the CI step and the README section are all absent. Annotated, not built.029T020: the server-edition Linux matrix rows are commented out inrelease.yml.019T046:ActivityWidget.vueexists but is never imported intoDashboard.vue— "add X and wire it into Y" is not satisfied by X alone.The diff is checkbox-only: task text is byte-identical on both sides. Unchecked tasks that are documented scope-outs were left alone.
2.
feat(roadmap)— archive cold epics, make the DAG browsableroadmap.yamlgrew monotonically becausedoneepics never left. It is now the working set; cold shipped epics sweep intoroadmap.archive.yamlviagen-roadmap.py --archive(done + all PRs MERGED + newest merge ≥--min-age-days, default 14). Blocks move as raw YAML text, so notes and comments survive byte-for-byte.depends_onnow resolves against both files — an edge into the archive is satisfied by definition; an id in neither is a new hard ERROR in--check-github, which previously never validateddepends_onat all.The DAG was one
graph TDwith ~70 nodes and rendered as an illegible postage stamp on GitHub. Now split by zoom level: agraph LRoverview of only the 9 epics with cross-epic edges (drawing all 24 gave an 896×2238 column of disconnected boxes), dependency-free epics listed as text, and per-epic collapsed<details>blocks with a mini task DAG + table. Verified in a browser: overview 896×606, 16px labels, zero mermaid errors, no horizontal overflow.Also drops the Assignee column from the Epics table.
3.
docs(roadmap,spec-080)— corrected funnel metricsSpec 080's two headline numbers are measurement artifacts — the same failure class the spec exists to fix:
anonymous_id, which churns per process and fakes one-day installs. With machine-identity dedup: returned ≥ once 52.0%, one-and-done 48.0%, day-1 31.8%, day-7 16.7% — matching the dashboard retention page exactly. The same query on rawanonymous_idreproduces the artifact.Retention is still the biggest leak — half of installs never return — but it is roughly half as severe as stated, and no "retrieve-to-call conversion" work is warranted.
New items:
tpa-dbepic (offline TPA signature DB),action-log-tray-menu,tray-menu-open-telemetry(menu opens are invisible today — since Spec 048menuWillOpenmakes zero REST calls), andtelemetry-v7-realcall-flag(first_real_tool_call_ever, the root cause of #2).Epic reconciliation:
analytics-*andregistries-*statuses corrected to shipped reality;windows-tray's wrongspec: 002-windows-installerlink removed (that's the installer, not the tray); two spec links dropped that painted unbuilt UX tasks ~99% green.Checks:
gen-roadmap.py --checkbyte-idempotent ·--check-github0 errors (2 pre-existing warnings aboutwindows-traylacking a PR link) ·--archiveidempotent · pre-commitroadmap-verifyhook green on all three commits.Worth a human pass:
044-diagnostics-taxonomy(59/106) drew the most auditor/verifier disagreement.