Skip to content

Commit d3642da

Browse files
committed
feat(dogfood): seal 22-wave roam-on-roam dogfood loop — 71 surgical fixes across 59 files
Pattern 1B/1C/1D envelope discipline: - cmd_grep / cmd_refs_text / cmd_delete_check: 6 Pattern 1B/1D usage-error envelopes - cmd_diagnose: Pattern 1B isolated-in-graph + N×4 batch-loop hoist (cmd_diagnose.py:288) - cmd_attest: empty-changeset no_changes envelope (no more lying safe_to_merge:true) - cmd_ws: _require_workspace() canonical Pattern-1A envelope across 6 subcommands - cmd_annotate: read-path Pattern 1D resolution (W324 sibling) - cmd_invariants / cmd_laws: usage_error envelope + agent_contract parity - cmd_pr_bundle: emit verdict 156→69 chars (LAW 6 + LAW 4 truncation) - cmd_workflow: 3 next_commands populated (CONSTRAINT 12) - cmd_hotspots / cmd_ingest_trace: no-traces closed-enum + parse-error envelope - cmd_partition / cmd_path_coverage: silent-bucket on unknown filter values Pattern 2 silent-fallback closures: - cmd_health: silent-Healthy 100/100 on empty corpus - cmd_doctor: "all checks passed" on empty corpus - cmd_taint: silent-SAFE on empty corpus (security-critical) - cmd_fan: 3-state closed enum for empty-rows - cmd_runs verify --all: state="ok" when unsigned>0 (now state="unsigned") - cmd_fingerprint: text/JSON god_components 40x divergence Critical detector-correctness fixes: - graph/dark_matter.py: _RE_TABLE matched all Python imports as SQL FROM → 5→1 SHARED_DB (100% FP class fix) - cmd_secrets.py: Generic Bearer regex FP-storm (1-char+ → 20-char min + lookahead) - cmd_orphan_imports.py: pyproject parser + dist→import-name map → 578→21 (96.4% FP drop) - cmd_understand.py: framework FP-storm (gated on _SOURCE_LANGUAGES + path filter) - cmd_minimap.py / cmd_tour.py: test-fixture pollution in rankings/entry-points Critical argv-injection security fixes: - constitution/loader.py: tokenize-before-substitute (--symbol "x; rm -rf ." was injecting 7 argv tokens) - cmd_pr_replay.py: --range "-"-prefix guard + "--" separator - security/taint_rules: added CLI-context sources (sys.argv, click.option, etc.) to detect this class structurally Pattern 3a / Pattern 6 envelope volume: - cmd_fingerprint.py: 2.1MB → 30KB envelope via top-100 cluster truncate - cmd_plan.py: 6048 → 3050 tokens via _trim_signature() decorator stripping - formatter.py: added "cohesion" measurement_suffix + "risks" anchor terminal W361 PageRank truncation sweep (72% of nonzero values were rounding to 0.0): - cmd_partition / cmd_duplicates / cmd_codeowners / cmd_batch_search / cmd_agent_export / cmd_map: 4→6 decimals Complexity reductions: - cmd_health.py: _emit_health_findings 148→6 (-142) via emit-helper extraction - catalog/parallel_hierarchy.py: detect_parallel_hierarchy 141→112 (markers_and_union helper) LAW 4 / LAW 6 / CONSTRAINT 12: - cmd_impact.py: verdict reach_pct .0f→.1f (LAW 6 self-consistency) - cmd_orchestrate / cmd_mutate: docstring example flags (--n-agents → --agents, positional → required) - cmd_adversarial.py: LAW 4 verdict re-anchored on "challenges" terminal - next_steps.py: roam trace SOURCE TARGET (was 1-arg, Click rejected) Empty-catch tightening (Pattern 2 lineage): - cmd_pr_risk / cmd_clean / mcp_extras/completions / search/index_embeddings: typed-narrow handlers Pattern 2 playbook propagation: - index_embeddings.py: ONNX backend ImportError/RuntimeError sentinel - cmd_math --only/--exclude: silent-no-op fix with closest-match warning prefix cmd_invariants / cmd_laws docstring + agent_contract surfaces: - cmd_laws check / list / explain: agent_contract.next_commands parity Detector-pack expansions: - security/taint_rules/python_basic.yaml + python_path_traversal.yaml: 5 CLI-context sources + 2 sinks - formatter.py: added "cohesion" + "risks" to LAW 4 vocabulary (CLAUDE.md count 98→99, 115→116) Dead code: - search/tfidf.py: compute_tfidf_vectors() removed (0 callers) - output/framework_filter.py: filter_framework_rows() + count_filtered() removed (undocumented, 0 callers) - cmd_dead.py: degraded _parse_param_names → import canonical roam._signature_utils.parse_param_names - mcp_server.py: 2 hedge_comments_false_cycle fix - evidence/env_refs.py: false-cycle hedge fix (hoisted _detect_ci_env_id to module top) New tests: - tests/test_graph_layers.py (9 tests): detect_layers, format_layers - tests/test_vuln_store_ingest.py (6 tests): ingest_npm/pip/trivy/osv + W202 operator-precedence regression pin - tests/test_orphan_imports_filters.py (5 tests): W161 pyproject/dist-name/sibling discipline 22 dogfood agents executed; 71 surgical fixes accumulated across 59 modified files. ruff format + ruff check clean. Drift guards + JSON contracts + smoke all pass.
1 parent aad6eff commit d3642da

59 files changed

Lines changed: 2158 additions & 464 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

src/roam/catalog/parallel_hierarchy.py

Lines changed: 32 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,34 @@ def _finding(
170170
)
171171

172172

173+
def _markers_and_union(
174+
subs: list[tuple[int, str, Optional[str], Optional[int]]],
175+
super_name: str,
176+
) -> tuple[list[tuple[str, set[str]]], set[str]]:
177+
"""Build the per-subclass marker sets + their union for one hierarchy.
178+
179+
Each subclass name is tokenised; tokens that overlap with the
180+
superclass name are stripped so the comparison signal lives on the
181+
*variant* portion (``US``/``UK``/``Payroll``) rather than the shared
182+
root (``Employee``). Subclasses with no remaining markers are
183+
dropped — they can't contribute to the Jaccard signal.
184+
185+
The A-side and B-side inner loops in ``detect_parallel_hierarchy``
186+
are byte-identical apart from variable suffixes; this helper is the
187+
extracted common form.
188+
"""
189+
super_toks = _tokenize(super_name)
190+
markers: list[tuple[str, set[str]]] = []
191+
for _sid, name, _p, _l in subs:
192+
toks = _strip_super_token_overlap(_tokenize(name), super_toks)
193+
if toks:
194+
markers.append((name, toks))
195+
union: set[str] = set()
196+
for _n, toks in markers:
197+
union |= toks
198+
return markers, union
199+
200+
173201
def detect_parallel_hierarchy(
174202
conn: sqlite3.Connection,
175203
*,
@@ -242,35 +270,14 @@ def detect_parallel_hierarchy(
242270
eligible.sort(key=lambda t: t[0])
243271
for i in range(len(eligible)):
244272
super_a_id, super_a_name, subs_a = eligible[i]
245-
super_a_toks = _tokenize(super_a_name)
246-
# Build marker sets per subclass (parent tokens stripped).
247-
markers_a: list[tuple[str, set[str]]] = []
248-
for _sid, name, _p, _l in subs_a:
249-
markers = _strip_super_token_overlap(_tokenize(name), super_a_toks)
250-
if markers:
251-
markers_a.append((name, markers))
252-
if len(markers_a) < min_subclasses:
253-
continue
254-
union_a: set[str] = set()
255-
for _n, toks in markers_a:
256-
union_a |= toks
257-
if not union_a:
273+
markers_a, union_a = _markers_and_union(subs_a, super_a_name)
274+
if len(markers_a) < min_subclasses or not union_a:
258275
continue
259276

260277
for j in range(i + 1, len(eligible)):
261278
super_b_id, super_b_name, subs_b = eligible[j]
262-
super_b_toks = _tokenize(super_b_name)
263-
markers_b: list[tuple[str, set[str]]] = []
264-
for _sid, name, _p, _l in subs_b:
265-
markers = _strip_super_token_overlap(_tokenize(name), super_b_toks)
266-
if markers:
267-
markers_b.append((name, markers))
268-
if len(markers_b) < min_subclasses:
269-
continue
270-
union_b: set[str] = set()
271-
for _n, toks in markers_b:
272-
union_b |= toks
273-
if not union_b:
279+
markers_b, union_b = _markers_and_union(subs_b, super_b_name)
280+
if len(markers_b) < min_subclasses or not union_b:
274281
continue
275282

276283
similarity = _jaccard(union_a, union_b)

src/roam/commands/cmd_adversarial.py

Lines changed: 71 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
import click
1818

1919
from roam.capability import roam_capability
20+
from roam.catalog._shared import is_test_path as _is_test_path
2021
from roam.commands.changed_files import get_changed_files, resolve_changed_to_db
2122
from roam.commands.resolve import ensure_index
2223
from roam.db.connection import batched_in, find_project_root, open_db
@@ -268,15 +269,36 @@ def _check_anti_patterns(conn, changed_file_ids, status=None):
268269

269270
changed_fids = set(changed_file_ids)
270271

272+
# W1259 dogfood fix (CHALLENGE 57 HIGH loop-query): the original loop ran
273+
# one ``SELECT file_id FROM symbols WHERE id = ?`` per finding. On
274+
# roam-code itself ``run_detectors`` emits ~7900 findings, producing
275+
# ~7900 SQL round-trips here just to filter to the small set of changed
276+
# files. Pre-fetch the symbol->file_id map for ALL referenced symbols in
277+
# one batched query, then filter in Python.
278+
candidate_sids = {f.get("symbol_id") for f in findings if f.get("symbol_id")}
279+
sym_to_file: dict[int, int] = {}
280+
if candidate_sids:
281+
try:
282+
rows = batched_in(
283+
conn,
284+
"SELECT id, file_id FROM symbols WHERE id IN ({ph})",
285+
list(candidate_sids),
286+
)
287+
sym_to_file = {r["id"]: r["file_id"] for r in rows}
288+
except Exception as exc: # noqa: BLE001
289+
# Lineage: degrade loudly. If the batched lookup fails we
290+
# cannot safely scope findings to changed files, so mark the
291+
# check as errored rather than emit a misleading clean result.
292+
if status is not None:
293+
status["anti_patterns"] = f"errored:symbol_file_lookup:{type(exc).__name__}"
294+
return challenges
295+
271296
for f in findings:
272297
sym_id = f.get("symbol_id")
273298
if not sym_id:
274299
continue
275-
try:
276-
row = conn.execute("SELECT file_id FROM symbols WHERE id = ?", (sym_id,)).fetchone()
277-
except Exception:
278-
continue
279-
if not row or row["file_id"] not in changed_fids:
300+
file_id = sym_to_file.get(sym_id)
301+
if file_id is None or file_id not in changed_fids:
280302
continue
281303

282304
confidence = f.get("confidence", "medium")
@@ -437,9 +459,15 @@ def _check_orphaned_symbols(conn, changed_sym_ids, status=None):
437459
file_path = (sym["file_path"] or "").replace("\\", "/")
438460
name = sym["name"] or ""
439461

440-
# Skip test files and private symbols
441-
is_test = file_path.startswith("test") or "tests/" in file_path or "test/" in file_path or "spec/" in file_path
442-
if is_test:
462+
# W1259 dogfood fix (W907 cargo-cult guard + parity): the original
463+
# ad-hoc check (``startswith("test")`` + ``"tests/" in``) missed
464+
# ``_test.go`` / ``_test.py`` suffix files, ``__tests__/``
465+
# directories, and camelCase ``UserTest.java`` / ``UserSpec.scala``
466+
# / ``UserTests.cs`` basenames — all of which the canonical
467+
# ``roam.catalog._shared.is_test_path`` detects. Delegate to the
468+
# canonical helper so multi-language repos don't see test
469+
# symbols flagged as orphans here.
470+
if _is_test_path(file_path):
443471
continue
444472
if name.startswith("_"):
445473
continue
@@ -758,16 +786,28 @@ def adversarial(ctx, staged, commit_range, severity, fail_on_critical, fmt):
758786
# ------------------------------------------------------------------
759787
# Gather symbol IDs and file IDs for changed files
760788
# ------------------------------------------------------------------
789+
# W1259 dogfood fix (CHALLENGE 71/77/88 silent-swallow at line 769):
790+
# the original loop ran one ``SELECT id FROM symbols WHERE file_id =
791+
# ?`` per changed file and silently swallowed any failure. A SQLite
792+
# error here would leave ``changed_sym_ids`` partial, making every
793+
# downstream check (cycles / layers / cross-cluster / orphaned /
794+
# fan-out) emit degraded results indistinguishable from a clean
795+
# pass — the canonical Pattern-2 silent-fallback hole. Batch the
796+
# lookup into one query AND degrade loudly via ``check_status``
797+
# when it fails.
761798
changed_sym_ids: set[int] = set()
762-
changed_file_ids: set[int] = set()
763-
764-
for path, fid in file_map.items():
765-
changed_file_ids.add(fid)
799+
changed_file_ids: set[int] = set(file_map.values())
800+
sym_lookup_status = "ran"
801+
if changed_file_ids:
766802
try:
767-
syms = conn.execute("SELECT id FROM symbols WHERE file_id = ?", (fid,)).fetchall()
768-
changed_sym_ids.update(s["id"] for s in syms)
769-
except Exception:
770-
pass
803+
rows = batched_in(
804+
conn,
805+
"SELECT id FROM symbols WHERE file_id IN ({ph})",
806+
list(changed_file_ids),
807+
)
808+
changed_sym_ids = {r["id"] for r in rows}
809+
except Exception as exc: # noqa: BLE001
810+
sym_lookup_status = f"errored:symbol_lookup:{type(exc).__name__}"
771811

772812
# ------------------------------------------------------------------
773813
# Run all challenge generators
@@ -777,6 +817,10 @@ def adversarial(ctx, staged, commit_range, severity, fail_on_critical, fmt):
777817
# "changes look clean" when any check errored. Same shape as the
778818
# W832 cmd_critique guard and the X4 cmd_pr_prep guard.
779819
check_status: dict[str, str] = {}
820+
# W1259 dogfood: also surface the changed-symbol lookup status so a
821+
# SQL failure here cannot silently produce empty downstream results.
822+
if sym_lookup_status != "ran":
823+
check_status["symbol_lookup"] = sym_lookup_status
780824
challenges: list[dict] = []
781825
challenges.extend(_check_new_cycles(conn, changed_sym_ids, status=check_status))
782826
challenges.extend(_check_layer_violations(conn, changed_sym_ids, status=check_status))
@@ -810,6 +854,13 @@ def adversarial(ctx, staged, commit_range, severity, fail_on_critical, fmt):
810854
errored_checks = sorted(name for name, s in check_status.items() if s.startswith("errored:"))
811855
partial_success = bool(errored_checks)
812856

857+
# W1259 dogfood fix (LAW 4): the original verdicts ended on
858+
# ``critical`` / ``severity`` / ``info`` — none of which are in
859+
# ``_CONCRETE_NOUN_ANCHORS``. Static lint missed it because
860+
# ``facts = [verdict]`` is a Name reference, not a literal. At
861+
# runtime the verdict (which fact[0] copies) failed LAW 4
862+
# anchoring. Rephrase each verdict to terminate on ``challenges``
863+
# (anchored).
813864
if not challenges:
814865
if partial_success:
815866
verdict = (
@@ -820,13 +871,13 @@ def adversarial(ctx, staged, commit_range, severity, fail_on_critical, fmt):
820871
else:
821872
verdict = "No architectural challenges found -- changes look clean"
822873
elif critical > 0:
823-
verdict = f"{len(challenges)} challenge(s), {critical} critical"
874+
verdict = f"{critical} critical of {len(challenges)} challenges"
824875
elif high > 0:
825-
verdict = f"{len(challenges)} challenge(s), {high} high severity"
876+
verdict = f"{high} high-severity of {len(challenges)} challenges"
826877
elif warning > 0:
827-
verdict = f"{len(challenges)} challenge(s), {warning} warning(s)"
878+
verdict = f"{warning} warning(s) across {len(challenges)} challenges"
828879
else:
829-
verdict = f"{len(challenges)} challenge(s), {info} info"
880+
verdict = f"{info} info-level of {len(challenges)} challenges"
830881
if partial_success and challenges:
831882
# Append partial qualifier so consumers see BOTH the findings
832883
# count AND the cascade. Matches the W832 cmd_critique shape.

src/roam/commands/cmd_agent_export.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,7 +196,11 @@ def _gather_key_files(conn, limit=15):
196196
results.append(
197197
{
198198
"path": r["path"],
199-
"pagerank": round(r["max_pr"] or 0, 4),
199+
# W-dogfood (W336 sibling): 6-decimal precision —
200+
# `max_pr` is the MAX symbol-level PR per file. On
201+
# 5K+ symbol graphs the per-node PR floor is ~1.4e-05,
202+
# so 4-decimal rounding zeroes ~72% of nonzero values.
203+
"pagerank": round(r["max_pr"] or 0, 6),
200204
"fan_in": r["max_in"] or 0,
201205
"symbols": r["sym_count"],
202206
}

src/roam/commands/cmd_annotate.py

Lines changed: 42 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,12 @@ def annotations(ctx, target, tag, since):
184184
conditions = ["(expires_at IS NULL OR expires_at > datetime('now'))"]
185185
params = []
186186

187+
# W324 + dogfood — Pattern 1D: track resolution state for the read
188+
# path so the envelope can distinguish a legitimate "0 annotations
189+
# for a resolved target" from a degraded "target did not resolve, we
190+
# are querying a dangling qualified_name" silent-success. Mirrors the
191+
# write-path disclosure on ``annotate``.
192+
resolution: str | None = None # None when no target was supplied
187193
if target:
188194
# Try symbol resolution
189195
sym = find_symbol(conn, target)
@@ -192,6 +198,7 @@ def annotations(ctx, target, tag, since):
192198
qname = sym["qualified_name"] or sym["name"]
193199
conditions.append("(symbol_id = ? OR qualified_name = ?)")
194200
params.extend([sym_id, qname])
201+
resolution = "symbol"
195202
else:
196203
# Try file path
197204
frow = conn.execute(
@@ -201,10 +208,16 @@ def annotations(ctx, target, tag, since):
201208
if frow:
202209
conditions.append("file_path = ?")
203210
params.append(frow["path"])
211+
resolution = "file"
204212
else:
205-
# Try as qualified_name
213+
# Try as qualified_name. The target did NOT resolve as a
214+
# live symbol or file row; matching by literal qname can
215+
# still hit annotations stored on dangling names from
216+
# prior unresolved writes (relinks on reindex), but the
217+
# caller must be told this is degraded resolution.
206218
conditions.append("qualified_name = ?")
207219
params.append(target)
220+
resolution = "unresolved"
208221

209222
if tag:
210223
conditions.append("tag = ?")
@@ -235,17 +248,39 @@ def annotations(ctx, target, tag, since):
235248
for r in rows
236249
]
237250

251+
# Pattern 1D — silent-success guard. If the caller asked about a
252+
# specific target and we fell through to the dangling-qualified_name
253+
# path, the count we just returned is from the dangling-name shard
254+
# of the table, not from a resolved subject. Mark the envelope
255+
# partial_success and degrade the verdict so agents can tell.
256+
partial = resolution == "unresolved"
257+
if resolution == "unresolved":
258+
# LAW 4 / Pattern 1D: lead with a non-digit subject so the long-
259+
# sentence anchor rule fires, and surface that the count reflects
260+
# only literal qualified_name matches on a target that did NOT
261+
# resolve to a live symbol or file row.
262+
plural = "s" if len(ann_list) != 1 else ""
263+
verdict = f"target did not resolve to any symbol or file: {len(ann_list)} dangling-name annotation{plural}"
264+
else:
265+
verdict = f"{len(ann_list)} annotation{'s' if len(ann_list) != 1 else ''}"
266+
267+
summary: dict = {
268+
"verdict": verdict,
269+
"count": len(ann_list),
270+
"target": target,
271+
"tag_filter": tag,
272+
"partial_success": partial,
273+
}
274+
if resolution is not None:
275+
summary["resolution"] = resolution
276+
238277
if json_mode:
239278
click.echo(
240279
to_json(
241280
json_envelope(
242281
"annotations",
243-
summary={
244-
"verdict": f"{len(ann_list)} annotation{'s' if len(ann_list) != 1 else ''}",
245-
"count": len(ann_list),
246-
"target": target,
247-
"tag_filter": tag,
248-
},
282+
summary=summary,
283+
resolution=resolution,
249284
annotations=ann_list,
250285
)
251286
)

src/roam/commands/cmd_attest.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -682,10 +682,20 @@ def attest(ctx, commit_range, staged, output_format, sign, output_file):
682682

683683
changed = get_changed_files(root, staged=staged, commit_range=commit_range)
684684
if not changed:
685+
# Pattern 1D / Pattern 2: no-changes is a degraded-resolution path,
686+
# NOT a fully-assessed "safe to merge" verdict. An agent reading
687+
# ``summary.safe_to_merge`` must not see ``True`` when the underlying
688+
# check never ran (there was nothing to assess). Disclose state +
689+
# partial_success explicitly so the verdict and the field agree.
685690
label = commit_range or ("staged" if staged else "uncommitted")
686691
_attest_empty_envelope = json_envelope(
687692
"attest",
688-
summary={"verdict": f"no changes found for {label}", "safe_to_merge": True},
693+
summary={
694+
"verdict": f"no changes found for {label}",
695+
"state": "no_changes",
696+
"partial_success": True,
697+
"safe_to_merge": None,
698+
},
689699
)
690700
auto_log(_attest_empty_envelope, action="attest", target=label, repo_root=root)
691701
if json_mode or output_format == "json":

src/roam/commands/cmd_batch_search.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,10 @@ def _run_one(conn, q: str, limit: int, include_paths: bool) -> list[dict]:
7676
"kind": r["kind"],
7777
"file_path": r["file_path"],
7878
"line_start": r["line_start"],
79-
"pagerank": round(float(r["pagerank"] or 0), 4),
79+
# W-dogfood (W336 sibling): 6-decimal precision —
80+
# 4-decimal rounding zeroes ~72% of nonzero PR values on
81+
# 5K+ symbol graphs (per-node PR floor ~1.4e-05).
82+
"pagerank": round(float(r["pagerank"] or 0), 6),
8083
}
8184
for r in rows
8285
]

src/roam/commands/cmd_clean.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
from __future__ import annotations
1313

1414
import os
15+
import sqlite3
1516

1617
import click
1718

@@ -131,7 +132,10 @@ def clean(ctx):
131132
try:
132133
conn.execute("VACUUM")
133134
vacuumed = True
134-
except Exception:
135+
except sqlite3.OperationalError:
136+
# VACUUM can fail on an active transaction or locked DB
137+
# — that's the only legitimate failure mode here.
138+
# Programmer errors (NameError, etc.) propagate per W531.
135139
pass
136140

137141
verdict = (

0 commit comments

Comments
 (0)