v2.0.0 — Modernization: loop-first runtime, Verification 2.0, producer skills, enforcing gates#130
Merged
Merged
Conversation
…tration runtime, automation hooks Phase 0 — Foundation gates: - check-surface-sync.py: marker-driven enumerative-table-row gate (skills/agents tables) - .githooks/pre-commit + install-hooks.sh: real pre-commit quality gate (closes the direct-commit bypass) - pin model:/effort: across all 17 inherit agents per model-routing.md (9 Opus / 8 Sonnet) Phase 1 — Verification 2.0 (ground truth is a process): - audit-reproducibility: EXPLAINED-with-named-alternative disposition (fail-closed; hard floor preserved) - "manuscript is not the oracle" disagreement-as-signal guidance - claim-verifier: soften numeric/directional auto-HIGH-WARN with named-alternative escape (fabricated-citation floor intact) - two-strikes reuse for repeatedly-EXPLAINED claims; model-diversity challenger guardrail - verify-claims: citation-appropriateness scope; validate-bib stays existence-only and points here Phase 2 — Real orchestration runtime: - orchestration-schemas.md + agent-fleet.md (references; count-safe) - orchestrator-protocol.md rewritten from "pattern, not a runtime" into fan-out -> reduce -> judge(+hallucination gate) -> loop-until-dry + RUN_CONFIG - seven-pass/review-paper/qa-quarto/slide-excellence/deep-audit wired to the shared contract Phase 3 — Enforcing automation (hooks count held at 6): - log-reminder.py: auto-WRITES the session log instead of nagging (+ compile-completion note) - git-guardrails.py (PreToolUse): blocks reset --hard / clean -f / push --force / add -A / mass-discard; warns on hardcoded paths - retire verify-reminder.py; pre-compact DRAFT-block default ON; enriched statusline; .github/workflows CI All gates green: surface-sync (26 count + 3 table-row), skill-integrity, model-versions.
…(8 economist skills) New producer skills (skills 38→46), drafted to AEA/DCAS gold standard: - /replication-package — submission-ready DCAS/openICPSR deposit (README, manifest, env, Table→script:line map); gates on /audit-reproducibility - /capture-environment — renv.lock/requirements/uv.lock + Stata ado list + optional Dockerfile + seeds - /did-event-study — THIN wrapper over canonical packages (Callaway-Sant'Anna did, Sun-Abraham, HonestDiD; Stata twins); never reimplements an estimator - /power-analysis — RCT/cluster/multi-arm/simulation power + MDE; wired into /preregister - /disclosure-check — SDL pre-screen for restricted-data outputs (small cells, dominance, PII); gates on CRITICAL - /grant-proposal, /data-management-plan — compose confidential-data + capture-environment + DMP primitives - /coauthor-brief — multi-machine/collaborator handoff brief Plus: - .claude/rules/confidential-data.md (rules 28→29) — restricted-data protocol + multi-author git topology - journal-profiles.md: AEA Data Editor / DCAS policy note (all AEA-imprint journals) - preregister wired to /power-analysis for RCT sample plans Count-sync: skills 38→46 + rules 28→29 across README/CLAUDE/guide/docs/skill-template; +8 rows in both skill tables. All gates green (surface-sync, skill-integrity, model-versions).
…irst onboarding + retirements Phase 5 — background/scheduled/event-driven autonomy: - claim-reconcile.py (PostToolUse): event-driven — flags stale manuscript claims the moment an analysis script / _outputs changes (hooks 6→7) - references/scheduled-routines.md: nightly reproducibility, weekly lit-delta, monthly memory-promotion, daily inbox triage (Routines, push-on-failure) - scripts/nightly-repro-check.sh: cron-able passport staleness check - /triage-inbox skill: schedulable Gmail/Calendar triage + referee-obligations tracker (human-gated, never auto-sends) Phase 6 — teaching-at-scale (the repo's origin, previously a blind spot): - /syllabus, /teach-from-paper, /respond-to-eval, /scaffold-exercises (problem sets w/ solutions, imported from mattpocock/skills) Phase 7 — loop-first onboarding, output-styles, scaffolder, retirements: - README: "Goal-first, gate-enforced" reframe; fixed the stale "no pre-commit hook" claim; loop-until-dry - output-styles/: academic-writing.md + referee.md (repo shipped zero before) - /new-skill scaffolder (imported from mattpocock write-a-skill) — gate-passing frontmatter first try - RETIRE /prompt + /prompt-only → demoted to .claude/rules/prompt-shaping.md (ambient habit, not a command); prompt-formatting-core re-pointed Net inventory: skills 46→50 (+6 new, −2 retired), rules 29→30, hooks 6→7, agents 18. Count-sync across README/CLAUDE/guide/docs/skill-template; all gates green.
…dme template (exploration log, not a replication README)
…g stale) - CHANGELOG: v2.0.0 entry led by a "⭐ Why upgrade — read this even if you've used the template before" before→after table for returning forkers, full Added/Changed/Removed/Upgrading sections - guide/workflow-guide.qmd (+ re-rendered HTML → docs/): orchestrator "pattern" → real runtime; loop-until-dry; "no pre-commit hook" claim fixed; log-reminder now "auto-writes" not "nags"; hooks table adds git-guardrails + claim-reconcile, drops verify-reminder; model-routing "inherit default" → pinned; /prompt + /prompt-only marked retired - docs/index.html (landing): goal-first/gate-enforced + auditor + not-a-daemon framing; runtime fan-out→reduce→judge; 50/30/7 counts; "no /prompt command" - README: rules table +prompt-shaping (always-on) +confidential-data (path-scoped); version pin → v2.0.0 - v2.0-backlog.md: moved /syllabus, /teach-from-paper, /power-analysis, /replication-package, /respond-to-eval, /triage-inbox (+8 more) to "Shipped in v2.0.0"; marketplace re-affirmed as documented non-goal All gates green; comprehensive stale-term scan clean (verify-reminder, "pattern not a runtime", "no pre-commit", "model: inherit", "5 rounds max" → 0 hits).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5fe6968697
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+32
to
+34
| echo "── pre-commit: surface-sync gates ──" | ||
| ./scripts/check-surface-sync.sh | ||
| [ $? -ne 0 ] && FAIL=1 |
There was a problem hiding this comment.
When committing a partially staged change, this runs check-surface-sync.sh against the working tree rather than the index that will actually be committed. For example, if stale README counts are staged but the working tree already has the corrected README left unstaged, the hook passes even though the commit records the stale version; the inverse can also block a valid staged commit because of unrelated unstaged edits. The hook claims to gate staged changes, so it needs to either require a clean worktree or run the checks against a temporary checkout of the staged index.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Major v2.0.0 modernization that turns the template into a loop-first, verification-gated “runtime”: stronger enforcement (pre-commit + hooks + CI), expanded verification semantics (EXPLAINED), new “producer” skills (replication package, environment capture, teaching tools, etc.), and updated public surfaces/docs to match the new inventory and workflow.
Changes:
- Added enforcement layers: version-controlled pre-commit hook + install script, guardrail hooks, drift detection (table-row sync), and CI workflows for deterministic gates.
- Upgraded verification semantics across reproducibility and claim-checking (introducing
EXPLAINED) and refreshed orchestration/runtime docs + schemas. - Added a large set of new skills (replication package, environment capture, DiD/event-study wrapper, disclosure checks, teaching-at-scale skills, inbox triage) while retiring
/promptand/prompt-only, and pinned agent model/effort tiers.
Reviewed changes
Copilot reviewed 81 out of 81 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/skill-template.md | Updates stated skill inventory count to match v2.0.0. |
| templates/passport-template.yaml | Adds EXPLAINED to passport status semantics and documents downgrade requirements. |
| templates/archive-readme.md | Adds an abandonment log header clarifying intended usage/scope. |
| scripts/nightly-repro-check.sh | Adds local cron-able stale-claim detector against passports. |
| scripts/install-hooks.sh | Adds installer to enable repo-controlled git hooks via core.hooksPath. |
| scripts/check-surface-sync.py | Extends surface sync gate to enforce enumerative table row counts via markers. |
| docs/index.html | Updates landing page copy/counts to reflect loop-first, gate-enforced v2.0.0. |
| CLAUDE.md | Updates enforcement explanation and skills table (adds table marker; adds/removes commands). |
| CHANGELOG.md | Adds v2.0.0 release notes and migration/upgrading guidance. |
| .github/workflows/gates.yml | Adds CI workflow to run deterministic gate suite on PRs/branches. |
| .github/workflows/deploy.yml | Adds workflow to render guide to docs/ on main when guide sources change. |
| .githooks/pre-commit | Adds version-controlled pre-commit gate runner (surface-sync + quality scoring). |
| .claude/skills/verify-claims/SKILL.md | Clarifies complementarity with /validate-bib and adds EXPLAINED tier semantics. |
| .claude/skills/validate-bib/SKILL.md | Clarifies scope (existence/structure) and defers appropriateness to /verify-claims. |
| .claude/skills/triage-inbox/SKILL.md | New skill spec for human-gated Gmail/Calendar triage + referee obligations tracker. |
| .claude/skills/teach-from-paper/SKILL.md | New skill spec to turn a paper into teachable outline + slide skeleton + exercises brief. |
| .claude/skills/syllabus/SKILL.md | New skill spec to generate a structured syllabus + week→lecture work-list. |
| .claude/skills/slide-excellence/SKILL.md | Updates synthesis step to reduce typed findings/scorecards rather than rereading prose. |
| .claude/skills/seven-pass-review/SKILL.md | Moves synthesis toward reduce→judge with hallucination gate, using shared schemas. |
| .claude/skills/scaffold-exercises/SKILL.md | New skill spec to generate problem sets and separate solution keys. |
| .claude/skills/review-paper/SKILL.md | Updates iteration logic to loop-until-dry and adds hallucination-gate language. |
| .claude/skills/respond-to-eval/SKILL.md | New skill spec to turn course evals into a verified improvement plan. |
| .claude/skills/replication-package/SKILL.md | New skill spec to assemble DCAS/openICPSR-style replication package gated by reproducibility audit. |
| .claude/skills/qa-quarto/SKILL.md | Updates iteration logic to loop-until-dry and clarifies gate predicate. |
| .claude/skills/prompt/SKILL.md | Removes deprecated /prompt skill (replaced by ambient prompt-shaping rule). |
| .claude/skills/prompt-only/SKILL.md | Removes deprecated /prompt-only skill (replaced by ambient prompt-shaping rule). |
| .claude/skills/preregister/SKILL.md | Adds cross-reference guidance to /power-analysis for numeric sample plans. |
| .claude/skills/power-analysis/SKILL.md | New skill spec for analytical/simulation power, scripts, and prereg-ready paragraph. |
| .claude/skills/new-skill/SKILL.md | New skill spec to scaffold convention-compliant skills that pass integrity gates. |
| .claude/skills/disclosure-check/SKILL.md | New skill spec for SDL pre-screen of restricted-data outputs with gating semantics. |
| .claude/skills/did-event-study/SKILL.md | New thin-wrapper DiD/event-study skill spec using canonical packages + diagnostics. |
| .claude/skills/deep-audit/SKILL.md | Updates loop phase to loop-until-dry wording and references two-strikes escalation. |
| .claude/skills/data-management-plan/SKILL.md | New skill spec to draft funder-compliant DMPs integrating confidentiality + reproducibility. |
| .claude/skills/coauthor-brief/SKILL.md | New skill spec for collaborator handoff briefs with reproducibility + restricted-data steps. |
| .claude/skills/capture-environment/SKILL.md | New skill spec for environment capture (lockfiles, seeds, optional Dockerfile) with verification. |
| .claude/skills/audit-reproducibility/SKILL.md | Adds PASS/FAIL/EXPLAINED semantics and downgrade rules; updates report schema. |
| .claude/settings.json | Adds PreToolUse guardrails hook and swaps verify-reminder for claim reconciliation. |
| .claude/scripts/statusline.sh | Enriches statusline with branch/dirty/plan/context% badges. |
| .claude/rules/summary-parity.md | Generalizes “two-strikes” escalation pattern across workflows. |
| .claude/rules/replication-protocol.md | Adds EXPLAINED semantics and clarifies mismatch framing (code vs paper). |
| .claude/rules/quality-gates.md | Updates enforcement docs to reflect optional real pre-commit gate. |
| .claude/rules/prompt-shaping.md | Adds ambient prompt-shaping rule replacing /prompt skills. |
| .claude/rules/orchestrator-protocol.md | Rewrites orchestrator from “pattern” to runtime primitives + hallucination gate + loop-until-dry. |
| .claude/rules/model-routing.md | Adds corollary guardrail about challenger vs auditor tier separation in future designs. |
| .claude/rules/confidential-data.md | Adds restricted-data handling protocol and links to disclosure check and collaboration topology. |
| .claude/references/v2.0-backlog.md | Updates backlog to reflect shipped v2.0.0 items and remaining deferrals. |
| .claude/references/scheduled-routines.md | Adds scheduled routines reference and links to local nightly drift script and event-driven hook. |
| .claude/references/prompt-formatting-core.md | Re-points reference from retired /prompt skills to prompt-shaping rule. |
| .claude/references/orchestration-schemas.md | Adds shared FINDING/SCORECARD/RUN_CONFIG schema and hallucination gate contract. |
| .claude/references/journal-profiles.md | Adds AEA DCAS/Data Editor policy note and links to replication tooling. |
| .claude/references/agent-fleet.md | Adds an explicit agent fleet manifest with model tiers and dispatch mapping. |
| .claude/output-styles/referee.md | Adds output style for referee-report tone/structure. |
| .claude/output-styles/academic-writing.md | Adds output style for publication-grade academic prose discipline. |
| .claude/hooks/verify-reminder.py | Removes deprecated verify-reminder hook implementation. |
| .claude/hooks/pre-compact.py | Changes default to block once on DRAFT plans before compaction (opt-out). |
| .claude/hooks/git-guardrails.py | Adds PreToolUse guardrails to block destructive git ops and warn/deny path leaks. |
| .claude/hooks/context-monitor.py | Persists context usage estimate for statusline consumption. |
| .claude/hooks/claim-reconcile.py | Adds PostToolUse hook to flag possibly-stale passport claims when scripts/outputs change. |
| .claude/agents/verifier.md | Pins agent model/effort (removes inherit). |
| .claude/agents/tikz-reviewer.md | Pins agent model/effort (removes inherit). |
| .claude/agents/slide-auditor.md | Pins agent model/effort (removes inherit). |
| .claude/agents/sim-reviewer.md | Pins agent model/effort (removes inherit). |
| .claude/agents/r-reviewer.md | Pins agent model/effort (removes inherit). |
| .claude/agents/r-package-reviewer.md | Pins agent model/effort (removes inherit). |
| .claude/agents/quarto-fixer.md | Pins agent model/effort (removes inherit). |
| .claude/agents/quarto-critic.md | Pins agent model/effort (removes inherit). |
| .claude/agents/proofreader.md | Pins agent model/effort (removes inherit). |
| .claude/agents/pedagogy-reviewer.md | Pins agent model/effort (removes inherit). |
| .claude/agents/methods-referee.md | Pins agent model/effort (removes inherit). |
| .claude/agents/humanize-auditor.md | Pins agent model/effort (removes inherit). |
| .claude/agents/editor.md | Pins agent model/effort (removes inherit). |
| .claude/agents/domain-reviewer.md | Pins agent model/effort (removes inherit). |
| .claude/agents/domain-referee.md | Pins agent model/effort (removes inherit). |
| .claude/agents/claim-verifier.md | Pins agent model/effort and adds author_alternative/EXPLAINED semantics. |
| .claude/agents/beamer-translator.md | Pins agent model/effort (removes inherit). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+66
to
+69
| ctx="" | ||
| hash="$(printf '%s' "$cwd" | python3 -c 'import sys,hashlib; print(hashlib.md5(sys.stdin.read().encode()).hexdigest()[:8])' 2>/dev/null)" | ||
| pct_file="$HOME/.claude/sessions/${hash}/context-pct.txt" | ||
| [ -f "$pct_file" ] && ctx="ctx $(cat "$pct_file" 2>/dev/null)%" |
Comment on lines
+92
to
+93
| content = ti.get("content") or ti.get("new_string") or "" | ||
| if isinstance(content, str): |
Comment on lines
+24
to
+26
| pdir = root / "quality_reports" / "passports" | ||
| if not pdir.is_dir(): | ||
| print("no passports/ — nothing to check"); sys.exit(0) |
- statusline.sh: hash CLAUDE_PROJECT_DIR / git-toplevel (not raw cwd) so the context-% badge resolves the same sessions folder context-monitor.py writes to — fixes the badge vanishing in subdirectories (Copilot) - git-guardrails.py: scan MultiEdit's edits[*].new_string list, not just content/new_string — closes the path-leak miss on MultiEdit (Copilot) - nightly-repro-check.sh: print the real path (quality_reports/passports/) in the empty-state message (Copilot) - .githooks/pre-commit: stash unstaged *tracked* changes (--keep-index, trap-restored) so the gate validates the STAGED index, not the working tree — fixes partial-staging false-pass/false-block (Codex P2). Untracked files left in place; stash failure never blocks.
…entions rule; adversarial-hardening defect fixes DiD flagship (grounded ONLY in Pedro Sant'Anna's PUBLIC materials — packages, JEL guide, What's-Trending, did-resources; vault kept private): - Rewrote /did-event-study into a thin orchestrator over the canonical packages (did/DRDID/didFF/contdid; Stata csdid/drdid) — never reimplements an estimator. 8-phase practitioner workflow with exact API (att_gt/aggte/ggdid, drdid estMethod="imp", contdid), DR default, control-group rule, uniform-band inference, mandatory pre-trend/HonestDiD/didFF diagnostics, replicate-and-verify-to-1e-6, and a GRADED credibility verdict (actor→critic, after the Netflix causal-inference pattern). - New rule did-conventions.md (rules 30→31) encoding his HARD standards + the pitfalls he warns against (no TWFE pre-test, no already-treated controls, no over-reading pre-trends, functional-form matters, debug-the-wrapper-first). - Opinionated defaults flagged [CONFIRM-PEDRO] inline (TWFE framing, HonestDiD Mbar vs M, staggered-pkg default, control-group default, few-cluster software, public URLs) — nothing opinionated ships unconfirmed. Adversarial-hardening fixes (from the find→verify→synthesize audit; 13 confirmed, 28% noise dropped): - CRITICAL .githooks/pre-commit: the stash --keep-index/pop round-trip could write conflict markers into a file with BOTH staged+unstaged changes (git add -p). Now skips the stash on same-file overlap (validates worktree + warns), keeps it for different-files, and makes pop loud+recoverable. - git-guardrails: --force(?![\w-]) so --force-with-lease (the safe alternative) is ALLOWED; clean guard now catches --force / -d -f / later-token -f. - claim-reconcile: match/throttle on project-relative path, not basename. - validate-setup.sh: warn if .githooks/pre-commit lost its exec bit or core.hooksPath isn't set (the gate going silently dark). - power-analysis: Bonferroni divisor = family size (m=K-1 vs K(K-1)/2). guide: orchestrator ~93 lines, +prompt-shaping in always-on tree, 26 path-scoped, abstract line-count framing. All gates green (50 skills / 18 agents / 31 rules / 7 hooks); guide re-rendered.
…estDiD with relative-magnitudes Mbar (also M), staggered as option/att_gt default, TWFE benchmark-only; clear all CONFIRM-PEDRO flags
…ONFIRM flags remain)
…bug found by running it) Validated against Pedro's DiD_book Card-Krueger to his 1e-6 standard: - PRINCIPLE PASSES: DRDID reproduces his canonical 2x2 DiD (2.913982357…) to ~1e-14. - REAL BUG: the skill prescribed DRDID(panel=TRUE, idname=id), which ERRORS on Card-Krueger — an unbalanced panel (409 stores, 389 in both waves) with 2 duplicate (id,wave) rows. Fixes: - Phase 3: pre-flight idname-unique-by-period + balance check; full-sample 2x2 via panel=FALSE + row-unique id (matches feols ~1e-10); balancing → panel=TRUE flagged as a DIFFERENT estimand (2.914 full-sample vs 2.972 balanced — recorded as a named alternative, the EXPLAINED pattern). - did-conventions: idname-unique-by-period + check-balance-before-panel-mode now HARD. - quality_reports/did_validation_card_krueger.md: the verification record. Not yet validated: staggered att_gt path + HonestDiD/didFF sensitivity (packages not installed).
…ated HonestDiD recipe End-to-end validation of the staggered path (did::mpdta, the canonical CS example) with his defaults: - att_gt (notyettreated/dr/universal/bootstrap+cband) → Overall ATT -0.0323 (his default); nevertreated -0.0328 matches the did vignette ~-0.031; event study is the canonical pattern; ggdid plots. - HonestDiD relative-magnitudes sensitivity runs via the direct path; didFF p=0.998. 2nd real bug fixed: honest_did() is a NON-exported internal S3 method in HonestDiD 0.2.8 (bare call errors). Skill now ships the validated direct recipe (createSensitivityResults_relativeMagnitudes with betahat + IF-based sigma from aggte(dynamic)). Verdict: pipeline validated end-to-end on real data (2x2 to 1e-14 + staggered+sensitivity), driving his packages, 2 real bugs found-by-running and fixed. Open: contdid path + a strict R↔Stata 1e-6 staggered cross-check.
…+ contdid path PASS - R↔Stata dual-software (his DiD_book standard): csdid asinr (Stata-MP) vs did::att_gt (R) on mpdta agree to 4.65e-08 on point estimates AND 4.11e-08 on analytic SEs; overall ATT identical. PASS at 1e-6. - contdid continuous-treatment: installed (needs npiv); reproduces his README set.seed(1234) example exactly (Overall ACRT 0.1341 == documented 0.1341). Verdict: /did-event-study validated across EVERY prescribed path (2x2, staggered, sensitivity, dual-software, continuous) against his packages + reference outputs. 2 real bugs found-by-running and fixed. Renamed report did_validation.md.
…erse Per Pedro: his R packages (did/DRDID/didFF/contdid) are the canonical implementations. - Skill + did-conventions: cross-language verification reframed — Stata (csdid/drdid via asinr = "as in R") and Python ports MUST reproduce R to 1e-6 (point + analytic SE). The asinr option exists precisely because R is the reference. - Kept the distinct case explicit: replicating a *published paper* matches THAT paper's original author code (often Stata) — separate from benchmarking his estimators. - Validation report reframed: "Stata-matches-R" (R is the benchmark), not a symmetric cross-check.
…ross-refs, inference-robustness rule From the adversarial-hardening audit's nice-to-have list (now applied): - CTX-1: CLAUDE.md 169→125 lines (back under its ~150 every-session budget) — the 50-row skills table replaced by a compact categorized list + README pointer; the CLAUDE.md surface-sync-table marker removed (README remains the gated complete index; gate now checks 2 table markers). - LINK-1: scheduled-routines.md /schedule dead links → code spans (/schedule is host-provided, not a repo skill). - DOC-1: CONTRIBUTING + PR template now document install-hooks.sh + the surface-sync-table row requirement and check-surface-sync.sh gate. - WIRE-1/2: onward cross-references added — create-lecture → translate-to-quarto/qa-quarto/deploy/scaffold-exercises; review-paper + audit-reproducibility → replication-package/capture-environment/disclosure-check. - SUBST-1: new inference-robustness.md rule (rules 31→32) — multiple-testing (FWER/Romano-Wolf vs FDR/Anderson sharpened-q, pre-register the family) + specification-curve/leave-one-out/wild-cluster-bootstrap robustness; cross-linked from review-paper/preregister/power-analysis/did-conventions. Count-sync: rules 31→32, guide path-scoped 26→27; +README rules row. All gates green; guide re-rendered.
…survived → ~12 distinct) CRITICAL: - pre-commit gate was INVERTED: quality_score.py exits 2 on auto-fail (compile/syntax FAILURE) and 1 on score<80; the hook treated rc>=2 as a non-blocking "infra error", so an uncompilable deck sailed through while a 79/100 blocked. Now ANY nonzero rc blocks. HIGH: - git-guardrails bypassed by `git -C <dir>`/`git -c k=v` global options (patterns required git+subcommand adjacent) → added a global-options prefix; also catches `git add -- .` / `git add :/`. Verified: all bypass forms now DENY, safe forms (force-with-lease, commit, -C status) ALLOW. MEDIUM: - pre-compact + log-reminder classified plan status by WHOLE-FILE substring (APPROVED checked before DRAFT) → a DRAFT plan mentioning "APPROVED"/"COMPLETED" (a checklist item or the status legend) was mislabelled. Now parse the Status FIELD via regex. - pre-compact DRAFT-block default reverted ON→OFF (opt-in): default-ON blocked the harness's AUTOMATIC compaction, which can strand a user at the context ceiling (post-compact-restore re-injects the plan anyway). - statusline ctx-% path now mirrors context-monitor's CLAUDE_PROJECT_DIR-unset fallback (sessions/default/) exactly, instead of hashing the git toplevel. - nightly-repro: date-only last_verified_on now treated as END-of-day, so a same-day file edit isn't flagged a false STALE drift. Deferred (cleanup/altitude, no incorrect output): duplicated get_session_dir/passport helpers, redundant statusline subprocess spawns, the documented working-tree-on-partial-stage trade-off.
…dard coverage audit GOLD-STANDARD AUDIT (multi-agent, 63 agents / 6 coverage finders + adversarial verify / 5 coherence finders + verify / synthesis + completeness critic): - Verdict: gold standard for DiD specifically + a strong general scaffold, but NOT yet for empirical econ in general on the METHODS axis (IV, RD, matching/PS, panel-CRE, RCT analysis are thin/absent). Coherence side is healthy — work is additive, not subtractive. - Full report + standing roadmap: quality_reports/audits/2026-06-09_gold-standard-coverage-audit.md - Methods build-out DEFERRED by decision. When built (critic-adopted order): Step-0 shared quasi-experimental rule → /iv-2sls → /matching-weighting (ahead of RDD) → /rdd-analysis → /panel-models → /rct-analysis as a first-class P0. New flagships are COMMUNITY-STANDARD (orchestrate rdrobust/ivreg/MatchIt, validate against those packages) — only DiD carries the 1e-6 "his-name-on-it" bar. COHERENCE FIXES (the cheap, confirmed cleanup): - CHANGELOG count drift: v2.0 inventory line said "30 rules / +2 rules" → 32 / +4 (applied summary-parity.md: re-verified every claim in the paragraph vs disk, not a surgical swap). - deep-audit/SKILL.md:87 "rules/guide" read as a literal path → reworded to the concrete targets (.claude/rules/*.md and the guide). - Inspected but intentionally NOT changed (gate-green non-issues): r-package-check missing Write (it saves via Bash), argument-hint on 2 no-arg skills (optional, no house convention), seven-pass-review positioning note (already present at line 15). NEW SKILL — /diagnose (the one net-new idea from a fresh pass over mattpocock/skills; the other transferables — /new-skill, /scaffold-exercises, git-guardrails — were already taken in v2.0, and the rest are TS-specific or already covered better here): - Root-cause a wrong/failing empirical result: reproduce → minimise → hypothesise → instrument → fix, tuned for research-code bugs (type coercion, NA/merge blow-ups, clustering/SE, seed/version drift). Competing-hypothesis fan-out via Task; --no-fix. - Full count-surface wiring (skills 50→51): README count + skills table row, CLAUDE.md quick-ref, guide source + rendered HTML, landing page, skill template, CHANGELOG. - Gates green: surface-sync (26 assertions + 2 table rows), check-skill-integrity, model-versions.
guide/workflow-guide.html is the source-side twin of the (gated) docs/workflow-guide.html; the surface-sync gate only checks the docs/ copy, so this in-source render had drifted unstaged. Now matches docs/ (51 skills) — the two are identical but for one deployment- specific line. Picked up by the pre-commit partial-staging stash guard last commit.
… blind spots, fix count drift
Adversarial gold-standard audit (38 agents, 7 dimensions, 24% noise, convention+coherence
came back 100% clean). Verdict was "one focused fixing pass away" — this is that pass.
COUNT DRIFT (must-fix) — 4 stragglers the gate's regexes missed, all → 51:
- guide/workflow-guide.qmd:2391 ("The guide includes 50 skills") — a SOURCE line.
- guide + docs/workflow-guide.html (rendered twins) + the "this template's 50" callout
(docs HTML used a typographic apostrophe ’ that the straight-quote regex never matched).
- .github/CONTRIBUTING.md prose example.
SURFACE-SYNC GATE HARDENED so this drift class can't recur (was the root cause):
- this template['’]s — now matches the curly apostrophe Quarto emits in rendered HTML.
- "guide includes N skills" — new regex for the prose form that matched none of the gate's
patterns. Gate now validates 29 count assertions (was 26); regexes proven to bite.
/diagnose DEEPENED to gold-standard (skill-depth + functional + safety findings):
- Phase 3 gains the 3 high-frequency silent-bug classes a top econometrician expects —
numerical stability/convergence (singular Hessian, VIF, optimizer flag), weighting/
aggregation, bad-controls/colliders. Phase 3b adds a hypothesis-reduce + confidence gate.
- Phase 4 names concrete diagnostic primitives (str/summary, before/after counts, table(),
cor()/VIF, weight diagnostics). Phase 5 adds a per-bug-class prevention-guard table AND a
pre-edit confidence gate (the anti-laundering rule — no fix on an unproven hypothesis).
- Adds a WORKED EXAMPLE (no sibling ships without code fences — this was the depth gap), a
cross-language R/Stata/Python note (multi-discipline accessibility), a sharpened
/audit-reproducibility boundary, tolerance-by-source heuristic, and a fuller exit table.
RECIPROCAL DISCOVERABILITY (altitude) — /diagnose was link-out-only; now findable from the
failures it resolves: audit-reproducibility (FAIL → hand off) and replication-protocol.md
("If Mismatch" → localize which step) point back to it.
FORK-LEAK / PRIVACY (critic, missed by synthesis): quality_reports/audits/ + the new
diagnoses/ are now gitignored with .gitkeep (consistent with plans/specs/merges) — diagnoses
will hold real data values, and the repo-specific coverage audit is noise for a forker. The
committed audit report is untracked (kept local + in history); reconciled its 50-skills
snapshot + the false "/learn is a template skill" finding. /diagnose now mkdir -p's its dir.
Gates: surface-sync (29 + 2 table rows), check-skill-integrity, model-versions — all green.
Round-2 adversarial audit caught a real arithmetic contradiction in the worked example: '3 duplicate rows per id' implies ~4x inflation (12,400 -> ~49,600), but the example showed 12,400 -> 12,933. Reframed to 'duplicate covars rows for a subset of ids -> +533 rows', which is internally consistent and matches the displayed merge output. 'many-to-many merge' kept (correct for a panel: many periods per id x duplicated covars per id). Round 2 is now dry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A paradigm-shift major release: from a prompt-craft contractor (craft a prompt → invoke a skill → read a report) to a verification-gated research lab — you state a goal, a fleet of specialist agents does the labor under gates that enforce themselves, and you act as the auditor of the disagreements they surface. Converges "loops, not prompts" with "ground truth is a process, not a dataset." Modernizes orchestration, not substance (passport, simulation contract, referees untouched).
Inventory: 50 skills · 18 agents · 30 rules · 7 hooks (was 38/18/28/6). Full notes in CHANGELOG.md — which now leads with a "⭐ Why upgrade — read this even if you've used the template before" before→after table.
Highlights by phase
.githooks/pre-commit+install-hooks.sh(closes the direct-git commitbypass) · all 17inheritagents pinned to model+effort.EXPLAINED-with-named-alternative (a defensible mismatch is recorded, not auto-failed) · "manuscript isn't the oracle" · softenedclaim-verifier· citation-appropriateness.orchestrator-protocolrewritten from "pattern, not a runtime" → fan-out → reduce → judge (+ hallucination gate) → loop-until-dry; neworchestration-schemas+agent-fleetreferences.log-remindernow auto-writes the session log ·git-guardrails(PreToolUse) ·claim-reconcile(event-driven) · enriched statusline · CI./replication-package(AEA DCAS/openICPSR) ·/capture-environment·/did-event-study(thin wrapper, no reimplementation) ·/power-analysis·/disclosure-check·/grant-proposal·/data-management-plan·/coauthor-brief+confidential-datarule.nightly-repro-check.sh·/triage-inbox./syllabus·/teach-from-paper·/respond-to-eval·/scaffold-exercises.academic-writing+refereeoutput-styles ·/new-skillscaffolder · retired/prompt+/prompt-only→prompt-shapingrule; retiredverify-reminder.Documented non-goals (discipline, not omissions)
No autonomous daemon · no plugin marketplace · one
/did-event-study(not an estimator fleet) · no challenger→auditor→ledger pipeline (zero-costEXPLAINEDinstead) ·MEMORY.mdstays the committed memory backend.Test plan
./scripts/check-surface-sync.shgreen — 26 count + 3 table-row assertions; skill-integrity; model-versions (50/18/30/7).check-skill-integrity(flag + allowed-tools parity).git-guardrailsblocks reset --hard / clean -f / push --force / add -A while allowing normal push/commit.docs/; comprehensive stale-term scan clean (verify-reminder, "pattern not a runtime", "no pre-commit", "model: inherit", "5 rounds max" → 0 hits)../scripts/install-hooks.shto activate the pre-commit gate.🤖 Generated with Claude Code