Skip to content

Commit a02b4c7

Browse files
odunbarodunbar
committed
math-auditor skill
1 parent aed68a4 commit a02b4c7

2 files changed

Lines changed: 316 additions & 0 deletions

File tree

‎.claude/skills/math-auditor/SKILL.md‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
---
2+
name: math-auditor
3+
description: >
4+
Run an adversarial mathematical-accuracy review of a Julia package's src/ and
5+
test/ directories, producing a dated markdown report plus concise, self-contained
6+
fix-prompt markdowns suitable for handing to a smaller model (e.g. Sonnet) in a
7+
later session. Use this skill whenever the user asks for an adversarial review,
8+
a math audit, a full code review focused on mathematical/numerical correctness,
9+
a check of algorithmic consistency with the literature, or says things like
10+
"review the math in src/", "is the algebra right?", "audit the update equations",
11+
"check the statistics/linear algebra for bugs", or "construct a code review as
12+
markdown". Trigger even when the user does not say "audit" — any request for a
13+
correctness-focused sweep of a scientific Julia codebase qualifies.
14+
---
15+
16+
# Math Audit
17+
18+
Adversarial review of a scientific Julia package for **mathematical accuracy and
19+
consistency** — not software architecture (flag architecture only when it causes
20+
mathematical wrongness, e.g. mutation aliasing, accidental type demotion, or
21+
inconsistent conventions between modules).
22+
23+
The output is written for the package's own developers: findings must cite exact
24+
`file:line`, state the correct mathematics, and give a concrete failure scenario.
25+
A finding that can't survive an attempt at refutation doesn't ship.
26+
27+
## What "adversarial" means here
28+
29+
Each reviewer's job is to *break* the code, not describe it. Concretely, hunt for:
30+
31+
- **Wrong equations**: emulator predictive mean/covariance formulas, kernel and
32+
feature-map definitions, MCMC proposal/acceptance ratios, likelihood and prior
33+
terms that differ from the cited papers or from the docstring's own LaTeX.
34+
Derive the correct expression independently and diff it against the code.
35+
- **Convention drift**: rows-vs-columns for ensemble members/input points,
36+
`N-1` vs `N` normalization, factor-of-2 / sign errors, Cholesky `L` vs `U`,
37+
covariance vs precision, whitened vs unwhitened space, whether decorrelation
38+
is applied in input- or output-space — especially *inconsistencies between
39+
modules that must agree* (e.g. `GaussianProcess.jl` vs `ScalarRandomFeature.jl`
40+
vs `VectorRandomFeature.jl` on the same emulator interface).
41+
- **Statistical validity**: is observational/model noise sampled and scaled
42+
correctly? Are means/covariances computed over the right dimension (samples
43+
vs output components)? Do scalar and vector emulator variants agree in
44+
expectation? Do the SVD/PCA-based decorrelation and elementwise scaling
45+
transforms in `Utilities/` compose correctly with the emulator and sampler
46+
that consume them?
47+
- **Numerical soundness**: unguarded `inv`/`\` on possibly-singular matrices,
48+
loss of symmetry/PSD-ness, subtraction-based variance formulas, missing
49+
regularization/jitter, `sqrt` of negative-by-roundoff eigenvalues.
50+
- **Edge cases the math must survive**: single training point, single output
51+
dimension (scalar vs matrix degeneracy), zero variance, degenerate/rank-
52+
deficient covariance after decorrelation, empty or singleton minibatches,
53+
MCMC chains that reject every proposal.
54+
- **Test-math consistency**: do the tests actually pin the mathematics
55+
(analytic solutions, invariants, convergence rates), or just check shapes and
56+
"it runs"? A wrong equation whose test only checks `size()` is a *double*
57+
finding: the bug and the missing test.
58+
59+
## Workflow
60+
61+
### 1. Partition
62+
63+
First establish what is actually **live**: grep the top-level module (and the
64+
files it includes, recursively) for `include(...)`, and compare against
65+
`git ls-files src/`. Files never included in the build, or untracked, are dead
66+
code — assign them skim-only status and cap their findings at minor/hygiene
67+
(they can't produce wrong results today, but note them in the report so future
68+
reviewers and greps don't confuse dead sampler math with live code).
69+
70+
List the live `src/*.jl` and `test/**/*.jl` with line counts. Group into 4–8
71+
review units of roughly comparable size, pairing each source module with the
72+
tests that exercise it. Group modules that *share mathematical conventions*
73+
together so the reviewer can catch cross-module inconsistencies — in this
74+
package that typically means: the emulator variants (`Emulator.jl` plus
75+
`MachineLearningTools/GaussianProcess.jl`, `RandomFeature.jl`,
76+
`ScalarRandomFeature.jl`, `VectorRandomFeature.jl`) in one or two units since
77+
they must agree on the same predict/covariance interface; the MCMC samplers in
78+
another; and the `Utilities/` preprocessing transforms (decorrelation,
79+
elementwise scaling, canonical correlation, likelihood-informed subspace) in
80+
another, since they feed both the emulator and the sampler and a convention
81+
mismatch there propagates silently downstream.
82+
83+
Reserve the smallest unit as a **cross-cutting unit**. Alongside its own files
84+
(module wiring, display code, package-wide greps for `dims=`, `corrected=`,
85+
jitter constants, symmetrization idioms), give it an explicit mandate to
86+
independently re-derive the *single highest-risk formula* in each other unit's
87+
territory — MCMC acceptance ratios, structure-matrix congruence transforms,
88+
predictive-covariance noise semantics. This overlap is deliberate: in one
89+
audit the cross-cutting unit independently found and numerically verified the
90+
same critical pCN acceptance-ratio bug as the MCMC unit, on a *different* toy
91+
problem — two matching numerical confirmations settle a critical finding in a
92+
way a single one never can. Tell that agent the overlap is intentional so it
93+
doesn't defer to the "owning" unit.
94+
95+
### 2. Fan out reviewers (parallel agents)
96+
97+
Spawn one agent per unit, in a single message so they run concurrently. Build
98+
each prompt from `references/unit-agent-prompt.md`: fill the placeholders
99+
(unit file list, domain context, unit-specific extra hunts, scratch directory)
100+
and paste in the full "What adversarial means here" list above — agents don't
101+
see this skill. The template carries the rules that have earned their keep in
102+
past audits (the numerical-verification tiering, the warn/throw grep before
103+
any "silent" claim, severity-rule reporting) and fixes the required
104+
three-section output: the findings JSON, a CONVENTIONS section, and a
105+
TEST-COVERAGE section. The CONVENTIONS lines are the raw material for the
106+
step-3 matrix — without them you must re-read the code yourself to build it.
107+
108+
As each agent's report arrives, save its raw findings verbatim to a scratchpad
109+
file (one per unit). A full audit plus verification is long enough that
110+
context summarization mid-run can silently lose findings; the scratchpad files
111+
are the durable record the report is assembled from.
112+
113+
### 3. Verify
114+
115+
For each critical/major finding, attempt refutation before it enters the
116+
report: re-read the cited lines yourself, re-derive the math, and check whether
117+
a test or an upstream transformation already accounts for it (common false
118+
positives: a transpose hidden in a helper, normalization done at construction
119+
time, a convention documented elsewhere, a `@warn` at the constructor that the
120+
reviewer never read — re-run the warn/throw grep yourself for any "silent"
121+
claim). Prioritise findings tagged `verified: inspection`; numerically verified
122+
ones usually need only a sanity re-read. Spawn skeptic agents for findings you
123+
can't settle from the main context. Demote or drop findings that don't survive;
124+
mark surviving ones **CONFIRMED** vs **PLAUSIBLE** (couldn't fully verify).
125+
126+
Then build a small **conventions matrix** from the agents' CONVENTIONS
127+
sections before writing anything: rows = modules, columns = the conventions
128+
they reported (covariance normalization N vs N−1, whitened vs unwhitened
129+
space, whether predictive covariance includes observational noise,
130+
symmetrization idiom, jitter constants, RNG threading, rows-vs-columns). Any
131+
mismatched cell between modules that must agree is a finding candidate in
132+
itself — in practice the worst bugs are a scale factor or transform applied in
133+
one emulator variant or preprocessing step but not its sibling, and they only
134+
become visible side by side.
135+
136+
### 4. Write the report
137+
138+
Create `full-code-review/<YYYY-MM-DD>/` (date from `date +%F`, never from
139+
memory). Before writing, copy every numerical-verification scratch script into
140+
`full-code-review/<date>/verification/<unit>/` and cite those repo-relative
141+
paths in the report — scratchpad paths die with the session, and a numerical
142+
finding whose evidence has evaporated is just an inspection finding with extra
143+
steps.
144+
145+
Write `review.md`:
146+
147+
```markdown
148+
# Adversarial Mathematical Review — <Package> (<date>)
149+
## Scope and method <!-- files covered, units, verification policy -->
150+
## Summary table <!-- ID | severity | verdict | file:line | one-line claim -->
151+
## Critical findings <!-- full detail: evidence, math, failure scenario, fix sketch -->
152+
## Major findings
153+
## Minor findings & hygiene <!-- terser -->
154+
## Cross-module consistency notes
155+
## Test-coverage gaps <!-- LEAD with 2-3 "keystone tests" — see below -->
156+
## What was checked and found sound <!-- credit where due; prevents re-auditing -->
157+
```
158+
159+
In **Test-coverage gaps**, lead with the 2–3 *keystone tests* whose absence
160+
masked the most findings, ranked by findings-caught-per-test. In practice a
161+
handful of missing test archetypes hide many bugs at once — e.g. an analytic
162+
linear-Gaussian posterior test asserting the MCMC mean **and variance** would
163+
have caught two critical sampler bugs in one audit, and a single test of the
164+
default noise-flag semantics would have caught three emulator findings. This
165+
list is the highest-leverage part of the report for maintainers.
166+
167+
Findings get stable IDs used everywhere, including fix prompts: `C1…` critical,
168+
`M1…` major, `m1…` minor, `h1…` hygiene; grouped-minor fix prompts get `G1…`.
169+
170+
### 5. Write fix prompts
171+
172+
For each finding with an actionable fix (usually critical + major, plus grouped
173+
minors), write `full-code-review/<date>/fix-prompts/<ID>-<slug>.md`. These are
174+
consumed by a *smaller model in a fresh session with no context*, so each must
175+
be self-contained:
176+
177+
```markdown
178+
# Fix <ID>: <one-line title>
179+
**File**: `src/Foo.jl`, function `bar!`, around line NNN.
180+
**Problem**: <2–4 sentences: what the code does vs what the math requires.
181+
Include the incorrect snippet verbatim.>
182+
**Required change**: <exact edit, or precise description with the correct formula>
183+
**Do not**: <guardrails — e.g. "do not change the API", "do not touch other methods">
184+
**Verify**: <the test to run or add, with the invariant it should pin>
185+
```
186+
187+
Keep each under ~40 lines. One finding per file; group only truly mechanical
188+
repeats (e.g. the same typo pattern in five docstrings) into one prompt.
189+
Quote enough of the offending snippet that the fixer can locate it by function
190+
name + snippet — line numbers drift between the audit and the fix session, so
191+
present them as hints, not anchors.
192+
Also write `fix-prompts/README.md` listing prompts in recommended application
193+
order (independent fixes first, same-file prompts sequenced, conflicting ones
194+
flagged) and naming which fixes *intentionally change numerical results* — so
195+
the fixer checks a failing loose regression test against the analytic reference
196+
in the prompt before "fixing" the test.
197+
198+
### 6. Report back
199+
200+
Final message: lead with the headline (how many confirmed critical/major
201+
findings and the single worst one), then the report path, then a compact
202+
summary table. Do not paste the whole report into the chat.
203+
204+
## Calibration
205+
206+
- Severity: **critical** = produces mathematically wrong results in mainstream
207+
use; **major** = wrong in common configurations or silently degrades
208+
statistical properties; **minor** = wrong in edge cases, misleading docs
209+
math, dead/misnamed math; **hygiene** = style-level (only if math-adjacent).
210+
- A numerically demonstrated wrong stationary distribution, wrong acceptance
211+
ratio, or wrong variance scaling in *exported* functionality is **critical**
212+
even if the feature looks niche — samplers and estimators exist to produce
213+
distributions, and a wrong distribution is the worst failure they have.
214+
(Agents report which calibration rule they applied, so when two units rate
215+
the same bug differently you arbitrate on the rule, not the adjective.)
216+
- A docstring–code mismatch is a real finding even when the code is right —
217+
users implement against docstrings.
218+
- A statistically unjustified combination that the package explicitly warns
219+
about (e.g. a constructor `@warn "... experimental ..."`) caps at **minor**
220+
unless the warning itself is wrong — the trap isn't silent.
221+
- Don't pad. If a unit is sound, the report says so in one paragraph; an audit
222+
that cries wolf gets ignored next time.
223+
224+
## Improving this skill
225+
226+
After delivering the report, offer: "Would you like to improve the
227+
**math-auditor** skill itself using skill-creator? You can share suggestions, or
228+
I can analyse this run — finding quality, false-positive rate, fix-prompt
229+
usability — to refine the skill for next time."

‎.claude/skills/math-auditor/references/unit-agent-prompt.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Unit-agent prompt template
2+
3+
Fill the `{PLACEHOLDERS}` and send one such prompt per review unit, all in a
4+
single message so the agents run concurrently. `{HUNTING_LIST}` is the full
5+
"What adversarial means here" bullet list from SKILL.md, pasted verbatim —
6+
agents cannot see the skill. `{EXTRA_HUNTS}` is where unit-specific attack
7+
targets go (derive them from the unit's domain: e.g. for an MCMC unit, spell
8+
out the correct pCN/Barker/MALA acceptance-ratio forms; for preprocessing
9+
units, the congruence-transform and round-trip invariants; for the
10+
cross-cutting unit, the deliberate-overlap mandate and the package-wide greps).
11+
12+
---
13+
14+
You are an adversarial mathematical reviewer for the Julia package {PACKAGE}
15+
at {REPO_PATH}. Your job is to BREAK the code mathematically, not describe it.
16+
Focus on mathematical accuracy and consistency, NOT software architecture
17+
(flag architecture only when it causes mathematical wrongness, e.g. mutation
18+
aliasing, accidental type demotion, inconsistent conventions).
19+
20+
YOUR UNIT (read every line of these):
21+
{FILES — with line counts, and which test files pair with which source files.
22+
Mark any dead/not-in-build files as skim-only.}
23+
24+
Context: {DOMAIN_CONTEXT — 3-6 sentences: what the package does, what this
25+
unit's math is supposed to compute, the key papers/algorithms it implements,
26+
and how this unit's outputs are consumed by the rest of the pipeline. Name
27+
adjacent files the agent may read for interface contracts but that another
28+
reviewer audits in depth.}
29+
30+
HUNT FOR:
31+
{HUNTING_LIST}
32+
{EXTRA_HUNTS}
33+
34+
RULES:
35+
- Prefer few, well-evidenced findings over many speculative ones — but do
36+
report genuine minor inconsistencies. If the module's math is correct, say
37+
so and note the strongest invariants the tests pin.
38+
- Verify cheap claims numerically and tag them `verified: numerical` —
39+
numerically verified findings are worth far more than inspection-only ones.
40+
Tiering, best first: (1) reproduce the behavior in-package
41+
(`julia --project={REPO_PATH}`; a Manifest.toml usually resolves — worth the
42+
precompile wait for behavioral claims like state-sharing, crashes, or
43+
option-ignored bugs); (2) standalone script using only
44+
LinearAlgebra/Statistics/Random/Distributions that reimplements the
45+
questioned formula next to the correct one (right tool for formula and
46+
statistical-scaling claims — e.g. a tiny MH chain on a 1D conjugate Gaussian
47+
exposes a wrong acceptance ratio in seconds); (3) inspection, only when
48+
running code is genuinely expensive. Put scratch scripts in {SCRATCH_DIR}
49+
and cite them per finding.
50+
- Before claiming anything is 'silent' or 'has no warning/guard', grep for
51+
`@warn`, `@error`, and `throw` at the *constructors and call sites* of the
52+
code path, not just the function you are reading — guards often live at
53+
construction time.
54+
- Check code against docstrings/comments AND against the standard form of the
55+
algorithm from the literature. A docstring–code mismatch is a real finding
56+
even when the code is right — users implement against docstrings.
57+
- Severity calibration: critical = mathematically wrong results in mainstream
58+
use — and any numerically demonstrated wrong stationary distribution,
59+
acceptance ratio, or variance scaling in exported functionality is critical
60+
even if the feature looks niche; major = wrong in common configurations or
61+
silently degrades statistical properties; minor = edge cases, misleading
62+
docs math, dead/misnamed math; hygiene = math-adjacent style. A trap the
63+
package explicitly @warns about caps at minor. For each finding, state
64+
WHICH rule you applied.
65+
66+
OUTPUT (your final message is raw data for synthesis, not prose for a human) —
67+
three sections, all required:
68+
69+
1. FINDINGS: a JSON-like list, each entry with: `file`, `line`, `severity`
70+
(critical/major/minor/hygiene) plus the calibration rule applied, `claim`
71+
(one sentence), `evidence` (quote the offending snippet vs the correct
72+
math), `failure_scenario` (concrete inputs → wrong output),
73+
`verified` (numerical / inspection, with script path if numerical),
74+
`suggested_fix` (optional, a few lines).
75+
76+
2. CONVENTIONS: one line per convention this unit assumes — data orientation
77+
(columns = samples?), covariance normalization (N vs N−1), whitened /
78+
encoded-space handling, whether predictive covariance includes
79+
observational noise, symmetrization idiom (`Symmetric` vs `hermitianpart`),
80+
jitter/regularization constants and where they apply, RNG threading.
81+
Explicitly note any DISAGREEMENT between files inside your unit. These
82+
lines feed a cross-module consistency matrix, so state values, not vibes.
83+
84+
3. TEST-COVERAGE: which mathematical properties the unit's tests pin
85+
(analytic values, invariants, statistical tolerances) vs leave unpinned —
86+
and for each of your findings, whether an existing test could have caught
87+
it (if not, that is a double finding: the bug and the missing test).

0 commit comments

Comments
 (0)