Skip to content

.eval-notes-F.md

Latest commit

 

History

History
97 lines (79 loc) · 4.01 KB

.eval-notes-F.md

File metadata and controls

97 lines (79 loc) · 4.01 KB
Raw

Eval F — benchdiff-rs

Cycle: F (cycle 1 of 3 clean) Agent: fresh, hostile, did not build, did not run A-E Result: CLEAN

Scope

Hostile independent audit per Round 79 Phase 2 protocol. Re-verify eval-E's config auto-discovery fix and probe README feature claims against actual binary behavior using the built target/release/benchdiff.exe.

Checklist

  1. Read every source file: main.rs, cli.rs, config.rs, baseline.rs, compare.rs, error.rs, lib.rs, parsers/mod.rs, report/mod.rs, report/text.rs, report/markdown.rs, report/json.rs.
  2. Re-verified eval-E's Config::load_or_default with unit test + live binary probe.
  3. Verified --config path overrides auto-discovered ./benchdiff.toml.
  4. Verified a malformed auto-discovered config fails loudly (exit 3, full TOML parse diagnostic, not swallowed).
  5. Verified benchdiff list reads baseline_dir from the auto-discovered config (not hardcoded).
  6. Verified benchdiff init prints a valid, round-trippable template.
  7. Verified Cargo.toml version (1.0.0) matches README badge, CHANGELOG, and --version output.
  8. Ran each documented CLI example (save/compare/list/init/summary/inspect) against fresh CSV fixtures.
  9. Ran cargo test — 180 passed, 0 failed (9 test binaries).
  10. Cross-module consistency: Error variants ↔ classify_exit_code exit codes; text/markdown/json renderers use shared fmt_ns/fmt_pct/ fmt_d with saturation sentinels.

Eval-E fix re-verification

  • Unit test load_or_default_auto_discovers_cwd passes.
  • Live: wrote C:\...\benchdiff-rs\benchdiff.toml with baseline_dir = "cwd_baselines" and ran benchdiff list — output was no baselines in cwd_baselines, proving auto-discovery took effect.
  • Live: with CWD config in place, benchdiff --config /c/tmp/other.toml list (where other.toml had baseline_dir = "other_baselines") returned no baselines in other_baselines — --config wins.
  • Live: wrote benchdiff.toml with alpha = not_a_number and ran benchdiff list — exit 3, full TOML parse error surfaced. Not silenced.

Live probes against release binary

benchdiff --version                      → benchdiff 1.0.0
benchdiff init                           → valid TOML template
benchdiff save bench1.csv --label v1     → bdb\v1.json (via cwd config)
benchdiff list                           → v1
benchdiff compare bench2.csv --against v1 --output markdown
    → 8-column GFM table, 1 regression (parse_small +50.00%,
      p <1e-4, d=35.36), exit 1
benchdiff compare ... --output json
    → valid JSON, full p_value precision (3.278258399375255e-14),
      no null numerics, snake_case field names
benchdiff summary bench2.csv --against v1 --top 2
    → UTF8 box table, expected rows
benchdiff inspect bench1.csv
    → "source: csv / benchmarks: 2 / ..."

Observations (non-bugs, noted for future)

  • Verdict serializes to JSON as PascalCase ("Regression") while the library's own Verdict::as_str() returns snake_case ("regression"). Minor consistency nit, not README-promised. Not fixing in F (out of hostile-audit scope).
  • Markdown table's first header column is an empty cell (emoji slot has no label). Cosmetic; the table is still valid GFM.
  • --config /nonexistent.toml silently falls back to the default config (documented in source as deliberate for prior-behavior compatibility). Mildly surprising but intentional.

False alarm cleared

First probe showed duplicated TOML error in the binary output, which would have invalidated eval-C's fix. Root cause: target/release/benchdiff.exe was stale — eval-C's source fix was in place but the release binary had not been rebuilt since. cargo build --release followed by re-running the probe gave the expected single-copy error output. Not a current bug; note added here so future cycles remember to force-rebuild before binary-level probes.

Test result

cargo test — 180 passed, 0 failed, 0 ignored.

Fixes

None. Clean audit.

Verdict

CLEAN — cycle 1 of 3.