Add visual eval metric baseline tracking with regression detection

[Trecek]

Summary

As we add optimizations to the Rust spectral init (RustNative mode, SIMD kernels, solver tuning), we need automated tracking of how our visual eval metrics change over time. Currently the visual eval pipeline produces per-dataset metrics JSON files, but there's no mechanism to:

1. Record a baseline snapshot of metric values for the current Rust implementation
2. Detect regressions when a code change significantly worsens a key metric
3. Track improvement as optimizations bring us closer to Python parity

Current Baseline (2026-03-28, commit ca700e9)

Dataset	n	dims	Procrustes	Pairwise corr	Trust (Py)	Trust (Rust)	Silh (Py)	Silh (Rust)
circles_1000	1000	2	0.4718	0.6143	0.9982	0.9982	0.5840	0.5431
swiss_roll_2000	2000	3	0.1137	0.9325	0.9989	0.9989	0.4057	0.4398
two_moons_1000	1000	2	0.4994	0.5945	0.9976	0.9975	0.5797	0.5437
pendigits	1797	64	0.1794	0.8879	0.9867	0.9869	0.6171	0.6240
mnist_10k	10000	784	0.0646	0.9536	0.9543	0.9541	0.3785	0.3550
fashion_mnist_10k	10000	784	0.0733	0.9766	0.9780	0.9776	0.1708	0.1578

Key observations:

• Trustworthiness deltas are all < 0.001 — excellent parity
• Silhouette deltas are all < 0.04 — good parity
• Procrustes/pairwise vary widely per dataset (geometric alignment, not quality)
• High-dim datasets (MNIST, Fashion-MNIST) show best geometric alignment

Proposed Design

1. Baseline file (tests/visual_eval/baseline_metrics.json)

A checked-in JSON file recording the accepted metric values per dataset per commit:

{
  "version": 1,
  "recorded_at": "2026-03-28",
  "commit": "ca700e9",
  "datasets": {
    "mnist_10k": {
      "procrustes": 0.0646,
      "pairwise_corr": 0.9536,
      "trustworthiness_py": 0.9543,
      "trustworthiness_rust": 0.9541,
      "silhouette_py": 0.3785,
      "silhouette_rust": 0.3550
    }
  }
}

2. Regression detection in Phase 2

After computing metrics in run_compare(), compare against the baseline file. For each metric, flag:

• Regression warning: quality metric (trustworthiness, silhouette) worsened by more than a configurable threshold (e.g., trustworthiness delta > 0.005, silhouette delta > 0.02)
• Improvement note: metric improved beyond previous baseline
• Geometric shift: Procrustes or pairwise correlation changed significantly (informational, not a failure)

3. Output

• Print a per-dataset regression/improvement summary after Phase 2
• Write a {name}_regression.json if any regressions detected
• Exit with non-zero if quality regressions exceed threshold (for CI integration)

4. Baseline update workflow

After accepting a new optimization:

python tests/visual_eval/generate_umap_comparisons.py --update-baseline

This re-records the baseline from the current *_metrics.json files.

Key Metrics to Track

Quality metrics (regressions are bugs):

• Trustworthiness (Rust) — absolute value and delta from Python
• Silhouette (Rust) — absolute value and delta from Python

Alignment metrics (informational, track trends):

• Procrustes disparity vs Python
• Pairwise distance correlation vs Python

Scope

• Create baseline_metrics.json with current values
• Add --compare-baseline flag to Phase 2 that loads baseline and prints delta report
• Add --update-baseline flag to re-record baseline from current metrics
• Define regression thresholds (trustworthiness delta > 0.005, silhouette delta > 0.02)
• Print clear REGRESSION/IMPROVED/STABLE per-metric summary
• Non-zero exit code on quality regression for CI gating