pystamps/docs/verification.html at main · ESA-PhiLab/pystamps · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Verification - pySTAMPS Docs</title>
  <link rel="stylesheet" href="assets/styles.css" />
  <script defer src="assets/scripts.js"></script>
</head>
<body>
<div class="site">
  <aside class="sidebar">
    <div class="logo"><img class="brand-logo" src="assets/pystamps-logo.svg" alt="pySTAMPS logo" /><h1>pySTAMPS Docs</h1></div>
    <small>Output comparison and parity checks.</small>
    <nav>
      <div class="nav-group"><h2>Guide</h2><ul class="nav-list"><li><a href="index.html">Introduction</a></li><li><a href="stages.html">Stages and Code Paths</a></li><li><a href="pipeline-science-guide.html">Pipeline Guide</a></li><li><a href="getting-started.html">Getting Started</a></li><li><a href="installation.html">Installation</a></li><li><a href="quickstart.html">Quick Start</a></li><li><a href="native-cli.html">Native Rust CLI</a></li><li><a href="architecture.html">Architecture</a></li><li><a href="configuration.html">Configuration</a></li><li><a href="usage.html">Usage</a></li><li><a href="testing.html">Testing</a></li><li><a class="active" href="verification.html">Verification</a></li><li><a href="contributing.html">Contributing</a></li><li><a href="faq.html">FAQ</a></li></ul></div>
      <div class="nav-group"><h2>Reference</h2><ul class="nav-list"><li><a href="function-reference.html">Function Reference</a></li><li><a href="api/pystamps.html">Package API</a></li><li><a href="api/cli.html">cli</a></li><li><a href="api/verify.html">verify</a></li><li><a href="api/parity_contract.html">parity_contract</a></li></ul></div>
    </nav>
  </aside>
  <main class="content">
    <div class="breadcrumb">Home / Verification</div>

    <section class="hero">
      <span class="badge">Result checks</span>
      <h1>Validate generated outputs with deterministic comparison workflows.</h1>
      <p>Verification in pySTAMPS has two practical layers: a direct CLI comparison against a reference dataset, and parity-audit scripts that prepare fresh run copies before validation.</p>
    </section>

    <section class="card">
      <h2>CLI verification</h2>
      <p>Use <span class="code-inline">pystamps verify</span> once a run copy exists.</p>
      <pre><button class="copy-btn">Copy</button><code class="language-bash">uv run pystamps verify --run /path/to/run_copy --golden /path/to/reference_dataset</code></pre>
      <ul>
        <li>Returns <span class="code-inline">0</span> when all comparisons pass and non-zero when mismatches are found.</li>
        <li>Compares numeric MATLAB payloads with tolerance settings and classifies failure families by artifact.</li>
        <li>Use repository shortcuts only for fixed in-repo references.</li>
      </ul>
    </section>

    <section class="card">
      <h2>Native full-chain verification</h2>
      <p>Use this gate for the Rust native execution path. It prepares a clean run copy, executes native stages, checks performance budgets, and compares outputs against the golden dataset.</p>
      <pre><button class="copy-btn">Copy</button><code class="language-bash">make native-full-chain-verify</code></pre>
      <pre><button class="copy-btn">Copy</button><code class="language-bash">make native-full-chain-verify \
  DATASET=/path/to/source_dataset \
  GOLDEN=/path/to/reference_dataset \
  RUN=/path/to/clean_run_dataset \
  THREADS=8</code></pre>
      <p>Read <span class="code-inline">RUN/_native_gate_reports/</span> for coverage, run output, timings, and verification details.</p>
    </section>

    <section class="card">
      <h2>Repository shortcut caveat</h2>
      <p><span class="code-inline">make verify</span> is a convenience for fixed reference paths in this repository. For arbitrary runs, use the CLI command above so the path pair is explicit.</p>
    </section>

    <section class="card">
      <h2>Parity audit workflow</h2>
      <p>Parity audits prepare and track fresh run copies, then report the copy path for explicit verification.</p>
      <p><span class="code-inline">make audit</span> is the repo-local convenience wrapper. The authoritative driver is <span class="code-inline">scripts/validate_audit.py</span>, and omitting <span class="code-inline">--datasets</span> makes it use the manifest-backed required dataset set from <span class="code-inline">pystamps/data/audited_workflow_manifest.json</span>.</p>
      <pre><button class="copy-btn">Copy</button><code class="language-bash">make audit</code></pre>
      <pre><button class="copy-btn">Copy</button><code class="language-bash">OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 PYTHONPATH=. \
  uv run python scripts/validate_audit.py \
    --output /path/to/latest_audit.json</code></pre>
      <pre><button class="copy-btn">Copy</button><code class="language-bash">RUN_COPY="$(uv run python - <<'PY'
import json
from pathlib import Path

payload = json.loads(Path('/path/to/latest_audit.json').read_text(encoding='utf-8'))
print(payload['audits'][0]['run_root'])
PY
)"
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 PYTHONPATH=. \
  uv run pystamps verify --run "$RUN_COPY" --golden /path/to/reference_dataset</code></pre>
      <ul>
        <li>Fresh audit output should be treated as active input for subsequent checks.</li>
        <li>The required done-gate datasets are owned by <span class="code-inline">pystamps/data/audited_workflow_manifest.json</span>, including the small-baseline stage-7 audit targets.</li>
        <li>Do not rerun against stale copies; interrupted audits or manual restart paths invalidate prior results.</li>
      </ul>
    </section>

    <section class="card">
      <h2>Parity loop and oracle precedence</h2>
      <p><span class="code-inline">make parity-loop</span> wraps <span class="code-inline">scripts/parity_bug_loop.py</span>. The loop consumes a matching <span class="code-inline">latest_audit.json</span> when one already exists, then ranks the next target from the saved first-divergent-boundary trace instead of guessing from downstream symptoms.</p>
      <p>Oracle precedence comes from <span class="code-inline">pystamps/data/oracle_contract.json</span>: <span class="code-inline">cpp_wrapper</span>, then <span class="code-inline">matlab_source</span>, then <span class="code-inline">manual_references</span>. When the pinned StaMPS wrapper intentionally differs from the MATLAB scripts, the wrapper-backed path is the practical parity oracle and should be recorded in the emitted audit trace.</p>
    </section>

    <section class="card">
      <h2>Release flow references</h2>
      <p>For release gate ordering, use the same parity-audit and verify steps from the verification flow. The release page now links here for full command context.</p>
      <p>See the full API contract fields for parity in <a href="api/parity_contract.html">api/parity_contract</a> and the command-level verification API in <a href="api/verify.html">api/verify</a>.</p>
    </section>
  </main>
</div>
</body>
</html>