PCS benchmark suite aligned with pcs-core, reproducibility benchmark, and smoke packet.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: fraware

.github/workflows/pcs.yml

@@ -83,20 +83,33 @@ jobs:
         run: python examples/pcs_qc_release/scripts/ci_validate_formalization.py
       - name: Generate and validate benchmark cases
         run: |
+          labtrust generate-benchmark-cases \
+            --workflow hospital_lab.qc_release \
+            --out /tmp/labtrust-bench \
+            --release-dir examples/pcs_qc_release/release \
+            --seed 42
+          labtrust verify-benchmark-cases --benchmark-dir /tmp/labtrust-bench
           labtrust generate-benchmark-cases \
             --workflow hospital_lab.qc_release \
             --out examples/pcs_qc_release/benchmark \
             --release-dir examples/pcs_qc_release/release \
             --seed 42
           python examples/pcs_qc_release/scripts/ci_validate_benchmark_cases.py
-      - name: Reproducibility benchmark (hash stability)
+          python examples/pcs_qc_release/scripts/ci_validate_benchmark_pcs_core.py
+      - name: pcs-bench validate fixtures (when pcs-core checkout present)
+        continue-on-error: true
         run: |
-          labtrust benchmark-reproducibility \
-            --pcs-core pcs-core/examples/labtrust-release \
-            --release-dir examples/pcs_qc_release/release \
-            --runs 3 \
-            --seed 42 \
-            --out /tmp/labtrust_repro_benchmark
+          if command -v pcs >/dev/null 2>&1; then
+            pcs benchmark validate
+          elif [ -d pcs-core/python ]; then
+            python -m pcs_core.cli benchmark validate
+          else
+            echo "skip pcs-bench validate (no pcs-core CLI)"
+          fi
+      - name: Reproducibility benchmark
+        run: python examples/pcs_qc_release/scripts/ci_benchmark_reproducibility.py
+      - name: Benchmark packet fixtures
+        run: python examples/pcs_qc_release/scripts/generate_benchmark_packet.py
       - name: Verify LabTrust release protocol package
         run: |
           labtrust verify-release-protocol \

docs/contracts/cli_contract.md

@@ -72,9 +72,9 @@ This document defines the contract for all LabTrust-Gym CLI commands: exit codes
 | regenerate-release-protocol | `--out <release-dir>`; `--certifyedge-bin`; optional `--pcs-core`; `--workflow-profile`; `--json-summary`; `--summary-out` | 0 | Full protocol package; writes `<out>/regeneration_report.json`, `proof_obligation_hints.json`, `proof_obligation_identifiers.json`, `formalization_readiness_report.json` (pcs-bench; Lean extraction inputs only) | examples/pcs_qc_release/RUNBOOK.md, schemas_or_docs/FormalizationReadinessReport.v0.md |
 | regenerate-release-chain | alias for `regenerate-release-protocol` | 0 | same as above | examples/pcs_qc_release/RUNBOOK.md |
 | generate-failure-gallery | `--workflow <id\|property_id>`; `--out <dir>`; optional `--release-dir`; `--workflow-profile` | 0 | Per-case `failure_case_manifest.json`, `README.md`, `artifacts/`, legacy `expected_failure.json` / `repair_hint.json` | examples/pcs_qc_release/failures/ |
-| generate-benchmark-cases | `--workflow <id\|property_id>`; `--out <dir>`; `--seed` (default 42); optional `--release-dir`; `--workflow-profile` | 0 | `benchmark_case.v0.json`, `input_artifacts/`, `expected_failure.json`, `expected_repair_hint.json`, `benchmark_index.json`, `coverage_report.v0.json` | docs/labtrust-benchmark-profile.md |
-| benchmark-reproducibility | `--out <dir>`; `--runs` (default 5); `--seed`; `--mode hash_stability\|full_regeneration`; optional `--pcs-core`, `--release-dir`, `--certifyedge-bin` | 0 | `benchmark_run.v0.json`, `coverage_report.v0.json` | docs/labtrust-benchmark-profile.md |
-| verify-benchmark-cases | `--benchmark-dir <dir>`; `--json` | 0 | (stderr OK labels) | docs/labtrust-benchmark-profile.md |
+| generate-benchmark-cases | `--workflow <id\|property_id>`; `--out <dir>`; `--seed` (default 42); optional `--release-dir`; `--workflow-profile` | 0 | `benchmark_case.v0.json`, `labtrust_benchmark_extension.v0.json`, `input_artifacts/`, `expected_failure.json`, `expected_repair_hint.json`, `benchmark_index.json`, `coverage_report.v0.json` | docs/labtrust-benchmark-profile.md |
+| benchmark-reproducibility | `--out <dir>`; `--runs` (default 5); `--seed`; `--mode full_regeneration` (default) or `hash_stability`; optional `--pcs-core`, `--release-dir`, `--certifyedge-bin` | 0 | `benchmark_run.v0.json`, `coverage_report.v0.json`, `regeneration_report.json` | docs/labtrust-benchmark-profile.md |
+| verify-benchmark-cases | `--benchmark-dir <dir>`; optional `--pcs-core` (auto-detect); `--json` | 0 | (stderr OK labels) | docs/labtrust-benchmark-profile.md |
 
 **PCS CI scripts (not `labtrust` subcommands):** `run_pcs_ci_local.{sh,ps1}` runs `pytest tests/pcs`, `ci_validate_workflow_profile.py`, `ci_validate_formalization.py`, export/release validation, `check-status-policy`, failure gallery generate/verify, and optional `verify-release-protocol` (matches `.github/workflows/pcs.yml`). `materialize_workflow_profile.py` refreshes `workflow_profile.v0.json` digest. `materialize_formalization_artifacts.py` rebuilds formalization JSON from an existing `release/` tree. `generate_golden.py` regenerates `expected/` in deterministic mode.
 

docs/labtrust-benchmark-profile.md

@@ -23,16 +23,21 @@ deployment safety.
 
 | Artifact | Role |
 |----------|------|
-| `benchmark_case.v0.json` | pcs-bench case descriptor (failure localization) |
-| `input_artifacts/` | Release-shaped inputs for the case |
+| `benchmark_case.v0.json` | pcs-core `BenchmarkCase.v0` (status, failure code, provenance, digest) |
+| `labtrust_benchmark_extension.v0.json` | LabTrust-only: detection layer, protocol failure code |
+| `input_artifacts/` | Release-shaped inputs (`input_artifacts.release_directory`) |
 | `expected_failure.json` | Protocol-level failing check and code |
-| `expected_repair_hint.json` | `hint_kind`, operator hint, `repair_command` |
+| `expected_repair_hint.json` | `responsible_component`, `repair_hint.kind`, `repair_command` |
 | `benchmark_index.json` | Suite index |
 | `coverage_report.v0.json` | Taxonomy coverage (case kinds, detection layers) |
-| `benchmark_run.v0.json` | Reproducibility aggregate (per-run hashes, duration) |
+| `benchmark_run.v0.json` | Reproducibility summary (pcs-bench compatible) |
+| `regeneration_report.json` | Full reproducibility report (`ReproducibilityBenchmarkReport.v0`) |
 
 Schemas: `policy/schemas/pcs/BenchmarkCase.v0.schema.json`, `BenchmarkRun.v0.schema.json`,
-`CoverageReport.v0.schema.json`.
+`CoverageReport.v0.schema.json`, `ReproducibilityBenchmarkReport.v0.schema.json`.
+
+Minimal smoke packet: `examples/pcs_qc_release/benchmark_packet/` (`valid_release`,
+`invalid_trace_hash_tamper`, `reproduce.sh`, `expected_report.json`).
 
 ## Failure case taxonomy
 
@@ -86,11 +91,13 @@ labtrust benchmark-reproducibility \
   --seed 42
 ```
 
-Default mode `hash_stability` copies the committed `release/` tree and verifies identical
-hashes and validation across runs (CI-safe). `full_regeneration` is reserved for local benches
-with CertifyEdge.
+Default mode `full_regeneration` re-runs `regenerate-release-protocol` each iteration and
+records artifact/canonical hash stability, CertifyEdge success, release protocol validation,
+status policy, and pcs-core validation. CI falls back to `hash_stability` when CertifyEdge is
+unavailable (`examples/pcs_qc_release/scripts/ci_benchmark_reproducibility.py`).
 
 ## pcs-bench integration
 
-Load `benchmark_index.json`, iterate `cases`, and for each case read `benchmark_case.v0.json`
-plus `input_artifacts/` — no LabTrust-specific adapters required beyond the published schemas.
+Suite id `labtrust-qc-release-v0`. Load `benchmark_index.json`, iterate `cases`, and for each
+case read `benchmark_case.v0.json` plus `input_artifacts/`. Use `labtrust_benchmark_extension.v0.json`
+for detection-layer assertions in simulate mode.

examples/pcs_qc_release/RUNBOOK.md

@@ -306,19 +306,29 @@ labtrust generate-failure-gallery \
   --workflow hospital_lab.qc_release \
   --out examples/pcs_qc_release/failures
 
-# pcs-bench suite (BenchmarkCase.v0 — failure localization + repair hints):
+# Align release/ artifact bytes with pcs-core RC (when canonical tree drifts):
+python examples/pcs_qc_release/scripts/sync_release_from_pcs_core.py
+python examples/pcs_qc_release/scripts/materialize_downstream_release_artifacts.py
+
+# pcs-bench suite (BenchmarkCase.v0 — pcs-core aligned + LabTrust extensions):
 labtrust generate-benchmark-cases \
   --workflow hospital_lab.qc_release \
   --out examples/pcs_qc_release/benchmark \
   --seed 42
-labtrust verify-benchmark-cases --benchmark-dir examples/pcs_qc_release/benchmark
+labtrust verify-benchmark-cases --benchmark-dir examples/pcs_qc_release/benchmark --pcs-core ../pcs-core
 python examples/pcs_qc_release/scripts/ci_validate_benchmark_cases.py
+python examples/pcs_qc_release/scripts/ci_validate_benchmark_pcs_core.py
+
+# Minimal two-case packet for pcs-bench smoke (no full gallery):
+python examples/pcs_qc_release/scripts/generate_benchmark_packet.py
 
 labtrust benchmark-reproducibility \
   --pcs-core ../pcs-core \
   --runs 5 \
+  --mode full_regeneration \
   --out benchmark_runs/labtrust_reproducibility \
   --seed 42
+python examples/pcs_qc_release/scripts/ci_benchmark_reproducibility.py
 
 labtrust check-status-policy \
   --release-dir examples/pcs_qc_release/release

examples/pcs_qc_release/benchmark/benchmark_index.json

@@ -28,9 +28,8 @@
     {
       "benchmark_case_id": "labtrust-missing-qc-result-v0",
       "case_id": "missing_qc_result",
-      "case_kind": "missing_qc_step",
+      "case_kind": "invalid_render",
       "directory": "missing_qc_result",
-      "expected_detection_layer": "LabTrust",
       "input_artifacts": [
         "run_meta.json",
         "trace.json",
@@ -40,9 +39,8 @@
     {
       "benchmark_case_id": "labtrust-unauthorized-release-v0",
       "case_id": "unauthorized_release",
-      "case_kind": "unauthorized_actor",
+      "case_kind": "invalid_render",
       "directory": "unauthorized_release",
-      "expected_detection_layer": "LabTrust",
       "input_artifacts": [
         "run_meta.json",
         "trace.json",
@@ -54,7 +52,6 @@
       "case_id": "trace_hash_tamper",
       "case_kind": "invalid_hash_mismatch",
       "directory": "trace_hash_tamper",
-      "expected_detection_layer": "LabTrust",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -72,9 +69,8 @@
     {
       "benchmark_case_id": "labtrust-certificate-id-tamper-v0",
       "case_id": "certificate_id_tamper",
-      "case_kind": "certificate_tamper",
+      "case_kind": "invalid_certificate",
       "directory": "certificate_id_tamper",
-      "expected_detection_layer": "CertifyEdge",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -92,9 +88,8 @@
     {
       "benchmark_case_id": "labtrust-stale-trace-after-certificate-v0",
       "case_id": "stale_trace_after_certificate",
-      "case_kind": "status_transition",
+      "case_kind": "stale_release",
       "directory": "stale_trace_after_certificate",
-      "expected_detection_layer": "LabTrust",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -112,9 +107,8 @@
     {
       "benchmark_case_id": "labtrust-legacy-handoff-file-v0",
       "case_id": "legacy_handoff_file",
-      "case_kind": "handoff_tamper",
+      "case_kind": "invalid_handoff",
       "directory": "legacy_handoff_file",
-      "expected_detection_layer": "LabTrust",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -133,9 +127,8 @@
     {
       "benchmark_case_id": "labtrust-placeholder-commit-v0",
       "case_id": "placeholder_commit",
-      "case_kind": "provenance_invalid",
+      "case_kind": "invalid_registry",
       "directory": "placeholder_commit",
-      "expected_detection_layer": "LabTrust",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -153,9 +146,8 @@
     {
       "benchmark_case_id": "labtrust-lean-trace-hash-mismatch-v0",
       "case_id": "lean_trace_hash_mismatch",
-      "case_kind": "formal_check_failure",
+      "case_kind": "invalid_formal_check",
       "directory": "lean_trace_hash_mismatch",
-      "expected_detection_layer": "Lean trust kernel",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -173,9 +165,8 @@
     {
       "benchmark_case_id": "labtrust-lean-rejected-certificate-v0",
       "case_id": "lean_rejected_certificate",
-      "case_kind": "formal_check_failure",
+      "case_kind": "invalid_formal_check",
       "directory": "lean_rejected_certificate",
-      "expected_detection_layer": "Lean trust kernel",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -193,9 +184,8 @@
     {
       "benchmark_case_id": "labtrust-lean-stale-certificate-v0",
       "case_id": "lean_stale_certificate",
-      "case_kind": "formal_check_failure",
+      "case_kind": "invalid_formal_check",
       "directory": "lean_stale_certificate",
-      "expected_detection_layer": "Lean trust kernel",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -213,9 +203,8 @@
     {
       "benchmark_case_id": "labtrust-lean-signed-hash-mismatch-v0",
       "case_id": "lean_signed_hash_mismatch",
-      "case_kind": "formal_check_failure",
+      "case_kind": "invalid_formal_check",
       "directory": "lean_signed_hash_mismatch",
-      "expected_detection_layer": "Provability Fabric",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -235,9 +224,8 @@
     {
       "benchmark_case_id": "labtrust-scientific-memory-import-failure-v0",
       "case_id": "scientific_memory_import_failure",
-      "case_kind": "scientific_memory_import_failure",
+      "case_kind": "invalid_import",
       "directory": "scientific_memory_import_failure",
-      "expected_detection_layer": "Scientific Memory",
       "input_artifacts": [
         "trace.json",
         "runtime_receipt.json",
@@ -258,6 +246,7 @@
   "out_dir": "benchmark",
   "pcs_bench": {
     "case_descriptor": "benchmark_case.v0.json",
+    "case_extension": "labtrust_benchmark_extension.v0.json",
     "coverage_report": "coverage_report.v0.json",
     "expected_failure": "expected_failure.json",
     "expected_repair_hint": "expected_repair_hint.json",
@@ -269,7 +258,7 @@
   "schema_version": "v0",
   "seed": 42,
   "status": "passed",
-  "task_id": "labtrust-qc-release-failure-localization-v0",
+  "task_id": "labtrust-qc-release-v0",
   "workflow_id": "hospital_lab.qc_release",
   "workflow_profile": "examples/pcs_qc_release/workflow_profile.v0.json"
 }

examples/pcs_qc_release/benchmark/certificate_id_tamper/README.md

@@ -4,4 +4,4 @@ Certificate id propagation fails after trace_certificate tampering.
 
 - Benchmark case: `labtrust-certificate-id-tamper-v0`
 - Detection layer: `CertifyEdge`
-- Case kind: `certificate_tamper`
+- Case kind: `invalid_certificate`

examples/pcs_qc_release/benchmark/certificate_id_tamper/benchmark_case.v0.json

@@ -1,17 +1,74 @@
 {
   "case_id": "labtrust-certificate-id-tamper-v0",
-  "case_kind": "certificate_tamper",
-  "expected_detection_layer": "CertifyEdge",
-  "expected_failing_check": "certificate_id_propagation",
+  "case_kind": "invalid_certificate",
   "expected_failure_code": "certificate_id_mismatch",
-  "expected_protocol_failure_code": "CERTIFICATE_ID_MISMATCH",
-  "expected_repair_command": "certifyedge emit-pcs-certificate ... && labtrust attach-certificate --bundle science_claim_bundle.pending.json --certificate trace_certificate.json",
-  "expected_repair_hint_kind": "re_emit_certificate",
-  "expected_responsible_component": "certifyedge.certificate",
-  "expected_status": "Rejected",
-  "gallery_case_id": "certificate_id_tamper",
-  "profile_workflow_id": "labtrust.qc_release_v0.1",
+  "expected_repair_hint_kind": "align_certificate_id",
+  "expected_responsible_component": "certificate_producer",
+  "expected_status": "failed",
+  "input_artifacts": {
+    "artifacts": [
+      {
+        "artifact_type": "HandoffManifest.v0",
+        "path": "handoff_to_certifyedge.json",
+        "role": "handoff_to_certifyedge_json"
+      },
+      {
+        "artifact_type": "HandoffManifest.v0",
+        "path": "handoff_to_pf.json",
+        "role": "handoff_to_pf_json"
+      },
+      {
+        "artifact_type": "ComponentReleaseFragment.v0",
+        "path": "labtrust_release_fragment.json",
+        "role": "labtrust_release_fragment_json"
+      },
+      {
+        "artifact_type": "ReleaseManifest.v0",
+        "path": "manifest.json",
+        "role": "manifest_json"
+      },
+      {
+        "artifact_type": "RuntimeReceipt.v0",
+        "path": "runtime_receipt.json",
+        "role": "runtime_receipt_json"
+      },
+      {
+        "artifact_type": "ScienceClaimBundle.v0",
+        "path": "science_claim_bundle.certified.json",
+        "role": "science_claim_bundle_certified_json"
+      },
+      {
+        "artifact_type": "ScienceClaimBundle.v0",
+        "path": "science_claim_bundle.pending.json",
+        "role": "science_claim_bundle_pending_json"
+      },
+      {
+        "artifact_type": "LabTrustTrace.v0",
+        "path": "trace.json",
+        "role": "trace_json"
+      },
+      {
+        "artifact_type": "TraceCertificate.v0",
+        "path": "trace_certificate.json",
+        "role": "trace_certificate_json"
+      },
+      {
+        "artifact_type": "TraceHashAlignment.v0",
+        "path": "trace_hash_alignment.json",
+        "role": "trace_hash_alignment_json"
+      },
+      {
+        "artifact_type": "WorkflowProfile.v0",
+        "path": "workflow_profile.v0.json",
+        "role": "workflow_profile_v0_json"
+      }
+    ],
+    "release_directory": "input_artifacts"
+  },
   "schema_version": "v0",
-  "task_id": "labtrust-qc-release-failure-localization-v0",
+  "signature_or_digest": "sha256:5e3d37f28e5bbb97edf48e38ec746afdec64476f98a0f15b905a5758ef3e6e5d",
+  "source_commit": "186c65b252bfaade6ee321b0a74c15980852cdd2",
+  "source_repo": "https://github.com/fraware/LabTrust-Gym",
+  "task_id": "labtrust-qc-release-v0",
   "workflow_id": "hospital_lab.qc_release"
 }

examples/pcs_qc_release/benchmark/certificate_id_tamper/expected_failure.json

@@ -1,5 +1,6 @@
 {
   "case_id": "certificate_id_tamper",
   "expected_failing_check": "certificate_id_propagation",
-  "expected_failure_code": "CERTIFICATE_ID_MISMATCH"
+  "expected_failure_code": "certificate_id_mismatch",
+  "expected_protocol_failure_code": "CERTIFICATE_ID_MISMATCH"
 }

examples/pcs_qc_release/benchmark/certificate_id_tamper/expected_repair_hint.json

@@ -1,5 +1,10 @@
 {
-  "hint": "Re-run CertifyEdge emit-pcs-certificate and attach-certificate without editing certificate_id.",
-  "hint_kind": "re_emit_certificate",
-  "repair_command": "certifyedge emit-pcs-certificate ... && labtrust attach-certificate --bundle science_claim_bundle.pending.json --certificate trace_certificate.json"
+  "expected_detection_layer": "CertifyEdge",
+  "failure_code": "certificate_id_mismatch",
+  "repair_hint": {
+    "command": "certifyedge emit-pcs-certificate ... && labtrust attach-certificate --bundle science_claim_bundle.pending.json --certificate trace_certificate.json",
+    "description": "Re-run CertifyEdge emit-pcs-certificate and attach-certificate without editing certificate_id.",
+    "kind": "align_certificate_id"
+  },
+  "responsible_component": "certificate_producer"
 }