Introduce Stage 5 release candidate schemas

Author: anth-volk

changelog.d/1022.added

@@ -0,0 +1 @@
+Added typed Stage 5 release candidate schemas and Stage 4 candidate bundle readers.

docs/engineering/skills/README.md

@@ -24,3 +24,10 @@ Current skills:
 Stage-specific AI-facing engineering guides live under `docs/engineering/stages/`.
 Use them alongside these cross-cutting skills when modifying a stage-specific
 pipeline path.
+
+Current stage guides:
+
+- `build_outputs.md`: Stage 4 output-build library boundaries and test
+  expectations.
+- `release_promotion.md`: Stage 5 release candidate identity, validation-report
+  schema, rerun comparison material, and side-effect boundaries.

docs/engineering/stages/release_promotion.md

@@ -0,0 +1,85 @@
+# Release Promotion Stage AI Guide
+
+This guide is for AI agents and maintainers modifying Stage 5
+(`5_validate_and_promote_release`) code. Stage 5 validates a staged release
+candidate, promotes the exact candidate to public Hugging Face and GCS
+destinations, writes release/version/completion metadata, and cleans staging
+only after completion is certified.
+
+## Candidate Identity
+
+Use `policyengine_us_data.release_promotion.ReleasePromotionContext` as the
+typed Stage 5 identity boundary. The context must keep these values distinct:
+
+- `run_id`: the canonical publication run correlation key.
+- `candidate_version`: the candidate staging scope used in Hugging Face staging
+  paths such as `staging/{candidate_version}-{run_id}/...`.
+- `release_version`: the final stable public release version.
+- `base_release_version` and `release_bump`: optional provenance for how the
+  candidate scope was chosen.
+
+Do not resolve a different run ID from the environment inside lower-level
+release-promotion logic. Environment resolution belongs at orchestration edges;
+Stage 5 library code should receive explicit context.
+
+## Release Candidate Bundles
+
+Use `ReleaseCandidateInputBundle` to describe the artifacts Stage 5 is allowed
+to validate and promote. Each artifact should be represented by a
+`ReleaseArtifactSpec` with a production-relative path, artifact family, source
+stage, and optional checksum/size metadata.
+
+The current compatibility path may build a bundle from the legacy staged path
+set produced by Modal orchestration. Mark that reader as compatibility-only and
+keep it retirable.
+
+The Stage 4 contract/inventory reader API now exists for migration work:
+`build_release_candidate_bundle_from_stage4_contract()` accepts an in-memory
+Stage 4 contract plus inventory records, and
+`read_stage4_release_candidate_bundle()` reads the same shape from files.
+Production Stage 5 code should not depend on Stage 4 contracts until the
+contract and inventory are canonical, complete, and populated with semantic
+artifact identity plus checksum/size material.
+
+## Validation Reports
+
+Stage 5 must use the shared validation schema for durable validation output:
+
+- `policyengine_us_data.stage_contracts.ValidationReport`
+- `policyengine_us_data.stage_contracts.ValidationFinding`
+- `policyengine_us_data.stage_contracts.DiagnosticRef`
+
+Do not create a Stage 5-specific durable validation report, check, finding, or
+error schema for contracts, diagnostics, release candidates, status endpoints,
+or step manifests. Release-specific details such as missing staged artifacts,
+missing validation reports, finalized-release conflicts, version mismatches, or
+destination conflicts should live in canonical finding metadata.
+
+## Rerun Comparison Material
+
+Before public writes, rerun and reuse decisions should compare semantic
+candidate identity rather than only checking whether output files exist. The
+comparison material should include:
+
+- run ID, candidate version, release version, HF repository, and GCS bucket;
+- Stage 4 output contract fingerprint when available;
+- output inventory paths/checksums when available;
+- validation report paths and their identities when available;
+- expected production-relative artifact paths;
+- the Stage 5 candidate bundle fingerprint.
+
+When required artifacts only have paths and no checksum/size identity, treat
+the bundle as path-only and do not use its fingerprint for promotion reuse
+decisions.
+
+Already-finalized releases are an idempotency case, not a shortcut around
+candidate identity. A finalized release can be reused only when its completion
+marker is valid and it matches the requested candidate.
+
+## Side Effects
+
+Candidate builders, schema adapters, and rerun comparison helpers should not
+perform Hugging Face writes, GCS uploads, Modal calls, staging cleanup, or
+release-manifest publication. Keep those operations behind explicit adapters or
+services so tests can exercise candidate shape and validation logic without
+credentials or network access.

docs/pipeline_map.yaml

@@ -759,7 +759,7 @@ stages:
     node_type: artifact
     description: Policy target database copied into the pipeline volume
   - id: hf_staging_base_s1g
-    label: HuggingFace staging/{candidate_version}/{run_id}
+    label: HuggingFace staging/{candidate_version}-{run_id}
     node_type: external
     description: Run-scoped staging prefix for base datasets
   - id: stage_base_datasets
@@ -1504,7 +1504,7 @@ stages:
     node_type: artifact
     description: Output set from substage 5a
   - id: hf_staging_s5b
-    label: HuggingFace staging/{candidate_version}/{run_id}
+    label: HuggingFace staging/{candidate_version}-{run_id}
     node_type: external
     description: Run-scoped staging prefix containing validated artifacts
   - id: out_hf_prod

policyengine_us_data/release_promotion/__init__.py

@@ -0,0 +1,42 @@
+"""Typed Stage 5 release promotion boundaries.
+
+This package starts with release-candidate identity and candidate-bundle
+schemas. Promotion side effects still live in the existing transaction engine
+until later Stage 5 migration slices move them behind typed services.
+"""
+
+from .artifacts import (
+    BASE_RELEASE_ARTIFACT_PATHS,
+    ReleaseArtifactSpec,
+    dedupe_normalized_release_paths,
+    infer_artifact_identity,
+    infer_release_artifact_spec,
+    logical_name_for_release_path,
+    normalize_release_path,
+    strip_staging_prefix,
+)
+from .candidate import (
+    ReleaseCandidateInputBundle,
+    build_legacy_release_candidate_bundle,
+    build_release_candidate_bundle_from_stage4_contract,
+    read_stage4_release_candidate_bundle,
+)
+from .context import ReleasePromotionContext
+from .validation import build_release_candidate_shape_report
+
+__all__ = [
+    "BASE_RELEASE_ARTIFACT_PATHS",
+    "ReleaseArtifactSpec",
+    "ReleaseCandidateInputBundle",
+    "ReleasePromotionContext",
+    "build_legacy_release_candidate_bundle",
+    "build_release_candidate_bundle_from_stage4_contract",
+    "build_release_candidate_shape_report",
+    "dedupe_normalized_release_paths",
+    "infer_artifact_identity",
+    "infer_release_artifact_spec",
+    "logical_name_for_release_path",
+    "normalize_release_path",
+    "read_stage4_release_candidate_bundle",
+    "strip_staging_prefix",
+]