Merge remote-tracking branch 'origin/main' into agent/stage-1/pr-1-specs-foundation

Author: anth-volk

changelog.d/1022.added

@@ -0,0 +1 @@
+Added typed Stage 5 release candidate schemas and Stage 4 candidate bundle readers.

docs/engineering/skills/README.md

@@ -24,3 +24,10 @@ Current skills:
 Stage-specific AI-facing engineering guides live under `docs/engineering/stages/`.
 Use them alongside these cross-cutting skills when modifying a stage-specific
 pipeline path.
+
+Current stage guides:
+
+- `build_outputs.md`: Stage 4 output-build library boundaries and test
+  expectations.
+- `release_promotion.md`: Stage 5 release candidate identity, validation-report
+  schema, rerun comparison material, and side-effect boundaries.

docs/engineering/stages/release_promotion.md

@@ -0,0 +1,92 @@
+# Release Promotion Stage AI Guide
+
+This guide is for AI agents and maintainers modifying Stage 5
+(`5_validate_and_promote_release`) code. Stage 5 validates a staged release
+candidate, promotes the exact candidate to public Hugging Face and GCS
+destinations, writes release/version/completion metadata, and cleans staging
+only after completion is certified.
+
+## Candidate Identity
+
+Use `policyengine_us_data.release_promotion.ReleasePromotionContext` as the
+typed Stage 5 identity boundary. The context must keep these values distinct:
+
+- `run_id`: the canonical publication run correlation key.
+- `candidate_version`: the candidate staging scope used in Hugging Face staging
+  paths such as `staging/{candidate_version}-{run_id}/...`.
+- `release_version`: the final stable public release version.
+- `base_release_version` and `release_bump`: optional provenance for how the
+  candidate scope was chosen.
+
+Do not resolve a different run ID from the environment inside lower-level
+release-promotion logic. Environment resolution belongs at orchestration edges;
+Stage 5 library code should receive explicit context.
+
+## Release Candidate Bundles
+
+Use `ReleaseCandidateInputBundle` to describe the artifacts Stage 5 is allowed
+to validate and promote. Each artifact should be represented by a
+`ReleaseArtifactSpec` with a production-relative path, artifact family, source
+stage, and optional checksum/size metadata.
+
+The current compatibility path may build a bundle from the legacy staged path
+set produced by Modal orchestration. Mark that reader as compatibility-only and
+keep it retirable.
+
+The Stage 4 contract/inventory reader API now exists for migration work:
+`build_release_candidate_bundle_from_stage4_contract()` accepts an in-memory
+Stage 4 contract plus inventory records, and
+`read_stage4_release_candidate_bundle()` reads the same shape from files.
+Production Stage 5 code should not depend on Stage 4 contracts until the
+contract and inventory are canonical, complete, and populated with semantic
+artifact identity plus checksum/size material.
+
+Candidate bundles may record validation reports as path-only
+`validation_report_paths` for compatibility. When Stage 4 or another upstream
+producer can provide report checksums, prefer `validation_report_refs` with
+canonical `DiagnosticRef` / `ArtifactRef` identity so rerun comparison can
+distinguish an overwritten report at the same diagnostics path.
+
+## Validation Reports
+
+Stage 5 must use the shared validation schema for durable validation output:
+
+- `policyengine_us_data.stage_contracts.ValidationReport`
+- `policyengine_us_data.stage_contracts.ValidationFinding`
+- `policyengine_us_data.stage_contracts.DiagnosticRef`
+
+Do not create a Stage 5-specific durable validation report, check, finding, or
+error schema for contracts, diagnostics, release candidates, status endpoints,
+or step manifests. Release-specific details such as missing staged artifacts,
+missing validation reports, finalized-release conflicts, version mismatches, or
+destination conflicts should live in canonical finding metadata.
+
+## Rerun Comparison Material
+
+Before public writes, rerun and reuse decisions should compare semantic
+candidate identity rather than only checking whether output files exist. The
+comparison material should include:
+
+- run ID, candidate version, release version, HF repository, and GCS bucket;
+- Stage 4 output contract fingerprint when available;
+- output inventory paths/checksums when available;
+- validation report paths and `DiagnosticRef` checksum identities when
+  available;
+- expected production-relative artifact paths;
+- the Stage 5 candidate bundle fingerprint.
+
+When required artifacts only have paths and no checksum/size identity, treat
+the bundle as path-only and do not use its fingerprint for promotion reuse
+decisions.
+
+Already-finalized releases are an idempotency case, not a shortcut around
+candidate identity. A finalized release can be reused only when its completion
+marker is valid and it matches the requested candidate.
+
+## Side Effects
+
+Candidate builders, schema adapters, and rerun comparison helpers should not
+perform Hugging Face writes, GCS uploads, Modal calls, staging cleanup, or
+release-manifest publication. Keep those operations behind explicit adapters or
+services so tests can exercise candidate shape and validation logic without
+credentials or network access.

docs/pipeline_map.yaml

@@ -759,7 +759,7 @@ stages:
     node_type: artifact
     description: Policy target database copied into the pipeline volume
   - id: hf_staging_base_s1g
-    label: HuggingFace staging/{candidate_version}/{run_id}
+    label: HuggingFace staging/{candidate_version}-{run_id}
     node_type: external
     description: Run-scoped staging prefix for base datasets
   - id: stage_base_datasets
@@ -1554,7 +1554,7 @@ stages:
     node_type: artifact
     description: Output set from substage 5a
   - id: hf_staging_s5b
-    label: HuggingFace staging/{candidate_version}/{run_id}
+    label: HuggingFace staging/{candidate_version}-{run_id}
     node_type: external
     description: Run-scoped staging prefix containing validated artifacts
   - id: out_hf_prod

policyengine_us_data/release_promotion/__init__.py

@@ -0,0 +1,44 @@
+"""Typed Stage 5 release promotion boundaries.
+
+This package starts with release-candidate identity and candidate-bundle
+schemas. Promotion side effects still live in the existing transaction engine
+until later Stage 5 migration slices move them behind typed services.
+"""
+
+from .artifacts import (
+    BASE_RELEASE_ARTIFACT_PATHS,
+    ReleaseArtifactSpec,
+    dedupe_normalized_release_paths,
+    infer_artifact_identity,
+    infer_release_artifact_spec,
+    logical_name_for_release_path,
+    normalize_release_path,
+    strip_staging_prefix,
+)
+from .candidate import (
+    ReleaseCandidateInputBundle,
+)
+from .candidate_builders import build_legacy_release_candidate_bundle
+from .context import ReleasePromotionContext
+from .stage4_reader import (
+    build_release_candidate_bundle_from_stage4_contract,
+    read_stage4_release_candidate_bundle,
+)
+from .validation import build_release_candidate_shape_report
+
+__all__ = [
+    "BASE_RELEASE_ARTIFACT_PATHS",
+    "ReleaseArtifactSpec",
+    "ReleaseCandidateInputBundle",
+    "ReleasePromotionContext",
+    "build_legacy_release_candidate_bundle",
+    "build_release_candidate_bundle_from_stage4_contract",
+    "build_release_candidate_shape_report",
+    "dedupe_normalized_release_paths",
+    "infer_artifact_identity",
+    "infer_release_artifact_spec",
+    "logical_name_for_release_path",
+    "normalize_release_path",
+    "read_stage4_release_candidate_bundle",
+    "strip_staging_prefix",
+]

policyengine_us_data/release_promotion/artifact_uris.py

@@ -0,0 +1,81 @@
+"""URI parsing helpers for Stage 5 release candidate artifacts."""
+
+from __future__ import annotations
+
+from urllib.parse import ParseResult, urlparse
+
+from .artifacts import (
+    BASE_RELEASE_ARTIFACT_PATHS,
+    normalize_release_path,
+    strip_staging_prefix,
+)
+from .context import ReleasePromotionContext
+
+
+def release_path_from_artifact_uri(
+    uri: str,
+    *,
+    context: ReleasePromotionContext,
+) -> str | None:
+    """Return a release-relative path from a staged artifact URI."""
+
+    parsed = urlparse(uri)
+    if parsed.scheme or parsed.netloc:
+        validate_uri_repo(parsed, context)
+    raw_path = parsed.path.lstrip("/") if parsed.scheme else uri
+    candidate_paths = [raw_path]
+    if parsed.netloc:
+        candidate_paths.append(f"{parsed.netloc}/{raw_path}")
+    for candidate in candidate_paths:
+        if context.hf_staging_prefix and context.hf_staging_prefix in candidate:
+            return strip_staging_prefix(
+                candidate[candidate.index(context.hf_staging_prefix) :],
+                context.hf_staging_prefix,
+            )
+        if parsed.scheme or parsed.netloc:
+            if contains_release_artifact_path(candidate):
+                raise ValueError(
+                    "external artifact URI must point under the expected staging prefix"
+                )
+            continue
+        if candidate.startswith("staging/"):
+            return strip_staging_prefix(candidate, context.hf_staging_prefix)
+        for prefix in ("states/", "districts/", "cities/", "national/"):
+            if prefix in candidate:
+                return normalize_release_path(candidate[candidate.index(prefix) :])
+        for path in BASE_RELEASE_ARTIFACT_PATHS:
+            if candidate == path or candidate.endswith(f"/{path}"):
+                return normalize_release_path(
+                    candidate[candidate.rindex(path) :],
+                )
+    return None
+
+
+def contains_release_artifact_path(candidate: str) -> bool:
+    """Return whether a path-shaped string names a release artifact."""
+
+    if any(
+        prefix in candidate
+        for prefix in ("states/", "districts/", "cities/", "national/")
+    ):
+        return True
+    return any(
+        candidate.endswith(f"/{path}") or candidate == path
+        for path in BASE_RELEASE_ARTIFACT_PATHS
+    )
+
+
+def validate_uri_repo(
+    parsed_uri: ParseResult,
+    context: ReleasePromotionContext,
+) -> None:
+    """Require external artifact URIs to reference the expected HF repository."""
+
+    if not parsed_uri.scheme or not parsed_uri.netloc:
+        return
+    path_parts = parsed_uri.path.strip("/").split("/")
+    if not path_parts or not path_parts[0]:
+        return
+    repo_name = f"{parsed_uri.netloc}/{path_parts[0]}"
+    if repo_name != context.hf_repo_name:
+        raise ValueError("external artifact URI repo must match context.hf_repo_name")