Problem
A code-only deploy can pass PR CI but fail later in Release to Modal because PR CI runs tests in the full development environment, while release deploys lightweight gateway runtimes with smaller dependency sets.
This happened in the 0.25.0 observability release. PR CI passed, but the release run failed in staging:
The deployed logs showed startup import failures:
- Modal gateway:
ModuleNotFoundError: No module named 'numpy'
- Import path:
modal_release/gateway_app.py -> modal_release/gateway.py -> observability/flask.py -> utils.config_loader -> utils/__init__.py -> utils/json.py.
utils/json.py imports NumPy, but the Modal gateway image is intentionally lightweight and does not install NumPy.
- Cloud Run gateway:
ModuleNotFoundError: No module named 'policyengine_observability'
failover/cloud_run_gateway.py imports observability code, but gcp/cloud_run/gateway.Dockerfile did not install policyengine-observability[flask].
Why PR CI missed it
.github/workflows/pr.yml installs the full project and runs unit tests. That environment includes dependencies that the lightweight gateway images do not include, so import errors in the deployable artifacts were masked.
The release workflow is the first place that actually builds/deploys the minimal Modal gateway and Cloud Run gateway runtimes and checks their endpoints.
There is also a smaller drift: PR CI uses Python 3.12, while release uses Python 3.13.
Proposed safeguard
Add targeted PR-level smoke checks for deployable runtime bootability. These should be path-triggered and cheap, not a full staging release.
Suggested checks:
-
Modal gateway minimal import smoke
- Create an isolated environment matching
household_api_gateway_image() dependencies.
- Import
policyengine_household_api.modal_release.gateway_app or policyengine_household_api.modal_release.gateway.
- This would catch imports that pull in dependencies missing from the lightweight Modal gateway image.
-
Cloud Run gateway Docker boot smoke
- Build
gcp/cloud_run/gateway.Dockerfile.
- Run the image locally in CI.
- Curl
/liveness_check.
- This would catch missing Dockerfile dependencies before merge.
-
Align PR CI Python with release Python where possible (3.13).
Suggested path triggers include:
policyengine_household_api/modal_release/**
policyengine_household_api/failover/**
policyengine_household_api/observability/**
policyengine_household_api/utils/**
gcp/cloud_run/gateway.Dockerfile
gcp/cloud_run/gateway_start.sh
.github/scripts/cloud-run-deploy-failover.sh
.github/workflows/deploy-staged.yml
Non-goals
This should not run the full Release to Modal workflow on every PR. The goal is only to catch basic artifact boot/import failures before merge. Release CI should still own real staging deploys, deployed integration tests, Cloud Run fallback validation, and production gating.
Problem
A code-only deploy can pass PR CI but fail later in
Release to Modalbecause PR CI runs tests in the full development environment, while release deploys lightweight gateway runtimes with smaller dependency sets.This happened in the
0.25.0observability release. PR CI passed, but the release run failed in staging:Integration tests against Modal stagingtimed out loading/versions/us.Deploy Cloud Run failover to stagingdeployed the gateway but/liveness_checkreturned HTTP 503.The deployed logs showed startup import failures:
ModuleNotFoundError: No module named 'numpy'modal_release/gateway_app.py->modal_release/gateway.py->observability/flask.py->utils.config_loader->utils/__init__.py->utils/json.py.utils/json.pyimports NumPy, but the Modal gateway image is intentionally lightweight and does not install NumPy.ModuleNotFoundError: No module named 'policyengine_observability'failover/cloud_run_gateway.pyimports observability code, butgcp/cloud_run/gateway.Dockerfiledid not installpolicyengine-observability[flask].Why PR CI missed it
.github/workflows/pr.ymlinstalls the full project and runs unit tests. That environment includes dependencies that the lightweight gateway images do not include, so import errors in the deployable artifacts were masked.The release workflow is the first place that actually builds/deploys the minimal Modal gateway and Cloud Run gateway runtimes and checks their endpoints.
There is also a smaller drift: PR CI uses Python 3.12, while release uses Python 3.13.
Proposed safeguard
Add targeted PR-level smoke checks for deployable runtime bootability. These should be path-triggered and cheap, not a full staging release.
Suggested checks:
Modal gateway minimal import smoke
household_api_gateway_image()dependencies.policyengine_household_api.modal_release.gateway_apporpolicyengine_household_api.modal_release.gateway.Cloud Run gateway Docker boot smoke
gcp/cloud_run/gateway.Dockerfile./liveness_check.Align PR CI Python with release Python where possible (
3.13).Suggested path triggers include:
Non-goals
This should not run the full
Release to Modalworkflow on every PR. The goal is only to catch basic artifact boot/import failures before merge. Release CI should still own real staging deploys, deployed integration tests, Cloud Run fallback validation, and production gating.