Skip to content

Add PR-level smoke checks for lightweight gateway runtimes #1583

Description

@hua7450

Problem

A code-only deploy can pass PR CI but fail later in Release to Modal because PR CI runs tests in the full development environment, while release deploys lightweight gateway runtimes with smaller dependency sets.

This happened in the 0.25.0 observability release. PR CI passed, but the release run failed in staging:

The deployed logs showed startup import failures:

  • Modal gateway: ModuleNotFoundError: No module named 'numpy'
    • Import path: modal_release/gateway_app.py -> modal_release/gateway.py -> observability/flask.py -> utils.config_loader -> utils/__init__.py -> utils/json.py.
    • utils/json.py imports NumPy, but the Modal gateway image is intentionally lightweight and does not install NumPy.
  • Cloud Run gateway: ModuleNotFoundError: No module named 'policyengine_observability'
    • failover/cloud_run_gateway.py imports observability code, but gcp/cloud_run/gateway.Dockerfile did not install policyengine-observability[flask].

Why PR CI missed it

.github/workflows/pr.yml installs the full project and runs unit tests. That environment includes dependencies that the lightweight gateway images do not include, so import errors in the deployable artifacts were masked.

The release workflow is the first place that actually builds/deploys the minimal Modal gateway and Cloud Run gateway runtimes and checks their endpoints.

There is also a smaller drift: PR CI uses Python 3.12, while release uses Python 3.13.

Proposed safeguard

Add targeted PR-level smoke checks for deployable runtime bootability. These should be path-triggered and cheap, not a full staging release.

Suggested checks:

  1. Modal gateway minimal import smoke

    • Create an isolated environment matching household_api_gateway_image() dependencies.
    • Import policyengine_household_api.modal_release.gateway_app or policyengine_household_api.modal_release.gateway.
    • This would catch imports that pull in dependencies missing from the lightweight Modal gateway image.
  2. Cloud Run gateway Docker boot smoke

    • Build gcp/cloud_run/gateway.Dockerfile.
    • Run the image locally in CI.
    • Curl /liveness_check.
    • This would catch missing Dockerfile dependencies before merge.
  3. Align PR CI Python with release Python where possible (3.13).

Suggested path triggers include:

policyengine_household_api/modal_release/**
policyengine_household_api/failover/**
policyengine_household_api/observability/**
policyengine_household_api/utils/**
gcp/cloud_run/gateway.Dockerfile
gcp/cloud_run/gateway_start.sh
.github/scripts/cloud-run-deploy-failover.sh
.github/workflows/deploy-staged.yml

Non-goals

This should not run the full Release to Modal workflow on every PR. The goal is only to catch basic artifact boot/import failures before merge. Release CI should still own real staging deploys, deployed integration tests, Cloud Run fallback validation, and production gating.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions