Skip to content

RELEASE.md

Latest commit

 

History

History
155 lines (100 loc) · 7.16 KB

RELEASE.md

File metadata and controls

155 lines (100 loc) · 7.16 KB
Raw

Release (Foundry)

This repo is public. Do not put real Foundry URLs, dataset RIDs, tokens, or API keys in git.

Preconditions

  • A pipeline-mode Compute Module is created in Foundry
  • Input dataset and output dataset exist (and their schemas are decided)
  • You know the alias names you will configure in the pipeline (e.g. input, output)

Required Configuration

Environment Variables

Foundry injects:

  • BUILD2_TOKEN (file path)
  • RESOURCE_ALIAS_MAP (file path)

Foundry service discovery should provide:

  • FOUNDRY_SERVICE_DISCOVERY_V2: file path containing api_gateway and stream_proxy base URLs

You must provide (via compute module configuration):

  • GEMINI_MODEL: Gemini model name (do not hardcode in code; configure per environment)

FOUNDRY_URL remains supported as a local-dev shorthand when service discovery is unavailable.

Gemini API key (recommended: Foundry Sources):

  • Option A (Sources, recommended): configure a Source that stores the Gemini key and injects egress policy.
    • Foundry will mount a credentials JSON file and set SOURCE_CREDENTIALS to its file path.
    • Configure:
      • GEMINI_SOURCE_API_NAME: the Source "API name" you configured in Data Connection
      • GEMINI_SOURCE_SECRET_NAME: the secret name inside that Source (optional if the key can be inferred)
  • Option B (env var / secret): set GEMINI_API_KEY directly.
    • GEMINI_API_KEY may be the literal key or a file path containing the key.

Optional knobs:

  • WORKERS (int)
  • MAX_RETRIES (int)
  • REQUEST_TIMEOUT (duration)
  • FAIL_FAST (bool)
  • RATE_LIMIT_RPS (float)
  • GEMINI_CAPTURE_AUDIT (bool)
  • GEMINI_BASE_URL (string; optional base URL override for proxies/testing, not recommended in Foundry)

Output Write Semantics (Pipeline Mode)

This module supports two output types:

  • Snapshot dataset output (dataset transactions + file upload)
  • Stream output (stream-proxy JSON record publish)

By default, the binary runs in --output-write-mode=auto, which probes stream-proxy to decide which write path to use.

Dataset Output (Transactions)

When executing in pipeline mode with a snapshot dataset output, Foundry may open a transaction on the configured output dataset for the duration of the build. During this time you may be unable to create a new transaction on that output dataset/branch.

This binary handles that by:

  • Attempting to create a transaction
  • If Foundry responds with OpenTransactionAlreadyExists, listing transactions (preview endpoint) and using the latest OPEN transaction
  • Uploading the output file into that transaction
  • If the transaction was created by Foundry (the OpenTransactionAlreadyExists case), do not commit; Foundry will commit as part of the build. If the binary created the transaction (local harness), it will commit after a successful upload.

Stream Output (Stream-Proxy)

If the configured output is a stream, this binary writes output rows through the legacy stream-proxy backend. The backend boundary is foundryio.StreamBackend; high-scale streams support is not implemented yet.

Egress Policy

Compute modules run in a zero-trust network model: by default they have no external network access (including other Foundry services). You must explicitly configure Sources (network policies) for any network access the module needs.

At minimum, expect to allowlist:

  • generativelanguage.googleapis.com (Gemini API)

Confirm exact domains from the client library / runtime behavior before locking the policy.

In addition, if your module calls Foundry REST APIs (this repo does), you should plan to allow access to the Foundry service hosts resolved by FOUNDRY_SERVICE_DISCOVERY_V2 (or the FOUNDRY_URL shorthand in local development). This does not mean "leaving Foundry"; it is simply allowing the container to make HTTPS requests to the stack's API gateway and stream-proxy.

Practical pattern:

  • Create a REST API Source pointing at your Foundry stack hostname (e.g. <stack>.palantirfoundry.com:443)
  • Enable code import for compute modules on that Source
  • Attach that Source to the compute module alongside the Gemini Source

Note: for Sources used in code alongside Foundry inputs/outputs, you may need to enable Source exports and allow the required markings/org.

Compute Module Probes + Logs

Two common pitfalls when iterating on compute modules:

  • Stdout log capture: Foundry's stdout log capture can require /bin/sh and tee. Very minimal images (e.g. distroless) may run fine but produce no logs.
  • Readiness probe: avoid probes like echo (not present in distroless). Prefer an exec probe that calls your binary directly, for example ['/enricher','--help'].

Streaming Output Idempotency

Streaming outputs are append-only. If the compute module is restarted/redeployed, the container may re-run the pipeline and republish the same records.

Options:

  • Write to a snapshot dataset instead of a stream for one-shot jobs.
  • Add idempotency/deduplication in your stream sink (e.g. publish a stable key or filter against a checkpoint/state dataset).
  • Keep the compute module process alive after a successful run so the platform doesn't restart it and duplicate writes.

Image Publishing

This repo publishes compute-module images to Foundry via .github/workflows/publish-foundry.yml. Release version bumps and Git tag creation are automated by .github/workflows/release-version.yml.

GitHub Secrets Required

  • FOUNDRY_ARTIFACT_REPOSITORY_RID: artifact repository RID used as the docker login username.
  • FOUNDRY_TOKEN: token used as the docker login password.
  • FOUNDRY_REGISTRY_HOST: Foundry container registry host (example: <your-stack>-container-registry.palantirfoundry.com).
  • FOUNDRY_DOCKER_IMAGE_NAME: image name in the registry (example: email-enrichment-google).

FOUNDRY_TOKEN must be current when publishing. An expired token will fail the registry login/push stage even if all build and test stages pass.

Tagging Behavior

  • Pull requests run a Foundry-image build only; they do not push to the Foundry registry.
  • After successful CI on main, release-version bumps internal/version/version.go with a bot PR, auto-merges with a [skip ci] squash commit, and creates tag vX.Y.Z from that merge commit.
  • Release tags v* trigger publish-foundry.yml, which publishes image tag <X.Y.Z> (the leading v is removed).
  • The separate ci.yml Docker job builds the public GHCR image and pushes only on push events.

Permissions

For push to succeed, the CI identity backing FOUNDRY_TOKEN needs Edit on the target Artifact repository. View alone can pull but cannot push. For automated version bumps, the repository must allow GitHub Actions to open and merge PRs on main (or grant equivalent bypass/merge permissions for github-actions[bot]).

Rollout Checklist

  • First run against a tiny fixture (5-10 emails)
  • Validate output schema matches the dataset schema exactly
  • Validate logs contain no tokens/API keys and avoid raw emails
  • Validate failure policy (fail-fast vs partial output) matches pipeline expectations
  • Confirm egress policy is minimal and sufficient

Rollback

  • Revert compute module image tag to the previously known-good tag
  • Re-run pipeline build