feat: migrate app pipelines to release-resource based workflow

[blarghmatey]

What are the relevant tickets?

N/A — linked to discussion https://github.com/mitodl/hq/discussions/10465

Description (What does it do?)

Replaces the Doof-era release-candidate/release git branch model for app deployments with a Concourse-native release workflow.

Before: Doof pushed commits to release-candidate to trigger RC builds, and pushed to release to trigger production deploys.

After: A dedicated release Concourse resource (from the ol-concourse release resource) drives the full lifecycle:

1. Release resource triggers build-{app}-release-image
   • Runs bump_version_task to update version strings in app source
   • Puts the release resource (action: create) to commit the version bump and create a release branch + tag in GitHub
   • Builds and pushes a versioned Docker image to DockerHub and ECR
2. New versioned image triggers QA Pulumi deploy
   • Before deploy: starts a GitHub Deployment for the RC environment
   • After deploy: opens a GitHub Release Issue (body from checklist.md) and marks the RC GitHub Deployment as successful
3. Release manager closes the GitHub Issue to gate production
   • The release-gate resource (github-issues, closed state) triggers production when the issue is closed
   • Before deploy: starts a GitHub Deployment for the Production environment
   • After deploy: marks the Production GitHub Deployment as successful

Key changes to docker_pulumi.py:

• AppPipelineParams: removed repo_rc_branch/repo_release_branch; added github_repo (defaults to mitodl/{repo_name})
• _define_git_resources: simplified to main + ol-infra repos only
• Added _define_release_resources: creates release, release-gate, release-issue, and GitHub Deployment resources
• Replaced _build_image_job(branch_type=release_candidate) with _build_release_image_job
• pulumi_jobs_chain wired with custom_dependencies (per-stack) and additional_post_steps for GitHub Deployments and Release Issues
• Imports release_resource_type and github_deployments_resource from ol-concourse 0.7.0 (already pinned as dependency)

How can this be tested?

Generated pipeline JSON was validated by running the script for mitxonline and verifying the jobs/resources/resource_types structure is correct. A test pipeline was created at https://cicd.odl.mit.edu/teams/infrastructure/pipelines/docker-pulumi-micromasters-relesae-test

To validate locally:

uv run python src/ol_concourse/pipelines/infrastructure/k8s_apps/docker_pulumi.py mitxonline
# Inspect definition.json for expected jobs, resources, and resource_types

Additional Context

This PR requires ol-concourse 0.7.0 (which adds release_resource_type()), already reflected in pyproject.toml. App repos will need bumpver configured for bump_version_task to succeed — teams will be notified separately.

[Copilot]

Pull request overview

Migrates the k8s app docker+Pulumi Concourse pipeline generator away from the legacy release-candidate/release branch promotion model to a release-resource-driven workflow (release creation, QA deploy, GitHub issue gate, production deploy).

Changes:

• Simplifies git resources to just app main + ol-infrastructure, removing RC/release branch git resources.
• Introduces release workflow resources (release, release-gate, release-issue, GitHub Deployments) and a new build-{app}-release-image job that bumps version + creates the release + publishes versioned images.
• Rewires QA/Production Pulumi deployment chain to use the release resources for checklist/issue creation and deployment status reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

release_gate matches any closed issue with prefix/title Release {app_name} and Production uses only that as the manual gate. Because the issue title/prefix don’t include the release version, closing an older outstanding release issue (or having multiple releases in flight) can promote the wrong image tag to production. Include the version in the issue title/prefix (e.g., Release {app_name} vX.Y.Z) and/or pass the gated version through so Production deploys the exact release being approved.

[blarghmatey]

Good catch — this is a real design risk worth documenting clearly. Here's the detailed breakdown and the options for addressing it.

The concrete failure scenario

1. Release 0.24.0 ships: QA deploys, issue "Release {app}" is opened, team closes it → Production deploys 0.24.0. ✅
2. Release 0.25.0 ships the next day. QA deploys, a new "Release {app}" issue is opened.
3. Someone notices the still-open 0.24.0 issue from step 1 was never closed (or an old abandoned release issue exists) and closes it.
4. release_gate fires on that closure → Production deploys... but which image? The shared dependencies image GetStep has passed=[qa_job_name], so Concourse picks the latest image version that successfully passed through QA — which at this point is 0.25.0, not 0.24.0. So closing the stale 0.24.0 issue accidentally triggers a 0.25.0 prod deploy before the team signed off.

The passed constraint prevents deploying untested images, but it doesn't pin the gate to the specific release being approved.

Partial mitigation already in place

• The skip_if_labeled: [abandoned] on release_gate handles the explicit-abandonment case.
• Any prod deploy triggered prematurely still deploys a QA-validated image, so correctness isn't broken — just the timing control is.

Approaches to fully resolve it

Option A — Include version in the issue title (recommended)

Pass ((.:release_version)) into the release_issue put step's title so issues are named "Release {app} v0.25.0". The release_gate resource would need a matching issue_prefix, but since the prefix must be static at pipeline-generation time, this only works if we update the github_issues resource to support a runtime-variable prefix — or if we use set_pipeline to re-emit the pipeline with the version baked in (complex).

Verdict: Clean conceptually but requires either a dynamic pipeline re-emit or a github_issues resource that supports runtime prefixes. Not straightforward today.

Option B — Use version: every on the release_gate

Add version="every" to GetStep(get=release_gate.name). This tells Concourse to process each new closed-issue version separately rather than jumping to the latest. Each release issue closure would fan-trigger at most one Production deployment. It doesn't fully prevent the wrong-issue problem but ensures no closure is silently skipped or batched with another.

Verdict: Low-effort improvement, reduces (but doesn't eliminate) the race. Easy to add now.

Option C — Make release_gate the same resource as release_issue with passed constraint

If release_gate and release_issue are the same underlying resource (same name), a GetStep on the gate with passed=[qa_job_name] would require the specific issue version that was created in QA to now be in the closed state. This is the most watertight solution.

Concretely: rename release_issue to e.g. {app}-release-checklist and make release_gate the single canonical github_issues resource (open when written, closed when gated). Use passed on the Production gate GetStep to pin it to the version created by the QA PutStep.

Verdict: Architecturally correct and idiomatic Concourse. Requires restructuring the resource names so QA writes and Production reads the same versioned resource.

Option D — Accept and document (current state)

The risk only materialises when a stale release issue is closed out-of-band. The skip_if_labeled: [abandoned] label discipline partially covers this. Documenting the operational rule ("always label un-merged release issues as 'abandoned' before closing") reduces real-world risk to near-zero without code changes.

Verdict: Acceptable for an initial rollout; Option B (version: every) is a cheap complement.

---

My recommendation: land Option B as a quick hardening measure in this PR, and track Option C as a follow-up once the pipeline is running in production and we can verify the passed-constraint approach works with the github_issues resource.

[blarghmatey]

Update: Option C is not viable with this resource.

After verifying with the resource implementation: the github_issues resource version is purely ID-based (derived from the issue number/ID), and is not affected by state transitions. Opening and closing the same issue produce the same version fingerprint.

This breaks Option C completely. Here's the failure mode:

1. QA job: PutStep creates issue Bump boto3 from 1.17.69 to 1.17.70 #142, implicit get registers version {id: 142} as consumed by the QA job.
2. Production job: GetStep(passed=[qa_job]) asks Concourse: "has version {id: 142} been seen by the QA job?" — YES, immediately, because QA just consumed it.
3. Production triggers right after QA finishes. The gate is completely bypassed.

The passed mechanism works when a version needs to flow through jobs sequentially (e.g., an image built in job A used in job B). It cannot model "wait for this specific thing to change state" because state changes don't produce new versions here.

The two-resource architecture is therefore correct by necessity. release_gate must be a separate resource watching for newly-closed issues because that state transition is the only event that produces a new version for Concourse to trigger on. There is no single-resource equivalent.

What would be needed for Option C to work: the github_issues resource would need to version on state transitions — e.g., version {id: 142, state: open} and {id: 142, state: closed} as distinct versions. That would require a change to the resource itself, which is out of scope here.

Current best posture: Option B (version: every, already implemented) + operational discipline with the abandoned label. This is the ceiling of what's achievable without modifying the resource type.