Skip to content

feat: add Prefect readiness init container#60

Open
PhillSimonds wants to merge 2 commits into
stablefrom
001-prefect-readiness-gate
Open

feat: add Prefect readiness init container#60
PhillSimonds wants to merge 2 commits into
stablefrom
001-prefect-readiness-gate

Conversation

@PhillSimonds

@PhillSimonds PhillSimonds commented Apr 17, 2026

Copy link
Copy Markdown

Fixes #59

Add an opt-out init container to infrahubTaskWorker and infrahubServer deployments that blocks main-container startup until Prefect's /api/health endpoint responds. Resolves the CrashLoopBackOff observed on fresh installs when task worker pods race ahead of the bundled prefect-server.

  • New waitForPrefect values block on both deployments (enabled=true, busybox:1.37, http://prefect-server:4200/api/health, pollSeconds=5, timeoutSeconds=300, explicit resource requests/limits)
  • Shared infrahub-helm.waitForPrefect.initContainer helper in _helpers.tpl so both deployments render the same shape
  • charts/infrahub: 4.20.2 -> 4.21.0 (MINOR, additive)
  • charts/infrahub-enterprise: 4.6.3 -> 4.6.4 (PATCH); infrahub subchart dep 4.20.2 -> 4.21.0 (enterprise inherits the gate transparently)
  • README values table updated with 18 new rows

Validated on OrbStack:

  • Community and enterprise default installs: zero httpx.ConnectError messages in task-worker or server logs
  • Gate-disabled render is byte-identical to pre-feature baseline outside the init-container region (only helm.sh/chart label and regenerated rabbitmq-erlang-cookie differ)
  • Unreachable gate surfaces Init:Error at exactly timeoutSeconds (30s test confirmed)

Note: one task-worker replica still observed restarting once on fresh installs from prefect.exceptions.ObjectAlreadyExists (a race between replicas creating the Prefect work pool). This is separate from the httpx.ConnectError symptom covered by this fix and warrants a follow-up issue.

Add an opt-out init container to infrahubTaskWorker and infrahubServer
deployments that blocks main-container startup until Prefect's
/api/health endpoint responds. Resolves the CrashLoopBackOff observed
on fresh installs when task worker pods race ahead of the bundled
prefect-server.

- New waitForPrefect values block on both deployments (enabled=true,
  busybox:1.37, http://prefect-server:4200/api/health, pollSeconds=5,
  timeoutSeconds=300, explicit resource requests/limits)
- Shared infrahub-helm.waitForPrefect.initContainer helper in
  _helpers.tpl so both deployments render the same shape
- charts/infrahub: 4.20.2 -> 4.21.0 (MINOR, additive)
- charts/infrahub-enterprise: 4.6.3 -> 4.6.4 (PATCH); infrahub subchart
  dep 4.20.2 -> 4.21.0 (enterprise inherits the gate transparently)
- README values table updated with 18 new rows

Validated on OrbStack:
- Community and enterprise default installs: zero httpx.ConnectError
  messages in task-worker or server logs
- Gate-disabled render is byte-identical to pre-feature baseline
  outside the init-container region (only helm.sh/chart label and
  regenerated rabbitmq-erlang-cookie differ)
- Unreachable gate surfaces Init:Error at exactly timeoutSeconds
  (30s test confirmed)

Note: one task-worker replica still observed restarting once on
fresh installs from prefect.exceptions.ObjectAlreadyExists (a race
between replicas creating the Prefect work pool). This is separate
from the httpx.ConnectError symptom covered by this fix and warrants
a follow-up issue.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@PhillSimonds
PhillSimonds force-pushed the 001-prefect-readiness-gate branch from 791d6d5 to 792b56d Compare April 17, 2026 04:56
@PhillSimonds
PhillSimonds requested a review from fatih-acar April 17, 2026 04:57
@PhillSimonds PhillSimonds changed the title feat(infrahub): add Prefect readiness init container (fixes #59) feat: add Prefect readiness init container Apr 17, 2026
@fatih-acar

Copy link
Copy Markdown
Contributor

Thanks @PhillSimonds
Could we make the init container disabled by default instead?
This case usually only happens on the first install and is not supposed to happen during upgrades and app lifecycle, I'm afraid it could cause more troubles than it fixes.
Users could still make use of it by enabling it manually.

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 4.6.4

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should bump to 4.7.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Prefect Readiness gate to task worker and server startup

2 participants