feat(experiments): route metrics recalculation to its own temporal queue

[rodrigoi]

Problem

The experiment metrics recalculation workflow runs on the shared general-purpose-task-queue, so its activities compete for worker slots with unrelated workflows on that queue. A burst of recalcs can starve the other workloads, and the shared worker's concurrency caps how wide a single recalc can fan out. That fan-out was pinned at 4 concurrent metrics, which makes recalculating an experiment with many metrics slow.

This routes recalc to its own dedicated queue and worker so it is isolated from the general queue and can fan out wider per run.

Changes

Moves the metrics recalculation workflow off general-purpose-task-queue onto a dedicated experiments-recalculation-task-queue, served by a new worker (provisioned in the companion charts PR). The timeseries recalculation workflow and the precompute canary stay on the general queue.

Queue routing

Adds the EXPERIMENTS_RECALCULATION_TASK_QUEUE setting and registers the recalc workflow and activities on it in start_temporal_worker.py, removing them from the general-purpose entry. The workflow start in the metrics_recalculation view and the metrics interceptor's task_queue both point at the new queue; the interceptor must match the registration queue or is_task_queue_supported filters it out and Prometheus metrics stop emitting.

• posthog/settings/temporal.py
• posthog/management/commands/start_temporal_worker.py
• products/experiments/backend/presentation/views.py
• products/experiments/backend/temporal/recalculation_metrics.py
• products/experiments/backend/temporal/test_recalculation_metrics.py

Wider per-run fan-out

Raises MAX_CONCURRENT_METRICS from 4 to 14. The dedicated worker bounds total ClickHouse load across concurrent runs, so a single run can fan out wide enough to finish most experiments in one concurrent wave. The old cap of 4 existed because recalc shared the general queue's ClickHouse budget; that constraint no longer applies.

• products/experiments/backend/temporal/recalculation_workflow.py

CI image wiring

Adds a change-detection step and a repository-dispatch to PostHog/charts for the temporal-worker-experiments-recalculation release, so the new worker's image state tracks builds independently. Recalc's temporal path moves off the general-purpose worker's change filter since it no longer runs there. There is no new image build; every temporal worker shares the posthog-cloud image and only the dispatched release name differs.

• .github/workflows/container-images-cd.yml

Rollback

Revert this PR to route recalc back to general-purpose-task-queue. The workflow itself is unchanged, only the queue it runs on and the fan-out width.

Deploy order

The companion charts PR that stands up the dedicated worker must merge and the worker must be healthy before this PR merges. If this merges first, recalc workflows start but their activities sit on a queue no worker polls until the worker exists.

Recalc activities are removed from the general-purpose worker here, so a recalc run already in-flight on general-purpose-task-queue at the cutover keeps its activities on that queue (activities inherit the workflow's task queue) and can stall once the general worker stops registering them. Recalc runs are short and user-triggered, so either let in-flight runs drain before rolling the general worker, or accept that the few runs straddling the cutover need re-triggering. New runs started after this deploys are unaffected.

How did you test this code?

pnpm turbo run backend:test --filter=@posthog/products-experiments

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Model: Opus 4.8
Manually refactored: no

Skills used:

• /query-clickhouse-via-metabase (local)
• /writing-pull-requests (local)

Relevant decisions:

• Raised MAX_CONCURRENT_METRICS to 14 so a typical experiment's metrics recalculate in a single concurrent wave. The dedicated worker's activity-slot cap, set in the charts PR, bounds total ClickHouse load across concurrent runs.
• Kept the timeseries recalc workflow and the precompute canary on general-purpose-task-queue; only the metrics recalc workflow moves, since only it needs the isolation and wider fan-out.
• The CI dispatch keeps the new worker's image current, but the worker already boots on the shared posthog-cloud image via its seeded charts state file, so queue routing works from the first deploy.

[rodrigoi]

• chore(ci): build a dedicated experiments recalculation worker image #67261
• feat(experiments): route metrics recalculation to its own temporal queue #67257 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[greptile-apps]

_{Reviews (1): Last reviewed commit: "feat(experiments): configure the metrics..." | Re-trigger Greptile}

[greptile-apps]

_{Reviews (2): Last reviewed commit: "fix(experiments): updating misleading co..." | Re-trigger Greptile}

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-03 16:51 UTC	Run
prod-us	✅ Deployed	2026-07-03 17:03 UTC	Run
prod-eu	✅ Deployed	2026-07-03 17:08 UTC	Run