fix(services): HealthChecker.Register immediately populates cache when already running by Fletch153 · Pull Request #1976 · smartcontractkit/chainlink-common

Fletch153 · 2026-04-10T21:24:50Z

Summary

Root cause: When Register() is called on an already-started HealthChecker (e.g. by the job spawner after starting a new job service), the new service's health state is absent from IsHealthy() / IsReady() for up to 15 seconds — the full polling interval — even though Ready() already returns nil.
Fix: Register() now calls IfStarted after adding the service, immediately populating c.ready and c.healthy for that service.
The servicesMu write lock is scoped to an inner closure and released before calling IfStarted to avoid a deadlock with Start(), which holds the StateMachine lock while calling update() (which acquires servicesMu.RLock).

Changes

pkg/services/health.go — Register() triggers an immediate single-service health check when the checker is already running
pkg/services/health_test.go — two new tests covering healthy and unhealthy services registered on a running checker

Test plan

go test ./pkg/services/... -run TestHealthChecker_Register — both new tests pass
go test ./pkg/services/... — no regressions

🤖 Generated with Claude Code

github-actions · 2026-04-10T21:26:12Z

⚠️ API Diff Results - `github.com/smartcontractkit/chainlink-common`

⚠️ Breaking Changes (7)

`package github` (1)

com/smartcontractkit/chainlink-common/script — 🗑️ Removed

`pkg/capabilities/v2/actions/confidentialrelay.SecretsRequestParams` (1)

OrgID — 🗑️ Removed

`pkg/capabilities/v2/actions/confidentialworkflow.(*WorkflowExecution)` (1)

GetOrgId — 🗑️ Removed

`pkg/capabilities/v2/actions/confidentialworkflow.WorkflowExecution` (1)

OrgId — 🗑️ Removed

`pkg/workflows` (3)

GenerateExecutionIDWithTriggerIndex — 🗑️ Removed
GetTriggerIndexFromReferenceID — 🗑️ Removed
GetTriggerReferenceID — 🗑️ Removed

✅ Compatible Changes (3)

`pkg/workflows/dontime/pb` (1)

Observations — ➕ Added

`pkg/workflows/dontime/pb.(*Observation)` (1)

GetPruneExecutions — ➕ Added

`pkg/workflows/dontime/pb.Observation` (1)

PruneExecutions — ➕ Added

📄 View full apidiff report

HealthChecker.Register() now signals the polling goroutine to run update() immediately when the checker is already running, eliminating the up-to-15-second window where a dynamically registered service (e.g. a spawned job) was absent from IsHealthy() / IsReady() even though Ready() already returned nil. Fixes CORE-2386. Companion chainlink-common PR: smartcontractkit/chainlink-common#1976

Replace the chRefresh channel mechanism with direct map population in Register(). Instead of signaling the run() goroutine to call update() (which re-checks ALL services), Register() now directly populates c.ready and c.healthy for just the newly registered service. This is simpler and avoids unnecessary re-polling of all services.

Fix missing servicesMu.Unlock() in Unregister that would cause a deadlock on every call. Revert unrelated whitespace alignment changes to struct definition and New() constructor.

HealthChecker.Register() now signals the polling goroutine to run update() immediately when the checker is already running, eliminating the up-to-15-second window where a dynamically registered service (e.g. a spawned job) was absent from IsHealthy() / IsReady() even though Ready() already returned nil. Fixes CORE-2386. Companion chainlink-common PR: smartcontractkit/chainlink-common#1976

Update chainlink-common to v0.11.2-0.20260413215716-a931c3f2f162 which is the merge commit of smartcontractkit/chainlink-common#1976 on main, replacing the pre-merge branch reference.

Bump chainlink-common to v0.11.2-0.20260413215716-a931c3f2f162 (merge commit of smartcontractkit/chainlink-common#1976) to resolve stale cache in HealthChecker.Register() (CORE-2386).

…running (#22056) * fix: bump chainlink-common to include HealthChecker Register fix Bump chainlink-common to v0.11.2-0.20260413215716-a931c3f2f162 (merge commit of smartcontractkit/chainlink-common#1976) to resolve stale cache in HealthChecker.Register() (CORE-2386). * chore: retrigger CI * ci: retrigger (runner shutdown on previous run) * fix: bump chainlink-deployments-framework to v0.95.0 in load/go.sum

Fletch153 requested a review from a team as a code owner April 10, 2026 21:24

product-security-plaid-production Bot requested a review from DylanTinianov April 10, 2026 21:25

Fletch153 temporarily deployed to integration April 10, 2026 21:25 — with GitHub Actions Inactive

Fletch153 temporarily deployed to integration April 10, 2026 21:33 — with GitHub Actions Inactive

Fletch153 force-pushed the fix/health-checker-register-immediate-update branch from f6b9ba9 to e76df7d Compare April 10, 2026 21:43

Fletch153 temporarily deployed to integration April 10, 2026 21:43 — with GitHub Actions Inactive

Fletch153 force-pushed the fix/health-checker-register-immediate-update branch from e76df7d to d2ac04e Compare April 10, 2026 21:52

Fletch153 temporarily deployed to integration April 10, 2026 21:53 — with GitHub Actions Inactive

Fletch153 force-pushed the fix/health-checker-register-immediate-update branch from d2ac04e to 07a72b4 Compare April 10, 2026 22:26

Fletch153 temporarily deployed to integration April 10, 2026 22:26 — with GitHub Actions Inactive

mchain0 previously approved these changes Apr 13, 2026

View reviewed changes

Fletch153 dismissed mchain0’s stale review via 1e41cda April 13, 2026 09:13

Fletch153 temporarily deployed to integration April 13, 2026 09:13 — with GitHub Actions Inactive

Fletch153 had a problem deploying to integration April 13, 2026 09:13 — with GitHub Actions Failure

Fletch153 dismissed jmank88’s stale review via 1fdf090 April 13, 2026 12:49

Fletch153 force-pushed the fix/health-checker-register-immediate-update branch from cbc6906 to 1fdf090 Compare April 13, 2026 12:49

Fletch153 had a problem deploying to integration April 13, 2026 12:49 — with GitHub Actions Error

Fletch153 force-pushed the fix/health-checker-register-immediate-update branch from 1fdf090 to 111cd0b Compare April 13, 2026 12:52

Fletch153 temporarily deployed to integration April 13, 2026 12:53 — with GitHub Actions Inactive

Fletch153 force-pushed the fix/health-checker-register-immediate-update branch from 111cd0b to 575541a Compare April 13, 2026 13:17

Fletch153 temporarily deployed to integration April 13, 2026 13:18 — with GitHub Actions Inactive

fix(services): restore Unregister unlock and revert cosmetic changes

faff2bf

Fix missing servicesMu.Unlock() in Unregister that would cause a deadlock on every call. Revert unrelated whitespace alignment changes to struct definition and New() constructor.

Fletch153 temporarily deployed to integration April 13, 2026 13:39 — with GitHub Actions Inactive

mchain0 approved these changes Apr 13, 2026

View reviewed changes

jmank88 approved these changes Apr 13, 2026

View reviewed changes

jmank88 added this pull request to the merge queue Apr 13, 2026

Merged via the queue into main with commit a931c3f Apr 13, 2026
30 of 32 checks passed

jmank88 deleted the fix/health-checker-register-immediate-update branch April 13, 2026 22:04

Fletch153 mentioned this pull request Apr 13, 2026

fix: HealthChecker.Register immediately populates cache when already running smartcontractkit/chainlink#21979

Closed

2 tasks

Fletch153 mentioned this pull request Apr 16, 2026

fix: HealthChecker.Register immediately populates cache when already running smartcontractkit/chainlink#22056

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(services): HealthChecker.Register immediately populates cache when already running#1976

fix(services): HealthChecker.Register immediately populates cache when already running#1976
jmank88 merged 2 commits intomainfrom
fix/health-checker-register-immediate-update

Fletch153 commented Apr 10, 2026

Uh oh!

github-actions Bot commented Apr 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Fletch153 commented Apr 10, 2026

Summary

Changes

Test plan

Uh oh!

github-actions Bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ API Diff Results - github.com/smartcontractkit/chainlink-common

⚠️ Breaking Changes (7)

package github (1)

pkg/capabilities/v2/actions/confidentialrelay.SecretsRequestParams (1)

pkg/capabilities/v2/actions/confidentialworkflow.(*WorkflowExecution) (1)

pkg/capabilities/v2/actions/confidentialworkflow.WorkflowExecution (1)

pkg/workflows (3)

✅ Compatible Changes (3)

pkg/workflows/dontime/pb (1)

pkg/workflows/dontime/pb.(*Observation) (1)

pkg/workflows/dontime/pb.Observation (1)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Apr 10, 2026 •

edited

Loading

⚠️ API Diff Results - `github.com/smartcontractkit/chainlink-common`

`package github` (1)

`pkg/capabilities/v2/actions/confidentialrelay.SecretsRequestParams` (1)

`pkg/capabilities/v2/actions/confidentialworkflow.(*WorkflowExecution)` (1)

`pkg/capabilities/v2/actions/confidentialworkflow.WorkflowExecution` (1)

`pkg/workflows` (3)

`pkg/workflows/dontime/pb` (1)

`pkg/workflows/dontime/pb.(*Observation)` (1)

`pkg/workflows/dontime/pb.Observation` (1)