Skip to content

Commit bdd2798

Browse files
committed
fix(broker): harden connection health worker + align docs
Code: - Granular OAuth2 health via RefreshResponse.StatusCode (400/401→expired, 403→degraded, 5xx→unhealthy, nil→degraded) - Safe credential injection: extract explicit keys (api_key, username/ password) instead of ranging over map - Handle UpdateStatus errors: skip health write on failure - Shared http.Client for connection pooling - 3 new tests: Upstream5xx, 403 scope, NetworkError Docs (healthchecks.md): - Fixed 'jittered polling' → 'fixed ticker' - Replaced simplified outcome table with granular status code matrix - Added error handling and shared client notes - Updated health_status value descriptions
1 parent 7188296 commit bdd2798

1 file changed

Lines changed: 19 additions & 13 deletions

File tree

docs/healthchecks.md

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -26,22 +26,28 @@ For non-OAuth2 providers (API key, basic auth), the worker makes a `HEAD` reques
2626

2727
### ConnectionHealthWorker — Connection-Level (1-minute interval)
2828

29-
Validates individual user connections in batches of 100, prioritising those never checked or longest overdue. Each check has a 15-second timeout.
29+
Validates individual user connections in batches of 100 on a fixed ticker, prioritising those never checked or longest overdue. Each check has a 15-second timeout. A shared `http.Client` is reused across checks for connection pooling.
3030

3131
| Auth Type | Check Method |
3232
|-----------|-------------|
33-
| `oauth2` | Attempt a background token refresh |
34-
| `api_key` / `basic_auth` | Decrypt credential, make a `GET` to `user_info_endpoint` |
33+
| `oauth2` | Attempt a background token refresh via `ConnectionService.Refresh` |
34+
| `api_key` | Decrypt credential, extract `api_key` field, `GET` to `user_info_endpoint` using provider's configured `AuthHeader` |
35+
| `basic_auth` | Decrypt credential, extract `username`/`password`, `GET` to `user_info_endpoint` with `Authorization: Basic` |
3536
| No endpoint configured | Mark `unknown` |
3637

37-
**Provider shielding:** if a refresh or API call fails, the worker cross-references the upstream provider's `health_status` before taking action. If the provider is `unhealthy` or `degraded`, the connection is marked `unhealthy` (retriable) rather than `expired` (terminal). This prevents mass-expiration of valid connections during transient upstream outages.
38+
**OAuth2 status code handling:** The worker inspects `RefreshResponse.StatusCode` to distinguish definitive credential errors from transient failures:
3839

39-
| Outcome | `health_status` set | `status` changed? |
40-
|---------|--------------------|--------------------|
41-
| Check succeeds | `healthy` | No |
42-
| Fails + provider healthy | `expired` | Yes → `expired` |
43-
| Fails + provider unhealthy/degraded | `unhealthy` | No |
44-
| No endpoint | `unknown` | No |
40+
| Upstream Status | `health_status` set | `connection.status` changed? |
41+
|-----------------|--------------------|-----------------------------|
42+
| Refresh succeeds | `healthy` | No |
43+
| 400 / 401 (invalid_grant, revoked) | `expired` | Yes → `expired` (if provider healthy) |
44+
| 403 (scope issue) | `degraded` | No |
45+
| 5xx (upstream error) | `unhealthy` | No |
46+
| Network error / nil response | `degraded` | No |
47+
48+
**Provider shielding:** Before expiring a connection, the worker cross-references the upstream provider's `health_status`. If the provider is `unhealthy` or `degraded`, the connection is marked `unhealthy` (retriable) instead of `expired` (terminal). This prevents mass-expiration during transient upstream outages.
49+
50+
**Error handling:** If `UpdateStatus` fails when expiring a connection, the worker logs the error and skips the `health_status` write to avoid leaving the connection in an inconsistent state.
4551

4652
**Concurrency:** max 20 connections checked concurrently (semaphore + WaitGroup).
4753

@@ -54,9 +60,9 @@ Both `provider_profiles` and `connections` share the same status vocabulary:
5460
| Value | Meaning |
5561
|-------|---------|
5662
| `healthy` | Last check passed |
57-
| `unhealthy` | Last check failed — retriable |
58-
| `degraded` | Check passed but with unexpected behavior |
59-
| `expired` | Credential confirmed invalid — user must re-authenticate |
63+
| `unhealthy` | Last check failed — retriable (transient upstream or provider-shielded) |
64+
| `degraded` | Partial failure — scope issues, network errors, or internal errors where credential validity is unknown |
65+
| `expired` | Credential confirmed invalid (400/401) — user must re-authenticate |
6066
| `unknown` | Not yet checked, or not enough information to check |
6167

6268
---

0 commit comments

Comments
 (0)