Commit 39ff054
feat(worker): flow_synthetic continuous-monitoring runner (P0 matrix → metrics + NR event + reap) (#91)
* feat(worker): flow_synthetic continuous-monitoring runner (P0 matrix → metrics + NR event + reap)
The capstone of the observability pillar: a River-scheduled job that runs the
P0 user-flow matrix against prod every 5 min and pushes a green/red matrix to
NR. Mirrors the auth_probe.go / deploy_probe.go prober pattern (per-flow
result enum pass|fail|degraded, audit_log row on fail + slog ERROR NR
fallback) and adds the two things the single-flow probers don't need:
- the common/analyticsevent backend→NR bridge (RecordFlowTest →
InstantFlowTest custom event, cohort="synthetic" so real dashboards
exclude it), and
- a create→assert→REAP lifecycle (rule-24 cleanup ledger): every resource
a flow provisions is deleted via the real DELETE /api/v1/resources/:id
path + a synthetic.reaped audit row, with a 30m orphan-sweep backstop.
Flows (this PR, all prod-safe / API layer):
- healthz (anon) GET /healthz → 200 + commit_id (also the per-tick
commitId correlation for every event, rule 14/15)
- auth_me (human) GET /auth/me with a minted session JWT → 200 + email
- provision_reap(agent) POST /db/new → DELETE resource round-trip
Brevo-free auth: the runner mints an HS256 session JWT locally (JWT_SECRET,
the same value the api verifies against — like api/e2e makeSessionJWT), and
idempotently seeds a deterministic cohort-tagged synthetic team + primary
user, elevating its tier via the same UpgradeTeamAllTiers logic the Razorpay
webhook uses. The cohort flag means every team-iterating background job (the
already-shipped test_cohort.go guards) no-ops for it.
Flag-gated OFF (FLOW_SYNTHETIC_ENABLED, default false — the DoD habit): the
whole layer is inert in prod until an operator lights it. Per-flow kill list
(FLOW_SYNTHETIC_DISABLED) + a Team-tier sub-flag (FLOW_SYNTHETIC_TEAM_ENABLED,
off until Team GA). Registered in buildPeriodicJobs with UniqueOpts
(replicas:2 safe), RunOnStart=true.
Metrics (rule 25 — alert + tile + catalog land in the paired infra PR):
instant_flow_test_total{flow,actor,tier,layer,result}
instant_flow_test_latency_seconds{flow,actor,tier,layer}
instant_flow_synthetic_reaped_total{flow,outcome} (leaked MUST stay 0)
Tests: flag-off no-op, full happy-path matrix + reap + event assertions,
per-flow fail/degraded/leak paths, kill switch, no-JWT/seed-failure degrade,
orphan-sweep backstop, the Prom impl, and a DB-gated *Integration test proving
the cohort-team seed + reaper soft-delete against real Postgres. make gate
green (build + vet + go test ./... -short -count=1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(worker): flow_synthetic patch coverage → 100% (test seams + reachable branches)
The patch-coverage gate (diff-cover --fail-under=100) flagged defensive/error
branches in flow_synthetic.go. Make every changed line covered:
- injectable per-flow budget override (SetBudgetOverrideForTest) → drives the
degraded-latency branches deterministically (0 budget = always over).
- extract flowSyntheticNoRedirect + RunPanickingFlowForTest + ReapNilDBForTest
test seams for the default-client CheckRedirect, the recover() boundary
(also covers flowActorForFlow's default arm via an unknown flow id), and the
db==nil guards in recordReap/emitFlowTestFailed.
- tests for bad-base-URL build_request, closed-port http_error, seed
sub-errors (begin/user/elevate/commit), reap http_error (conn hijack),
audit-insert error (non-fatal), orphan scan/rows/update errors.
- remove two genuinely-unreachable defensive branches (reapResource
build_request — url.PathEscape sanitises; mintSessionJWT marshal error —
a string/int map can't fail to marshal), matching the deploy_probe `_ =`
posture. workers.go: Factory's Override path can't error (analyticsnr.New
is never nil) → discard the structurally-dead advisory branch.
diff-cover: internal/{config,jobs,workers}.go all 100% on the patch (0 missing).
make gate green; golangci-lint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 55c7b9f commit 39ff054
6 files changed
Lines changed: 2475 additions & 27 deletions
File tree
- internal
- config
- jobs
- metrics
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
103 | | - | |
104 | | - | |
105 | | - | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| |||
151 | 151 | | |
152 | 152 | | |
153 | 153 | | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
154 | 170 | | |
155 | 171 | | |
156 | 172 | | |
| |||
180 | 196 | | |
181 | 197 | | |
182 | 198 | | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
201 | 217 | | |
202 | 218 | | |
203 | 219 | | |
| |||
253 | 269 | | |
254 | 270 | | |
255 | 271 | | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
256 | 285 | | |
257 | 286 | | |
258 | 287 | | |
| |||
0 commit comments