Commit 618810c
authored
fix(agents-server): stop orphaned cron wakes (#4559)
Fixes orphaned cron wakes in Agents Server after a manifest-backed
schedule is deleted. Users should no longer see agents repeatedly woken
by a cron schedule that `list_schedules` correctly reports as absent.
## Root Cause
Cron schedules are represented by two linked pieces of state:
1. a manifest-backed schedule row on the agent stream, used by tools
like `list_schedules`
2. wake-registry + scheduler side effects, used to deliver cron ticks
For `/horton/RbYyjlX3ey`, the manifest schedule had been deleted, so
`list_schedules` returned `[]`. However, cron wake delivery kept going
because stale wake/scheduler side effects could outlive the manifest
deletion:
- `WakeRegistry` removed cached registrations on Electric shape deletes
by deriving the DB id from the shape message key. Shape keys are
protocol-level identifiers and are not guaranteed to be the table
primary key, so the DB row could be gone while the in-memory
registration remained.
- Cron ticks rescheduled themselves indefinitely, even when no
`wake_registrations` rows still subscribed to that cron stream.
## Approach
This PR makes the cron side effects self-healing in both places.
### Wake registry delete handling
`WakeRegistry.applyShapeMessage` now treats `old_value.id` as
authoritative for delete messages. The shape uses `replica: full`, so
deletes should carry the deleted row; if the id is missing, the registry
resets its cache to fail closed instead of keeping a potentially stale
wake registration alive.
```ts
const oldValue = (message as unknown as {
old_value?: { id?: unknown }
}).old_value
const oldId = Number(oldValue?.id)
if (Number.isFinite(oldId)) {
this.removeCachedRegistrationByDbId(oldId)
} else {
this.resetCachedRegistrations()
}
```
### Cron chain rescheduling
Before inserting the next cron tick, the scheduler checks whether any
wake registration still points at that cron stream:
```ts
select 1 as exists
from wake_registrations
where tenant_id = ${tenantId}
and source_url = ${streamPath}
limit 1
```
If there are no subscribers, it completes the current task and stops the
chain. A future schedule registration will seed a fresh tick through
`getOrCreateCronStream` / rehydration.
## Key Invariants
- The manifest remains the user-visible source of truth for schedule
tools.
- A cron stream should only keep producing future ticks while at least
one wake registration subscribes to it.
- Deleting a manifest-backed cron schedule must remove both durable wake
registrations and any effective in-memory wake registrations.
- Shared cron streams remain reusable: deleting the last subscriber
stops the chain; adding a subscriber later recreates it.
## Non-goals
- This does not change the public schedule tool API.
- This does not delete historical cron stream events.
- This does not introduce per-schedule cron streams; cron streams remain
shared by expression/timezone.
- This does not attempt to clean up already-delivered wake events from
agent timelines.
## Trade-offs
An alternative would be to always reschedule cron ticks forever for
every cron stream ever created. That is simpler, but it allows orphaned
cron streams to keep waking stale in-memory subscribers and creates
unnecessary scheduler churn. Checking for subscribers before
rescheduling keeps cron streams lazy and aligns their lifecycle with
active wake registrations.
Another option would be to rely only on wake-registry cache
invalidation. The scheduler guard is still useful as a second line of
defense and prevents orphaned cron task chains even if a process
restarts or a wake cache bug reappears.
## Verification
Commands run:
```bash
pnpm install
pnpm --filter @electric-ax/agents-runtime build
pnpm --filter @electric-ax/agents-mcp build
pnpm --filter @electric-sql/client build
pnpm --filter @electric-ax/agents-server typecheck
cd packages/agents-server && pnpm test test/scheduler.test.ts
cd packages/agents-server && pnpm test test/wake-registry.test.ts -t "removes cached registrations"
GITHUB_BASE_REF=main node scripts/check-changeset.mjs
git diff --check
```
Notes:
- `pnpm install` completed dependency linking but reported warnings,
including `examples/.shared prepare` failing due SST.
- The focused wake-registry unit test passes. Running all of
`wake-registry.test.ts` locally reached the integration section and
timed out with a webhook delivery `fetch failed` warning, so the new
unit path was verified with `-t`.
- `check-changeset` passes.
## Files changed
- `packages/agents-server/src/wake-registry.ts` — remove cached
registrations using `old_value.id` from shape delete messages; reset the
cache if a delete id is unavailable.
- `packages/agents-server/src/scheduler.ts` — stop rescheduling cron
ticks when no registrations subscribe to the cron stream; clarify cron
payload stream-path extraction and subscriber lookup.
- `packages/agents-server/test/scheduler.test.ts` — cover
subscriber-present and no-subscriber cron reschedule paths.
- `packages/agents-server/test/wake-registry.test.ts` — cover non-id
shape keys with delete ids coming from `old_value.id`.
- `.changeset/fix-orphaned-cron-wakes.md` — patch changeset for
`@electric-ax/agents-server`.1 parent 0947230 commit 618810c
5 files changed
Lines changed: 117 additions & 2 deletions
File tree
- .changeset
- packages/agents-server
- src
- test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
228 | 228 | | |
229 | 229 | | |
230 | 230 | | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
231 | 239 | | |
232 | 240 | | |
233 | 241 | | |
| |||
680 | 688 | | |
681 | 689 | | |
682 | 690 | | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
683 | 709 | | |
684 | 710 | | |
685 | 711 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
739 | 739 | | |
740 | 740 | | |
741 | 741 | | |
742 | | - | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
743 | 759 | | |
744 | 760 | | |
745 | 761 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
317 | | - | |
| 317 | + | |
318 | 318 | | |
319 | 319 | | |
320 | 320 | | |
| |||
346 | 346 | | |
347 | 347 | | |
348 | 348 | | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
349 | 385 | | |
350 | 386 | | |
351 | 387 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
201 | 201 | | |
202 | 202 | | |
203 | 203 | | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
204 | 236 | | |
205 | 237 | | |
206 | 238 | | |
| |||
0 commit comments