Commit 0898c23
committed
refactor: convert osv-sync from standalone bin to Temporal workflow
ADR-0001 §Worker architecture (decided 2026-05-25) standardized
packages_worker sub-workers on the Temporal shape: workflows.ts +
activities.ts + schedule.ts per sub-worker, registered against the
shared packages-worker task queue, with ScheduleOverlapPolicy.SKIP
so a slow run does not queue a second concurrent execution.
This branch originally implemented osv-sync as a standalone bin
following the github-repos-enricher precedent and proposed ADR-0004
to codify that pattern. Rebasing onto current main surfaced the
conflict: github-repos-enricher is now explicitly listed as a
legacy exception with migration deferred, and osv/ is named
alongside npm/ and deps-dev/ as Temporal-based from the start.
ADR-0004 is removed (superseded by ADR-0001 before merging).
Changes:
- src/osv/workflows.ts: osvSync workflow definition. Loops the
ecosystems passed via schedule args, calls osvSyncEcosystem
activity per ecosystem (per-ecosystem failure logged but does not
abort the pass), then osvDeriveCriticalFlag. proxyActivities with
retry policy (3 attempts, exp backoff) and nonRetryableErrorTypes
for NOT_FOUND / PARSE.
- src/osv/activities.ts: osvSyncEcosystem and osvDeriveCriticalFlag
activities wrapping the existing pure-function pipeline
(fetchEcosystemZip + parseOsvRecord + upsertAdvisoryBatch +
deriveCriticalFlag). Heartbeats every 1000 records keep Temporal
aware during the ~1-hour npm pass. FetchError(NOT_FOUND|PARSE) is
translated to ApplicationFailure.nonRetryable so RetryPolicy
short-circuits.
- src/osv/schedule.ts: scheduleOsvSync registers an idempotent
daily schedule at 03:30 UTC (offset from npm-registry-ingest at
03:15). 4-hour workflowExecutionTimeout, SKIP overlap.
- src/workflows/index.ts + src/activities.ts: re-export the new
workflow + activities for the worker bundle / runtime.
- src/bin/packages-worker.ts: call scheduleOsvSync on startup
alongside scheduleNpmIngest.
Deleted:
- src/bin/osv-sync.ts (entry point no longer needed)
- src/osv/index.ts (runOsvSync sleep loop replaced by Temporal)
- scripts/services/osv-sync.yaml (separate compose service gone;
packages-worker container serves osv via the shared task queue)
- docs/adr/0004 (superseded by ADR-0001 §Worker architecture)
Env vars no longer needed (replaced by Temporal scheduling):
- OSV_SYNC_INTERVAL_HOURS (cron expression owns cadence)
- OSV_IDLE_SLEEP_SEC (no internal sleep loop)
- getOsvConfig() removed from src/config.ts
The pure-function pipeline survives untouched — extractSeverity,
cvssScoring, parseOsvRecord, fetchEcosystemZip, upsertAdvisory,
deriveCriticalFlag, versionCompare, types. All 68 unit tests still
pass. The Temporal activity is a thin orchestrator over the same
functions, so the migration is shape-only — no business logic
changes.
Signed-off-by: Joan Reyero <joan@reyero.io>1 parent 0b7ccb4 commit 0898c23
15 files changed
Lines changed: 303 additions & 354 deletions
File tree
- backend
- docs/adr
- scripts/services
- services/apps/packages_worker
- src
- bin
- osv
- workflows
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
184 | 184 | | |
185 | 185 | | |
186 | 186 | | |
187 | | - | |
| 187 | + | |
188 | 188 | | |
189 | 189 | | |
190 | | - | |
191 | | - | |
192 | 190 | | |
193 | 191 | | |
194 | 192 | | |
Lines changed: 0 additions & 62 deletions
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
16 | 15 | | |
17 | 16 | | |
18 | 17 | | |
19 | 18 | | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
8 | 7 | | |
9 | 8 | | |
10 | | - | |
11 | 9 | | |
12 | 10 | | |
13 | | - | |
14 | 11 | | |
15 | 12 | | |
16 | 13 | | |
| |||
24 | 21 | | |
25 | 22 | | |
26 | 23 | | |
| 24 | + | |
27 | 25 | | |
28 | 26 | | |
29 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| 8 | + | |
7 | 9 | | |
8 | 10 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
0 commit comments