Skip to content

Commit 0898c23

Browse files
committed
refactor: convert osv-sync from standalone bin to Temporal workflow
ADR-0001 §Worker architecture (decided 2026-05-25) standardized packages_worker sub-workers on the Temporal shape: workflows.ts + activities.ts + schedule.ts per sub-worker, registered against the shared packages-worker task queue, with ScheduleOverlapPolicy.SKIP so a slow run does not queue a second concurrent execution. This branch originally implemented osv-sync as a standalone bin following the github-repos-enricher precedent and proposed ADR-0004 to codify that pattern. Rebasing onto current main surfaced the conflict: github-repos-enricher is now explicitly listed as a legacy exception with migration deferred, and osv/ is named alongside npm/ and deps-dev/ as Temporal-based from the start. ADR-0004 is removed (superseded by ADR-0001 before merging). Changes: - src/osv/workflows.ts: osvSync workflow definition. Loops the ecosystems passed via schedule args, calls osvSyncEcosystem activity per ecosystem (per-ecosystem failure logged but does not abort the pass), then osvDeriveCriticalFlag. proxyActivities with retry policy (3 attempts, exp backoff) and nonRetryableErrorTypes for NOT_FOUND / PARSE. - src/osv/activities.ts: osvSyncEcosystem and osvDeriveCriticalFlag activities wrapping the existing pure-function pipeline (fetchEcosystemZip + parseOsvRecord + upsertAdvisoryBatch + deriveCriticalFlag). Heartbeats every 1000 records keep Temporal aware during the ~1-hour npm pass. FetchError(NOT_FOUND|PARSE) is translated to ApplicationFailure.nonRetryable so RetryPolicy short-circuits. - src/osv/schedule.ts: scheduleOsvSync registers an idempotent daily schedule at 03:30 UTC (offset from npm-registry-ingest at 03:15). 4-hour workflowExecutionTimeout, SKIP overlap. - src/workflows/index.ts + src/activities.ts: re-export the new workflow + activities for the worker bundle / runtime. - src/bin/packages-worker.ts: call scheduleOsvSync on startup alongside scheduleNpmIngest. Deleted: - src/bin/osv-sync.ts (entry point no longer needed) - src/osv/index.ts (runOsvSync sleep loop replaced by Temporal) - scripts/services/osv-sync.yaml (separate compose service gone; packages-worker container serves osv via the shared task queue) - docs/adr/0004 (superseded by ADR-0001 §Worker architecture) Env vars no longer needed (replaced by Temporal scheduling): - OSV_SYNC_INTERVAL_HOURS (cron expression owns cadence) - OSV_IDLE_SLEEP_SEC (no internal sleep loop) - getOsvConfig() removed from src/config.ts The pure-function pipeline survives untouched — extractSeverity, cvssScoring, parseOsvRecord, fetchEcosystemZip, upsertAdvisory, deriveCriticalFlag, versionCompare, types. All 68 unit tests still pass. The Temporal activity is a thin orchestrator over the same functions, so the migration is shape-only — no business logic changes. Signed-off-by: Joan Reyero <joan@reyero.io>
1 parent 0b7ccb4 commit 0898c23

15 files changed

Lines changed: 303 additions & 354 deletions

File tree

backend/.env.dist.local

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -184,11 +184,9 @@ ENRICHER_BATCH_SIZE=100
184184
ENRICHER_REPO_UPDATE_INTERVAL_HOURS=24
185185
ENRICHER_IDLE_SLEEP_SEC=60
186186

187-
# osv-sync
187+
# osv-sync (Temporal-scheduled; see services/apps/packages_worker/src/osv/schedule.ts)
188188
OSV_BULK_BASE_URL=https://osv-vulnerabilities.storage.googleapis.com
189189
OSV_ECOSYSTEMS=npm,Maven
190-
OSV_SYNC_INTERVAL_HOURS=24
191-
OSV_IDLE_SLEEP_SEC=60
192190
OSV_TMP_DIR=/tmp/osv
193191
OSV_BATCH_SIZE=500
194192
OSV_DERIVE_BATCH_SIZE=1000

docs/adr/0004-standalone-bin-vs-temporal-for-batch-sub-workers.md

Lines changed: 0 additions & 62 deletions
This file was deleted.

docs/adr/README.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,17 +6,17 @@ Use the `/adr` skill in Claude Code to record new ADRs or query past decisions.
66

77
## Index
88

9-
| ADR | Title | Status | Date |
10-
| --- | ----- | ------ | ---- |
11-
| [ADR-0001](./0001-oss-packages-design-decisions.md) | OSS packages — design decisions (living) | living | 2026-05-27 |
12-
| [ADR-0003](./0003-has-critical-vulnerability-semantics.md) | Semantics of `packages.has_critical_vulnerability` | accepted | 2026-05-27 |
13-
| [ADR-0004](./0004-standalone-bin-vs-temporal-for-batch-sub-workers.md) | Standalone bin vs Temporal for batch sub-workers in packages_worker | accepted | 2026-05-27 |
14-
| [ADR-0005](./0005-cvss-scoring-strategy.md) | CVSS scoring strategy for OSV ingestion (inline v3.1, defer v4) | accepted | 2026-05-27 |
15-
| [ADR-0006](./0006-advisory-affected-ranges-uniqueness-scope.md) | Uniqueness scope of `advisory_affected_ranges` | accepted | 2026-05-27 |
9+
| ADR | Title | Status | Date |
10+
| --------------------------------------------------------------- | --------------------------------------------------------------- | -------- | ---------- |
11+
| [ADR-0001](./0001-oss-packages-design-decisions.md) | OSS packages — design decisions (living) | living | 2026-05-27 |
12+
| [ADR-0003](./0003-has-critical-vulnerability-semantics.md) | Semantics of `packages.has_critical_vulnerability` | accepted | 2026-05-27 |
13+
| [ADR-0005](./0005-cvss-scoring-strategy.md) | CVSS scoring strategy for OSV ingestion (inline v3.1, defer v4) | accepted | 2026-05-27 |
14+
| [ADR-0006](./0006-advisory-affected-ranges-uniqueness-scope.md) | Uniqueness scope of `advisory_affected_ranges` | accepted | 2026-05-27 |
1615

1716
## Why ADRs?
1817

1918
The codebase is in active transition across several axes (see `CLAUDE.md`). ADRs provide a durable record of:
19+
2020
- Why old patterns are being replaced (e.g. Sequelize → pg-promise)
2121
- What alternatives were considered before choosing the current approach
2222
- What trade-offs were accepted

pnpm-lock.yaml

Lines changed: 13 additions & 10 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

scripts/services/osv-sync.yaml

Lines changed: 0 additions & 71 deletions
This file was deleted.

services/apps/packages_worker/package.json

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,10 @@
44
"scripts": {
55
"start:packages-worker": "CROWD_TEMPORAL_TASKQUEUE=packages-worker CROWD_TEMPORAL_NAMESPACE=$CROWD_PACKAGES_TEMPORAL_NAMESPACE SERVICE=packages-worker tsx src/bin/packages-worker.ts",
66
"start:github-repos-enricher": "SERVICE=github-repos-enricher tsx src/bin/github-repos-enricher.ts",
7-
"start:osv-sync": "SERVICE=osv-sync tsx src/bin/osv-sync.ts",
87
"dev:packages-worker": "CROWD_TEMPORAL_TASKQUEUE=packages-worker CROWD_TEMPORAL_NAMESPACE=$CROWD_PACKAGES_TEMPORAL_NAMESPACE SERVICE=packages-worker LOG_LEVEL=trace nodemon --watch src --watch ../../libs --ext ts --exec tsx --inspect=0.0.0.0:9233 src/bin/packages-worker.ts",
98
"dev:github-repos-enricher": "SERVICE=github-repos-enricher LOG_LEVEL=trace nodemon --watch src --watch ../../libs --ext ts --exec tsx --inspect=0.0.0.0:9234 src/bin/github-repos-enricher.ts",
10-
"dev:osv-sync": "SERVICE=osv-sync LOG_LEVEL=trace nodemon --watch src --watch ../../libs --ext ts --exec tsx --inspect=0.0.0.0:9235 src/bin/osv-sync.ts",
119
"dev:packages-worker:local": "set -a && . ../../../backend/.env.dist.local && . ../../../backend/.env.override.local && set +a && CROWD_TEMPORAL_TASKQUEUE=packages-worker CROWD_TEMPORAL_NAMESPACE=$CROWD_PACKAGES_TEMPORAL_NAMESPACE SERVICE=packages-worker LOG_LEVEL=trace nodemon --watch src --watch ../../libs --ext ts --exec tsx --inspect=0.0.0.0:9233 src/bin/packages-worker.ts",
1210
"dev:github-repos-enricher:local": "set -a && . ../../../backend/.env.dist.local && . ../../../backend/.env.override.local && set +a && SERVICE=github-repos-enricher LOG_LEVEL=trace nodemon --watch src --watch ../../libs --ext ts --exec tsx --inspect=0.0.0.0:9234 src/bin/github-repos-enricher.ts",
13-
"dev:osv-sync:local": "set -a && . ../../../backend/.env.dist.local && . ../../../backend/.env.override.local && set +a && SERVICE=osv-sync LOG_LEVEL=trace nodemon --watch src --watch ../../libs --ext ts --exec tsx --inspect=0.0.0.0:9235 src/bin/osv-sync.ts",
1411
"lint": "npx eslint --ext .ts src --max-warnings=0",
1512
"format": "npx prettier --write \"src/**/*.ts\"",
1613
"format-check": "npx prettier --check .",
@@ -24,6 +21,7 @@
2421
"@crowd/data-access-layer": "workspace:*",
2522
"@crowd/database": "workspace:*",
2623
"@crowd/logging": "workspace:*",
24+
"@temporalio/activity": "~1.11.8",
2725
"@temporalio/client": "~1.11.8",
2826
"@temporalio/workflow": "~1.11.8",
2927
"semver": "^7.6.0",
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
export { sayHiNpm } from './npm/activities'
2+
export { osvSyncEcosystem, osvDeriveCriticalFlag } from './osv/activities'

services/apps/packages_worker/src/bin/osv-sync.ts

Lines changed: 0 additions & 47 deletions
This file was deleted.
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
import { scheduleNpmIngest } from '../npm/schedule'
2+
import { scheduleOsvSync } from '../osv/schedule'
23
import { svc } from '../service'
34

45
setImmediate(async () => {
56
await svc.init()
67
await scheduleNpmIngest()
8+
await scheduleOsvSync()
79
await svc.start()
810
})

services/apps/packages_worker/src/config.ts

Lines changed: 0 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -32,20 +32,3 @@ export function getEnricherConfig() {
3232
idleSleepSec: requireEnvInt('ENRICHER_IDLE_SLEEP_SEC'),
3333
}
3434
}
35-
36-
export function getOsvConfig() {
37-
const ecosystems = requireEnv('OSV_ECOSYSTEMS')
38-
.split(',')
39-
.map((e) => e.trim())
40-
.filter(Boolean)
41-
42-
return {
43-
bulkBaseUrl: requireEnv('OSV_BULK_BASE_URL'),
44-
ecosystems,
45-
syncIntervalHours: requireEnvInt('OSV_SYNC_INTERVAL_HOURS'),
46-
idleSleepSec: requireEnvInt('OSV_IDLE_SLEEP_SEC'),
47-
tmpDir: requireEnv('OSV_TMP_DIR'),
48-
batchSize: requireEnvInt('OSV_BATCH_SIZE'),
49-
deriveBatchSize: requireEnvInt('OSV_DERIVE_BATCH_SIZE'),
50-
}
51-
}

0 commit comments

Comments
 (0)