You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| Source of truth: deps.dev vs registries / OSV | decided |
26
+
| deps.dev coverage and gaps | decided |
27
+
| Downloads timeline by tier | decided |
27
28
28
29
---
29
30
@@ -79,7 +80,7 @@ Adding a new data source means adding `src/{worker}/` and a new compose service
79
80
80
81
Tier 2 enriches a critical slice of the npm and Maven ecosystems — not the full registry. We need a signal-rich, affordable source to rank the ~4–5M packages and decide which top-N to fully enrich.
81
82
82
-
We use the [deps.dev BigQuery public datasets](https://deps.dev) — specifically `PackageVersionsLatest`, `DependentsLatest`, `PackageVersionToProjectLatest`, and `ProjectsLatest` — filtered to `System IN ('NPM', 'MAVEN')` as the universe input. The BigQuery data is exported to Parquet files and imported into `packages_universe` on a weekly cadence. A scoring + ranking job then promotes the top-N per ecosystem by setting `is_critical = true` and copying `criticality_score` onto the full `packages` table.
83
+
We use the [deps.dev BigQuery public datasets](https://deps.dev) — specifically `PackageVersionsLatest`, `DependentsLatest`, `PackageVersionToProjectLatest`, and `ProjectsLatest` — filtered to `System IN ('NPM', 'MAVEN')` as the universe input. The BigQuery data is exported to Parquet files and imported into `packages_universe` on a weekly cadence aligned with deps.dev's own refresh interval. The first run is a one-time full backfill; subsequent weekly imports only pull rows whose deps.dev snapshot date has advanced since the previous import, so the export size and write volume are scoped to actual diffs rather than the full universe. A scoring + ranking job then promotes the top-N per ecosystem by setting `is_critical = true` and copying `criticality_score` onto the full `packages` table.
83
84
84
85
The current scoring formula and per-ecosystem critical-package quotas are **not yet finalized** — both are still under discussion. The ranking function takes `critical_top_n_by_ecosystem` as a JSONB parameter and weights as numeric inputs, so thresholds and formula coefficients can be tuned at call time without a schema change.
85
86
@@ -96,7 +97,7 @@ Five sub-workers run concurrently (npm, Maven, OSV, GitHub, Docker Hub), all wri
|`packages`| Upsert on `purl`. Each worker only writes columns it owns; ecosystem isolation means column-level conflicts cannot occur in practice. |
99
-
|`packages_universe`|Full truncate-and-replace on the weekly rank job. No other worker writes to this table. The truncate + bulk-insert must be wrapped in a single transaction or shadow-table swap to avoid a window of emptiness visible to the promotion query. |
100
+
|`packages_universe`|Incremental upsert keyed on `purl`. The deps.dev import only touches rows whose underlying deps.dev snapshot date has advanced since the previous import (initial run is a one-time full backfill).|
100
101
|`versions`| Append-only via `INSERT … ON CONFLICT DO NOTHING`. Yanked/deprecated status is a separate targeted `UPDATE (is_yanked = true) WHERE …`. |
101
102
|`repos`| Registry workers (npm, Maven) do **not** write directly to `repos`. They write `package_repos` rows. The GitHub enricher — triggered when `repos.last_synced_at IS NULL` — upserts `repos` with metadata. Docker Hub worker adds `docker_*` columns on top. |
102
103
|`package_repos`| Composite PK `(package_id, repo_url)`. Each `source` value ('declared', 'deps_dev', 'heuristic', 'manual') is a separate row — sources do not overwrite each other. |
@@ -211,7 +212,7 @@ A range `(introduced, fixed, last_affected)` matches `latest_version` when:
211
212
-`fixed IS NULL OR latest_version < fixed`, AND
212
213
-`last_affected IS NULL OR latest_version <= last_affected`.
213
214
214
-
This is **option (b)** (latest_version inside an active range), plus a **MAL- override** so malicious-package reports flip the flag regardless of CVSS — the XZ-style maintainer-compromise case from the Osprey memo. ~213k of 220k npm OSV records are `MAL-*` with `cvss = NULL`, so option (b) on its own would miss the dominant security signal.
215
+
This is **option (b)** (latest_version inside an active range), plus a **MAL- override** so malicious-package reports flip the flag regardless of CVSS — the XZ-style maintainer-compromise case. ~213k of 220k npm OSV records are `MAL-*` with `cvss = NULL`, so option (b) on its own would miss the dominant security signal.
215
216
216
217
**Why not option (a)** (any critical advisory exists for the package name, regardless of version): option (a) over-reports — a CVE patched in v1.0 flags a package now on v9.0 — and under-reports when an advisory has multiple `affected[]` ranges where only some are patched. The actionable consumer question is "is the version I'd install today vulnerable?", and that's option (b).
217
218
@@ -288,6 +289,51 @@ Docker Hub runs after registry workers are stable. Docker dependents (which imag
288
289
289
290
---
290
291
292
+
### Source of truth: deps.dev backfill vs registries / OSV
293
+
294
+
deps.dev gives us a single low-cost source covering packages, versions, package → repo mappings, repos, and advisories. The same data is also available from the underlying registries (npm, Maven Central, etc.) and from OSV directly. We need a clear ownership rule so two sources don't quietly disagree.
295
+
296
+
We split responsibility by lifecycle stage, not by table:
297
+
298
+
-**Initial backfill (one-time, per ecosystem)**: deps.dev populates `packages`, `versions`, `package_repos`, `repos`, `advisories`, and `advisory_affected_ranges` in a single bulk pass. This is how we get from zero to a workable dataset without standing up every registry worker first.
299
+
-**New packages discovered after backfill**: deps.dev remains the bootstrap source. On each weekly deps.dev import, any newly-seen `purl` lands in all relevant tables from deps.dev fields, exactly as it did during backfill. This avoids a window where a new package exists in deps.dev but is invisible to us until the registry worker happens to see it.
300
+
-**Ongoing updates to existing rows**: registries are the source of truth for package and version data; OSV is the source of truth for advisories; the GitHub enricher remains the source of truth for `repos` metadata beyond what `ProjectsLatest` provides. Registry / OSV workers may overwrite values that deps.dev wrote at backfill time — that is the intended direction of drift.
The §deps.dev coverage and gaps table below remains the authoritative per-column ownership reference; this section is the lifecycle rule that sits on top of it.
315
+
316
+
#### Data-source provenance: `package_source_log`
317
+
318
+
To make drift between deps.dev and the registry / OSV workers observable, every package carries one row per writing source in `package_source_log`. Each worker that touches a package upserts its `(package_id, source)` row in the same transaction as the data write — bumping `last_synced_at` and recording the set of `table.column` paths it owns for that package.
|`package_id`| FK to `packages(id)`. Part of PK. |
323
+
|`source`| Writing source: `'deps_dev'`, `'npm-registry'`, `'maven-central'`, `'osv'`, `'github-enricher'`, `'manual'`. Part of PK. |
324
+
|`columns`| Array of `table.column` paths this source wrote for this package (e.g. `packages.latest_version`, `versions.is_yanked`). |
325
+
|`last_synced_at`| Timestamp of the most recent write by this source for this package. |
326
+
327
+
Primary key is `(package_id, source)` — one row per package per source, updated in place. Cardinality is bounded by `|packages| × |sources|` (a small constant per package), well under what an append-only event log would generate.
328
+
329
+
The shape answers the two questions the live data tables cannot: "which source last touched this package, and when?" and "is deps.dev still writing columns that a registry worker should now own?". It deliberately does not preserve history — a worker overwriting its own row drops the previous `columns` set. The present-tense "who currently owns what for this package" view is what the intended consumers (drift alerting, debugging the registry-vs-deps.dev handoff) need.
330
+
331
+
Updating `package_source_log` is a social contract on every worker that writes to packages-domain tables — no Postgres constraint enforces it. Code review and the §Source of truth lifecycle table are the enforcement mechanisms.
332
+
333
+
**Decided**: 2026-05-29
334
+
335
+
---
336
+
291
337
### deps.dev coverage and gaps
292
338
293
339
deps.dev is the primary source for package identity, dependents, advisories, and `ProjectsLatest`-derived repo metadata. The table below is the canonical reference for which sub-worker owns which column. Nullable columns in the schema reflect these boundaries.
@@ -368,8 +414,6 @@ Downloads is the strongest criticality signal, but the ~4–5M packages in `pack
368
414
369
415
Tier 2 (`packages`) downloads are stored in `downloads_daily` — one row per `(package_id, date)`, consumers sum over any window they need. Tier 3 (`packages_universe`) downloads are stored in `downloads_last_30d` — one row per `(purl, end_date)` capturing a rolling 30-day window — with the latest window's count also cached on `packages_universe.downloads_last_30d bigint` for direct use by `rank_packages_universe()` without a join.
370
416
371
-
`downloads_last_30d` is keyed by `purl` (not `packages_universe.id`) so rows survive the weekly truncation of `packages_universe`.
|`downloads_daily`| 2 (`packages`) | one row per package per day |`(package_id, date)`|`(id, date)`| RANGE on `date` via pg_partman |
@@ -396,7 +440,6 @@ A package promoted from Tier 3 to Tier 2 (becomes critical) will have rolling-wi
396
440
## Open questions / in-flight
397
441
398
442
-**Sonatype Central Stats API access** — not confirmed as of 2026-05-27. If unavailable by day 5, Maven download counts will be absent from the week-2 demo (`downloads_last_month` NULL for Maven rows; disclose to stakeholders).
399
-
-**criticality_score formula** — the placeholder formula (`X * downloadsCount + Y * dependentCount`) has not been validated against known critical packages. Final formula is yet to be defined.
400
443
-**pg_partman + pg_cron setup** — must be confirmed active in the OCI environment before download workers start; `downloads_daily` and `downloads_last_30d` inserts will fail if monthly partitions are not pre-created.
401
444
402
445
---
@@ -405,6 +448,7 @@ A package promoted from Tier 3 to Tier 2 (becomes critical) will have rolling-wi
405
448
406
449
- 2026-05-27 — initial record
407
450
- 2026-05-28 — folded standalone ADR-0003 (`has_critical_vulnerability` semantics), ADR-0005 (CVSS scoring strategy), and ADR-0006 (`advisory_affected_ranges` uniqueness scope) into this living record; standalone files removed. Resolved the prior open question on `has_critical_vulnerability` (option b + MAL- override). ADR-0004 (standalone-bin vs Temporal) was removed before merging — the worker architecture decision in this ADR supersedes it.
451
+
- 2026-05-29 — clarified `packages_universe` import semantics (one-time backfill + weekly snapshot-diff incrementals; the ranking job updates score/flag columns in place). Added §Source of truth: deps.dev backfill vs registries / OSV with lifecycle ownership rules and the agreed `package_source_log` provenance table (`(package_id, source)` PK; `columns` array tracks `table.column` paths each source writes).
0 commit comments