|
| 1 | +# ADR-0009: Packagist worker — design decisions (living) |
| 2 | + |
| 3 | +**Date**: 2026-07-13 |
| 4 | +**Status**: accepted |
| 5 | +**Deciders**: Anil Bostanci |
| 6 | + |
| 7 | +_Living, consolidated record for the Packagist (PHP/Composer) worker. Record further |
| 8 | +Packagist-worker decisions here as new `### subsections` under Decisions rather than opening a |
| 9 | +new ADR per decision — mirrors ADR-0001 (oss-packages living ADR) and the ADR-0005 consolidation |
| 10 | +note. Each entry is stamped with its own `**Decided**` date; the Changelog at the bottom tracks |
| 11 | +when entries are added or revised._ |
| 12 | + |
| 13 | +## Context |
| 14 | + |
| 15 | +Packagist is the primary registry for PHP/Composer. Unlike npm/maven/pypi/etc., it is **not |
| 16 | +covered by deps.dev** — there is no `PACKAGIST` system in `bigquery-public-data.deps_dev_v1` — so we |
| 17 | +cannot seed it from BigQuery the way ADR-0001's universe ingest does for other ecosystems. We crawl |
| 18 | +Packagist directly instead (list.json seed → dynamic stats endpoint → static p2 metadata endpoint). |
| 19 | +A consequence that shapes several decisions: Packagist hands us **per-package Composer manifests** |
| 20 | +(the raw `require` / `require-dev` maps a package declared), **not a pre-resolved dependency graph** |
| 21 | +like the one deps.dev exposes in BigQuery. |
| 22 | + |
| 23 | +## Scope and current status |
| 24 | + |
| 25 | +| Decision area | Status | |
| 26 | +| --------------- | ------- | |
| 27 | +| Dependency model | decided | |
| 28 | +| Metadata enrichment scope | decided | |
| 29 | +| Lane architecture & cadence | decided | |
| 30 | + |
| 31 | +--- |
| 32 | + |
| 33 | +## Decisions |
| 34 | + |
| 35 | +### Dependency model: direct edges + declared constraints, resolved at query time |
| 36 | + |
| 37 | +We store, in `package_dependencies`, **only the direct dependency edges a version declares** — one |
| 38 | +row per `(version_id, depends_on_id, dependency_kind)` — and we **resolve concrete versions and |
| 39 | +transitive closure at query time**, never at ingest. |
| 40 | + |
| 41 | +Per stored edge (for **all** packages — dependency edges ride the same p2 metadata fetch as |
| 42 | +versions; see the Metadata enrichment scope decision): |
| 43 | + |
| 44 | +- `require` → `dependency_kind = 'direct'`, `require-dev` → `dependency_kind = 'dev'`. |
| 45 | +- `version_constraint` = the **declared** constraint string verbatim (e.g. `^3.0 || ^2.0`). |
| 46 | +- `depends_on_id` = the depended-on **package** row (resolved by name; edges whose target isn't a |
| 47 | + known packages row are counted and skipped, not errored). |
| 48 | +- `depends_on_version_id` = **NULL** — we do not resolve the concrete satisfying version at ingest. |
| 49 | +- Platform requirements (`php`, `hhvm`, `ext-*`, `lib-*`, `composer-*` — anything with no `/`) are |
| 50 | + excluded; they aren't packages. |
| 51 | + |
| 52 | +**Provenance.** The query-time transitive model is a **stated requirement**, quoted verbatim from |
| 53 | +the Packagist change brief: |
| 54 | + |
| 55 | +> We store only direct dependencies (the requirements declared in each version's composer.json) and |
| 56 | +> walk the tree at query time when transitive resolution is needed. We do not need to compute or |
| 57 | +> store the resolved graph at ingest. |
| 58 | +
|
| 59 | +Leaving `depends_on_version_id` NULL is the column-level **implementation** of that last sentence — |
| 60 | +the resolved concrete version *is* part of "the resolved graph" — locked at the plan-approval gate |
| 61 | +and consistent with the pre-existing schema comment (`depends_on_version_id bigint, -- resolved |
| 62 | +version; NULL if unknown`). Storing the declared `version_constraint` but not a pinned version means |
| 63 | +the two derived views — *which version a constraint resolves to* and *the full transitive tree* — |
| 64 | +are both computed on read. |
| 65 | + |
| 66 | +**Why not pin `depends_on_version_id` at ingest (as deps.dev does):** a resolved version is *derived |
| 67 | +and volatile*. `^3.0` resolves to whatever `psr/log` patch is newest this week; a stored pin goes |
| 68 | +stale the moment a patch ships and would need continuous rewriting. deps.dev's own worker documents |
| 69 | +exactly this churn — re-resolving `^4.17.0` every snapshot is what produced its ~555M-row edge |
| 70 | +exports and forced a snapshot-diff to suppress it. We sidestep it entirely by storing the immutable |
| 71 | +declared constraint and resolving on demand. deps.dev pins because it *already* has a fully-resolved |
| 72 | +graph in BigQuery for free; we only get manifests, so resolving eagerly would mean building and |
| 73 | +continuously re-running a Composer resolver for no durable benefit. |
| 74 | + |
| 75 | +**Why store `dev` edges when deps.dev doesn't:** deps.dev's resolved graph is runtime-only — every |
| 76 | +edge lands as `'direct'`, it has no dev/peer notion. Because we read the raw manifest we can split |
| 77 | +`require-dev` into `'dev'`, so Packagist edges are *richer per-edge* (dev deps + declared |
| 78 | +constraints) even though they are *shallower* (no resolved target version) than deps.dev's. |
| 79 | + |
| 80 | +**Consequences.** |
| 81 | + |
| 82 | +_Positive:_ |
| 83 | + |
| 84 | +- Write path is small and touches only immutable declared facts — no re-resolution churn, no |
| 85 | + billion-row resolved-graph materialization for Packagist. |
| 86 | +- Query-time resolution and transitive walks always reflect current data (new patches, newly-added |
| 87 | + deep dependencies) with no re-ingest. |
| 88 | +- Captures dev-dependencies (`kind='dev'`), a dimension deps.dev-seeded ecosystems lack. |
| 89 | +- Schema-compatible with the deps.dev-populated `package_dependencies`: same table, same unique key |
| 90 | + `(version_id, depends_on_id, dependency_kind)`; the `depends_on_id` HASH-partitioning (ADR-0001) |
| 91 | + keeps the upstream "who depends on X?" hot path fast for Packagist edges too. |
| 92 | + |
| 93 | +_Negative / trade-offs:_ |
| 94 | + |
| 95 | +- `depends_on_version_id` is NULL for every Packagist edge — a consumer that wants the exact version |
| 96 | + a dependency pulls in must resolve the constraint itself; there is no per-edge pinned version the |
| 97 | + way there is for npm/maven/pypi/cargo seeded from deps.dev. |
| 98 | +- Transitive questions ("everything X depends on") pay a recursive walk over direct edges at read |
| 99 | + time rather than a single indexed lookup. |
| 100 | + |
| 101 | +_Risks:_ |
| 102 | + |
| 103 | +- If an LFX product later needs per-edge **resolved** versions for Packagist, that is a |
| 104 | + **requirements change** — it reopens the "do not store the resolved graph at ingest" decision and |
| 105 | + would require standing up a Composer-style resolver (and accepting the re-resolution churn deps.dev |
| 106 | + fights). Flagged here so the trade-off is revisited deliberately, not silently. |
| 107 | + |
| 108 | +**Decided**: 2026-07-13 |
| 109 | + |
| 110 | +--- |
| 111 | + |
| 112 | +### Metadata enrichment scope: all packages, not the critical slice |
| 113 | + |
| 114 | +The metadata lane (p2 manifests → `versions`, `package_dependencies` edges, and the package-level |
| 115 | +aggregates `licenses` / `latest_version` / `versions_count` / `first_release_at` / |
| 116 | +`latest_release_at` / `homepage`) runs for **all ~500K Packagist packages**, not only the critical |
| 117 | +slice: `CROWD_PACKAGES_PACKAGIST_RUN_ONLY_FOR_CRITICAL` defaults to `false` (setting it `true` |
| 118 | +narrows back to the critical slice). The same lane additionally links `repos` / |
| 119 | +`package_repos` rows for all packages (`'declared'` @ `0.8`, the manifest-declared convention |
| 120 | +shared with npm/pypi/maven/cargo), so the shared github-repos-enricher picks Packagist repos up. |
| 121 | + |
| 122 | +**Provenance.** Requested in the deps.dev-parity review (Joana): deps.dev has zero Packagist data, |
| 123 | +so unlike every other ecosystem there is no external source backfilling the non-critical majority — |
| 124 | +the tiering left most of the catalog thin for no data-availability reason. The goal is best-effort |
| 125 | +parity with what deps.dev populates for its ecosystems. |
| 126 | + |
| 127 | +**Why this deliberately diverges from pypi:** the pypi sibling documents critical-only as the |
| 128 | +"intended steady state" with all-packages as a temporary bootstrap mode. That design assumes |
| 129 | +deps.dev provides package-level data for the whole ecosystem, so per-package registry enrichment |
| 130 | +only needs to deepen the critical slice. Packagist has no such backstop; the registry crawl *is* |
| 131 | +the universe source. The p2 endpoint is a static, CDN-served file designed to be mirrored (it is |
| 132 | +how Composer clients resolve), with `If-Modified-Since`/304 replay — so the steady-state weekly |
| 133 | +cost after the first full pass is dominated by 304s, not re-parses. |
| 134 | + |
| 135 | +**What stays critical-only (deliberate, not parity gaps):** `downloads_daily` and |
| 136 | +maintainers. deps.dev carries no downloads at all, and npm/pypi both scope daily downloads and |
| 137 | +maintainer enrichment to the critical slice; ungating maintainers would also cost ~3–4M per-row |
| 138 | +DB round-trips per weekly cycle (`upsertPackageMaintainers` is 2M+2 queries per package) — a |
| 139 | +batching rewrite is a prerequisite before that scope can ever widen. |
| 140 | + |
| 141 | +**Consequences.** |
| 142 | + |
| 143 | +_Positive:_ versions, direct dependency edges, licenses/latest-version/release-date aggregates, |
| 144 | +homepage, and repo links exist for the whole catalog — matching deps.dev's coverage shape; ranking |
| 145 | +inputs and downstream consumers see a fully-populated ecosystem rather than a thin one. |
| 146 | + |
| 147 | +_Negative / trade-offs:_ the weekly metadata drain walks ~500K purls (~10k keyset batches, ~500 |
| 148 | +continueAsNew generations) instead of ~56K; `versions` grows by an estimated 5–10M rows and |
| 149 | +`package_dependencies` by ~20–40M edges (both well within the tables' 90M+/1.15B+ design points); |
| 150 | +the two packagist workers now default to opposite scopes, mitigated by explicit comments at both |
| 151 | +flip points. |
| 152 | + |
| 153 | +_Risks:_ the all-packages due-selection abandons the partial `is_critical` index and rides the |
| 154 | +`purl` btree keyset walk — late-cycle batches skip progressively more already-scanned rows; if the |
| 155 | +tail slows measurably, revisit with an `EXPLAIN` (flagged in the rollout checklist). |
| 156 | + |
| 157 | +**Decided**: 2026-07-16 |
| 158 | + |
| 159 | +--- |
| 160 | + |
| 161 | +### Lane architecture: seed → chained enrichment, downloads as dedicated lanes |
| 162 | + |
| 163 | +Four workflows instead of the earlier seed/stats/metadata split (the `ingestPackagistStats` |
| 164 | +workflow, whose name and mixed responsibilities were unclear, was removed): |
| 165 | + |
| 166 | +1. **`seedPackagistPackages`** — weekly cron. Discovery only; on completion **chain-starts the |
| 167 | + enrichment drain as a child workflow** (`ParentClosePolicy.ABANDON`) instead of a second cron, |
| 168 | + so "after seed" is an event, not a clock offset (same idiom as `bootstrapOsspckgs`). |
| 169 | +2. **`ingestPackagistMetadata`** — the merged enrichment lane: per package it fetches **both** |
| 170 | + registry endpoints — dynamic (package info, counters, repo link, maintainers) and p2 (versions, |
| 171 | + dependencies, aggregates, homepage) — under one `metadata_last_run_at` watermark. |
| 172 | +3. **`ingestPackagistDownloads30d`** — monthly cron **on the 1st**: captures the observed rolling |
| 173 | + 30d value as the month's `downloads_last_30d` window row (mirrored to the packages column), |
| 174 | + npm's breadth pattern (run-scoped cutoff, per-purl watermark). Running on the boundary replaces |
| 175 | + the earlier "first scan on/after the 1st" heuristic. No history/backfill lane — Packagist keeps |
| 176 | + no historical series, so a missed month is unrecoverable by design. |
| 177 | +4. **`ingestPackagistDownloadsDaily`** — daily cron, critical slice only. |
| 178 | + |
| 179 | +Each lane owns its own watermark + run result in `packagist_package_state` |
| 180 | +(`metadata_* / downloads_30d_* / daily_downloads_*`; the ambiguous `p2_last_modified` renamed to |
| 181 | +`metadata_last_modified`). Trade-offs: the enrichment lane makes two HTTP calls per package |
| 182 | +(bounded by the dynamic endpoint's 10-concurrency), the dynamic endpoint is fetched by three lanes |
| 183 | +at different cadences (~500K extra fetches/month for the 30d lane), and a hard seed failure skips |
| 184 | +that week's enrichment (recoverable via the manual trigger). |
| 185 | + |
| 186 | +**Decided**: 2026-07-16 |
| 187 | + |
| 188 | +--- |
| 189 | + |
| 190 | +## Changelog |
| 191 | + |
| 192 | +- **2026-07-13** — ADR created. First entry: _Dependency model: direct edges + declared constraints, |
| 193 | + resolved at query time_. |
| 194 | +- **2026-07-16** — Added _Metadata enrichment scope: all packages, not the critical slice_ (also |
| 195 | + covers repo linking for all packages and the deliberate critical-only carve-outs). |
| 196 | +- **2026-07-16** — Added _Lane architecture: seed → chained enrichment, downloads as dedicated |
| 197 | + lanes_ (removes the stats lane; renames `p2_last_modified` → `metadata_last_modified`). |
| 198 | +- **2026-07-17** — Corrected stale "critical slice only" wording left over in the _Dependency |
| 199 | + model_ entry from before the _Metadata enrichment scope_ decision widened it to all packages; |
| 200 | + set `Status` to `accepted` (matching ADR-0005's precedent for a living/consolidated doc). |
0 commit comments