|
| 1 | +# ADR-0004: Compute GO/NUGET transitive dependent counts via exact reverse closure (over HLL approximation) |
| 2 | + |
| 3 | +**Date**: 2026-06-23 |
| 4 | +**Status**: accepted |
| 5 | +**Deciders**: Uroš Marolt |
| 6 | + |
| 7 | +## Context |
| 8 | +deps.dev publishes its reverse-dependent index (`Dependents`) and its resolved/transitive |
| 9 | +dependency graph (`Dependencies`, `DependencyGraphEdges`) for only four ecosystems — |
| 10 | +NPM, MAVEN, PYPI, CARGO (verified via BQ 2026-06-23: each `GROUP BY System` returns exactly |
| 11 | +those four). GO and NUGET have only raw direct manifests (`GoRequirementsLatest`, |
| 12 | +`NuGetRequirementsLatest`). The `dependent_counts` ingest was therefore split three ways |
| 13 | +(edges / go / nuget — see the dependent-counts split). Direct `dependent_count` for GO/NUGET |
| 14 | +is a cheap manifest invert, but `transitive_dependent_count` (MinimumDepth>1) and the all-depth |
| 15 | +`dependent_repos_count` require the reverse **transitive closure**, which deps.dev does not |
| 16 | +provide for these two ecosystems — so we must compute it ourselves from the direct edges. |
| 17 | + |
| 18 | +## Decision |
| 19 | +Compute the **exact** reverse transitive closure for GO/NUGET — a semi-naive fixpoint over |
| 20 | +module-level edges (names hashed to INT64 via `FARM_FINGERPRINT`, persisted + clustered) iterated |
| 21 | +to convergence — so GO/NUGET transitive counts are true integers matching the edge-system |
| 22 | +methodology. We chose exact over an HLL-sketch approximation, accepting the higher cost for now. |
| 23 | + |
| 24 | +## Alternatives Considered |
| 25 | + |
| 26 | +### Alternative 1: HLL sketch propagation (approximate) |
| 27 | +- **Pros**: ~cents/run, seconds, tiny tables (one HyperLogLog sketch per subject, ~188K rows for |
| 28 | + GO, merged along edges to a fixpoint — never materializes the billions of pairs). Full-depth. |
| 29 | +- **Cons**: probabilistic cardinality estimate with bounded relative error ≈ `1.04/sqrt(2^p)` |
| 30 | + (~1.6% at precision 12, ~0.4% at 15). Methodology differs from the exact edge-system counts, so |
| 31 | + GO/NUGET would be ~1% approximate while the other four ecosystems are exact. |
| 32 | +- **Why not**: the decision is to have identical, exact integer methodology across all six |
| 33 | + ecosystems. The measured exact cost (below) is acceptable for now. HLL remains the documented |
| 34 | + fallback if weekly cost becomes prohibitive. |
| 35 | + |
| 36 | +### Alternative 2: Bounded-depth closure (cap at K hops) |
| 37 | +- **Pros**: K deterministic joins in a single statement, dry-runnable, cost-bounded, no scripting. |
| 38 | +- **Cons**: undercounts. The GO closure needs **31 hops** to converge; new pairs peak at hop 8 |
| 39 | + (~507M) and only decay to zero by hop 31. Capping at a small K drops hundreds of millions of |
| 40 | + genuine transitive pairs. |
| 41 | +- **Why not**: a low cap is materially wrong; a cap high enough to be correct is no cheaper than |
| 42 | + full convergence. |
| 43 | + |
| 44 | +## Consequences |
| 45 | + |
| 46 | +### Positive |
| 47 | +- Exact integer counts, full depth, identical methodology to NPM/MAVEN/PYPI/CARGO. |
| 48 | +- Deterministic and reproducible; no estimator error to explain to downstream consumers. |
| 49 | +- Termination is guaranteed by construction: each iteration's frontier is anti-joined against the |
| 50 | + accumulated `reach`, so only brand-new pairs survive and the finite pair space converges (proven: |
| 51 | + GO converged at hop 31 with `new_pairs → 0`). |
| 52 | + |
| 53 | +### Negative |
| 54 | +- This is by far the heaviest job in the packages pipeline. Measured exact runs (2026-06-23, |
| 55 | + INT64-fingerprint-optimized; raw string names would be ~5×). The build-`reach` figures are from the |
| 56 | + closure probe; the full-pipeline figures (incl. the all-depth repos aggregation + per-subject |
| 57 | + counts → `_export_data`) are from the end-to-end validation run of the production script: |
| 58 | + - **GO**: 5.81B closure pairs, 31 iterations. Build reach ≈ 1.86 TB; **full pipeline = 2.31 TB |
| 59 | + billed (~$14.5 at $6.25/TiB), 99 slot-hours, 16 min wall, 132 child statements** (validated). |
| 60 | + - **NUGET**: 114M closure pairs, 19 iterations, ~4 min wall, ~32 GB billed (~$0.19); full pipeline |
| 61 | + validated → 191,762 subject rows. |
| 62 | + - GO dominates (its graph is ~50× larger and deeper). Combined ≈ **~$15/run, ~20 min, weekly**. |
| 63 | + For comparison the edge `dependent_counts` job scans ~310 GB (~$2). |
| 64 | +- Requires a semi-naive BQ scripting job (not a single exportable SELECT), so it does not fit the |
| 65 | + existing one-query→GCS pipeline shape as-is (integrated via an `isScript` mode — see below). |
| 66 | + |
| 67 | +### Risks |
| 68 | +- Cost/runtime grow with the GO/NUGET graphs. Mitigations: INT64 fingerprints (~5× cheaper), |
| 69 | + clustering on the join keys, and a documented escape hatch to switch GO/NUGET to HLL (Alternative |
| 70 | + 1) if cost becomes prohibitive — counts in the hundred-thousands make ~1% error invisible for |
| 71 | + popularity/ranking. |
| 72 | +- `FARM_FINGERPRINT` collisions are negligible at ~1–2M nodes (~1e-6) but non-zero; acceptable for |
| 73 | + these metrics. Switch to a dictionary-encoded INT id if exactness must be collision-proof. |
| 74 | + |
| 75 | +## Pipeline integration (decided 2026-06-23) |
| 76 | +- **No persistent scratch dataset.** The closure runs as one multi-statement scripting job using |
| 77 | + session-scoped `CREATE TEMP TABLE`s (edges → INT64-fingerprinted edges → `reach` fixpoint → |
| 78 | + fingerprint→name lookup → `_export_data`). Temp tables are clustered on the join keys and are |
| 79 | + auto-dropped when the script's session ends — on success *and* failure — so there is no lingering |
| 80 | + storage to bill or clean up. (The interactive probe used regular tables in |
| 81 | + `lfx-insights:scratch_closure_probe`; that is leftover, dropped separately — not the pipeline.) |
| 82 | +- **Reuses the existing job path, not a new kind.** `bqExportToGcs` gains an `isScript` mode: when |
| 83 | + set, `sql` is the full script (ending in `CREATE TEMP TABLE _export_data AS …`) and the activity |
| 84 | + appends only the `EXPORT DATA … AS SELECT * FROM _export_data` statement rather than wrapping |
| 85 | + `sql` in a subquery. Everything downstream (listParquetFiles → guard → chunked |
| 86 | + gcsParquetToStaging → mergeStagingToTable) is unchanged, and the `dependent_counts_go` / |
| 87 | + `dependent_counts_nuget` kinds, guard baselines, and monitor entries stay continuous. |
| 88 | +- **Cost guard.** A dry-run cannot price a `WHILE` loop, so script mode skips it and enforces the |
| 89 | + byte ceiling server-side via `maximumBytesBilled` (= the per-variant `maxBytesGb`). The script |
| 90 | + also carries an iteration cap as the deterministic runaway guard (convergence already proven: |
| 91 | + GO hop 31, NUGET hop 19, both `new_pairs → 0`). |
| 92 | +- **Final output → counts.** `_export_data` emits `(purl, dependent_count, |
| 93 | + transitive_dependent_count, dependent_repos_count)`: fingerprints in `reach` are mapped back to |
| 94 | + names via a `names` temp table, joined to `purl_map` (name→purl). `transitive_dependent_count = |
| 95 | + COUNT(DISTINCT dep over reach) − direct_count`; `dependent_repos_count` = distinct |
| 96 | + `PackageVersionToProject` SOURCE_REPO_TYPE repos over the all-depth dependent set. |
| 97 | +- **Weekly trigger.** No new schedule — the `go`/`nuget` variants are already child workflows of |
| 98 | + `bootstrapOsspckgs`, which the existing `osspckgs-bootstrap-weekly` schedule runs incrementally. |
| 99 | + |
| 100 | +Related: ADR-0003 (deps BQ table selection — same root cause: GO/NUGET absent from the resolved-graph tables). |
0 commit comments