feat: split GO/NUGET dependent counts, incremental edge-diff + snapshot guard (CM-1281)

[themarolt]

Summary

Three deps.dev ingestion fixes for CM-1281. (1) The incremental package_dependencies
export keyed on the re-resolved to_version, so stable edges churned out ~555M rows every
run (0.5% PG yield, ~5h). (2) deps.dev shipped corrupt resolved-graph snapshots (06-11/06-15:
every version collapsed to 1–2 edges duplicated ~100×) and nothing stopped a run from burning
5h on garbage. (3) deps.dev's Dependents table excludes GO/NUGET entirely, so those two
ecosystems had no dependent counts at all.

Changes

• Incremental edge-diff (depsSql.ts): match today-vs-watermark on the edge identity
  (ecosystem, root_name, root_version, to_name), dropping the re-resolved to_version from
  the diff key. No-op for PG (unique key omits depends_on_version_id, merge is
  ON CONFLICT DO NOTHING) — just stops exporting churn PG was discarding. ~555M → low-tens-
  of-millions, ~5h → minutes. Edge-level (not version-level) diff so late-resolved edges (~14%)
  aren't permanently missed.
• Edge-snapshot quality guard (checkEdgeSnapshotQuality.ts): probes high-fanout canary
  packages on the today partition before the export; aborts (Slack alert, existing rows
  preserved) when ≥half show the ×100 duplication ratio. Cluster-pruned filter keeps it at
  ~pennies. Soft-fails only the deps kind — rest of the bootstrap proceeds.
• dependent_counts 3-way split (dependentCountsSql.ts, ingestDependentCounts.ts):
  edge systems (NPM/MAVEN/PYPI/CARGO) keep the Dependents reverse-index query under the
  existing dependent_counts kind; GO and NUGET get an exact reverse transitive closure
  (semi-naive fixpoint BQ script) as new kinds dependent_counts_go / dependent_counts_nuget.
  Disjoint purl spaces, per-kind guard baselines, independent soft-fail. See ADR-0004.

Type of change

• New feature
• Performance improvement

JIRA ticket

https://linuxfoundation.atlassian.net/browse/CM-1281

---

Note

High Risk
Touches core package dependency and popularity metrics with expensive GO closure BQ jobs (~$15/week) and changed incremental diff semantics; guards limit corrupt-data blast radius but wrong closure or diff logic would skew rankings silently.

Overview
Fixes three deps.dev ingestion problems: incremental dependency churn, corrupt resolved-graph snapshots, and missing GO/NUGET dependent metrics.

Incremental package_dependencies now diffs today vs watermark on edge identity (ecosystem, root_name, root_version, to_name) without to_version, so weekly re-resolution churn no longer drives ~555M-row exports. Today CTEs use GROUP BY + MAX for resolved fields.

Pre-export edge snapshot guard (checkEdgeSnapshotQuality) probes high-fanout canaries on the same BQ source as the ingest; corrupt ×100 duplication or mass collapse aborts before export with Slack + soft-fail so bootstrap continues. Wired for Option A only.

Dependent counts split three ways: NPM/MAVEN/PYPI/CARGO stay on Dependents as dependent_counts; GO and NUGET get new kinds with exact reverse transitive closure BQ scripts (isScript on bqExportToGcs, maximumBytesBilled, per-kind row guards). ADR-0004 documents the closure choice and cost.

Pipeline hardening: ingest jobs mark failed on BQ/staging/merge errors; weekly bootstrap uses workflowRunTimeout: 24h per attempt; noop cleanup schedule removed; local pg-copy-streams types replace abandoned @types.

^{Reviewed by Cursor Bugbot for commit 7dd4205. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 22 out of 25 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

[Copilot]

Pull request overview

Copilot reviewed 23 out of 26 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

[Copilot]

Pull request overview

Copilot reviewed 25 out of 28 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4a7e5ca. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 25 out of 28 changed files in this pull request and generated 7 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

[Copilot]

Pull request overview

Copilot reviewed 25 out of 28 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file