feat: split GO/NUGET dependent counts, incremental edge-diff + snapshot guard (CM-1281) - #4273
Merged
Merged
Conversation
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
…ev-ingestion-CM-1281 # Conflicts: # services/apps/packages_worker/src/deps-dev/queries/depsSql.ts # services/apps/packages_worker/src/deps-dev/workflows/bootstrapOsspckgs.ts # services/apps/packages_worker/src/deps-dev/workflows/ingestDependencies.ts
Signed-off-by: Uroš Marolt <uros@marolt.me>
…ev-ingestion-CM-1281
…ev-ingestion-CM-1281
Signed-off-by: Uroš Marolt <uros@marolt.me>
…ev-ingestion-CM-1281
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
…ev-ingestion-CM-1281
mbani01
previously approved these changes
Jun 29, 2026
Signed-off-by: Uroš Marolt <uros@marolt.me>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4a7e5ca. Configure here.
…ev-ingestion-CM-1281
Signed-off-by: Uroš Marolt <uros@marolt.me>
…ev-ingestion-CM-1281
skwowet
pushed a commit
that referenced
this pull request
Jul 3, 2026
…ot guard (CM-1281) (#4273) Signed-off-by: Uroš Marolt <uros@marolt.me> Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three deps.dev ingestion fixes for CM-1281. (1) The incremental
package_dependenciesexport keyed on the re-resolved
to_version, so stable edges churned out ~555M rows everyrun (0.5% PG yield, ~5h). (2) deps.dev shipped corrupt resolved-graph snapshots (06-11/06-15:
every version collapsed to 1–2 edges duplicated ~100×) and nothing stopped a run from burning
5h on garbage. (3) deps.dev's
Dependentstable excludes GO/NUGET entirely, so those twoecosystems had no dependent counts at all.
Changes
depsSql.ts): match today-vs-watermark on the edge identity(ecosystem, root_name, root_version, to_name), dropping the re-resolvedto_versionfromthe diff key. No-op for PG (unique key omits
depends_on_version_id, merge isON CONFLICT DO NOTHING) — just stops exporting churn PG was discarding. ~555M → low-tens-of-millions, ~5h → minutes. Edge-level (not version-level) diff so late-resolved edges (~14%)
aren't permanently missed.
checkEdgeSnapshotQuality.ts): probes high-fanout canarypackages on the today partition before the export; aborts (Slack alert, existing rows
preserved) when ≥half show the ×100 duplication ratio. Cluster-pruned filter keeps it at
~pennies. Soft-fails only the deps kind — rest of the bootstrap proceeds.
dependentCountsSql.ts,ingestDependentCounts.ts):edge systems (NPM/MAVEN/PYPI/CARGO) keep the
Dependentsreverse-index query under theexisting
dependent_countskind; GO and NUGET get an exact reverse transitive closure(semi-naive fixpoint BQ script) as new kinds
dependent_counts_go/dependent_counts_nuget.Disjoint purl spaces, per-kind guard baselines, independent soft-fail. See ADR-0004.
Type of change
JIRA ticket
https://linuxfoundation.atlassian.net/browse/CM-1281
Note
High Risk
Touches core package dependency and popularity metrics with expensive GO closure BQ jobs (~$15/week) and changed incremental diff semantics; guards limit corrupt-data blast radius but wrong closure or diff logic would skew rankings silently.
Overview
Fixes three deps.dev ingestion problems: incremental dependency churn, corrupt resolved-graph snapshots, and missing GO/NUGET dependent metrics.
Incremental
package_dependenciesnow diffs today vs watermark on edge identity(ecosystem, root_name, root_version, to_name)withoutto_version, so weekly re-resolution churn no longer drives ~555M-row exports. Today CTEs useGROUP BY+MAXfor resolved fields.Pre-export edge snapshot guard (
checkEdgeSnapshotQuality) probes high-fanout canaries on the same BQ source as the ingest; corrupt ×100 duplication or mass collapse aborts before export with Slack + soft-fail so bootstrap continues. Wired for Option A only.Dependent counts split three ways: NPM/MAVEN/PYPI/CARGO stay on
Dependentsasdependent_counts; GO and NUGET get new kinds with exact reverse transitive closure BQ scripts (isScriptonbqExportToGcs,maximumBytesBilled, per-kind row guards). ADR-0004 documents the closure choice and cost.Pipeline hardening: ingest jobs mark
failedon BQ/staging/merge errors; weekly bootstrap usesworkflowRunTimeout: 24hper attempt; noop cleanup schedule removed; localpg-copy-streamstypes replace abandoned@types.Reviewed by Cursor Bugbot for commit 7dd4205. Bugbot is set up for automated code reviews on this repo. Configure here.