feat: fix maven repo gap (CM-1305) - #4311
Merged
Merged
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR closes a gap in how Maven package repository links are derived. It rewrites normalizeScmUrl into a stricter, host-validated canonicalizer that only emits https://<host>/<owner>/<repo> for known SCM hosts (returning null for homepages, placeholders, and non-SCM hosts), and adds a one-shot backfill that re-runs this normalizer over already-stored declared_repository_url values — without re-fetching POMs — to fill previously-dropped repository_url values (Gap B) and clear non-repository values (Gap C).
Changes:
- Reworked
normalizeScmUrlto canonicalize SCM URLs against an allow-list of hosts (github,gitlab,bitbucket,gitee,codeberg), handling scheme-less,git+, SCP, andssh://forms, with case-folding for GitHub/GitLab. - Added DAL helpers (
listMavenPackagesForRepoUrlRecompute,updateMavenRepositoryUrls) plus a resumable, dry-run-capable backfill (backfillMavenRepositoryUrls) and CLI entrypoint. - Extended unit tests for the new normalizer behavior and added
npmscripts for the backfill.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| services/apps/packages_worker/src/maven/extract.ts | Rewrites normalizeScmUrl into a host-validated canonicalizer; core of the fix. |
| services/libs/data-access-layer/src/osspckgs/packages.ts | Adds keyset scan + batched clear/set UPDATE helpers used by the backfill. |
| services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts | New resumable/idempotent backfill loop recomputing repository_url from stored data. |
| services/apps/packages_worker/src/bin/maven-repo-url-backfill.ts | CLI entrypoint with arg parsing, graceful shutdown, and DB connect. |
| services/apps/packages_worker/src/maven/tests/normalize.test.ts | Adds Gap B/Gap C test cases for the new normalizer. |
| services/apps/packages_worker/package.json | Adds backfill:maven-repo-url npm scripts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+158
to
+176
| export type MavenRepoUrlRow = { | ||
| id: number | ||
| declaredRepositoryUrl: string | null | ||
| repositoryUrl: string | null | ||
| } | ||
|
|
||
| /** | ||
| * Keyset-paginated scan of Maven rows that carry a repository link (declared or | ||
| * canonical). Rows with neither are skipped — there is nothing to recompute. | ||
| * Used by the repository_url backfill to re-run the normalizer over stored data | ||
| * without re-fetching POMs from the registry. | ||
| * | ||
| * `criticalOnly` restricts the scan to is_critical rows (index-backed by the | ||
| * partial index on is_critical) — used for a fast, consumer-facing first pass. | ||
| */ | ||
| export async function listMavenPackagesForRepoUrlRecompute( | ||
| qx: QueryExecutor, | ||
| options: { afterId: number; limit: number; criticalOnly?: boolean }, | ||
| ): Promise<MavenRepoUrlRow[]> { |
Comment on lines
+105
to
+111
| await qx.tx(async (t) => { | ||
| await updateMavenRepositoryUrls(t, updates) | ||
| await deleteMavenPackageRepoLinks(t, pruneTargets) | ||
| for (const target of linkTargets) { | ||
| await writeRepoLink(t, target.id, target.repositoryUrl) | ||
| } | ||
| }) |
ulemons
force-pushed
the
fix/maven-repo-gap
branch
from
July 10, 2026 08:27
b700368 to
53455dd
Compare
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
ulemons
force-pushed
the
fix/maven-repo-gap
branch
from
July 13, 2026 07:46
53455dd to
97ade13
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 97ade13. Configure here.
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the Maven half of the
repository_urlnormalization problem:packages.repository_urlmust always hold a canonicalhttps://<host>/<owner>/<repo>link orNULL, never a website, placeholder, or free-form string. The Maven normalizer (normalizeScmUrl) had two defects:startsWith('https://')discarded inputs whose repo was clearly reconstructable:scm:git:prefixes without a scheme, barehost/owner/repovalues, andhttp://URLs that were never upgraded tohttps://. This leftrepository_urlNULL for ~18.8k critical Maven rows where the declared value was a valid GitHub URL.https://meson.ai/) passed straight through and were stored as if they were repository links.Since the public API falls back to
declared_repository_urlwhenrepository_urlis NULL, these gaps meant clients received either nothing or raw, non-normalized registry values.This PR rewrites
normalizeScmUrlto parse and validate instead of regex-guess, and adds a DB-only backfill to correct existing rows.Scope: Maven only. Cargo/npm/NuGet gaps, the shared normalizer + ADR, and the API fallback removal are tracked separately.
Changes
normalizeScmUrl(services/apps/packages_worker/src/maven/extract.ts) to normalize viaURL()parsing rather than a regex chain. It now: stripsscm:git:/scm:/git+prefixes and SCP/ssh:///git://forms; prependshttps://to schemeless inputs and upgradeshttp://→https://(Gap B); requires a known SCM host and anowner/repopath shape, returning NULL otherwise (Gap C); and lower-cases the path on case-insensitive hosts (github/gitlab).SCM_HOSTS/CASE_INSENSITIVE_HOSTSare markedTODO(CM)pending product confirmation of the final host list. The trade-off: an allowlist reliably rejects doc-sites but also drops self-hosted git (e.g.gitbox.apache.org); needs a decision before rollout.maven/__tests__/normalize.test.ts). All pre-existing cases still pass, includingsvn://→ NULL.backfill:maven. The existing backfill re-fetches every POM over the network (~18.8k+ HTTP calls) and — critically — cannot clear Gap C junk, because the enrichment upsertCOALESCEs and can never write NULL. The new path re-runs the normalizer over the already-storeddeclared_repository_url(zero network) and applies a direct UPDATE, so it both fills recoverable NULLs (Gap B) and clears non-repo values (Gap C).services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts— keyset-paginated, idempotent, resumable orchestration; counts filled/cleared/rewritten/unchanged/linked/pruned; supports--dry-run.repository_url. This matters because consumers like the security-contacts pipeline read the repository throughrepos ⋈ package_repos, notpackages.repository_url— so a fill that only touchedpackageswould stay invisible to them. On rewrites and clears the stalesource='declared'link is pruned; on fills and rewrites the correct link is (re)written via the enrichment loop'swriteRepoLink(now exported fromrunMavenEnrichmentLoop.ts). This is deliberately stricter than the incremental enrichment path, which is upsert-only and never prunes — so the backfill also cleans up pre-existing stale links rather than leaving zombies.services/apps/packages_worker/src/bin/maven-repo-url-backfill.ts— bin entrypoint with graceful shutdown and positive-integer validation of the batch-size env var.services/libs/data-access-layer/src/osspckgs/packages.ts— newlistMavenPackagesForRepoUrlRecompute+updateMavenRepositoryUrls(splits clears from sets to avoid NULLs inside atext[]literal).services/libs/data-access-layer/src/osspckgs/repos.ts— newdeleteMavenPackageRepoLinks(removes onlysource='declared'links; never deletes sharedreposrows).package.json— newbackfill:maven-repo-url[:local]scripts.last_synced_at, so it doesn't disturb the enrichment freshness window.Type of change
JIRA ticket
CM-1305
Note
Medium Risk
Bulk updates to
packages.repository_url,package_repos, andlast_synced_ataffect security-contacts and Tinybird CDC; host allowlist is provisional (TODO CM) so rollout may drop or remap links until confirmed.Overview
Rewrites Maven
normalizeScmUrlto parse withURL(), recover previously dropped SCM shapes (schemeless/http/SCP), reject non-repo URLs via host allowlists and dedicated handlers (Apache gitweb, GitLab namespaces, GitHub Pages, etc.), and addinterpolatePropertiesso POM extraction resolves${...}in SCM URLs from merged parent-chain properties.Adds a DB-only repo-url backfill (
backfill:maven-repo-url) that recomputesrepository_urlfrom storeddeclared_repository_url(no POM fetch), can NULL out junk values the enrichment upsert cannot clear, and keepspackage_reposin sync via transactional UPDATE + prune/relink ofsource='declared'links.backfill:maven --forceruns a terminating id-keyset full critical POM re-extraction (runMavenCriticalForceBackfill+listMavenCriticalPackagesById) for rolling out interpolation/normalizer changes without re-fetching only via the staleness queue.Reviewed by Cursor Bugbot for commit fc09d51. Bugbot is set up for automated code reviews on this repo. Configure here.