feat: fix maven repo gap (CM-1305) by ulemons · Pull Request #4311 · linuxfoundation/crowd.dev

ulemons · 2026-07-07T07:37:12Z

Summary

Fixes the Maven half of the repository_url normalization problem: packages.repository_url must always hold a canonical https://<host>/<owner>/<repo> link or NULL, never a website, placeholder, or free-form string. The Maven normalizer (normalizeScmUrl) had two defects:

Gap B — recoverable inputs dropped to NULL. A regex chain gated on startsWith('https://') discarded inputs whose repo was clearly reconstructable: scm:git: prefixes without a scheme, bare host/owner/repo values, and http:// URLs that were never upgraded to https://. This left repository_url NULL for ~18.8k critical Maven rows where the declared value was a valid GitHub URL.
Gap C — non-repository URLs written as repos. After the https gate there was no shape validation, so homepages (e.g. https://meson.ai/) passed straight through and were stored as if they were repository links.

Since the public API falls back to declared_repository_url when repository_url is NULL, these gaps meant clients received either nothing or raw, non-normalized registry values.

This PR rewrites normalizeScmUrl to parse and validate instead of regex-guess, and adds a DB-only backfill to correct existing rows.

Scope: Maven only. Cargo/npm/NuGet gaps, the shared normalizer + ADR, and the API fallback removal are tracked separately.

Changes

Rewrote normalizeScmUrl (services/apps/packages_worker/src/maven/extract.ts) to normalize via URL() parsing rather than a regex chain. It now: strips scm:git:/scm:/git+ prefixes and SCP/ssh:///git:// forms; prepends https:// to schemeless inputs and upgrades http:// → https:// (Gap B); requires a known SCM host and an owner/repo path shape, returning NULL otherwise (Gap C); and lower-cases the path on case-insensitive hosts (github/gitlab).
The SCM host allowlist is provisional — SCM_HOSTS/CASE_INSENSITIVE_HOSTS are marked TODO(CM) pending product confirmation of the final host list. The trade-off: an allowlist reliably rejects doc-sites but also drops self-hosted git (e.g. gitbox.apache.org); needs a decision before rollout.
Added tests for the ticket's Gap B/Gap C cases (maven/__tests__/normalize.test.ts). All pre-existing cases still pass, including svn:// → NULL.
Added a recompute-from-DB backfill rather than reusing backfill:maven. The existing backfill re-fetches every POM over the network (~18.8k+ HTTP calls) and — critically — cannot clear Gap C junk, because the enrichment upsert COALESCEs and can never write NULL. The new path re-runs the normalizer over the already-stored declared_repository_url (zero network) and applies a direct UPDATE, so it both fills recoverable NULLs (Gap B) and clears non-repo values (Gap C).
- services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts — keyset-paginated, idempotent, resumable orchestration; counts filled/cleared/rewritten/unchanged/linked/pruned; supports --dry-run.
Keeps the repos / package_repos link tables consistent with the recomputed repository_url. This matters because consumers like the security-contacts pipeline read the repository through repos ⋈ package_repos, not packages.repository_url — so a fill that only touched packages would stay invisible to them. On rewrites and clears the stale source='declared' link is pruned; on fills and rewrites the correct link is (re)written via the enrichment loop's writeRepoLink (now exported from runMavenEnrichmentLoop.ts). This is deliberately stricter than the incremental enrichment path, which is upsert-only and never prunes — so the backfill also cleans up pre-existing stale links rather than leaving zombies.
- services/apps/packages_worker/src/bin/maven-repo-url-backfill.ts — bin entrypoint with graceful shutdown and positive-integer validation of the batch-size env var.
- services/libs/data-access-layer/src/osspckgs/packages.ts — new listMavenPackagesForRepoUrlRecompute + updateMavenRepositoryUrls (splits clears from sets to avoid NULLs inside a text[] literal).
- services/libs/data-access-layer/src/osspckgs/repos.ts — new deleteMavenPackageRepoLinks (removes only source='declared' links; never deletes shared repos rows).
- package.json — new backfill:maven-repo-url[:local] scripts.
The backfill does not bump last_synced_at, so it doesn't disturb the enrichment freshness window.

Type of change

JIRA ticket

CM-1305

Note

Medium Risk
Bulk updates to packages.repository_url, package_repos, and last_synced_at affect security-contacts and Tinybird CDC; host allowlist is provisional (TODO CM) so rollout may drop or remap links until confirmed.

Overview
Rewrites Maven normalizeScmUrl to parse with URL(), recover previously dropped SCM shapes (schemeless/http/SCP), reject non-repo URLs via host allowlists and dedicated handlers (Apache gitweb, GitLab namespaces, GitHub Pages, etc.), and add interpolateProperties so POM extraction resolves ${...} in SCM URLs from merged parent-chain properties.

Adds a DB-only repo-url backfill (backfill:maven-repo-url) that recomputes repository_url from stored declared_repository_url (no POM fetch), can NULL out junk values the enrichment upsert cannot clear, and keeps package_repos in sync via transactional UPDATE + prune/relink of source='declared' links.

backfill:maven --force runs a terminating id-keyset full critical POM re-extraction (runMavenCriticalForceBackfill + listMavenCriticalPackagesById) for rolling out interpolation/normalizer changes without re-fetching only via the staleness queue.

^{Reviewed by Cursor Bugbot for commit fc09d51. Bugbot is set up for automated code reviews on this repo. Configure here.}

github-actions

Conventional Commits FTW!

Copilot

Pull request overview

This PR closes a gap in how Maven package repository links are derived. It rewrites normalizeScmUrl into a stricter, host-validated canonicalizer that only emits https://<host>/<owner>/<repo> for known SCM hosts (returning null for homepages, placeholders, and non-SCM hosts), and adds a one-shot backfill that re-runs this normalizer over already-stored declared_repository_url values — without re-fetching POMs — to fill previously-dropped repository_url values (Gap B) and clear non-repository values (Gap C).

Changes:

Reworked normalizeScmUrl to canonicalize SCM URLs against an allow-list of hosts (github, gitlab, bitbucket, gitee, codeberg), handling scheme-less, git+, SCP, and ssh:// forms, with case-folding for GitHub/GitLab.
Added DAL helpers (listMavenPackagesForRepoUrlRecompute, updateMavenRepositoryUrls) plus a resumable, dry-run-capable backfill (backfillMavenRepositoryUrls) and CLI entrypoint.
Extended unit tests for the new normalizer behavior and added npm scripts for the backfill.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
services/apps/packages_worker/src/maven/extract.ts	Rewrites `normalizeScmUrl` into a host-validated canonicalizer; core of the fix.
services/libs/data-access-layer/src/osspckgs/packages.ts	Adds keyset scan + batched clear/set UPDATE helpers used by the backfill.
services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts	New resumable/idempotent backfill loop recomputing `repository_url` from stored data.
services/apps/packages_worker/src/bin/maven-repo-url-backfill.ts	CLI entrypoint with arg parsing, graceful shutdown, and DB connect.
services/apps/packages_worker/src/maven/tests/normalize.test.ts	Adds Gap B/Gap C test cases for the new normalizer.
services/apps/packages_worker/package.json	Adds `backfill:maven-repo-url` npm scripts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 8 comments.

+export type MavenRepoUrlRow = {
+  id: number
+  declaredRepositoryUrl: string | null
+  repositoryUrl: string | null
+}
+
+/**
+ * Keyset-paginated scan of Maven rows that carry a repository link (declared or
+ * canonical). Rows with neither are skipped — there is nothing to recompute.
+ * Used by the repository_url backfill to re-run the normalizer over stored data
+ * without re-fetching POMs from the registry.
+ *
+ * `criticalOnly` restricts the scan to is_critical rows (index-backed by the
+ * partial index on is_critical) — used for a fast, consumer-facing first pass.
+ */
+export async function listMavenPackagesForRepoUrlRecompute(
+  qx: QueryExecutor,
+  options: { afterId: number; limit: number; criticalOnly?: boolean },
+): Promise<MavenRepoUrlRow[]> {


Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 3 comments.

+      await qx.tx(async (t) => {
+        await updateMavenRepositoryUrls(t, updates)
+        await deleteMavenPackageRepoLinks(t, pruneTargets)
+        for (const target of linkTargets) {
+          await writeRepoLink(t, target.id, target.repositoryUrl)
+        }
+      })


Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 5 comments.

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 97ade13. Configure here.}

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 9 comments.

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 6 comments.

Copilot AI review requested due to automatic review settings July 7, 2026 07:37

github-actions Bot reviewed Jul 7, 2026

View reviewed changes

Copilot started reviewing on behalf of ulemons July 7, 2026 07:37 View session

ulemons self-assigned this Jul 7, 2026

ulemons added the Bug Created by Linear-GitHub Sync label Jul 7, 2026

cursor Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/extract.ts

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts Outdated

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts

Comment thread services/apps/packages_worker/src/maven/extract.ts

ulemons changed the title ~~feat: fix maven repo gap~~ feat: fix maven repo gap (CM-1305) Jul 7, 2026

Copilot AI review requested due to automatic review settings July 7, 2026 08:51

Copilot started reviewing on behalf of ulemons July 7, 2026 08:51 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts Outdated

Copilot AI review requested due to automatic review settings July 7, 2026 13:25

Copilot started reviewing on behalf of ulemons July 7, 2026 13:25 View session

cursor Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts Outdated

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/extract.ts Outdated

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts Outdated

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts Outdated

Copilot AI review requested due to automatic review settings July 8, 2026 11:32

Copilot started reviewing on behalf of ulemons July 8, 2026 11:33 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 08:51

Copilot started reviewing on behalf of ulemons July 9, 2026 08:52 View session

cursor Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/runMavenEnrichmentLoop.ts

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 09:36

Copilot started reviewing on behalf of ulemons July 9, 2026 09:37 View session

cursor Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/extract.ts

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread services/libs/data-access-layer/src/osspckgs/packages.ts

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts Outdated

Copilot AI review requested due to automatic review settings July 10, 2026 08:03

Copilot started reviewing on behalf of ulemons July 10, 2026 08:03 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 08:27

ulemons force-pushed the fix/maven-repo-gap branch from b700368 to 53455dd Compare July 10, 2026 08:27

Copilot started reviewing on behalf of ulemons July 10, 2026 08:28 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

ulemons added 11 commits July 13, 2026 09:46

feat: fix maven repo gap

dd15ccd

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: remove zombies

a62f998

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: critical onlhy

0bf3fa2

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: add allowlist

ee73e33

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: add allowlist

f680b52

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: add force to backfill

7d6771d

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: add apache normalize

ed269a3

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: add eclipse normalize

5fb4a60

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: add eclipse normalize

2441129

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: comments review

5e5c010

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

fix: refactor query executor

97ade13

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

Copilot AI review requested due to automatic review settings July 13, 2026 07:46

ulemons force-pushed the fix/maven-repo-gap branch from 53455dd to 97ade13 Compare July 13, 2026 07:46

Copilot started reviewing on behalf of ulemons July 13, 2026 07:47 View session

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread services/apps/packages_worker/src/maven/backfillRepositoryUrl.ts

Copilot AI reviewed Jul 13, 2026

View reviewed changes

fix: comments

fc09d51

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

Copilot AI review requested due to automatic review settings July 13, 2026 08:35

Copilot started reviewing on behalf of ulemons July 13, 2026 08:36 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

ulemons merged commit 7735a8d into main Jul 13, 2026
18 checks passed

ulemons deleted the fix/maven-repo-gap branch July 13, 2026 08:46

Uh oh!

Conversation

ulemons commented Jul 7, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Type of change

JIRA ticket

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ulemons commented Jul 7, 2026 •

edited by cursor Bot

Loading

github-actions Bot left a comment •

edited

Loading