feat: cargo registry implementation [CM-1264] - #4236
Conversation
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
|
|
There was a problem hiding this comment.
Pull request overview
This PR adds initial Cargo (crates.io) registry ingestion support to services/apps/packages_worker, introducing a dedicated Temporal worker/task-queue that downloads the crates.io DB dump, stages it into a schema in packages-db, enriches package metadata, and records per-field audit changes.
Changes:
- Added a new Cargo Temporal workflow + schedule + worker entrypoint (
cargo-worker) to run a daily registry sync. - Implemented dump download/extract + high-volume
COPY FROM STDINload into a staging schema, plus set-based enrichment SQL phases. - Updated local/dev wiring (Docker Compose + workspace scripts) and introduced
pg-copy-streamsfor streaming COPY.
Reviewed changes
Copilot reviewed 15 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| services/apps/packages_worker/tmp-test-versions.ts | Adds a temporary local script for manually running enrichVersions (needs cleanup before merge). |
| services/apps/packages_worker/src/workflows/index.ts | Exports the new cargoSyncWorkflow from the shared workflows index. |
| services/apps/packages_worker/src/db.ts | Adds getPackagesDbConnection() to expose the raw pg-promise connection for COPY streaming. |
| services/apps/packages_worker/src/config.ts | Adds getCargoConfig() (currently only CARGO_DUMP_URL). |
| services/apps/packages_worker/src/cargo/workflows.ts | Defines the cargoSyncWorkflow orchestration and activity timeouts/retries. |
| services/apps/packages_worker/src/cargo/types.ts | Adds Cargo-specific result/config types used across activities/enrichment. |
| services/apps/packages_worker/src/cargo/schedule.ts | Registers a daily Temporal Schedule to run the Cargo sync workflow on cargo-worker queue. |
| services/apps/packages_worker/src/cargo/loadDump.ts | Implements staging schema DDL + CSV COPY loading + aggregation/denormalization into enrich tables. |
| services/apps/packages_worker/src/cargo/enrich.ts | Implements enrichment phases (packages/versions/repos/maintainers/downloads) + audit logging. |
| services/apps/packages_worker/src/cargo/dump.ts | Downloads the crates.io tarball with timeout and extracts it to /tmp/cargo-dump. |
| services/apps/packages_worker/src/cargo/activities.ts | Wires dump download/load + enrichment phases into Temporal activities and cleanup. |
| services/apps/packages_worker/src/bin/cargo-worker.ts | New worker entrypoint that initializes the service, schedules Cargo sync, and starts processing. |
| services/apps/packages_worker/src/activities.ts | Re-exports Cargo activities from the root activities index. |
| services/apps/packages_worker/package.json | Adds start/dev scripts for cargo-worker and adds pg-copy-streams deps. |
| scripts/services/cargo-worker.yaml | Adds Docker Compose definitions for cargo-worker and cargo-worker-dev. |
| scripts/builders/packages.env | Adds cargo-worker to the packages image/service build list. |
| pnpm-lock.yaml | Locks pg-copy-streams (and types) and includes incidental lockfile metadata changes. |
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 45e776f. Configure here.
| SERVICE: cargo-worker | ||
| SHELL: /bin/sh | ||
| SUPPRESS_NO_CONFIG_WARNING: 'true' | ||
| CROWD_TEMPORAL_TASKQUEUE: cargo-worker |
There was a problem hiding this comment.
Missing packages Temporal namespace
Medium Severity
The cargo worker compose env block sets CROWD_TEMPORAL_TASKQUEUE but not CROWD_TEMPORAL_NAMESPACE: ${CROWD_PACKAGES_TEMPORAL_NAMESPACE} like npm-worker.yaml and osv-worker.yaml. The worker and schedule then use the default Temporal namespace instead of the packages namespace when those differ in deployment.
Reviewed by Cursor Bugbot for commit 45e776f. Configure here.
| export async function flushAudit(qx: QueryExecutor): Promise<number> { | ||
| return qx.result( | ||
| `INSERT INTO audit_field_changes (worker, purl, changed_fields) | ||
| SELECT $(worker), p.purl, array_agg(DISTINCT ac.field) |
| export async function loadDump( | ||
| qx: QueryExecutor, | ||
| db: DbConnection, | ||
| dumpDir: string, | ||
| ): Promise<LoadResult> { | ||
| const startedAt = Date.now() | ||
| const dataDir = path.join(dumpDir, 'data') |
| const STAGING_DDL = ` | ||
| DROP TABLE IF EXISTS ${STAGING_SCHEMA}.crates CASCADE; | ||
| CREATE TABLE ${STAGING_SCHEMA}.crates ( | ||
| created_at text, description text, documentation text, homepage text, | ||
| id integer, max_features text, max_upload_size text, name text, | ||
| readme text, repository text, trustpub_only text, updated_at text | ||
| ); |
This pull request introduces support for a new Cargo package worker in the system, including its configuration, build, runtime scripts, and all necessary dependencies. The changes enable the service to download, extract, load, enrich, and clean up Cargo package data, integrating it into the existing packages worker infrastructure.
Major changes include:
Cargo Worker Integration
scripts/services/cargo-worker.yaml) to define build, development, and runtime environments for the Cargo worker service.scripts/builders/packages.env) to includecargo-workerin the list of services to be built and deployed.services/apps/packages_worker/package.jsonwith new scripts for starting and developing the Cargo worker, and includedpg-copy-streamsand its types as dependencies. [1] [2] [3]Cargo Worker Implementation
src/activities.tsto expose all Cargo-related activities for workflow orchestration.src/cargo/activities.tswith functions to handle downloading, loading, enriching, and cleaning up Cargo data, leveraging shared database utilities and logging.src/cargo/dump.tsto handle downloading and extracting Cargo database dumps, including robust error handling and cleanup logic.Dependency and Lockfile Updates
pnpm-lock.yamlto addpg-copy-streams,@types/pg-copy-streams, and related dependencies, ensuring the new worker has the required packages for PostgreSQL streaming and type support. [1] [2] [3] [4] [5] [6]Other Dependency Adjustments
pnpm-lock.yamlto keep AWS SDK and other packages consistent, though these are less central to the Cargo worker addition. [1] [2] [3] [4] [5] [6] [7] [8] [9]These changes collectively provide the infrastructure and code needed to support Cargo package ingestion and enrichment as a first-class workflow in the system.
Note
Medium Risk
Large, scheduled bulk SQL writes to core package tables (including maintainer link replacement and
synchronous_commitoff) could affect data correctness or DB load if matching or dump quality regresses; workflow is designed as idempotent with COALESCE and audit.Overview
Adds a cargo-worker that ingates crates.io’s daily PostgreSQL dump into the shared packages database on a Temporal schedule (06:00 UTC,
cargo-registry-sync).The
cargoSyncWorkflowruns load → enrich → audit → cleanup: download/extract the tarball (CARGO_DUMP_URL, defaultstatic.crates.io),COPYdump CSVs into acargo_syncstaging schema viapg-copy-streams(newgetPackagesDbConnection), aggregate/denormalize to staging enrich tables matched bypkg:cargo/PURLs, then bulk-updatepackages,versions,repos/package_repos,maintainers/package_maintainers, anddownloads_daily, with field-levelaudit_field_changesand staging cleanup.Wiring includes
cargo-workerin packages build/deploy, Docker Compose service YAML, pnpm scripts, activity exports, andgetCargoConfig.Reviewed by Cursor Bugbot for commit 140581a. Bugbot is set up for automated code reviews on this repo. Configure here.