feat: go packages worker [CM-1275]

[mbani01]

This pull request adds support for processing Go packages in the packages_worker service. It introduces new batch enrichment workflows for Go package versions and status, integrates these into the existing Temporal-based system, and provides all necessary configuration, scripts, and Docker Compose setup to run and develop the new Go worker.

Go package enrichment support:

• Added new batch activities, workflows, and scheduling for Go package version and status enrichment, including database updates and audit logging. (src/go/activities.ts [1] src/go/workflows.ts [2] src/go/schedule.ts [3]
• Implemented Go module proxy and pkg.go.dev clients to fetch latest versions and status, handling rate limits, errors, and module path encoding. (src/go/proxyClient.ts [1] src/go/pkgGoDevClient.ts [2] src/go/types.ts [3]
• Exposed new Go enrichment activities in the main activities export. (src/activities.ts services/apps/packages_worker/src/activities.tsR27)

Configuration and orchestration:

• Added Go worker configuration options and environment variable support. (src/config.ts services/apps/packages_worker/src/config.tsR66-R72)
• Created Docker Compose service definition for the Go worker, including development setup with volume mounts for live code reloading. (scripts/services/go-worker.yaml scripts/services/go-worker.yamlR1-R66)
• Registered the Go worker in the list of package services for Docker builds. (scripts/builders/packages.env scripts/builders/packages.envL4-R4)

Scripts and commands:

• Added npm scripts for triggering Go enrichment tasks and for starting/dev'ing the Go worker, including local variants. (package.json [1] [2]

These changes collectively enable the system to keep Go package metadata up-to-date, similar to existing support for other ecosystems, and provide an integrated development and deployment workflow for the new Go worker.

---

Note

Medium Risk
Writes package metadata in the shared packages DB and depends on external APIs/rate limits; failures are mostly per-package skips, but bad updates could affect downstream package data.

Overview
Adds a go-worker Temporal service that keeps Go package rows in sync with proxy.golang.org (latest version, release time, repo URL) and pkg.go.dev (deprecated/active status, versions count).

Two cursor-based batch workflows (enrichGoVersions, enrichGoStatus) scan ecosystem = 'go' packages, update the packages table, and record audit field changes. Daily cron schedules register at worker startup (02:00 versions, 05:00 status). A trigger-go script can start the same workflows manually.

Deployment/dev: go-worker is added to package Docker builds, Compose (go-worker / go-worker-dev), and package.json start/dev/trigger scripts. Config adds getGoConfig() (GO_FETCH_TIMEOUT_MS, GO_PROXY_CONCURRENCY); pkg.go.dev client throttles to ~40 QPS with 429 retries.

^{Reviewed by Cursor Bugbot for commit 0946688. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

Adds a new Go packages enrichment worker to services/apps/packages_worker, integrating it into the existing Temporal scheduling + activity/workflow structure so Go package metadata can be periodically refreshed from external Go services.

Changes:

• Introduces Go-specific Temporal workflows/activities to batch-enrich Go package versions (proxy.golang.org) and module status (pkg.go.dev).
• Wires up scheduling + a new go-worker entrypoint to register cron schedules and run the worker.
• Adds local/dev tooling (pnpm scripts + Docker Compose service + build service list entry) to run and trigger the Go enrichments.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
services/apps/packages_worker/src/workflows/index.ts	Exposes Go workflows via the central workflows export.
services/apps/packages_worker/src/scripts/triggerGoEnrich.ts	Adds a CLI script to manually trigger Go enrichment workflows.
services/apps/packages_worker/src/go/workflows.ts	Adds cursor-based scan workflows with continueAsNew for long runs.
services/apps/packages_worker/src/go/types.ts	Defines Go client result types and a FetchError shape.
services/apps/packages_worker/src/go/schedule.ts	Registers cron schedules for versions/status enrichment.
services/apps/packages_worker/src/go/proxyClient.ts	Implements proxy.golang.org @latest client + module path escaping.
services/apps/packages_worker/src/go/pkgGoDevClient.ts	Implements pkg.go.dev versions/status client with throttling + 429 backoff.
services/apps/packages_worker/src/go/activities.ts	Implements Temporal activities to batch-scan Go packages and update packages with audit logging.
services/apps/packages_worker/src/config.ts	Adds Go worker config (timeout + proxy concurrency).
services/apps/packages_worker/src/bin/go-worker.ts	Adds the Go worker entrypoint that initializes service + schedules + worker start.
services/apps/packages_worker/src/activities.ts	Re-exports Go activities from the main activities module.
services/apps/packages_worker/package.json	Adds pnpm scripts for starting/dev’ing and triggering the Go worker/enrichments.
scripts/services/go-worker.yaml	Adds Docker Compose definitions for go-worker and go-worker-dev.
scripts/builders/packages.env	Registers go-worker in the packages image build service list.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

Cursor mismatches batch ordering

High Severity

getGoBatch orders rows by last_synced_at then purl, but pagination only applies purl > $(after) and the workflow cursor is the last row’s purl. After a batch, every package with a smaller purl that was not in that batch is excluded for the rest of the run, so the scan can finish early and leave Go packages unenriched until the next scheduled workflow.

Additional Locations (1)

• services/apps/packages_worker/src/go/activities.ts#L91-L92

 

^{Reviewed by Cursor Bugbot for commit 8f85149. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0946688. Configure here.}

[cursor]

Transient fetch errors swallowed

Medium Severity

Version and status enrichment treat every FetchError, including RATE_LIMIT and TRANSIENT, as a warn-and-skip. Unlike the npm worker, those cases never throw, so Temporal does not retry the activity and the batch cursor still advances past packages that were not updated.

Additional Locations (1)

• services/apps/packages_worker/src/go/activities.ts#L107-L113

 

^{Reviewed by Cursor Bugbot for commit 0946688. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.