Skip to content

feat: add remote repository to handle DockerHub rate limits#1266

Merged
dobrac merged 8 commits into
mainfrom
handle-dockerhub-rate-limits-for-general-usage-eng-2933
Oct 1, 2025
Merged

feat: add remote repository to handle DockerHub rate limits#1266
dobrac merged 8 commits into
mainfrom
handle-dockerhub-rate-limits-for-general-usage-eng-2933

Conversation

@dobrac
Copy link
Copy Markdown
Contributor

@dobrac dobrac commented Sep 29, 2025

Add Docker Hub remote repository proxy to cache Docker images and handle Docker Hub rate limits. https://docs.docker.com/docker-hub/usage/

We can add login to Docker Hub for the remote repository in a following PR.


Note

Adds a Docker Hub remote repository proxy (GCP/AWS/Noop) and wires it into template builds, Terraform modules, and Nomad to cache/pull public images and mitigate rate limits.

  • Infra (Terraform/GCP):
    • Add remote-repository module creating google_artifact_registry_repository in remote mode with IAM and output dockerhub_remote_repository_url.
    • Wire module conditionally via var.remote_repository_enabled; add variable remote_repository_enabled and Makefile REMOTE_REPOSITORY_ENABLED tfvar.
    • Pass dockerhub_remote_repository_url into module.nomad and Nomad job env DOCKERHUB_REMOTE_REPOSITORY_URL.
  • Nomad/Template Manager:
    • Extend job/template to accept and export DOCKERHUB_REMOTE_REPOSITORY_URL.
    • Add dockerhub_remote_repository_url variable and plumb through nomad/main.tf to the template-manager job.
  • Orchestrator:
    • Introduce docker remote repository usage in build flow: plumb dockerhubRepository through builder, base phase, rootfs, and OCI image fetch.
    • oci.GetPublicImage now supports pulling via remote repository proxy when unauthenticated and using default registry.
    • Template server manages closers for clean shutdown.
  • Shared package:
    • New packages/shared/pkg/dockerhub with providers: GCP Artifact Registry, AWS ECR, and Noop; auto-select via env and expose GetImage/Close.
  • Tests:
    • Update OCI tests to use noop repository; add unit tests for tag parsing in dockerhub package.

Written by Cursor Bugbot for commit c7307d4. This will update automatically on new commits. Configure here.

@linear
Copy link
Copy Markdown

linear Bot commented Sep 29, 2025

@dobrac dobrac added the feature New feature label Sep 29, 2025
@dobrac dobrac force-pushed the handle-dockerhub-rate-limits-for-general-usage-eng-2933 branch 3 times, most recently from f9de1df to d8e3e6b Compare September 29, 2025 21:33
@dobrac dobrac changed the title feat: add Docker remote repository to handle DockerHub rate limits feat: add remote repository to handle DockerHub rate limits Sep 29, 2025
@dobrac dobrac marked this pull request as ready for review September 29, 2025 22:07
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

@jakubno jakubno self-assigned this Sep 30, 2025
Copy link
Copy Markdown
Member

@jakubno jakubno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, small nits

Comment thread packages/shared/pkg/dockerhub/repository_gcp.go
Comment thread packages/orchestrator/cmd/build-template/main.go Outdated
Comment thread packages/shared/pkg/dockerhub/repository.go
Comment thread packages/shared/pkg/dockerhub/repository_gcp.go
Comment thread packages/shared/pkg/dockerhub/repository_gcp.go Outdated
Comment thread packages/shared/pkg/dockerhub/repository_aws.go
Comment thread packages/shared/pkg/dockerhub/repository.go Outdated
@dobrac dobrac force-pushed the handle-dockerhub-rate-limits-for-general-usage-eng-2933 branch from 0182d36 to b48444e Compare September 30, 2025 09:53
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

@dobrac dobrac requested a review from jakubno September 30, 2025 10:18
cursor[bot]

This comment was marked as outdated.

@dobrac dobrac force-pushed the handle-dockerhub-rate-limits-for-general-usage-eng-2933 branch from 5d6268f to d113898 Compare October 1, 2025 09:46
@e2b-request-same-site-reviewers e2b-request-same-site-reviewers Bot requested review from jakubno and sitole and removed request for ValentaTomas and jakubno October 1, 2025 09:47
@dobrac dobrac force-pushed the handle-dockerhub-rate-limits-for-general-usage-eng-2933 branch from d113898 to 798d958 Compare October 1, 2025 09:54
@e2b-request-same-site-reviewers e2b-request-same-site-reviewers Bot requested review from jakubno and removed request for jakubno October 1, 2025 09:54
Comment thread packages/orchestrator/internal/template/server/main.go Outdated
@e2b-request-same-site-reviewers e2b-request-same-site-reviewers Bot requested review from jakubno and removed request for jakubno October 1, 2025 10:39
@dobrac dobrac enabled auto-merge (squash) October 1, 2025 10:43
@dobrac dobrac merged commit c07e336 into main Oct 1, 2025
27 checks passed
@dobrac dobrac deleted the handle-dockerhub-rate-limits-for-general-usage-eng-2933 branch October 1, 2025 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants