fix(ci): authenticate the nightly dependency-bump API calls#11042
Merged
Conversation
The "Bump Backend dependencies" workflow has failed every night for the last two weeks. #11012 fixed one cause (repos renamed under localai-org); what is left is rate limiting. bump_deps.sh fans out to ~25 parallel matrix jobs that each query api.github.com anonymously. Anonymous calls are capped at 60/hour per source IP and GitHub-hosted runners egress through shared NAT addresses, so a random handful of jobs draw HTTP 403 and die at curl exit 22 with an empty response. Last night that hit ggml-org/whisper.cpp and mudler/depth-anything.cpp -- both public and resolvable, nothing wrong with either pin. Route every bump script through a shared gh_curl helper that sends GITHUB_TOKEN when present (1000/hour instead of 60) and retries transient failures, including the 403s that plain --retry ignores. The helper suppresses xtrace around the call so the Authorization header cannot land in a public job log. bump_docs.sh had a sharper version of the same bug: it piped an unchecked response into `jq -r .tag_name`, so a throttled request resolved to the string "null" and would have been published as the docs version. It now refuses to write anything it cannot resolve to a tag. Verified locally by running all four scripts end to end against their real upstreams: correct SHAs/tags written, exit 0; a nonexistent repo now fails with a named diagnostic instead of a bare exit 22 and leaves the pinned file untouched; the token is absent from the xtrace output; and the scripts still work unauthenticated. Assisted-by: Claude:opus-4.8 [Claude Code]
Contributor
|
Collaborator
Author
|
@mudler CI triage: every change-specific check is green. The Yamllint failure is inherited from current master: this PR head 7402cba has parent 5c96e09 (current master) and changes no gallery files; the errors are all in gallery/index.yaml already present in that parent, so a rerun cannot fix it. DCO is also ACTION_REQUIRED because the commit has no human Signed-off-by; repo policy forbids the bot from adding or forging that attestation. Please sign off/amend the commit when you review it. I have not mixed the unrelated gallery repair into this CI-auth PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Bump Backend dependencies workflow has been failing every single night for the last 12+ days. Master push CI and the v4.7.1 release jobs are green — this scheduled workflow is the only red.
#11012 fixed one cause yesterday (repos renamed under
localai-org). What remains is rate limiting.Why it fails
bump_deps.shfans out to ~25 parallel matrix jobs, each queryingapi.github.comanonymously. Anonymous API calls are capped at 60 requests/hour per source IP, and GitHub-hosted runners egress through shared NAT addresses — so a random handful of jobs draw HTTP 403,curl -fexits 22 with an empty response, and the job dies.Last night's two casualties were
ggml-org/whisper.cppandmudler/depth-anything.cpp. Both are public and resolve fine:Nothing is wrong with either pin — it is pure rate-limit noise. Confirming the anonymous ceiling:
What this changes
.github/gh_curl.sh: a shared curl wrapper used by all four bump scripts. It sendsGITHUB_TOKENwhen present (1000/hour instead of 60) and retries transient failures with--retry-all-errors, which covers the 403 rate-limit responses that plain--retryskips.xtracearound the call, so theAuthorizationheader can never land in a public job log (these scripts run underset -x).GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}wired into all four bump steps. Read-only use of the default token — no new secret.bump_deps.shnow reports a named diagnostic instead of aborting at a bareexit 22.Drive-by:
bump_docs.shsilent corruptionIt piped an unchecked response straight into
jq -r '.tag_name'. On a throttled request that resolves to the string"null", which would have been published as the docs version:It now refuses to write a version it could not resolve.
Verification
All four scripts run end to end against their real upstreams:
bump_deps.sh→ggml-org/whisper.cpp080bbbe8…bump_deps.sh→mudler/depth-anything.cppf4e17dea…bump_vllm_wheel.shvllm==0.25.1+ matching wheel URLbump_vllm_metal.shv0.3.0.dev20260721061415(matches #11034)bump_docs.shv4.7.1bump_docs.shon bad repoversion.jsonnot overwritten withnullAuthorizationstring in the traceGITHUB_TOKENsetBoth workflow YAMLs re-parsed and each bump step confirmed to carry the env var.
bash -nclean on all five scripts.Notes
.agents/ai-coding-assistants.mdan AI must not add aSigned-off-by. @mudler please sign off if you take this.Assisted-by: Claude:opus-4.8 [Claude Code]