Excavator - Handle GitHub API rate limiting

[o-l-a-v]

Seems Excavator does not handle GitHub API rate limiting very well. Here is a run from Extras bucket:

https://github.com/ScoopInstaller/Extras/actions/runs/18555159620/job/52890740764

Autoupdating appflowy
The remote server returned an error: (403) rate limit exceeded.
URL https://api.github.com/repos/AppFlowy-IO/AppFlowy/releases is not valid

[o-l-a-v]

Or maybe it does, as it managed to update it anyway.

Could not find hash in https://api.github.com/repos/AppFlowy-IO/AppFlowy/releases
Downloading AppFlowy-0.10.1-windows-x86_64.zip to compute hashes!
VERBOSE: Requested HTTP/1.1 GET with 0-byte payload
VERBOSE: Received HTTP/1.1 response of content type application/json of unknown size
VERBOSE: Content encoding: utf-8
Downloading https://github.com/AppFlowy-IO/AppFlowy/releases/download/0.10.1/AppFlowy-0.10.1-windows-x86_64.zip (59.8 MB)...
Computed hash: 82b98f7943b2548b5b2e507ac6ff7b093801d842bbdf511ac1d1ee44ea5a6d44
Writing updated appflowy manifest

[z-Fng]

We still need to handle GitHub API rate limiting isuue.

as it managed to update it anyway.

When the hash extraction fails due to API rate limits, it will fall back to downloading.
However, if it happens in version detection (checkver), it will fail directly.

Possible solution:

• Use multiple tokens.
• Obtain a higher rate-limit GitHub token? currently: 5,000 requests per hour, 100 concurrent requests
• Split the Main/Extras bucket into multiple buckets.

[z-Fng]

The solution I suggested earlier was going in the wrong direction.

From my tests, most issues were caused by retrieving hashes from GitHub API without token and hitting concurrent request limits (100 concurrent requests), rather than the hourly request limit.

The missing token issue will be fixed in ScoopInstaller/Scoop#6535. After it's merged, I'll retest to see if we need to adjust the concurrency limits.

[z-Fng]

The missing token issue will be fixed in ScoopInstaller/Scoop#6535. After it's merged, I'll retest to see if we need to adjust the concurrency limits.

From my tests, even with ScoopInstaller/Scoop#6535, our Extras bucket still hits GitHub API rate limits constantly. We may need to implemente concurrency control next.

Make too many concurrent requests. No more than 100 concurrent requests are allowed. This limit is shared across the REST API and GraphQL API.

[o-l-a-v]

One idea: We could add fallback to PAT (personal access token) if GITHUB_TOKEN has reached its rate limit + PAT is present in the bucket repo?

From my tests, most issues were caused by retrieving hashes from GitHub API without token and hitting concurrent request limits (100 concurrent requests), rather than the hourly request limit.

Does checkver hit the GitHub API once to get latest version, and then later autoupdate hit the GitHub API for each hash it tries to find, vs. getting releases/latest ONCE and reuse that JSON for the next steps?

Could save some hits if caching the response from checkver.