Improve Subway resilience: re-add LuckyFriday, tune timeouts for endpoint rotation#655
Merged
Conversation
…oint rotation All 4 Acala Foundation endpoints (aca-api.network) resolve to the same 3 IPs, giving Subway zero failover diversity. With client timeout at 90s and server ceiling at 120s, Subway could only attempt ~1.3 endpoints before the server killed the request — making the 4-endpoint list useless. Changes: - Re-add wss://rpc-acala.luckyfriday.io (independent Cloudflare infra) - Lower client request_timeout_seconds 90→60 (still generous for state queries, but allows more rotation attempts) - Raise server request_timeout_seconds 120→240 (allows ≥3 full upstream attempts before the hard ceiling)
Contributor
|
No issues found |
rockbmb
added
the
endpoints
rockbmb
added
the
infrastructure
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
rockbmb
force-pushed
the
fix-acala-subway-resilience
branch
from
aa523c1 to
ac204bf
Compare
Without this, @polkadot/rpc-provider's WsProvider gives up after 90s while subway is still rotating through endpoints (up to 240s). The client disconnects before subway has a chance to find a responsive upstream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
All 4 Acala Foundation endpoints (acala-rpc-{0,1,3}.aca-api.network, acala-rpc.aca-api.network) resolve to the same 3 IPs in AWS eu-west-1. Subway has zero failover diversity: when the cluster stalls, all "different" endpoints fail together. This has caused 9 out of 10 recent update-known-good runs to fail on Acala chain_getBlockHash timeouts.
The previous timeout configuration made it worse: with client=90s and server=120s, Subway could only attempt ~1.3 endpoints before the server ceiling killed the request, making the 4-endpoint list meaningless.
Changes
Re-add
wss://rpc-acala.luckyfriday.io. Independent infrastructure (Cloudflare), provides actual failover when the Acala Foundation cluster is unresponsive. Was removed in Fix Acala endpoints: add numbered Acala Foundation RPCs, remove dead Dwellir and OnFinality #654 because it was 678k blocks behind head; it has since fully re-synced.Lower client
request_timeout_secondsfrom 90 to 60. Still generous for state queries (Subway's default was 30s before Install Subway fromacala/subway:v0.1.1Docker image; set 90s per-upstream timeout #622), but allows more rotation attempts within the server ceiling.Raise server
request_timeout_secondsfrom 120 to 240. Allows 3+ full upstream attempts (3 x 65s = 195s < 240s) before the hard ceiling, giving Subway a real chance to find a responsive endpoint.Timeout math