Implement adaptive request throttling on HTTP 429 (Too Many Requests) responses

[offtherailz]

Description

MapStore currently handles HTTP 429 responses as a common error. The HTTP protocol defines clear semantics for rate limiting MapStore should respect them.

Acceptance criteria

When a resource returns 429, MapStore should:

• Respect the Retry-After header if present (both Retry-After: and Retry-After: forms per RFC 9110)
• Apply exponential backoff when Retry-After is absent, starting from a configurable base delay (e.g. 1s) with a configurable cap (e.g. 60s)
• Queue or defer pending requests to that same origin/resource during the backoff window, rather than dropping or firing them immediately

[tdipisa]

@Binabh please provide here a tech proposal explaining how you have in mind to implement what requested by this issue. Many thanks.

[Binabh]

I propose the following to solve this issue:

1. Creare a helper function to handle throttle/defer logic in case of 429 error.
   1.1. This helper will handle parsing and using of Retry-After header if present or exponential backoff if header is not there.
   1.2. Also this function will have logic to bucket the requests based on passed key like optional msRateLimitKey value or by default bucket based on the origin of the url
2. Use this function in axios interceptors in web/client/libs/ajax.js as this is common axios object used by plugins.
3. Run geoserver and see if ratelimiting is supported with 429 error and Retry-After header. With quick search I could find this plugin used for rate-limiting but I will have to find if this can be used to replicate our specific case. We may have to write a small python wrapper to replicate and use exact scenario. I am thinking of going with wrapper as it gives more flexibility and we will have to test the case where Retry-After header is absent also.
4. Write regression test for this added feature (may need to do with api mocks). Mainly we need to test that backoff logic is being handled appropriately.
5. Run relavent/whole test suite. As we are modifying central axios config. It will be good idea to verify none of the tests break.
6. Manual verification for WMS layer and also make sure other services remain unaffected after the change(i.e. properly handle 429 error but no change in existing behavior)

@tdipisa @offtherailz Please let me know if I am thinking in right direction or if you have any suggestion in the plan itself or any step needs modification or further clarification.
If plan looks good I will move forward with the implementation.

[offtherailz]

Hi @Binabh, the direction is right but there are a few gaps to address. Please adapt your proposal and estimation on this:

Rate limit tests

IDK if control-flow replies correctly. The original problem comes from a source that is not geoserver, AFAIK. @tdipisa can you add more information about the server that has these rate limits to have a full case list of possible scenarios for @Binabh ?

Anyway control-flow is widely used so it can be a good alternative test system.

Critical: most WMS tile requests bypass axios

loadFunction in OL's WMSLayer.js has three paths:

• axios.post — long URL → interceptor works
• axios.get — custom headers → interceptor works
• img.src = newSrc — base case (most common) → interceptor doesn't see it

We should intercept the error after src or if a resource is already in rate-limit, to queue the request somehow, also in the case of src (or switch to axios.get in case of error).

Single tile vs multi-tile (OL)

Same loadFunction feeds both imageLoadFunction and tileLoadFunction: one fix covers both. tileLoadFunction is called before the HTTP request fires, so a pre-check is possible: delay the fetch, tile stays in LOADING state. Tiles already in-flight when 429 arrives will retry independently after backoff.

Cesium

Consider also this case

Bucketing scope

RFC 6585 leaves rate-limit scope to the server: the 429 doesn't communicate whether throttling is per-layer, per-service, or per-origin. Raw URL is wrong for tiles (BBOX/size vary per tile).

Anyway controlflow allow to apply rate-limits per service/layer.
We should be conservative and do by default the bucketing per-origin.
Maybe allow configurable "rules" per server (e.g. in localConfig.json) to allow bucketing - per-path or per layer.

Multi-layer / bucketing

Use a module-level singleton RateLimitManager and axios interceptor.

[Binabh]

@offtherailz Thank You for your feedback and Identifying the gaps I missed.

I mainly see these as added effort from my side:

1. img.src = newSrc — base case (most common)
2. Cesium case : This will need effort on implementation, testing and manual verification
3. Allow configurable "rules" per server (e.g. in localConfig.json) to allow bucketing: This will also need more effort as this will need implementation, testing, verification and documentation

I will increase the estimation to 4D Keeping more room for verification and testing. Let me know if I am over/under estimating. I will adjust accordingly.

I will work with the information we currently have and adjust if we get info about the server that has these rate limits to have a full case list of possible scenarios from @tdipisa

Shall I start implementation incorporating feedback from @offtherailz if plan looks good?