[TEP-953] Tier / Priority-based model/provider routing with fallback retry | 基于优先级的路由负载

[sfu2]

Version

v0.26

System Information

Ubuntu 22.04

Category

Other

Scenario

Keywords: Model/provider routing, load balancing, and failover

Tingly-Box is currently using round-robin routing to switch between models and providers. I would like to request a priority-based routing feature similar to the behavior in Claude Code Hub.

Each model/provider should be configurable with a priority. When handling a request, Tingly-Box should always choose the available model/provider with the highest priority. If the highest-priority option is unavailable, the system should automatically fall back to the next highest-priority available option.

After a configurable timeout, for example X minutes, Tingly-Box should check whether a higher-priority model/provider has become available again. If so, routing should switch back to the higher-priority option.

In short, the gateway should always prefer the best available model/provider, fall back when necessary, and periodically retry higher-priority options.

Proposed Solution

No response

[FFengIll]

Design

Part 1

提供基于优先级的策略，支持按优先级分组，高优先级先使用，同优先级round robin。

#952

Part 2

提供尽可能服务策略

1. Pre-Request：请求发出失败，立刻切换其他重试
2. Pre-Stream：对stream，Firtst token 失败，客户端没有收到消息，服务端可自行重试（切换其他可用）
3. Mid-Stream：对stream，中间过程失败，无法自然重试，但考虑做有限buffer，在特定范围的buffer接受时间内，如果error则自行重试
4. Let it Crash：无法兜底，立即放弃
5. Mark Unhealth：对于已经失败的provider，标记非健康（一段时间），回避使用以使其恢复

#959
#1069

Part 3

分组策略简化为 Tier，分层，Tier 0 为最高优先。
高优先先选中，除非不可用，才会落入低优先。
由此支持 用量感知， 错误感知， 失败兜底等多类场景。

层间多级兜底，层内动态路由，尽可能服务策略感知错误，即时切换。

#1096