Support configurable retry rules per provider via retryRules

Author: ericdallo

AGENTS.md

@@ -30,4 +30,4 @@ ECA Agent Guide (AGENTS.md)
 - Use java class typing to avoid GraalVM reflection issues
 - Avoid adding too many comments, only add essential or when you think is really important to mention something. 
 - ECA's protocol specification of client <-> server lives in docs/protocol.md
-- When adding support to a new feature or fixing a existing github issue, add a entry to Unreleased in CHANGELOG.md if not already there.
+- When adding support to a new feature or fixing a existing github issue, add a entry to Unreleased in CHANGELOG.md if not already there, be concise like the rest.

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 - Fix MCP server initialization crash (`String cannot be cast to IPersistentCollection`) when OAuth metadata endpoint returns a non-JSON or error response.
 - Auto-approve `eca__read_file` for tool call output cache files (`~/.cache/eca/toolCallOutputs`).
+- Support configurable retry rules per provider via `retryRules`, allowing users to define custom HTTP status and/or body pattern matching for automatic retries with optional labels.
 
 ## 0.110.3
 

docs/config.json

@@ -424,6 +424,36 @@
             "okhttp"
           ]
         },
+        "retryRules": {
+          "type": "array",
+          "description": "Custom retry rules. Each rule can match by HTTP status code and/or response body regex pattern. When matched, the request is retried with exponential backoff.",
+          "markdownDescription": "Custom retry rules. Each rule can match by HTTP status code and/or response body regex pattern. When matched, the request is retried with exponential backoff.",
+          "items": {
+            "type": "object",
+            "properties": {
+              "status": {
+                "type": "integer",
+                "description": "HTTP status code to match.",
+                "markdownDescription": "HTTP status code to match."
+              },
+              "bodyPattern": {
+                "type": "string",
+                "description": "Regex pattern to match against the response body (case-insensitive).",
+                "markdownDescription": "Regex pattern to match against the response body (case-insensitive)."
+              },
+              "label": {
+                "type": "string",
+                "description": "Human-readable label shown in the retry progress message (e.g. 'Rate limited').",
+                "markdownDescription": "Human-readable label shown in the retry progress message (e.g. 'Rate limited')."
+              }
+            },
+            "anyOf": [
+              { "required": ["status"] },
+              { "required": ["bodyPattern"] }
+            ],
+            "additionalProperties": false
+          }
+        },
         "models": {
           "type": "object",
           "description": "Available models for this provider. Each key is the model alias.",

docs/config/models.md

@@ -78,6 +78,7 @@ Schema:
 | `thinkTagStart`                   | string  | Optional override the think start tag tag for openai-chat (Default: "<think>") api                           | No       |
 | `thinkTagEnd`                     | string  | Optional override the think end tag for openai-chat (Default: "</think>") api                                | No       |
 | `httpClient`                      | map     | Allow customize the http-client for this provider requests, like changing http version                       | No       |
+| `retryRules`                      | array   | Custom retry rules that match by HTTP status and/or body regex pattern (see [Retry Rules](#retry-rules))    | No       |
 | `models`                          | map     | Key: model name, value: its config                                                                           | Yes      |
 | `models <model> extraPayload`     | map     | Extra payload sent in body to LLM                                                                            | No       |
 | `models <model> extraHeaders`     | map     | Extra headers sent to LLM request                                                                            | No       |
@@ -230,6 +231,42 @@ Notes:
 - Authentication priority (short): `key` (with dynamic string pase support) > OAuth.
 - All providers with API key auth can use credential files.
 
+### Retry Rules
+
+ECA automatically retries requests on common transient errors (429, 500, 502, 503, 529) with exponential backoff. You can define custom retry rules per provider using `retryRules` to handle additional status codes or response body patterns.
+
+Each rule can match by:
+
+- **`status`** (integer): HTTP status code to match
+- **`bodyPattern`** (string): Regex pattern to match against the response body (case-insensitive)
+- **`label`** (string, optional): Human-readable text shown in the retry progress message
+
+At least one of `status` or `bodyPattern` is required. When both are specified, both must match. Custom rules are checked before built-in classification, so they can override default behavior.
+
+```javascript title="~/.config/eca/config.json"
+{
+  "providers": {
+    "my-company": {
+      "api": "openai-chat",
+      "url": "${env:MY_COMPANY_API_URL}",
+      "key": "${env:MY_COMPANY_API_KEY}",
+      "retryRules": [
+        {"status": 418, "label": "Corporate proxy throttle"},
+        {"bodyPattern": "capacity.*exceeded", "label": "Capacity exceeded"},
+        {"status": 503, "bodyPattern": "maintenance", "label": "Under maintenance"}
+      ],
+      "models": {
+        "gpt-5": {}
+      }
+    }
+  }
+}
+```
+
+When a retry rule matches, the chat shows a progress message like:
+
+> ⏳ Corporate proxy throttle. Retrying in 2s (attempt 1/10)
+
 ## Providers examples
 
 === "Anthropic"

src/eca/features/chat.clj

@@ -1299,12 +1299,13 @@
                 :variant (:variant chat-ctx)
                 :subagent? (some? (get-in @db* [:chats chat-id :subagent]))
                 :cancelled? (fn [] (identical? :stopping (get-in @db* [:chats chat-id :status])))
-                :on-retry (fn [{:keys [attempt max-retries delay-ms error-data]}]
-                            (let [{error-type :error/type} (llm-providers.errors/classify-error error-data)
-                                  reason (case error-type
-                                           :rate-limited "Rate limited"
-                                           :overloaded "Provider overloaded"
-                                           "Transient error")]
+                :on-retry (fn [{:keys [attempt max-retries delay-ms classified]}]
+                            (let [{error-type :error/type error-label :error/label} classified
+                                  reason (or error-label
+                                             (case error-type
+                                               :rate-limited "Rate limited"
+                                               :overloaded "Provider overloaded"
+                                               "Transient error"))]
                               (send-content! chat-ctx :system
                                              {:type :progress
                                               :state :running

src/eca/llm_api.clj

@@ -371,9 +371,12 @@
                            (when-not (:silent? (ex-data exception))
                              (logger/error args)
                              (on-error args)))
+        provider-config (get-in config [:providers provider])
+        retry-rules (:retryRules provider-config)
         maybe-retry (fn [error-data attempt on-give-up retry-prompt-fn]
-                      (let [{error-type :error/type} (llm-providers.errors/classify-error error-data)]
-                        (if (and (contains? #{:rate-limited :overloaded} error-type)
+                      (let [{error-type :error/type
+                             :as classified} (llm-providers.errors/classify-error error-data retry-rules)]
+                        (if (and (contains? #{:rate-limited :overloaded :retryable-custom} error-type)
                                  (< attempt default-max-retries)
                                  (not (cancelled?)))
                           (let [delay-ms (retry-delay-ms attempt)]
@@ -387,14 +390,14 @@
                                 (on-retry {:attempt (inc attempt)
                                            :max-retries default-max-retries
                                            :delay-ms delay-ms
-                                           :error-data error-data})
+                                           :error-data error-data
+                                           :classified classified})
                                 (catch Exception e
                                   (logger/warn logger-tag "on-retry callback failed" {:exception e}))))
                             (if (sleep-with-cancel delay-ms cancelled?)
                               (retry-prompt-fn (inc attempt))
                               (on-give-up error-data)))
                           (on-give-up error-data))))
-        provider-config (get-in config [:providers provider])
         model-config (get-in provider-config [:models model])
         model-config (update model-config :variants #(config/effective-model-variants config provider model %))
         api-handler (provider->api-handler provider model config)

src/eca/llm_providers/errors.clj

@@ -68,35 +68,60 @@
 
       :else nil)))
 
+(defn ^:private classify-by-custom-rules
+  "Checks user-configured retry rules. Each rule may have :status (int),
+   :body-pattern (regex string, case-insensitive), and :label (string).
+   Returns {:error/type :retryable-custom :error/label label} on first match, nil otherwise."
+  [{:keys [status body]} retry-rules]
+  (when (seq retry-rules)
+    (some (fn [{rule-status :status rule-body-pattern :bodyPattern rule-label :label}]
+            (let [status-matches? (if rule-status
+                                    (= rule-status status)
+                                    true)
+                  body-matches? (if rule-body-pattern
+                                  (when (string? body)
+                                    (re-find (re-pattern (str "(?i)" rule-body-pattern)) body))
+                                  true)
+                  has-condition? (or rule-status rule-body-pattern)]
+              (when (and has-condition? status-matches? body-matches?)
+                (cond-> {:error/type :retryable-custom}
+                  rule-label (assoc :error/label rule-label)))))
+          retry-rules)))
+
 (defn classify-error
   "Classifies an error map into a semantic error type.
 
    Accepts the standard on-error map shape: {:message :status :body :exception}.
+   Optional `retry-rules` seq of user-configured rules checked before built-in classification.
    Returns a map with :error/type — one of:
+     :retryable-custom  — matched a user-configured retry rule (with optional :error/label)
      :context-overflow  — prompt exceeds model context window
      :rate-limited      — 429 or rate limit pattern in body/message
      :overloaded        — provider overloaded (503, 529, etc.)
      :auth              — authentication/authorization failure (401, 403)
      :unknown           — unclassified error"
-  [{:keys [status exception] :as error-data}]
-  (or (when status
-        (classify-by-status-and-body error-data))
-      (classify-by-message error-data)
-      (when exception
-        (classify-by-message {:message (ex-message exception)}))
-      {:error/type :unknown}))
+  ([error-data] (classify-error error-data nil))
+  ([{:keys [status exception] :as error-data} retry-rules]
+   (or (classify-by-custom-rules error-data retry-rules)
+       (when status
+         (classify-by-status-and-body error-data))
+       (classify-by-message error-data)
+       (when exception
+         (classify-by-message {:message (ex-message exception)}))
+       {:error/type :unknown})))
 
 (defn context-overflow?
   "Returns true if the error is a context window overflow."
   [error-data]
   (= :context-overflow (:error/type (classify-error error-data))))
 
 (def ^:private retryable-error-types
-  #{:rate-limited :overloaded})
+  #{:rate-limited :overloaded :retryable-custom})
 
 (defn retryable?
   "Returns true if the error is transient and the request can be retried
-   (rate-limited or provider overloaded)."
-  [error-data]
-  (contains? retryable-error-types
-             (:error/type (classify-error error-data))))
+   (rate-limited, provider overloaded, or matched a custom retry rule)."
+  ([error-data] (retryable? error-data nil))
+  ([error-data retry-rules]
+   (contains? retryable-error-types
+              (:error/type (classify-error error-data retry-rules)))))

test/eca/llm_api_test.clj

@@ -338,3 +338,87 @@
            :on-message-received identity})))
       (is (= 1 @attempt*))
       (is (true? @on-error-called*)))))
+
+(deftest sync-retry-on-custom-retry-rule-test
+  (testing "retries when custom retryRules status matches"
+    (let [attempt* (atom 0)
+          retry-events* (atom [])
+          on-error-called* (atom false)]
+      (with-redefs [eca.llm-api/prompt! (fn [_opts]
+                                          (let [attempt (swap! attempt* inc)]
+                                            (if (= 1 attempt)
+                                              {:error {:status 418
+                                                       :body "I'm a teapot"
+                                                       :message "LLM response status: 418"}}
+                                              {:output-text "success"
+                                               :usage {:input-tokens 10 :output-tokens 5}})))
+                    eca.llm-api/sleep-with-cancel (fn [_ cancelled?] (not (cancelled?)))]
+        (llm-api/sync-or-async-prompt!
+         (make-prompt-opts
+          {:stream false
+           :config {:providers {"anthropic" {:key "test-key"
+                                             :url "http://test"
+                                             :retryRules [{:status 418 :label "Proxy throttle"}]
+                                             :models {"claude-sonnet-4-6" {:extraPayload {:stream false}}}}}}
+           :on-retry (fn [event] (swap! retry-events* conj event))
+           :on-error (fn [_] (reset! on-error-called* true))
+           :on-message-received identity})))
+      (is (= 2 @attempt*))
+      (is (= 1 (count @retry-events*)))
+      (is (= :retryable-custom (get-in (first @retry-events*) [:classified :error/type])))
+      (is (= "Proxy throttle" (get-in (first @retry-events*) [:classified :error/label])))
+      (is (false? @on-error-called*)))))
+
+(deftest async-retry-on-custom-retry-rule-body-pattern-test
+  (testing "retries async when custom retryRules bodyPattern matches"
+    (let [attempt* (atom 0)
+          retry-events* (atom [])
+          received-text* (atom "")
+          on-error-called* (atom false)]
+      (with-redefs [eca.llm-api/prompt! (fn [{:keys [on-message-received on-error]}]
+                                          (let [attempt (swap! attempt* inc)]
+                                            (if (= 1 attempt)
+                                              (on-error {:status 500
+                                                         :body "server capacity exceeded"
+                                                         :message "LLM response status: 500"})
+                                              (do
+                                                (on-message-received {:type :text :text "hello"})
+                                                (on-message-received {:type :finish :finish-reason "stop"})))))
+                    eca.llm-api/sleep-with-cancel (fn [_ cancelled?] (not (cancelled?)))]
+        (llm-api/sync-or-async-prompt!
+         (make-prompt-opts
+          {:config {:providers {"anthropic" {:key "test-key"
+                                             :url "http://test"
+                                             :retryRules [{:bodyPattern "capacity.*exceeded"
+                                                           :label "Capacity exceeded"}]
+                                             :models {"claude-sonnet-4-6" {}}}}}
+           :on-retry (fn [event] (swap! retry-events* conj event))
+           :on-error (fn [_] (reset! on-error-called* true))
+           :on-message-received (fn [{:keys [type text]}]
+                                  (when (= :text type)
+                                    (swap! received-text* str text)))})))
+      (is (= 2 @attempt*))
+      (is (= 1 (count @retry-events*)))
+      (is (= "Capacity exceeded" (get-in (first @retry-events*) [:classified :error/label])))
+      (is (false? @on-error-called*))
+      (is (= "hello" @received-text*))))
+
+  (testing "does not retry when no custom rule matches"
+    (let [attempt* (atom 0)
+          on-error-called* (atom false)]
+      (with-redefs [eca.llm-api/prompt! (fn [{:keys [on-error]}]
+                                          (swap! attempt* inc)
+                                          (on-error {:status 418
+                                                     :body "I'm a teapot"
+                                                     :message "LLM response status: 418"}))
+                    eca.llm-api/sleep-with-cancel (fn [_ _] true)]
+        (llm-api/sync-or-async-prompt!
+         (make-prompt-opts
+          {:config {:providers {"anthropic" {:key "test-key"
+                                             :url "http://test"
+                                             :retryRules [{:status 599 :label "Something else"}]
+                                             :models {"claude-sonnet-4-6" {}}}}}
+           :on-error (fn [_] (reset! on-error-called* true))
+           :on-message-received identity})))
+      (is (= 1 @attempt*))
+      (is (true? @on-error-called*)))))