improvements to retry feature

Author: ericdallo

CHANGELOG.md

@@ -4,6 +4,9 @@
 
 - Fix MCP OAuth credentials cache not invalidating when the server URL changes.
 - Add `isSubagent` condition variable for chat system instructions
+- **Breaking:** Replace `bodyPattern` with `errorPattern` in `retryRules`, which matches against any error text (response body, error message, or exception message).
+  - Retry on "Remote host terminated the handshake" TLS errors.
+  - Fix empty "Error: " message on connection failures (e.g. DNS resolution, connection refused) and retry them as transient errors.
 
 ## 0.112.0
 

docs/config.json

@@ -477,8 +477,8 @@
         },
         "retryRules": {
           "type": "array",
-          "description": "Custom retry rules. Each rule can match by HTTP status code and/or response body regex pattern. When matched, the request is retried with exponential backoff.",
-          "markdownDescription": "Custom retry rules. Each rule can match by HTTP status code and/or response body regex pattern. When matched, the request is retried with exponential backoff.",
+          "description": "Custom retry rules. Each rule can match by HTTP status code and/or error text regex pattern. When matched, the request is retried with exponential backoff.",
+          "markdownDescription": "Custom retry rules. Each rule can match by HTTP status code and/or error text regex pattern. When matched, the request is retried with exponential backoff.",
           "items": {
             "type": "object",
             "properties": {
@@ -487,10 +487,10 @@
                 "description": "HTTP status code to match.",
                 "markdownDescription": "HTTP status code to match."
               },
-              "bodyPattern": {
+              "errorPattern": {
                 "type": "string",
-                "description": "Regex pattern to match against the response body (case-insensitive).",
-                "markdownDescription": "Regex pattern to match against the response body (case-insensitive)."
+                "description": "Regex pattern to match against any error text — response body, error message, or exception message (case-insensitive). Useful for connection-level errors that have no HTTP status.",
+                "markdownDescription": "Regex pattern to match against any error text — response body, error message, or exception message (case-insensitive). Useful for connection-level errors that have no HTTP status."
               },
               "label": {
                 "type": "string",
@@ -500,7 +500,7 @@
             },
             "anyOf": [
               { "required": ["status"] },
-              { "required": ["bodyPattern"] }
+              { "required": ["errorPattern"] }
             ],
             "additionalProperties": false
           }

docs/config/models.md

@@ -78,7 +78,7 @@ Schema:
 | `thinkTagStart`                   | string  | Optional override the think start tag tag for openai-chat (Default: "<think>") api                           | No       |
 | `thinkTagEnd`                     | string  | Optional override the think end tag for openai-chat (Default: "</think>") api                                | No       |
 | `httpClient`                      | map     | Allow customize the http-client for this provider requests, like changing http version                       | No       |
-| `retryRules`                      | array   | Custom retry rules that match by HTTP status and/or body regex pattern (see [Retry Rules](#retry-rules))    | No       |
+| `retryRules`                      | array   | Custom retry rules that match by HTTP status and/or error pattern (see [Retry Rules](#retry-rules))    | No       |
 | `models`                          | map     | Key: model name, value: its config                                                                           | Yes      |
 | `models <model> extraPayload`     | map     | Extra payload sent in body to LLM                                                                            | No       |
 | `models <model> extraHeaders`     | map     | Extra headers sent to LLM request                                                                            | No       |
@@ -233,15 +233,15 @@ Notes:
 
 ### Retry Rules
 
-ECA automatically retries requests on common transient errors (429, 500, 502, 503, 529) with exponential backoff. You can define custom retry rules per provider using `retryRules` to handle additional status codes or response body patterns.
+ECA automatically retries requests on common transient errors (429, 500, 502, 503, 529) with exponential backoff. You can define custom retry rules per provider using `retryRules` to handle additional status codes or error patterns.
 
 Each rule can match by:
 
 - **`status`** (integer): HTTP status code to match
-- **`bodyPattern`** (string): Regex pattern to match against the response body (case-insensitive)
+- **`errorPattern`** (string): Regex pattern to match against any error text — response body, error message, or exception message (case-insensitive). Useful for both HTTP response errors and connection-level errors (e.g. TLS handshake failures)
 - **`label`** (string, optional): Human-readable text shown in the retry progress message
 
-At least one of `status` or `bodyPattern` is required. When both are specified, both must match. Custom rules are checked before built-in classification, so they can override default behavior.
+At least one of `status` or `errorPattern` is required. When both are specified, both must match. Custom rules are checked before built-in classification, so they can override default behavior.
 
 ```javascript title="~/.config/eca/config.json"
 {
@@ -252,8 +252,9 @@ At least one of `status` or `bodyPattern` is required. When both are specified,
       "key": "${env:MY_COMPANY_API_KEY}",
       "retryRules": [
         {"status": 418, "label": "Corporate proxy throttle"},
-        {"bodyPattern": "capacity.*exceeded", "label": "Capacity exceeded"},
-        {"status": 503, "bodyPattern": "maintenance", "label": "Under maintenance"}
+        {"errorPattern": "capacity.*exceeded", "label": "Capacity exceeded"},
+        {"status": 503, "errorPattern": "maintenance", "label": "Under maintenance"},
+        {"errorPattern": "terminated.*handshake", "label": "TLS handshake failed"}
       ],
       "models": {
         "gpt-5": {}

src/eca/features/chat.clj

@@ -1473,11 +1473,11 @@
                                 (do
                                   (when compacting?
                                     (swap! db* update-in [:chats chat-id] dissoc :auto-compacting? :compacting?))
-                                  (send-content! chat-ctx :system {:type :text :text (or message (str "Error: " (ex-message exception)))})
+                                  (send-content! chat-ctx :system {:type :text :text (or message (str "Error: " (or (ex-message exception) (.getName (class exception)))))})
                                   (finish-chat-prompt! :idle (dissoc chat-ctx :on-finished-side-effect))))))})
               (catch Exception e
                 (logger/error e)
-                (send-content! chat-ctx :system {:type :text :text (str "Error: " (ex-message e))})
+                (send-content! chat-ctx :system {:type :text :text (str "Error: " (or (ex-message e) (.getName (class e))))})
                 (finish-chat-prompt! :idle (dissoc chat-ctx :on-finished-side-effect))))))))))
 
 (defn ^:private send-mcp-prompt!

src/eca/llm_providers/anthropic.clj

@@ -88,14 +88,14 @@
                      (llm-util/log-response logger-tag rid "response-error" body)
                      (reset! response* error-data)))]
     (llm-util/log-request logger-tag rid url body headers)
-    (let [{:keys [status body]} (http/post
-                                 url
-                                 {:headers headers
-                                  :body (json/generate-string body)
-                                  :throw-exceptions? false
-                                  :http-client (client/merge-with-global-http-client http-client)
-                                  :as (if on-stream :stream :json)})]
-      (try
+    (try
+      (let [{:keys [status body]} (http/post
+                                   url
+                                   {:headers headers
+                                    :body (json/generate-string body)
+                                    :throw-exceptions? false
+                                    :http-client (client/merge-with-global-http-client http-client)
+                                    :as (if on-stream :stream :json)})]
         (if (not= 200 status)
           (let [body-str (if on-stream (slurp body) body)]
             (logger/warn logger-tag "Unexpected response status: %s body: %s" status body-str)
@@ -110,9 +110,10 @@
             (do
               (llm-util/log-response logger-tag rid "response" body)
               (reset! response*
-                      {:output-text (:text (last (:content body)))}))))
-        (catch Exception e
-          (on-error {:exception e}))))
+                      {:output-text (:text (last (:content body)))})))))
+      (catch Exception e
+        (on-error {:exception e
+                   :message (format "Connection error: %s" (or (ex-message e) (.getName (class e))))})))
     @response*))
 
 (defn ^:private normalize-messages [past-messages supports-image?]

src/eca/llm_providers/errors.clj

@@ -25,6 +25,14 @@
    #"(?i)throttl"
    #"(?i)overloaded_error"])
 
+(def ^:private overloaded-patterns
+  "Regex patterns matching transient connection/infrastructure errors across providers."
+  [#"(?i)remote host terminated the handshake"
+   #"(?i)host is unreachable"
+   #"(?i)connection error"
+   #"(?i)connection refused"
+   #"(?i)UnresolvedAddressException"])
+
 (defn ^:private matches-any-pattern? [^String text patterns]
   (when text
     (some #(re-find % text) patterns)))
@@ -66,24 +74,34 @@
       (matches-any-pattern? message rate-limited-patterns)
       {:error/type :rate-limited}
 
+      (matches-any-pattern? message overloaded-patterns)
+      {:error/type :overloaded}
+
       :else nil)))
 
 (defn ^:private classify-by-custom-rules
   "Checks user-configured retry rules. Each rule may have :status (int),
-   :body-pattern (regex string, case-insensitive), and :label (string).
+   :errorPattern (regex string, case-insensitive, matched against body, message,
+   and exception message), and :label (string).
    Returns {:error/type :retryable-custom :error/label label} on first match, nil otherwise."
-  [{:keys [status body]} retry-rules]
+  [{:keys [status body message exception]} retry-rules]
   (when (seq retry-rules)
-    (some (fn [{rule-status :status rule-body-pattern :bodyPattern rule-label :label}]
+    (some (fn [{rule-status :status rule-error-pattern :errorPattern rule-label :label}]
             (let [status-matches? (if rule-status
                                     (= rule-status status)
                                     true)
-                  body-matches? (if rule-body-pattern
-                                  (when (string? body)
-                                    (re-find (re-pattern (str "(?i)" rule-body-pattern)) body))
-                                  true)
-                  has-condition? (or rule-status rule-body-pattern)]
-              (when (and has-condition? status-matches? body-matches?)
+                  error-matches? (if rule-error-pattern
+                                   (let [pattern (re-pattern (str "(?i)" rule-error-pattern))]
+                                     (or (when (string? body)
+                                           (re-find pattern body))
+                                         (when (string? message)
+                                           (re-find pattern message))
+                                         (when exception
+                                           (some-> (ex-message exception)
+                                                   (as-> msg (re-find pattern msg))))))
+                                   true)
+                  has-condition? (or rule-status rule-error-pattern)]
+              (when (and has-condition? status-matches? error-matches?)
                 (cond-> {:error/type :retryable-custom}
                   rule-label (assoc :error/label rule-label)))))
           retry-rules)))

src/eca/llm_providers/ollama.clj

@@ -65,14 +65,14 @@
                      (llm-util/log-response logger-tag rid "response-error" body)
                      (reset! response* error-data)))]
     (llm-util/log-request logger-tag rid url body headers)
-    (let [{:keys [status body]} (http/post
-                                 url
-                                 {:headers headers
-                                  :body (json/generate-string body)
-                                  :throw-exceptions? false
-                                  :http-client (client/merge-with-global-http-client {})
-                                  :as (if on-stream :stream :json)})]
-      (try
+    (try
+      (let [{:keys [status body]} (http/post
+                                   url
+                                   {:headers headers
+                                    :body (json/generate-string body)
+                                    :throw-exceptions? false
+                                    :http-client (client/merge-with-global-http-client {})
+                                    :as (if on-stream :stream :json)})]
         (if (not= 200 status)
           (let [body-str (if on-stream (slurp body) body)]
             (logger/warn logger-tag (format "Unexpected response status: %s body: %s" status body-str))
@@ -87,9 +87,10 @@
             (do
               (llm-util/log-response logger-tag rid "response" body)
               (reset! response*
-                      {:output-text (:content (:message body))}))))
-        (catch Exception e
-          (on-error {:exception e}))))
+                      {:output-text (:content (:message body))})))))
+      (catch Exception e
+        (on-error {:exception e
+                   :message (format "Connection error: %s" (or (ex-message e) (.getName (class e))))})))
     @response*))
 
 (defn ^:private ->tools [tools]

src/eca/llm_providers/openai.clj

@@ -72,14 +72,14 @@
                      (llm-util/log-response logger-tag rid "response-error" body)
                      {:error error-data}))]
     (llm-util/log-request logger-tag rid url body headers)
-    (let [{:keys [status body]} (http/post
-                                 url
-                                 {:headers headers
-                                  :body (json/generate-string body)
-                                  :throw-exceptions? false
-                                  :http-client (client/merge-with-global-http-client http-client)
-                                  :as (if on-stream :stream :json)})]
-      (try
+    (try
+      (let [{:keys [status body]} (http/post
+                                   url
+                                   {:headers headers
+                                    :body (json/generate-string body)
+                                    :throw-exceptions? false
+                                    :http-client (client/merge-with-global-http-client http-client)
+                                    :as (if on-stream :stream :json)})]
         (if (not= 200 status)
           (let [body-str (if on-stream (slurp body) body)]
             (logger/warn logger-tag "Unexpected response status: %s body: %s" status body-str)
@@ -93,9 +93,10 @@
                 (on-stream event data)))
             (do
               (llm-util/log-response logger-tag rid "response" body)
-              (response-body->result body))))
-        (catch Exception e
-          (on-error {:exception e}))))))
+              (response-body->result body)))))
+      (catch Exception e
+        (on-error {:exception e
+                   :message (format "Connection error: %s" (or (ex-message e) (.getName (class e))))})))))
 
 (defn ^:private normalize-messages [messages supports-image?]
   (keep (fn [{:keys [role content] :as msg}]

src/eca/llm_providers/openai_chat.clj

@@ -129,14 +129,14 @@
                          "Content-Type" "application/json"}
                         extra-headers))]
     (llm-util/log-request logger-tag rid url body headers)
-    (let [{:keys [status body]} (http/post
-                                 url
-                                 {:headers headers
-                                  :body (json/generate-string body)
-                                  :throw-exceptions? false
-                                  :http-client (client/merge-with-global-http-client http-client)
-                                  :as (if on-stream :stream :json)})]
-      (try
+    (try
+      (let [{:keys [status body]} (http/post
+                                   url
+                                   {:headers headers
+                                    :body (json/generate-string body)
+                                    :throw-exceptions? false
+                                    :http-client (client/merge-with-global-http-client http-client)
+                                    :as (if on-stream :stream :json)})]
         (if (not= 200 status)
           (let [body-str (if on-stream (slurp body) body)]
             (logger/warn logger-tag rid "Unexpected response status: %s body: %s" status body-str)
@@ -151,9 +151,10 @@
               (on-stream "stream-end" {}))
             (do
               (llm-util/log-response logger-tag rid "full-response" body)
-              (response-body->result body on-tools-called-wrapper))))
-        (catch Exception e
-          (on-error {:exception e}))))))
+              (response-body->result body on-tools-called-wrapper)))))
+      (catch Exception e
+        (on-error {:exception e
+                   :message (format "Connection error: %s" (or (ex-message e) (.getName (class e))))})))))
 
 (defn ^:private transform-message
   "Transform a single ECA message to OpenAI format. Returns nil for unsupported roles.