Retry transient full-scan upload failures (502/503/504/408, dropped connections)

Production api-v0 backends occasionally stop reading an upload mid-request
(pod event-loop stalls during load episodes, e.g. the 2026-06-08 'Degraded
API Performance' incident): the connection freezes mid-body, Cloudflare
returns a 502 to the client after ~30s, and the backend later 408s the
half-read request. The CLI previously made exactly one attempt, so an entire
tier-1 reachability run died on a single transient failure even though a
retried upload routed to a healthy backend almost always succeeds.

create_full_scan now retries the fullscans POST up to 3 total attempts with
increasing waits (~10s, then ~30s, plus jitter) on transient failures only:
APIBadGateway (502), APIConnectionError, APITimeout, and catch-all APIFailure
whose embedded original_status_code is 408/503/504. Dedicated 4xx classes,
catch-all 400s, and error payloads are never retried. Because the server
never finished reading the request body, no scan was created server-side,
so a retry cannot duplicate a scan.

The retry loop lives inside the existing try/finally so the brotli-compressed
.socket.facts.json.br temp files survive until every attempt has finished;
fullscans.post rebuilds its lazy file loaders from the plain paths on every
call, so re-invoking it per attempt is safe.

Assisted-by: Claude Code:claude-opus-4-8

Author: mtorp

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 2.4.8
+
+### Fixed: retry transient full-scan upload failures
+
+- The full-scan upload (`POST /orgs/<org>/full-scans`) now retries transient
+  gateway/connection failures — HTTP 502/503/504/408, dropped or reset connections, and
+  request timeouts — up to 3 total attempts with increasing waits (~10s, then ~30s, plus
+  jitter). Production gateways occasionally drop an upload mid-request when a backend pod
+  stalls and stops reading the body (the client sees a 502 after ~30s); these episodes are
+  transient and a retried upload almost always succeeds. Since the server never finished
+  reading the request body, no scan was created, so retrying cannot duplicate a scan.
+  Non-transient errors (400/401/403/404/429 and error payloads) are never retried. Each
+  retry logs a warning explaining what failed and when the next attempt happens.
+
 ## 2.4.7
 
 ### Changed: pin @coana-tech/cli version; auto-update is now opt-in

pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "socketsecurity"
-version = "2.4.7"
+version = "2.4.8"
 requires-python = ">= 3.11"
 license = {"file" = "LICENSE"}
 dependencies = [

socketsecurity/__init__.py

@@ -1,3 +1,3 @@
 __author__ = 'socket.dev'
-__version__ = '2.4.7'
+__version__ = '2.4.8'
 USER_AGENT = f'SocketPythonCLI/{__version__}'

socketsecurity/core/__init__.py

@@ -1,5 +1,6 @@
 import logging
 import os
+import random
 import re
 import sys
 import tarfile
@@ -13,7 +14,12 @@
 if TYPE_CHECKING:
     from socketsecurity.config import CliConfig
 from socketdev import socketdev
-from socketdev.exceptions import APIFailure
+from socketdev.exceptions import (
+    APIBadGateway,
+    APIConnectionError,
+    APIFailure,
+    APITimeout,
+)
 from socketdev.fullscans import FullScanParams, SocketArtifact
 from socketdev.org import Organization
 from socketdev.repos import RepositoryInfo
@@ -76,6 +82,46 @@
 TIER1_FINALIZE_MAX_ATTEMPTS = 3
 TIER1_FINALIZE_BACKOFF_SECONDS = 1.0
 
+# Full scan upload retry policy. Production gateways occasionally drop an upload mid-request
+# (a backend pod stalls and stops reading the body; the client then sees a 502/408 or a reset
+# connection). Those episodes are transient and pod-local: a retried upload routed to another
+# backend almost always succeeds, and because the server never finished reading the request
+# body, no scan was created, so retrying cannot duplicate a scan.
+FULL_SCAN_UPLOAD_MAX_ATTEMPTS = 3
+# Wait before retry attempt 2 and attempt 3 respectively (plus a little jitter so a fleet of
+# CI jobs hitting the same episode doesn't retry in lock-step).
+FULL_SCAN_UPLOAD_BACKOFF_SCHEDULE_SECONDS = (10.0, 30.0)
+FULL_SCAN_UPLOAD_BACKOFF_JITTER_SECONDS = 2.0
+# Transient gateway/timeout HTTP statuses that the SDK does NOT raise as a dedicated
+# exception class (502 has APIBadGateway; 408/503/504 surface as the catch-all APIFailure
+# with the status only present in the message text - see _is_transient_full_scan_upload_error).
+FULL_SCAN_UPLOAD_RETRYABLE_STATUS_CODES = frozenset({408, 503, 504})
+# Matches the status code the SDK embeds in catch-all APIFailure messages
+# (socketdev/core/api.py: "Bad Request: HTTP original_status_code:<code> ...").
+_API_FAILURE_STATUS_CODE_RE = re.compile(r"original_status_code:(\d{3})")
+
+
+def _is_transient_full_scan_upload_error(error: Exception) -> bool:
+    """Whether a full-scan upload failure is transient and safe to retry.
+
+    Transient means the failure happened at the gateway/connection level before the server
+    finished reading the request body (so no scan was created server-side): HTTP 502/503/504/408,
+    client-side timeouts, and dropped/reset connections. 4xx client errors (400/401/403/404/429)
+    and success responses carrying an error payload are never retried.
+    """
+    if isinstance(error, (APIBadGateway, APIConnectionError, APITimeout)):
+        # 502 / connection reset-dropped / request timeout - the SDK raises dedicated classes.
+        return True
+    if type(error) is APIFailure:
+        # The SDK raises 408/503/504 (and every other status without a dedicated class,
+        # including 400) as the catch-all APIFailure, so match on the exact class plus the
+        # status code embedded in the message. Subclasses (APIAccessDenied, APIResourceNotFound,
+        # APIInsufficientQuota, ...) are deliberately excluded - those are never transient.
+        match = _API_FAILURE_STATUS_CODE_RE.search(str(error))
+        if match:
+            return int(match.group(1)) in FULL_SCAN_UPLOAD_RETRYABLE_STATUS_CODES
+    return False
+
 
 def _humanize_alert_type(alert_type: str) -> str:
     """Convert a camelCase/PascalCase alert type into a Title-Cased label.
@@ -787,7 +833,33 @@ def create_full_scan(self, files: List[str], params: FullScanParams, base_paths:
         # facts file under the per-file upload size cap. See _compress_facts_files_for_upload.
         upload_files, compressed_temp_files = self._compress_facts_files_for_upload(files)
         try:
-            res = self.sdk.fullscans.post(upload_files, params, use_types=True, use_lazy_loading=True, max_open_files=50, base_paths=base_paths)
+            # Retry transient gateway/timeout failures (502/503/504/408, dropped connections,
+            # timeouts) with increasing waits; a stalled backend pod recovers or gets routed
+            # around within minutes, and since it never finished reading the request body no
+            # scan was created, so a retry cannot duplicate one. fullscans.post() rebuilds its
+            # lazy file loaders from the plain paths in upload_files on every call, so simply
+            # calling it again per attempt is safe. The loop must stay inside this try so the
+            # temp .br files (cleaned up in the finally below) outlive every attempt.
+            for attempt in range(1, FULL_SCAN_UPLOAD_MAX_ATTEMPTS + 1):
+                try:
+                    res = self.sdk.fullscans.post(upload_files, params, use_types=True, use_lazy_loading=True, max_open_files=50, base_paths=base_paths)
+                    break
+                except APIFailure as error:
+                    if attempt >= FULL_SCAN_UPLOAD_MAX_ATTEMPTS or not _is_transient_full_scan_upload_error(error):
+                        raise
+                    backoff_index = min(attempt, len(FULL_SCAN_UPLOAD_BACKOFF_SCHEDULE_SECONDS)) - 1
+                    wait_seconds = FULL_SCAN_UPLOAD_BACKOFF_SCHEDULE_SECONDS[backoff_index] + random.uniform(
+                        0, FULL_SCAN_UPLOAD_BACKOFF_JITTER_SECONDS
+                    )
+                    # SDK error messages can span many lines (path + response headers); the
+                    # first line carries the status, which is all the warning needs.
+                    error_summary = str(error).strip().splitlines()[0] if str(error).strip() else ""
+                    log.warning(
+                        f"Full scan upload failed with {type(error).__name__}({error_summary}), "
+                        f"retrying in {wait_seconds:.0f}s "
+                        f"(attempt {attempt + 1}/{FULL_SCAN_UPLOAD_MAX_ATTEMPTS})"
+                    )
+                    time.sleep(wait_seconds)
         finally:
             for temp_file in compressed_temp_files:
                 try: