fix(agent): kill ECS baseline-build hang + early Linear ACK (ABCA-707)

A Linear task against the full ABCA monorepo wedged silently for 50+ min:
the pre-agent baseline build (mise run build → the agent pytest suite) hung
on the ECS substrate, and the issue showed no reaction/comment/state the
whole time. Root-caused live on ABCA-707. Three fixes:

1. Kill the hang at its root (agent/tests/conftest.py). The ECS agent task
   def sets AGENT_SESSION_ROLE_ARN. With it set, aws_session resolves a
   *scoped* session and tenant_client returns session.client(...), which
   BYPASSES a @patch("boto3.client") mock. test_attachments then makes a
   REAL S3 get_object that blocks forever on the ECS network (no egress) in
   a socket read SIGALRM can't interrupt. Add AGENT_SESSION_ROLE_ARN to the
   _clean_env scrub list so every test resolves the unscoped path where the
   mock intercepts. reset_session_cache() alone was insufficient — a cold
   get_session() re-resolves scoped while the var is still set. + regression
   test (TestConftestScrubsScopingEnv).

2. Make the hang watchdog actually reap (agent/tests/conftest.py). The prior
   faulthandler.dump_traceback_later(1200, exit=True) never fired: pytest's
   faulthandler_timeout re-arms faulthandler's single timer per-test WITHOUT
   exit=True, cancelling the session-level exit timer. Replace with an
   independent daemon-thread reaper that dumps all stacks and os._exit(1)s at
   600s, so a future hang fails the build fast instead of burning to the
   3600s ceiling.

3. Early Linear ACK (agent/src/pipeline.py). Move token resolve + 👀 reaction
   + Backlog→In Progress + start comment to BEFORE setup_repo() so a
   large-repo task shows immediate feedback during the multi-minute baseline
   build instead of looking dead. As a side benefit, a setup-phase failure
   now has a 👀 for the existing outer crash handler to swap to ❌ — closing
   the silent-stuck-issue gap. configure_channel_mcp stays after the clone
   (it needs the repo dir).

Author: isadeks

agent/src/pipeline.py

@@ -953,28 +953,23 @@ def _on_trace_truncated(max_bytes: int, first_dropped: int) -> None:
             if prompt_version:
                 os.environ["PROMPT_VERSION"] = prompt_version
 
-            # Setup repo (deterministic pre-hooks)
-            with task_span("task.repo_setup") as setup_span:
-                setup = setup_repo(config, progress=progress)
-                setup_span.set_attribute("build.before", setup.build_before)
-            progress.write_agent_milestone(
-                "repo_setup_complete",
-                f"branch={setup.branch} build_before={setup.build_before}",
-            )
-
-            system_prompt = build_system_prompt(config, setup, hc, system_prompt_overrides)
-
-            # Channel-specific MCP wiring. Must happen before
-            # discover_project_config so the scan picks up the file we just
-            # wrote. Resolve the per-channel access token from Secrets
-            # Manager *before* writing .mcp.json so the child SDK process
-            # inherits the env var that the MCP server entry references
-            # (${LINEAR_API_TOKEN} / ${JIRA_API_TOKEN}).
+            # ── Early ACK (ABCA-707) ─────────────────────────────────────────
+            # Acknowledge the task is picked up BEFORE the (potentially long)
+            # pre-agent baseline build in setup_repo(). On a large repo that
+            # baseline is minutes (up to the build-verify ceiling); posting the
+            # 👀 only *after* it left the issue looking dead for the whole phase
+            # (the ABCA-707 symptom: no reaction, comment, or state change for
+            # 30+ min). None of these calls needs the cloned repo — they act on
+            # the channel issue via its API token + issue id from channel
+            # metadata — so they belong before the clone/build.
+            #
+            # Resolve the per-channel access token from Secrets Manager first
+            # (react_task_started/comment_task_started read the env var it sets).
+            # configure_channel_mcp DOES need setup.repo_dir, so it stays below.
             if config.channel_source == "linear":
                 resolve_linear_api_token(config.channel_metadata)
             elif config.channel_source == "jira":
                 resolve_jira_oauth_token(config.channel_metadata)
-            configure_channel_mcp(setup.repo_dir, config.channel_source)
 
             # 👀 on the Linear issue — acknowledges the task is picked up.
             # No-op for non-Linear tasks. Best-effort; failures are logged
@@ -1000,6 +995,31 @@ def _on_trace_truncated(max_bytes: int, first_dropped: int) -> None:
                 config.channel_metadata,
             )
 
+            # Setup repo (deterministic pre-hooks). A failure/timeout/OOM in the
+            # pre-agent baseline build raises here; it needs no local handler —
+            # the outer ``except Exception`` at the bottom of this ``try`` writes
+            # the task FAILED, swaps the 👀 (posted above) to ❌, and posts the
+            # failure comment. Before the Early-ACK move the 👀 didn't exist yet
+            # at this point, so a setup failure left the issue silently stuck
+            # (the ABCA-707 symptom); posting the 👀 earlier is what makes the
+            # outer handler's ❌-swap actually visible for setup failures.
+            with task_span("task.repo_setup") as setup_span:
+                setup = setup_repo(config, progress=progress)
+                setup_span.set_attribute("build.before", setup.build_before)
+            progress.write_agent_milestone(
+                "repo_setup_complete",
+                f"branch={setup.branch} build_before={setup.build_before}",
+            )
+
+            system_prompt = build_system_prompt(config, setup, hc, system_prompt_overrides)
+
+            # Channel-specific MCP wiring. Must happen before
+            # discover_project_config so the scan picks up the file we just
+            # wrote — and after the clone, since it writes .mcp.json into the
+            # repo dir. (Token resolution + the 👀/start ACK moved earlier so
+            # the user gets immediate feedback; see the Early ACK block above.)
+            configure_channel_mcp(setup.repo_dir, config.channel_source)
+
             # Download attachments from S3 (version-pinned, integrity-verified)
             prepared_attachments: list = []
             if config.attachments:

agent/tests/conftest.py

@@ -1,27 +1,55 @@
 """Shared fixtures for agent unit tests."""
 
 import faulthandler
+import os
 import sys
+import threading
 from types import SimpleNamespace
 
 import pytest
 
 from models import TaskConfig
 
-# Session-wide hang backstop, immune to what pytest-timeout's SIGALRM cannot
-# reach. SIGALRM fires only in the MAIN thread during a test's *call* phase, so a
-# deadlock in a WORKER thread, a fixture, collection, or a C-level socket read the
-# main thread never returns from stalls the whole `mise run build` silently — up
-# to the platform's 3600s build-verify ceiling (the ECS-only stall seen on
-# ABCA-684/686/688 and the warm-cache run: 53 min of dead air, signal never
-# fired). faulthandler runs a dedicated C watchdog thread that the GIL and blocked
-# syscalls can't stall: at 1200s (20 min — comfortably inside the 3600s ceiling,
-# far above the whole suite's normal ~3 min) it dumps EVERY thread's Python stack
-# to stderr and hard-exits, converting a blind stall into a self-diagnosing crash
-# that names the exact file:line. Complements the per-test faulthandler_timeout in
-# pyproject.toml (which dumps but does not exit). ``exit=True`` guarantees the
-# non-zero exit even if the hang is uninterruptible.
-faulthandler.dump_traceback_later(1200, exit=True, file=sys.stderr)
+# Session-wide hang backstop. SIGALRM (pytest-timeout method="signal") fires only
+# in the MAIN thread during a test's *call* phase, so a deadlock in a WORKER
+# thread, a fixture, collection, or a C-level socket read the main thread never
+# returns from stalls the whole `mise run build` silently — up to the platform's
+# 3600s build-verify ceiling (the ECS-only stall on ABCA-684/686/688, and again
+# on ABCA-707: 40+ min of dead air, container never reaped).
+#
+# The obvious instrument — ``faulthandler.dump_traceback_later(1200, exit=True)``
+# — does NOT work here: faulthandler has a SINGLE internal timer, and pytest's
+# ``faulthandler_timeout`` (pyproject.toml) RE-ARMS it at the start of every test
+# WITHOUT ``exit=True``. So a session-level exit timer is cancelled by the first
+# test, the per-test timer only DUMPS, and the suite hangs forever anyway
+# (exactly what happened on ABCA-707: a 300s dump fired, no exit followed).
+#
+# So own the reaper on a dedicated daemon thread pytest cannot touch. A blocked
+# socket read (the ABCA-707 failure mode) releases the GIL, so this thread runs;
+# it dumps every thread's stack for diagnosis and then HARD-EXITS the process, so
+# `mise run build` returns non-zero within seconds of the deadline instead of
+# burning to the 3600s ceiling. Deadline 600s: ~200x the whole suite's normal
+# ~3s… well above any legitimate run (per-test cap is 300s) yet far under the
+# build ceiling.
+_HANG_REAP_DEADLINE_S = 600
+
+
+def _reap_on_hang() -> None:
+    faulthandler.dump_traceback(all_threads=True, file=sys.stderr)
+    print(
+        f"\nCONFTEST HANG WATCHDOG: test session exceeded {_HANG_REAP_DEADLINE_S}s "
+        "— dumped all thread stacks above and hard-exiting so the build fails "
+        "fast instead of stalling to the build-verify ceiling.",
+        file=sys.stderr,
+        flush=True,
+    )
+    os._exit(1)
+
+
+# daemon=True so a clean, fast suite exit is never blocked waiting on this timer.
+_hang_watchdog = threading.Timer(_HANG_REAP_DEADLINE_S, _reap_on_hang)
+_hang_watchdog.daemon = True
+_hang_watchdog.start()
 
 
 class FakeRunCmd:
@@ -111,6 +139,16 @@ def make_task_config(**overrides) -> TaskConfig:
     "LOG_GROUP_NAME",
     "MEMORY_ID",
     "ENABLE_CLI_TELEMETRY",
+    # Per-session IAM scoping (PR #209). When this is set, ``aws_session`` resolves
+    # a *scoped* session and ``tenant_client`` returns ``session.client(...)`` —
+    # which BYPASSES a ``@patch("boto3.client")`` mock. On the ECS substrate the
+    # task def sets this, so a test that mocks ``boto3.client`` (e.g.
+    # ``test_attachments``) instead makes a REAL S3 call that hangs on the network
+    # (no egress). Stripping it here forces every test onto the unscoped path,
+    # where the ``boto3.client`` mock actually intercepts. Resetting the session
+    # cache below is NOT enough on its own — a fresh ``get_session()`` re-resolves
+    # as scoped while this var is still set. (Live-caught on ABCA-707, 2026-07-15.)
+    "AGENT_SESSION_ROLE_ARN",
 ]
 
 
@@ -121,15 +159,19 @@ def _clean_env(monkeypatch):
     The env cleanup keeps host state from leaking into agent code that reads
     ``os.environ`` at import/call time.
 
-    The session reset closes a cross-test leak: ``aws_session`` caches the
-    resolved boto3 session in a MODULE GLOBAL (``_session``/``_scoped``), and
-    ``tenant_client`` returns ``session.client(...)`` from that cache when
-    ``_scoped`` is True — bypassing a downstream ``@patch("boto3.client")``. If an
-    earlier test left a scoped session cached (only ``test_aws_session`` cleans up
-    after itself), a later test like ``test_attachments`` would get a REAL client
-    despite its mock → a live S3 call that hangs on the ECS network (no route/no
-    creds) in a socket read SIGALRM can't interrupt. Resetting the cache before
-    every test guarantees each test resolves the session under its own patches.
+    The env cleanup + session reset TOGETHER close a scoped-session leak that
+    hangs the suite on the ECS substrate: ``aws_session`` caches the resolved
+    boto3 session in a MODULE GLOBAL (``_session``/``_scoped``), and
+    ``tenant_client`` returns ``session.client(...)`` when ``_scoped`` is True —
+    bypassing a downstream ``@patch("boto3.client")``. Two things make a test
+    resolve *scoped*: a stale cached session (fixed by ``reset_session_cache``),
+    OR ``AGENT_SESSION_ROLE_ARN`` still being set when the cache is cold (fixed by
+    stripping it in ``_AGENT_ENV_VARS`` above — the ECS task def sets it, so on
+    that substrate the reset alone re-resolves scoped and the leak persists). With
+    the var gone AND the cache reset, every test resolves the unscoped path where
+    its ``boto3.client`` mock intercepts. Otherwise a mocked test (e.g.
+    ``test_attachments``) makes a REAL S3 call that hangs on the ECS network
+    (no egress) in a socket read SIGALRM can't interrupt. (Live-caught ABCA-707.)
     """
     for var in _AGENT_ENV_VARS:
         monkeypatch.delenv(var, raising=False)

agent/tests/test_aws_session.py

@@ -79,6 +79,35 @@ def test_blank_role_arn_treated_as_unset(self, monkeypatch):
         assert is_scoped() is False
 
 
+class TestConftestScrubsScopingEnv:
+    """Regression guard for the ECS test-hang (ABCA-707).
+
+    The bug: the ECS agent task def sets ``AGENT_SESSION_ROLE_ARN``. If that var
+    leaks into the test process, ``get_session()`` resolves a *scoped* session and
+    ``tenant_client`` returns ``session.client(...)`` — bypassing a
+    ``@patch("boto3.client")`` mock. A mocked test (e.g. ``test_attachments``)
+    then makes a REAL S3 call that hangs on the ECS network (no egress) in a
+    socket read SIGALRM can't interrupt → the whole ``mise run build`` stalls
+    silently for 40+ min. The ``_clean_env`` autouse fixture in conftest MUST
+    strip the var (resetting the session cache alone is not enough — a cold
+    ``get_session()`` re-resolves scoped while the var is still set).
+    """
+
+    def test_session_role_arn_not_visible_to_tests(self):
+        import os
+
+        # No monkeypatch here: this asserts the AUTOUSE fixture already scrubbed
+        # the var, even if the parent (ECS) environment had it set.
+        assert os.environ.get(SESSION_ROLE_ARN_ENV) is None
+
+    def test_session_resolves_unscoped_by_default(self):
+        # With the var scrubbed and the cache reset (both by _clean_env), a bare
+        # get_session() must be unscoped — the path where boto3.client mocks work.
+        with patch("boto3.Session", return_value=MagicMock()):
+            get_session()
+        assert is_scoped() is False
+
+
 # ---------------------------------------------------------------------------
 # Scoped: SessionRole ARN set → refreshable assumed-role session
 # ---------------------------------------------------------------------------