diffctx hangs unbounded (no output, OOM/SIGKILL) on large repos for trivial diffs; 300s timeout ineffective

[nikolay-e]

Summary

diffctx . --diff HEAD~1 hangs indefinitely (no output, eventually OOM/SIGKILL,
rc=137/142) on large real-world repos for a trivial diff. The documented
300s pipeline timeout does not bound total runtime.

Repro (diffctx 1.10.0, pipx binary)

Real-world test repos, each diffctx . --diff HEAD~1:

repo	diff size	result
gitpod (Go/TS monorepo)	11 files, 12+/12- (24 lines)	>200s, 0 bytes output, killed (rc=142 at a hard 200s cap; rc=137 after ~8000s unbounded)
pytorch	7 files	~1848s then SIGKILL (rc=137), 0 output
react-native (similar monorepo)	5 files	rc=0 in 2s ✓
elasticsearch	1 file	rc=0 in 37s ✓
sentry / numpy / llama.cpp / …	1–5 files	rc=0, seconds ✓

So it is repo-specific (size/shape), not diff-size driven: a 24-line gitpod diff
never completes while a 5-file react-native diff is instant.

Notable

• The cost is repo-size-proportional, not diff-proportional — points at the
  candidate-discovery / parse / graph-build phase scanning the whole tree, not
  the (tiny) diff.
• The pipeline timeout (default 300s, DEFAULT_PIPELINE_TIMEOUT) does not
  abort it — gitpod ran ~8000s in one run. The timeout does not cover the
  hanging phase (or isn't enforced over total wall-clock). A bounded run should
  fail fast with the actionable-error contract, not hang for hours.

Expected

Either complete in seconds for a trivial diff (like react-native), or the
timeout aborts with a clear error well before SIGKILL. Hanging unbounded +
OOM is the bug.

Likely related

Same family as #65 (the engine pulls in / scans far more of the repo than the
diff warrants); here it crosses from over-selection into non-termination on the
largest repos.

[nikolay-e]

Root-cause diagnosis (QA pass)

The pipeline timeout (default 300s) is wired only to git subprocess calls —
pipeline.rs calls git::set_git_timeout(timeout) and the various
Instant::now() markers (t_parse_changed, t_universe, t_discovery,
t_parse_discovered, t_tokenization) are telemetry only, not deadline
checks. So the in-process phases (candidate-universe discovery, parsing,
tokenization, graph build) have no wall-clock bound — that's why gitpod runs
~8000s without the 300s timeout firing.

Running gitpod with --log-level debug and a 70s cap produced zero output
before the cap — the process hangs in an early phase before any phase log
flushes (git-diff/changed-file parse or the candidate "universe" enumeration),
not in selection/scoring. Combined with cost being repo-size- not
diff-proportional, the likely culprit is whole-repo candidate enumeration/parse
on a very large monorepo.

Suggested fix direction (needs design + benchmark validation, not a quick patch)

1. Enforce a real wall-clock deadline across ALL phases (thread
   deadline = t0 + timeout and check it inside the hot discovery/parse
   loops, not just between phases — gitpod is stuck mid-phase, so a between-phase
   check wouldn't fire), aborting with the actionable-error contract instead of
   hanging → SIGKILL.
2. Bound candidate-universe size / discovery work by repo size so large repos
   degrade gracefully (ties into diffctx over-selection: PPR/edge relevance filter pulls in unrelated files (whole-tree over-dump + co-located k8s manifests) #65's over-reach).

Filed from the mandatory real-world test-repo sweep; not fixed in this QA pass
(core-engine change to a freshly-released 1.10.0 — needs deliberate design +
the yaml_cases/benchmark validation, same class as #65).

[nikolay-e]

Extra data point (overnight unbounded run)

Letting it run without a cap: gitpod eventually completed after 17471s
(~4.9 hours) — 11-file diff → 54 out-files / 116 fragments / 19k tokens, range
correctly honored. So it is not a true deadlock but pathological
O(repo-size) slowness that, under memory pressure, gets OOM-killed first
(pytorch: SIGKILL rc=137 at ~169s in a tighter-memory run, ~1848s in another).
Net effect for any user on a large monorepo: minutes-to-hours hang or OOM on a
trivial diff. Reframe: 'unbounded/too-slow + OOM', the fix direction (real
wall-clock deadline + bounded discovery) stands.

(Note: linux in the sweep returned exit 3 instantly — that clone has no
commits/HEAD, and diffctx correctly emits the environment-error contract. Not a
bug; just excluding it from the sweep.)

[nikolay-e]

Wall-clock deadline landed (3725f51) — symptom bounded

The titled defect (unbounded hang / --timeout not bounding total runtime) is fixed in 3725f51b: the CLI now runs the pipeline on a worker thread and enforces --timeout as a true total wall-clock ceiling, not just a per-git-subprocess timeout. On expiry it fails fast with the actionable-error contract and rc=124 instead of hanging until OOM/SIGKILL.

Verified on the known-hang repo:

$ diffctx . --diff HEAD~1 --timeout 15   # gitpod
diffctx: pipeline exceeded 15s wall-clock deadline; aborting before OOM/SIGKILL. ...
# terminated at 15.2s, rc=124 (was: 0 bytes output, ~8000s then SIGKILL)

Scope note: this bounds the symptom (no more unbounded hang/OOM). It does NOT make a trivial diff on a giant monorepo complete — the underlying repo-size-proportional cost (whole-repo candidate enumeration / parse / graph build scanning far more than the diff warrants) is the over-scan root cause tracked in #65. The fix is deliberately CLI-only (main.rs) so the library/MCP paths are unaffected.

Keeping this open until the over-scan perf in #65 lets large repos finish fast, rather than only fail fast.

[nikolay-e]

Fresh sweep data point (QA pass) — fix confirmed on a third giant monorepo

Real-world test-repo sweep hit the cap on elasticsearch (fresh pull, HEAD~1 diff): the pipx 1.10.0 binary (predates 3725f51) hung past the external 200s perl alarm cap, rc=142, zero output — same class as gitpod/pytorch.

Re-ran the same range on the dev build (has the 3725f51 wall-clock fix) with --timeout 60:

$ diffctx . --diff HEAD~1 --format yaml --timeout 60   # elasticsearch
diffctx: pipeline exceeded 60s wall-clock deadline; aborting before OOM/SIGKILL. ...
# terminated at 60.7s wall, rc=124 (94.8s of CPU time burned across threads before the deadline check fired)

Confirms the fix generalizes beyond the two originally-reported repos — third independent giant monorepo, same fast-fail behavior, no hang. Sweep stopped here per QA.md stopping condition (cap-hit = issue; this is the already-tracked #70/#65 root cause, not a new defect, so filing here rather than opening a duplicate).

[nikolay-e]

New reproduction datapoints (QA sweep 2026-07-07)

The unbounded hang is not limited to large repos with trivial diffs — it
also fires on large diffs (60–100+ changed files), on repos that handle
their own HEAD~1 fine. Same symptom: zero stdout, empty stderr, runs until an
external cap kills it (the pipx-published Python wrapper still exposes no
--timeout / has no wall-clock watchdog — verified on 1.10.2).

Verified (external 150s perl alarm cap on the pipx binary
/Users/nikolay/.local/bin/diffctx):

repo	commit	diff size	result
onnxruntime	22ae796 ([CUDA] Promote CUDA plugin EP as default provider #29544)	63 files, +963/−365	rc=142, elapsed 150s (cap), 0 bytes stdout, empty stderr
elasticsearch	e24492c (ESQL: serve warm COUNT/MIN/MAX … #150920)	108 files, +11799/−1915	hang, killed at 150s cap

Repro:

git clone https://github.com/microsoft/onnxruntime && cd onnxruntime
git checkout 22ae796471889b9e1318b67537320203ea880151
perl -e 'alarm 150; exec @ARGV' diffctx . --diff HEAD~1 -f yaml
# → rc=142 after 150s, no output at all

Note the same repos complete other (smaller) commits in this sweep in a few
seconds, so the trigger correlates with diff/graph size, not merely repo size.
Reinforces the need for the Python-wrapper watchdog around the pyo3
compute_scored_state call (the Rust main.rs --timeout never crossed into
the Python CLI).