fix: detach analyze_target's per-child recon loop for large firmware (#278)

* fix: detach analyze_target's per-child recon loop for large firmware

Found while designing directory-import: promote_file and reveal_dir got
detached earlier tonight (#274, #275), but the loop they both actually
detach — analyze_target's sequential per-child sandbox recon — was never
fixed at its OWN primary call site. `ingest_and_analyze` (hexgraph ingest /
target_ingest / the web UI's upload endpoint) calls analyze_target
synchronously, inline, with no aggregate bound — a large firmware still
blocks the initial ingest call for however long sequential per-child recon
takes, the exact same bug class, just one call site earlier in the chain.

Above CHILD_RECON_DETACH_THRESHOLD (25 — comfortably above every existing
test fixture's few-file unpacks, comfortably below what a real large
firmware produces) unpacked children, recon now runs as ONE detached
process working through them sequentially, reusing the exact
create_task/spawn_detached_task infrastructure built for promote_file and
reveal_dir tonight. Child target rows still exist immediately (unpack's own
registration is a cheap host-side copy+hash, not the bottleneck) and are
returned in the summary; only their recon facts land later. A small
firmware (every existing fixture) is completely unaffected — same
synchronous behavior, same summary shape, recon_status="done".

Threaded through every entry point that calls analyze_target/
ingest_and_analyze directly: CLI (`hexgraph ingest`), MCP (`target_ingest`),
and the web UI's upload endpoint (`api_add_target`) all now surface
`recon_status` and handle the "queued" case without breaking their existing
synchronous-case output.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: mention recon_status in target_ingest's MCP catalog description

Flagged in #278's review: docs/mcp.md, vr_skill.py, and ingest()'s own
docstring were updated to explain the new detached-recon behavior, but the
advertised MCP catalog description (the one an agent actually sees under
deferred/name-only tool loading) was missed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: branover

docs/mcp.md

@@ -79,7 +79,10 @@ agent's context small:
   its own target), `task_run`, `finding_verify_poc`, `fuzz_verify_artifact`,
   `fuzz_start`/`fuzz_resume`, `src_build`, and more. `target_ingest` returns a bounded summary
   (the child count plus a preview of the first ~20 children, since firmware can unpack into
-  hundreds); call `target_list(project_id)` for the full target tree. `target_promote_file`
+  hundreds); call `target_list(project_id)` for the full target tree. Above a couple dozen
+  unpacked children, per-child recon runs detached in the background instead of blocking the
+  call — `recon_status` in the response is "done" or "queued" (children exist, their recon
+  facts land later; `target_facts` on one to check). `target_promote_file`
   returns as soon as the child target exists — analysis runs detached, since promoting a large,
   deeply-nested container can mean thousands of sequential sandbox runs; call it again with the
   same arguments to poll `analysis_status` (queued/running/succeeded/failed) rather than assuming

src/hexgraph/agent/mcp_catalog.py

@@ -109,7 +109,7 @@
      {"type": "object", "properties": {"project_id": {"type": "string"}, "include_hidden": {"type": "boolean", "description": "also list hidden firmware children (default false)"}}, "required": ["project_id"]}),
     ("read", "target_facts", _t.target_facts, "The cached recon SUMMARY for a target — its detected architecture and file format, the key imported/exported symbols, and the security mitigations recon already computed at ingest. The cheapest first orientation (it reads stored recon, no sandbox run). Note its `dangerous_imports` to find the risky libc calls fast. For the AUTHORITATIVE, uncapped symbol table + relocations + full mitigations, run re_binutils_facts instead.",
      {"type": "object", "properties": {"target_id": {"type": "string"}}, "required": ["target_id"]}),
-    ("run", "target_ingest", _t.ingest, "INGEST a binary/firmware from a local path as a new target (the bytes are processed in the sandbox; firmware unpacks into child targets). Use this to bring a binary/firmware artifact into HexGraph. Creates a project if none is given. Returns a BOUNDED summary — {root_target_id, children_count, a preview of the first ~20 children, format}; call target_list(project_id) for the FULL child set (firmware can unpack into hundreds).",
+    ("run", "target_ingest", _t.ingest, "INGEST a binary/firmware from a local path as a new target (the bytes are processed in the sandbox; firmware unpacks into child targets). Use this to bring a binary/firmware artifact into HexGraph. Creates a project if none is given. Above a couple dozen unpacked children, per-child recon runs DETACHED in the background instead of blocking this call — `recon_status` is \"done\" or \"queued\" (child targets exist either way; their recon facts land later for a large firmware). Returns a BOUNDED summary — {root_target_id, children_count, a preview of the first ~20 children, format, recon_status}; call target_list(project_id) for the FULL child set (firmware can unpack into hundreds).",
      {"type": "object", "properties": {"path": {"type": "string", "description": "local filesystem path to the binary/firmware"}, "name": {"type": "string", "description": "optional target name (defaults from the path)"}, "project_id": {"type": "string", "description": "optional; a NEW project is created if omitted"}}, "required": ["path"]}),
     ("run", "target_promote_file", _t.promote_file, "PROMOTE one file from a firmware target's unpacked filesystem into its OWN child target — the bridge from browsing the rootfs (fs_list/fs_read_file) to ANALYZING a binary in it (re_decompile_function/re_list_functions/task_run/fuzz_start). `path` is relative to the extracted root; an entry's `is_elf` flags a binary worth promoting, `added` means it's already a target. Idempotent per path. Returns as soon as the child target exists (seconds) — analysis (recon; for a CONTAINER, unpack + recon of every nested file) runs DETACHED in the sandbox when Docker is up, since a large deeply-nested firmware package can be thousands of sequential sandbox runs, minutes to hours. `analysis_status` reports queued/running/succeeded/failed; CALL AGAIN with the same target_id/path to poll — returns the same child, no duplicate work. Promoting a CONTAINER (.pkg/squashfs/cpio — e.g. a large vendor firmware package) registers its inner binaries as HIDDEN child targets once analysis succeeds; the result then reports `registered_children` (+ a note: target_list(include_hidden=true) to see them, target_set_visible to analyze one) and `packed_containers` for any still-nested containers. Returns {id, name, kind, parent_id, arch, analysis_status?, registered_children?, packed_containers?, note?}.",
      {"type": "object", "properties": {"target_id": {"type": "string", "description": "the firmware target whose filesystem holds the file"}, "path": {"type": "string", "description": "file path relative to the extracted root (see fs_list)"}}, "required": ["target_id", "path"]}),

src/hexgraph/agent/mcp_tools.py

@@ -2464,7 +2464,13 @@ def ingest(path: str, name: str | None = None, project_id: str | None = None) ->
     """Ingest a binary/firmware from a local path as a target (firmware unpacks into
     children), running recon in the sandbox. Creates a project if none is given. Returns a
     bounded summary (children_count + a preview of the first ~20 children); call
-    target_list(project_id) for the full set."""
+    target_list(project_id) for the full set.
+
+    For a large firmware (more than a couple dozen unpacked children), per-child recon runs
+    DETACHED in the background instead of blocking this call — `recon_status` is "done"
+    (small — recon already ran) or "queued" (large — the child target rows already exist and
+    are in `children`/`children_count`, but their recon facts land later; poll via
+    `target_facts` on a child, or just re-list with target_list once you expect it's done)."""
     import os
 
     from hexgraph.engine.targets.ingest import create_project, ingest_file
@@ -2490,9 +2496,12 @@ def ingest(path: str, name: str | None = None, project_id: str | None = None) ->
             "root_target_id": summary["root_target_id"],
             "children_count": summary.get("children_count", len(children)),
             "children": children[:_INGEST_CHILD_PREVIEW],
+            "recon_status": summary.get("recon_status", "done"),
         }
         if summary.get("format"):
             result["format"] = summary["format"]
+        if summary.get("recon_status") == "failed":
+            result["warning"] = "background recon could not be started for this firmware's children"
         # G01: a large blob whose container format the unpacker didn't recognize, and the carve
         # attempt extracted nothing — surface it LOUDLY (a top-level warning + the magic bytes) so
         # the operator isn't left with a silent 0-child result and no idea why.
@@ -2509,7 +2518,12 @@ def ingest(path: str, name: str | None = None, project_id: str | None = None) ->
                 "(squashfs/cpio/.pkg) are still in the tree and were NOT recursed — the deeper "
                 "service surface (e.g. the service runtime: web UI/SSH/SNMP) may be inside. "
                 "target_promote_file one to extract + register its inner binaries.")
-        if len(children) > _INGEST_CHILD_PREVIEW:
+        if summary.get("recon_status") == "queued":
+            result["note"] = (f"{len(children)} children found; recon is running in the "
+                              f"background (large firmware — sequential per-child sandbox "
+                              f"analysis can take a while). Check target_facts on a child, "
+                              f"or re-list later, to see recon facts land.")
+        elif len(children) > _INGEST_CHILD_PREVIEW:
             result["note"] = (f"{len(children)} children unpacked; showing the first "
                               f"{_INGEST_CHILD_PREVIEW}. Use target_list(project_id) for the "
                               f"full set.")

src/hexgraph/agent/vr_skill.py

@@ -83,7 +83,11 @@
   **Firmware unpacks into child targets**: the extracted binaries become their own targets and
   the rootfs becomes browsable. Those children are registered **hidden** (a firmware unpacks
   into hundreds of ELFs; a visible child each would bury the graph), but each is still
-  recon-enriched, searchable, and addressable. `target_list(project_id)` shows the firmware
+  recon-enriched, searchable, and addressable. Above a couple dozen children, per-child recon
+  runs DETACHED in the background instead of blocking this call (`recon_status` in the
+  response: "done" or "queued" — the child targets exist either way, their recon facts just
+  land later for a large firmware; `target_facts` on one to check, or just come back to it).
+  `target_list(project_id)` shows the firmware
   plus the **revealed** children; `target_list(project_id, include_hidden=true)` (or `fs_list`,
   whose entries carry `added`/`revealed`) shows the full set. Pick the binaries worth analyzing
   (httpd, cgi-bin handlers, daemons, the libraries they link) and **reveal** them —

src/hexgraph/api/routers/targets.py

@@ -71,6 +71,10 @@ def api_add_target(
                 summary = analyze_target(s, project, target, executor.get_executor())
                 build_links_against(s, project)
                 result["children"] = summary.get("children", [])
+                # A large firmware's per-child recon runs detached (engine.pipeline.
+                # CHILD_RECON_DETACH_THRESHOLD) — "queued" means the children above are
+                # registered but their recon facts land later, not that this call failed.
+                result["recon_status"] = summary.get("recon_status", "done")
             return result
     finally:
         os.unlink(tmp)

src/hexgraph/cli.py

@@ -80,10 +80,18 @@ def _cmd_ingest(args: argparse.Namespace) -> int:
         print(f"target  {summary['root_target_id']}  {summary['name']}")
         for child in summary["children"]:
             print(f"  child {child['target_id']}  {child['name']}")
-        print(
-            f"recon complete: {1 + len(summary['children'])} target(s), "
-            f"{summary['links_against_edges']} links_against edge(s)"
-        )
+        if summary.get("recon_status") == "queued":
+            print(f"{1 + len(summary['children'])} target(s) registered; recon for "
+                  f"{len(summary['children'])} child(ren) is running in the background "
+                  f"(large firmware) — check `hexgraph targets {project_id}` later.")
+        elif summary.get("recon_status") == "failed":
+            print(f"{1 + len(summary['children'])} target(s) registered, but background recon "
+                  f"could not be started for the children — re-ingest to retry.", file=sys.stderr)
+        else:
+            print(
+                f"recon complete: {1 + len(summary['children'])} target(s), "
+                f"{summary['links_against_edges']} links_against edge(s)"
+            )
     return 0
 
 

src/hexgraph/engine/pipeline.py

@@ -26,6 +26,18 @@
 _FIRMWARE_FORMATS = {"squashfs", "cpio", "disk_image"}
 _ENRICHABLE_KINDS = {"executable", "shared_library"}
 
+# How many unpacked children to recon INLINE before switching to a detached background task
+# instead. Existing test fixtures unpack into single digits (unaffected); a real large
+# firmware can unpack into thousands — reconning each is its own sandboxed container
+# spin-up, so a small firmware still analyzes fully synchronously (unchanged behavior, exact
+# same summary shape as before) while a large one returns fast with the child TARGET ROWS
+# already created (unpack_firmware's own registration is a cheap host-side copy+hash, not
+# the bottleneck — see _ensure_children_recon_detached) and a task id to poll instead of
+# blocking the ingest call for what can be hours. Same bug class already fixed tonight for
+# promote_file/reveal_dir, just at the INITIAL-ingest call site (`ingest_and_analyze` /
+# `hexgraph ingest` / `target_ingest` never routed through the detached-task system at all).
+CHILD_RECON_DETACH_THRESHOLD = 25
+
 
 def _record_progress(session: Session, target: Target, stage: str, **extra) -> None:
     """Emit a coarse ingest-progress signal so the multi-minute unpack+recon isn't a silent
@@ -69,6 +81,37 @@ def _maybe_enrich_ghidra(session: Session, project: Project, target: Target, fac
         pass
 
 
+def _ensure_children_recon_detached(session: Session, project: Project, target: Target,
+                                    children: list[Target]) -> str:
+    """Kick off recon for `children` as ONE detached process working through them
+    SEQUENTIALLY — same "one process for the whole batch, not one per item" reasoning as
+    `engine.targets.reveal._ensure_batch_ghidra_enrichment` (a directory/firmware can have
+    thousands of children; spawning one INDEPENDENT process per child would launch that many
+    CONCURRENT sandbox containers and contend hard for host resources). The Task is anchored
+    on `target` (the firmware/parent being analyzed); child ids travel in `params_json`.
+    Returns "queued", or "failed" if the spawn itself raised (the Task ends up marked
+    failed rather than stuck queued forever — see engine.targets.reveal for the same
+    self-heal reasoning)."""
+    from hexgraph.engine.tasks import create_task, mark_failed
+    from hexgraph.engine.worker import spawn_detached_task
+
+    task = create_task(session, project=project, target_id=target.id, type="recon_children_batch",
+                       params={"target_ids": [c.id for c in children]})
+    session.commit()
+    try:
+        spawn_detached_task(task.id)
+    except Exception:  # noqa: BLE001 — never let a spawn failure break the ingest call
+        mark_failed(task, "failed to spawn detached recon-children process")
+        session.commit()
+        return "failed"
+    meta = dict(target.metadata_json or {})
+    meta["recon_children_task_id"] = task.id
+    target.metadata_json = meta
+    flag_modified(target, "metadata_json")
+    session.commit()
+    return "queued"
+
+
 def analyze_target(
     session: Session,
     project: Project,
@@ -79,10 +122,16 @@ def analyze_target(
 
     Emits coarse per-stage progress on the root target's metadata (recon → unpacking →
     recon i/N children → done) so the multi-minute firmware path isn't a silent black box
-    (F05). The signal is advisory — a poller watches `metadata_json["ingest_progress"]`."""
+    (F05). The signal is advisory — a poller watches `metadata_json["ingest_progress"]`.
+
+    Above `CHILD_RECON_DETACH_THRESHOLD` children, per-child recon runs DETACHED (see
+    `_ensure_children_recon_detached`) instead of inline. `summary["recon_status"]` is
+    "done" (small — recon already ran, exactly like before), "queued" (large — child TARGET
+    ROWS exist and are already in `summary["children"]`, but their recon facts land later —
+    poll via `target_facts`/re-ingesting), or "failed" (the detach itself couldn't start)."""
     _record_progress(session, target, "recon")
     facts = run_recon(session, project, target, runner)
-    summary = {"target_id": target.id, "name": target.name, "children": []}
+    summary = {"target_id": target.id, "name": target.name, "children": [], "recon_status": "done"}
 
     # G01: a recognized firmware format OR a large blob whose format we couldn't recognize —
     # in the latter case ATTEMPT a binwalk carve anyway (it often recognizes vendor wrappers our
@@ -100,14 +149,18 @@ def analyze_target(
         # Materialize the child list first so we can report "recon i/N children" with a known N.
         children = list(unpack_firmware(session, project, target, runner))
         total = len(children)
-        for i, child in enumerate(children, start=1):
-            _record_progress(session, target, "recon_children", done=i - 1, total=total)
-            # Children are registered HIDDEN by unpack_firmware: recon ENRICHES each
-            # (metadata + a recon Observation) but materializes no graph nodes — a
-            # hidden child contributes nothing to the graph until revealed.
-            child_facts = run_recon(session, project, child, runner)
-            _maybe_enrich_ghidra(session, project, child, child_facts)
-            summary["children"].append({"target_id": child.id, "name": child.name})
+        if total > CHILD_RECON_DETACH_THRESHOLD:
+            summary["children"] = [{"target_id": c.id, "name": c.name} for c in children]
+            summary["recon_status"] = _ensure_children_recon_detached(session, project, target, children)
+        else:
+            for i, child in enumerate(children, start=1):
+                _record_progress(session, target, "recon_children", done=i - 1, total=total)
+                # Children are registered HIDDEN by unpack_firmware: recon ENRICHES each
+                # (metadata + a recon Observation) but materializes no graph nodes — a
+                # hidden child contributes nothing to the graph until revealed.
+                child_facts = run_recon(session, project, child, runner)
+                _maybe_enrich_ghidra(session, project, child, child_facts)
+                summary["children"].append({"target_id": child.id, "name": child.name})
         # F07: flag packed containers the unpack left in the tree (a large vendor firmware image leaves
         # the real web UI/SSH/SNMP runtime in nested .pkg/squashfs that aren't auto-recursed).
         # Without this, "N children unpacked" reads as "fully unpacked" and a researcher hunts the
@@ -143,7 +196,8 @@ def analyze_target(
     else:
         _maybe_enrich_ghidra(session, project, target, facts)
     summary["children_count"] = len(summary["children"])
-    _record_progress(session, target, "done", children=summary["children_count"])
+    final_stage = "recon_children_queued" if summary["recon_status"] == "queued" else "done"
+    _record_progress(session, target, final_stage, children=summary["children_count"])
     return summary
 
 

src/hexgraph/engine/worker.py

@@ -70,6 +70,31 @@ def _dispatch(session: Session, project: Project, target: Target, task: Task) ->
         analyze_target(session, project, target, get_executor())
         build_links_against(session, project)
         return
+    if task.type == "recon_children_batch":
+        # analyze_target's large-firmware path (engine.pipeline.CHILD_RECON_DETACH_THRESHOLD):
+        # ONE detached process recons every child SEQUENTIALLY — same "one process for the
+        # whole batch, not one per item" reasoning as ghidra_enrich_batch below. `target`
+        # here is the PARENT (firmware) being analyzed, not any one of the actual children —
+        # those travel in params_json.
+        from hexgraph.engine.pipeline import _maybe_enrich_ghidra, _record_progress
+        from hexgraph.engine.re.recon import run_recon
+
+        child_ids = (task.params_json or {}).get("target_ids", [])
+        total = len(child_ids)
+        for i, cid in enumerate(child_ids, start=1):
+            # _record_progress commits — this doubles as the per-child checkpoint, same as
+            # the inline (small-firmware) loop already relied on.
+            _record_progress(session, target, "recon_children", done=i - 1, total=total)
+            child = session.get(Target, cid)
+            if child is None:
+                continue
+            try:
+                child_facts = run_recon(session, project, child, get_executor())
+                _maybe_enrich_ghidra(session, project, child, child_facts)
+            except Exception:  # noqa: BLE001 — one bad child must not abort the rest
+                pass
+        _record_progress(session, target, "done", children=total)
+        return
     if task.type == "ghidra_enrich":
         from hexgraph.engine.re.ghidra import enrich_target