Add node lab debug status and attach UX

Author: nshkrdotcom

.blitz/test_state_v1/indexes/task_states.ndjson

@@ -0,0 +1,43 @@
+# Distributed Cleanup Runbook
+
+Use this runbook after local distributed StackLab proofs, especially after a
+failed run or interrupted terminal session.
+
+## Normal Cleanup
+
+```bash
+mix stack_lab.gn_ten.node_lab.down --json
+```
+
+The command removes the node-lab run-state file. Peer-mode proofs also stop
+peers before the original command returns and record cleanup receipts.
+
+## Verify EPMD State
+
+```bash
+epmd -names
+```
+
+No expected `stack_lab` proof node should remain after cleanup. If a local
+debug session is still attached, close it before treating the cleanup as
+complete.
+
+## Inspect Status
+
+```bash
+mix stack_lab.gn_ten.node_lab.status --json
+```
+
+A clean state reports `no_active_run`. If a run state remains, inspect the
+status summary for log paths, cleanup posture, and node ids, then rerun
+`down`.
+
+## Worktree Hygiene
+
+```bash
+git status --short
+```
+
+Generated runtime output belongs under ignored or temporary paths. A proof run
+must not create tracked source, fixture, or docs changes unless the phase
+explicitly updates deterministic fixtures or proof-matrix rows.

docs/runbooks/distributed_cleanup.md

@@ -34,11 +34,33 @@ retention. That claim belongs to a later daemon or release-path controller.
 ```bash
 mix stack_lab.gn_ten.node_lab.status --json
 mix stack_lab.gn_ten.node_lab.probe --node fixture_profile_0 --json
+mix stack_lab.gn_ten.node_lab.attach --node fixture_profile_0 --json
 mix stack_lab.gn_ten.node_lab.down --json
 ```
 
-`status` reads the latest run-state receipt. `probe` reads one logical node
-receipt from that state. `down` removes the run-state file and is idempotent.
+`status` reads the latest run-state receipt and reports a debug summary:
+logical node ids, profiles, peer node names, started OTP apps, owner `:pg`
+membership, facade readiness, current connection posture, log path, latest
+receipt refs, cleanup posture, and artifact hygiene. `probe` reads one logical
+node receipt from that state. `attach` prints a redacted local debug-shell
+recipe for the node. `down` removes the run-state file and is idempotent.
+
+## Attach
+
+```bash
+mix stack_lab.gn_ten.node_lab.attach --node fixture_profile_0 --json
+```
+
+The attach receipt names the node and profile and prints only a redacted
+manual shell shape:
+
+```text
+iex --sname debug_shell_<run> --cookie <redacted_run_cookie> --remsh <node>
+```
+
+Phase 15 does not store or print cookie values. It also does not claim durable
+cross-command peer retention; current peer-mode proofs clean up peers before
+returning unless a future retained controller owns that stronger behavior.
 
 ## Safety Notes
 

docs/runbooks/distributed_node_lab.md

@@ -0,0 +1,68 @@
+# Distributed Debugging
+
+StackLab distributed debugging is a local proof-harness workflow. It helps
+inspect peer-mode runs without turning Erlang distribution into a production
+security model or a platform business API.
+
+## Status
+
+```bash
+mix stack_lab.gn_ten.node_lab.status --json
+```
+
+The status receipt reports:
+
+- run id and topology ref;
+- logical node id, profile, and BEAM node name;
+- started OTP apps;
+- owner-defined `:pg` membership;
+- facade readiness;
+- current connection posture;
+- log path;
+- latest receipt refs;
+- cleanup posture;
+- artifact hygiene for run-state and log paths.
+
+It must not include Erlang cookie values. The word `cookie` may appear in
+posture fields, but the secret value is never printed.
+
+## Attach
+
+```bash
+mix stack_lab.gn_ten.node_lab.attach --node mezzanine_workflow_0 --json
+```
+
+The attach command prints a redacted local shell recipe for the selected
+logical node:
+
+```text
+iex --sname debug_shell_<run> --cookie <redacted_run_cookie> --remsh <node>
+```
+
+This is a development convenience. It is not tenant authority, not product
+authority, and not a production operations model. Current peer-mode runs clean
+up peers before returning, so attach receipts can be `not_attached` while still
+documenting the safe recipe and node target.
+
+## Logs
+
+Node-lab logs are written under generated or temporary paths such as:
+
+```text
+tmp/stack_lab/<run_id>/logs/distributed.log
+```
+
+Each line includes timestamp, profile, logical node id, BEAM node name, stream,
+correlation ref, and a redacted message. Cookies, credentials, raw prompts, raw
+memory, and raw provider payloads are not valid log output.
+
+## Local Inspection
+
+Observer and low-level Erlang tracing are local-only debugging aids:
+
+```elixir
+:observer.start()
+```
+
+Automated proof assertions use owner receipts, StackLab receipts, and AITrace
+exports. They do not depend on Observer, remote PIDs, or ad hoc `:dbg` output.

guides/distributed_debugging.md

@@ -0,0 +1,53 @@
+defmodule Mix.Tasks.StackLab.GnTen.NodeLab.Attach do
+  @moduledoc "Prints a redacted debug attach receipt for a gn-ten node-lab node."
+
+  use Mix.Task
+
+  alias StackLab.GnTenNodeLab.{RunState, Runner}
+
+  @shortdoc "Prints a redacted gn-ten node-lab attach recipe"
+
+  @impl Mix.Task
+  def run(args) do
+    Mix.Task.run("app.start")
+
+    {opts, _argv, invalid} =
+      OptionParser.parse(args,
+        strict: [
+          node: :string,
+          state: :string,
+          json: :boolean
+        ]
+      )
+
+    if invalid != [], do: Mix.raise("invalid options: #{inspect(invalid)}")
+
+    node_id = Keyword.get(opts, :node) || Mix.raise("expected --node <logical_node_id>")
+    runner_opts = [state_path: Keyword.get(opts, :state, RunState.default_path())]
+
+    case Runner.attach(node_id, runner_opts) do
+      {:ok, receipt} ->
+        print(receipt, Keyword.get(opts, :json, false), :info)
+
+      {:error, receipt} ->
+        print(receipt, Keyword.get(opts, :json, false), :error)
+        exit({:shutdown, 1})
+    end
+  end
+
+  defp print(receipt, true, level) do
+    encoded = Jason.encode!(receipt, pretty: true)
+    if level == :error, do: Mix.shell().error(encoded), else: Mix.shell().info(encoded)
+  end
+
+  defp print(receipt, false, :info) do
+    Mix.shell().info("stack_lab.gn_ten.node_lab.attach #{receipt["status"]}")
+    Mix.shell().info("node=#{receipt["node"]}")
+    Mix.shell().info("cookie=<redacted>")
+  end
+
+  defp print(receipt, false, :error) do
+    Mix.shell().error("stack_lab.gn_ten.node_lab.attach failed")
+    Mix.shell().error("failures=#{inspect(receipt["failures"])}")
+  end
+end

lib/mix/tasks/stack_lab.gn_ten.node_lab.attach.ex

@@ -158,12 +158,14 @@ defmodule StackLab.Workspace.MixProject do
         "guides/generalized_stack.md",
         "guides/qc_and_operations.md",
         "guides/code_smell_remediation.md",
+        "guides/distributed_debugging.md",
         "docs/gn_ten_proof_matrix.md",
         "docs/review/gn_ten_batch_review.md",
         "docs/review/shared_library_governed_adapter_review.md",
         "docs/runbooks/up_single.md",
         "docs/runbooks/up_multi.md",
         "docs/runbooks/distributed_node_lab.md",
+        "docs/runbooks/distributed_cleanup.md",
         "docs/runbooks/faults.md",
         "docs/runbooks/tre_lane_acceptance.md",
         "CHANGELOG.md",
@@ -181,6 +183,7 @@ defmodule StackLab.Workspace.MixProject do
           "guides/generalized_stack.md",
           "guides/qc_and_operations.md",
           "guides/code_smell_remediation.md",
+          "guides/distributed_debugging.md",
           "docs/gn_ten_proof_matrix.md",
           "docs/review/gn_ten_batch_review.md",
           "docs/review/shared_library_governed_adapter_review.md"
@@ -189,6 +192,7 @@ defmodule StackLab.Workspace.MixProject do
           "docs/runbooks/up_single.md",
           "docs/runbooks/up_multi.md",
           "docs/runbooks/distributed_node_lab.md",
+          "docs/runbooks/distributed_cleanup.md",
           "docs/runbooks/faults.md",
           "docs/runbooks/tre_lane_acceptance.md"
         ],

mix.exs

@@ -15,7 +15,8 @@ It owns local test-harness mechanics:
 - checked-in topology fixture loading;
 - required app boot probes;
 - owner-defined facade host and `:pg` readiness checks;
-- JSON admin receipts for `preflight`, `up`, `status`, `probe`, and `down`;
+- JSON admin receipts for `preflight`, `up`, `status`, `probe`, `attach`, and
+  `down`;
 - distributed envelope scanning for tenant, authority, trace, idempotency,
   payload-mode, redaction, local-term, raw-payload, cross-tenant, stale-schema,
   and direct-lower-import defects;
@@ -48,6 +49,7 @@ mix stack_lab.gn_ten.node_lab.up \
   --json
 mix stack_lab.gn_ten.node_lab.status --json
 mix stack_lab.gn_ten.node_lab.probe --node fixture_profile_0 --json
+mix stack_lab.gn_ten.node_lab.attach --node fixture_profile_0 --json
 mix stack_lab.gn_ten.node_lab.down --json
 ```
 
@@ -57,6 +59,13 @@ and recorded as an explicit intent, but cross-command peer retention is not a
 v2 Phase 6 claim. A later daemon or release-path controller must own that
 stronger claim.
 
+`status` returns a debug summary with node ids, profile names, node names,
+started apps, owner `:pg` membership, facade readiness, log path, latest
+receipt refs, cleanup posture, and artifact hygiene. `attach` prints a
+redacted local debug-shell recipe for a logical node. It intentionally omits
+the Erlang cookie value and records that Phase 15 does not prove production
+security or cross-command peer retention.
+
 ## Distributed Envelope Scanner
 
 The package exposes `StackLab.GnTenNodeLab.scan_envelope/2` and