|
| 1 | +# UAT Report — 2026-03-20 |
| 2 | + |
| 3 | +## Session Summary |
| 4 | +- **Range**: Full 27-VM cyberrange (6 networks, perimeter→office→datacenter→management→vault) |
| 5 | +- **Persona**: CTF Player transitioning to Pentester critique |
| 6 | +- **Objective**: Validate logs feature (v0.13.0), tunnel connectivity, auto-routing, and MCP tool UX |
| 7 | +- **Outcome**: Logs feature works end-to-end. Single-hop pivot to office network confirmed. Found 2 bugs (neli ACK parsing, missing disconnect log) and 1 MCP error code issue. |
| 8 | + |
| 9 | +## Pontoon MCP — Findings |
| 10 | + |
| 11 | +### Tool Completeness |
| 12 | +Sufficient for range lifecycle and VM interaction. `range_up`, `vm_exec`, `vm_cp`, `vm_exec_bg`, `vm_tail` covered all needs. |
| 13 | + |
| 14 | +### UX & Discoverability |
| 15 | +Tool names are clear and consistent. `vm_port_probe` runs from the host, not inside the VM — this caused initial confusion (probed office network from host, got "closed" because the host isn't routed through the tunnel). Not a bug, but worth documenting. |
| 16 | + |
| 17 | +### Specific Issues |
| 18 | +| Severity | Finding | Suggested Fix | |
| 19 | +|----------|---------|---------------| |
| 20 | +| 🟢 Suggestion | `vm_port_probe` description doesn't clarify it runs from the host perspective, not inside the VM | Add "(from the host)" to description | |
| 21 | + |
| 22 | +## Wallhack MCP — Findings |
| 23 | + |
| 24 | +### Tool Completeness |
| 25 | +All management operations covered: info, peers, routes, stats, logs, connect, listen, disconnect, route_add, route_del, hint_set, hint_set_auto, shutdown. The new `logs` tool fills the observability gap — can now diagnose peer connections, route installation, and errors without SSH access. |
| 26 | + |
| 27 | +### Logging Quality |
| 28 | +- **Startup**: clear and informative (version, capabilities, listen addr, PSK warning) |
| 29 | +- **Peer lifecycle**: connect events logged with name, addr, TUN interface, route count |
| 30 | +- **Missing**: peer disconnect events not logged at info level — only connects appear in the ring buffer |
| 31 | +- **Verbosity**: right level for info — no data-plane noise, just lifecycle and errors |
| 32 | + |
| 33 | +### Terminology Consistency |
| 34 | +No issues found in this session. Role names (entry/exit), peer/route terminology all consistent. |
| 35 | + |
| 36 | +### UX & Workflow |
| 37 | +- Auto-routing on connect is excellent — office network was immediately reachable after gateway-perimeter connected |
| 38 | +- Route warning when adding a CIDR not advertised by the peer is helpful and actionable |
| 39 | +- `hint_set` tool name is confusing for first-time users — "hint" is protocol jargon. The REPL uses `role` which is much clearer. MCP should match. |
| 40 | + |
| 41 | +### Error Messages |
| 42 | +- "peer not found: nonexistent" — clear message, but wrapped as MCP internal_error (-32603) instead of invalid_params (-32602). All daemon `ErrorResponse` messages are mapped to internal_error regardless of cause. |
| 43 | + |
| 44 | +### Specific Issues |
| 45 | +| Severity | Finding | Suggested Fix | |
| 46 | +|----------|---------|---------------| |
| 47 | +| 🔴 Critical | neli 0.7 ACK parsing broken — `recv_netlink_ack` returns "Unexpected payload in ACK" for route add/remove OS operations. Kernel routes silently fail to install for manually added routes. Auto-routes may also be affected. | Fix the raw buffer parsing in `recv_netlink_ack` — the 0.7 `Buffer` payload may include the full nlmsghdr, not just the error code. Compare with neli 0.6 behavior. | |
| 48 | +| 🟡 Minor | Peer disconnect events not captured in log ring buffer — connects show "Peer connected: ..." but disconnects produce no corresponding info-level log line | Add `info!("Peer disconnected: name={name}")` in the disconnect path | |
| 49 | +| 🟡 Minor | MCP error codes: all daemon errors mapped to `-32603` (internal_error) regardless of cause. "peer not found" and "invalid CIDR" should be `-32602` (invalid_params) | Map `ErrorResponse` codes to appropriate MCP error types in `convert::format_response` | |
| 50 | +| 🟡 Minor | `hint_set` / `hint_set_auto` MCP tool names don't match REPL (`role` command). Per interface parity, MCP should expose `role` not `hint` | Rename MCP tools or add `role` as the primary name | |
| 51 | +| 🟢 Suggestion | `logs` output could include timestamps for each line (currently just `level: message`) | Prepend compact timestamp to each buffered line | |
| 52 | + |
| 53 | +## Cross-Cutting Observations |
| 54 | + |
| 55 | +- **Hot-patching workflow works but is manual** — kill old process, cp binary, restart with correct flags. The initrd binary is v0.11.0 while we're testing v0.13.0. A `pontoon build` + `range_up` cycle would be cleaner but slower. |
| 56 | +- **CLI changed between initrd (subcommand: `wallhack entry`) and current (flag: `wallhack --role entry`)** — the clusterfuck merge changed the CLI surface. Help text says "Use a subcommand" but they're actually flags now. This will confuse users who read old docs. |
| 57 | +- **The logs feature delivered exactly what was needed** — during this session, `logs` was the primary diagnostic tool for understanding peer connections, route installation, and error conditions. Without it, debugging would have required SSH + reading stderr. |
| 58 | + |
| 59 | +## Session Transcript (Condensed) |
| 60 | + |
| 61 | +▶ Brought range up (27 VMs) |
| 62 | + → All VMs ready. Wallhack daemon auto-started on attacker (entry) and gateway-perimeter (exit). |
| 63 | + |
| 64 | +▶ `wallhack info` via MCP |
| 65 | + → Connected to attacker daemon (v0.11.0 — stale initrd binary) |
| 66 | + |
| 67 | +▶ `wallhack logs` on stale binary |
| 68 | + → "empty request" — logs feature doesn't exist in v0.11.0. Expected. |
| 69 | + |
| 70 | +▶ Hot-patched attacker and gateway-perimeter with v0.13.0 binary |
| 71 | + → Restarted both daemons with `--role entry/exit` flags |
| 72 | + |
| 73 | +▶ `wallhack logs` on v0.13.0 |
| 74 | + → Returns startup lines, peer connect events, route installation. Feature works. |
| 75 | + |
| 76 | +▶ Curled intranet (10.99.2.80) from attacker through tunnel |
| 77 | + → Page returned with DB creds. Full tunnel chain working. |
| 78 | + |
| 79 | +▶ `route_add` with nonexistent peer |
| 80 | + → "peer not found: nonexistent" — clear message, wrong MCP error code (-32603 vs -32602) |
| 81 | + |
| 82 | +▶ `route_add` for CIDR not advertised by peer |
| 83 | + → "OK" with warning about unreachable destination. Excellent UX. |
| 84 | + → Bug: OS route operations fail with "Unexpected payload in ACK" (neli 0.7 parsing) |
| 85 | + |
| 86 | +▶ Disconnected web-filter peer |
| 87 | + → Disconnect succeeded, peer removed from list, but no log line captured for the event |
| 88 | + |
| 89 | +▶ Tunnel still functional after disconnect — office network still reachable |
0 commit comments