docs(decisions): ADR-0005 — Elixir REST to Zig FFI transport

Proposes OS-process-port (Option A) for Phase 2 of the Elixir REST
port. Rejects NIF (Option B) on fault-isolation grounds — a segfault
in any of 99 cartridge FFIs would kill the entire BEAM VM — and rejects
Unix-socket side-car (Option C) as duplicating work the REST server is
already doing one layer up.

Holds NIF open as a future option once uapi_gnosis_set_handler lands
upstream and the boundary consolidates to a single gnosis handler
rather than 99 per-cartridge surfaces.

Status: Proposed. Consequence and migration-path sections make the
Phase 2 implementation concrete (boj_invoke_cli.zig + Invoker GenServer
pool, 5s timeout default, pool size 4).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: hyperpolymath

docs/decisions/0005-elixir-to-zig-ffi-transport.md

@@ -0,0 +1,157 @@
+<!-- SPDX-License-Identifier: PMPL-1.0-or-later -->
+<!-- Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> -->
+
+# 5. Choose Transport for Elixir REST → Zig Cartridge FFI
+
+Date: 2026-04-18
+
+## Status
+
+Proposed
+
+## Context
+
+`elixir/` (added 2026-04-18) brings up a Cowboy+Plug HTTP listener that
+serves cartridge metadata but returns `501 invocation-not-yet-wired` for
+`POST /cartridge/:name/invoke`. The remaining Phase 2 work is: given a
+cartridge name, a tool name, and a JSON arg blob, call into the
+cartridge's Zig FFI and return the result.
+
+Each cartridge already ships:
+
+- `cartridges/<name>/ffi/<name>_ffi.zig` with C-ABI exports.
+- `cartridges/<name>/ffi/zig-out/lib/lib<name>.so` produced by
+  `zig build` (per `STATE.a2ml §maintenance`: 99/99 built).
+- A documented tool catalogue in `cartridge.json`.
+
+The trunk already has `ffi/zig/src/loader.zig` which handles mount /
+unmount / hash verification of these `.so` files — so cartridge
+**loading** is a solved problem at the Zig layer.
+
+The question this ADR answers is: **how does an Elixir process sitting
+behind the REST router call a function in that loaded `.so`?**
+
+There are three established BEAM ↔ C-ABI transports, each with a
+different safety / latency / complexity trade-off.
+
+### Option A — OS process port (`System.cmd` / `Port.open`)
+
+Spawn a Zig CLI binary per invocation (or a long-lived one pooled in a
+GenServer). The CLI reads `<cartridge> <tool> <json>` on argv/stdin,
+loads the `.so` via `dlopen`, dispatches, emits JSON on stdout.
+
+- **Safety**: a cartridge crash kills the child OS process. BEAM is
+  isolated. Strongest fault-containment.
+- **Latency**: ~1–5 ms per shell-out (fork/exec + dlopen overhead). A
+  pool of long-lived children amortises this.
+- **Complexity**: one new Zig CLI binary; pool management is a
+  standard GenServer pattern.
+- **Licensing / packaging**: the CLI is a small Zig target; no new
+  upstream dependencies.
+
+### Option B — NIF (Native Implemented Function) via `zigler`
+
+Compile the cartridge FFI directly into a BEAM NIF and call it
+in-process.
+
+- **Safety**: a segfault in a NIF crashes the entire BEAM VM. No
+  schedulers survive. Unacceptable for a server that hosts 99
+  cartridges of varying maturity.
+- **Latency**: sub-microsecond. Fastest possible.
+- **Complexity**: `zigler` adds a toolchain; per-cartridge build
+  integration; NIF contract (dirty scheduler, resource management) is
+  a real engineering commitment.
+- **Observation**: ADR-0002 positions BoJ to consume `libzig_api.so`
+  upstream with the `uapi_gnosis_*` unified handler. If that consolidation
+  lands, a single NIF boundary at the gnosis layer (not 99 per-cartridge
+  NIFs) becomes attractive — but that is not today.
+
+### Option C — Local HTTP / gRPC to a side-car
+
+Run a Zig-based side-car process (existing `uapi_gnosis` binary or a
+new one) listening on Unix domain sockets. Elixir POSTs invocation
+requests to it.
+
+- **Safety**: side-car crash doesn't kill BEAM, symmetrical to A.
+- **Latency**: ~100 µs per Unix-socket round-trip; faster than A.
+- **Complexity**: requires a protocol definition (HTTP/JSON or gRPC)
+  between Elixir and the side-car — architecturally this is the same
+  shape the REST server is already solving, just pushed one layer
+  down.
+- **Observation**: this is effectively what
+  `http-capability-gateway` (ADR-0004) already does at tier 2. Pushing
+  the Elixir→Zig bridge into gateway territory would conflate two
+  concerns.
+
+## Decision
+
+**Adopt Option A (OS process port) for Phase 2**, with Option B held
+open for a future phase when the `uapi_gnosis` unified handler is
+live.
+
+Concretely:
+
+1. Add `ffi/zig/src/boj_invoke_cli.zig` — a short Zig binary that
+   takes `<cartridge> <tool> <json-args>` on argv, loads the cartridge
+   `.so` via the existing `loader.zig`, dispatches to the named tool,
+   emits the result as JSON on stdout, and exits with a documented
+   error-code convention (matching the trunk-wide `IsUnbreakable`
+   proof scheme).
+2. Add `elixir/lib/boj_rest/invoker.ex` as a `GenServer`-managed pool
+   of long-lived CLI children (not fork-per-request). The pool size
+   is configurable; default 4.
+3. Replace the 501 in `elixir/lib/boj_rest/router.ex` with a call to
+   `Invoker.dispatch(cartridge, tool, args)`.
+4. Timeouts: 5 s default per invocation, configurable per cartridge
+   via `cartridge.json`.
+
+## Consequences
+
+### Positive
+
+- Fault isolation: any cartridge panic stays in the child OS process.
+  BEAM's supervisor restarts the pool slot; the REST request returns
+  a 502 with the error classification.
+- Incremental path: Phase 2 ships without committing to the NIF
+  toolchain, and the CLI binary is reusable by non-Elixir clients
+  (CI smoke tests, panic-attacker).
+- No new protocol: the CLI uses argv + stdout JSON, no gRPC/Thrift.
+
+### Negative
+
+- Latency floor ~1 ms per invocation. For tools expected to run
+  hundreds of times per second per cartridge, the pool size needs
+  tuning. Not every cartridge has that load profile — most are
+  interactive (MCP tool call rate is one-per-user-keystroke at
+  worst).
+- Double error-code surface: the CLI's exit codes plus the `.so`'s
+  return codes. ADR follow-up: map both into a single
+  `{:ok, body} | {:error, classified}` Elixir tuple in `Invoker`.
+
+### Migration path
+
+When `uapi_gnosis_set_handler` stabilises upstream (ADR-0002 open
+question #1), re-evaluate: keep the OS-port pool and have it link
+`libzig_api.so` internally, or promote to a single NIF boundary at
+the gnosis handler. That decision will merit its own ADR.
+
+## Non-decisions
+
+- **Federation invocation** (cross-node) is out of scope — it belongs
+  to `Boj.Federation` at the trunk ABI, not the Elixir REST layer.
+- **HTTP capability governance** on the REST endpoint itself is
+  ADR-0004's territory (`http-capability-gateway` at tier 2). This ADR
+  covers only what happens *after* the gateway lets the request through.
+- **Cartridge credential isolation** is `Boj.CredentialIsolation`
+  (BJ2) — this ADR does not change that model; the CLI must carry the
+  cartridge's vault partition index through to the FFI.
+
+## References
+
+- ADR-0002 (`align-unified-zig-api-stack`) — open question #1.
+- ADR-0004 (`adopt-http-capability-gateway`) — tier 2 attachment
+  point; independent of this ADR.
+- `elixir/README.adoc` §Phase plan — Phase 2 description.
+- `docs/practice/DOGFOOD-LOG.adoc` 2026-04-18 entry — original
+  "REST API unreachable" finding that motivated the port.
+- `ffi/zig/src/loader.zig` — existing cartridge `.so` loader.