Skip to content

PROPOSAL-PANTOGRAPH-INTEGRATION.md

Latest commit

 

History

History
423 lines (318 loc) · 20.2 KB

PROPOSAL-PANTOGRAPH-INTEGRATION.md

File metadata and controls

423 lines (318 loc) · 20.2 KB

Proposal: Pantograph Integration for PumaBot Agent Orchestration

Context

PumaBot needs an agent orchestration framework to power its MCP server and Claude Code orchestrator (see PROPOSAL-MCP-ORCHESTRATION.md). Rather than building a custom orchestrator from scratch, we want to use Pantograph's workflow engine indirectly through its binding crates. PumaBot constructs and runs Pantograph workflows, passing data through input/output nodes and receiving events. Inference, RAG, embeddings, and model management are all consumed via Pantograph's built-in workflow nodes — PumaBot never imports Pantograph's Rust crates directly.

This document covers:

  1. The full integration architecture across PumaBot, Pantograph, and Pumas-Library
  2. Specific asks for the Pantograph team — including cross-language binding crates
  3. Pumas-core integration strategy

Architecture Overview

PumaBot (Elixir/Phoenix OTP)
│
├── PumaBot.MCP.Server            ── Hermes MCP tools (compile, test, docs_lookup)
├── PumaBot.Terminal.Session       ── PTY sessions for Claude Code instances
├── PumaBot.Terminal.Orchestrator  ── High-level task dispatch (Elixir GenServer)
│
├── PumaBot.Workflow.Native        ── Elixir module consuming pantograph-rustler NIFs
├── PumaBot.Workflow.Engine        ── Elixir wrapper (create/edit/run workflows, I/O)
├── PumaBot.Workflow.Callbacks     ── Custom node logic in Elixir (MCP, terminal)
│
└── PumaBotWeb                     ── Phoenix LiveView dashboard
    ├── TerminalDashboardLive      ── Terminal grid with xterm.js
    └── WorkflowDashboardLive      ── Workflow graph visualization

Pantograph (Rust workspace)
│
├── crates/node-engine/           ── Workflow execution engine (library)
├── crates/workflow-nodes/        ── Built-in node implementations (library)
├── crates/inference/             ── LLM inference backends (library)
│
├── crates/pantograph-rustler/    ── NEW: Elixir/Erlang NIFs (like pumas-rustler)
└── crates/pantograph-uniffi/     ── NEW: Cross-language bindings (like pumas-uniffi)
    (PumaBot consumes Pantograph only via these binding crates)

Pumas-Library (Rust workspace) — existing, consumed by Pantograph
│
├── crates/pumas-core/            ── Model management library
├── crates/pumas-rustler/         ── Elixir NIFs
└── crates/pumas-uniffi/          ── Cross-language bindings (mature)

Integration Pattern: Pantograph owns its own bindings

Following the same pattern as Pumas-Library, Pantograph provides its own binding crates. PumaBot (and any other Elixir project) consumes pantograph-rustler NIFs — PumaBot writes zero Rust code.

The bindings expose full workflow CRUD (create, edit, run) plus workflow I/O — the ability to pass data into Pantograph input nodes and receive results from Pantograph output nodes. This is the primary integration surface.

Custom node logic (MCP tools, terminal management) that is PumaBot-specific is implemented in Elixir via a callback mechanism (see Ask 8). RAG, embeddings, model management, inference, and other AI capabilities are provided by Pantograph's own workflow nodes — PumaBot simply builds and runs workflows that use them.

Pumas-Core Integration: Through Pantograph

Pantograph has existing PumaLib node(s) for utilizing pumas-core within workflows (workflow-nodes/src/input/model_provider.rs). These need revision and full implementation (see Ask 4). PumaBot accesses model management capabilities through Pantograph workflows — not via direct pumas-rustler NIFs. This keeps pumas-core as a Pantograph concern and avoids PumaBot needing a separate Rust dependency.

Specific Asks for the Pantograph Team

Ask 1: Headless Library Mode for All Crates

Problem: Pantograph is structured as a Tauri desktop app. The crates have implicit assumptions about running inside Tauri (ProcessSpawner trait, Tauri event channels). All crates need to work as pure headless libraries so the binding crates can wrap them.

What we need:

  1. node-engine crate — Already mostly headless. Needs:

    • Ensure no Tauri dependencies leak in (currently clean — confirmed)
    • Stable public API surface: DemandEngine, WorkflowExecutor, TaskExecutor trait, EventSink trait, orchestration types
    • Add a library feature flag that excludes any desktop-specific functionality

  2. inference crate — Has ProcessSpawner abstraction but needs:

    • The std-process feature flag (using StdProcessSpawner) must work fully without Tauri
    • Ensure llama.cpp sidecar and Ollama backends work in headless mode

  3. workflow-nodes crate — Needs:

    • Feature-gate any nodes that depend on Tauri (ComponentPreview, LinkedInput)
    • Ensure all processing nodes (Inference, Validation, Embedding, ToolExecutor, ToolLoop, ReadFile, WriteFile, VectorDb) work as pure library code
    • The ModelProviderTask needs pumas-core integration (see Ask 4)

Files to modify:

  • crates/node-engine/Cargo.toml — Add library feature
  • crates/inference/Cargo.toml — Verify std-process feature completeness
  • crates/workflow-nodes/Cargo.toml — Feature-gate desktop-only nodes

Ask 2: Custom Node Registration API

Problem: PumaBot needs workflow nodes that don't belong in Pantograph (MCP tool invocation, terminal/PTY session management). Currently all nodes are defined inside workflow-nodes with no mechanism for external consumers to register their own node types.

What we need:

  1. A node registry abstraction in node-engine:
pub struct NodeRegistry {
    executors: HashMap<String, Box<dyn TaskExecutorFactory>>,
}

pub trait TaskExecutorFactory: Send + Sync {
    fn create(&self, task_id: &str, config: serde_json::Value) -> Box<dyn TaskExecutor>;
    fn metadata(&self) -> TaskMetadata;
}

impl NodeRegistry {
    pub fn register<F: TaskExecutorFactory + 'static>(&mut self, node_type: &str, factory: F);
    pub fn create_executor(&self, node_type: &str, task_id: &str, config: Value) -> Option<Box<dyn TaskExecutor>>;
    pub fn available_node_types(&self) -> Vec<TaskMetadata>;
}

  1. Composable registries — Pantograph's built-in nodes + consumer-registered nodes unified into a single executor lookup.

  2. WorkflowExecutor::demand() should use the registry to resolve node types dynamically.

Files to modify:

  • crates/node-engine/src/registry.rs (new)
  • crates/node-engine/src/engine.rs — Use registry in demand path

Ask 3: Complete Orchestration Graph Implementation

Problem: Orchestration graphs (node-engine/src/orchestration/) are not fully implemented / don't work yet. PumaBot needs orchestration for multi-step agent workflows (e.g., compile → check errors → fix → recompile loop). The types and executor skeleton exist but the system is not functional.

Current state (from code review):

  • types.rs — OrchestrationGraph, OrchestrationNode, OrchestrationNodeType (Start, End, Condition, Loop, DataGraph, Merge) — defined
  • executor.rs — OrchestrationExecutor with DataGraphExecutor trait — partially implemented
  • nodes.rs — Node execution logic — exists
  • store.rs — Persistence — exists

What we need:

  1. Complete the orchestration executor so it reliably runs orchestration graphs end-to-end
  2. Condition nodes correctly evaluate boolean context values and branch
  3. Loop nodes correctly iterate with max iteration limits and exit conditions
  4. DataGraph nodes correctly delegate to data graph execution and propagate outputs
  5. Merge nodes correctly combine multiple execution paths
  6. Error handling and recovery — when a data graph fails, route to error handling
  7. Integration tests covering: linear flow, conditional branching, loops with exit conditions, data graph delegation, error paths

Files to modify:

  • crates/node-engine/src/orchestration/executor.rs
  • crates/node-engine/src/orchestration/nodes.rs
  • crates/node-engine/src/orchestration/store.rs

Ask 4: PumaLib Node Revision (pumas-core integration)

Problem: The PumaLib/ModelProvider node(s) (workflow-nodes/src/input/model_provider.rs) have a TODO to integrate pumas-core. Currently the ModelProviderTask just passes through a model name string with no validation or registry lookup. This needs a full revision.

What we need:

  1. ModelProviderTask revised to:

    • Accept a PumasApi handle via workflow context
    • Call pumas_core::list_models() and search_models() for model discovery
    • Validate that requested models exist and resolve their file paths
    • Output full ModelInfo including path, type, family, hashes, security tier

  2. inference crate optionally uses pumas-core for:

    • Model path resolution (given a model name, find the GGUF/safetensors file)
    • Backend selection (pumas-core knows which backends are running: Ollama, llama.cpp)
    • System resource checks before inference (GPU memory, disk space)

  3. Feature-gated behind model-library so Pantograph can still work without it.

Files to modify:

  • crates/workflow-nodes/Cargo.toml — Add pumas-library dependency behind feature flag
  • crates/workflow-nodes/src/input/model_provider.rs — Full revision with pumas-core integration
  • crates/inference/Cargo.toml — Optional pumas-library dependency
  • crates/inference/src/gateway.rs — Model resolution via pumas-core when available

Ask 5: Broadcast/Channel EventSink Implementations

Problem: The EventSink trait (node-engine/src/events.rs) only has NullEventSink and VecEventSink. External consumers need real-time event streaming.

What we need:

  1. BroadcastEventSink — sends to multiple subscribers via tokio::sync::broadcast
  2. CallbackEventSink — calls a user-provided closure per event (critical for NIF bridging)
  3. CompositeEventSink — fans out events to multiple sinks

Files to modify:

  • crates/node-engine/src/events.rs

Ask 6: Workflow Serialization and Builder

Problem: PumaBot needs to store workflows in PostgreSQL and construct them programmatically from Elixir.

What we need:

  1. Confirm WorkflowGraph/OrchestrationGraph serde round-trip cleanly (no information loss).

  2. A workflow builder API for programmatic construction:

WorkflowBuilder::new("wf-1", "Compile and Test")
    .add_node("process", "compile", json!({"command": "mix", "args": ["compile"]}))
    .add_node("process", "test", json!({"command": "mix", "args": ["test"]}))
    .connect("compile", "stdout", "test", "stdin")
    .build()?  // validates graph integrity
  1. Graph validation: no cycles, required ports connected, port type compatibility, exactly one Start node in orchestration graphs.

Files to create:

  • crates/node-engine/src/builder.rs
  • crates/node-engine/src/validation.rs

Ask 7: Process/Command Execution Node

Problem: PumaBot needs to spawn and manage long-running processes (Claude Code CLI, mix compile, mix test). Pantograph doesn't have a general-purpose process execution node.

What we need: A ProcessTask workflow node:

  • Spawns external process with configurable command, args, cwd, env vars
  • Streams stdout/stderr as TaskStream events
  • Supports stdin input (for interactive processes)
  • Handles timeouts and graceful termination
  • Outputs exit code, stdout, stderr

Files to create:

  • crates/workflow-nodes/src/system/process.rs
  • crates/workflow-nodes/src/system/mod.rs

Ask 8: Rustler and UniFFI Binding Crates

Problem: Pantograph currently has no cross-language bindings. For PumaBot (and other projects) to consume Pantograph's workflow engine, it needs the same binding pattern that Pumas-Library provides.

Reference implementation: Pumas-Library's binding crates:

  • pumas-rustler (Pumas-Library/rust/crates/pumas-rustler/) — Rustler NIF crate, cdylib, wraps pumas-library for Elixir
  • pumas-uniffi (Pumas-Library/rust/crates/pumas-uniffi/) — UniFFI crate, cdylib + lib, wraps pumas-library for Python/C#/Swift/Kotlin/Ruby/Go. Uses FfiPumasApi object with Arc<PumasApi>, FFI-safe wrapper types, FfiError conversion from internal errors.

What we need:

pantograph-rustler crate (Elixir/Erlang NIFs)

crates/pantograph-rustler/
├── Cargo.toml       # [lib] crate-type = ["cdylib"], depends on node-engine, workflow-nodes, inference internally
└── src/
    └── lib.rs       # Rustler NIF functions + FFI types

Exposed API surface (NIF functions):

Workflow CRUD:
  create_workflow(id, name) -> ResourceArc<FfiWorkflowExecutor>
  load_workflow_json(json_string) -> ResourceArc<FfiWorkflowExecutor>
  export_workflow_json(executor) -> String
  add_node(executor, node_type, id, config_json) -> :ok
  remove_node(executor, node_id) -> :ok
  connect(executor, source, source_port, target, target_port) -> :ok
  disconnect(executor, edge_id) -> :ok
  update_node_data(executor, node_id, data_json) -> :ok
  validate_graph(executor) -> {:ok, []} | {:error, [validation_errors]}

Workflow I/O (the primary integration surface):
  set_input(executor, node_id, port, value_json) -> :ok
    # Sets data on an input node's port before or during execution
  get_output(executor, node_id, port) -> {:ok, value_json} | {:error, reason}
    # Reads data from an output node's port after execution

Execution:
  execute_workflow(executor, output_node_id, callback_pid) -> :ok
    # Demand-driven: computes everything needed for the requested output
    # Runs on dirty scheduler, streams WorkflowEvent messages to callback_pid
    # callback_pid receives: {:workflow_event, event_map}
  execute_orchestration(executor, orch_json, initial_data, callback_pid) -> :ok

Node Registry:
  list_node_types() -> [%{node_type, category, label, description, inputs, outputs}]
  register_callback_node(executor, node_type, handler_pid) -> :ok
    # When this node executes, it sends {:node_execute, task_id, inputs_json}
    # to handler_pid and waits for {:node_result, task_id, outputs_json} response

Utility:
  version() -> String

Key pattern — Elixir callback nodes: Instead of PumaBot writing custom Rust nodes, pantograph-rustler provides a generic CallbackTask that bridges back to BEAM:

/// When executed in a workflow, sends inputs to an Elixir PID and
/// waits for a response. This allows custom node logic to live in Elixir.
struct ElixirCallbackTask {
    task_id: String,
    handler_pid: LocalPid,
    node_type: String,
    env: OwnedEnv,
}

// On execution:
// 1. Sends {:node_execute, task_id, node_type, inputs_json} to handler_pid
// 2. Waits on a tokio::sync::oneshot for the Elixir process to reply
// 3. Returns the reply as task outputs

This means PumaBot implements only PumaBot-specific node logic in Elixir (MCP tools, terminal sessions). RAG, inference, embeddings, and model management are all handled by Pantograph's built-in workflow nodes.

defmodule PumaBot.Workflow.Callbacks do
  use GenServer

  # Called by pantograph-rustler when a callback node executes
  def handle_info({:node_execute, task_id, "mcp_compile", inputs}, state) do
    result = PumaBot.MCP.Tools.Compile.execute(Jason.decode!(inputs))
    Pantograph.Native.node_result(task_id, Jason.encode!(result))
    {:noreply, state}
  end

  def handle_info({:node_execute, task_id, "terminal_session", inputs}, state) do
    result = PumaBot.Terminal.Session.run_task(inputs)
    Pantograph.Native.node_result(task_id, Jason.encode!(result))
    {:noreply, state}
  end
end

pantograph-uniffi crate (cross-language, optional but recommended)

Same pattern as pumas-uniffi. Exposes:

  • FfiWorkflowEngine object (Arc<WorkflowExecutor>)
  • FfiNodeRegistry for node type listing
  • FfiWorkflowBuilder for programmatic construction
  • FfiError with variants for engine errors
  • FFI-safe wrapper types for WorkflowEvent, GraphNode, GraphEdge, etc.

Files to create:

  • crates/pantograph-rustler/Cargo.toml
  • crates/pantograph-rustler/src/lib.rs
  • crates/pantograph-uniffi/Cargo.toml
  • crates/pantograph-uniffi/src/lib.rs
  • crates/pantograph-uniffi/uniffi.toml
  • crates/pantograph-uniffi/src/bin/uniffi_bindgen.rs

Add to workspace Cargo.toml members:

members = [..., "crates/pantograph-rustler", "crates/pantograph-uniffi"]

PumaBot Side: What We Build (Elixir only, no Rust)

With Pantograph providing pantograph-rustler NIFs, PumaBot only writes Elixir:

1. Elixir NIF wrapper module

defmodule PumaBot.Workflow.Native do
  use Rustler, otp_app: :puma_bot, crate: "pantograph_rustler"
  # NIF stubs auto-generated by Rustler
end

2. Workflow engine wrapper

lib/puma_bot/workflow/
├── engine.ex           # High-level API: create/edit/run workflows, set inputs, read outputs
├── graph.ex            # Elixir-friendly graph construction helpers
├── event_handler.ex    # GenServer receiving events from NIF → PubSub broadcast
├── callbacks.ex        # PumaBot-specific callback node handlers (MCP, terminal)
└── templates.ex        # Pre-built workflow templates (compile-test-fix loop, etc.)

3. Callback nodes (PumaBot-specific logic only)

  • mcp_compile — Calls PumaBot.MCP.Tools.Compile.execute/1
  • mcp_test — Calls PumaBot.MCP.Tools.Test.execute/1
  • terminal_session — Wraps PumaBot.Terminal.Session GenServer

RAG, embeddings, inference, model management, and other AI capabilities are handled by Pantograph's built-in workflow nodes — PumaBot just constructs workflows that use them and passes data via input/output nodes.

Implementation Phases

Phase 1: Headless Library Extraction (Pantograph team) — Ask 1

  • Feature-gate Tauri-dependent code in all 3 crates
  • Verify crates build and test with --no-default-features
  • Validation: cargo test --no-default-features -p node-engine -p workflow-nodes

Phase 2: Core Engine Improvements (Pantograph team) — Asks 2, 3, 5, 6, 7

  • Complete orchestration graph implementation (Ask 3) — this is blocking
  • Implement NodeRegistry with TaskExecutorFactory (Ask 2)
  • Implement BroadcastEventSink, CallbackEventSink, CompositeEventSink (Ask 5)
  • Implement WorkflowBuilder + graph validation (Ask 6)
  • Implement ProcessTask node (Ask 7)
  • Validation: Build orchestration workflow programmatically, register custom node, execute with loops and conditions, receive streamed events

Phase 3: Binding Crates (Pantograph team) — Ask 8

  • Create pantograph-rustler with NIF surface (workflow CRUD, I/O, execution, callback nodes)
  • Create pantograph-uniffi with FFI surface (matching rustler capabilities)
  • Follow pumas-rustler/pumas-uniffi patterns exactly
  • Validation: From Elixir IEx — create workflow, set inputs, execute, read outputs, receive events

Phase 4: PumaLib Node Revision (Pantograph + Pumas-Library teams) — Ask 4

  • Revise ModelProviderTask with full pumas-core integration
  • Optional pumas-core model resolution in inference crate
  • Validation: ModelProvider node lists real models, resolves paths, validates existence

Phase 5: PumaBot Integration (PumaBot team, after Phases 1-3)

  • Add pantograph-rustler as dependency
  • Build Elixir wrapper modules (engine, graph, event_handler, callbacks)
  • Wire workflow events to Phoenix PubSub → LiveView
  • Create workflow templates for MCP orchestration use cases
  • Validation: End-to-end — create compile-test workflow in IEx, pass data to input nodes, observe events in LiveView, read results from output nodes

Verification Plan

  1. Crate tests: Each Pantograph crate passes cargo test with library feature
  2. Orchestration test: Orchestration graph with conditions, loops, and data graph delegation runs correctly
  3. Binding smoke test: pantograph-rustler NIF loads in Elixir, basic round-trip works
  4. Workflow I/O test: Create workflow, set input node values from Elixir, execute, read output node values
  5. Callback test: Register Elixir callback node, execute workflow, verify Elixir handler called and result returned
  6. Event streaming test: Execute workflow, verify events arrive at subscribed Elixir PID
  7. End-to-end: PumaBot creates a "compile-and-test" orchestration workflow with MCP callback nodes, passes project path via input node, results stream to LiveView dashboard