Recorded Auditability and logging for expert and no expert communication for processed flow

[h-network]

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

This is a framework proposal, not code. Current state doesn't have auditability or task tracking, which is essential for agent testing.

Parallel execution agent management.

Task Bus (bus.py) — State Machine

queued → running → completed/failed/aborted/timed_out
Every state transition:

1. SET hcli:task:{id}:state with TTL
2. PUBLISH hcli:task:{id}:notify for subscribers
3. HMAC-SHA256 signed results to prevent spoofing

Dispatcher (dispatcher.py) — Concurrent Execution

• ThreadPoolExecutor with semaphore gating (MAX_CONCURRENT_TASKS)
• Per-chat serialization — tasks from the same chat are serialized via locks to protect session history
• BLPOP on Redis queue with backpressure (semaphore acquired before popping)
• Heartbeat file for health monitoring
• Graceful SIGTERM shutdown with in-flight task drain

Worker (worker.py) — Task Execution with Full Context

• Builds per-task system prompts with session memory + skill injection
• Spawns Claude as a subprocess with start_new_session=True (own process group for clean kill)
• Abort mechanism: Redis pub/sub control channel per task, listener thread that os.killpg() on abort
• Timeout enforcement (TASK_TIMEOUT, default 600s)
• Session chunking to disk when size exceeds MAX_SESSION_BYTES
• Conversation history tracking in Redis with idle sweep

The ANNOUNCEMENT/REPLY Protocol (development process)

This is separate from the runtime — it's how the codebase itself was built. The docs/decisions/ directory contains the actual
artifacts:

• architect-report-redis.md — The architect agent's report after 3 discussion rounds with 4 expert teams
• core-reply.md — Core team's analysis of how the dispatcher split affects their MCP contract
• orchestration-reply.md — Orchestration team's analysis of why single-container is correct
• interface-reply.md, llm-reply.md — Other teams' responses

These show the actual protocol in action:

1. Architect pushes ANNOUNCEMENT.md to each team's branch
2. Each team analyzes independently, pushes REPLY.md
3. Architect synthesizes into architectural decisions (AD-1 through AD-12)
4. Contracts between teams are documented explicitly

For example, the architect-report shows how 4 AI expert teams debated MCP-over-Redis across 3 rounds and ultimately rejected it
(AD-12), with clear reasoning from each team about risks, call chains, and container topology.

---

The Fundamental Gap

┌─────────────────────────────┬────────────────────────────────────────┬────────────────────────────────────────────────────────┐
│ Capability │ NeMo Agent Toolkit │ h-cli │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Parallel execution │ asyncio.gather() on tool calls │ ThreadPoolExecutor + semaphore + per-chat locks │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Task state tracking │ None — fire and forget │ Full state machine with Redis persistence + TTLs │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Result integrity │ Plain string concatenation │ HMAC-SHA256 signed results │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Abort/cancel │ Not supported │ Redis control channel + os.killpg() │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Crash recovery │ Not supported │ Startup scan marks orphaned running → failed │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Agent coordination protocol │ LLM chooses next tool via ReAct prompt │ Git branches + Redis pub/sub + ANNOUNCEMENT/REPLY docs │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Session continuity │ None between agent calls │ Redis session history + disk chunking + idle sweep │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Health monitoring │ None │ Heartbeat file + Docker healthcheck │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Observability │ logger.info() with duration │ Redis counters + TimescaleDB + Grafana │
├─────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Notification │ None │ PUBLISH on every state transition │
└─────────────────────────────┴────────────────────────────────────────┴────────────────────────────────────────────────────────┘

Describe your ideal solution

The Setup

One tmux session. One operator. One architect agent. Eight expert teams — each an independent Claude instance in its own tmux pane.

flowchart LR
    OP["Operator\n(human)"] -->|"direction"| AR["Architect\n(Claude)"]
    AR -->|"tasks via\ngit + Redis"| teams
    teams -->|"branches +\ndone signals"| AR
    AR -->|"results"| OP

    subgraph teams ["Expert Teams — each a separate Claude instance"]
        T1["orchestration"] ~~~ T2["interface"] ~~~ T3["core"] ~~~ T4["llm"]
        T5["monitor"] ~~~ T6["hssh"] ~~~ T7["knowledge"] ~~~ T8["security"]
    end

    style OP fill:#c62828,color:#fff,stroke:#b71c1c
    style AR fill:#1565c0,color:#fff,stroke:#0d47a1
    style teams fill:#1a1a2e,color:#e0e0e0,stroke:#4a4a6a

Loading

How It Works

The operator tells the architect what to build. The architect breaks it into scoped tasks, writes an ANNOUNCEMENT.md for each team, pushes branches, and notifies teams via Redis. Each expert reads its task, implements it in its own directory, pushes a branch with a REPLY.md, and signals done. The architect reviews, merges, and reports back.

No expert ever talks to another expert. All coordination flows through the architect.

sequenceDiagram
    participant O as Operator
    participant A as Architect
    participant R as Redis
    participant E1 as Core Team
    participant E2 as Interface Team

    O->>A: "Add output sanitization"
    A->>A: Create branches with ANNOUNCEMENT.md
    A->>R: PUBLISH round "core interface"
    A->>R: PUBLISH msg:core "pull branch, check ANNOUNCEMENT.md"
    A->>R: PUBLISH msg:interface "pull branch, check ANNOUNCEMENT.md"
    R->>E1: Task notification
    R->>E2: Task notification

    par Parallel execution
        E1->>E1: Read task, implement, push branch
        E2->>E2: Read task, implement, push branch
    end

    E1->>R: PUBLISH done "core"
    E2->>R: PUBLISH done "interface"
    R->>A: "All teams done: core interface"
    A->>A: Review branches, merge to main
    A->>O: Done — here's what changed

Loading

The tmux Layout

┌─────────────────────────────────────────────────────┐
│ h-cli-development:architect    ← Architect agent    │
├─────────────────────────────────────────────────────┤
│ h-cli-development:orchestration                     │
│ h-cli-development:interface                         │
│ h-cli-development:core                              │
│ h-cli-development:llm          ← Expert agents      │
│ h-cli-development:monitor        (one per pane)     │
│ h-cli-development:hssh                              │
│ h-cli-development:knowledge                         │
│ h-cli-development:security                          │
├─────────────────────────────────────────────────────┤
│ h-cli-development:redis        ← Conductor          │
└─────────────────────────────────────────────────────┘

Each pane is a separate Claude Code instance. They share only the git repo and a Redis instance. The conductor (a small shell script on the Redis pane) tracks which teams have signaled done and notifies the architect when a round is complete.

The Rules

Strict conventions prevent chaos:

• Experts stay in scope. Each team owns one directory. Edits outside it are forbidden unless the task explicitly allows it.
• Communication is async. Tasks go out via git branches + Redis. Results come back via git branches + Redis. No shared state, no direct messaging.
• Rounds are atomic. The architect declares a round, all teams execute in parallel, all signal done, then the architect merges. No partial merges mid-round.
• Main stays clean. No communication artifacts (ANNOUNCEMENT.md, REPLY.md) reach the main branch. The architect strips them during merge.
• Pull before work. Every team pulls main before starting its task branch — prevents divergence.
• Push before signal. A "done" signal without a pushed branch is useless. Push first, signal second.

What This Means

The entire codebase — 12 Docker services, 45 security hardening items, two network topologies, an Asimov-inspired AI firewall, session management, skill teaching, vector memory, and monitoring — was built through this process. One human steering, AI agents executing in parallel, strict protocols preventing them from stepping on each other.

The operator never wrote code. The architect never read implementation details. The experts never coordinated directly. Each role stayed in its lane, and the system grew commit by commit.

(private)670+ commits. Zero merge conflicts from scope violations (after the first week).

Additional context

https://github.com/h-network/h-cli/blob/main/docs/H-CLI-DEVELOPMENT-EXPLAINED.md

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[willkill07]

@h-network how exactly is h-cli related to NeMo Agent Toolkit when you clearly state things like:

• Each pane is a separate Claude Code instance. They share only the git repo and a Redis instance. The conductor (a small shell script on the Redis pane) tracks which teams have signaled done and notifies the architect when a round is complete.
• One human steering, AI agents executing in parallel, strict protocols preventing them from stepping on each other.

NeMo Agent Toolkit is a library for writing, observing, profiling, and optimizing agents that are written in one of several supported frameworks: https://docs.nvidia.com/nemo/agent-toolkit/latest/components/integrations/frameworks.html

Can you please update this issue to be less of a Claude Code dump and a clear proposal for what you are explicitly proposing?

[h-network]

@willkill07 The issue isn't about claude code or h-cli in particular. It just happens that the documentation is there. And h-cli was built with this "framework".
The issue is more a framework, communication and traceability description/proposal. The bus.py, dispatcher.py, worker.py, etc logic.

The framework describes how the agents communicate, keep track of their decisions and all the commands they have done during the process. This would mean, you can have 1 repo, and different LLMs running in seperate "windows", and you could becnhmark and track better than current state.

The framework solves 1785, 1645, if i'm not mistaken
My apologies if the "claude code" threw you off, this could be any python script interacting with vllm/ollama or the LLM itself.

[willkill07]

The framework solves 1785, 1645, if i'm not mistaken

• There is already a PR open that solves 1785 by the contributor who filed that issue.
• This proposal does not address 1645 in any way, as the UI is orthogonal to the execution.

The issue is more a framework, communication and traceability description/proposal. The bus.py, dispatcher.py, worker.py, etc logic.

If you could please reword your issue in a clear and concise way then to reflect your actual proposed feature without extra cruft, it would be greatly appreciated. Specifically, how does the architecture of the toolkit change. "Please provide a clear description of problem this feature solves" really needs to be addressed.

[h-network]

Problem this feature solves:

NeMo Agent Toolkit ships nvidia_nat_redis for vector memory, but Redis capabilities beyond memory search are untapped. Agent executions are currently fire-and-forget, no state tracking, no external abort, no crash recovery, no
session continuity between invocations. These are production requirements for teams deploying agents as long-running services.

Redis natively supports all of these patterns with infrastructure NeMo users already run.

Proposed solution:

Extend nvidia_nat_redis (or create nvidia_nat_redis_orchestration) to provide:

1. Task lifecycle state machine
   Redis strings with TTL for execution state (running → completed | failed | timed_out | aborted). Pub/Sub notifications on state transitions. SCAN-based crash recovery to detect orphaned running states on startup.

2. External abort
   Redis Pub/Sub control channel per task. External callers publish abort, middleware cancels execution. Enables UI kill buttons, API cancellation, inter-agent coordination.

3. Session continuity
   Redis lists for conversation history with TTL-based expiry. Size-based rotation when sessions exceed configurable limits. Working memory (Redis, hot) with automatic compaction to long-term storage.

4. Execution metrics
   Redis hashes with daily counters (tasks, tokens, cost, errors). Complements existing profiler/observability middleware with persistent aggregate stats.

All implemented as middleware (DynamicFunctionMiddleware) + a Redis-backed state provider. Fully opt-in , no changes to existing agents or the current nvidia_nat_redis memory plugin.

middleware:
orchestration:
_type: redis_orchestration
redis_url: redis://localhost:6379
enable_state_tracking: true
enable_abort: true
enable_session_continuity: true
session_max_bytes: 102400
session_ttl: 28800
state_ttl: 3600

Architecture impact:

Layers on top of existing middleware and builder infrastructure. Does not modify existing agents, executors, or the nvidia_nat_redis memory plugin. Teams that don't need orchestration are unaffected.

Additional context:

This complements #1635 (Redis Agent Memory Server) and #1786 (sequential executor tracing). Memory Server adds advanced memory management; this adds execution infrastructure using the same Redis dependency. Together they make Redis a complete production backend for NeMo deployments.

[h-network]

Stress tested with 100 different agents, per agent tracing and memory, and the kill-switch.

https://github.com/h-network/h-NeMo-Agent-Toolkit/tree/h-dev

[h-network]

Used the wrong github account for the other issue, my apologies (#1811)

[willkill07]

nvidia_nat_redis ships more than just a memory interface. There's also an object store implementation. So to me, any additions here for orchestration would be best-fit under the nvidia_nat_redis package.

Alternatively, this could be shipped as a standalone orchestrator plugin that wouldn't have to be directly upstreamed into NeMo Agent Toolkit.

We’re currently improving our documentation and the visibility of external/community plugins, and we’d be very happy to work with you to make sure any community plugin created is:

• easy to install (should always be just pip install)
• well documented within the NeMo Agent Toolkit repository and documentation site (clear table/list of community plugins)

Would you be open to this approach? (for this along with your other issue -- #1811)

[h-network]

Hi Will,

Thank you for the response.

I have no issue with this approach. In fact, I would prefer for the orchestration functionality to be included in the main NAT Redis module, and I think it would also be better for the community if that functionality is maintained by NVIDIA directly.

Regarding the other issue (#1811), I completely understand it being categorized as a community plugin, and I agree that this would be the better outcome since it is a less impactful module.

So, to summarize:
• orchestration in the current Redis module
• #1811 as a community plugin that I maintain

Would that work for you?

[willkill07]

I would prefer for the orchestration functionality to be included in the main NAT Redis module

We need a comprehensive proposal for such a feature that would require review and sign-off. An issue proposing the feature with a matching PR is not enough. This is why I'm first suggesting an external plugin that -- provided enough usage/adoption -- could then be integrated into the toolkit.

[h-network]

Makes sense.
Since I documented this in detail during my own implementation,
is there a proposal template or format I should follow for the comprehensive review?"