[lfx-mentorship-2026-June-August] Support multi-AgentCube Capability

[LiZhenCheng9527]

• Description: At present, AgentCube launches a single agent for both Code Interpretation and agent runtime. However, we have now entered the era of multi-agent collaboration. Consequently, AgentCube aims to support multi-AgentCube orchestration, enabling multiple agents to collaborate on a single task, with AgentCube providing unified management of the lifecycle of these agents.
• Expected Outcome:
  1. Proposal
  2. Code(Including Unit test and E2E test)
  3. User guide and another related docs.
• Recommended Skills:
  • Go, Kubernetes, agent
• Mentor(s):
  • Zhonghu xu(@hzxuzhonghu, zhhxu2011@gmail.com)

[Pasta-coder]

i was waiting for this project to get approved in cncf mentoring repo . i am very excited to apply and contribute to this project .

@hzxuzhonghu are there any prerequisites to this project ? Or should i just explain my plan for this project ? Or should i directly send the application ?

thankyou

[Abhinav-kodes]

Hi @hzxuzhonghu
I've been studying the AgentCube codebase (recently had PR #311 merged)
I'm very interested in this mentorship and I'm working through the design questions around the multi-agent CRD model and session lifecycle.
Would it be appropriate to share a draft proposal here for early feedback?

Thank You

[Abhinav-kodes]

Multi-Agent Runtime Support (Draft Design Proposal)

Hi @hzxuzhonghu,

Since the application window is approaching, I'll go ahead and share my draft design here for early feedback.
happy to revise based on your input.

I've been studying the codebase deeply and have mapped out a concrete implementation plan. I'm planning to dedicate full-time hours to this mentorship, so I've scoped the proposal ambitiously across 12 weeks.

The existing example/pcap-analyzer/ already demonstrates a planner + code-interpreter pattern, MultiAgentRuntime should support exactly this use case natively with unified lifecycle management.

---

Core Insight

The existing createSandbox() in handlers.go is already fully transactional, it handles placeholder storage, K8s resource creation, readiness probing, rollback, and store finalization atomically. The multi-agent implementation should sit above this layer, not replace it.

---

Proposed Architecture

MultiAgentRuntime CRD
        ↓  orchestrates
AgentRuntime CRDs  (one per role, existing CRDs)
        ↓  instantiates via existing createSandbox()
Sandbox / Pod

Multi-agent orchestration is a pure composition layer (the entire existing sandbox lifecycle stays untouched).

---

New CRD: MultiAgentRuntime

apiVersion: runtime.agentcube.volcano.sh/v1alpha1
kind: MultiAgentRuntime
metadata:
  name: research-team
  namespace: default
spec:
  startupPolicy: Atomic          # Atomic (default) or BestEffort
  roles:
    - name: planner
      runtimeRef: planner-agent  # references existing AgentRuntime CRD
      isCoordinator: true        # only coordinator entrypoints exposed externally
      warmPoolSize: 2            # pre-warm 2 pods for this role
    - name: researcher
      runtimeRef: researcher-agent
      warmPoolSize: 3
      dependencies: [planner]    # waits for planner to be ready; planner endpoint injected as env var
    - name: coder
      runtimeRef: coder-agent
      dependencies: [planner, researcher]
  sessionTimeout: 15m
  maxSessionDuration: 8h

Key design decisions:

• runtimeRef points to an existing AgentRuntime CRD --> no inline pod spec duplication, full reuse of existing security/resource config
• Flat roles[] list instead of hardcoded orchestrator/workers split --> supports DAGs, swarms, peer topologies
• dependencies[] enables topological startup ordering --> dependency endpoints injected into dependent pods as env vars at boot time
• warmPoolSize per role mirrors the existing CodeInterpreter warm pool pattern (SandboxTemplate + SandboxWarmPool) --> group creation latency drops from minutes to seconds
• startupPolicy: Atomic (default) --> any failure rolls back all roles; BestEffort --> coordinator must succeed, workers are optional
• Exactly one isCoordinator: true --> validated at request time; only its entrypoints are returned externally
• Workers communicate over cluster-internal DNS directly --> the external Router is never involved in inter-agent traffic

---

Store Layout

Each sandbox is stored independently (GC and routing unchanged), plus a lightweight group manifest:

sandbox:{sessionID-planner}    --> SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "planner" }
sandbox:{sessionID-researcher} --> SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "researcher" }
sandbox:{sessionID-coder}      --> SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "coder" }

agentgroup:{grp-xxx} → {
    roles: [
        { name: "planner",    sessionID: "...", endpoint: "10.0.0.4:8080", status: "ready" },
        { name: "researcher", sessionID: "...", endpoint: "10.0.0.5:8080", status: "ready" },
        { name: "coder",      sessionID: "...", endpoint: "10.0.0.6:8080", status: "failed" }  <-- BestEffort only
    ]
}

The router externally tracks only the coordinator's sessionID. Worker pod IPs are stored in the group manifest and used for internal endpoint discovery and dependency injection.

---

Core New Function: createSandboxGroup()

func (s *Server) createSandboxGroup(ctx context.Context, ...) (*types.CreateAgentGroupResponse, error) {
    groupSessionID := "grp-" + uuid.New().String()
    var created []createdRole

    needGroupRollback := true
    defer func() {
        if !needGroupRollback { return }
        for _, c := range created {
            s.rollbackSandboxCreation(dynamicClient, c.sandbox, nil, c.sessionID) // called unchanged
        }
    }()

    // Topological sort by dependencies (cycle detection returns descriptive error)
    orderedRoles, err := topoSort(mar.Spec.Roles)
    if err != nil {
        return nil, err // e.g. "dependency cycle: researcher --> coder --> researcher"
    }

    for _, role := range orderedRoles {
        // buildSandboxByAgentRuntime is called unchanged
        sandbox, sandboxEntry, err := buildSandboxByAgentRuntime(mar.Namespace, role.RuntimeRef, s.informers)
        sandboxEntry.GroupSessionID = groupSessionID
        sandboxEntry.Role = role.Name

        // Inject dependency pod IPs as env vars before pod creation
        if len(role.Dependencies) > 0 {
            injectDependencyEndpoints(&sandbox.Spec.PodTemplate, role.Dependencies, created)
        }

        resultChan := s.sandboxController.WatchSandboxOnce(ctx, sandbox.Namespace, sandbox.Name)
        defer s.sandboxController.UnWatchSandbox(sandbox.Namespace, sandbox.Name)

        // createSandbox is called unchanged
        resp, err := s.createSandbox(ctx, dynamicClient, sandbox, nil, sandboxEntry, resultChan)
        if err != nil {
            if mar.Spec.StartupPolicy == BestEffort && !role.IsCoordinator {
                recordRoleFailure(groupSessionID, role.Name)
                continue
            }
            return nil, fmt.Errorf("role %s: %w", role.Name, err)
        }
        created = append(created, createdRole{name: role.Name, resp: resp})
    }

    s.storeClient.SaveAgentGroup(ctx, groupSessionID, buildGroupManifest(created))
    needGroupRollback = false
    return response, nil
}

buildSandboxByAgentRuntime, createSandbox, and rollbackSandboxCreation are called without any modification.

---

Complete File Change Map

New Files

File	Description
pkg/apis/runtime/v1alpha1/multiagentruntime_types.go	CRD: MultiAgentRuntime, AgentRoleSpec, MultiAgentRuntimeStatus with kubebuilder markers
pkg/workloadmanager/multiagent_controller.go	MultiAgentRuntimeReconciler -- self-healing (wired in Phase 4)
pkg/workloadmanager/multiagent_controller_test.go	Reconciler unit tests
sdk-python/agentcube/multi_agent.py	MultiAgentRuntimeClient -- create group, get topology, delete
sdk-python/examples/multi_agent_usage.py	End-to-end usage example
test/e2e/multi_agent_runtime.yaml	E2E fixtures: 3 AgentRuntime CRDs + 1 MultiAgentRuntime
docs/design/multi-agent-design.md	Architecture design doc
manifests/charts/base/crds/runtime.agentcube.volcano.sh_multiagentruntimes.yaml	Auto-generated by make gen-crd (called via make generate) -- committed

Modified Files

File	Change
pkg/apis/runtime/v1alpha1/register.go	Add MultiAgentRuntimeKind, MultiAgentRuntimeListKind, MultiAgentRuntimeGroupVersionKind (mirror existing AgentRuntime block)
pkg/apis/runtime/v1alpha1/zz_generated.deepcopy.go	Regenerated by make generate (controller-gen object)
pkg/common/types/types.go	Add MultiAgentRuntimeKind constant
pkg/common/types/sandbox.go	Add GroupSessionID, Role to SandboxInfo; add AgentGroupManifest, group request/response types
pkg/api/errors.go	Add ErrMultiAgentRuntimeNotFound + multiAgentRuntimeResource in workloadResource() switch
pkg/workloadmanager/informers.go	Add MultiAgentRuntimeGVR + informer wiring + cache sync
pkg/workloadmanager/workload_builder.go	Add GroupSessionID, Role to sandboxEntry struct (package-private, same package)
pkg/workloadmanager/sandbox_helper.go	Propagate group fields in buildSandboxPlaceHolder() + buildSandboxInfo()
pkg/workloadmanager/handlers.go	Add handleMultiAgentRuntimeCreate, createSandboxGroup, handleDeleteAgentGroup, handleGetGroupTopology
pkg/workloadmanager/handlers_test.go	Add group creation + rollback test cases
pkg/workloadmanager/server.go	Add 3 routes: POST /v1/multi-agent-runtime, DELETE /v1/multi-agent-runtime/sessions/:groupSessionId, GET /v1/multi-agent-runtime/groups/:groupSessionId/topology
pkg/workloadmanager/garbage_collection.go	Group manifest cleanup when last member sandbox is GC'd
pkg/store/interface.go	Add SaveAgentGroup, GetAgentGroup, DeleteAgentGroup, UpdateAgentGroupRoleStatus to interface
pkg/store/store_redis.go	Implement all 4 group methods
pkg/store/store_redis_test.go	Group CRUD test cases
pkg/store/store_valkey.go	Implement all 4 group methods (Valkey backend)
pkg/store/store_valkey_test.go	Group CRUD test cases (parallel to Redis test file)
pkg/router/session_manager.go	Add MultiAgentRuntimeKind case in endpoint switch -- routes cold-path requests to /v1/multi-agent-runtime
cmd/workload-manager/main.go	Phase 1: wire HTTP routes only; Phase 4: additionally wire MultiAgentRuntimeReconciler into setupControllers()
sdk-python/agentcube/__init__.py	Export MultiAgentRuntimeClient
test/e2e/e2e_test.go	Add TestMultiAgentRuntimeCreate, TestMultiAgentRuntimeRollback

---

Planned Deliverables (Phased Across 12 Weeks)

Phase 1 -- Core Foundation (Weeks 1-4)

• MultiAgentRuntime CRD + register.go + run make gen-crd and make generate (deepcopy + CRD YAML manifest)
• createSandboxGroup() with Atomic rollback only
• Store: all 4 group methods on both Redis and Valkey backends + tests
• Router cold-path case for MultiAgentRuntimeKind
• cmd/workload-manager/main.go --> HTTP route wiring only (no controller yet)
• GC group manifest cleanup
• Unit tests: atomic rollback on partial failure, store CRUD, coordinator validation
• E2E test on kind cluster
• User guide: YAML example + kubectl workflow

Phase 2 -- Warm Pools Per Role (Weeks 5–6)

• Add warmPoolSize to AgentRoleSpec
• Controller creates SandboxTemplate + SandboxWarmPool per role (exact mirror of CodeInterpreterReconciler)
• Group creation uses SandboxClaim for warm roles -- startup drops from minutes to seconds
• Unit tests for warm vs cold creation paths

Phase 3 -- DAG Startup + Topology (Weeks 7–8)

• Add dependencies []string to AgentRoleSpec
• Topological sort in createSandboxGroup() with cycle detection
• Dependency endpoint injection into pod env vars before creation
• GET /topology endpoint -- returns all role internal endpoints for coordinator discovery
• Python SDK: get_topology() method on MultiAgentRuntimeClient

Phase 4 -- StartupPolicy + Self-Healing (Weeks 9–11)

• startupPolicy: BestEffort -- failed workers recorded in manifest, don't trigger rollback
• MultiAgentRuntimeReconciler -- Atomic: tears down group on role crash; BestEffort: restarts failed role
• MultiAgentRuntimeStatus with per-role conditions + ActiveGroups int32
• cmd/workload-manager/main.go -- additionally wire MultiAgentRuntimeReconciler into setupControllers()

Phase 5 -- Observability + Docs (Week 12)

• Prometheus metrics: agentcube_group_creation_duration_seconds, agentcube_group_role_failures_total, agentcube_active_groups
• docs/design/multi-agent-design.md -- architecture, API reference, troubleshooting guide
• Final PR polish + review cycle iterations

---

What Stays Completely Unchanged

• createSandbox() -- called as-is
• rollbackSandboxCreation() -- called as-is
• All existing AgentRuntime / CodeInterpreter flows
• All existing store methods
• GC idle/expiry logic for single-agent sandboxes
• Router proxy logic

---

I'm happy to adjust scope or prioritization based on your feedback. One specific question: does the /groups/:id/topology endpoint align with your planned approach for inter-agent communication, or do you have a different mechanism in mind?

Thank you for considering my application.

--- Abhinav (@Abhinav-kodes)

[ShubhamTaple]

Multi-Agent Runtime Support (Design Proposal)

Hi @hzxuzhonghu,

I'd like to share my design proposal for the Multi-Agent Runtime Support mentorship task ahead of the application deadline. Happy to revise based on your feedback.

I've been studying the AgentCube codebase and have mapped out a concrete implementation plan across 12 weeks. I'm committed to full-time hours for the duration of the mentorship.

One note on my background that I think is directly relevant here: I have 5 merged pull requests into the Redis upstream codebase (#14987, #15042, #15094, #15095, #15151). The store backends for this feature (store_redis.go, store_valkey.go) are an area where I can contribute with genuine depth, not just surface-level familiarity.

---

Core Insight

The existing createSandbox() in handlers.go is already a complete, self-contained transaction — it handles placeholder storage, Kubernetes resource creation, readiness probing, rollback, and store finalization atomically. MultiAgentRuntime should be a composition layer above this, not a replacement. Every existing flow stays untouched.

A second observation: the existing SandboxTemplate + SandboxWarmPool pattern in the CodeInterpreter reconciler already solves per-role warm pool pre-warming. The multi-agent implementation should mirror this pattern per role rather than inventing a new one.

---

Proposed Architecture

MultiAgentRuntime CRD
        ↓  orchestrates
AgentRuntime CRDs  (one per role — existing CRDs, unchanged)
        ↓  instantiates via existing createSandbox()
Sandbox / Pod

Multi-agent orchestration is a pure composition layer. The entire existing sandbox lifecycle is preserved.

---

New CRD: MultiAgentRuntime

apiVersion: runtime.agentcube.volcano.sh/v1alpha1
kind: MultiAgentRuntime
metadata:
  name: research-team
  namespace: default
spec:
  startupPolicy: Atomic          # Atomic (default) or BestEffort
  roles:
    - name: planner
      runtimeRef: planner-agent  # references existing AgentRuntime CRD
      isCoordinator: true        # only coordinator entrypoints exposed externally
      warmPoolSize: 2
    - name: researcher
      runtimeRef: researcher-agent
      warmPoolSize: 3
      dependencies: [planner]    # dependency endpoints injected as env vars at boot
    - name: coder
      runtimeRef: coder-agent
      dependencies: [planner, researcher]
  sessionTimeout: 15m
  maxSessionDuration: 8h

Key design decisions:

• runtimeRef references an existing AgentRuntime CRD — no inline pod spec duplication, full reuse of existing security and resource configuration
• Flat roles[] list rather than a hardcoded orchestrator/workers split — supports DAGs, swarms, and peer topologies
• dependencies[] drives topological startup ordering — dependency pod IPs injected as env vars before creation
• warmPoolSize per role mirrors the existing CodeInterpreter warm pool pattern exactly — group creation latency drops from minutes to seconds
• startupPolicy: Atomic (default) — any role failure triggers full rollback; BestEffort — coordinator must succeed, worker failures are recorded but don't block
• Exactly one isCoordinator: true required — validated at admission time; only coordinator entrypoints are returned to the caller
• Workers communicate over cluster-internal DNS — the external Router is never in the inter-agent data path

---

Store Layout

Each sandbox is stored independently (GC and routing unchanged). A lightweight group manifest sits alongside:

sandbox:{sessionID-planner}    → SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "planner" }
sandbox:{sessionID-researcher} → SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "researcher" }
sandbox:{sessionID-coder}      → SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "coder" }
agentgroup:{grp-xxx} → {

roles: [

{ name: "planner",    sessionID: "...", endpoint: "10.0.0.4:8080", status: "ready" },

{ name: "researcher", sessionID: "...", endpoint: "10.0.0.5:8080", status: "ready" },

{ name: "coder",      sessionID: "...", endpoint: "10.0.0.6:8080", status: "failed" }  // BestEffort only

]

}

The router externally tracks only the coordinator's sessionID. Worker pod IPs live in the group manifest and are used for internal endpoint discovery and dependency injection.

Store implementation note: Because I have working familiarity with Redis internals from upstream contributions, I plan to implement the group manifest using a Redis Hash (HSET agentgroup:{grp-xxx} roles <json>) rather than a plain string key. This gives us atomic field-level updates for role status changes (HSET agentgroup:{grp-xxx} role:coder:status "ready") without rewriting the full manifest on every reconciliation tick. I'll implement the same pattern on the Valkey backend in parallel.

---

Core New Function: createSandboxGroup()

func (s *Server) createSandboxGroup(ctx context.Context, ...) (*types.CreateAgentGroupResponse, error) {
    groupSessionID := "grp-" + uuid.New().String()
    var created []createdRole
needGroupRollback := true
defer func() {
    if !needGroupRollback { return }
    for _, c := range created {
        s.rollbackSandboxCreation(dynamicClient, c.sandbox, nil, c.sessionID) // called unchanged
    }
}()

// Topological sort by dependencies (cycle detection returns descriptive error)
orderedRoles, err := topoSort(mar.Spec.Roles)
if err != nil {
    return nil, err // e.g. "dependency cycle: researcher → coder → researcher"
}

for _, role := range orderedRoles {
    // buildSandboxByAgentRuntime called unchanged
    sandbox, sandboxEntry, err := buildSandboxByAgentRuntime(mar.Namespace, role.RuntimeRef, s.informers)
    sandboxEntry.GroupSessionID = groupSessionID
    sandboxEntry.Role = role.Name

    // Inject dependency pod IPs as env vars before pod creation
    if len(role.Dependencies) > 0 {
        injectDependencyEndpoints(&sandbox.Spec.PodTemplate, role.Dependencies, created)
    }

    resultChan := s.sandboxController.WatchSandboxOnce(ctx, sandbox.Namespace, sandbox.Name)
    defer s.sandboxController.UnWatchSandbox(sandbox.Namespace, sandbox.Name)

    // createSandbox called unchanged
    resp, err := s.createSandbox(ctx, dynamicClient, sandbox, nil, sandboxEntry, resultChan)
    if err != nil {
        if mar.Spec.StartupPolicy == BestEffort && !role.IsCoordinator {
            recordRoleFailure(groupSessionID, role.Name)
            continue
        }
        return nil, fmt.Errorf("role %s: %w", role.Name, err)
    }
    created = append(created, createdRole{name: role.Name, resp: resp})
}

s.storeClient.SaveAgentGroup(ctx, groupSessionID, buildGroupManifest(created))
needGroupRollback = false
return response, nil

}

buildSandboxByAgentRuntime, createSandbox, and rollbackSandboxCreation are called without any modification.

---

Complete File Change Map

New Files

File	Description
pkg/apis/runtime/v1alpha1/multiagentruntime_types.go	CRD: MultiAgentRuntime, AgentRoleSpec, MultiAgentRuntimeStatus with kubebuilder markers
pkg/workloadmanager/multiagent_controller.go	MultiAgentRuntimeReconciler — self-healing (wired in Phase 4)
pkg/workloadmanager/multiagent_controller_test.go	Reconciler unit tests
sdk-python/agentcube/multi_agent.py	MultiAgentRuntimeClient — create group, get topology, delete
sdk-python/examples/multi_agent_usage.py	End-to-end usage example
test/e2e/multi_agent_runtime.yaml	E2E fixtures: 3 AgentRuntime CRDs + 1 MultiAgentRuntime
docs/design/multi-agent-design.md	Architecture design doc
manifests/charts/base/crds/runtime.agentcube.volcano.sh_multiagentruntimes.yaml	Auto-generated by make gen-crd — committed

---

Planned Deliverables (Phased Across 12 Weeks)

Phase 1 — Core Foundation (Weeks 1–4)

• MultiAgentRuntime CRD + register.go + run make gen-crd and make generate
• createSandboxGroup() with Atomic rollback only
• Store: all 4 group methods on both Redis and Valkey backends + tests (Redis Hash implementation for atomic role status updates)
• Router cold-path case for MultiAgentRuntimeKind
• HTTP route wiring in cmd/workload-manager/main.go (no controller yet)
• GC group manifest cleanup
• Unit tests: atomic rollback on partial failure, store CRUD, coordinator validation
• E2E test on kind cluster
• User guide: YAML example + kubectl workflow

Phase 2 — Warm Pools Per Role (Weeks 5–6)

• Add warmPoolSize to AgentRoleSpec
• Controller creates SandboxTemplate + SandboxWarmPool per role (mirrors CodeInterpreterReconciler exactly)
• Group creation uses SandboxClaim for warm roles — startup latency drops from minutes to seconds
• Unit tests for warm vs cold creation paths

Phase 3 — DAG Startup + Topology API (Weeks 7–8)

• Add dependencies []string to AgentRoleSpec
• Topological sort in createSandboxGroup() with cycle detection (descriptive error messages)
• Dependency endpoint injection into pod env vars before creation
• GET /topology endpoint returning all role internal endpoints for coordinator discovery
• Python SDK: get_topology() method on MultiAgentRuntimeClient

Phase 4 — StartupPolicy + Self-Healing (Weeks 9–11)

• startupPolicy: BestEffort — failed workers recorded in manifest, rollback not triggered
• MultiAgentRuntimeReconciler — Atomic: tears down group on role crash; BestEffort: restarts failed role independently
• MultiAgentRuntimeStatus with per-role conditions + ActiveGroups int32
• Wire MultiAgentRuntimeReconciler into setupControllers() in main.go

Phase 5 — Observability + Docs (Week 12)

• Prometheus metrics: agentcube_group_creation_duration_seconds, agentcube_group_role_failures_total, agentcube_active_groups
• docs/design/multi-agent-design.md — architecture, API reference, troubleshooting guide
• Final PR polish + review cycle iterations

---

What Stays Completely Unchanged

• createSandbox() — called as-is
• rollbackSandboxCreation() — called as-is
• All existing AgentRuntime / CodeInterpreter flows
• All existing store methods
• GC idle/expiry logic for single-agent sandboxes
• Router proxy logic

---

One specific question: for the /groups/:id/topology endpoint, is there a preference on response format — should worker endpoints be returned as a flat map of role → endpoint, or is a richer graph structure (including the dependencies edges) more useful for coordinator-side discovery?

Thank you for considering my application.

— Shubham (@ShubhamTaple)

# Multi-Agent Runtime Support (Design Proposal)

Hi [@hzxuzhonghu](https://github.com/hzxuzhonghu),

I'd like to share my design proposal for the Multi-Agent Runtime Support mentorship task ahead of the application deadline. Happy to revise based on your feedback.

I've been studying the AgentCube codebase and have mapped out a concrete implementation plan across 12 weeks. I'm committed to full-time hours for the duration of the mentorship.

One note on my background that I think is directly relevant here: I have 5 merged pull requests into the Redis upstream codebase ([#14987](redis/redis#14987), [#15042](redis/redis#15042), [#15094](redis/redis#15094), [#15095](redis/redis#15095), [#15151](redis/redis#15151)). The store backends for this feature (store_redis.go, store_valkey.go) are an area where I can contribute with genuine depth, not just surface-level familiarity.

---

Core Insight

The existing createSandbox() in handlers.go is already a complete, self-contained transaction — it handles placeholder storage, Kubernetes resource creation, readiness probing, rollback, and store finalization atomically. MultiAgentRuntime should be a composition layer above this, not a replacement. Every existing flow stays untouched.

A second observation: the existing SandboxTemplate + SandboxWarmPool pattern in the CodeInterpreter reconciler already solves per-role warm pool pre-warming. The multi-agent implementation should mirror this pattern per role rather than inventing a new one.

---

Proposed Architecture

MultiAgentRuntime CRD
        ↓  orchestrates
AgentRuntime CRDs  (one per role — existing CRDs, unchanged)
        ↓  instantiates via existing createSandbox()
Sandbox / Pod

Multi-agent orchestration is a pure composition layer. The entire existing sandbox lifecycle is preserved.

---

New CRD: MultiAgentRuntime

apiVersion: runtime.agentcube.volcano.sh/v1alpha1
kind: MultiAgentRuntime
metadata:
  name: research-team
  namespace: default
spec:
  startupPolicy: Atomic          # Atomic (default) or BestEffort
  roles:
    - name: planner
      runtimeRef: planner-agent  # references existing AgentRuntime CRD
      isCoordinator: true        # only coordinator entrypoints exposed externally
      warmPoolSize: 2
    - name: researcher
      runtimeRef: researcher-agent
      warmPoolSize: 3
      dependencies: [planner]    # dependency endpoints injected as env vars at boot
    - name: coder
      runtimeRef: coder-agent
      dependencies: [planner, researcher]
  sessionTimeout: 15m
  maxSessionDuration: 8h

Key design decisions:

• runtimeRef references an existing AgentRuntime CRD — no inline pod spec duplication, full reuse of existing security and resource configuration
• Flat roles[] list rather than a hardcoded orchestrator/workers split — supports DAGs, swarms, and peer topologies
• dependencies[] drives topological startup ordering — dependency pod IPs injected as env vars before creation
• warmPoolSize per role mirrors the existing CodeInterpreter warm pool pattern exactly — group creation latency drops from minutes to seconds
• startupPolicy: Atomic (default) — any role failure triggers full rollback; BestEffort — coordinator must succeed, worker failures are recorded but don't block
• Exactly one isCoordinator: true required — validated at admission time; only coordinator entrypoints are returned to the caller
• Workers communicate over cluster-internal DNS — the external Router is never in the inter-agent data path

---

Store Layout

Each sandbox is stored independently (GC and routing unchanged). A lightweight group manifest sits alongside:

sandbox:{sessionID-planner}    → SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "planner" }
sandbox:{sessionID-researcher} → SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "researcher" }
sandbox:{sessionID-coder}      → SandboxInfo{ ..., GroupSessionID: "grp-xxx", Role: "coder" }

agentgroup:{grp-xxx} → {
    roles: [
        { name: "planner",    sessionID: "...", endpoint: "10.0.0.4:8080", status: "ready" },
        { name: "researcher", sessionID: "...", endpoint: "10.0.0.5:8080", status: "ready" },
        { name: "coder",      sessionID: "...", endpoint: "10.0.0.6:8080", status: "failed" }  // BestEffort only
    ]
}

The router externally tracks only the coordinator's sessionID. Worker pod IPs live in the group manifest and are used for internal endpoint discovery and dependency injection.

Store implementation note: Because I have working familiarity with Redis internals from upstream contributions, I plan to implement the group manifest using a Redis Hash (HSET agentgroup:{grp-xxx} roles <json>) rather than a plain string key. This gives us atomic field-level updates for role status changes (HSET agentgroup:{grp-xxx} role:coder:status "ready") without rewriting the full manifest on every reconciliation tick. I'll implement the same pattern on the Valkey backend in parallel.

---

Core New Function: createSandboxGroup()

func (s *Server) createSandboxGroup(ctx context.Context, ...) (*types.CreateAgentGroupResponse, error) {
    groupSessionID := "grp-" + uuid.New().String()
    var created []createdRole

    needGroupRollback := true
    defer func() {
        if !needGroupRollback { return }
        for _, c := range created {
            s.rollbackSandboxCreation(dynamicClient, c.sandbox, nil, c.sessionID) // called unchanged
        }
    }()

    // Topological sort by dependencies (cycle detection returns descriptive error)
    orderedRoles, err := topoSort(mar.Spec.Roles)
    if err != nil {
        return nil, err // e.g. "dependency cycle: researcher → coder → researcher"
    }

    for _, role := range orderedRoles {
        // buildSandboxByAgentRuntime called unchanged
        sandbox, sandboxEntry, err := buildSandboxByAgentRuntime(mar.Namespace, role.RuntimeRef, s.informers)
        sandboxEntry.GroupSessionID = groupSessionID
        sandboxEntry.Role = role.Name

        // Inject dependency pod IPs as env vars before pod creation
        if len(role.Dependencies) > 0 {
            injectDependencyEndpoints(&sandbox.Spec.PodTemplate, role.Dependencies, created)
        }

        resultChan := s.sandboxController.WatchSandboxOnce(ctx, sandbox.Namespace, sandbox.Name)
        defer s.sandboxController.UnWatchSandbox(sandbox.Namespace, sandbox.Name)

        // createSandbox called unchanged
        resp, err := s.createSandbox(ctx, dynamicClient, sandbox, nil, sandboxEntry, resultChan)
        if err != nil {
            if mar.Spec.StartupPolicy == BestEffort && !role.IsCoordinator {
                recordRoleFailure(groupSessionID, role.Name)
                continue
            }
            return nil, fmt.Errorf("role %s: %w", role.Name, err)
        }
        created = append(created, createdRole{name: role.Name, resp: resp})
    }

    s.storeClient.SaveAgentGroup(ctx, groupSessionID, buildGroupManifest(created))
    needGroupRollback = false
    return response, nil
}

buildSandboxByAgentRuntime, createSandbox, and rollbackSandboxCreation are called without any modification.

---

Complete File Change Map

New Files

File	Description
pkg/apis/runtime/v1alpha1/multiagentruntime_types.go	CRD: MultiAgentRuntime, AgentRoleSpec, MultiAgentRuntimeStatus with kubebuilder markers
pkg/workloadmanager/multiagent_controller.go	MultiAgentRuntimeReconciler — self-healing (wired in Phase 4)
pkg/workloadmanager/multiagent_controller_test.go	Reconciler unit tests
sdk-python/agentcube/multi_agent.py	MultiAgentRuntimeClient — create group, get topology, delete
sdk-python/examples/multi_agent_usage.py	End-to-end usage example
test/e2e/multi_agent_runtime.yaml	E2E fixtures: 3 AgentRuntime CRDs + 1 MultiAgentRuntime
docs/design/multi-agent-design.md	Architecture design doc
manifests/charts/base/crds/runtime.agentcube.volcano.sh_multiagentruntimes.yaml	Auto-generated by make gen-crd — committed

Modified Files

File	Change
pkg/apis/runtime/v1alpha1/register.go	Add MultiAgentRuntimeKind, MultiAgentRuntimeListKind, MultiAgentRuntimeGroupVersionKind
pkg/apis/runtime/v1alpha1/zz_generated.deepcopy.go	Regenerated by make generate
pkg/common/types/types.go	Add MultiAgentRuntimeKind constant
pkg/common/types/sandbox.go	Add GroupSessionID, Role to SandboxInfo; add AgentGroupManifest, group request/response types
pkg/api/errors.go	Add ErrMultiAgentRuntimeNotFound + multiAgentRuntimeResource in workloadResource() switch
pkg/workloadmanager/informers.go	Add MultiAgentRuntimeGVR + informer wiring + cache sync
pkg/workloadmanager/workload_builder.go	Add GroupSessionID, Role to sandboxEntry struct
pkg/workloadmanager/sandbox_helper.go	Propagate group fields in buildSandboxPlaceHolder() + buildSandboxInfo()
pkg/workloadmanager/handlers.go	Add handleMultiAgentRuntimeCreate, createSandboxGroup, handleDeleteAgentGroup, handleGetGroupTopology
pkg/workloadmanager/handlers_test.go	Group creation + rollback test cases
pkg/workloadmanager/server.go	Add 3 routes: POST /v1/multi-agent-runtime, DELETE /v1/multi-agent-runtime/sessions/:groupSessionId, GET /v1/multi-agent-runtime/groups/:groupSessionId/topology
pkg/workloadmanager/garbage_collection.go	Group manifest cleanup when last member sandbox is GC'd
pkg/store/interface.go	Add SaveAgentGroup, GetAgentGroup, DeleteAgentGroup, UpdateAgentGroupRoleStatus to interface
pkg/store/store_redis.go	Implement all 4 group methods using Redis Hash for atomic per-role status updates
pkg/store/store_redis_test.go	Group CRUD + concurrent status update test cases
pkg/store/store_valkey.go	Implement all 4 group methods (Valkey backend — parallel to Redis)
pkg/store/store_valkey_test.go	Group CRUD test cases
pkg/router/session_manager.go	Add MultiAgentRuntimeKind case in endpoint switch
cmd/workload-manager/main.go	Phase 1: wire HTTP routes only; Phase 4: wire MultiAgentRuntimeReconciler into setupControllers()
sdk-python/agentcube/__init__.py	Export MultiAgentRuntimeClient
test/e2e/e2e_test.go	Add TestMultiAgentRuntimeCreate, TestMultiAgentRuntimeRollback

---

Planned Deliverables (Phased Across 12 Weeks)

Phase 1 — Core Foundation (Weeks 1–4)

• MultiAgentRuntime CRD + register.go + run make gen-crd and make generate
• createSandboxGroup() with Atomic rollback only
• Store: all 4 group methods on both Redis and Valkey backends + tests (Redis Hash implementation for atomic role status updates)
• Router cold-path case for MultiAgentRuntimeKind
• HTTP route wiring in cmd/workload-manager/main.go (no controller yet)
• GC group manifest cleanup
• Unit tests: atomic rollback on partial failure, store CRUD, coordinator validation
• E2E test on kind cluster
• User guide: YAML example + kubectl workflow

Phase 2 — Warm Pools Per Role (Weeks 5–6)

• Add warmPoolSize to AgentRoleSpec
• Controller creates SandboxTemplate + SandboxWarmPool per role (mirrors CodeInterpreterReconciler exactly)
• Group creation uses SandboxClaim for warm roles — startup latency drops from minutes to seconds
• Unit tests for warm vs cold creation paths

Phase 3 — DAG Startup + Topology API (Weeks 7–8)

• Add dependencies []string to AgentRoleSpec
• Topological sort in createSandboxGroup() with cycle detection (descriptive error messages)
• Dependency endpoint injection into pod env vars before creation
• GET /topology endpoint returning all role internal endpoints for coordinator discovery
• Python SDK: get_topology() method on MultiAgentRuntimeClient

Phase 4 — StartupPolicy + Self-Healing (Weeks 9–11)

• startupPolicy: BestEffort — failed workers recorded in manifest, rollback not triggered
• MultiAgentRuntimeReconciler — Atomic: tears down group on role crash; BestEffort: restarts failed role independently
• MultiAgentRuntimeStatus with per-role conditions + ActiveGroups int32
• Wire MultiAgentRuntimeReconciler into setupControllers() in main.go

Phase 5 — Observability + Docs (Week 12)

• Prometheus metrics: agentcube_group_creation_duration_seconds, agentcube_group_role_failures_total, agentcube_active_groups
• docs/design/multi-agent-design.md — architecture, API reference, troubleshooting guide
• Final PR polish + review cycle iterations

---

One specific question: for the /groups/:id/topology endpoint, is there a preference on response format — should worker endpoints be returned as a flat map of role → endpoint, or is a richer graph structure (including the dependencies edges) more useful for coordinator-side discovery?

Thank you for considering my application.

— Shubham ([@ShubhamTaple](https://github.com/ShubhamTaple))

[Oxidaner]

Hi @YaoZengzeng, I would like to work on this LFX project.

After reading the current Kthena Router code and related documentation, my understanding is that the existing router already has a solid foundation for OpenAI-compatible traffic. The current path roughly works as:

Client request
  -> /v1/* Router entry
  -> request parsing
  -> auth / rate limit / access log / metrics
  -> ModelRoute matching
  -> ModelServer selection
  -> pod discovery
  -> scheduler
  -> proxy to selected in-cluster model pod

For third-party LLM APIs, I think we can extend this path without introducing another gateway or bypassing the current router pipeline. My proposal is to treat external LLM services as a first-class backend target in Kthena Router, while keeping the existing in-cluster ModelServer path unchanged.

Proposed Architecture

flowchart TD
    Client[OpenAI-compatible Client] --> Router[Kthena Router /v1/*]

    Router --> Filters[Auth / RateLimit / AccessLog / Metrics]
    Filters --> RouteMatch[ModelRoute Matching]

    RouteMatch -->|In-cluster target| ModelServer[ModelServer]
    ModelServer --> Store[Datastore: Pods + Runtime Metrics]
    Store --> Scheduler[Scheduler Plugins]
    Scheduler --> PodProxy[Proxy to selected model pod]

    RouteMatch -->|External target| ProviderRef[External Provider Target]
    ProviderRef --> ProviderManager[Provider Manager]
    ProviderManager --> Adapter[Provider Adapter]
    Adapter --> ExternalAPI[OpenAI / DeepSeek / Gemini / HuggingFace / Custom API]

    PodProxy --> Response[OpenAI-compatible Response]
    ExternalAPI --> Response

Loading

1. API Design

I suggest adding an ExternalModelProvider CRD, or an equivalent provider section if the community prefers extending the existing networking APIs.

Example:

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ExternalModelProvider
metadata:
  name: openai-provider
  namespace: default
spec:
  providerType: OpenAICompatible
  endpoint: https://api.openai.com
  auth:
    secretRef:
      name: openai-api-key
      key: apiKey
  models:
    - publicName: gpt-4o-mini
      upstreamName: gpt-4o-mini
  timeout: 60s
  retry:
    attempts: 2

Then ModelRoute can support both internal and external targets:

targetModels:
  - modelServerName: qwen-local
    weight: 80
  - externalProviderRef:
      name: openai-provider
      model: gpt-4o-mini
    weight: 20

The two target types should be mutually exclusive:

• modelServerName: current in-cluster model serving path.
• externalProviderRef: third-party LLM API path.

This keeps backward compatibility for existing users and avoids overloading ModelServer, which currently represents Kubernetes workloads selected by labels.

2. Router Changes

I suggest introducing a small internal routing abstraction:

type RouteDestination struct {
    Type RouteDestinationType // ModelServer or ExternalProvider

    ModelServerName types.NamespacedName
    ExternalProviderRef *ExternalProviderRef
}

Current flow:

Parse request
  -> MatchModelServer
  -> Get pods
  -> Schedule
  -> Proxy to pod

Proposed flow:

Parse request
  -> Apply auth / rate limit / access log / metrics
  -> Match route
  -> If destination is ModelServer:
         use existing scheduling + proxy path
     If destination is ExternalProvider:
         resolve provider config + secret
         normalize request
         forward to external API
         normalize response / stream back to client

This means most existing logic remains reusable:

• request parsing
• model-based route matching
• weighted traffic split
• rate limiting
• access log
• metrics
• OpenAI-compatible response handling

Only the final upstream execution path needs to branch.

3. Provider Adapter Layer

I would add a provider package like:

pkg/kthena-router/provider/
  provider.go
  manager.go
  openai_compatible.go
  gemini.go
  huggingface.go
  transport.go

Suggested interface:

type Provider interface {
    Name() string
    BuildRequest(ctx context.Context, original *http.Request, modelRequest router.ModelRequest) (*http.Request, error)
    Do(ctx context.Context, req *http.Request) (*http.Response, error)
}

For the first milestone, I think OpenAICompatibleProvider should be implemented first because OpenAI, DeepSeek, and many other services expose similar /v1/chat/completions, /v1/completions, and /v1/embeddings APIs.

Provider-specific adapters can be added incrementally:

• OpenAICompatibleProvider: OpenAI, DeepSeek, compatible self-hosted services
• GeminiProvider: request/response transformation if needed
• HuggingFaceProvider: model endpoint style transformation if needed
• CustomHTTPProvider: user-defined endpoint + headers for simple cases

4. Secret and Security Model

API keys should not be stored directly in CRDs or ConfigMaps.

The provider should reference a Kubernetes Secret:

auth:
  secretRef:
    name: deepseek-secret
    key: apiKey

The router should:

• read the secret through informer/client cache
• inject the API key into the upstream request header
• redact sensitive headers in access logs
• avoid printing endpoint credentials in error messages
• support timeout and retry policy per provider
• optionally validate that external endpoints use HTTPS by default

5. Streaming Support

Since the router already handles streaming and non-streaming responses, the external provider path should preserve this behavior.

For OpenAI-compatible providers:

• if stream: true, forward SSE chunks directly to the client
• parse usage data when available
• if usage is not available in stream chunks, keep current behavior without failing the request
• map upstream error responses to downstream OpenAI-compatible errors where possible

6. Observability

The external path should reuse existing router observability instead of adding a separate mechanism.

Possible new labels or fields:

backend_type = internal | external
provider = openai | deepseek | gemini | huggingface | custom
upstream_model = actual provider model name
route = namespace/name of ModelRoute

Metrics can include:

• downstream request count
• upstream external request count
• upstream latency
• upstream error count by provider/status code
• rate-limit result
• token usage if available

Access logs can add:

target_type=external
provider=deepseek
upstream_model=deepseek-chat

but should never log API keys or sensitive request headers.

7. Testing Plan

Because stable free external LLM APIs are not always available, I would avoid depending on real third-party services in CI.

Unit tests:

• CRD validation: internal target and external target are mutually exclusive
• provider config parsing
• secret resolution
• OpenAI-compatible request building
• header injection and redaction
• streaming response forwarding with httptest.Server
• upstream error mapping
• timeout and retry behavior
• weighted routing between internal and external targets

Integration-style tests:

• use a local fake OpenAI-compatible server
• create ExternalModelProvider
• create ModelRoute
• send /v1/chat/completions
• verify the router forwards to the fake provider and returns the expected response

This should satisfy the project requirement without requiring real paid API credentials.

8. Implementation Milestones

Milestone 1: Proposal and API design

• Write the design proposal under docs/proposal/
• Confirm CRD shape with maintainers
• Decide whether to add ExternalModelProvider CRD or extend existing APIs

Milestone 2: Router internal abstraction

• Introduce RouteDestination
• Extend route matching to return internal/external destination
• Keep existing ModelServer scheduling path unchanged

Milestone 3: Provider manager and OpenAI-compatible provider

• Implement provider registry/manager
• Implement OpenAI-compatible external provider
• Support non-streaming and streaming request forwarding
• Support timeout, retry, and error mapping

Milestone 4: Security and observability

• Add Secret-based API key loading
• Redact sensitive headers in logs
• Add provider-level metrics and access-log fields

Milestone 5: Tests and docs

• Add unit tests with fake HTTP servers
• Add user guide examples
• Add troubleshooting docs for auth, timeout, and provider errors

Why This Design

I think this design fits Kthena Router because it preserves the current in-cluster model-aware scheduling path, while adding a clean external-provider path for cases where users want to mix local models and public LLM APIs.

It also enables useful scenarios such as:

• fallback from local model to external provider
• canary traffic between local and external models
• unified auth/rate-limit/observability for both local and third-party LLM APIs
• OpenAI-compatible entrypoint for users regardless of where the model is actually served

I am willing to start by drafting the proposal and a small prototype for the OpenAI-compatible provider path, then iterate based on community feedback.

[sicaario]

Hi @hzxuzhonghu ,

I’m applying for this LFX project. I’ve been reading through the AgentCube codebase and contributing around the repo recently, and the multi-agent runtime problem space looks really interesting to work on.

Looking forward to learning more from the maintainers and contributing incrementally throughout the mentorship process.