|
| 1 | += Redpanda Agentic Data Plane Overview |
| 2 | +:description: Enterprise-grade infrastructure for building, deploying, and governing AI agents at scale with compliance-grade audit trails. |
| 3 | +:page-topic-type: overview |
| 4 | +:page-categories: AI, Agentic AI |
| 5 | +:personas: evaluator, ai_agent_developer, platform_admin |
| 6 | +:learning-objective-1: Identify the key components of Redpanda ADP and their purposes |
| 7 | +:learning-objective-2: Describe how each component addresses enterprise governance and reliability requirements |
| 8 | +:learning-objective-3: Determine whether Redpanda ADP fits your organization's requirements for AI agent deployment |
| 9 | + |
| 10 | +glossterm:AI agent[,AI agents] are moving from demos to production. Enterprises need governance, reliability, and cost control to deploy them safely. Redpanda Agentic Data Plane (ADP) combine a streaming-native immutable log, 300+ proven data connectors, and declarative glossterm:AI agent[,AI agents] into a unified platform with built-in compliance-grade audit trails. |
| 11 | + |
| 12 | +After reading this page, you will be able to: |
| 13 | + |
| 14 | +* [ ] {learning-objective-1} |
| 15 | +* [ ] {learning-objective-2} |
| 16 | +* [ ] {learning-objective-3} |
| 17 | +
|
| 18 | +NOTE: The Agentic Data Plane is supported on BYOC clusters running with AWS and Redpanda version 25.3 and later. |
| 19 | + |
| 20 | +== AI Agents |
| 21 | + |
| 22 | +Instead of writing Python or JavaScript, you declare the agent behavior you want and Redpanda handles execution and orchestration. You define behaviors in YAML, orchestrate multiple specialized glossterm:subagent[,sub-agents], or bring your own frameworks like LangChain or LlamaIndex. |
| 23 | + |
| 24 | +What makes this practical at scale is xref:develop:connect/about.adoc[Redpanda Connect]. More than 300 connectors with built-in filtering, enrichment, and routing give declarative definitions real power. Upcoming templates will provide default behaviors for common domains such as customer success, legal, and finance. |
| 25 | + |
| 26 | +The result is faster time-to-production, lower maintenance (declarative definitions instead of imperative code), and organizational consistency across teams. |
| 27 | + |
| 28 | +== MCP Servers |
| 29 | + |
| 30 | +glossterm:MCP server[,MCP servers] translate agent intent into connections to databases, queues, HRIS, CRMs, and other business systems. They are the simplest way to give agents context and capabilities without writing glue code. |
| 31 | + |
| 32 | +Under the hood, MCP servers wrap the same proven connectors that power some of the world's largest e-commerce, EV, electricity, and AI companies. Built on xref:develop:connect/about.adoc[Redpanda Connect], they are lightweight, support OIDC-based authentication, and enforce deterministic policies at the tool level. You define tools in YAML, and policy enforcement programmatically prevents prompt injection, SQL injection, and other agent-based attacks. |
| 33 | + |
| 34 | +With over 300 connectors out-of-the-box and real-time debugging capabilities, you reduce integration time while getting enterprise-grade security. You can reuse your existing infrastructure and data sources rather than building new integrations from scratch. |
| 35 | + |
| 36 | +== Transcripts |
| 37 | + |
| 38 | +Every agent action is recorded in an end-to-end execution log. A single glossterm:transcript[] can span multiple agents, tools, and models, covering interactions that last minutes to days. |
| 39 | + |
| 40 | +Transcripts are the keystone of agent governance. They are built on Redpanda's immutable log with Raft consensus and TLA+ correctness proofs. No gaps, no tampering. For regulated industries that require multi-year audit trails, this provides a compliance-grade record of every decision an agent makes and every data source it uses. |
| 41 | + |
| 42 | +Redpanda captures 100% of agent actions through OpenTelemetry standards, with end-to-end lineage across the entire execution chain. You can materialize execution logs to Iceberg tables for long-term retention and analysis, or replay them to evaluate and improve agent performance over time. |
| 43 | + |
| 44 | +== AI Gateway |
| 45 | + |
| 46 | +The AI Gateway manages LLM provider access with two priorities: keeping your application up and keeping costs under control. |
| 47 | + |
| 48 | +For high availability, the gateway provides provider-agnostic routing with intelligent failover. Your users don't care which provider serves a request. They care that the application stays up. For fiscal control, you get per-tenant budgets and rate limiting, so there are no runaway costs and no surprise bills. |
| 49 | + |
| 50 | +The gateway also supports tenancy modeling for teams, individuals, applications, and service accounts, giving you chargeback transparency for internal cost allocation. You can proxy both models and MCP gateways, centralizing compliance for all LLM interactions without locking into any single provider. |
| 51 | + |
| 52 | +== Enterprise governance |
| 53 | + |
| 54 | +Redpanda ADP addresses critical enterprise requirements across all components. |
| 55 | + |
| 56 | +*Security by design*:: MCP servers enforce policies at the tool level, programmatically preventing prompt injection, SQL injection, and other agent-based attacks. Policy enforcement is deterministic and controlled. Agents cannot bypass security constraints even through creative prompting. |
| 57 | + |
| 58 | +*Unified authorization*:: All components use OIDC-based authentication with an on-behalf-of authorization model. When a user invokes an agent, the agent inherits the intersection of its own permissions and the user's permissions. This ensures proper data access scoping. |
| 59 | + |
| 60 | +*Complete observability*:: Redpanda ADP provides two levels of inspection. Execution logs (transcripts) capture every agent action with 100% sampling using OpenTelemetry standards. Real-time debugging tools allow you to inspect individual MCP server calls down to individual tool invocations with full timing data. You can view detailed agent actions in glossterm:Redpanda Console[] and replay data for agent evaluations. |
| 61 | + |
| 62 | +*Compliance and audit*:: For industries requiring multi-year audit trails, Redpanda ADP records every agent action and data source used in decision-making. Execution logs are stored in Redpanda topics and can be materialized to Iceberg tables for long-term retention and analysis. |
| 63 | + |
| 64 | +== Use cases |
| 65 | + |
| 66 | +Organizations use Redpanda ADP to: |
| 67 | + |
| 68 | +* *Automate operational workflows*: Create specialized agents for building management, infrastructure monitoring, compliance reporting, and other domain-specific tasks. |
| 69 | +* *Monitor manufacturing and operations*: Deploy multi-agent systems that analyze factory machine telemetry in real-time, detect anomalies, search equipment manuals, and create maintenance tickets automatically. |
| 70 | +* *Extend enterprise productivity tools*: Integrate Microsoft Copilot or other workplace agents with internal data sources and systems that are otherwise inaccessible. |
| 71 | + |
| 72 | +== Next steps |
| 73 | + |
| 74 | +* xref:ai-agents:mcp/overview.adoc[] |
0 commit comments