Merge pull request #3 from redpanda-data/reapply-pr560-transcripts-rewrite

Reapply PR #560: Transcripts documentation rewrite for ADP GA

Author: micheleRP

modules/observability/pages/concepts.adoc

@@ -1,13 +1,12 @@
 = Transcripts and AI Observability
-:page-aliases: redpanda-cloud:ai-agents:observability/concepts.adoc
 :description: Understand how Redpanda captures end-to-end execution transcripts on an immutable distributed log for agent governance and observability.
 :page-topic-type: concepts
 :personas: evaluator, agent_developer, platform_admin, data_engineer
 :learning-objective-1: Explain how transcripts and spans capture execution flow
 :learning-objective-2: Interpret transcript structure for debugging and monitoring
 :learning-objective-3: Distinguish between transcripts and audit logs
 
-include::ROOT:partial$adp-la.adoc[]
+include::ai-agents:partial$adp-la.adoc[]
 
 Redpanda provides complete observability and governance for AI agents through automated glossterm:transcript[] capture. Every agent execution, from simple tool calls to complex multi-agent, multi-turn workflows, generates a permanent, write-once record stored on Redpanda's glossterm:log[distributed log]. This captures all agent reasoning, tool invocations, model interactions, and data flows with 100% sampling and no gaps. 
 
@@ -226,6 +225,8 @@ AI SDK layer (following https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-a
 - `gen_ai.agent.name`: Sub-agent name for multi-agent systems
 - `gen_ai.provider.name`, `gen_ai.request.model`: LLM provider and model
 - `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`: Token consumption
+- `gen_ai.usage.input_tokens_cost_usd`, `gen_ai.usage.output_tokens_cost_usd`: USD cost per call, derived from token counts and a per-model pricing table
+// TODO: Confirm exact USD-cost attribute names + pricing-table coverage with team-ai before GA.
 - `gen_ai.tool.name`, `gen_ai.tool.call.arguments`: Tool execution details
 - `gen_ai.input.messages`, `gen_ai.output.messages`: Full LLM conversation context
 
@@ -321,9 +322,10 @@ Redpanda manages both the `redpanda.otel_traces` topic and its schema automatica
 
 The `redpanda.otel_traces` topic has a predefined retention policy. Configuration changes to this topic are not supported. If you modify settings, Redpanda reverts them to the default values.
 
-The topic persists in your cluster even after all agents and MCP servers are deleted, allowing you to retain historical trace data for analysis.
+The topic persists even after all agents and MCP servers are deleted, allowing you to retain historical trace data for analysis.
 
-Transcripts may contain sensitive information from your tool inputs and outputs. Consider implementing appropriate glossterm:ACL[access control lists (ACLs)] for the `redpanda.otel_traces` topic, and review the data in transcripts before sharing or exporting to external systems.
+Transcripts may contain sensitive information from your tool inputs and outputs. Review the data in transcripts before sharing or exporting to external systems.
+// TODO: Re-add guidance on ACL scoping for `redpanda.otel_traces` once the standalone-ADP permission model lands. Today's wording assumed users manage topic ACLs on their own Redpanda Cloud cluster, which won't apply when ADP is a separate product surface.
 
 == Transcripts compared to audit logs
 
@@ -338,8 +340,15 @@ Transcripts provide:
 
 Transcripts are optimized for execution-level observability and governance. For user-level accountability tracking ("who initiated what"), use the session and task topics for agents, which provide records of agent conversations and task execution.
 
+[[history-reconstruction]]
+== Reconstructed transcript history
+
+Trace data on `redpanda.otel_traces` is subject to a retention policy. When a transcript covers a long-running conversation whose earliest spans have already been evicted, Redpanda reconstructs the missing turns from the LLM message context carried on later spans (`gen_ai.input.messages`) and sets the boolean field `is_reconstructed` to `true` on each affected turn. The UI surfaces this as a **Reconstructed** badge on those turns — `is_reconstructed` is the backing data field; "Reconstructed" is the visible label. Reconstructed turns preserve the high-level intent and role ordering of the conversation, but do not preserve byte-level fidelity: token counts, per-turn latency, and tool-call arguments are unavailable for the reconstructed range.
+
+// TODO: Verify model names on this page against the GA build (gpt-5.2, claude-sonnet-4-5, gpt-5-nano, gemini-3-flash-preview) — replace any that ship with a different GA identifier.
+
 == Next steps
 
-* xref:transcripts.adoc[]
-* xref:agents:monitor.adoc[]
-* xref:mcp:remote/monitor-mcp-servers.adoc[]
+* xref:ai-agents:observability/transcripts.adoc[]
+* xref:ai-agents:agents/monitor-agents.adoc[]
+* xref:ai-agents:mcp/remote/monitor-mcp-servers.adoc[]

modules/observability/pages/ingest-custom-traces.adoc

@@ -5,7 +5,9 @@
 :learning-objective-2: Validate trace data format and compatibility with existing MCP server traces
 :learning-objective-3: Secure the ingestion endpoint using authentication mechanisms
 
-include::ROOT:partial$adp-la.adoc[]
+include::ai-agents:partial$adp-la.adoc[]
+
+// TODO (standalone-ADP rewrite): This page was written when ADP lived inside a Redpanda Cloud cluster, so the setup flow deploys a Redpanda Connect pipeline on the user's own cluster and references cluster-id-shaped URLs (`<pipeline-id>.pipelines.<cluster-id>.clusters.rdpa.co`). Post-2026-04-21, ADP ships as its own product surface and users won't necessarily have a Redpanda Cloud cluster. The whole ingestion flow (prereqs, pipeline deployment, endpoint URL format, auth) needs a rewrite once the standalone-ADP ingestion path is defined. Tracking under ADP Docs Plan Workflow #11 (BYOA telemetry). Until then, the content below is accurate for the federated-in-cloud-ui preview but will mislead standalone-ADP users.
 
 You can extend Redpanda's transcript observability to custom agents built with frameworks like LangChain or instrumented with OpenTelemetry SDKs. By ingesting traces from external applications into the `redpanda.otel_traces` topic, you gain unified visibility across all agent executions, from Redpanda's declarative agents, Remote MCP servers, to your own custom implementations.
 
@@ -17,9 +19,9 @@ After reading this page, you will be able to:
 
 == Prerequisites
 
-* A BYOC cluster
-* Ability to manage secrets in Redpanda Cloud
-* The latest version of xref:redpanda-cloud:manage:rpk/rpk-install.adoc[`rpk`] installed
+* A Redpanda Connect pipeline host (today: a BYOC Redpanda Cloud cluster with Connect enabled). Ability to manage secrets on that host.
+// TODO: Replace with the standalone-ADP ingestion target once defined (may no longer require a Redpanda Cloud cluster).
+* The latest version of xref:manage:rpk/rpk-install.adoc[`rpk`] installed
 * Custom agent or application instrumented with OpenTelemetry SDK
 * Basic understanding of the https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/[OpenTelemetry span format^] and https://opentelemetry.io/docs/specs/otlp/[OpenTelemetry Protocol (OTLP)^]
 
@@ -29,7 +31,8 @@ If you're using LangChain with OpenTelemetry tracing, you can send traces to Red
 
 . Configure LangChain's OpenTelemetry integration by following the https://docs.langchain.com/langsmith/trace-with-opentelemetry[LangChain documentation^]. 
 
-. Deploy a Redpanda Connect pipeline using the `otlp_http` input to receive OTLP traces over HTTP. Create the pipeline in the **Connect** page of your cluster, or see the <<configure-the-ingestion-pipeline,Configure the ingestion pipeline>> section below for a sample configuration.
+. Deploy a Redpanda Connect pipeline using the `otlp_http` input to receive OTLP traces over HTTP. Today, create the pipeline in the **Connect** page of your Redpanda Cloud cluster. See the <<configure-the-ingestion-pipeline,Configure the ingestion pipeline>> section below for a sample configuration.
+// TODO: Update the deployment entry point once the standalone-ADP ingestion flow is defined.
 
 . Configure your OTEL exporter to send traces to your Redpanda Connect pipeline using environment variables:
 +
@@ -51,19 +54,19 @@ By default, traces are sent to both LangSmith and your Redpanda Connect pipeline
 export LANGSMITH_OTEL_ONLY="true"
 ----
 
-Your LangChain application will send traces to the `redpanda.otel_traces` topic, making them visible in the Transcripts view in your cluster  alongside Remote MCP server and declarative agent traces.
+Your LangChain application will send traces to the `redpanda.otel_traces` topic, making them visible in the Transcripts view alongside Remote MCP server and declarative agent traces.
 
 For non-LangChain applications or custom instrumentation, continue with the sections below.
 
 == About custom trace ingestion
 
 Custom agents are applications with OpenTelemetry instrumentation that operate independently of Redpanda's Remote MCP servers or declarative agents (such as LangChain, CrewAI, or manually instrumented applications).
 
-When these agents send traces to `redpanda.otel_traces`, you gain unified observability alongside Remote MCP server and declarative agent traces. See xref:concepts.adoc#cross-service-transcripts[Cross-service transcripts] for details on how traces correlate across services.
+When these agents send traces to `redpanda.otel_traces`, you gain unified observability alongside Remote MCP server and declarative agent traces. See xref:ai-agents:observability/concepts.adoc#cross-service-transcripts[Cross-service transcripts] for details on how traces correlate across services.
 
 === Trace format requirements
 
-Custom agents must emit traces in OTLP format. The xref:redpanda-cloud:develop:connect/components/inputs/otlp_http.adoc[`otlp_http`] input accepts both OTLP Protobuf (`application/x-protobuf`) and JSON (`application/json`) payloads. For <<use-grpc,gRPC transport>>, use the xref:redpanda-cloud:develop:connect/components/inputs/otlp_grpc.adoc[`otlp_grpc`] input.
+Custom agents must emit traces in OTLP format. The xref:develop:connect/components/inputs/otlp_http.adoc[`otlp_http`] input accepts both OTLP Protobuf (`application/x-protobuf`) and JSON (`application/json`) payloads. For <<use-grpc,gRPC transport>>, use the xref:develop:connect/components/inputs/otlp_grpc.adoc[`otlp_grpc`] input.
 
 Each trace must follow the OTLP specification with these required fields:
 
@@ -95,7 +98,7 @@ Optional but recommended fields:
 - `parentSpanId` for hierarchical traces
 - `attributes` for contextual information
 
-For complete trace structure details, see xref:concepts.adoc#understand-the-transcript-structure[Understand the transcript structure].
+For complete trace structure details, see xref:ai-agents:observability/concepts.adoc#understand-the-transcript-structure[Understand the transcript structure].
 
 == Configure the ingestion pipeline
 
@@ -168,17 +171,20 @@ NOTE: Clients must include the authentication token in gRPC metadata as `authori
 
 The OTLP input automatically handles format conversion, so no processors are needed for basic trace ingestion. Each span becomes a separate message in the `redpanda.otel_traces` topic.
 
-=== Deploy the pipeline in Redpanda Cloud
+=== Deploy the ingestion pipeline
+
+// TODO (standalone-ADP): Steps below assume a Redpanda Cloud cluster with Connect enabled. The standalone ADP product surface may expose a different deployment entry point. Revisit the whole subsection under Workflow #11.
 
 . In the *Connect* page of your Redpanda Cloud cluster, click *Create Pipeline*.
 . For the input, select the *otlp_http* (or *otlp_grpc*) component.
 . Skip to *Add a topic* and select `redpanda.otel_traces` from the list of existing topics. Leave the default advanced settings.
 . In the *Add permissions* step, create a service account with write access to the `redpanda.otel_traces` topic.
-. In the *Create pipeline* step, enter a name for your pipeline and paste the configuration. Redpanda Cloud automatically handles authentication for incoming requests.
+. In the *Create pipeline* step, enter a name for your pipeline and paste the configuration. Authentication for incoming requests is handled automatically.
 
 == Send traces from your custom agent
 
-Configure your custom agent to send OpenTelemetry traces to the pipeline endpoint. After deploying the pipeline, you can find its URL in the Redpanda Cloud UI on the pipeline details page.
+Configure your custom agent to send OpenTelemetry traces to the pipeline endpoint. After deploying the pipeline, you can find its URL on the pipeline details page in the host UI.
+// TODO (standalone-ADP): Confirm where users find the pipeline URL once the ingestion path moves out of the Redpanda Cloud UI.
 
 [cols="1,3", options="header"]
 |===
@@ -193,7 +199,8 @@ Configure your custom agent to send OpenTelemetry traces to the pipeline endpoin
 
 === Authenticate to the pipeline
 
-The OTLP pipeline uses the same authentication mechanism as the Redpanda Cloud API. Obtain an access token using your service account credentials as described in xref:redpanda-cloud:security:cloud-authentication.adoc#authenticate-to-the-cloud-api[Authenticate to the Cloud API].
+The OTLP pipeline uses the same authentication mechanism as the Redpanda Cloud API (today, while ingestion runs on a Redpanda Cloud cluster). Obtain an access token using your service account credentials as described in xref:redpanda-cloud:security:cloud-authentication.adoc#authenticate-to-the-cloud-api[Authenticate to the Cloud API].
+// TODO (standalone-ADP): Update the auth model when the standalone ADP ingestion path is defined.
 
 Include the token in your requests:
 
@@ -558,25 +565,25 @@ Check that traces are being published to the `redpanda.otel_traces` topic:
 rpk topic consume redpanda.otel_traces --offset end -n 10
 ----
 
-You can also view the `redpanda.otel_traces` topic in the *Topics* page of Redpanda Cloud UI.
+You can also view the `redpanda.otel_traces` topic in the *Topics* page of the host UI (today, the Redpanda Cloud UI).
 
 Look for spans with your custom `instrumentationScope.name` to identify traces from your agent.
 
 === View traces in Transcripts 
 
-After your custom agent sends traces through the pipeline, they appear in your cluster's *Agentic AI > Transcripts* view alongside traces from Remote MCP servers, declarative agents, and AI Gateway.
+After your custom agent sends traces through the pipeline, they appear in the *Transcripts* view of the ADP UI alongside traces from Remote MCP servers, declarative agents, and AI Gateway.
 
 ==== Identify custom agent transcripts
 
-Custom agent transcripts are identified by the `service.name` resource attribute, which differs from Redpanda's built-in services (`ai-agent` for declarative agents, `mcp-{server-id}` for MCP servers). See xref:concepts.adoc#cross-service-transcripts[Cross-service transcripts] to understand how the `service.name` attribute identifies transcript sources.
+Custom agent transcripts are identified by the `service.name` resource attribute, which differs from Redpanda's built-in services (`ai-agent` for declarative agents, `mcp-{server-id}` for MCP servers). See xref:ai-agents:observability/concepts.adoc#cross-service-transcripts[Cross-service transcripts] to understand how the `service.name` attribute identifies transcript sources.
 
 Your custom agent transcripts display with:
 
 * **Service name** in the service filter dropdown (from your `service.name` resource attribute)
 * **Agent name** in span details (from the `gen_ai.agent.name` attribute)
 * **Operation names** like `"invoke_agent my-assistant"` indicating agent executions
 
-For detailed instructions on filtering, searching, and navigating transcripts in the UI, see xref:transcripts.adoc[View Transcripts].
+For detailed instructions on filtering, searching, and navigating transcripts in the UI, see xref:ai-agents:observability/transcripts.adoc[View Transcripts].
 
 ==== Token usage tracking
 
@@ -614,7 +621,7 @@ If requests succeed but traces do not appear in `redpanda.otel_traces`:
 
 == Next steps
 
-* xref:transcripts.adoc[]
-* xref:agents:monitor.adoc[Observability for declarative agents]
-* xref:redpanda-cloud:develop:connect/components/inputs/otlp_http.adoc[OTLP HTTP input reference] - Complete configuration options for the `otlp_http` component
-* xref:redpanda-cloud:develop:connect/components/inputs/otlp_grpc.adoc[OTLP gRPC input reference] - Alternative gRPC-based trace ingestion
+* xref:ai-agents:observability/transcripts.adoc[]
+* xref:ai-agents:agents/monitor-agents.adoc[Observability for declarative agents]
+* xref:develop:connect/components/inputs/otlp_http.adoc[OTLP HTTP input reference] - Complete configuration options for the `otlp_http` component
+* xref:develop:connect/components/inputs/otlp_grpc.adoc[OTLP gRPC input reference] - Alternative gRPC-based trace ingestion