Skip to content

Commit 8c6ee9e

Browse files
Move profiler agent example from the primary repo
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
1 parent 7406837 commit 8c6ee9e

18 files changed

Lines changed: 1672 additions & 0 deletions

examples/profiler_agent/README.md

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
<!--
2+
SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
-->
17+
18+
# NeMo Agent Toolkit Profiler Agent
19+
20+
**Complexity:** 🟨 Intermediate
21+
22+
The profiler agent is a tool that allows you to analyze the performance of NeMo Agent Toolkit workflows. It uses the Phoenix server to store and retrieve traces of workflow runs.
23+
24+
> [!NOTE]
25+
> This example is a snapshot copy from `NVIDIA/NeMo-Agent-Toolkit` commit `c402fd35b73145e90db03c040d1aa652fbe7c32b`.
26+
27+
## Table of Contents
28+
29+
- [NeMo Agent Toolkit Profiler Agent](#nemo-agent-toolkit-profiler-agent)
30+
- [Table of Contents](#table-of-contents)
31+
- [Key Features](#key-features)
32+
- [Installation and Setup](#installation-and-setup)
33+
- [Install this Workflow:](#install-this-workflow)
34+
- [Set Up API Keys](#set-up-api-keys)
35+
- [Run the Workflow](#run-the-workflow)
36+
- [Features](#features)
37+
38+
## Key Features
39+
40+
- **Workflow Performance Analysis:** Demonstrates a specialized agent that analyzes NeMo Agent Toolkit workflow performance using Phoenix server traces for comprehensive performance monitoring.
41+
- **Token Usage Tracking:** Shows how to retrieve and analyze token consumption patterns across multiple workflow runs, providing insights into LLM resource utilization.
42+
- **Trace Visualization:** Generates flowcharts and visual representations of workflow execution patterns from stored Phoenix traces using natural language queries.
43+
- **Phoenix Server Integration:** Demonstrates integration with Phoenix observability platform for storing, retrieving, and analyzing workflow telemetry data.
44+
- **Natural Language Interface:** Provides a conversational interface for querying performance metrics, making complex trace analysis accessible through simple questions.
45+
46+
## Installation and Setup
47+
48+
If you have not already done so, follow the NeMo Agent Toolkit installation guide:
49+
https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/main/docs/source/get-started/installation.md
50+
51+
### Install this Workflow:
52+
53+
From the root directory of the NeMo Agent Toolkit Examples repository, run:
54+
55+
```bash
56+
uv pip install --prerelease=allow -e examples/profiler_agent
57+
```
58+
59+
### Set Up API Keys
60+
If you have not already done so, follow the API key instructions:
61+
https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/main/docs/source/get-started/quick-start.md#obtaining-api-keys
62+
63+
```bash
64+
export NVIDIA_API_KEY=<YOUR_API_KEY>
65+
```
66+
67+
## Run the Workflow
68+
69+
1. Start the Phoenix server if not already running. If you are using a remote Phoenix server, you can skip this step and modify the configs/config.yml file to point to the URL.
70+
```bash
71+
docker run -p 6006:6006 -p 4317:4317 -i -t arizephoenix/phoenix:latest
72+
```
73+
74+
2. Ensure that there are traces in the Phoenix server. You can use any workflow instrumented with Phoenix telemetry to generate traces.
75+
> [!NOTE]
76+
Then, run an instrumented workflow to generate traces.
77+
```bash
78+
# Example placeholder:
79+
# nat run --config_file <your-phoenix-enabled-config.yml> --input "<your test prompt>"
80+
```
81+
82+
3. Run the profiler agent:
83+
```
84+
nat serve --config_file=examples/profiler_agent/configs/config.yml
85+
```
86+
87+
4. Launch the NeMo Agent Toolkit UI by following:
88+
https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/main/docs/source/run-workflows/launching-ui.md
89+
90+
5. Query the agent with natural language via the UI:
91+
```
92+
Show me the token usage of last run
93+
```
94+
95+
![Sample Response](https://raw.githubusercontent.com/NVIDIA/NeMo-Agent-Toolkit/main/docs/source/_static/profiler-agent.png "Sample Response UI Image")
96+
97+
More examples:
98+
```
99+
Show me flowchart of last 3 runs
100+
```
101+
102+
```
103+
Analyze the last 2 runs
104+
```
105+
106+
## Features
107+
108+
- Query Phoenix traces with natural language
109+
- Analyze LLM application performance metrics
110+
- Generate trace visualizations
111+
- Extract user queries across trace spans

examples/profiler_agent/configs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
src/nat_profiler_agent/configs
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
[build-system]
17+
build-backend = "setuptools.build_meta"
18+
requires = ["setuptools >= 64", "setuptools-scm>=8"]
19+
20+
[tool.setuptools_scm]
21+
git_describe_command = "git describe --long --first-parent"
22+
root = "../.."
23+
24+
[tool.setuptools.packages.find]
25+
where = ["src"]
26+
27+
[project]
28+
name = "nat_profiler_agent"
29+
dynamic = ["version"]
30+
requires-python = ">=3.11,<3.14"
31+
description = "NeMo Agent Toolkit Profiler Agent"
32+
dependencies = [
33+
"nvidia-nat[langchain,profiling,opentelemetry,phoenix,ragaai,weave,test]>=1.4.0,<1.6.0",
34+
"arize-phoenix-client~=1.21",
35+
]
36+
keywords = ["ai", "rag", "agents"]
37+
classifiers = ["Programming Language :: Python"]
38+
authors = [{ name = "NVIDIA Corporation" }]
39+
maintainers = [{ name = "NVIDIA Corporation" }]
40+
41+
[project.entry-points.'nat.components']
42+
nat_profiler_agent = "nat_profiler_agent.register"
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
import logging
17+
import uuid
18+
from typing import Any
19+
from typing import TypedDict
20+
21+
from langchain_core.language_models import BaseChatModel
22+
from langchain_core.messages import AIMessage
23+
from langchain_core.messages import BaseMessage
24+
from langchain_core.messages import HumanMessage
25+
from langchain_core.messages import ToolMessage
26+
from langchain_core.output_parsers import PydanticOutputParser
27+
from langchain_core.runnables import RunnableConfig
28+
from langchain_core.tools import BaseTool
29+
from langgraph.graph import StateGraph
30+
31+
from nat_profiler_agent.data_models import ExecPlan
32+
from nat_profiler_agent.data_models import TraceInfo
33+
from nat_profiler_agent.tool.flow_chart import FlowChartOutput
34+
from nat_profiler_agent.tool.px_query import PxQueryOutput
35+
from nat_profiler_agent.tool.token_usage import TokenUsageOutput
36+
37+
logger = logging.getLogger(__name__)
38+
39+
40+
class ProfilerAgentState(TypedDict):
41+
"""State for the ProfilerAgent."""
42+
43+
exec_plan: ExecPlan
44+
messages: list[BaseMessage]
45+
df_path: str | None = None
46+
trace_infos: dict[str, TraceInfo] | None = None
47+
end_condition: bool = False
48+
retry_count: int = 0
49+
user_query: str | None = None
50+
51+
52+
class ProfilerAgent:
53+
"""Agent for profiling LLM traces."""
54+
55+
def __init__(
56+
self,
57+
llm: BaseChatModel,
58+
tools: dict[str, BaseTool],
59+
response_composer_tool: BaseTool,
60+
detailed_logs: bool = False,
61+
max_retries: int = 3,
62+
retry_prompt: str = "",
63+
):
64+
self.llm = llm
65+
self.detailed_logs = detailed_logs
66+
self.tools = tools
67+
self.callbacks = []
68+
self.max_retries = max_retries
69+
self.retry_prompt = retry_prompt
70+
# pydantic parser
71+
self.output_parser = PydanticOutputParser(pydantic_object=ExecPlan)
72+
self.response_composer = response_composer_tool
73+
self.graph = None
74+
logger.info("ProfilerAgent initialized")
75+
76+
async def conditional_edge(self, state: ProfilerAgentState):
77+
try:
78+
logger.debug("Starting the Tool Calling Conditional Edge")
79+
if "exec_plan" in state and len(state["exec_plan"].tools) > 0:
80+
return "executor"
81+
else:
82+
return "response_composer"
83+
except Exception as ex:
84+
if "retry_count" in state and state["retry_count"] >= self.max_retries:
85+
logger.warning("Max retries reached, returning without meaningful output")
86+
state["messages"].append(AIMessage(content="No meaningful output, please try again with another query"))
87+
return "__end__"
88+
else:
89+
state.setdefault("retry_count", 1)
90+
logger.warning(
91+
"Error in the conditional edge: %s, retrying %d times out of %d",
92+
ex,
93+
state["retry_count"],
94+
self.max_retries,
95+
)
96+
return "agent"
97+
98+
async def build_graph(self):
99+
try:
100+
logger.debug("Building and compiling the Agent Graph")
101+
graph = StateGraph(ProfilerAgentState)
102+
graph.add_node("agent", self.agent_node)
103+
graph.add_node("response_composer", self.response_composer_node)
104+
graph.add_node("executor", self.executor_node)
105+
graph.add_conditional_edges(
106+
"agent",
107+
self.conditional_edge,
108+
)
109+
graph.add_conditional_edges(
110+
"executor",
111+
self.conditional_edge,
112+
)
113+
graph.set_entry_point("agent")
114+
graph.set_finish_point("response_composer")
115+
self.graph = graph.compile()
116+
logger.info("ProfilerAgent Graph built and compiled successfully")
117+
return self.graph
118+
except Exception as ex:
119+
logger.error("Failed to build ProfilerAgent Graph: %s", ex)
120+
raise
121+
122+
async def agent_node(self, state: ProfilerAgentState):
123+
try:
124+
logger.debug("Starting Agent Node")
125+
logger.info("Calling agent to plan the execution")
126+
if len(state["messages"]) == 0:
127+
raise RuntimeError('No input received in state: "messages"')
128+
129+
response = await self.llm.ainvoke(state["messages"], config=RunnableConfig(callbacks=self.callbacks))
130+
if self.detailed_logs:
131+
logger.debug("The agent's input was:\n%s", state["messages"])
132+
logger.debug("The agent's output is:\n%s", response)
133+
# parse the response to get the exec_plan
134+
try:
135+
exec_plan = self.output_parser.parse(response.text())
136+
logger.info("Agent planned the execution: %s", exec_plan)
137+
state["exec_plan"] = exec_plan
138+
except Exception as ex:
139+
logger.warning("Failed to parse the agent's output: %s", response.text())
140+
state.setdefault("retry_count", 0)
141+
message = self.retry_prompt.format(error=ex, output_parser=self.output_parser.get_format_instructions())
142+
state["messages"].append(HumanMessage(content=message))
143+
return state
144+
except Exception as ex:
145+
logger.error("Failed to call agent_node: %s", ex)
146+
raise
147+
148+
async def executor_node(self, state: ProfilerAgentState):
149+
# check if the tool is px_query
150+
try:
151+
if state["exec_plan"].tools[0] == "px_query":
152+
query_result = await self.tools["px_query"].ainvoke(input={**state["exec_plan"].model_dump()})
153+
self.update_state(state, query_result)
154+
state["exec_plan"].tools.popleft()
155+
else:
156+
tool_name = state["exec_plan"].tools.popleft()
157+
tool_result = await self.tools[tool_name].ainvoke(input={"df_path": state["df_path"]})
158+
self.update_state(state, tool_result)
159+
except Exception as ex:
160+
logger.error("Failed to call executor_node: %s", ex)
161+
raise
162+
return state
163+
164+
async def response_composer_node(self, state: ProfilerAgentState):
165+
try:
166+
if len(state["trace_infos"]) == 0:
167+
state["messages"].append(HumanMessage(content="No traces retrieved. Exiting..."))
168+
else:
169+
tool_response = await self.response_composer.ainvoke(input={"trace_infos": state["trace_infos"]})
170+
self.update_state(state, tool_response)
171+
return state
172+
except Exception as ex:
173+
logger.error("Failed to call response_composer_node: %s", ex)
174+
raise
175+
176+
def update_state(self, state: ProfilerAgentState, tool_response: Any) -> ProfilerAgentState:
177+
"""Update the state with the tool response."""
178+
match tool_response:
179+
case PxQueryOutput():
180+
state["df_path"] = tool_response.df_path
181+
for trace_id, user_query in tool_response.user_queries.items():
182+
state["trace_infos"].setdefault(trace_id, TraceInfo()).user_query = user_query
183+
state["messages"].append(
184+
HumanMessage(content=f"PxQuery returned a PxDataFrame with {tool_response.row_count} rows"
185+
f"you can use it to analyze the traces by calling tools, you can omit the "
186+
f"dataframe parameter in tool calls, it will be automatically added. "
187+
f"Don't call px_query tool again! "
188+
f"You should call all analysis tools unless user specifies otherwise."))
189+
190+
case FlowChartOutput():
191+
# update the trace_infos with the flow chart
192+
for trace_id, flow_info in tool_response.trace_id_to_flow_info.items():
193+
state["trace_infos"].setdefault(trace_id, TraceInfo()).flow_info = flow_info
194+
num_traces = len(tool_response.trace_id_to_flow_info)
195+
state["messages"].append(
196+
HumanMessage(content=f"FlowChartOutput returned a FlowChartOutput with {num_traces} traces"))
197+
case TokenUsageOutput():
198+
# update the trace_infos with the token usage
199+
for trace_id, token_usage in tool_response.trace_id_to_token_usage.items():
200+
state["trace_infos"].setdefault(trace_id, TraceInfo()).token_usage_info = token_usage
201+
state["messages"].append(
202+
HumanMessage(content=f"TokenUsageOutput returned a TokenUsageOutput with "
203+
f"{len(tool_response.trace_id_to_token_usage)} traces"))
204+
case str():
205+
state["messages"].append(ToolMessage(content=tool_response, tool_call_id=uuid.uuid4()))
206+
case _:
207+
raise ValueError(f"Unsupported tool response type: {type(tool_response)}")
208+
209+
return state

0 commit comments

Comments
 (0)