Skip to content

LOW_CODE_TRACING.md

Latest commit

 

History

History
294 lines (216 loc) · 7.58 KB

LOW_CODE_TRACING.md

File metadata and controls

294 lines (216 loc) · 7.58 KB
Raw

Kaizen Low-Code Tracing Guide

Enable Phoenix tracing for your LLM agents with minimal code changes.

Architecture Overview

flowchart TB
    subgraph Agent["Your Agent"]
        A1["import kaizen.auto"]
        A2["LLM Calls (OpenAI, LiteLLM, etc.)"]
    end
    
    subgraph Kaizen["kaizen.auto Module"]
        K1{"KAIZEN_AUTO_ENABLED?"}
        K2{"Already instrumented?"}
        K3["phoenix.otel.register()"]
        K4["OpenInference Instrumentors"]
    end
    
    subgraph Phoenix["Phoenix Server"]
        P1["Trace Collector"]
        P2["Trace Storage"]
    end
    
    subgraph Sync["kaizen sync phoenix"]
        S1["Fetch Traces"]
        S2["Generate Tips"]
        S3["Store in DB"]
    end
    
    A1 --> K1
    K1 -->|No| SKIP["No-op"]
    K1 -->|Yes| K2
    K2 -->|Yes| SKIP
    K2 -->|No| K3
    K3 --> K4
    K4 -.->|"Patches"| A2
    A2 -->|"Traces"| P1
    P1 --> P2
    P2 --> S1
    S1 --> S2
    S2 --> S3
Loading

Integration

Add one import at the top of your agent:

try:
    import kaizen.auto # noqa: F401
except ImportError:
    pass

# Your agent code - all LLM calls are now traced
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(...)

Environment variables:

export KAIZEN_AUTO_ENABLED=true
export KAIZEN_TRACING_PROJECT=my-agent  # Optional, defaults to "kaizen-agent"
export KAIZEN_TRACING_ENDPOINT=http://localhost:6006/v1/traces  # Optional

# For Kaizen example scripts only (e.g. examples/low_code/smolagents_demo.py):
export KAIZEN_EXAMPLE_AGENT_MODEL="Azure/gpt-4.1" # Overrides default tips model for agent execution

Note: Auto-patching will skip if existing tracing is detected. Use enable_tracing(force=True) to override.

Example: Simple OpenAI Script

Use this when you are tracing raw API calls directly using the openai library. Kaizen will capture the individual inputs and outputs of the LLM.

try:
    import kaizen.auto # noqa: F401
except ImportError:
    pass

from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Example: LiteLLM (Multi-Provider)

Use this when using LiteLLM to abstract across multiple providers. Kaizen traces the unified call interface.

try:
    import kaizen.auto # noqa: F401
except ImportError:
    pass

import litellm

# Call Azure
response = litellm.completion(
    model="Azure/gpt-4",
    messages=[{"role": "user", "content": "What is 2+2?"}]
)

# Call Anthropic
response = litellm.completion(
    model="claude-3-opus-20240229",
    messages=[{"role": "user", "content": "Explain recursion."}]
)

# All calls from any provider are traced!

Example: Smolagents (HuggingFace)

Use this for agentic workflows built with HuggingFace's smolagents. Kaizen traces the agent's steps, tool calls, and final answers.

try:
    import kaizen.auto # noqa: F401
except ImportError:
    pass

from smolagents import CodeAgent, HfApiModel

model = HfApiModel()
agent = CodeAgent(tools=[], model=model, max_steps=5)

# Agent execution is traced
result = agent.run("What is the capital of France?")
print(result)

Example: OpenAI Agents SDK

Use this for the OpenAI Agents framework (agents). Kaizen traces the high-level agent lifecycle, including runs, turns, and tool executions.

try:
    import kaizen.auto # noqa: F401
except ImportError:
    pass

from agents import Agent, Runner

agent = Agent(
    name="helper",
    instructions="You are a helpful assistant."
)

runner = Runner()
result = runner.run(agent, "Write a haiku about coding.")
print(result)

Testing & Validation

1. Start Phoenix Server

pip install arize-phoenix
phoenix serve
# Server runs at http://localhost:6006

2. Run Your Agent

KAIZEN_AUTO_ENABLED=true KAIZEN_TRACING_PROJECT=test-agent python your_agent.py

3. Verify Traces in Phoenix

# Check if project exists
curl http://localhost:6006/v1/projects

# Check spans
curl "http://localhost:6006/v1/projects/test-agent/spans?limit=5"

4. Sync to Kaizen

cd kaizen_repo
KAIZEN_BACKEND=filesystem \
KAIZEN_TIPS_MODEL="gpt-4" \
uv run python -m kaizen.frontend.cli.cli sync phoenix \
    --project test-agent \
    --include-errors

5. Verify Generated Tips

KAIZEN_BACKEND=filesystem \
uv run python -m kaizen.frontend.cli.cli entities list kaizen --type guideline

6. Understanding Tip Provenance (Metadata)

When Kaizen generates tips from traced trajectories (or from explicit save_trajectory calls), it automatically injects provenance metadata into the resulting guideline entities. This helps you track exactly where a tip came from and how it was created.

{
  "type": "guideline",
  "content": "Always verify the record exists before updating.",
  "metadata": {
    "creation_mode": "auto-phoenix",
    "source_task_id": "0df020ed0bd2e...",
    "source_span_id": "9218e1003f...",
    "category": "optimization"
  }
}
  • creation_mode: Describes the origin of the tip.
    • "auto-phoenix": Auto-generated from observability traces via kaizen sync phoenix.
    • "auto-mcp": Auto-generated when an agent directly calls the Kaizen save_trajectory MCP tool.
    • "manual": Explicitly created by a human or agent (e.g., via the create_entity MCP tool).
  • source_task_id: The originating trace ID (for Phoenix) or task ID (for MCP), linking the tip back to the specific execution that inspired it.

End-to-End Verification

Kaizen includes a comprehensive E2E verification suite to ensure that tracing and tip generation work correctly across all supported agents.

Running the E2E Pipeline

You can run the full regression suite using pytest:

KAIZEN_E2E=true uv run pytest tests/e2e/test_e2e_pipeline.py -s

Running Specific Tests

To test a specific agent framework:

# Test smolagents
KAIZEN_E2E=true uv run pytest tests/e2e/test_e2e_pipeline.py -k smolagents -s

# Test OpenAI Agents
KAIZEN_E2E=true uv run pytest tests/e2e/test_e2e_pipeline.py -k openai_agents -s

What It Tests

The pipeline performs the following for each agent:

  1. Executes the Agent: Run the agent script (e.g., smolagents_demo.py) with auto-instrumentation enabled.
  2. Verifies Traces: Checks the Phoenix server for the existence of traces in a unique, timestamped project.
  3. Generates Tips: Runs kaizen sync on the generated traces to verify that tips are successfully created from the agent's execution.

This ensures the entire "Agent -> Traces -> Tips" loop is functional.

Troubleshooting

Issue Solution
ModuleNotFoundError: kaizen.auto Install: pip install -e path/to/kaizen_repo or add to PYTHONPATH
No traces appearing Check KAIZEN_AUTO_ENABLED=true is set
Wrong project name Set KAIZEN_TRACING_PROJECT=your-name
Existing tracer conflict Use explicit mode with force=True
Phoenix not running Start with phoenix serve

Supported Frameworks

Kaizen.auto automatically instruments these frameworks when detected:

  • OpenAI (Example) - ChatCompletion, Completion, Embeddings
  • LiteLLM (Example) - All providers (Azure, Anthropic, etc.)
  • Smolagents (Example) - HuggingFace agents
  • OpenAI Agents SDK (Example) - OpenAI's agent framework