LLM_Operations_and_Observability.md

Back to the main index

LLM Operations and Observability

Tools for building, evaluating, and operating applications built on top of large language models — often grouped under the term LLMOps. These sit one layer above the models themselves: they help you assemble LLM and RAG pipelines, route and cache requests, trace what your application does in production, and measure output quality over time.

For the models and providers these tools run on, see Foundation Models; for autonomous-agent frameworks (CrewAI, LangGraph, AutoGen), see Automation.

Table of contents

• Application Frameworks
• Observability and Tracing
• Evaluation and Testing
• AI Gateways and Routing

Application Frameworks

Libraries for composing LLM calls, retrieval (RAG), and tools into applications. For agent-runtime frameworks, see the Automation section.

Tool	Description	License	Pricing
LangChain	A widely used framework for building LLM applications by chaining models, prompts, retrieval, and tools.
LlamaIndex	A data framework for connecting LLMs to your own data, focused on retrieval and RAG pipelines.
Haystack	An open-source framework from deepset for building production RAG and search pipelines.
DSPy	A framework from Stanford for programming (rather than prompting) and automatically optimizing LLM pipelines.
Pydantic AI	A type-safe Python agent framework from the Pydantic team, with structured, validated outputs.

Observability and Tracing

Tools to log, trace, and monitor LLM and agent applications in production — inspecting prompts, responses, latency, cost, and errors.

Tool	Description	License	Pricing
Langfuse	An open-source LLM engineering platform for tracing, prompt management, and evaluation.
LangSmith	LangChain's platform for tracing, testing, and monitoring LLM applications (works with or without LangChain).
Helicone	An open-source observability platform that proxies LLM requests to log, monitor, and cache them.
Arize Phoenix	An open-source tool for tracing, evaluating, and debugging LLM applications, with strong RAG support.
Lunary	An open-source platform for monitoring, prompt management, and analytics of LLM apps.
Langtrace	An open-source, OpenTelemetry-based tracing tool for LLM applications and frameworks.

Evaluation and Testing

Tools to measure output quality, regression-test prompts, and catch issues such as hallucinations before and after deployment.

Tool	Description	License	Pricing
Promptfoo	An open-source tool for testing, evaluating, and red-teaming prompts and LLM applications.
DeepEval	An open-source evaluation framework for LLM outputs, with a wide set of metrics and unit-test-style assertions.
Ragas	An open-source framework for evaluating retrieval-augmented generation (RAG) pipelines.
Braintrust	A platform for evaluating, testing, and iterating on LLM applications with datasets and scoring.
Confident AI	The cloud platform behind DeepEval, for running, tracking, and sharing LLM evaluations.
Galileo	An enterprise platform for evaluating and monitoring LLM and agent applications.

AI Gateways and Routing

Tools that sit between your application and the model providers to give you a single API, with routing, fallbacks, caching, rate limiting, and cost controls.

Tool	Description	License	Pricing
LiteLLM	An open-source gateway exposing 100+ providers behind a single OpenAI-compatible API, with routing and cost tracking.
Portkey	An open-source AI gateway with routing, caching, guardrails, and built-in observability.
Cloudflare AI Gateway	A gateway that adds caching, rate limiting, logging, and analytics in front of model providers.
OpenRouter	A unified API that routes requests across many proprietary and open models from a single account.

Back to the main index