diff --git a/content/academy/ai-engineering-loop.mdx b/content/academy/ai-engineering-loop.mdx new file mode 100644 index 000000000..f9812b193 --- /dev/null +++ b/content/academy/ai-engineering-loop.mdx @@ -0,0 +1,86 @@ +--- +title: AI Engineering Loop +description: A high-level map of the AI engineering lifecycle, from tracing and monitoring to building datasets, experimenting, and evaluating. +--- + +import { Activity, BadgeCheck, Database, FlaskConical, Route } from "lucide-react"; + +# The AI Engineering Loop + +The AI Engineering Loop is how teams approach the continuous evolution and improvement of their AI-powered systems. It connects what happens in production directly to the work of improving quality, cost, latency, and reliability during development. + +Many of the underlying concepts mirror traditional software engineering, but a key differentiator is the probabilistic nature of LLM outputs and the sheer number of paths a system can take. You cannot unit-test your way to confidence. You need a systematic way to observe, learn, and improve. + +![The AI engineering loop](/images/academy/loop-overview.png) + +The loop clusters into two areas of work. + +## 1. Understanding what's happening in production + +The first part is about visibility. What is your system actually doing in the real world? Which requests are going well, and which are failing in ways that matter? + + + } + arrow + > + Capture the full path of a request, including prompts, retrieved context, tool calls, outputs, latency, and cost. Tracing is the raw record of what your system actually did. + + } + arrow + > + Track how the system behaves over time and surface the traces that deserve attention. Monitoring turns a stream of raw data into an ongoing understanding of how the system evolves. + + + +## 2. Improving systematically during development + +The second part is about turning what you have observed into improvements you can trust — without degrading the parts of the system that are already working. + + + } + arrow + > + Turn real scenarios surfaced through monitoring into repeatable test cases. Instead of testing against a handful of hand-picked examples, you build a set that reflects how the system actually gets used. + + } + arrow + > + Change one variable at a time — a prompt, a model, a retrieval strategy — and compare it against a stable baseline. That way you know what actually improved instead of guessing. + + } + arrow + > + Decide whether results are good enough to ship using manual review, code-based checks, or LLM judges. Evaluation is how you turn a comparison into a decision. + + + +Once you ship a change, the cycle starts again. The updated system produces new traces, new monitoring signals, and new opportunities to improve. + +## You don't have to close the full loop on day one + +Most teams don't start with all five steps in place. That is fine. + +The value of the loop is cumulative. Each step you add gives you better signal, more systematic coverage, and more confidence in what you are shipping. The goal is not to implement everything at once — it is to understand where you are and take the next step toward closing the loop. + +{/* TODO: Link blog article about patterns of AI engineering lifecycle adoption once written */} + +## Start with tracing + +The natural place to begin is tracing. You cannot monitor what you cannot see, and you cannot improve what you cannot measure. Tracing is the foundation everything else builds on. + +[→ Start with Tracing](/academy/tracing) diff --git a/content/academy/datasets/meta.json b/content/academy/datasets/meta.json index 68592e861..6dd956a9b 100644 --- a/content/academy/datasets/meta.json +++ b/content/academy/datasets/meta.json @@ -1,5 +1,5 @@ { - "title": "Datasets", + "title": "Building Datasets", "pages": [ "overview" ] diff --git a/content/academy/evaluate/meta.json b/content/academy/evaluate/meta.json index f1bf37585..bd19c5454 100644 --- a/content/academy/evaluate/meta.json +++ b/content/academy/evaluate/meta.json @@ -1,5 +1,5 @@ { - "title": "Evaluate", + "title": "Evaluating", "pages": [ "overview" ] diff --git a/content/academy/experiments/meta.json b/content/academy/experiments/meta.json index 64169544b..c3e2443c7 100644 --- a/content/academy/experiments/meta.json +++ b/content/academy/experiments/meta.json @@ -1,5 +1,5 @@ { - "title": "Experiments", + "title": "Experimenting", "pages": [ "overview" ] diff --git a/content/academy/index.mdx b/content/academy/index.mdx index 2b9e99035..bdd33502c 100644 --- a/content/academy/index.mdx +++ b/content/academy/index.mdx @@ -1,44 +1,42 @@ --- title: Langfuse Academy -description: Build a mental model for AI engineering. Learn the core disciplines teams rely on as LLM applications move from prototype to production. +description: Understand why LLM engineering is different and how to navigate the full AI engineering lifecycle. --- # Welcome to Langfuse Academy -This is the place to build a mental model for AI engineering. We'll introduce the core disciplines teams rely on as LLM applications move from prototype to production. +Building with LLMs changes what it means for a system to work. Outputs are probabilistic. A system can run fine and still produce responses that are wrong, off-brand, or useless. Teams need to reason about quality, cost, latency, and the tradeoffs between them. -Rather than focusing on individual product features, Academy is meant to help you understand the bigger picture, and how teams can work with that change in a systematic way. +Langfuse Academy maps the AI engineering lifecycle so you understand how the pieces fit and what it takes to ship from prototype to production. -## Why LLM observability is different -Traditional observability remains essential. Teams still need to know whether their systems are up, whether requests are slow, whether dependencies are failing, and whether costs are under control. Those questions do not disappear when an application starts using LLMs. -But LLM applications introduce a different kind of challenge. Their behavior is probabilistic: the same input can produce different outputs, and a response can look plausible even when it is wrong, incomplete, off-brand, unsafe, or simply unhelpful. In other words, a request can succeed technically and still fail for the user. +## What you will find here - -TODO: insert a visual or an example here to break up the text - - -AI engineering is not only about reliability. It is also about quality. Teams need to understand whether the output was useful, grounded, safe, and worth the cost. Observability for LLM applications therefore sits closer to product quality and iteration than traditional application monitoring usually does. - -Modern observability platforms for LLM systems increasingly treat prompts, responses, token usage, quality signals, and model-specific behavior as first-class telemetry. +The Langfuse Academy follows the AI engineering lifecycle from first visibility into production behavior all the way to structured improvement and evaluation. The goal is to explain why each step exists, what problem it solves, and how the steps connect. -## The AI engineering loop +Start with [The AI Engineering Loop](/academy/ai-engineering-loop) for the high-level map, then go deeper into the individual parts: -Because of this, AI engineering is iterative. Teams do not build once, ship once, and assume the work is done. They observe behavior, learn from it, improve the system, and evaluate the result over time. +- [Tracing](/academy/tracing) +- [Monitoring](/academy/monitoring) +- [Building Datasets](/academy/datasets) +- [Experimenting](/academy/experiments) +- [Evaluating](/academy/evaluate) -![The AI engineering loop](/images/academy/loop-overview.png) +Some pages explain the high-level concepts. Others are deeper dives into individual parts of the lifecycle. You can read the full sequence or jump to the topic that is most relevant to your team right now. - -TODO: replace with final loop visual; should we explain each step in 1-2 sentences? Keep it concise - - -## What comes next +## Academy and docs do different jobs -The rest of Langfuse Academy goes deeper into each step of the loop. +Academy focuses on high-level concepts and how the lifecycle fits together. The [docs](/docs) and [guides](/guides) cover Langfuse features, product implementation details, and step-by-step how-tos. -Each section is designed to work on its own: it gives you an overview first, and then lets you go deeper if and when that makes sense for your use case. You can follow the full loop, or focus only on the parts that are most relevant for your team right now. +Use Academy to understand the lifecycle. Use the docs and guides when you are ready to implement it in Langfuse. -You also do not need to adopt everything at once. Most teams improve their setup iteratively over time, adding new practices as they become useful. Doing part of this loop is already better than having no LLM engineering practices at all. + +- AI engineers and software engineers building LLM applications and agentic systems +- Product managers who need to reason about quality, iteration, and tradeoffs +- Technical and business leaders who need a working understanding of how AI systems are built and improved +- AI agents that support humans in understanding AI engineering concepts and workflows + -Let's dive in! +## Why we are publishing this +Langfuse is open source, and we want to open source the conceptual side of AI engineering too. The Academy is our way of making the core ideas, vocabulary, and workflows behind LLM application development easier to access for everyone. diff --git a/content/academy/meta.json b/content/academy/meta.json index 895de021c..74f055207 100644 --- a/content/academy/meta.json +++ b/content/academy/meta.json @@ -2,6 +2,7 @@ "title": "Academy", "pages": [ "index", + "ai-engineering-loop", "---The Loop---", "tracing", "monitoring",