Tracking LLM usage

[sky-2002]

Hi haystack team,
I have been using haystack for my workflows and I have components that make LLM API calls.
I wanted to ask how do I add tracking for these components so that I can see usage for every run.
Please point me to any resources or docs if any.

[sky-2002]

Also is there a way to cache component outputs, for use cases like starting from earlier state if some component fails etc, would love to know.

@julian-risch

[sjrl]

Hey @sky-2002 glad to hear you've been using Haystack! I'd recommend checking out our different tracing options like Langfuse. Depending on the LLM components that you are using you should see token tracking and cost tracking within the langfuse dashboard.

[sjrl]

For starting from an earlier state we do have a pipeline breakpoints feature coming soon to Haystack. You can find an experimental implementation of it in our haystack-experimental repo. Here is the discussion on it and here is a google colab for it.

[sky-2002]

Hey @SRJL
Thanks for the info, will check the options and breakpoints feature, thanks for the links and resources

[sjrl]

@sky-2002 moved this to be a discussion since that feels more relevant than an issue currenty

[onestardao]

Awesome to see this thread — I've run into this exact pain point while building multi-hop RAG pipelines.

Tracking usage is one thing... but making sense of which step derailed the semantic flow is a whole different beast, especially once dynamic retrieval + fallback reranking enters the picture.

We’ve been testing a “semantic traceback” system (kind of a WFGY-style symbolic path debugger) — it lets you not only track LLM/token costs, but reconstruct where the logic flow diverged, even when caching kicks in or prompts vary subtly.

If helpful, here’s how we do it:
https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md

(esp. Problem #17, which covers caching misalignment + shadow trace fallback)

Still early stage, but might be useful if you’re trying to debug RAG workflows with more than one failpoint. Happy to trade notes!

[e6o]

Adding to @sjrl's recommendation for Langfuse — here's a complementary approach that goes beyond tracking into actual cost reduction:

The tracking → reduction pipeline:

1. Track per-component costs first. Langfuse or similar tools will show you which Haystack components burn the most tokens. Typical multi-hop RAG pipelines have a cost distribution like:

   • Generator/LLM calls: 60-70% of total cost
   • Embedding generation: 15-20%
   • Reranking: 10-15%

2. Identify the "cost innefficiency gap". Once you see per-component costs, check if all LLM calls use the same model. If your generator uses GPT-4-class for everything (simple extraction, summarization, complex reasoning), you're overpaying by 2-3x.

3. Tier your models by task complexity:

   • Entity extraction, classification → smaller/cheaper model (GPT-4.1-mini, Haiku)
   • Summarization, rewriting → mid-tier (Sonnet, GPT-4.1)
   • Complex reasoning, multi-step → top-tier (GPT-5.5, Opus)

   In Haystack, you can set different generators per pipeline node. This alone typically cuts costs 40-50% with negligible quality impact.

4. For the caching question: Haystack's upcoming breakpoints feature helps, but for LLM response caching specifically, a routing proxy with built-in semantic caching (cache similar prompts and reuse responses) can eliminate 20-30% of redundant API calls entirely.

The biggest win isn't tracking costs — it's using the tracking data to route tasks to the right model. Track first, then optimize.