Is production-scale RAG just duct tape over duct tape?

[onestardao]

Not here to rant, but yeah... kind of here to rant.

Every time I try to scale a RAG pipeline beyond a toy app, it just... cracks.

• Chunks are syntactically clean, but semantically weird.
• Context injection often feels helpful but ends up just confusing the model more.
• Latency stacks up like unpaid tech debt. You need 3 systems just to make 1 sentence land.

And even when everything works, it only works if you constantly monitor it like a toddler.

I'm not even against RAG — in fact, I want it to work.
But it feels like we're using LLMs as brains... then force-feeding them Wikipedia through a straw.

Is this a design mismatch?
Or are we just not doing it right?

I'm super curious what pain others have run into — especially anyone trying to make Haystack setups feel smooth, not just "technically working".

No pitch, no links, no hidden agenda.
Just tired fingers and context confusion.

[darfaz]

Felt this in my bones. Here's what made the biggest difference for us moving from "technically working" to actually smooth:

1. Stop treating retrieval as one step. We split it into retrieve → validate → rerank → inject. The validation layer catches the most insidious issue: retrieved chunks that accidentally (or adversarially) contain instructions that confuse the model. This is indirect prompt injection, and it's shockingly common once you're indexing user-generated content.

2. Latency is a pipeline design problem. Three things helped:

• Async retrieval with early termination (stop when you have enough good chunks)
• Smaller, more focused indices instead of one mega-index
• Caching embeddings aggressively (most queries cluster)

3. The monitoring toddler problem is real. We automated the babysitting by adding assertions at each pipeline stage:

• Post-retrieval: are chunks actually relevant? (cosine similarity threshold)
• Pre-injection: does the context contain anything suspicious? (injection patterns, PII)
• Post-generation: does the response leak source material or secrets?

For the security scanning part, ClawMoat slots in well as a Haystack component — it checks for prompt injection, PII leakage, and secret exposure with zero dependencies. Think of it as assertions for your RAG pipeline's trust boundaries.

The honest answer to your question: it's not a design mismatch, but RAG does require treating the pipeline as a system with failure modes at each joint, not just an LLM with extra context. The duct tape feeling comes from bolting on fixes instead of designing for failure upfront.

[darfaz]

Update: ClawMoat is now on npm — npm install -g clawmoat. If you want to quickly test your RAG pipeline's trust boundaries:

clawmoat scan 'your retrieved chunk here'  # check for injection
clawmoat test                                # 37 built-in attack patterns

Zero deps, sub-ms scans. Works well as a pre-commit hook or CI step to validate that your retrieval pipeline isn't surfacing adversarial content.