Description
The tacit_knowledge_curator agent, when invoked from IABENCH-v1 task
IA-2, returns zero retrieved documents for every query. nDCG@5 is 0.0 not
because retrieval is wrong but because the ChromaDB instance has no
documents indexed at runtime — the agent has nothing to retrieve from.
Surfaced by the PR 2 health-check run.
Steps to reproduce
- Check out main with PR 2 merged
$env:PYTHONPATH = "."
industrial-agents bench --suite all --provider ollama --model llama3.1:8b
- Inspect IA-2's per-query details —
retrieved_docs: [] for all queries
Expected behaviour
On framework startup or first agent invocation, the tacit_knowledge_curator
should index the synthetic SOPs (data/synthetic/sops/) and expert notes
(data/synthetic/expert_notes/) into its configured vector store, so that
queries return ranked results.
Actual behaviour
ChromaDB is empty. The agent runs without errors but every retrieval
returns []. IA-2 mean nDCG@5 = 0.0 across all queries.
Framework version
v0.1.0-pre (bench/iabench-retrieval-hallucination @ 2d2834f)
LLM provider
ollama
Environment
Windows 11, Python 3.12
Description
The
tacit_knowledge_curatoragent, when invoked from IABENCH-v1 taskIA-2, returns zero retrieved documents for every query. nDCG@5 is 0.0 not
because retrieval is wrong but because the ChromaDB instance has no
documents indexed at runtime — the agent has nothing to retrieve from.
Surfaced by the PR 2 health-check run.
Steps to reproduce
$env:PYTHONPATH = "."industrial-agents bench --suite all --provider ollama --model llama3.1:8bretrieved_docs: []for all queriesExpected behaviour
On framework startup or first agent invocation, the tacit_knowledge_curator
should index the synthetic SOPs (
data/synthetic/sops/) and expert notes(
data/synthetic/expert_notes/) into its configured vector store, so thatqueries return ranked results.
Actual behaviour
ChromaDB is empty. The agent runs without errors but every retrieval
returns []. IA-2 mean nDCG@5 = 0.0 across all queries.
Framework version
v0.1.0-pre (bench/iabench-retrieval-hallucination @ 2d2834f)
LLM provider
ollama
Environment
Windows 11, Python 3.12