Problem
rulechef-savings only understands classification traffic: it compares the rule's label string to the observed llm_label (rulechef/savings.py, the chef.extract(...)["label"] comparison). NER/extraction traffic — where the LLM returns entity lists — can't be analyzed, even though that's exactly the TAB-style use case.
Current behavior
label = (out or {}).get("label") or ""
ok = str(label).strip().lower() == str(r.get("llm_label", "")).strip().lower()A traffic row with {"text": ..., "llm_entities": [...]} is silently treated as unanswered.
What to do
- Accept NER traffic rows:
{"text": ..., "llm_entities": [{"text","start","end","type"}, ...]}.
- Define "answered": rules produced a non-empty entity list. Define "agreement": span-set comparison against
llm_entities — reuse _match_entities from rulechef/evaluation.py (mode "text" or "partial") and report micro-F1 vs the LLM instead of exact string match.
- Keep the report layout; the per-rule table's "agreement" column becomes F1-vs-LLM for NER traffic.
- Auto-detect the traffic type from the first row (
llm_label vs llm_entities).
Acceptance
rulechef-savings --rules <TAB rules> --traffic <ner_traffic.jsonl> produces a report with sensible coverage/agreement; add a test next to tests/test_reports.py::test_savings_cli with a 2-row NER traffic file.
Start here
git clone https://github.com/KRLabsOrg/rulechef.git && cd rulechef
pip install -e ".[dev,grex]" && pytest tests/test_reports.py -q
rulechef-savings --rules benchmarks/results/results_banking77_new.json \
--traffic benchmarks/results/sample_traffic_banking77.jsonl --out s.html # the classification path that works today
Problem
rulechef-savingsonly understands classification traffic: it compares the rule'slabelstring to the observedllm_label(rulechef/savings.py, thechef.extract(...)["label"]comparison). NER/extraction traffic — where the LLM returns entity lists — can't be analyzed, even though that's exactly the TAB-style use case.Current behavior
A traffic row with
{"text": ..., "llm_entities": [...]}is silently treated as unanswered.What to do
{"text": ..., "llm_entities": [{"text","start","end","type"}, ...]}.llm_entities— reuse_match_entitiesfromrulechef/evaluation.py(mode"text"or"partial") and report micro-F1 vs the LLM instead of exact string match.llm_labelvsllm_entities).Acceptance
rulechef-savings --rules <TAB rules> --traffic <ner_traffic.jsonl>produces a report with sensible coverage/agreement; add a test next totests/test_reports.py::test_savings_cliwith a 2-row NER traffic file.Start here