Skip to content

Commit 58f313f

Browse files
adris-misraclaude
adris-misra
and
claude
authored
bench(iabench): implement IA-2 retrieval and IA-5 hallucination tasks (#12)
* bench(iabench): implement IA-2 retrieval and IA-5 hallucination tasks - Add expert-notes corpus: EN-001 (motor maintenance) and EN-002 (hydraulic system) in data/synthetic/expert_notes/ - Add query set for IA-2: 12 operator queries with gold qrels in benchmarks/data/ia2_queries.json - Add grounded query set for IA-5: 18 queries with expected key facts and forbidden facts in benchmarks/data/ia5_grounded_queries.json - Implement _run_task_ia2(): invokes TacitKnowledgeCuratorAgent, normalises returned source_documents to corpus IDs, computes nDCG@5 per query - Implement _run_task_ia5(): invokes TacitKnowledgeCuratorAgent then LLM-as-judge to detect hallucinations and measure recall - Add _ndcg_at_5() and _normalize_doc_id() helpers for IA-2 scoring - Add _judge_hallucination() async helper for IA-5 LLM-as-judge - Add --judge-model CLI flag to bench command; same-model limitation documented in task_ia_5.yaml and code docstring - Update task_ia_2.yaml and task_ia_5.yaml: status→implemented, reference_implementation filled in, roadmap field removed - Update industrial_agent_benchmark.md: IA-2 and IA-5 marked Implemented - Fix pre-existing ruff PT001/PT023 errors in tests/unit/ (auto-fix) - Fix pre-existing mypy unused-ignore in sparkplug_client.py - Add TestNormalizeDocId and TestNdcgAt5 unit tests in test_iabench.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): remove spurious blank line in test_iabench imports (ruff I001) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): PT001 fixture parens and paho callback type alignment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * build(ci): pin ruff to 0.15.16 across CI, pre-commit, and dev deps; remove version-mismatch band-aids Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(types): use paho MQTTMessage type in on_message callback signature Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 35b5b09 commit 58f313f

20 files changed

Lines changed: 1001 additions & 98 deletions

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ jobs:
2020
with:
2121
python-version: "3.12"
2222
- name: Install ruff
23-
run: pip install ruff==0.4.7
23+
run: pip install ruff==0.15.16
2424
- name: Ruff check
2525
run: ruff check src/ tests/ benchmarks/ examples/
2626
- name: Ruff format check

‎.pre-commit-config.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ repos:
1515
args: ["--branch", "main"]
1616

1717
- repo: https://github.com/astral-sh/ruff-pre-commit
18-
rev: v0.4.7
18+
rev: v0.15.16
1919
hooks:
2020
- id: ruff
2121
args: ["--fix"]

‎benchmarks/data/ia2_queries.json‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
[
2+
{
3+
"id": "q001",
4+
"query": "What vibration RMS threshold triggers motor bearing replacement on motor_01?",
5+
"gold_labels": {
6+
"SOP-MAINT-001": 2,
7+
"EN-001": 1
8+
},
9+
"notes": "SOP-MAINT-001 states the 4.5 mm/s threshold directly; EN-001 corroborates with expert context."
10+
},
11+
{
12+
"id": "q002",
13+
"query": "What PPE is required when handling bearings during motor maintenance?",
14+
"gold_labels": {
15+
"SOP-MAINT-001": 2,
16+
"EN-001": 1
17+
},
18+
"notes": "SOP specifies nitrile gloves; EN-001 adds the reason (prevent skin-oil contamination)."
19+
},
20+
{
21+
"id": "q003",
22+
"query": "What is the part number for the drive end bearing on motor_01?",
23+
"gold_labels": {
24+
"SOP-MAINT-001": 2,
25+
"EN-001": 1
26+
},
27+
"notes": "Part BRG-6310-2RS appears in the SOP parts table; EN-001 reaffirms with additional context."
28+
},
29+
{
30+
"id": "q004",
31+
"query": "What temperature should bearings be heated to before installation, and what tool is used?",
32+
"gold_labels": {
33+
"SOP-MAINT-001": 2,
34+
"EN-001": 2
35+
},
36+
"notes": "SOP and EN-001 both specify 80-100 degrees C and induction heater TOOL-HEAT-01."
37+
},
38+
{
39+
"id": "q005",
40+
"query": "What hydraulic system pressure should be restored after filter replacement on hydraulic_01?",
41+
"gold_labels": {
42+
"SOP-MAINT-002": 2,
43+
"EN-002": 1
44+
},
45+
"notes": "SOP-MAINT-002 acceptance: 200 +/- 5 bar; EN-002 elaborates on troubleshooting pressure issues."
46+
},
47+
{
48+
"id": "q006",
49+
"query": "When should the hydraulic filter element be replaced according to the PM schedule?",
50+
"gold_labels": {
51+
"SOP-MAINT-002": 2,
52+
"EN-002": 1
53+
},
54+
"notes": "SOP-MAINT-002 states 2000-hour PM schedule; EN-002 notes typical filter life 1600-1800 hours."
55+
},
56+
{
57+
"id": "q007",
58+
"query": "What is the torque specification for the hydraulic filter bowl?",
59+
"gold_labels": {
60+
"SOP-MAINT-002": 2,
61+
"EN-002": 2
62+
},
63+
"notes": "Both SOP and EN-002 specify 35 Nm with torque wrench."
64+
},
65+
{
66+
"id": "q008",
67+
"query": "What qualifications and certifications are required to perform motor bearing replacement?",
68+
"gold_labels": {
69+
"SOP-MAINT-001": 2
70+
},
71+
"notes": "SOP-MAINT-001 section 2 specifies Electrical Journeyman (NFPA 70E) and Mechanical Technician Level 2."
72+
},
73+
{
74+
"id": "q009",
75+
"query": "What causes hydraulic filter clogging and what are the consequences of ignoring it?",
76+
"gold_labels": {
77+
"SOP-MAINT-002": 1,
78+
"EN-002": 2
79+
},
80+
"notes": "EN-002 gives expert detail on contamination causes; SOP-MAINT-002 references FM-HYD-003 failure mode."
81+
},
82+
{
83+
"id": "q010",
84+
"query": "How do I verify motor alignment after reinstallation, and what is the acceptance tolerance?",
85+
"gold_labels": {
86+
"SOP-MAINT-001": 2,
87+
"EN-001": 2
88+
},
89+
"notes": "Both specify < 0.05 mm TIR; EN-001 gives practical tips on using dial indicator."
90+
},
91+
{
92+
"id": "q011",
93+
"query": "What LOTO and safety steps are required before starting hydraulic filter maintenance?",
94+
"gold_labels": {
95+
"SOP-MAINT-002": 2,
96+
"SOP-MAINT-001": 1
97+
},
98+
"notes": "SOP-MAINT-002 covers hydraulic depressurization; SOP-MAINT-001 covers LOTO as a general reference."
99+
},
100+
{
101+
"id": "q012",
102+
"query": "What are the acceptance criteria for vibration and temperature after motor bearing replacement?",
103+
"gold_labels": {
104+
"SOP-MAINT-001": 2,
105+
"EN-001": 1
106+
},
107+
"notes": "SOP specifies vibration < 2.5 mm/s, bearing temp < 70 C above ambient; EN-001 describes the run-in procedure."
108+
}
109+
]

‎benchmarks/data/ia5_grounded_queries.json‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
[
2+
{
3+
"id": "g001",
4+
"question": "What vibration RMS threshold triggers bearing replacement on motor_01?",
5+
"source_doc": "SOP-MAINT-001",
6+
"expected_key_facts": ["4.5 mm/s", "4.5"],
7+
"forbidden_facts": ["5 mm/s", "3 mm/s", "3.5 mm/s", "6 mm/s"],
8+
"notes": "Exact value from SOP-MAINT-001 section 1: 'vibration RMS exceeds 4.5 mm/s'"
9+
},
10+
{
11+
"id": "g002",
12+
"question": "What is the part number for the drive end bearing on motor_01?",
13+
"source_doc": "SOP-MAINT-001",
14+
"expected_key_facts": ["BRG-6310-2RS", "6310-2RS"],
15+
"forbidden_facts": ["BRG-6208-2RS", "6208-2RS"],
16+
"notes": "Part table in SOP-MAINT-001: drive end is BRG-6310-2RS"
17+
},
18+
{
19+
"id": "g003",
20+
"question": "What is the torque specification for reinstalling the hydraulic filter bowl on hydraulic_01?",
21+
"source_doc": "SOP-MAINT-002",
22+
"expected_key_facts": ["35 Nm", "35"],
23+
"forbidden_facts": ["25 Nm", "45 Nm", "30 Nm", "50 Nm"],
24+
"notes": "SOP-MAINT-002 step 9: torque to 35 Nm"
25+
},
26+
{
27+
"id": "g004",
28+
"question": "What maximum temperature must the hydraulic oil be below before starting filter maintenance?",
29+
"source_doc": "SOP-MAINT-002",
30+
"expected_key_facts": ["50", "50°C", "50 C"],
31+
"forbidden_facts": ["60°C", "40°C", "70°C", "80°C"],
32+
"notes": "SOP-MAINT-002 safety precaution 3: allow oil to cool below 50 degrees C"
33+
},
34+
{
35+
"id": "g005",
36+
"question": "What is the normal operating pressure of the hydraulic system on hydraulic_01?",
37+
"source_doc": "SOP-MAINT-002",
38+
"expected_key_facts": ["200", "200 bar", "200 ± 5 bar"],
39+
"forbidden_facts": ["150 bar", "250 bar", "180 bar", "220 bar"],
40+
"notes": "SOP-MAINT-002 acceptance criteria and EN-002: 200 +/- 5 bar"
41+
},
42+
{
43+
"id": "g006",
44+
"question": "What vibration level must be achieved during the uncoupled run-in after motor bearing replacement?",
45+
"source_doc": "SOP-MAINT-001",
46+
"expected_key_facts": ["2.5 mm/s"],
47+
"forbidden_facts": ["4.5 mm/s", "5 mm/s", "3 mm/s"],
48+
"notes": "SOP-MAINT-001 step 9 and section 6: vibration < 2.5 mm/s"
49+
},
50+
{
51+
"id": "g007",
52+
"question": "What bearing temperature triggers replacement on motor_01?",
53+
"source_doc": "SOP-MAINT-001",
54+
"expected_key_facts": ["90°C", "90 C", "90"],
55+
"forbidden_facts": ["70°C", "80°C", "100°C", "60°C"],
56+
"notes": "SOP-MAINT-001 section 1: bearing temperature exceeds 90 degrees C"
57+
},
58+
{
59+
"id": "g008",
60+
"question": "What temperature range should bearings be heated to before installation?",
61+
"source_doc": "SOP-MAINT-001",
62+
"expected_key_facts": ["80", "100", "80-100", "80 to 100"],
63+
"forbidden_facts": ["150°C", "200°C", "50°C"],
64+
"notes": "SOP-MAINT-001 step 6 and EN-001: heat to 80-100 degrees C"
65+
},
66+
{
67+
"id": "g009",
68+
"question": "How long must the accumulator bleed valve be held open during hydraulic depressurization?",
69+
"source_doc": "SOP-MAINT-002",
70+
"expected_key_facts": ["30 second", "30-second", "30"],
71+
"forbidden_facts": ["60 second", "15 second", "45 second", "10 second"],
72+
"notes": "SOP-MAINT-002 step 3: manual bleed valve 30-second hold"
73+
},
74+
{
75+
"id": "g010",
76+
"question": "What is the maximum filter differential pressure accepted after hydraulic filter replacement?",
77+
"source_doc": "SOP-MAINT-002",
78+
"expected_key_facts": ["0.5 bar"],
79+
"forbidden_facts": ["2.5 bar", "1.0 bar", "1 bar", "2 bar"],
80+
"notes": "SOP-MAINT-002 step 10 and section 5: filter dP < 0.5 bar"
81+
},
82+
{
83+
"id": "g011",
84+
"question": "What is the minimum drain pan capacity required for hydraulic filter maintenance?",
85+
"source_doc": "SOP-MAINT-002",
86+
"expected_key_facts": ["5L", "5 L", "5 litre", "5 liter", "minimum 5"],
87+
"forbidden_facts": ["2L", "10L", "1L"],
88+
"notes": "SOP-MAINT-002 step 5: drain pan minimum 5L"
89+
},
90+
{
91+
"id": "g012",
92+
"question": "What qualifications are required to perform motor bearing replacement?",
93+
"source_doc": "SOP-MAINT-001",
94+
"expected_key_facts": ["Electrical Journeyman", "NFPA 70E", "Mechanical Technician"],
95+
"forbidden_facts": [],
96+
"notes": "SOP-MAINT-001 section 2: Electrical Journeyman (NFPA 70E) and Mechanical Technician Level 2"
97+
},
98+
{
99+
"id": "g013",
100+
"question": "What is the FMEA reference number for hydraulic filter clogging?",
101+
"source_doc": "SOP-MAINT-002",
102+
"expected_key_facts": ["FM-HYD-003"],
103+
"forbidden_facts": ["FM-MOT-001", "FM-HYD-001"],
104+
"notes": "SOP-MAINT-002 section 6: FM-HYD-003 (Filter Clog / Bypass)"
105+
},
106+
{
107+
"id": "g014",
108+
"question": "Which SOP covers bearing replacement for motor_01 and motor_02?",
109+
"source_doc": "SOP-MAINT-001",
110+
"expected_key_facts": ["SOP-MAINT-001"],
111+
"forbidden_facts": ["SOP-MAINT-002"],
112+
"notes": "SOP-MAINT-001 header: Applies To motor_01, motor_02"
113+
},
114+
{
115+
"id": "g015",
116+
"question": "What is the maximum bearing temperature rise above ambient allowed after motor bearing replacement?",
117+
"source_doc": "SOP-MAINT-001",
118+
"expected_key_facts": ["70°C", "70 C", "70"],
119+
"forbidden_facts": ["90°C", "50°C", "60°C", "40°C"],
120+
"notes": "SOP-MAINT-001 section 6: bearing temperature < 70 degrees C above ambient"
121+
},
122+
{
123+
"id": "g016",
124+
"question": "How many technicians are required to work on motors greater than 5.5 kW?",
125+
"source_doc": "SOP-MAINT-001",
126+
"expected_key_facts": ["two", "second technician", "2"],
127+
"forbidden_facts": ["one", "three", "alone"],
128+
"notes": "SOP-MAINT-001 safety precaution 5 and EN-001: second technician required for motors > 5.5 kW"
129+
},
130+
{
131+
"id": "g017",
132+
"question": "What grease is specified for use on motor bearings?",
133+
"source_doc": "SOP-MAINT-001",
134+
"expected_key_facts": ["Mobil Polyrex EM", "Polyrex EM"],
135+
"forbidden_facts": [],
136+
"notes": "SOP-MAINT-001 parts table: LUB-POLYREX (Mobil Polyrex EM)"
137+
},
138+
{
139+
"id": "g018",
140+
"question": "What is the part number for the non-drive end bearing on motor_01?",
141+
"source_doc": "SOP-MAINT-001",
142+
"expected_key_facts": ["BRG-6208-2RS", "6208-2RS"],
143+
"forbidden_facts": ["BRG-6310-2RS", "6310-2RS"],
144+
"notes": "SOP-MAINT-001 parts table: non-drive end is BRG-6208-2RS"
145+
}
146+
]

0 commit comments

Comments
 (0)