Skip to content

Commit 1f8265b

Browse files
guyernestclaude
guyernest
and
claude
committed
feat(browser-agent): Add select_by_llm action and harden persistent server mode (v0.5.2)
- Add select_by_llm action: extracts all <select> options via DOM, sends to text-only LLM for matching, uses select_option() to fire native change event (fixes Angular submit button staying disabled) - Harden server mode: wrap execute_script in try/except to prevent unhandled errors from killing the persistent Python process - Add browser health check with auto-recovery in server mode loop - Use json.dumps(default=str) to handle non-serializable result objects - Update address selection strategies: regex (fast) -> select_by_llm (reliable) -> vision LLM (fallback) - Bump version to 0.5.2 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 82d42b4 commit 1f8265b

7 files changed

Lines changed: 259 additions & 176 deletions

File tree

‎lambda/tools/local-browser-agent/examples/03_bt_broadband_bournemouth.json‎

Lines changed: 8 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -550,7 +550,7 @@
550550
"description": "Progressive address matching: regex -> fuzzy -> vision with scroll",
551551
"strategies": [
552552
{
553-
"name": "Strategy 1: Regex - building (with range) + first word only",
553+
"name": "Strategy 1: Regex - building number + first word",
554554
"steps": [
555555
{
556556
"action": "click",
@@ -563,45 +563,19 @@
563563
]
564564
},
565565
{
566-
"name": "Strategy 2: Regex - building (with range) + street flexible",
566+
"name": "Strategy 2: DOM extract + LLM text matching",
567567
"steps": [
568568
{
569-
"action": "click",
570-
"locator": {
571-
"strategy": "selector",
572-
"value": "text=/(?:\\d+-)?143(?:-\\d+)?[A-Za-z]?[\\s,]+Belle Vue Road/i",
573-
"nth": 0
574-
}
575-
}
576-
]
577-
},
578-
{
579-
"name": "Strategy 3: Regex - building (with range) + street anywhere",
580-
"steps": [
581-
{
582-
"action": "click",
583-
"locator": {
584-
"strategy": "selector",
585-
"value": "text=/(?:\\d+-)?143(?:-\\d+)?[A-Za-z]?.*Belle Vue Road/i",
586-
"nth": 0
587-
}
588-
}
589-
]
590-
},
591-
{
592-
"name": "Strategy 4: Exact text match",
593-
"steps": [
594-
{
595-
"action": "click",
596-
"locator": {
597-
"strategy": "text",
598-
"value": "143 Belle Vue Road"
599-
}
569+
"action": "select_by_llm",
570+
"description": "Extract all options from address dropdown, send to LLM for text matching, use select_option() to properly trigger Angular change event",
571+
"select_selector": "select.custom-select",
572+
"target": "143 Belle Vue Road, Bournemouth BH6 3EN",
573+
"context_hint": "Building number: 143, Street: Belle Vue Road, Postcode: BH6 3EN"
600574
}
601575
]
602576
},
603577
{
604-
"name": "Strategy 5: Vision LLM with scroll capability",
578+
"name": "Strategy 3: Vision LLM with scroll capability (fallback)",
605579
"escalate": {
606580
"type": "vision_llm",
607581
"prompt": "Find and click the address matching '143 Belle Vue Road, Bournemouth BH6 3EN' in the address list. CRITICAL INSTRUCTIONS:\n\n1. FIRST SEARCH: Look carefully at ALL visible addresses for one containing:\n - Building number: '143'\n - Street name: 'Belle Vue Road'\n - Format may vary (case, commas, word order)\n\n2. IF FOUND: Click it immediately\n\n3. IF NOT VISIBLE:\n a) The list is likely SCROLLABLE - look for:\n - A dropdown/select element that can be scrolled\n - Scroll arrows (up/down)\n - A scrollbar on the address list\n b) SCROLL DOWN within the list (not the whole page)\n c) After scrolling, SEARCH AGAIN for the address\n d) Repeat: scroll -> search -> scroll -> search (up to 3 scroll attempts)\n\n4. MATCHING RULES:\n - Accept variations: '143 Belle Vue Road', '143, Belle Vue Road', '143 Belle Vue Road, City'\n - Case-insensitive matching\n - Must have BOTH building number AND street name\n\n5. MAX ACTIONS: 8 actions total (including scrolls and clicks)\n\n6. IF STILL NOT FOUND: Report failure.",

‎lambda/tools/local-browser-agent/examples/04_bt_broadband_bolton.json‎

Lines changed: 8 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -550,7 +550,7 @@
550550
"description": "Progressive address matching: regex -> fuzzy -> vision with scroll",
551551
"strategies": [
552552
{
553-
"name": "Strategy 1: Regex - building (with range) + first word only",
553+
"name": "Strategy 1: Regex - building number + first word",
554554
"steps": [
555555
{
556556
"action": "click",
@@ -563,45 +563,19 @@
563563
]
564564
},
565565
{
566-
"name": "Strategy 2: Regex - building (with range) + street flexible",
566+
"name": "Strategy 2: DOM extract + LLM text matching",
567567
"steps": [
568568
{
569-
"action": "click",
570-
"locator": {
571-
"strategy": "selector",
572-
"value": "text=/(?:\\d+-)?34(?:-\\d+)?[A-Za-z]?[\\s,]+Market Street/i",
573-
"nth": 0
574-
}
575-
}
576-
]
577-
},
578-
{
579-
"name": "Strategy 3: Regex - building (with range) + street anywhere",
580-
"steps": [
581-
{
582-
"action": "click",
583-
"locator": {
584-
"strategy": "selector",
585-
"value": "text=/(?:\\d+-)?34(?:-\\d+)?[A-Za-z]?.*Market Street/i",
586-
"nth": 0
587-
}
588-
}
589-
]
590-
},
591-
{
592-
"name": "Strategy 4: Exact text match",
593-
"steps": [
594-
{
595-
"action": "click",
596-
"locator": {
597-
"strategy": "text",
598-
"value": "34 Market Street"
599-
}
569+
"action": "select_by_llm",
570+
"description": "Extract all options from address dropdown, send to LLM for text matching, use select_option() to properly trigger Angular change event",
571+
"select_selector": "select.custom-select",
572+
"target": "34-40 Market Street, Westhoughton, Bolton BL5 3AN",
573+
"context_hint": "Building number: 34, Street: Market Street, Postcode: BL5 3AN"
600574
}
601575
]
602576
},
603577
{
604-
"name": "Strategy 5: Vision LLM with scroll capability",
578+
"name": "Strategy 3: Vision LLM with scroll capability (fallback)",
605579
"escalate": {
606580
"type": "vision_llm",
607581
"prompt": "Find and click the address matching '34-40 Market Street, Westhoughton, Bolton BL5 3AN' in the address list. CRITICAL INSTRUCTIONS:\n\n1. FIRST SEARCH: Look carefully at ALL visible addresses for one containing:\n - Building number: '34'\n - Street name: 'Market Street'\n - Format may vary (case, commas, word order)\n\n2. IF FOUND: Click it immediately\n\n3. IF NOT VISIBLE:\n a) The list is likely SCROLLABLE - look for:\n - A dropdown/select element that can be scrolled\n - Scroll arrows (up/down)\n - A scrollbar on the address list\n b) SCROLL DOWN within the list (not the whole page)\n c) After scrolling, SEARCH AGAIN for the address\n d) Repeat: scroll -> search -> scroll -> search (up to 3 scroll attempts)\n\n4. MATCHING RULES:\n - Accept variations: '34 Market Street', '34, Market Street', '34 Market Street, City'\n - Case-insensitive matching\n - Must have BOTH building number AND street name\n\n5. MAX ACTIONS: 8 actions total (including scrolls and clicks)\n\n6. IF STILL NOT FOUND: Report failure.",

‎lambda/tools/local-browser-agent/examples/05_bt_broadband_peacehaven.json‎

Lines changed: 8 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -550,7 +550,7 @@
550550
"description": "Progressive address matching: regex -> fuzzy -> vision with scroll",
551551
"strategies": [
552552
{
553-
"name": "Strategy 1: Regex - building (with range) + first word only",
553+
"name": "Strategy 1: Regex - building number + first word",
554554
"steps": [
555555
{
556556
"action": "click",
@@ -563,45 +563,19 @@
563563
]
564564
},
565565
{
566-
"name": "Strategy 2: Regex - building (with range) + street flexible",
566+
"name": "Strategy 2: DOM extract + LLM text matching",
567567
"steps": [
568568
{
569-
"action": "click",
570-
"locator": {
571-
"strategy": "selector",
572-
"value": "text=/(?:\\d+-)?282(?:-\\d+)?[A-Za-z]?[\\s,]+South Coast Road/i",
573-
"nth": 0
574-
}
575-
}
576-
]
577-
},
578-
{
579-
"name": "Strategy 3: Regex - building (with range) + street anywhere",
580-
"steps": [
581-
{
582-
"action": "click",
583-
"locator": {
584-
"strategy": "selector",
585-
"value": "text=/(?:\\d+-)?282(?:-\\d+)?[A-Za-z]?.*South Coast Road/i",
586-
"nth": 0
587-
}
588-
}
589-
]
590-
},
591-
{
592-
"name": "Strategy 4: Exact text match",
593-
"steps": [
594-
{
595-
"action": "click",
596-
"locator": {
597-
"strategy": "text",
598-
"value": "282 South Coast Road"
599-
}
569+
"action": "select_by_llm",
570+
"description": "Extract all options from address dropdown, send to LLM for text matching, use select_option() to properly trigger Angular change event",
571+
"select_selector": "select.custom-select",
572+
"target": "282 South Coast Road, Peacehaven BN10 8LA",
573+
"context_hint": "Building number: 282, Street: South Coast Road, Postcode: BN10 8LA"
600574
}
601575
]
602576
},
603577
{
604-
"name": "Strategy 5: Vision LLM with scroll capability",
578+
"name": "Strategy 3: Vision LLM with scroll capability (fallback)",
605579
"escalate": {
606580
"type": "vision_llm",
607581
"prompt": "Find and click the address matching '282 South Coast Road, Peacehaven BN10 8LA' in the address list. CRITICAL INSTRUCTIONS:\n\n1. FIRST SEARCH: Look carefully at ALL visible addresses for one containing:\n - Building number: '282'\n - Street name: 'South Coast Road'\n - Format may vary (case, commas, word order)\n\n2. IF FOUND: Click it immediately\n\n3. IF NOT VISIBLE:\n a) The list is likely SCROLLABLE - look for:\n - A dropdown/select element that can be scrolled\n - Scroll arrows (up/down)\n - A scrollbar on the address list\n b) SCROLL DOWN within the list (not the whole page)\n c) After scrolling, SEARCH AGAIN for the address\n d) Repeat: scroll -> search -> scroll -> search (up to 3 scroll attempts)\n\n4. MATCHING RULES:\n - Accept variations: '282 South Coast Road', '282, South Coast Road', '282 South Coast Road, City'\n - Case-insensitive matching\n - Must have BOTH building number AND street name\n\n5. MAX ACTIONS: 8 actions total (including scrolls and clicks)\n\n6. IF STILL NOT FOUND: Report failure.",

‎lambda/tools/local-browser-agent/examples/06_bt_broadband_template_test.json‎

Lines changed: 9 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -550,58 +550,32 @@
550550
"description": "Progressive address matching: regex -> fuzzy -> vision with scroll",
551551
"strategies": [
552552
{
553-
"name": "Strategy 1: Regex - building (with range) + first word only",
553+
"name": "Strategy 1: Regex - building number + first word",
554554
"steps": [
555555
{
556556
"action": "click",
557557
"locator": {
558558
"strategy": "selector",
559-
"value": "text=/^\\s*(?:\\d+-)?396(?:-\\d+)?[A-Za-z]?\\s+\\w+/i",
559+
"value": "text=/^\\s*(?:\\d+-)?{{building_number}}(?:-\\d+)?[A-Za-z]?\\s+\\w+/i",
560560
"nth": 0
561561
}
562562
}
563563
]
564564
},
565565
{
566-
"name": "Strategy 2: Regex - building (with range) + street flexible",
566+
"name": "Strategy 2: DOM extract + LLM text matching",
567567
"steps": [
568568
{
569-
"action": "click",
570-
"locator": {
571-
"strategy": "selector",
572-
"value": "text=/(?:\\d+-)?396(?:-\\d+)?[A-Za-z]?[\\s,]+Wimborne Road/i",
573-
"nth": 0
574-
}
575-
}
576-
]
577-
},
578-
{
579-
"name": "Strategy 3: Regex - building (with range) + street anywhere",
580-
"steps": [
581-
{
582-
"action": "click",
583-
"locator": {
584-
"strategy": "selector",
585-
"value": "text=/(?:\\d+-)?396(?:-\\d+)?[A-Za-z]?.*Wimborne Road/i",
586-
"nth": 0
587-
}
588-
}
589-
]
590-
},
591-
{
592-
"name": "Strategy 4: Exact text match",
593-
"steps": [
594-
{
595-
"action": "click",
596-
"locator": {
597-
"strategy": "text",
598-
"value": "93 Wimborne Road"
599-
}
569+
"action": "select_by_llm",
570+
"description": "Extract all options from address dropdown, send to LLM for text matching, use select_option() to properly trigger Angular change event",
571+
"select_selector": "select.custom-select",
572+
"target": "93 Wimborne Road, Bournemouth BH8 8BN",
573+
"context_hint": "Building number: 93, Street: Wimborne Road, Postcode: BH8 8BN"
600574
}
601575
]
602576
},
603577
{
604-
"name": "Strategy 5: Vision LLM with scroll capability",
578+
"name": "Strategy 3: Vision LLM with scroll capability (fallback)",
605579
"escalate": {
606580
"type": "vision_llm",
607581
"prompt": "Find and click the address matching '93 Wimborne Road, Bournemouth BH8 8BN' in the address list. CRITICAL INSTRUCTIONS:\n\n1. FIRST SEARCH: Look carefully at ALL visible addresses for one containing:\n - Building number: '93'\n - Street name: 'Wimborne Road'\n - Format may vary (case, commas, word order)\n\n2. IF FOUND: Click it immediately\n\n3. IF NOT VISIBLE:\n a) The list is likely SCROLLABLE - look for:\n - A dropdown/select element that can be scrolled\n - Scroll arrows (up/down)\n - A scrollbar on the address list\n b) SCROLL DOWN within the list (not the whole page)\n c) After scrolling, SEARCH AGAIN for the address\n d) Repeat: scroll -> search -> scroll -> search (up to 3 scroll attempts)\n\n4. MATCHING RULES:\n - Accept variations: '93 Wimborne Road', '93, Wimborne Road', '93 Wimborne Road, City'\n - Case-insensitive matching\n - Must have BOTH building number AND street name\n\n5. MAX ACTIONS: 8 actions total (including scrolls and clicks)\n\n6. IF STILL NOT FOUND: Report failure.",

0 commit comments

Comments
 (0)