Skip to content

Commit 10fa90e

Browse files
guyernestclaude
andcommitted
feat(browser-agent): Add tag-based profile resolution and security checks
Profile Resolution: - Integrated ProfileManager into OpenAI Playwright Executor - Added _resolve_profile_from_script() method with centralized resolution - Resolution priority: CLI --user-data-dir > --profile > session.profile_name > session.required_tags > temporary profile > error - Tag matching uses AND logic (profile must have ALL required tags) - Most recently used profile selected when multiple profiles match Security: - Validates profile directory exists before browser launch - Fails with helpful error if no profile found and allow_temp_profile: false - Clear logging at each step of profile resolution - Guides users on how to create profiles with required tags CLI Enhancements: - Added --profile flag for profile name lookup - Added --list-profiles flag to list available profiles - Improved help text with examples and priority documentation Example Fixes: - Fixed 'type' vs 'action' field usage across all example scripts - Updated escalation test to use example.com (more reliable than Google) - Fixed execute_js script syntax (removed invalid 'return') Template Updates: - Added required_tags and allow_temp_profile to bt_broadband template - Updated workflow examples to demonstrate tag-based selection - Added profile selection documentation to README 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent abf1c92 commit 10fa90e

14 files changed

Lines changed: 775 additions & 147 deletions

lambda/tools/local-browser-agent/examples/README_PROGRESSIVE_EXAMPLES.md

Lines changed: 64 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,8 @@ This directory contains example scripts demonstrating the Progressive Escalation
3535
**Migration from Nova Act**:
3636
| Old (Nova Act) | New (Progressive) | Improvement |
3737
|----------------|-------------------|-------------|
38-
| `action: "act_with_schema"` | `type: "extract"` + `method: "vision"` | Explicit, predictable |
39-
| `action: "act"` for navigation | `type: "navigate"` | Clearer intent |
38+
| `action: "act_with_schema"` | `action: "extract"` + `method: "vision"` | Explicit, predictable |
39+
| `action: "act"` for navigation | `action: "navigate"` | Clearer intent |
4040
| Single agent call for clicks | Progressive escalation chain | 10x faster, free |
4141

4242
**Key improvements**:
@@ -106,7 +106,7 @@ This directory contains example scripts demonstrating the Progressive Escalation
106106
### New Format (Progressive Escalation)
107107
```json
108108
{
109-
"type": "fill",
109+
"action": "fill",
110110
"description": "Fill customer name field",
111111
"escalation_chain": [
112112
{
@@ -245,7 +245,7 @@ python openai_playwright_executor.py \
245245

246246
// New
247247
{
248-
"type": "click",
248+
"action": "click",
249249
"description": "Click Next button",
250250
"escalation_chain": [
251251
{"method": "playwright_locator", "locator": {"strategy": "text", "value": "Next"}},
@@ -261,7 +261,7 @@ python openai_playwright_executor.py \
261261

262262
// New
263263
{
264-
"type": "fill",
264+
"action": "fill",
265265
"escalation_chain": [
266266
{"method": "playwright_locator", "locator": {"strategy": "selector", "value": "input[type='email']"}},
267267
{"method": "vision_find_element", "prompt": "Find the email field"}
@@ -276,7 +276,7 @@ python openai_playwright_executor.py \
276276
{"action": "act_with_schema", "prompt": "Extract book titles", "schema": {...}}
277277

278278
// New
279-
{"type": "extract", "method": "vision", "prompt": "Extract book titles", "schema": {...}}
279+
{"action": "extract", "method": "vision", "prompt": "Extract book titles", "schema": {...}}
280280
```
281281

282282
6. **Add wait steps after navigation**:
@@ -287,11 +287,67 @@ python openai_playwright_executor.py \
287287

288288
7. **Add screenshots for debugging**:
289289
```json
290-
{"type": "screenshot", "save_to": "01_step_name.png"}
290+
{"action": "screenshot", "save_to": "01_step_name.png"}
291291
```
292292

293293
---
294294

295+
## Browser Profile Selection
296+
297+
Scripts can specify which browser profile to use for authenticated sessions.
298+
299+
### Session Configuration
300+
301+
Add a `session` block to your script to specify profile requirements:
302+
303+
```json
304+
{
305+
"name": "My Script",
306+
"session": {
307+
"profile_name": "My-Profile",
308+
"required_tags": ["example.com", "authenticated"],
309+
"allow_temp_profile": false,
310+
"clone_for_parallel": false
311+
},
312+
"steps": [...]
313+
}
314+
```
315+
316+
### Profile Resolution Priority
317+
318+
1. **Command-line `--user-data-dir`**: Explicit path (highest priority)
319+
2. **Command-line `--profile`**: Profile name lookup
320+
3. **Script `session.profile_name`**: Exact name match
321+
4. **Script `session.required_tags`**: Tag-based matching (AND logic)
322+
5. **Temporary profile**: If `allow_temp_profile: true` (default)
323+
6. **Error**: If no profile found and temp not allowed
324+
325+
### Tag Matching Logic
326+
327+
- **AND logic by default**: Profile must have ALL required tags
328+
- **Most recently used**: Selected when multiple profiles match
329+
- **Case-sensitive**: Tags are compared exactly
330+
331+
### Security Recommendation
332+
333+
For scripts that require authenticated sessions, always set:
334+
```json
335+
"session": {
336+
"required_tags": ["domain.com", "authenticated"],
337+
"allow_temp_profile": false
338+
}
339+
```
340+
341+
This prevents accidental execution without the correct profile.
342+
343+
### List Available Profiles
344+
345+
```bash
346+
python openai_playwright_executor.py --list-profiles
347+
```
348+
349+
---
350+
295351
## Best Practices Summary
296352

297353
1. **Always escalate progressively**: Playwright first, vision last
@@ -301,6 +357,7 @@ python openai_playwright_executor.py \
301357
5. **Add screenshots**: At key stages for debugging
302358
6. **Structured schemas**: Clear prompts, required fields
303359
7. **Cost-conscious**: Aim for <$0.02 per run, <5 vision calls
360+
8. **Use profile tags**: Set `required_tags` and `allow_temp_profile: false` for authenticated scripts
304361

305362
---
306363

lambda/tools/local-browser-agent/examples/bt_broadband_password_manager.json

Lines changed: 31 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -16,17 +16,17 @@
1616
"state": "networkidle"
1717
},
1818
{
19-
"type": "screenshot",
19+
"action": "screenshot",
2020
"description": "Screenshot of initial page",
2121
"save_to": "01_initial_page.png"
2222
},
2323
{
24-
"type": "execute_js",
24+
"action": "execute_js",
2525
"description": "Check page state - DOM method (fastest)",
2626
"script": "(function() { var title = document.title.toLowerCase(); return { title: document.title, url: window.location.href, isLoginPage: title.includes('secure authentication') || title.includes('login') || title.includes('sign in'), hasLoginForm: !!document.querySelector('input[type=\"password\"]'), hasSearchForm: !!document.querySelector('input[name=\"postcode\"], input[id*=\"postcode\"], input[placeholder*=\"postcode\"]') }; })()"
2727
},
2828
{
29-
"type": "extract",
29+
"action": "extract",
3030
"description": "Identify page type using vision (if DOM unclear)",
3131
"method": "vision",
3232
"prompt": "Analyze this BT Wholesale webpage and determine: 1) Is this a login/authentication page (might show 'Secure Authentication'), a search/checker page, or results page? 2) What is the main call-to-action? 3) Are there any error messages or notices visible?",
@@ -59,7 +59,7 @@
5959
}
6060
},
6161
{
62-
"type": "fill",
62+
"action": "fill",
6363
"description": "Enter username (if on login page)",
6464
"escalation_chain": [
6565
{
@@ -89,7 +89,7 @@
8989
"value": "nterizakis"
9090
},
9191
{
92-
"type": "click",
92+
"action": "click",
9393
"description": "Click Next/Continue button after username",
9494
"escalation_chain": [
9595
{
@@ -126,7 +126,7 @@
126126
]
127127
},
128128
{
129-
"type": "wait",
129+
"action": "wait",
130130
"description": "Wait for password field to appear",
131131
"locator": {
132132
"strategy": "selector",
@@ -140,17 +140,17 @@
140140
"state": "networkidle"
141141
},
142142
{
143-
"type": "screenshot",
143+
"action": "screenshot",
144144
"description": "Screenshot of password page",
145145
"save_to": "01b_after_username.png"
146146
},
147147
{
148-
"type": "wait",
148+
"action": "wait",
149149
"description": "Wait for browser autofill to complete",
150150
"duration": 2000
151151
},
152152
{
153-
"type": "click",
153+
"action": "click",
154154
"description": "Click password field to activate password manager",
155155
"escalation_chain": [
156156
{
@@ -164,28 +164,28 @@
164164
]
165165
},
166166
{
167-
"type": "wait",
167+
"action": "wait",
168168
"description": "Wait for password manager dropdown to appear",
169169
"duration": 1000
170170
},
171171
{
172-
"type": "press",
172+
"action": "press",
173173
"description": "Select password from manager (Arrow Down + Enter)",
174174
"keys": ["ArrowDown", "Enter"],
175175
"delay": 500
176176
},
177177
{
178-
"type": "wait",
178+
"action": "wait",
179179
"description": "Wait for password to fill from manager",
180180
"duration": 1500
181181
},
182182
{
183-
"type": "execute_js",
183+
"action": "execute_js",
184184
"description": "Verify password was filled by password manager",
185185
"script": "(function() { var pwdField = document.querySelector('input[type=\"password\"]'); return { passwordFilled: pwdField ? pwdField.value.length > 0 : false, passwordLength: pwdField ? pwdField.value.length : 0, hasPasswordField: !!pwdField }; })()"
186186
},
187187
{
188-
"type": "click",
188+
"action": "click",
189189
"description": "Click Next button on password page",
190190
"escalation_chain": [
191191
{
@@ -220,7 +220,7 @@
220220
"state": "networkidle"
221221
},
222222
{
223-
"type": "wait",
223+
"action": "wait",
224224
"description": "Wait for SSO redirect to complete (BT portal)",
225225
"duration": 3000
226226
},
@@ -230,17 +230,17 @@
230230
"state": "networkidle"
231231
},
232232
{
233-
"type": "screenshot",
233+
"action": "screenshot",
234234
"description": "Screenshot after login and SSO redirect",
235235
"save_to": "02_after_login.png"
236236
},
237237
{
238-
"type": "execute_js",
238+
"action": "execute_js",
239239
"description": "Verify we're on BT wholesale portal",
240240
"script": "(function() { var url = window.location.href; var hash = window.location.hash; return { url: url, hash: hash, onBTWholesale: url.includes('bbactotl.btwholesale.com') || url.includes('btwholesale'), currentPage: hash }; })()"
241241
},
242242
{
243-
"type": "navigate",
243+
"action": "navigate",
244244
"description": "Navigate to Address Checker page",
245245
"url": "https://bbactotl.btwholesale.com/bbac-ui/totl/totlSearch/#/ADSL/AddressHome"
246246
},
@@ -250,22 +250,22 @@
250250
"state": "networkidle"
251251
},
252252
{
253-
"type": "screenshot",
253+
"action": "screenshot",
254254
"description": "Screenshot of Address Checker page",
255255
"save_to": "02b_address_checker.png"
256256
},
257257
{
258-
"type": "execute_js",
258+
"action": "execute_js",
259259
"description": "Verify Address Checker page loaded",
260260
"script": "(function() { var url = window.location.href; var hash = window.location.hash; var postcodeInput = document.querySelector('input[name*=\"postcode\"], input[id*=\"postcode\"], input[placeholder*=\"postcode\" i]'); return { url: url, hash: hash, hasPostcodeField: !!postcodeInput, postcodeFieldVisible: postcodeInput ? window.getComputedStyle(postcodeInput).display !== 'none' : false, pageTitle: document.title }; })()"
261261
},
262262
{
263-
"type": "wait",
263+
"action": "wait",
264264
"description": "Wait for Address Checker form to be ready",
265265
"duration": 2000
266266
},
267267
{
268-
"type": "fill",
268+
"action": "fill",
269269
"description": "Enter postcode in search field",
270270
"escalation_chain": [
271271
{
@@ -304,7 +304,7 @@
304304
"value": "{{postcode}}"
305305
},
306306
{
307-
"type": "click",
307+
"action": "click",
308308
"description": "Click search/check availability button",
309309
"escalation_chain": [
310310
{
@@ -346,12 +346,12 @@
346346
"state": "networkidle"
347347
},
348348
{
349-
"type": "screenshot",
349+
"action": "screenshot",
350350
"description": "Screenshot after search",
351351
"save_to": "02_after_search.png"
352352
},
353353
{
354-
"type": "extract",
354+
"action": "extract",
355355
"description": "Check if address selection is needed",
356356
"method": "vision",
357357
"prompt": "Look at this page after postcode search. Answer: 1) Does it show a list of addresses to select from? 2) Does it show broadband packages directly? 3) Is there an error or 'no results' message?",
@@ -383,7 +383,7 @@
383383
}
384384
},
385385
{
386-
"type": "click",
386+
"action": "click",
387387
"description": "Select matching address from list",
388388
"escalation_chain": [
389389
{
@@ -420,7 +420,7 @@
420420
]
421421
},
422422
{
423-
"type": "click",
423+
"action": "click",
424424
"description": "Click the submit/continue button after selecting the address",
425425
"escalation_chain": [
426426
{
@@ -470,7 +470,7 @@
470470
"state": "networkidle"
471471
},
472472
{
473-
"type": "wait",
473+
"action": "wait",
474474
"description": "Wait for JavaScript redirect to complete (address details page)",
475475
"duration": 3000
476476
},
@@ -480,12 +480,12 @@
480480
"state": "networkidle"
481481
},
482482
{
483-
"type": "screenshot",
483+
"action": "screenshot",
484484
"description": "Screenshot of packages page",
485485
"save_to": "03_packages.png"
486486
},
487487
{
488-
"type": "extract",
488+
"action": "extract",
489489
"description": "Extract all broadband packages with conditional logic",
490490
"method": "vision",
491491
"prompt": "Extract broadband availability information from the results page following these steps: 1. ALWAYS CAPTURE (if visible): - ALK (Access Line Key) - usually displayed near top - Exchange Code - BT exchange identifier 2. CHECK FEATURED PRODUCTS SECTION: Look for a section called 'Featured Products'. Search for a row/line containing 'WBC FTTP'. 3. IF 'WBC FTTP' IS FOUND: - Set fttp_avail = true - From the WBC FTTP row, capture 'Downstream Line Rate' (this is the SOADSL DSL speed in Mbps). Then navigate to OTHER SECTIONS: a) OTHER OFFERINGS section: - Check if ANY row has 'Service Availability' field = 'Enabled' - If found, set sogea_avail = true, otherwise false. b) ONT DETAILS section: - Find the LAST row in this section - Capture: 'Reference' field (this is ont_reference) - Capture: 'ONT Serial Number' field (this is ont_serial_no) - Capture: 'Port Service ID' field (this is port_service_id) c) PREMISE ENVIRONMENT section: - Capture: 'Recommended Minimum SVR NLP' field (this is install_type) - Valid values are typically 'Standard' or 'Premium'. 4. IF 'WBC FTTP' IS NOT FOUND: - Set fttp_avail = false - Leave all FTTP-related fields blank (install_type, sogea_avail, soadsl_dsl_speed, ont_reference, ont_serial_no, port_service_id). IMPORTANT: fttp_avail is REQUIRED and must always be set to true or false based on whether WBC FTTP is present in Featured Products.",
@@ -534,7 +534,7 @@
534534
}
535535
},
536536
{
537-
"type": "screenshot",
537+
"action": "screenshot",
538538
"description": "Final screenshot for verification",
539539
"save_to": "04_final.png"
540540
}

0 commit comments

Comments
 (0)