Skip to content

Commit 598469d

Browse files
razkenarirazkenari
committed
docs: add AgentCore Browser integration guide and contributors
- Add docs/TOOL_AC_BROWSER.md with architecture, browser-use hooks, frontend live view, streaming reliability, and available patterns - Add Hari Prasanna Das and Rahul Ghosh to CONTRIBUTORS.md
1 parent 3e78899 commit 598469d

2 files changed

Lines changed: 150 additions & 0 deletions

File tree

‎CONTRIBUTORS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,5 @@
1919
- Samaneh Aminikhanghahi
2020
- Abhilash Balachandran
2121
- Rajat Dogra
22+
- Hari Prasanna Das
23+
- Rahul Ghosh

‎docs/TOOL_AC_BROWSER.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# AgentCore Browser Integration
2+
3+
This document explains the architectural decisions for integrating Amazon Bedrock AgentCore Browser into FAST.
4+
5+
## What is AgentCore Browser?
6+
7+
Amazon Bedrock AgentCore Browser provides a secure, isolated cloud browser environment for AI agents to interact with web applications. Key features:
8+
9+
- Isolated Chromium browser sessions in containerized environments
10+
- CDP (Chrome DevTools Protocol) access for automation
11+
- Real-time live view streaming via DCV protocol
12+
- Session persistence (cookies, login state, tabs carry over across tool calls)
13+
- Web Bot Auth support for reducing CAPTCHAs
14+
15+
**Documentation**: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-tool.html
16+
17+
## Architecture
18+
19+
```
20+
User prompt
21+
→ Agent (Strands / LangGraph / Claude SDK)
22+
→ browser tool (async generator)
23+
→ AgentCore Browser session (CDP)
24+
→ browser-use Agent (autonomous browsing)
25+
→ on_step_start hook → asyncio.Queue → yield → tool_stream_event SSE
26+
→ on_step_end hook → asyncio.Queue → yield → tool_stream_event SSE
27+
→ Live View URL → DCV WebSocket → BrowserLiveView component (direct to client)
28+
→ tool result → agent continues reasoning
29+
→ Final text response to user
30+
```
31+
32+
The DCV live view stream flows **directly** from AgentCore to the user's browser — it does not pass through the application server.
33+
34+
## Why Direct Integration (Not Gateway)?
35+
36+
Same rationale as Code Interpreter — FAST integrates Browser **directly into agents** rather than through the Gateway:
37+
38+
- **Session management** — Browser sessions persist across tool calls (cookies, login state, tabs)
39+
- **Streaming** — Tool stream events (live view URL, browser actions) require async generator yields
40+
- **Lower latency** — No Gateway/Lambda hops for CDP commands
41+
- **Follows AWS patterns** — Matches official SDK documentation and sample apps
42+
43+
## Browser Automation: browser-use Library
44+
45+
The browser tool uses the [browser-use](https://github.com/browser-use/browser-use) library for autonomous browser control. Instead of scripting individual clicks and navigations, you describe the task in natural language and the LLM-driven agent figures out the UI interactions.
46+
47+
### Why browser-use?
48+
49+
- **Autonomous** — LLM decides what to click, type, scroll without manual scripting
50+
- **Resilient** — Handles UI changes (CSS class renames, layout shifts) by reading the DOM semantically
51+
- **Multi-step** — Handles complex workflows in a single tool call
52+
- **Session reuse** — Agent memory, page context, and action history persist between calls
53+
54+
### Lifecycle Hooks for Streaming
55+
56+
browser-use provides `on_step_start` and `on_step_end` hooks that fire before and after each agent step. FAST uses these hooks with `asyncio.Queue` to stream actions to the frontend in real-time:
57+
58+
```python
59+
step_queue: asyncio.Queue = asyncio.Queue()
60+
61+
async def on_step_end_hook(agent):
62+
# Extract completed action from agent history
63+
actions = agent.history.action_names()
64+
urls = agent.history.urls()
65+
extracted = agent.history.extracted_content()
66+
# Put into queue for immediate yield
67+
await step_queue.put({"type": "browser_action", "extracted_content": summary})
68+
69+
# Run agent with hooks in background task
70+
agent_task = asyncio.create_task(
71+
agent.run(on_step_start=on_step_start_hook, on_step_end=on_step_end_hook)
72+
)
73+
74+
# Yield from queue as events arrive
75+
while not agent_task.done():
76+
step = await asyncio.wait_for(step_queue.get(), timeout=0.5)
77+
yield step
78+
```
79+
80+
## Frontend: Live View Component
81+
82+
The frontend uses the official `bedrock-agentcore` npm package for the DCV live view:
83+
84+
```tsx
85+
import { BrowserLiveView } from 'bedrock-agentcore/browser/live-view'
86+
87+
<BrowserLiveView
88+
signedUrl={presignedUrl}
89+
remoteWidth={1380}
90+
remoteHeight={720}
91+
/>
92+
```
93+
94+
### Vite Configuration
95+
96+
The `BrowserLiveView` component requires DCV SDK aliases in your Vite config:
97+
98+
- `resolve.alias` — Points `dcv` and `dcv-ui` to vendored SDK files
99+
- `resolve.dedupe` — Forces shared deps (React, Cloudscape) to resolve from your `node_modules`
100+
- `viteStaticCopy` — Copies DCV runtime files (workers, WASM decoders) to build output
101+
102+
See `frontend/vite.config.ts` for the complete configuration.
103+
104+
### Custom Tool Renderer
105+
106+
`BrowserToolDisplay` is registered as a custom renderer for the `"browser"` tool name:
107+
108+
```tsx
109+
useToolRenderer("browser", props => <BrowserToolDisplay {...props} />)
110+
```
111+
112+
This keeps browser-specific UI (live view, action streaming, result parsing) decoupled from the generic tool display system. Other tools continue using the default `ToolCallDisplay`.
113+
114+
### Streaming Reliability
115+
116+
Two key fixes ensure streaming actions appear in real-time:
117+
118+
1. **`flushSync`** — Forces React to render immediately on each `tool_stream` event instead of batching
119+
2. **Parser priority** — `tool_stream_event` is checked before text `data` in the strands parser to prevent action content from leaking as chat text
120+
121+
## Available Patterns
122+
123+
| Pattern | Description | Status |
124+
|---------|-------------|--------|
125+
| `strands-browseruse-multiagent` | Strands + browser-use + Code Interpreter + Gateway tools | ✅ Available |
126+
| `strands-browser-single-agent` | Strands + browser tool only | 🔜 Planned |
127+
| `langgraph-browser-agent` | LangGraph + browser tool | 🔜 Planned |
128+
| `claude-browser-agent` | Claude Agent SDK + browser tool | 🔜 Planned |
129+
130+
## Configuration
131+
132+
In `infra-cdk/config.yaml`:
133+
134+
```yaml
135+
backend:
136+
pattern: strands-browseruse-multiagent
137+
deployment_type: docker
138+
```
139+
140+
The browser tool requires Docker deployment (`deployment_type: docker`), not zip.
141+
142+
## References
143+
144+
- [AgentCore Browser documentation](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-tool.html)
145+
- [BrowserLiveView blog post](https://aws.amazon.com/blogs/machine-learning/embed-a-live-ai-browser-agent-in-your-react-app-with-amazon-bedrock-agentcore/)
146+
- [browser-use library](https://github.com/browser-use/browser-use)
147+
- [browser-use hooks documentation](https://docs.browser-use.com/customize/hooks)
148+
- [Bedrock AgentCore TypeScript SDK](https://github.com/aws/bedrock-agentcore-sdk-typescript)

0 commit comments

Comments
 (0)