Version 26.0.1 · Manifest V3 · Service Worker background
WebBrain is a browser extension that gives an LLM controlled access to the browser tab the user is looking at. The user types a natural-language instruction in a side panel, chooses Ask, Act, or Dev mode, and an autonomous agent loop calls the LLM, executes allowed tool calls (click, type, navigate, inspect, etc.), feeds the results back to the LLM, and repeats until the task is done or a loop detector halts it.
┌─────────────┐ messages ┌─────────────┐ HTTP/JSON ┌──────────────┐
│ Side Panel │ ◄──────────────► │ Background │ ◄──────────────► │ LLM Provider│
│ (UI) │ chrome.runtime │ (Agent) │ fetch() │ (OpenAI / │
│ sidepanel │ .sendMessage │ agent.js │ │ Anthropic /│
│ .js │ │ background │ │ llama.cpp) │
└──────┬──────┘ │ .js │ └──────────────┘
│ └──────┬───────┘
│ │
│ ┌────────────────────────────┼──────────────────────┐
│ │ chrome.debugger (CDP) │ chrome.scripting │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────────────────────────────┐ │
│ │ CDP │ │ Content Scripts (injected) │ │
│ │ Client │ │ • accessibility-tree.js │ │
│ │ cdp- │ │ • content.js │ │
│ │ client.js│ └──────────────────────────────────┘ │
│ └──────────┘ │
│ │ │
└───────┴──────────── DOM / Page ───────────────────────┘
The src/chrome tree is also the Microsoft Edge build. Edge supports the same
Chromium extension APIs and the same chrome.* namespace used by this code.
src/chrome/
├── manifest.json # Manifest V3 config
├── src/
│ ├── background.js # Service worker — message router
│ ├── run-ui-journal.js # Detached-run replay + streamed-text snapshots
│ ├── agent/
│ │ ├── agent.js # Core agent loop + tool dispatch
│ │ ├── loop-detector.js # Browser-free loop detection, directly unit-tested
│ │ ├── image-budget.js # Browser-free screenshot sizing, directly unit-tested
│ │ ├── mutation-tools.js # This build's state-change + mutating tool sets
│ │ ├── tools.js # Tool schemas + system prompts
│ │ ├── skills.js # Settings skills + dynamic skill tool manifests
│ │ ├── planner.js # Plan-before-Act structured planner
│ │ ├── permission-gate.js # Capability x origin permission gate
│ │ ├── adapters.js # Per-site guidance
│ │ └── scheduler.js # ScheduledJobManager — alarms-backed deferred tasks
│ ├── cdp/
│ │ └── cdp-client.js # DevTools Protocol wrapper
│ ├── content/
│ │ ├── accessibility-tree.js # AX tree builder + ref_id registry
│ │ └── content.js # Injected DOM reader / typer / clicker
│ ├── network/
│ │ └── network-tools.js # fetch_url, research_url, downloads, skill HTTP tools
│ ├── offscreen/
│ │ ├── offscreen.html # Offscreen document host
│ │ └── offscreen.js # HTTP fetch proxy (localhost/PNA fallback)
│ ├── providers/
│ │ ├── base.js # Provider interface
│ │ ├── manager.js # Provider lifecycle
│ │ ├── openai.js # OpenAI-compatible
│ │ ├── azure-openai.js # Azure OpenAI deployments
│ │ ├── aws-bedrock.js # AWS Bedrock Converse
│ │ ├── anthropic.js # Anthropic Claude
│ │ ├── llamacpp.js # Local llama.cpp server
│ │ └── fetch-with-fallback.js # Uses offscreen proxy on direct-fetch failure
│ ├── trace/
│ │ └── recorder.js # Optional IndexedDB run recorder
│ └── ui/
│ ├── sidepanel.html
│ ├── sidepanel.js # Chat UI, verbose mode, deep verbose
│ ├── settings.html
│ ├── settings.js # Provider + display settings
│ ├── traces.html
│ └── traces.js # Trace viewer / model comparison UI
├── skills/ # Packaged default skills (removable after seeding)
└── icons/
{
"permissions": [
"sidePanel", "activeTab", "contextMenus", "tabs", "tabGroups",
"scripting", "storage", "webNavigation", "webRequest",
"debugger", "downloads", "alarms", "unlimitedStorage",
"offscreen", "privateNetworkAccess", "tabCapture",
"clipboardWrite", "clipboardRead"
],
"host_permissions": ["<all_urls>", "http://localhost/*", "http://127.0.0.1/*", "http://*/*"]
}| Permission | Why |
|---|---|
debugger |
CDP access — trusted mouse/keyboard, pixel-perfect screenshots, shadow-DOM piercing. The single most important differentiator in the Chrome/Edge build vs Firefox. |
webRequest |
Opt-in, in-memory same-tab XHR/fetch observer for repeated-click API shortcut hints and opaque same-origin replay. Off by default. |
alarms |
Scheduled tasks and scheduled resumes across browser sessions. |
unlimitedStorage |
Optional trace recorder persists agent runs (LLM I/O + screenshots) into IndexedDB. A multi-step run can be 1–10 MB; the default ~10 MB origin cap fills after a few runs. |
offscreen |
Localhost LLM servers (llama.cpp, LM Studio, Ollama) are unreachable from the MV3 service worker due to CORS / Private Network Access restrictions. An offscreen document hosts the fetch proxy AND the tab recorder. |
privateNetworkAccess |
Same motivation — allow calling http://localhost:8080 from the extension. |
tabCapture |
Optional "Record this tab" feature in the sidepanel. Pulls a MediaStream of the active tab's video+audio via chrome.tabCapture.getMediaStreamId(), hands it to the offscreen document which runs the MediaRecorder. |
Manual action-mode runs (Act or Dev) always call agent/planner.js before the
first tool loop. Off uses the compact structured intent schema; Try and Strict
use the full bounded JSON plan with steps, memory strategy, scheduling hints,
and risks. The side panel renders a full plan
as an editable approval card; approving it pins the plan to the scratchpad so it
survives context compaction. Rejecting, timing out, or pressing Stop cancels
before browser tools execute. In the default Try mode, invalid JSON after one
repair degrades only that turn to the Ask prompt and read-only tool catalog;
Strict mode still stops before tools. Scheduled runs can set
autoApprovePlanReview so the plan is pinned without blocking on the UI.
The feature defaults to Try; an explicit Off setting remains off.
Planner LLM requests are recorded in traces with phase: "planner" and use the
same cost allowance and abort checks as the main loop.
Planner prompts keep optional policy text mechanically gated. The base planner
prompt includes general repeated-task pacing, but API replay guidance is appended
only when the tab conversation already has /allow-api; unavailable paths should
not bloat every planner request.
Settings -> Skills stores enabled skills under customSkills. On first run,
background.js seeds packaged skills from skills/*; FreeSkillz.xyz is enabled
by default but is just a stored built-in skill, so the user can remove it. If a
packaged built-in skill changes and the user still has it enabled, startup
refreshes the stored copy without re-adding deleted skills.
agent/skills.js splits each skill into two surfaces:
- prompt instructions appended by
buildCustomSkillsPrompt(); - optional tool schemas declared in fenced
webbrain-toolsJSON blocks.
The manifest fence is stripped before prompt injection. Declared skill tools are
appended to getToolsForMode(...) at LLM-call time and executed through
executeHttpSkillTool() in network-tools.js. Current skill tools support
read-only HTTPS GET/POST integrations and HTTPS download-job integrations that
poll a same-origin status URL, save through browser Downloads, and clean up the
provider job. Requests use credentials: "omit", optional URL input allowlists,
and optional response limits.
Importing/enabling a skill is the trust boundary. After import, the declared
tool can contact its declared endpoint without a per-call permission prompt.
Download-job tools still run in action modes and use the normal Downloads
permission gate before saving files. Third-party results should use
resultPolicy: "untrusted" so the agent wraps and digests them like page
content instead of trusted instructions.
Recording is user-driven from slash commands, not model-callable tools. /record
captures the active tab's video + audio + (optionally) microphone into a single
webm file and shows the red side-panel banner/timer. Add --transcribe to
/record or /record --full-screen to run Whisper transcription after stop.
As an intentionally undiscoverable convenience, a normal prompt may end in
/record [--save-as <filename>]; the side panel strips that suffix, starts the
tab recording before dispatch, and automatically stops it from the run cleanup
path. --save-as supplies the Downloads filename (with .webm normalized).
/record --full-screen opens Chrome's screen/window picker from the offscreen
recorder context through getDisplayMedia(), shows the WebBrain recording banner
by default, and can be stopped by its Stop button or double Escape on WebBrain or
browser pages. Add --hide-recording-indicator to hide the banner; Chrome's
picker decides what can be captured, so the user must choose the browser window
or whole screen if they want the WebBrain panel in the video.
sidepanel.js [/record]
│ runtime.sendMessage {action:'start_tab_recording', tabId, options}
▼
background.js
├─ chrome.tabCapture.getMediaStreamId({targetTabId}) → streamId
└─ offscreen recorder-start {source:'tab', streamId, options}
sidepanel.js [/record --full-screen]
│ prepare_recording_host
│ runtime.sendMessage {action:'start_display_recording', options}
▼
background.js
└─ offscreen recorder-start {source:'display', options}
│
▼
offscreen/recorder.js
├─ display: navigator.mediaDevices.getDisplayMedia({audio:true, video:true})
├─ tab: navigator.mediaDevices.getUserMedia(chromeMediaSource:'tab', streamId)
├─ navigator.mediaDevices.getUserMedia({audio:true}) (mic, best-effort)
├─ AudioContext:
│ captured audio ─→ mixDestination
│ mic ─→ mixDestination
│ tab audio only ─→ audioContext.destination (passthrough so user hears the call)
├─ MediaStream(video tracks ∪ mixDestination audio tracks when audio exists)
└─ MediaRecorder(mimeType selected from actual final video/audio tracks)
└─ ondataavailable → chunks[]
→ on stop, Blob → dataURL → background
background.js (on recorder-stop)
├─ chrome.downloads.download(dataURL → requested name or webbrain-recording-<ts>.webm)
└─ if transcribeAfter → runTranscription()
├─ providerManager.providers → pick first OpenAI-compatible
│ (openai → whisper-1, groq → whisper-large-v3, …)
├─ POST /v1/audio/transcriptions (multipart: file + model + response_format)
└─ chrome.downloads.download(.txt sibling)
sidepanel listens for recording_update broadcast events:
started → show the banner unless full-screen capture explicitly hid it
stopped → banner hides, "saved to Downloads" toast
transcribing → "Transcribing audio with Whisper…"
transcribed → "Transcript saved" + Summarize button (Phase 3)
The 2-hour safety cap lives in recorder/host.js as a service-worker timeout
plus a chrome.alarms watchdog, so hidden display recordings remain bounded
after the side panel closes.
By default, when chrome.tabCapture is active, Chromium browsers reroute the tab's
audio into the capture stream — the user can no longer hear what's playing
in the tab. For a meeting recorder this is catastrophic (you can't follow
the call). offscreen/recorder.js works around this with Web Audio:
const tabAudioSource = audioContext.createMediaStreamSource(captureStream);
tabAudioSource.connect(mixDestination); // into the recording
tabAudioSource.connect(audioContext.destination); // back to the user's speakerMic, by contrast, is only piped into the recording (NOT to the speaker — that would feed back).
Chrome/Edge MV3 allows exactly one offscreen document per extension. The
localhost-fetch proxy already needs one for Private Network Access
workarounds. Rather than fight over it, offscreen/offscreen.html loads
both offscreen.js (fetch proxy) and recorder.js (tab recorder).
src/offscreen/ensure.js is the single creation helper, declaring all
reasons up front: LOCAL_STORAGE (fetch), DISPLAY_MEDIA (tab/display
capture), USER_MEDIA (mic). Each script binds its own runtime.onMessage filter
(offscreen-fetch vs recorder-*) so they don't collide.
There is intentionally no separate "Whisper provider" settings tab. The
existing OpenAI-compatible provider configs already cover everything we
need. src/agent/transcribe.js picks the first usable one in priority
order:
| Priority | Provider id | Default model |
|---|---|---|
| 1 | openai |
whisper-1 |
| 2 | groq |
whisper-large-v3 |
| 3 | lmstudio |
user-configured local whisper model |
| 4 | llamacpp |
user-configured local whisper model |
| 5 | Any other type:'openai' provider not on the blocklist |
The blocklist (anthropic, gemini, mistral, deepseek, xai, nvidia)
covers providers that don't host Whisper despite being OpenAI-compatible
for chat. If none of the eligible providers is configured, transcription
is skipped with a clear "configure OpenAI or Groq" error — the .webm is
still saved.
| Works | |
|---|---|
| Google Meet (browser) | ✓ |
Zoom web client (zoom.us/wc/...) |
✓ |
| Native Zoom desktop app | ✓ via /record --full-screen when the user selects the Zoom window or screen in Chrome's picker; /record tab capture cannot reach it. |
| DRM-protected video (Netflix, Disney+) | ✗ — the browser blocks the encoder at the platform level. |
| chrome:// / edge:// / chrome-extension:// pages | ✗ — tabCapture is not allowed there. |
| Background tabs at start time | ⚠ — getMediaStreamId requires the target tab to be active; we briefly activate it before capture. The user can switch away after capture starts. |
Not yet — Firefox MV2 has no chrome.tabCapture equivalent; browser.tabCapture
doesn't exist. Only getDisplayMedia is available, which always shows the
system picker. That's a different UX (and matches "desktop capture only" —
the user's explicit "tab only" requirement can't be honoured there). Tracked
as a separate follow-up.
The primary page-interaction path is now an AX tree + ref_id model, replacing the older index-based get_interactive_elements for almost all flows.
Injected before content.js via content_scripts. Exposes three globals on window:
-
__generateAccessibilityTree(filter, maxDepth, maxChars, ref_id)— walks the DOM and emits a flat, indented text tree:dialog [ref_166] heading "Add a product" [ref_167] button "Close" [ref_169] textbox "Name" [ref_170] type="text" placeholder="Product name" value="namaz" combobox "…" [ref_180] type="button" -
__wb_ax_lookup(ref_id)— resolves aref_Nstring back to the liveElement. Backed by awindow.__wbElementMapofWeakRefs, so ids stay stable across calls as long as the element stays in the DOM. -
__wb_ax_suggest(ref_id, n)— when a lookup misses, returns nearby still-valid refs so the error message can guide the model back on track.
| filter | Behavior |
|---|---|
all |
Whole DOM. Useful when anchoring at a specific ref_id. |
visible |
Default for the agent. In-viewport, visible nodes only, soft-truncated at 3000 chars. |
interactive |
Only clickable/typeable things. |
- Overlay hoisting. Open dialogs / listboxes / menus /
[aria-expanded=true]comboboxes are emitted first under an[open overlays]banner so portal-rendered popups (React / Radix / Stripe) survive the 3000-char soft cap. - Accessible-name resolution (
getAccessibleName): priority is<select>selected option →aria-label→aria-labelledby(concatenates all referenced ids) →placeholder→title→alt→<label for>→ inputvalue(submit/button/reset only, never for text inputs — those emitvalue="..."separately) → direct text →innerTextfallback for buttons/links/summary →innerTextfallback for option/menuitem/tab/listitem/row/gridcell/cell → short visible descendant text for focusable, leaf-like generic token/chip wrappers → preceding-sibling text for unlabeled form fields ("Every 1 month(s)" pattern) → direct-text fallback. ref_idstability. WeakRef-backed registry with a monotonic counter means a ref you saw three turns ago still works if the element survived.
| Tool | Returns | Notes |
|---|---|---|
get_accessibility_tree |
Text tree + viewport | Primary page read path. |
click_ax({ref_id}) |
{success, method, tag, rect, name, href?, navigates?, hint?, trusted?, verified?, fallback?, observedHints?} |
Scrolls into view → focuses → el.click(). Chrome considers one CDP trusted-click fallback only after two stable page/target observations. URL, handler-focus, synchronous target mutation, and delayed semantic target state prove progress; broad page churn and delayed name/class/style/child changes are diagnostic hints only. Nearby mutating XHR/ping requests, new tabs, and downloads make the result inconclusive and veto retry, while background reads and obvious telemetry are ignored. Hidden, pointer-disabled, native, stateful/toggle, form, download, and potentially mutating controls never auto-retry. A silent app-internal success can still receive one trusted second activation; generic-only eligibility, settle time, and the one-shot rule bound but cannot eliminate that tradeoff. |
type_ax({ref_id, text, clear}) |
{success, method, rect} |
React-compatible: uses the native HTMLInputElement/HTMLTextAreaElement value setter. Rejects non-typeable INPUT subtypes (checkbox/radio/submit/file/...) with a clear error pointing at click_ax. |
set_field({ref_id, text, clear, submit}) |
{success, verified, ...} |
One-shot focus + clear + type + (optional) submit. Combobox-aware: if the element or an ancestor looks like a searchbox/combobox/open listbox, submit:true dispatches ArrowDown → Enter with small delays (Stripe-style virtualized pickers need the first option highlighted before Enter commits it). Bare text inputs still get Enter + form.requestSubmit(). |
Lives in agent.js, runs in the service worker. Same shape as always:
User message
│
▼
_enrichFirstUserMessage()
• Attach page URL + title
• Inject site adapter notes (if URL matches)
• Capture viewport screenshot (if provider supports vision)
• Build multimodal content [text, image_url]
│
▼
Main loop (max steps from Settings, default 130)
1. chatMainTurn(messages, {tools, temp, maxTokens})
• provider.chat() by default
• official OpenAI Responses streaming only for interactive Ask runs
2. If response has tool_calls:
a. _executeToolBatch() — run each tool
b. Push tool results into messages
c. Auto-screenshot if state changed AND provider has vision
(mode configurable: off / navigation / state_change / every_step)
d. Loop detection — nudge or stop
e. Navigation detection — warn on unintended SPA routing
f. Record trace event if tracing enabled
3. If response has text only → final answer
4. If done() called → summary (but see blockedDone below)
Production processMessage() lifecycle |
Legacy explicit processMessageStream() lifecycle |
|
|---|---|---|
| LLM call | provider.chat() by default; terminal-gated provider.chatStream() for eligible OpenAI Ask turns |
provider.chatStream() (SSE) |
| UI updates | Terminal text; eligible Ask turns also emit live text_delta |
Live text_delta |
| Tool calls | Returned only after a complete provider result | Accumulated by the separate streaming loop |
The Ask integration does not route through the legacy full streaming loop.
Normal side-panel sends still use detached chat_start, background-owned run
journaling/reconnect, attachment enrichment, and processMessage(). The stream
branch is eligible only for interactive Ask mode using the official OpenAI
Responses route and when the default-on Advanced kill switch is enabled. Act,
Dev, scheduled, cloud, and Continue runs stay on provider.chat().
During an eligible call, output text is forwarded live, but function calls,
usage, reasoning, and Responses output Items are buffered until
response.completed. Transport/protocol interruptions clear any partial
visible text, disable streaming for the remainder of that run, and retry the
generation via provider.chat(); terminal HTTP/API and response.incomplete
errors propagate without a duplicate fallback request. Incomplete tool calls
and partial assistant text are never executed or persisted. Live deltas remain
immediate, while reconnect snapshots coalesce on a 200 ms trailing interval;
terminal updates and pre-tool durability checkpoints flush immediately. The
journal also keeps separately bounded accumulated streamed text, allowing a
reopened panel to rebuild the in-progress Markdown even when early delta events
have already been acknowledged or trimmed from the replay window.
done() doesn't mean "done" if the page disagrees. Before accepting the summary, the agent probes for visible dialogs / forms / live-region messages. If the summary claims "created" / "saved" but a modal is still open, it returns {success: false, blockedDone: true} up to 2 times per tab, forcing the model to take another step. This stopped a class of "hallucinated success" failures on Stripe.
Three layered defenses keep the agent from interacting with the wrong surface when a dialog is open:
- Modal-scoped text click.
click({text: ...})now resolves_findTopmostModal()and scopes itsquerySelectorAll(plus label map, scroll retry, and contenteditable fallback) to that subtree. Previously this was only enforced for index-basedget_interactive_elements, so a text-click could still land on a dimmed background button (e.g. GitHub's "Publish release" while the "Create new tag" dialog was open). Error messages call out the scoping so the model knows why a page-wide lookup returns no match. - Post-click occlusion hit-test. For text / selector / index clicks (not x,y — the model chose those coords deliberately), after
scrollIntoViewbut before.click(), the resolver callselementFromPoint(cx, cy)at the target's center. If the topmost paint is neither the target nor a DOM ancestor/descendant, the click is refused with{occluded: true, occludedBy: {...}}and a suggestion to force-click via coordinates if the model really wants to hit what's on top. - Modal-aware ambiguity payload. When multiple elements match the text, the
ancestorfield in each candidate now identifies the containing dialog/form ("dialog: Create new tag"vs"form: release") so the model can pick by location instead of re-calling the same query.
System prompt has a new "MODALS & DIALOGS" section that describes the intended flow and the "dialog still open" failure pattern.
Before any submit-like text click, the agent checks a per-tab+URL 45-second window. Duplicate clicks in that window are blocked unless _allowResubmit is set; an acknowledged retry re-arms the window, so a further rapid duplicate needs its own acknowledgement. The browser-free guard is byte-identical in Chrome and Firefox, preventing the "clicked Create three times → three products created" failure mode in both builds.
When the API mutation observer setting is enabled, background.js records the
last 40 same-tab XHR/fetch requests using chrome.webRequest.onBeforeRequest.
The setting is off by default. When loop detection sees the same click /
click_ax repeat, _detectApiShortcut() checks whether each click produced the
same exact URL + method within 3 seconds. If so, the warning suggests
fetch_url({url, method}) instead of another click. This is advisory only:
POST/PUT/PATCH/DELETE still depend on the conversation's /allow-api state, and
GET/non-network capabilities still follow the normal permission gate.
When click({text}) or click({selector}) finds multiple matches, the error payload now carries full candidates (ref_id, rect, accessible name, ancestor summary) instead of just count. Lets the model pick the right one instead of looping.
The model-facing tool surface is selected by conversation mode plus provider tier:
- Ask: semantic/read-only page tools only. Ask intentionally excludes
clarify,read_page_source,inspect_element_styles, and action tools. - Act: the selected provider tier's normal browser-agent tools.
- Dev: Mid/Full only. Uses the selected Act tier, then adds source/style inspection, reversible CSS/DOM patches, CDP JavaScript/console/network/event diagnostics, temporary element highlighting, and Dev-extended shadow/frame inspection. Compact Dev is blocked.
get_accessibility_tree, read_page, read_pdf, get_window_info, get_interactive_elements, get_selection, extract_data, wait_for_stable
click_ax, type_ax, set_field, click (by text/selector/index/coords), type_text, press_keys, scroll, navigate, go_back, go_forward, new_tab, wait_for_element, iframe_read, iframe_click, iframe_type, upload_file
Full Act also adds advanced UI/DOM fallbacks: resize_window, hover (CDP-trusted, for reveal-on-hover menus), drag_drop (CDP-trusted pointer sequence, for Trello/Linear-style reordering), get_shadow_dom, shadow_dom_query, and get_frames. Mid Dev gets the shadow/frame inspection tools as Dev-extended debugging tools, but not hover/drag-drop.
Note: Chrome/Edge Dev mode implements
execute_jswith CDPRuntime.evaluate, not content-scripteval/new Function(). That keeps the MV3 extension-page CSP intact while allowing one-shot page-main-world evaluation. The tool remains absent from Ask and normal Act, is host-permission gated, and receives a fresh submit confirmation because arbitrary page JavaScript can submit forms.
fetch_url, research_url, list_downloads, read_downloaded_file, download_resource_from_page, download_files, download_social_media
verify_form, clarify, done, schedule_resume, schedule_task, scratchpad_write, progress_update, progress_read, solve_captcha
read_page_source, inspect_element_styles, inject_css, remove_injected_css, patch_element, revert_patch, execute_js, read_console, inspect_network_requests, inspect_event_listeners, and highlight_element are Dev-only: they do not appear in Ask or normal Act. CSS injection uses chrome.scripting.insertCSS/removeCSS and persists its patch metadata in chrome.storage.session; a navigation race removes the exact uniquely marked CSS from the replacement document before stale metadata is discarded. Structured element patches keep exact before/after values in the page's content-script world. Listener inspection briefly adds and restores an internal target attribute, follows open-shadow hosts for ancestor listeners, and highlighting inserts a temporary overlay, so both are gated as temporary page modifications. Console and network capture start when a Dev run attaches, use bounded buffers, omit headers/bodies by default, redact sensitive headers before storage, and are drained for every tracked tab when the panel leaves Dev mode.
cdp-client.js wraps chrome.debugger:
| Capability | CDP Domain | Use |
|---|---|---|
| Screenshot | Page.captureScreenshot |
Viewport + full-page |
| Click | Input.dispatchMouseEvent |
Trusted mouse events |
| JavaScript evaluation | Runtime.evaluate |
Dev-only one-shot async function bodies |
| Console diagnostics | Runtime.consoleAPICalled, Runtime.exceptionThrown, Log.entryAdded |
Bounded Dev console/error buffer |
| Network diagnostics | Network.* events and bounded body reads |
Dev request/status/timing inspection |
| Event listeners | DOMDebugger.getEventListeners |
Listener inspection for ref/selector targets |
| Keyboard | Input.dispatchKeyEvent |
Trusted keystrokes |
| Evaluate | Runtime.evaluate |
Run code in page context |
| DOM query | DOM.* |
Shadow DOM piercing |
CDP events are trusted (event.isTrusted === true). Many sites reject synthetic el.click(); CDP is what lets WebBrain work on those.
| CDP path | Content-script fallback | |
|---|---|---|
| How | Input.dispatchMouseEvent(x,y) |
el.click() |
| Trusted | Yes | No |
| Cross-origin | Works | Blocked |
| Used when | Default in Chrome/Edge, or click({x,y}) / click({text}) after coord resolution; one guarded fallback for a safe click_ax target after verified synthetic no-progress |
click_ax first attempt (focuses the exact element and dispatches in-page), plus Firefox's only click path |
class BaseProvider {
async chat(messages, options) // → { content, toolCalls, usage }
async *chatStream(messages, options) // yields { type, content }
get supportsTools()
get supportsVision()
async testConnection()
}| Provider | Endpoint | Vision detection |
|---|---|---|
OpenAIProvider |
/v1/chat/completions |
Model-name regex |
AzureOpenAIProvider |
Azure deployment chat completions | Manual/configured |
AwsBedrockProvider |
Bedrock Converse API | Disabled by default |
AnthropicProvider |
/v1/messages |
claude-(3|sonnet-4|opus-4) patterns |
LlamaCppProvider |
localhost:8080/v1/chat/completions |
Enabled by default, configurable |
| OpenAI-compatible configs | Provider-specific /v1 endpoint |
Model-name regex or explicit config |
ProviderManager seeds WebBrain Cloud, seven local backends, Azure OpenAI, AWS
Bedrock, direct cloud providers, and router providers. The canonical current ID
and default-model table is maintained in
docs/providers-and-models.md.
OpenAI format → Anthropic blocks: system → separate system field; assistant + tool_calls → assistant + tool_use blocks; tool role → user + tool_result blocks; image_url data URLs → image source.
providers/fetch-with-fallback.js tries a direct fetch first. On failure (typically a TypeError: Failed to fetch against localhost), it lazily creates an offscreen document and proxies through it. This is the only reason the offscreen permission exists.
ScheduledJobManager (Chrome/Edge build: src/chrome/src/agent/scheduler.js) is instantiated in background.js and uses chrome.alarms to fire deferred work.
Chromium/Edge-specific behavior vs Firefox:
- URL-target tasks open their tab in the background (
active: false) so the user isn't interrupted. - A service-worker keepalive (
startChromeAlarmKeepAlive) pingschrome.runtime.getPlatformInfoevery 20 s while a scheduled job is executing, keeping the MV3 service worker alive for the duration of the run. - Alarm names are prefixed
wb_scheduled_job:<jobId>; restored on service-worker startup viarestoreAlarms().
Job kinds, lifecycle, and tools are identical to the shared design — see docs/architecture.md § Scheduled Tasks.
| Limits | Value |
|---|---|
Min delay (schedule_resume) |
30 s |
Min delay (schedule_task) |
0 s to start now; otherwise 60 s |
| Max delay (both) | 7 days |
| Min recurring interval | 1 min |
| Max recurring interval | 1 year (525 600 min) |
| Max queue deferrals before failure | 120 (≈ 1 h of retries) |
The browser-free implementation lives in agent/loop-detector.js; Agent
inherits it, and the unit suite imports the same production class directly.
That module is byte-identical across builds, so the per-build tool surface it
classifies against lives in agent/mutation-tools.js instead.
Three independent detectors, strongest action wins:
- General repeat — last 6 tool calls by (name + args hash + outcome). Nudge at 3 identical or ABAB. Stop at 8 nudges without 2 consecutive healthy calls between.
- Coordinate click — 5-pixel bucketing. Nudge at 5 same-bucket clicks. Stop at 8.
- Navigation — snapshot URL before
click/navigate/iframe_click, compare 200 ms later. Unexpected change →[NAVIGATION OCCURRED]warning.
- Auto-trim (
_manageContext): triggered at >50 messages or >80,000 chars; keeps system prompt + LLM-summarized old messages (cap 2000 chars) + last 16 verbatim. - Emergency trim (
_emergencyTrim): on provider context-overflow error. - Image pruning (
_pruneOldImages): before every call, strip base64 images from all but the last 4 messages. - Tool result cap: individual results truncated at 8,000 chars with
[truncated]marker.
MV3 service workers and side panels can disappear while a run is active, so Chrome keeps three per-tab session records:
chrome.storage.session['agentConv:<tabId>'] // provider-facing conversation
chrome.storage.session['tabChat:<tabId>'] // rendered chat HTML
chrome.storage.session['runUi:<tabId>'] // detached-run status and replay
Agent history is debounced after changes and lazily hydrated before the next
message. The UI journal owns request/run identity, status, a 256-event replay
window, plan/tool state, terminal content, and separately bounded accumulated
streamed text. text_delta snapshots coalesce for 200 ms; non-delta, terminal,
and pre-tool checkpoints persist immediately. Reconnect replay is acknowledged
by sequence number, while the accumulated stream lets the panel reconstruct
in-progress Markdown after older delta events have been acknowledged.
Off by default. Enabled via Settings → Display → "Record traces". When on, every agent run writes to an IndexedDB database (webbrain-traces):
runsstore: one row per user message — model, provider, token totals, timestamps.eventsstore: one row per LLM request/response, tool call, screenshot. Rows are indexed by(runId, seq).
The Traces page (ui/traces.html) lists runs and renders their event timelines. Exporting produces a JSON blob identical to the ones used in this session's debugging. Data never leaves the machine — this is why unlimitedStorage is requested (a multi-step run with screenshots is 1–10 MB).
Settings → Display toggles (stored in chrome.storage.local):
| Setting | Effect |
|---|---|
| Verbose mode | Shows full tool args + JSON results in chat instead of compact labels. |
| Screenshot fallback | Capture a screenshot when DOM read fails or returns insufficient content. |
| Site adapters | Inject per-site guidance into the first user message (default on). |
| Auto-screenshot | off / navigation / state_change (default) / every_step. |
| Record traces | Enable the trace recorder (see above). |
| Completion sound | Play a chime in the side panel when the agent finishes. |
| Max Agent Steps | 5–195 finite steps or unlimited, default 130. |
58 adapters inject site-specific guidance into the first user message. Re-injected mid-conversation if the user navigates to a different matched site. Only ONE adapter fires at a time (the first matching match(url) wins), so the prompt cost is fixed regardless of total adapter count — what grows is the maintenance surface.
| Category | Sites |
|---|---|
| Code & Dev | GitHub, GitLab, Stack Overflow, Hacker News |
| Coding practice | LeetCode, HackerRank |
| Productivity | Gmail, Outlook, Google Docs, Google Sheets, Google Calendar, Slack, Notion, Jira, Trello |
| Social | Twitter/X, LinkedIn, Reddit, YouTube, Instagram, TikTok, Facebook |
| Messaging | Discord, WhatsApp Web, Telegram |
| Publishing | Medium, Substack, WordPress |
| Commerce | Amazon, eBay, Walmart, Target, Etsy |
| Travel | Airbnb, Booking.com, Expedia, Google Maps, Google Flights, Kayak, OpenTable |
| Cloud / Infra | AWS, GCP, Cloudflare, Vercel |
| News (paywalls) | NYT, WSJ, FT, Bloomberg, Economist, Washington Post |
| Job portals | Greenhouse, Workday |
| Finance | Stripe, Coinbase, Robinhood, TradingView, finance-generic (banks/exchanges/payments) |
Finance adapters carry a [FINANCE / HIGH-STAKES] banner and extra confirmation guidance. The finance-generic adapter matches a curated regex of bank, brokerage, crypto exchange, and payment domains as a catch-all when no site-specific adapter exists.
- Ask — read-only tools, analysis / Q&A.
- Act — selected provider tier's normal browser-action tools.
- Dev — Mid/Full action mode for source/style/page-debugging work; adds Dev tools and is blocked for Compact-tier providers.
- Normal — compact step labels.
- Verbose ON — full JSON args + truncated results.
- Deep verbose (Shift+click verbose button) — dump the full LLM request/response ring buffer (200 entries) to DevTools console with color-coded groups.
- A new question remains anchored while its answer grows.
- The floating control offers Follow response, Jump to latest, or Back to question based on the viewport position.
- Auto-follow resumes only after the reader reaches the bottom deliberately or explicitly jumps to the live edge. Blocking cards and slash-command output still force the relevant content into view.
- Restoring a panel or switching tabs re-establishes the latest turn's reading position and reconnects to any active background-owned run.
1. User types "create a product 'namaz' priced 500 CNY, recurring every 2 months"
2. sidepanel.js → sendRunWithReconnect('chat_start', {
text, mode:'act', tabId:42, requestId
})
3. background.js → begin/persist runUi:42 → detached `chat` lifecycle
→ agent.processMessage(42, text, onUpdate, 'act')
4. _enrichFirstUserMessage: attach URL/title + site adapter + viewport screenshot
5. provider.chat(messages, {tools, temp:0.15, maxTokens:4096})
→ trace recorder logs llm_request
6. LLM returns tool_calls: [{name:'get_accessibility_tree', args:{filter:'visible'}}]
7. _executeToolBatch → content script → window.__generateAccessibilityTree(...)
→ returns indented tree text
8. Agent appends tool result, auto-screenshots (state unchanged → skipped), loops
9. Over ~15 steps the agent: clicks "Create product", fills name via set_field,
fills price, opens currency combobox, types "CNY" with submit:true
(ArrowDown+Enter commits the filtered option), opens billing-period dropdown,
selects "Custom", sets count to 2 and unit to "month(s)", clicks "Add product".
10. done() returns the summary — blockedDone check passes since the dialog closed
and an alertdialog "Product created" is now visible.
11. sidepanel renders the assistant message + optional completion chime.
- Extension runs with the user's full browser permissions — no additional auth.
<all_urls>host permission → content-script injection anywhere.debugger→ trusted events on any tab.- Cross-origin iframes reachable via content-script injection (extension privilege).
- Plan before Act can require user approval before any action-mode tool executes.
/allow-apiflag required for API mutations (POST/PUT/PATCH/DELETE viafetch_url).- Finance adapters layer extra confirmation guidance.
- Tool results capped at 8 KB to limit prompt-injection surface.
- Offscreen proxy only forwards requests the user's own code initiated (provider SDK traffic).
- Trace data is local-only — never transmitted.