Skip to content

Commit 4b835b7

Browse files
variables for observe (browserbase#1808)
# why `act()` already supports variables, but `observe()` did not. That made safety-sensitive flows like login automation harder to use correctly: callers could inspect `observe()` results before executing them, but then had to guess which returned action should receive each secret value before calling `act()`. This change brings variable support to `observe()` so it can return placeholder-backed actions like `%username%` and `%password%`. That preserves the existing safe pattern of: 1. `observe()` candidate actions 2. validate the returned actions 3. `act()` the validated actions with real variable values at execution time # what changed - Added `variables?: Variables` support to `observe()` in the public SDK types and internal handler params. - Threaded `observe` variables through the local SDK path, inference layer, and prompt builder. - Updated the observe prompt so the model sees available variable names and returns `%variableName%` placeholders in action arguments instead of literal sensitive values. - Added observe variable support to the hosted/API path, including schema updates and flattening rich variable values to the existing wire format. - Updated the internal `fillForm` tool to pass variables into `observe()` as well as `act()`. - Added docs for `observe({ variables })` and the validate-then-act login flow. - Added a dedicated example at `packages/core/examples/observe_variables_login.ts` showing placeholder-based login planning with `observe()`, explicit validation, and execution via `act()`. # test plan - Ran targeted unit tests covering: - public `ObserveOptions` type support - observe variable forwarding into inference/prompting - placeholder preservation in returned observe actions - API client observe variable serialization - `fillForm` forwarding variables to `observe()` - Ran: - `pnpm --filter @browserbasehq/stagehand exec vitest run --config /tmp/stagehand-vitest-source.config.mjs tests/unit/public-api/public-types.test.ts tests/unit/agent-execution- model.test.ts tests/unit/timeout-handlers.test.ts tests/unit/api-client-observe-variables.test.ts` - Ran formatting checks on changed files with Prettier. - Added integration coverage for observe request schemas in both v3 and v4 server tests. - Full repo typecheck/build is still blocked by unrelated pre-existing issues in `packages/core/lib/v3/launch/browserbase.ts` and existing server test environment/type-resolution failures.
1 parent bfee1b8 commit 4b835b7

25 files changed

Lines changed: 712 additions & 37 deletions
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
/**
2+
* This example shows how to use observe({ variables }) to plan a sensitive
3+
* login flow, validate the returned placeholder actions, and then execute them
4+
* with act().
5+
*
6+
* observe() returns %variableName% placeholders in action arguments. That lets
7+
* you review the planned actions before any real secret values are used.
8+
*/
9+
import { Action, Stagehand } from "../lib/v3/index.js";
10+
import chalk from "chalk";
11+
12+
const variables = {
13+
username: "test@browserbase.com",
14+
password: "stagehand=goated",
15+
};
16+
17+
const loginInstruction = [
18+
"Fill the login form using the available variables.",
19+
"Use %username% for the email field.",
20+
"Use %password% for the password field.",
21+
"Include the field name in each action description.",
22+
].join(" ");
23+
24+
function findValidatedAction(
25+
observed: Action[],
26+
placeholder: string,
27+
keywords: string[],
28+
): Action {
29+
const matches = observed.filter((action) => {
30+
const description = action.description.toLowerCase();
31+
return (
32+
action.arguments?.includes(placeholder) === true &&
33+
keywords.some((keyword) => description.includes(keyword))
34+
);
35+
});
36+
37+
if (matches.length !== 1) {
38+
throw new Error(
39+
`Expected exactly one safe action for ${placeholder}, found ${matches.length}`,
40+
);
41+
}
42+
43+
return matches[0];
44+
}
45+
46+
async function observeVariablesLogin() {
47+
const stagehand = new Stagehand({
48+
env: "BROWSERBASE",
49+
verbose: 1,
50+
});
51+
52+
await stagehand.init();
53+
54+
try {
55+
const page = stagehand.context.pages()[0];
56+
57+
await page.goto("https://v0-modern-login-flow.vercel.app/", {
58+
waitUntil: "networkidle",
59+
timeoutMs: 30000,
60+
});
61+
62+
const observed = await stagehand.observe(loginInstruction, {
63+
variables,
64+
});
65+
66+
console.log(
67+
`${chalk.green("Observe:")} Placeholder actions found:\n${observed
68+
.map(
69+
(action) =>
70+
`${chalk.yellow(action.description)} -> ${chalk.blue(action.arguments?.join(", ") || "no arguments")}`,
71+
)
72+
.join("\n")}`,
73+
);
74+
75+
const emailAction = findValidatedAction(observed, "%username%", ["email"]);
76+
const passwordAction = findValidatedAction(observed, "%password%", [
77+
"password",
78+
]);
79+
80+
console.log(
81+
`\n${chalk.green("Validated:")} Safe actions to execute:\n${[
82+
emailAction,
83+
passwordAction,
84+
]
85+
.map(
86+
(action) =>
87+
`${chalk.yellow(action.description)} -> ${chalk.blue(action.arguments?.[0] || "no value")}`,
88+
)
89+
.join("\n")}`,
90+
);
91+
92+
await stagehand.act(emailAction, { variables });
93+
await stagehand.act(passwordAction, { variables });
94+
95+
const [submitButton] = await stagehand.observe("find the sign in button");
96+
97+
if (!submitButton) {
98+
throw new Error("Could not find the sign in button");
99+
}
100+
101+
await stagehand.act(submitButton);
102+
console.log(
103+
chalk.green(
104+
"\nSubmitted login form. Waiting 10 seconds before closing...",
105+
),
106+
);
107+
await page.waitForTimeout(10000);
108+
} finally {
109+
await stagehand.close();
110+
}
111+
}
112+
113+
(async () => {
114+
await observeVariablesLogin();
115+
})();

packages/core/lib/inference.ts

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ import type {
1717
StagehandZodObject,
1818
} from "./v3/zodCompat.js";
1919
import { SupportedUnderstudyAction } from "./v3/types/private/handlers.js";
20+
import type { Variables } from "./v3/types/public/agent.js";
2021

2122
// Re-export for backward compatibility
2223
export type { LLMParsedResponse, LLMUsage } from "./v3/llm/LLMClient.js";
@@ -245,6 +246,7 @@ export async function observe({
245246
logger,
246247
logInferenceToFile = false,
247248
supportedActions,
249+
variables,
248250
}: {
249251
instruction: string;
250252
domElements: string;
@@ -253,6 +255,7 @@ export async function observe({
253255
logger: (message: LogLine) => void;
254256
logInferenceToFile?: boolean;
255257
supportedActions?: string[];
258+
variables?: Variables;
256259
}) {
257260
const isGPT5 = llmClient.modelName.includes("gpt-5"); // TODO: remove this as we update support for gpt-5 configuration options
258261

@@ -297,7 +300,11 @@ export async function observe({
297300
type ObserveResponse = z.infer<typeof observeSchema>;
298301

299302
const messages: ChatMessage[] = [
300-
buildObserveSystemPrompt(userProvidedInstructions, supportedActions),
303+
buildObserveSystemPrompt(
304+
userProvidedInstructions,
305+
supportedActions,
306+
variables,
307+
),
301308
buildObserveUserMessage(instruction, domElements),
302309
];
303310

packages/core/lib/prompt.ts

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { ChatMessage } from "./v3/llm/LLMClient.js";
22
import type { Variables } from "./v3/types/public/agent.js";
3+
import { getVariablePromptEntries } from "./v3/agent/utils/variables.js";
34

45
export function buildUserInstructionsString(
56
userProvidedInstructions?: string,
@@ -112,10 +113,21 @@ Extracted content: ${JSON.stringify(extractionResponse, null, 2)}`,
112113
export function buildObserveSystemPrompt(
113114
userProvidedInstructions?: string,
114115
supportedActions?: string[],
116+
variables?: Variables,
115117
): ChatMessage {
116118
const actionsString = supportedActions?.length
117119
? `\n\nSupported actions: ${supportedActions.join(", ")}`
118120
: "";
121+
const variableEntries = getVariablePromptEntries(variables);
122+
const variablesString = variableEntries.length
123+
? `\n\nAvailable variables: ${variableEntries
124+
.map(({ name, description }) => {
125+
return description ? `%${name}% (${description})` : `%${name}%`;
126+
})
127+
.join(
128+
", ",
129+
)}. When an action needs a dynamic or sensitive value, return the matching %variableName% placeholder in the action arguments instead of a literal value`
130+
: "";
119131

120132
const observeSystemPrompt = `
121133
You are helping the user automate the browser by finding elements based on what the user wants to observe in the page.
@@ -125,7 +137,7 @@ You will be given:
125137
2. a hierarchical accessibility tree showing the semantic structure of the page. The tree is a hybrid of the DOM and the accessibility tree.
126138
127139
Return an array of elements that match the instruction if they exist, otherwise return an empty array.
128-
When returning elements, include the appropriate method from the supported actions list.${actionsString}. When choosing non-left click actions, provide right or middle as the argument.`;
140+
When returning elements, include the appropriate method from the supported actions list.${actionsString}${variablesString}. When choosing non-left click actions, provide right or middle as the argument.`;
129141
const content = observeSystemPrompt.replace(/\s+/g, " ");
130142

131143
return {

packages/core/lib/v3/agent/prompts/agentSystemPrompt.ts

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import type { AgentToolMode, Variables } from "../../types/public/agent.js";
22
import { CAPTCHA_SYSTEM_PROMPT_NOTE } from "../utils/captchaSolver.js";
3+
import { getVariablePromptEntries } from "../utils/variables.js";
34

45
export interface AgentSystemPromptOptions {
56
url: string;
@@ -214,17 +215,14 @@ export function buildAgentSystemPrompt(
214215
const variableToolsNote = isHybridMode
215216
? "Use %variableName% syntax in the type, fillFormVision, or act tool's value/text/action fields."
216217
: "Use %variableName% syntax in the act or fillForm tool's action fields.";
218+
const variableEntries = getVariablePromptEntries(variables);
217219
const variablesSection = hasVariables
218220
? `<variables>
219221
<note>You have access to the following variables. Use %variableName% syntax to substitute variable values. This is especially important for sensitive data like passwords.</note>
220222
<usage>${variableToolsNote}</usage>
221223
<example>To type a password, use: type %password% into the password field</example>
222-
${Object.entries(variables)
223-
.map(([name, v]) => {
224-
const description =
225-
typeof v === "object" && v !== null && "value" in v
226-
? v.description
227-
: undefined;
224+
${variableEntries
225+
.map(({ name, description }) => {
228226
return description
229227
? `<variable name="${name}">${description}</variable>`
230228
: `<variable name="${name}" />`;

packages/core/lib/v3/agent/tools/fillform.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,8 @@ export const fillFormTool = (
4646
.join(", ")}`;
4747

4848
const observeOptions = executionModel
49-
? { model: executionModel, timeout: toolTimeout }
50-
: { timeout: toolTimeout };
49+
? { model: executionModel, variables, timeout: toolTimeout }
50+
: { variables, timeout: toolTimeout };
5151
const observeResults = await v3.observe(instruction, observeOptions);
5252

5353
const completed = [] as unknown[];

packages/core/lib/v3/agent/utils/variables.ts

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,21 @@ export function getVariableDescription(v: VariableValue): string | undefined {
2222
return undefined;
2323
}
2424

25+
export interface VariablePromptEntry {
26+
name: string;
27+
description?: string;
28+
}
29+
30+
export function getVariablePromptEntries(
31+
variables?: Variables,
32+
): VariablePromptEntry[] {
33+
if (!variables) return [];
34+
return Object.entries(variables).map(([name, value]) => ({
35+
name,
36+
description: getVariableDescription(value),
37+
}));
38+
}
39+
2540
/**
2641
* Substitutes %variableName% tokens in text with resolved variable values.
2742
* Works with both simple and rich variable formats.

packages/core/lib/v3/handlers/observeHandler.ts

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ export class ObserveHandler {
6464
}
6565

6666
async observe(params: ObserveHandlerParams): Promise<Action[]> {
67-
const { instruction, page, timeout, selector, model } = params;
67+
const { instruction, page, timeout, selector, model, variables } = params;
6868

6969
const llmClient = this.resolveLlmClient(model);
7070

@@ -116,6 +116,7 @@ export class ObserveHandler {
116116
logger: v3Logger,
117117
logInferenceToFile: this.logInferenceToFile,
118118
supportedActions: Object.values(SupportedUnderstudyAction),
119+
variables,
119120
});
120121

121122
const {

packages/core/lib/v3/types/private/handlers.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ export interface ExtractHandlerParams<T extends StagehandZodSchema> {
2323
export interface ObserveHandlerParams {
2424
instruction?: string;
2525
model?: ModelConfiguration;
26+
variables?: Variables;
2627
timeout?: number;
2728
selector?: string;
2829
page: Page;

packages/core/lib/v3/types/public/api.ts

Lines changed: 28 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,12 @@
1010
*/
1111
import { z } from "zod/v4";
1212
import type Browserbase from "@browserbasehq/sdk";
13+
import { VariablesSchema } from "./variables.js";
14+
export {
15+
VariablePrimitiveSchema,
16+
VariableValueSchema,
17+
VariablesSchema,
18+
} from "./variables.js";
1319

1420
// =============================================================================
1521
// Shared Components
@@ -405,13 +411,17 @@ export const ActOptionsSchema = z
405411
description:
406412
"Model configuration object or model name string (e.g., 'openai/gpt-5-nano')",
407413
}),
408-
variables: z
409-
.record(z.string(), z.string())
410-
.optional()
411-
.meta({
412-
description: "Variables to substitute in the action instruction",
413-
example: { username: "john_doe" },
414-
}),
414+
variables: VariablesSchema.optional().meta({
415+
description:
416+
"Variables to substitute in the action instruction. Accepts flat primitives or { value, description? } objects.",
417+
example: {
418+
username: "john_doe",
419+
password: {
420+
value: "secret123",
421+
description: "The login password",
422+
},
423+
},
424+
}),
415425
timeout: z.number().optional().meta({
416426
description: "Timeout in ms for the action",
417427
example: 30000,
@@ -540,6 +550,17 @@ export const ObserveOptionsSchema = z
540550
description:
541551
"Model configuration object or model name string (e.g., 'openai/gpt-5-nano')",
542552
}),
553+
variables: VariablesSchema.optional().meta({
554+
description:
555+
"Variables whose names are exposed to the model so observe() returns %variableName% placeholders in suggested action arguments instead of literal values. Accepts flat primitives or { value, description? } objects.",
556+
example: {
557+
username: {
558+
value: "john@example.com",
559+
description: "The login email",
560+
},
561+
rememberMe: true,
562+
},
563+
}),
543564
timeout: z.number().optional().meta({
544565
description: "Timeout in ms for the observation",
545566
example: 30000,

packages/core/lib/v3/types/public/methods.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ export const pageTextSchema = z.object({
7373

7474
export interface ObserveOptions {
7575
model?: ModelConfiguration;
76+
variables?: Variables;
7677
timeout?: number;
7778
selector?: string;
7879
page?: PlaywrightPage | PuppeteerPage | PatchrightPage | Page;

0 commit comments

Comments
 (0)