Merge pull request #443 from alectimison-maker/feat/saved-workflows

feat: add guarded saved workflow replay

Author: esokullu

README.md

@@ -336,6 +336,10 @@ WebBrain accepts slash commands as the first thing on a line in the input box. T
 | `/memory` | Show saved user memory |
 | `/memory --add <text>` | Save a user preference to memory |
 | `/memory --forget <id>` | Forget a saved memory by ID |
+| `/workflow` | List saved workflows and their IDs |
+| `/workflow --save <name>` | Compile the latest successful traced run into a reusable, value-free workflow |
+| `/workflow --run <id>` | Run a saved workflow in Act mode, collecting any runtime parameters locally |
+| `/workflow --delete <id>` | Delete a saved workflow |
 | `/allow-api` | **Per-conversation API mutation override.** Lifts the UI-first restriction so the agent may use POST/PUT/PATCH/DELETE via `fetch_url` when UI is failing. Badge appears while active; clears on `/reset`. |
 | `/dangerously-skip-permissions` | **Global permission-prompt bypass.** Turns off `Ask before consequential actions` without opening Settings. WebBrain will act without per-site prompts until you re-enable the setting. |
 | `/compact` | Force context compaction for the current conversation |
@@ -373,6 +377,19 @@ the originating run tab before saving the after screenshot. If the recording
 or initial screenshot cannot be started and saved, the run is not sent.
 Standalone `/record` and `/screenshot` keep their existing behavior.
 
+Saved workflows use a separate `webbrain-workflow/1` schema; they are not raw
+trace replays. Historical `ref_id` values, action CSS selectors, coordinates,
+query strings, fragments, and typed field values are excluded. Typed values become runtime
+parameters, and each action is bound to the recorded origin and URL family.
+At run time WebBrain resolves a fresh accessibility-tree target and executes
+through the normal Act permission, submit-confirmation, and verification gates.
+Ambiguous targets fail closed. If an action may already have happened but its
+result is unknown, replay stops instead of retrying it. Runtime parameter values
+are not saved to the workflow, conversation, user memory, replay trace, or Agent
+fallback prompt; they are still delivered to the target page by the requested
+browser action. The original opt-in source trace remains separate and can
+contain raw tool arguments until the user deletes that trace.
+
 The default UI-first rule exists because API actions are invisible (you don't see what's being sent), often require separate auth tokens you may not have configured, and can have a much larger blast radius than a visible mis-click. Only use `/allow-api` when you've decided you want that tradeoff for a specific job.
 
 ## Keyboard Shortcuts

docs/architecture.md

@@ -432,6 +432,35 @@ job. A short queue drains best-effort through the active provider using the
 existing cost allowance guard; cost exhaustion skips extraction silently, and
 other failures retry once.
 
+### Saved workflows (`agent/workflows.js`)
+
+Saved workflows are compiled artifacts, not serialized trace events. The
+background reads the newest successful trace in the active conversation and
+normalizes its replayable actions into `webbrain-workflow/1`, stored under
+`wb_saved_workflows_v1`. Compilation removes historical element references,
+action CSS selectors, coordinates, query strings, fragments, and typed values. Every typed field
+value becomes a declared runtime parameter; unsupported or failed actions are
+skipped and reported to the user as save warnings.
+
+Each compiled step contains semantic target metadata (role, accessible name,
+label, field identity, link, or placeholder), an expected postcondition, and
+the origin/path family observed before that action. `/workflow --run <id>`
+collects parameters in an ephemeral side-panel form. The replay executor then:
+
+1. checks the current origin/path family before every step;
+2. reads a fresh accessibility tree and resolves exactly one semantic match;
+3. calls `_executeToolBatch()` so the existing permission, form-submit,
+   verification, abort, and action-normalization gates remain authoritative;
+4. validates the saved postcondition; and
+5. either continues deterministically, delegates a known-safe mismatch to the
+   normal Agent, or stops when a state-changing action has an unknown outcome.
+
+Replay does not set `currentRunId`, because ordinary tool tracing would retain
+runtime values. It creates a separate run containing sanitized notes and
+redacted UI tool events. Runtime parameter values are also omitted from the
+fallback prompt and user-memory extraction. Chrome and Firefox ship identical
+workflow schema/compiler code and the same replay policy.
+
 ### Scheduled Tasks (`scheduler.js`)
 
 The scheduler lets the agent defer work to a future browser session using the browser's `alarms` API. It lives in `src/chrome/src/agent/scheduler.js` (and the Firefox mirror) and is instantiated as `ScheduledJobManager` in the background script.

docs/privacy-and-data-flow.md

@@ -117,6 +117,30 @@ When enabled (Settings → Display → "Record traces"), every agent run is writ
 
 The Traces page (`ui/traces.html`) reads from local IndexedDB only. Export produces a JSON blob saved to the user's Downloads folder. **No trace data ever leaves the browser.**
 
+### Saved Workflows
+
+`/workflow --save <name>` locally compiles the latest successful trace into a
+separate `webbrain-workflow/1` record in browser local storage
+(`wb_saved_workflows_v1`). The saved record contains action names, sanitized
+arguments, semantic target descriptors, URL origin/path families,
+postconditions, and parameter descriptors. It does not contain typed field
+values, raw historical `ref_id` values, action CSS selectors, coordinates, URL query strings, or URL
+fragments.
+
+`/workflow --run <id>` collects declared values in a temporary side-panel form
+and sends them directly to the background replay executor. The values are not
+written to the workflow, chat text, retry payload, user memory, replay trace,
+or Agent fallback prompt. They necessarily reach the active page when the
+requested field action runs. A source trace is a separate opt-in record and may
+still contain the original raw tool arguments; saving a workflow does not
+delete or redact that source trace.
+
+Replay traces contain workflow/step IDs, semantic match status and score,
+postcondition status, fallback status, and estimated model calls saved. They do
+not contain runtime parameter values or freshly resolved element references.
+If deterministic replay cannot safely continue, a fallback Agent receives only
+saved metadata and must ask the user again for any still-needed value.
+
 ### Settings
 
 Provider configs (API keys, base URLs, model selections) are stored in `chrome.storage.local`. API keys are in plaintext — this is a personal-computer tool and the storage is sandboxed by the browser. The extension has no mechanism to exfiltrate these keys.

docs/security-model.md

@@ -167,6 +167,24 @@ The traces page (`ui/traces.html`) reads from local IndexedDB only. Export produ
 
 ---
 
+## Saved-Workflow Replay Boundary
+
+Saved workflows deliberately do not replay raw trace calls. Compilation uses an
+allowlist, replaces every typed value with a runtime parameter, discards raw
+references, action CSS selectors, coordinates and URL query or fragment data, and binds each action to
+an origin/path family plus a semantic target and postcondition.
+
+Before an action, replay must be on the recorded URL family and find one
+unambiguous target in a fresh accessibility tree. The action is dispatched
+through the same capability-by-origin permission gate, submit confirmation,
+form validation, and trusted-event path as a normal Act run. A known pre-action
+mismatch can be delegated to the Agent with no parameter values. A failed or
+unverified state-changing action whose dispatch cannot be disproved is treated
+as outcome-unknown and is never automatically retried. Replay telemetry and UI
+events redact runtime values and fresh `ref_id` values.
+
+---
+
 ## Firefox Differences
 
 Firefox has no CDP (`debugger` permission), so:

src/chrome/src/agent/agent.js

@@ -46,6 +46,17 @@ import { repairAssistantDisplayText, sanitizeText as sanitizePlannerText } from
 import { buildCustomSkillsPrompt, buildSkillLoaderDefinition, buildSkillToolDefinitions, buildSkillToolRegistry, getEligibleCustomSkills, getEligibleSkillCatalog, normalizeCustomSkills } from './skills.js';
 import { publicMediaUrlNeedsExplicitTarget } from './public-media-url.js';
 import { USER_MEMORY_DEFAULT_MAX_PROMPT_CHARS, formatUserMemoryPrompt, normalizeUserMemoryMaxPromptChars, normalizeUserMemoryStore } from './user-memory.js';
+import {
+  findWorkflowTarget,
+  parseAccessibilityTreeDescriptors,
+  redactWorkflowArgsForTelemetry,
+  redactWorkflowClarifyForTelemetry,
+  redactWorkflowResultForTelemetry,
+  resolveWorkflowArgs,
+  validateWorkflowStepResult,
+  workflowFallbackPrompt,
+  workflowUrlMatches,
+} from './workflows.js';
 import { mergeRedactionFrameRegions, mapRegionsToImage, pixelateDataUrl } from './screenshot-redaction.js';
 import { buildTrustedRuntimeContext, stripTrustedRuntimeContext } from './runtime-context.js';
 import { resolveSavedDownload } from '../download-result.js';
@@ -12381,6 +12392,212 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
     };
   }
 
+  async replaySavedWorkflow(tabId, workflow, parameters = {}, onUpdate = () => {}, runOptions = {}) {
+    if (this._runningTabs.has(tabId)) throw new Error('An agent run is already in progress for this tab.');
+    if (!workflow?.id || !Array.isArray(workflow.steps) || !workflow.steps.length) {
+      throw new Error('Saved workflow is missing or invalid.');
+    }
+    await this._hydrate(tabId);
+    // Align pre-run cleanup with processMessage so a prior Act turn cannot
+    // leak click-AX CDP fallbacks, plan guards, or active-skill state into
+    // deterministic replay (or the reverse on the next turn).
+    this._resetActiveSkillsForRun(tabId, { refreshPrompt: false });
+    this._clearRunLoopState(tabId);
+    this._clickAxCdpFallbacks?.delete(tabId);
+    this.abortFlags.delete(tabId);
+    this._prepareClarificationAuthorizationForRun(tabId);
+    this.permissions.beginTurn(tabId);
+    this.conversationModes.set(tabId, 'act');
+    const completionRunToken = this._beginCompletionInvariant(tabId);
+    this._runningTabs.add(tabId);
+    const startUrl = await this._currentUrl(tabId);
+    const conversationId = await this.ensureConversationId(tabId, 'act');
+    const traceRunId = await trace.startRun({
+      conversationId,
+      userMessage: `Run saved workflow: ${workflow.name}`,
+      tabUrl: startUrl,
+      mode: 'act',
+      model: this.providerManager?.getActive?.()?.model || '',
+      providerId: this.providerManager?.activeProviderId || '',
+    });
+    let traceStatus = 'workflow_stopped';
+    let finalContent = '';
+    let matchedSteps = 0;
+
+    const finishStopped = (reason, stepIndex = 0) => {
+      const summary = `Saved workflow "${workflow.name}" stopped safely at step ${stepIndex + 1}: ${reason}.`;
+      onUpdate('tool_result', {
+        name: 'done',
+        result: { success: false, done: true, outcome: 'failed', summary, workflowReplay: true },
+      });
+      traceStatus = 'workflow_stopped';
+      finalContent = summary;
+      return { status: 'stopped', summary, reason, stepIndex, matchedSteps };
+    };
+
+    try {
+      if (!workflowUrlMatches(workflow.start, startUrl)) {
+        trace.recordNote(traceRunId, 0, 'workflow_replay_start_miss', {
+          workflowId: workflow.id,
+          expectedOrigin: workflow.start?.origin || '',
+        });
+        return finishStopped('the current page is outside the saved origin or URL family', 0);
+      }
+
+      for (let index = 0; index < workflow.steps.length; index++) {
+        if (this._checkAbort(tabId)) return finishStopped('stopped by the user', index);
+        const step = workflow.steps[index];
+        const stepUrl = await this._currentUrl(tabId);
+        if (step.scope && !workflowUrlMatches(step.scope, stepUrl)) {
+          const reason = 'page scope mismatch';
+          trace.recordNote(traceRunId, index + 1, 'workflow_replay_scope_miss', {
+            workflowId: workflow.id,
+            stepId: step.id,
+            tool: step.tool,
+            expectedOrigin: step.scope.origin,
+            expectedPathFamily: step.scope.pathFamily,
+          });
+          traceStatus = 'workflow_fallback';
+          finalContent = `Deterministic replay paused at step ${index + 1}; continuing with the agent.`;
+          return {
+            status: 'fallback',
+            reason,
+            stepIndex: index,
+            matchedSteps,
+            prompt: workflowFallbackPrompt(workflow, index, reason),
+          };
+        }
+        let executionArgs;
+        try {
+          executionArgs = resolveWorkflowArgs(step.args, parameters);
+        } catch (error) {
+          return finishStopped(error?.message || 'a required runtime parameter is missing', index);
+        }
+
+        let targetMatch = null;
+        if (step.target) {
+          const treeResult = await this.executeTool(tabId, 'get_accessibility_tree', {
+            filter: 'all',
+            maxChars: 60000,
+          });
+          const treeText = typeof treeResult === 'string'
+            ? treeResult
+            : treeResult?.pageContent || treeResult?.tree || treeResult?.content || '';
+          const candidates = parseAccessibilityTreeDescriptors(treeText);
+          targetMatch = findWorkflowTarget(step.target, candidates);
+          if (targetMatch.status !== 'matched') {
+            trace.recordNote(traceRunId, index + 1, 'workflow_replay_target_miss', {
+              workflowId: workflow.id,
+              stepId: step.id,
+              tool: step.tool,
+              match: targetMatch.status,
+            });
+            const reason = `semantic target ${targetMatch.status}`;
+            traceStatus = 'workflow_fallback';
+            finalContent = `Deterministic replay paused at step ${index + 1}; continuing with the agent.`;
+            return {
+              status: 'fallback',
+              reason,
+              stepIndex: index,
+              matchedSteps,
+              prompt: workflowFallbackPrompt(workflow, index, reason),
+            };
+          }
+          if (['click_ax', 'set_checked', 'type_ax', 'set_field', 'scroll'].includes(step.tool)) {
+            executionArgs.ref_id = targetMatch.candidate.refId;
+          }
+        }
+
+        const toolCall = {
+          id: `workflow_${workflow.id}_${step.id}_${Date.now()}`,
+          type: 'function',
+          function: { name: step.tool, arguments: JSON.stringify(executionArgs) },
+        };
+        const messages = [{ role: 'assistant', content: null, tool_calls: [toolCall] }];
+        let rawResult = null;
+        const replayUpdate = (type, data = {}) => {
+          if (type === 'tool_result' && data?.name === step.tool) rawResult = data.result;
+          if (type === 'tool_call' && data?.name === step.tool) {
+            onUpdate(type, { ...data, args: redactWorkflowArgsForTelemetry(step.args, executionArgs), workflowReplay: true });
+            return;
+          }
+          if (type === 'tool_result' && data?.name === step.tool) {
+            onUpdate(type, { ...data, result: redactWorkflowResultForTelemetry(step.tool, data.result), workflowReplay: true });
+            return;
+          }
+          if (type === 'clarify') {
+            onUpdate(type, redactWorkflowClarifyForTelemetry(data));
+            return;
+          }
+          onUpdate(type, { ...(data && typeof data === 'object' ? data : {}), workflowReplay: true });
+        };
+        const beforeUrl = stepUrl;
+        const batch = await this._executeToolBatch(
+          tabId,
+          [toolCall],
+          messages,
+          replayUpdate,
+          this.providerManager?.getActive?.(),
+          null,
+          new Set([step.tool]),
+          index + 1,
+          runOptions,
+        );
+        const afterUrl = await this._currentUrl(tabId);
+        const validation = validateWorkflowStepResult(step.expected, rawResult, { beforeUrl, afterUrl, tool: step.tool });
+        trace.recordNote(traceRunId, index + 1, 'workflow_replay_step', {
+          workflowId: workflow.id,
+          stepId: step.id,
+          tool: step.tool,
+          targetMatch: targetMatch?.status || 'not_applicable',
+          targetScore: targetMatch?.score || 0,
+          validation: validation.ok ? 'passed' : validation.reason,
+        });
+
+        if (batch?.action === 'abort') return finishStopped('stopped by the user', index);
+        if (!validation.ok) {
+          if (validation.outcomeUnknown || rawResult?.denied || rawResult?.cancelled) {
+            return finishStopped(validation.reason, index);
+          }
+          traceStatus = 'workflow_fallback';
+          finalContent = `Deterministic replay paused at step ${index + 1}; continuing with the agent.`;
+          return {
+            status: 'fallback',
+            reason: validation.reason,
+            stepIndex: index,
+            matchedSteps,
+            prompt: workflowFallbackPrompt(workflow, index, validation.reason),
+          };
+        }
+        matchedSteps += 1;
+      }
+
+      const summary = `Saved workflow "${workflow.name}" completed ${matchedSteps} step${matchedSteps === 1 ? '' : 's'} with deterministic replay.`;
+      trace.recordNote(traceRunId, workflow.steps.length + 1, 'workflow_replay_complete', {
+        workflowId: workflow.id,
+        matchedSteps,
+        modelFallbacks: 0,
+        estimatedLlmCallsSaved: matchedSteps,
+      });
+      onUpdate('tool_result', {
+        name: 'done',
+        result: { success: true, done: true, outcome: 'success', summary, workflowReplay: true },
+      });
+      traceStatus = 'done';
+      finalContent = summary;
+      return { status: 'completed', summary, matchedSteps, estimatedLlmCallsSaved: matchedSteps };
+    } finally {
+      await trace.endRun(traceRunId, { status: traceStatus, finalContent });
+      this.currentCostState.delete(tabId);
+      this._planExecutionGuards.delete(tabId);
+      this._resetActiveSkillsForRun(tabId);
+      this._runningTabs.delete(tabId);
+      this._clearRunLoopState(tabId);
+      this._clickAxCdpFallbacks?.delete(tabId);
+      this._clearCompletionInvariant(tabId, completionRunToken);
+    }
+  }
+
   async executeTool(tabId, name, args, onUpdate = null, executionContext = null) {
     if (name === 'load_skill') {
       return this._loadSkillForRun(tabId, args || {});