Simplify TS computer-use templates with @onkernel/cua-agent

Replace the per-provider sampling loops and hand-written action
adapters in the Anthropic, OpenAI, and Gemini TypeScript templates
with the CuaAgent class from @onkernel/cua-agent. Each template now
provisions a Kernel browser, hands it to CuaAgent, and returns the
final answer, removing ~3500 lines of provider-specific tool
translation and screenshot-loop code.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Author: dprevoznik

pkg/templates/typescript/anthropic-computer-use/README.md

@@ -1,8 +1,8 @@
 # Kernel TypeScript Sample App - Anthropic Computer Use
 
-This is a Kernel application that implements a prompt loop using Anthropic Computer Use with Kernel's Computer Controls API.
+This is a Kernel application that runs Anthropic Computer Use against a Kernel cloud browser.
 
-It generally follows the [Anthropic Reference Implementation](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo) but uses Kernel's Computer Controls API instead of `xdotool` and `gnome-screenshot`.
+It uses [`@onkernel/cua-agent`](https://www.npmjs.com/package/@onkernel/cua-agent) to run the computer-use loop: the `CuaAgent` class translates Claude's computer-use tool calls into Kernel browser controls and feeds a fresh screenshot back on every turn. The app entry point just provisions a browser, hands it to `CuaAgent`, and returns the final answer.
 
 ## Setup
 
@@ -35,13 +35,8 @@ kernel invoke ts-anthropic-cua cua-task --payload '{"query": "Navigate to https:
 
 When enabled, the response will include a `replay_url` field with a link to view the recorded session.
 
-## Known Limitations
-
-### Cursor Position
-
-The `cursor_position` action is not supported with Kernel's Computer Controls API. If the model attempts to use this action, an error will be returned. This is a known limitation that does not significantly impact most computer use workflows, as the model typically tracks cursor position through screenshots.
-
 ## Resources
 
+- [@onkernel/cua-agent](https://www.npmjs.com/package/@onkernel/cua-agent)
 - [Anthropic Computer Use Documentation](https://docs.anthropic.com/en/docs/build-with-claude/computer-use)
 - [Kernel Documentation](https://www.kernel.sh/docs/quickstart)

pkg/templates/typescript/anthropic-computer-use/index.ts

@@ -1,5 +1,6 @@
 import { Kernel, type KernelContext } from '@onkernel/sdk';
-import { samplingLoop } from './loop';
+import { CuaAgent } from '@onkernel/cua-agent';
+import type { AssistantMessage } from '@onkernel/cua-ai';
 import { KernelBrowserSession } from './session';
 
 const kernel = new Kernel();
@@ -16,11 +17,40 @@ interface QueryOutput {
   replay_url?: string;
 }
 
-// LLM API Keys are set in the environment during `kernel deploy <filename> -e ANTHROPIC_API_KEY=XXX`
-// See https://www.kernel.sh/docs/launch/deploy#environment-variables
-const ANTHROPIC_API_KEY = process.env.ANTHROPIC_API_KEY;
+const CURRENT_DATE = new Intl.DateTimeFormat('en-US', {
+  weekday: 'long',
+  month: 'long',
+  day: 'numeric',
+  year: 'numeric',
+}).format(new Date());
+
+// System prompt optimized for the Kernel cloud browser environment.
+const SYSTEM_PROMPT = `<SYSTEM_CAPABILITY>
+* You are utilising an Ubuntu virtual machine using ${process.arch} architecture with internet access.
+* When you connect to the display, CHROMIUM IS ALREADY OPEN. The url bar is not visible but it is there.
+* If you need to navigate to a new page, use ctrl+l to focus the url bar and then enter the url.
+* You won't be able to see the url bar from the screenshot but ctrl-l still works.
+* As the initial step click on the search bar.
+* When viewing a page it can be helpful to zoom out so that you can see everything on the page.
+* Either that, or make sure you scroll down to see everything before deciding something isn't available.
+* Scroll action: scroll_amount and the tool result are in wheel units (not pixels).
+* When using your computer function calls, they take a while to run and send back to you.
+* Where possible/feasible, try to chain multiple of these calls all into one function calls request.
+* The current date is ${CURRENT_DATE}.
+* After each step, take a screenshot and carefully evaluate if you have achieved the right outcome.
+* Explicitly show your thinking: "I have evaluated step X..." If not correct, try again.
+* Only when you confirm a step was executed correctly should you move on to the next one.
+</SYSTEM_CAPABILITY>
+
+<IMPORTANT>
+* When using Chromium, if a startup wizard appears, IGNORE IT. Do not even click "skip this step".
+* Instead, click on the search bar on the center of the screen where it says "Search or enter address", and enter the appropriate search term or URL there.
+</IMPORTANT>`;
 
-if (!ANTHROPIC_API_KEY) {
+// LLM API keys are set in the environment during `kernel deploy <filename> -e ANTHROPIC_API_KEY=XXX`.
+// See https://www.kernel.sh/docs/launch/deploy#environment-variables
+// CuaAgent reads ANTHROPIC_API_KEY (or ANTHROPIC_OAUTH_TOKEN) from the environment by default.
+if (!process.env.ANTHROPIC_API_KEY) {
   throw new Error('ANTHROPIC_API_KEY is not set');
 }
 
@@ -42,44 +72,32 @@ app.action<QueryInput, QueryOutput>(
     console.log('Kernel browser live view url:', session.liveViewUrl);
 
     try {
-      // Run the sampling loop
-      const finalMessages = await samplingLoop({
-        model: 'claude-sonnet-4-6',
-        messages: [{
-          role: 'user',
-          content: payload.query,
-        }],
-        apiKey: ANTHROPIC_API_KEY,
-        thinkingBudget: 1024,
-        kernel,
-        sessionId: session.sessionId,
+      const agent = new CuaAgent({
+        browser: session.browser,
+        client: kernel,
+        initialState: {
+          model: 'anthropic:claude-sonnet-4-6',
+          systemPrompt: SYSTEM_PROMPT,
+        },
       });
 
-      // Extract the final result from the messages
-      if (finalMessages.length === 0) {
-        throw new Error('No messages were generated during the sampling loop');
-      }
-
-      const lastMessage = finalMessages[finalMessages.length - 1];
-      if (!lastMessage) {
-        throw new Error('Failed to get the last message from the sampling loop');
-      }
+      await agent.prompt(payload.query);
 
-      const result = typeof lastMessage.content === 'string'
-        ? lastMessage.content
-        : lastMessage.content.map(block =>
-          block.type === 'text' ? block.text : ''
-        ).join('');
+      const lastAssistant = [...agent.state.messages]
+        .reverse()
+        .find((message): message is AssistantMessage => message.role === 'assistant');
+      const result = lastAssistant?.content
+        .flatMap((block) => (block.type === 'text' ? [block.text] : []))
+        .join('') ?? '';
 
-      // Stop session and get replay URL if recording was enabled
       const sessionInfo = await session.stop();
 
       return {
         result,
         replay_url: sessionInfo.replayViewUrl,
       };
     } catch (error) {
-      console.error('Error in sampling loop:', error);
+      console.error('Error running CUA task:', error);
       await session.stop();
       throw error;
     }

pkg/templates/typescript/anthropic-computer-use/loop.ts

@@ -4,8 +4,9 @@
   "type": "module",
   "private": true,
   "dependencies": {
-    "@anthropic-ai/sdk": "^0.71.2",
-    "@onkernel/sdk": "^0.35.0"
+    "@onkernel/cua-agent": "^0.3.4",
+    "@onkernel/cua-ai": "^0.3.1",
+    "@onkernel/sdk": "0.49.0"
   },
   "devDependencies": {
     "@types/node": "^22.15.17",