Device Agents are the execution engines of UFO3's multi-device orchestration system. Each device agent operates as an autonomous, intelligent controller that translates high-level user intentions into low-level system commands. The architecture is designed for extensibility, safety, and scalability across heterogeneous computing environments.
UFO3 orchestrates tasks across multiple devices through a network of Device Agents. Originally designed as a Windows automation framework (UFO2), the architecture has evolved to support diverse platforms including Linux, macOS, and embedded systems. This document describes the abstract design principles and interfaces that enable this multi-platform capability.
Key Capabilities:
- Multi-Platform: Windows agents (HostAgent, AppAgent), Linux agent, extensible to macOS and embedded systems
- Safe Execution: Server-client separation isolates reasoning from system-level operations
- Scalable Architecture: Hierarchical agent coordination supports complex cross-device workflows
- LLM-Driven Reasoning: Dynamic decision-making using large language models
- Modular Design: Three-layer architecture (State, Strategy, Command) enables customization
Device agents implement a three-layer framework that separates concerns, promotes modularity, and enables extensibility:
graph TB
subgraph "Device Agent Architecture"
subgraph "Level-1: State Layer (FSM)"
S1[AgentState]
S2[State Machine]
S3[State Transitions]
S1 --> S2 --> S3
end
subgraph "Level-2: Strategy Layer (Execution Logic)"
P1[ProcessorTemplate<br/>Strategy Orchestrator]
P2[DATA_COLLECTION<br/>Strategies]
P3[LLM_INTERACTION<br/>Strategies]
P4[ACTION_EXECUTION<br/>Strategies]
P5[MEMORY_UPDATE<br/>Strategies]
P1 -->|manages & executes| P2
P2 --> P3 --> P4 --> P5
end
subgraph "Level-3: Command Layer (System Interface)"
C1[CommandDispatcher]
C2[MCP Tools]
C3[Atomic Commands]
C1 --> C2 --> C3
end
S3 -->|delegates to| P1
P5 -->|executes via| C1
end
LLM[Large Language Model]
P3 -.->|reasoning| LLM
LLM -.->|decisions| P4
| Layer | Level | Responsibility | Key Components | Extensibility |
|---|---|---|---|---|
| State | Level-1 | Finite State Machine governing agent lifecycle | AgentState, AgentStateManager, AgentStatus |
Register new states via @AgentStateManager.register |
| Strategy | Level-2 | Execution logic layer: processor manages sequence of modular strategies | ProcessorTemplate, ProcessingStrategy, ProcessingPhase, Middleware |
Compose custom strategies via ComposedStrategy, add middleware |
| Command | Level-3 | Atomic system operations mapped to MCP tools | BasicCommandDispatcher, Command, MCP integration |
Add new tools via client-side MCP server registration |
Design Rationale:
The three-layer separation ensures:
- State Layer (Level-1): Controls when and what to execute (state transitions, agent handoff)
- Strategy Layer (Level-2): Defines how to execute (processor orchestrates modular strategies)
- Command Layer (Level-3): Performs actual execution (deterministic system operations)
This separation allows replacing individual layers without affecting others.
The State Layer implements a Finite State Machine (FSM) that governs the agent's execution lifecycle. Each state encapsulates:
- A processor (strategy execution logic)
- Transition rules (to next state)
- Agent handoff logic (for multi-agent workflows)
stateDiagram-v2
[*] --> CONTINUE
CONTINUE --> CONTINUE: Success
CONTINUE --> PENDING: Wait for external event
CONTINUE --> CONFIRM: User confirmation needed
CONTINUE --> SCREENSHOT: Capture observation
CONTINUE --> FINISH: Task complete
CONTINUE --> FAIL: Error occurred
CONTINUE --> ERROR: Critical failure
PENDING --> CONTINUE: Event received
CONFIRM --> CONTINUE: User confirmed
CONFIRM --> FAIL: User rejected
SCREENSHOT --> CONTINUE: Screenshot captured
FINISH --> [*]
FAIL --> [*]
ERROR --> [*]
class AgentStatus(Enum):
"""Agent status enumeration"""
ERROR = "ERROR" # Critical error occurred
FINISH = "FINISH" # Task completed successfully
CONTINUE = "CONTINUE" # Normal execution
FAIL = "FAIL" # Task failed
PENDING = "PENDING" # Waiting for external event
CONFIRM = "CONFIRM" # Awaiting user confirmation
SCREENSHOT = "SCREENSHOT" # Screenshot capture neededState Registration:
New states can be registered dynamically using the @AgentStateManager.register decorator:
@AgentStateManager.register
class CustomState(AgentState):
async def handle(self, agent, context):
# Custom state logic
pass
def next_state(self, agent):
return AgentStateManager.get_state("CONTINUE")See State Layer Documentation for complete details.
The Strategy Layer implements the execution logic within each state. Each state encapsulates a processor that manages a sequence of strategies to implement step-level workflow. This layer consists of two key components:
The ProcessorTemplate orchestrates the execution of strategies:
- Registers Strategies: Configures which strategies execute in each phase
- Manages Middleware: Wraps strategy execution with logging, metrics, error handling
- Validates Dependencies: Ensures strategies have required data before execution
- Controls Execution: Sequences strategies through fixed workflow phases
graph TB
State[AgentState] -->|encapsulates| Processor[ProcessorTemplate<br/>Strategy Orchestrator]
Processor -->|1. Register| Strategies[ProcessingStrategies]
Processor -->|2. Wrap| Middleware[Middleware Chain]
Processor -->|3. Validate| Dependencies[Strategy Dependencies]
Processor -->|4. Execute| Workflow[Workflow Phases]
Workflow --> DC[DATA_COLLECTION]
Workflow --> LLM[LLM_INTERACTION]
Workflow --> AE[ACTION_EXECUTION]
Workflow --> MU[MEMORY_UPDATE]
DC & LLM & AE & MU -.->|implements| Strategies
Processor and Strategy Relationship:
- Processor: Framework that manages the sequence of strategies
- Strategy: Modular, reusable execution units
Together they form Level-2: Strategy Layer, which handles:
- Data collection and environment inspection
- Prompt construction and LLM reasoning
- Action planning and tool invocation
- Memory updates and context synchronization
ProcessingStrategies are modular execution units with a unified execute() interface:
graph LR
A[DATA_COLLECTION] --> B[LLM_INTERACTION]
B --> C[ACTION_EXECUTION]
C --> D[MEMORY_UPDATE]
A1[Screenshots<br/>UI Info<br/>System Status] --> A
B1[Prompt Construction<br/>LLM Call<br/>Response Parsing] --> B
C1[Command Dispatch<br/>MCP Execution<br/>Result Handling] --> C
D1[Memory Items<br/>Blackboard Update<br/>Context Sync] --> D
| Strategy Type | ProcessingPhase | Purpose | Examples |
|---|---|---|---|
| DATA_COLLECTION | data_collection |
Gather contextual information | Screenshot capture, UI tree extraction, system info |
| LLM_INTERACTION | llm_interaction |
Construct prompts, interact with LLM, parse responses | Prompt building, LLM reasoning, JSON parsing |
| ACTION_EXECUTION | action_execution |
Execute commands from LLM/toolkits | Click, type, scroll, API calls |
| MEMORY_UPDATE | memory_update |
Update short-term/long-term memory | Add memory items, update blackboard, sync context |
Strategy Layer Configuration Example:
Each state configures its processor with strategies and middleware:
class AppAgentProcessor(ProcessorTemplate):
def _setup_strategies(self):
# Register strategies for each phase
self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([
AppScreenshotCaptureStrategy(),
AppControlInfoStrategy()
])
self.strategies[ProcessingPhase.LLM_INTERACTION] = AppLLMInteractionStrategy()
self.strategies[ProcessingPhase.ACTION_EXECUTION] = AppActionExecutionStrategy()
self.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy()
def _setup_middleware(self):
# Add middleware for logging, metrics, error handling
self.middleware_chain = [
LoggingMiddleware(),
PerformanceMetricsMiddleware(),
ErrorHandlingMiddleware()
]See Processor Documentation and Strategy Documentation for complete details.
The Command Layer provides atomic, deterministic system operations. Each command maps to an MCP tool that executes on the device client.
sequenceDiagram
participant Agent as Device Agent (Server)
participant Dispatcher as CommandDispatcher
participant Protocol as AIP Protocol
participant Client as Device Client
participant MCP as MCP Tool
Agent->>Dispatcher: execute_commands([command1, command2])
Dispatcher->>Protocol: Send ServerMessage (COMMAND)
Protocol->>Client: WebSocket (AIP)
Client->>MCP: Route to MCP server
MCP->>MCP: Execute tool function
MCP->>Client: Return result
Client->>Protocol: Send ClientMessage (RESULT)
Protocol->>Dispatcher: Receive results
Dispatcher->>Agent: Return List[Result]
@dataclass
class Command:
"""Atomic command to be executed on device client"""
tool_name: str # MCP tool name (e.g., "click_element")
parameters: Dict[str, Any] # Tool arguments
tool_type: str # "data_collection" or "action"
call_id: str # Unique identifier!!! warning "Deterministic Execution" Commands are designed to be:
- **Atomic**: Single, indivisible operation
- **Deterministic**: Same inputs → same outputs
- **Auditable**: Full command history logged
- **Reversible**: Where possible, support undo operations
Extensibility:
New commands can be added by:
- Registering MCP tool on device client
- LLM dynamically selects tool from available MCP registry
- No server-side code changes required
See Command Layer Documentation for complete details.
Device agents use a server-client separation to balance safety, scalability, and functionality:
graph TB
subgraph "Server Side (UFO3 Orchestrator)"
Server[Device Agent Server]
State[State Machine]
Processor[Strategy Processor]
LLM[LLM Service]
Memory[Memory & Context]
Server --> State
Server --> Processor
Server --> Memory
Processor -.-> LLM
end
subgraph "Communication Layer"
AIP[AIP Protocol<br/>WebSocket]
end
subgraph "Client Side (Device)"
Client[Device Client]
Dispatcher[Command Dispatcher]
MCP[MCP Server Manager]
Tools[MCP Tools]
OS[Operating System]
Client --> Dispatcher
Dispatcher --> MCP
MCP --> Tools
Tools --> OS
end
Server <-->|Commands/Results| AIP
AIP <-->|Commands/Results| Client
| Component | Location | Responsibilities | Security Boundary |
|---|---|---|---|
| Agent Server | Orchestrator | State management, reasoning, planning, memory | Untrusted (LLM-driven decisions) |
| Device Client | Device | Command execution, MCP tool calls, resource access | Trusted (validated operations) |
| AIP Protocol | Communication | Message serialization, WebSocket transport, error handling | Secure channel (authentication, encryption) |
Why Server-Client Separation?
Safety: Isolates potentially unsafe LLM-generated decisions from direct system access. Clients validate all commands before execution.
Scalability: Single orchestrator server manages multiple device clients. Reduces per-device resource requirements.
Flexibility: Device clients can run on resource-constrained devices (embedded systems, mobile) while heavy reasoning occurs on server.
See Server-Client Architecture for complete details.
UFO3 currently supports Windows and Linux device agents, with architecture designed for extensibility to other platforms.
graph TB
subgraph "Windows Device (Two-Tier Hierarchy)"
Host[HostAgent<br/>Application Selection]
App1[AppAgent<br/>Word]
App2[AppAgent<br/>Excel]
App3[AppAgent<br/>Browser]
Host -->|delegates| App1
Host -->|delegates| App2
Host -->|delegates| App3
end
User[User Request] --> Host
HostAgent (Application-Level Coordinator):
- Selects appropriate application(s) for user request
- Decomposes tasks into application-specific subtasks
- Coordinates multiple AppAgents
- Manages application switching and data transfer
AppAgent (Application-Level Executor):
- Controls specific Windows application (Word, Excel, browser, etc.)
- Uses UI Automation for control element discovery
- Executes application-specific actions (type, click, scroll)
- Maintains application context and memory
!!! example "Windows Agent Example" User Request: "Create a chart from sales.xlsx and insert into report.docx"
1. **HostAgent** decomposes:
- Open Excel → Create chart → Copy chart
- Open Word → Paste chart
2. **AppAgent (Excel)**: Opens `sales.xlsx`, creates chart, copies to clipboard
3. **AppAgent (Word)**: Opens `report.docx`, pastes chart at cursor
graph TB
subgraph "Linux Device (Single-Tier Architecture)"
Linux[LinuxAgent<br/>Direct System Control]
Shell[Shell Commands]
Files[File Operations]
Apps[Application Launch]
Linux --> Shell
Linux --> Files
Linux --> Apps
end
User[User Request] --> Linux
LinuxAgent (System-Level Executor):
- Direct shell command execution
- File system operations
- Application launch and management
- Single-tier architecture (no application-level hierarchy)
!!! info "Architecture Difference" Windows uses two-tier hierarchy (HostAgent → AppAgent) due to:
- UI Automation framework's application-centric model
- Distinct application contexts requiring specialized agents
**Linux** uses single-tier architecture because:
- Shell provides unified interface to all system operations
- Application control occurs through same command-line interface
| Feature | Windows (UFO2) | Linux | macOS (Future) | Embedded (Future) |
|---|---|---|---|---|
| Agent Hierarchy | Two-tier (Host → App) | Single-tier | TBD | Single-tier |
| UI Control | UI Automation | X11/Wayland | Accessibility API | Platform-specific |
| Command Interface | MCP tools (Win32 API) | MCP tools (Shell) | MCP tools (AppleScript) | MCP tools (Custom) |
| Observation | Screenshot + UI tree | Screenshot + Shell output | Screenshot + UI tree | Sensor data |
| State Management | Shared FSM | Shared FSM | Shared FSM | Shared FSM |
| Strategy Layer | Processor framework | Processor framework | Processor framework | Processor framework |
| Current Status | ✅ Production | ✅ Production | 🔜 Planned | 🔜 Planned |
Extensibility Path:
Adding a new platform requires:
- Implement Agent Class: Extend
BasicAgent(inherit State layer, Processor framework) - Create Processor: Subclass
ProcessorTemplate, implement platform-specific strategies - Define MCP Tools: Register platform-specific MCP tools on device client
- Register Agent: Use
@AgentRegistry.registerdecorator
No changes to core State layer, Processor framework, or AIP protocol required.
See Agent Types Documentation for complete implementation details.
A typical device agent execution follows this lifecycle:
sequenceDiagram
participant User
participant Orchestrator
participant Agent
participant State
participant Processor
participant LLM
participant Dispatcher
participant Client
User->>Orchestrator: Submit task
Orchestrator->>Agent: Initialize agent (CONTINUE state)
loop Until FINISH/FAIL/ERROR
Agent->>State: handle(agent, context)
State->>Processor: execute strategies
Processor->>Processor: DATA_COLLECTION
Note over Processor: Screenshot, UI info
Processor->>LLM: LLM_INTERACTION
LLM-->>Processor: Action decision
Processor->>Dispatcher: ACTION_EXECUTION
Dispatcher->>Client: Execute commands
Client-->>Dispatcher: Results
Dispatcher-->>Processor: Results
Processor->>Processor: MEMORY_UPDATE
Note over Processor: Update memory, blackboard
State->>State: next_state(agent)
State->>Agent: Update agent status
end
Agent->>Orchestrator: Task complete/failed
Orchestrator->>User: Return result
- Initialization: Agent created with default state (
CONTINUE), processor, memory - State Handling: Current state's
handle()method invoked with agent and context - Strategy Execution: Processor runs strategies in sequence (DATA_COLLECTION → LLM_INTERACTION → ACTION_EXECUTION → MEMORY_UPDATE)
- State Transition: State's
next_state()determines next FSM state - Repeat/Terminate: Loop continues until terminal state (
FINISH,FAIL,ERROR)
!!! tip "Multi-Agent Handoff"
For multi-agent scenarios (e.g., Windows HostAgent → AppAgent), states implement next_agent():
```python
def next_agent(self, agent: BasicAgent) -> BasicAgent:
# HostAgent delegates to AppAgent
if agent.status == "DELEGATE_TO_APP":
return agent.create_app_agent(...)
return agent
```
Device agents maintain two types of memory:
Purpose: Track agent's execution history within a session
Implementation: Memory class with MemoryItem entries
class Memory:
"""Agent's short-term memory"""
_content: List[MemoryItem]
def add_memory_item(self, memory_item: MemoryItem):
"""Add new memory entry"""
self._content.append(memory_item)Content: Actions taken, observations made, results received
Lifetime: Single session (cleared between tasks)
Purpose: Share information across agents and sessions
Implementation: Blackboard class with multiple memory types
class Blackboard:
"""Multi-agent shared memory"""
_questions: Memory # Q&A history
_requests: Memory # Request history
_trajectories: Memory # Action trajectories
_screenshots: Memory # Visual observationsContent: Common knowledge, successful action patterns, user preferences
Lifetime: Persistent across sessions (can be saved/loaded)
Blackboard Usage Example:
Scenario: HostAgent delegates to AppAgent (Excel)
- HostAgent adds to blackboard:
- Request: "Create sales chart"
- Context: Previous analysis results
- AppAgent reads from blackboard:
- Retrieves request and context
- Adds action trajectories as executed
- Adds screenshot after chart creation
- HostAgent reads updated blackboard:
- Verifies chart creation
- Continues to next step (insert to Word)
See Memory System Documentation for complete details.
Device agents integrate with other UFO3 components:
graph TB
subgraph "UFO3 Architecture"
Session[Session/Round Manager]
Context[Global Context]
Agent[Device Agent]
Dispatcher[Command Dispatcher]
AIP[AIP Protocol]
Client[Device Client]
MCP[MCP Servers]
end
Session -->|manages lifecycle| Agent
Session -->|provides| Context
Agent -->|reads/writes| Context
Agent -->|sends commands| Dispatcher
Dispatcher -->|uses| AIP
AIP <-->|WebSocket| Client
Client -->|calls| MCP
| Component | Relationship | Description |
|---|---|---|
| Session Manager | Parent | Creates agents, manages agent lifecycle, coordinates multi-agent workflows |
| Round Manager | Sibling | Manages round-based execution, tracks round state, synchronizes with agent steps |
| Global Context | Shared State | Agent reads request/config, writes results/status, shares data across components |
| Command Dispatcher | Execution Interface | Agent sends commands, dispatcher routes to client, returns results |
| AIP Protocol | Communication | Serializes commands/results, manages WebSocket, handles errors/timeouts |
| Device Client | Executor | Receives commands, invokes MCP tools, returns results |
| MCP Servers | Tool Registry | Provides available tools, executes tool functions, returns structured results |
See Session Documentation, Context Documentation, and AIP Protocol for integration details.
Device agent architecture leverages several design patterns:
Purpose: Encapsulate state-specific behavior, enable dynamic state transitions
Implementation: AgentState abstract class, concrete state classes
class AgentState(ABC):
@abstractmethod
async def handle(self, agent, context):
"""Execute state-specific logic"""
pass
@abstractmethod
def next_state(self, agent):
"""Determine next state"""
passPurpose: Define family of algorithms (strategies), make them interchangeable
Implementation: ProcessingStrategy protocol, concrete strategy classes
class ProcessingStrategy(Protocol):
async def execute(self, agent, context) -> ProcessingResult:
"""Execute strategy logic"""
passPurpose: Define skeleton of algorithm, let subclasses override specific steps
Implementation: ProcessorTemplate abstract class
class ProcessorTemplate(ABC):
@abstractmethod
def _setup_strategies(self):
"""Subclass defines which strategies to use"""
pass
async def process(self, agent, context):
"""Template method - runs strategies in sequence"""
for phase, strategy in self.strategies.items():
result = await strategy.execute(agent, context)
# Handle result, update contextPurpose: Ensure single instance of state registry
Implementation: AgentStateManager with metaclass
class AgentStateManager(ABC, metaclass=SingletonABCMeta):
_state_mapping: Dict[str, Type[AgentState]] = {}
def get_state(self, status: str) -> AgentState:
"""Lazy load and return state instance"""
passPurpose: Register agent types, enable dynamic agent creation
Implementation: AgentRegistry decorator
@AgentRegistry.register(agent_name="appagent", processor_cls=AppAgentProcessor)
class AppAgent(BasicAgent):
passPurpose: Share data across multiple agents
Implementation: Blackboard class
class Blackboard:
_questions: Memory
_requests: Memory
_trajectories: Memory
_screenshots: Memory- Keep states focused: Each state should have single, clear responsibility
- Use rule-based transitions for deterministic flows, LLM-driven transitions for adaptive behavior
- Implement error states for graceful degradation
- Document state invariants and transition conditions
- Keep strategies atomic: Each strategy should perform one cohesive task
- Declare dependencies explicitly using
get_dependencies() - Use ComposedStrategy to combine multiple strategies within a phase
- Implement fail-fast for critical errors, continue-on-error for optional operations
- Keep commands atomic: Single, indivisible operation
- Design commands to be idempotent where possible
- Validate arguments on client side before execution
- Return structured results with success/failure status
- Use short-term memory for agent-specific execution history
- Use blackboard for multi-agent coordination and persistent knowledge
- Clear memory between sessions to avoid context pollution
- Implement memory pruning for long-running sessions
!!! warning "Security Considerations" - Validate all commands on client side before execution - Sanitize LLM outputs before converting to commands - Limit command scope via MCP tool permissions - Audit all actions for compliance and debugging - Isolate agents to prevent unauthorized cross-agent access
Deep Dive Into Layers:
- State Layer Documentation: FSM, AgentState, transitions, state registration
- Processor and Strategy Documentation: ProcessorTemplate, strategies, dependency management
- Command Layer Documentation: CommandDispatcher, MCP integration, atomic commands
Supporting Systems:
- Memory System Documentation: Memory, MemoryItem, Blackboard patterns
- Agent Types Documentation: Windows agents, Linux agent, platform-specific implementations
Integration Points:
- Server-Client Architecture: Server and client separation, communication patterns
- Server Architecture: Agent server, WebSocket manager, orchestration
- Client Architecture: Device client, MCP servers, command execution
- AIP Protocol: Agent Interaction Protocol for server-client communication
- MCP Integration: Model Context Protocol for tool execution
Key Takeaways:
✅ Three-Layer Architecture: State (FSM) → Strategy (Execution Logic) → Command (System Interface)
✅ Server-Client Separation: Safe isolation of reasoning (server) from execution (client)
✅ Multi-Platform Support: Windows (two-tier), Linux (single-tier), extensible to macOS and embedded
✅ LLM-Driven Reasoning: Dynamic decision-making with structured command output
✅ Modular & Extensible: Register new states, compose strategies, add MCP tools without core changes
✅ Memory Systems: Short-term (agent memory) and long-term (blackboard) for coordination
✅ Design Patterns: State, Strategy, Template Method, Singleton, Registry, Blackboard
The Device Agent architecture provides a robust, extensible foundation for multi-device automation. By separating concerns across three layers and isolating reasoning from execution, UFO3 achieves both safety and flexibility for orchestrating complex cross-device workflows.
Below is the reference for the BasicAgent class. All device agents inherit from BasicAgent and implement platform-specific processors and states:
::: agents.agent.basic.BasicAgent