Skip to content

overview.md

Latest commit

 

History

History
196 lines (140 loc) · 7.6 KB

overview.md

File metadata and controls

196 lines (140 loc) · 7.6 KB
Raw

HostAgent: Desktop Orchestrator

HostAgent serves as the centralized control plane of UFO². It interprets user-specified goals, decomposes them into structured subtasks, instantiates and dispatches AppAgent modules, and coordinates their progress across the system. HostAgent provides system-level services for introspection, planning, application lifecycle management, and multi-agent synchronization.

Architecture Overview

Operating atop the native Windows substrate, HostAgent monitors active applications, issues shell commands to spawn new processes as needed, and manages the creation and teardown of application-specific AppAgent instances. All coordination occurs through a persistent state machine, which governs the transitions across execution phases.

![HostAgent Architecture](../../img/hostagent2.png) Figure: HostAgent architecture showing the finite state machine, processing pipeline, and interactions with AppAgents through the Blackboard pattern.

Core Responsibilities

Task Decomposition

Given a user's natural language input, HostAgent identifies the underlying task goal and decomposes it into a dependency-ordered subtask graph.

Example: User request "Extract data from Word and create an Excel chart" becomes:

  1. Extract table from Word document
  2. Create chart in Excel with extracted data
![Task Decomposition](../../img/decomposition.png) Figure: HostAgent decomposes user requests into sequential subtasks, assigns each to the appropriate application, and orchestrates AppAgents to complete them in dependency order.

Application Lifecycle Management

For each subtask, HostAgent inspects system process metadata (via UIA APIs) to determine whether the target application is running. If not, it launches the program and registers it with the runtime.

AppAgent Instantiation

HostAgent spawns the corresponding AppAgent for each active application, providing it with task context, memory references, and relevant toolchains (e.g., APIs, documentation).

Task Scheduling and Control

The global execution plan is serialized into a finite state machine (FSM), allowing HostAgent to enforce execution order, detect failures, and resolve dependencies across agents. See State Machine Details for the FSM architecture.

Shared State Communication

HostAgent reads from and writes to a global blackboard, enabling inter-agent communication and system-level observability for debugging and replay.

Key Characteristics

  • Scope: Desktop-level orchestrator (system-wide, not application-specific)
  • Lifecycle: Single instance per session, persists throughout task execution
  • Hierarchy: Parent agent that manages multiple child AppAgents
  • Communication: Owns and coordinates the shared Blackboard
  • Control: 7-state finite state machine with 4-phase processing pipeline

Execution Workflow

sequenceDiagram
    participant User
    participant HostAgent
    participant Blackboard
    participant AppAgent1
    participant AppAgent2
    
    User->>HostAgent: "Extract Word table, create Excel chart"
    HostAgent->>HostAgent: Decompose into subtasks
    HostAgent->>Blackboard: Write subtask 1
    HostAgent->>AppAgent1: Create/Get Word AppAgent
    AppAgent1->>AppAgent1: Execute Word task
    AppAgent1->>Blackboard: Write result 1
    AppAgent1-->>HostAgent: Return FINISH
    
    HostAgent->>Blackboard: Read result 1
    HostAgent->>Blackboard: Write subtask 2
    HostAgent->>AppAgent2: Create/Get Excel AppAgent
    AppAgent2->>Blackboard: Read result 1
    AppAgent2->>AppAgent2: Execute Excel task
    AppAgent2->>Blackboard: Write result 2
    AppAgent2-->>HostAgent: Return FINISH
    
    HostAgent->>HostAgent: Verify completion
    HostAgent-->>User: Task completed
Loading

Deep Dive Topics

Input and Output

HostAgent Input

Input Description Type
User Request Natural language task description String
Application Information Active application metadata List of Dicts
Desktop Screenshots Visual context of desktop state Image
Previous Sub-Tasks Completed subtask history List of Dicts
Previous Plan Planned future subtasks List of Strings
Blackboard Shared memory space Dictionary

HostAgent Output

Output Description Type
Observation Desktop screenshot analysis String
Thought Reasoning process String
Current Sub-Task Active subtask description String
Message Information for AppAgent String
ControlLabel Selected application index String
ControlText Selected application name String
Plan Future subtask sequence List of Strings
Status Agent state (CONTINUE/ASSIGN/FINISH/etc.) String
Comment User-facing information String
Questions Clarification requests List of Strings
Bash System command to execute String

Example Output:

{
    "Observation": "Desktop shows Microsoft Word with document open containing a table",
    "Thought": "User wants to extract data from Word first",
    "Current Sub-Task": "Extract the table data from the document",
    "Message": "Starting data extraction from Word document",
    "ControlLabel": "0",
    "ControlText": "Microsoft Word - Document1",
    "Plan": ["Extract table from Word", "Create chart in Excel"],
    "Status": "ASSIGN",
    "Comment": "Delegating table extraction to Word AppAgent",
    "Questions": [],
    "Bash": ""
}

Related Documentation

Architecture & Design:

Configuration:

System Integration:

API Reference

:::agents.agent.host_agent.HostAgent

Summary

HostAgent is the desktop-level orchestrator that:

  • Decomposes tasks and coordinates AppAgents
  • Operates at system level, not application level
  • Uses a 7-state FSM: CONTINUE → ASSIGN → AppAgent → CONTINUE → FINISH
  • Executes a 4-phase pipeline: DATA_COLLECTION → LLM → ACTION → MEMORY
  • Creates, caches, and reuses AppAgent instances
  • Provides shared Blackboard memory for all agents
  • Maintains single instance per session managing multiple AppAgents

Next Steps:

  1. Read State Machine for FSM details
  2. Read Processing Strategy for pipeline architecture
  3. Read Command System for available desktop operations
  4. Read AppAgent for application-level execution