UFO³ — Weaving the Digital Agent Galaxy

From isolated device agents to interconnected constellations — Building the Digital Agent Galaxy

🚀 What is UFO³ Galaxy?

UFO³ Galaxy is a revolutionary cross-device orchestration framework that transforms isolated device agents into a unified digital ecosystem. It models complex user requests as Task Constellations (星座) — dynamic distributed DAGs where nodes represent executable subtasks and edges capture dependencies across heterogeneous devices.

🎯 The Vision

Building truly ubiquitous intelligent agents requires moving beyond single-device automation. UFO³ Galaxy addresses four fundamental challenges in cross-device agent orchestration:

🔄 Asynchronous Parallelism
Enabling concurrent task execution across multiple devices while maintaining correctness through event-driven coordination and safe concurrency control

⚡ Dynamic Adaptation
Real-time workflow evolution in response to intermediate results, transient failures, and runtime observations without workflow abortion

🌐 Distributed Coordination
Reliable, low-latency communication across heterogeneous devices via WebSocket-based Agent Interaction Protocol with fault tolerance

🛡️ Safety Guarantees
Formal invariants ensuring DAG consistency during concurrent modifications and parallel execution, verified through rigorous proofs

🏗️ Architecture

UFO³ Galaxy Layered Architecture — From natural language to distributed execution

Layered Design

UFO³ Galaxy follows a hierarchical orchestration model that separates global coordination from local execution. This architecture enables scalable cross-device orchestration while maintaining consistent control and responsiveness across diverse operating systems and network environments.

🎛️ Hierarchical Control Plane

ConstellationClient serves as the global control plane, maintaining a live registry of all connected device agents with their:

Capability profiles and system specifications
Runtime health metrics and availability status
Current load and resource utilization

This registry enables intelligent task placement based on device capabilities, avoiding mismatches between task requirements and device capacity.

Each device hosts a device agent server that manages local orchestration through persistent WebSocket sessions with ConstellationClient. The server:

Maintains execution contexts on the host
Provides unified interface to underlying tools via MCP servers
Handles task execution, telemetry streaming, and resource monitoring

Clean separation: Global orchestration policies are decoupled from device-specific heterogeneity, providing consistent abstraction across endpoints with different OS, hardware, or network conditions.

🔄 Orchestration Flow

DAG Synthesis: ConstellationClient invokes the Constellation Agent to construct a TaskConstellation—a dynamic DAG encoding task decomposition, dependencies, and device mappings
Device Assignment: Each TaskStar (DAG node) is assigned to suitable device agents based on capability profiles and system load
Asynchronous Execution: The Constellation Orchestrator executes the DAG in an event-driven manner:
- Task completions trigger dependent nodes
- Failures prompt retry, migration, or partial DAG rewrites
- Workflows adapt to real-time system dynamics (device churn, network variability)

Result: Highly parallel and resilient execution that sustains workflow completion even as subsets of devices fail or reconnect.

🔌 Cross-Agent Communication

The Agent Interaction Protocol (AIP) handles all cross-agent interactions:

Agent registration and capability synchronization
Task dispatch and progress reporting
Result aggregation and telemetry streaming

Built on persistent WebSocket channels, AIP provides:

Lightweight: Minimal overhead for control messages
Bidirectional: Full-duplex communication between client and agents
Multiplexed: Concurrent message streams over single connection
Low-latency: Fast propagation of control signals and state updates
Resilient: Maintains global consistency despite intermittent connectivity

Together, these design elements form a cohesive foundation for orchestrating large-scale, heterogeneous, and adaptive workflows across a resilient multi-device execution fabric.

✨ Core Design Principles

UFO³ Galaxy realizes cross-device orchestration through five tightly integrated design principles:

1. 🌟 Declarative Decomposition into Dynamic DAG (Task Constellation)

Natural-language or programmatic requests are decomposed by the Constellation Agent into a structured DAG of TaskStars (nodes) and TaskStarLines (edges) that encode workflow logic, dependencies, and device assignments. This declarative structure is amenable to automated scheduling, introspection, and dynamic modification throughout execution.

Key Benefits:

📋 Declarative structure for automated scheduling
🔍 Runtime introspection for workflow visibility
✏️ Dynamic rewriting throughout execution
🔄 Automated orchestration across heterogeneous devices

graph LR
    A[User Intent] --> B[Constellation Agent]
    B --> C[Task Constellation DAG]
    C --> D[TaskStar 1<br/>Windows]
    C --> E[TaskStar 2<br/>Linux GPU]
    C --> F[TaskStar 3<br/>Linux CPU]
    C --> G[TaskStar 4<br/>Mobile]
    E --> H[TaskStar 5]
    F --> H
    G --> H

Learn more →

2. 🔄 Continuous, Result-Driven Graph Evolution

The Task Constellation is a living data structure that evolves in response to execution feedback. Intermediate outputs, transient failures, and new observations trigger controlled rewrites—adding diagnostic TaskStars, creating fallbacks, rewiring dependencies, or pruning completed nodes—so the system adapts dynamically instead of aborting on errors.

Adaptation Mechanisms:

🩺 Diagnostic TaskStars added for debugging
🛡️ Fallback creation for error recovery
🔗 Dependency rewiring for workflow optimization
✂️ Node pruning after completion

The Constellation Agent operates in two modes:

Creation Mode: Synthesizes initial DAG from user request with device-aware task decomposition
Editing Mode: Incrementally refines constellation based on task completion events and runtime feedback

Learn more →

3. 🎯 Heterogeneous, Asynchronous, and Safe Orchestration

Each Task Star is matched to the most suitable device agent via rich Agent Profiles reflecting OS, hardware capabilities, and installed tools. The Constellation Orchestrator executes tasks asynchronously, allowing multiple TaskStars to progress in parallel.

Safety Guarantees:

🔒 Safe assignment locking prevents race conditions
📅 Event-driven scheduling monitors DAG readiness
✅ DAG consistency checks maintain structural integrity
🔄 Batched edits ensure atomicity
📐 Formal verification reinforces correctness
⏱️ Timeout protection prevents deadlocks

These mechanisms collectively ensure high efficiency without compromising reliability.

Learn more →

4. 🔌 Unified Agent Interaction Protocol (AIP)

Built atop persistent WebSocket channels, AIP provides a unified, secure, and fault-tolerant layer for the entire agent ecosystem.

Core Capabilities:

📝 Agent registry with capability profiles
🔐 Session management for secure communication
📤 Task dispatch with intelligent routing
🎯 Coordination primitives for distributed workflows
💓 Heartbeat monitoring for health tracking
🔌 Automatic reconnection under network fluctuations
🔄 Retry mechanisms for reliability

Architecture Benefits:

🪶 Lightweight interface for easy integration
🧩 Extensible design supports new agent types
🛡️ Fault tolerance ensures continuous operation

This protocol abstracts OS and network heterogeneity, enabling seamless collaboration among agents across desktops, servers, and edge devices, while allowing new agents to integrate seamlessly into the UFO³ ecosystem.

Learn more →

5. 🛠️ Template-Driven Framework for Device Agents

To democratize agent creation, UFO³ provides a lightweight development template and toolkit for rapidly building new device agents.

Development Framework:

📄 Capability declaration defines agent profiles
🔗 Environment binding connects to local systems
🧩 MCP server integration for tool augmentation
🔧 Modular design accelerates development

Model Context Protocol (MCP) Integration:

🎁 Tool packages via MCP servers
🔌 Plug-and-play capability extension
🌐 Cross-platform tool standardization
🚀 Rapid prototyping of new agents

This modular architecture maintains consistency across the constellation while enabling developers to extend UFO³ to new platforms (mobile, web, IoT, embedded systems, etc.) with minimal effort.

🔌 Extensibility: UFO³ is designed as a universal framework that supports developing new device agents for different platforms (mobile, web, IoT, embedded systems, etc.) and applications. Through the Agent Interaction Protocol (AIP), custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device automation. Want to build your own device agent? See our Creating Custom Device Agents tutorial to learn how to extend UFO³ to new platforms.

Learn more → | MCP Integration →

🎯 Key Capabilities

🌐 Cross-Device Collaboration

Execute workflows that span Windows desktops, Linux servers, GPU clusters, mobile devices, and edge nodes—all from a single natural language request.

⚡ Asynchronous Parallelism

Automatically identify parallelizable subtasks and execute them concurrently across devices through:

Event-driven scheduling that continuously monitors DAG topology for ready tasks
Non-blocking execution with Python asyncio for maximum concurrency
Dynamic adaptation that integrates new tasks without interrupting running execution

Result: Dramatically reduced end-to-end latency compared to sequential execution.

🛡️ Safety & Consistency

Three formal invariants (I1-I3) enforced at runtime for DAG correctness
Safe assignment locking prevents race conditions during concurrent modifications
Acyclicity validation ensures no circular dependencies
State merging algorithm preserves execution progress during dynamic edits
Timeout protection prevents deadlocks from agent failures

🔄 Dynamic Workflow Evolution

Dual-mode operation: Separate creation and editing phases with controlled transitions
Feedback-driven adaptation: Task completion events trigger intelligent constellation refinement
LLM-powered reasoning: ReAct architecture for context-aware DAG modifications
Undo/redo support: ConstellationEditor with command pattern for safe interactive editing

👁️ Rich Observability

Real-time constellation visualization with DAG topology updates
Event bus with publish-subscribe pattern for monitoring task progress
Detailed execution logs with markdown trajectory support
Task status tracking (pending, running, completed, failed, cancelled)
Dependency graph inspection and validation tools

🎨 Use Cases

🖥️ Software Development & Deployment

"Clone the repo on my laptop, build the Docker image on the GPU server, deploy to staging, and run the test suite on the CI cluster."

Workflow DAG:

graph LR
    A[Clone<br/>Windows] --> B[Build<br/>Linux GPU]
    B --> C[Deploy<br/>Linux Server]
    C --> D[Test<br/>Linux CI]

📊 Data Science Workflows

"Fetch the dataset from cloud storage, preprocess on the Linux workstation, train the model on the A100 node, and generate a visualization dashboard on my Windows machine."

Workflow DAG:

graph LR
    A[Fetch<br/>Any] --> B[Preprocess<br/>Linux]
    B --> C[Train<br/>Linux GPU]
    C --> D[Visualize<br/>Windows]

📝 Cross-Platform Document Processing

"Extract data from Excel on Windows, process with Python scripts on Linux, generate PDF reports, and send summary emails."

Workflow DAG:

graph LR
    A[Extract<br/>Windows] --> B[Process<br/>Linux]
    B --> C[Generate PDF<br/>Windows]
    B --> D[Send Email<br/>Windows]

🔬 Distributed System Monitoring

"Collect server logs from all Linux machines, analyze for errors, generate alerts, and create a consolidated report."

Workflow DAG:

graph LR
    A[Collect Logs<br/>Linux 1] --> D[Analyze Errors<br/>Any]
    B[Collect Logs<br/>Linux 2] --> D
    C[Collect Logs<br/>Linux 3] --> D
    D --> E[Generate Report<br/>Windows]

🏢 Enterprise Automation

"Query the database on the server, process the results, update Excel spreadsheets on Windows, and generate PowerPoint presentations."

Workflow DAG:

graph LR
    A[Query DB<br/>Linux] --> B[Process Data<br/>Any]
    B --> C[Update Excel<br/>Windows]
    B --> D[Create PPT<br/>Windows]

🗺️ Documentation Structure

🚀 Quick Start

Get UFO³ Galaxy up and running in minutes with our step-by-step guide

👥 Galaxy Client

Device coordination, connection management, and ConstellationClient API

🧠 Constellation Agent

LLM-driven task decomposition, DAG creation, and dynamic workflow evolution

⚙️ Constellation Orchestrator

Asynchronous execution engine, event-driven coordination, and safety guarantees

📊 Task Constellation

DAG structure, TaskStar nodes, TaskStarLine edges, and constellation editor

🆔 Agent Registration

Device registry, agent profiles, and registration flow

🌐 Agent Interaction Protocol

WebSocket messaging, protocol specification, and communication patterns

⚙️ Configuration

Device pools, capabilities, and orchestration policies

🚦 Getting Started

Ready to build your Digital Agent Galaxy? Follow these steps:

1. Install UFO³

# Clone the repository
git clone https://github.com/microsoft/UFO.git
cd UFO

# Install dependencies
pip install -r requirements.txt

2. Configure Device Pool

Create configuration files in config/galaxy/:

config/galaxy/devices.yaml - Define your devices:

devices:
  - device_id: "windowsagent"
    server_url: "ws://localhost:5005/ws"
    os: "windows"
    capabilities:
      - "web_browsing"
      - "office_applications"
      - "file_management"
    metadata:
      location: "home_office"
      os: "windows"
      performance: "medium"
    max_retries: 5
    
  - device_id: "linux_agent_1"
    server_url: "ws://localhost:5001/ws"
    os: "linux"
    capabilities:
      - "server"
      - "python"
      - "docker"
    metadata:
      os: "linux"
      performance: "high"
      logs_file_path: "/root/log/log1.txt"
    auto_connect: true
    max_retries: 5
    
  - device_id: "mobile_agent_1"
    server_url: "ws://localhost:5002/ws"
    os: "android"
    capabilities:
      - "mobile"
      - "adb"
      - "ui_automation"
    metadata:
      os: "android"
      performance: "medium"
      device_type: "smartphone"
    auto_connect: true
    max_retries: 5

config/galaxy/constellation.yaml - Configure runtime settings:

# Constellation Runtime Settings
CONSTELLATION_ID: "my_constellation"
HEARTBEAT_INTERVAL: 30.0  # Heartbeat interval in seconds
RECONNECT_DELAY: 5.0  # Delay before reconnecting in seconds
MAX_CONCURRENT_TASKS: 6  # Maximum concurrent tasks
MAX_STEP: 15  # Maximum steps per session

# Device Configuration
DEVICE_INFO: "config/galaxy/devices.yaml"

# Logging Configuration
LOG_TO_MARKDOWN: true

See Galaxy Configuration for complete documentation.

3. Start Device Agents

On each device, launch the Agent Server. For detailed setup instructions, see the respective quick start guides:

On Windows:

See Windows Agent (UFO²) Quick Start →

On Linux:

See Linux Agent Quick Start →

On Mobile (Android):

See Mobile Agent Quick Start →

4. Launch Galaxy Client

Interactive Mode:

python -m galaxy --interactive

Direct Request:

python -m galaxy "Your cross-device task here"

Programmatic API:

from galaxy.galaxy_client import GalaxyClient

async def main():
    client = GalaxyClient(session_name="my_session")
    await client.initialize()
    result = await client.process_request("Your task request")
    await client.shutdown()

For detailed instructions, see the Quick Start Guide.

🔧 System Components

UFO³ Galaxy consists of several integrated components working together:

Core Components

Component	Location	Responsibility
GalaxyClient	`galaxy/galaxy_client.py`	Session management, user interaction, orchestration coordination
ConstellationClient	`galaxy/client/constellation_client.py`	Device management, connection lifecycle, task assignment
ConstellationAgent	`galaxy/agents/constellation_agent.py`	LLM-driven DAG synthesis and evolution, state machine control
TaskConstellationOrchestrator	`galaxy/constellation/orchestrator/`	Asynchronous execution, event coordination, safety enforcement
TaskConstellation	`galaxy/constellation/task_constellation.py`	DAG data structure, validation, and modification APIs
DeviceManager	`galaxy/client/device_manager.py`	WebSocket connections, heartbeat monitoring, message routing

Supporting Infrastructure

Component	Purpose
Event Bus	Publish-subscribe system for constellation events
Observer Pattern	Event listeners for visualization and synchronization
Device Registry	Centralized device information and capability tracking
Agent Profile	Device metadata and capability declarations
MCP Servers	Tool augmentation via Model Context Protocol

For detailed component documentation, see the respective sections in Documentation Structure.

Technology Stack

Layer	Technologies
Programming	Python 3.10+, asyncio, dataclasses
Communication	WebSockets, JSON-RPC
LLM Integration	OpenAI API, Azure OpenAI, Gemini, Claude, Custom Models
Tool Augmentation	Model Context Protocol (MCP)
Configuration	YAML, Pydantic models
Logging	Python logging, Rich console, Markdown trajectory
Testing	pytest, mock agents

🌟 From Devices to Constellations to Galaxy

UFO³ represents a paradigm shift in intelligent automation:

Single Device → Isolated agents operating within one OS
Task Constellation → Coordinated multi-device workflows for one task
Digital Agent Galaxy → Interconnected constellations spanning your entire digital estate

Over time, multiple constellations can interconnect, weaving together agents, devices, and capabilities into a self-organizing Digital Agent Galaxy. This design elevates cross-device automation from a brittle engineering challenge to a unified orchestration paradigm, where multi-device workflows become naturally expressive, paving the way for large-scale, adaptive, and resilient intelligent ubiquitous computing systems.

📊 Performance Monitoring & Evaluation

UFO³ Galaxy provides comprehensive performance monitoring and evaluation tools to analyze multi-device workflow execution:

Automated Metrics Collection

Galaxy automatically collects detailed performance metrics during execution through an event-driven observer pattern:

Task Metrics: Execution times, success rates, bottleneck identification
Constellation Metrics: DAG statistics, parallelism analysis, critical path computation
Modification Metrics: Dynamic editing patterns and adaptation frequency
Device Metrics: Per-device performance and resource utilization

All metrics are captured in real-time without impacting execution performance and saved to structured JSON files for programmatic analysis.

Trajectory Report

Galaxy automatically generates a comprehensive Markdown trajectory report (output.md) documenting the complete execution lifecycle:

logs/galaxy/<task_name>/output.md

This human-readable report includes:

Step-by-step execution timeline with agent actions
Interactive DAG topology visualizations showing constellation evolution
Detailed task execution logs with results and errors
Device connection status and coordination events
Complete before/after constellation states at each step

The trajectory report provides visual debugging and workflow understanding, complementing the quantitative result.json metrics.

Result JSON Format

After each session, Galaxy also generates a comprehensive result.json file containing:

logs/galaxy/<task_name>/result.json

This file includes:

Complete session metadata and execution timeline
Task-by-task performance breakdown
Constellation statistics (parallelism ratio, critical path, max concurrency)
Modification history showing DAG evolution
Final results and outcomes

Example Key Metrics:

Metric	Description	Use Case
`parallelism_ratio`	Efficiency of parallel execution (total_work / critical_path)	Optimization target
`critical_path_length`	Minimum possible execution time	Theoretical performance limit
`average_task_duration`	Mean task execution time	Baseline performance
`modification_count`	Number of dynamic DAG edits	Adaptability analysis

Performance Analysis Tools

import json

# Load session results
with open("logs/galaxy/task_32/result.json", 'r') as f:
    result = json.load(f)

# Extract key metrics
metrics = result["session_results"]["metrics"]
task_stats = metrics["task_statistics"]
const_stats = result["session_results"]["final_constellation_stats"]

print(f"✅ Success Rate: {task_stats['success_rate'] * 100:.1f}%")
print(f"⏱️  Avg Task Duration: {task_stats['average_task_duration']:.2f}s")
print(f"🔀 Parallelism Ratio: {const_stats['parallelism_ratio']:.2f}")

Documentation:

Trajectory Report Guide - Complete guide to the human-readable execution log with DAG visualizations
Performance Metrics Guide - Comprehensive metrics documentation with analysis examples
Result JSON Reference - Complete schema reference and programmatic access guide

📚 Learn More

Research Paper: UFO³: Weaving the Digital Agent Galaxy (Coming Soon)
UFO² (Desktop AgentOS): Documentation
UFO (Original): GitHub Repository

🤝 Contributing

We welcome contributions! Whether you're building new device agents, improving orchestration algorithms, or enhancing the protocol, check out our Contributing Guide on GitHub.

📄 License

UFO³ Galaxy is released under the MIT License.

Transform your distributed devices into a unified digital collective.

UFO³ Galaxy — Where every device is a star, and every task is a constellation.

FilesExpand file tree

overview.md

Latest commit

History

overview.md

File metadata and controls

UFO³ — Weaving the Digital Agent Galaxy

🚀 What is UFO³ Galaxy?

🎯 The Vision

🏗️ Architecture

Layered Design

🎛️ Hierarchical Control Plane

🔄 Orchestration Flow

🔌 Cross-Agent Communication

✨ Core Design Principles

1. 🌟 Declarative Decomposition into Dynamic DAG (Task Constellation)

2. 🔄 Continuous, Result-Driven Graph Evolution

3. 🎯 Heterogeneous, Asynchronous, and Safe Orchestration

4. 🔌 Unified Agent Interaction Protocol (AIP)

5. 🛠️ Template-Driven Framework for Device Agents

🎯 Key Capabilities

🌐 Cross-Device Collaboration

⚡ Asynchronous Parallelism

🛡️ Safety & Consistency

🔄 Dynamic Workflow Evolution

👁️ Rich Observability

🎨 Use Cases

🖥️ Software Development & Deployment

📊 Data Science Workflows

📝 Cross-Platform Document Processing

🔬 Distributed System Monitoring

🏢 Enterprise Automation

🗺️ Documentation Structure

🚀 Quick Start

👥 Galaxy Client

🧠 Constellation Agent

⚙️ Constellation Orchestrator

📊 Task Constellation

🆔 Agent Registration

🌐 Agent Interaction Protocol

⚙️ Configuration

🚦 Getting Started

1. Install UFO³

2. Configure Device Pool

3. Start Device Agents

4. Launch Galaxy Client

🔧 System Components

Core Components

Supporting Infrastructure

Technology Stack

🌟 From Devices to Constellations to Galaxy

📊 Performance Monitoring & Evaluation

Automated Metrics Collection

Trajectory Report

Result JSON Format

Performance Analysis Tools

📚 Learn More

🤝 Contributing

📄 License