Skip to content

hardware_executor.md

Latest commit

 

History

History
506 lines (339 loc) · 9.04 KB

hardware_executor.md

File metadata and controls

506 lines (339 loc) · 9.04 KB
Raw

HardwareExecutor Server

Overview

HardwareExecutor provides hardware control capabilities including Arduino HID, BB-8 test fixture, robot arm, mouse control, and screenshot capture.

Server Type: Action
Deployment: HTTP (remote server)
Default Port: 8006
LLM-Selectable: ✅ Yes

Server Information

Property Value
Namespace HardwareExecutor
Server Name Echo Base MCP Server
Platform Cross-platform (requires hardware)
Tool Type action
Deployment HTTP server (stateless)

Tool Categories

1. Arduino HID Tools (Keyboard/Mouse Emulation)

2. Mouse Control Tools

3. BB-8 Test Fixture Tools

4. Robot Arm Tools

5. Screenshot Tool

Arduino HID Tools

arduino_hid_status

Get Arduino HID device status.

Returns: Dict[str, Any] with connected, status, device

arduino_hid_connect

Connect to Arduino HID device.

Returns: Dict[str, Any] with success message

arduino_hid_disconnect

Disconnect from Arduino HID device.

Returns: Dict[str, Any] with success message

type_text

Type a string of text via Arduino HID.

Parameters:

  • text (str): Text to type

Returns: Success message

Example:

await computer.run_actions([
    MCPToolCall(
        tool_key="action::type_text",
        tool_name="type_text",
        parameters={"text": "Hello, World!"}
    )
])

press_key_sequence

Press a sequence of keys.

Parameters:

  • keys (List[str]): List of key names
  • interval (float): Interval between key presses (default: 0.1)

Example:

await computer.run_actions([
    MCPToolCall(
        tool_key="action::press_key_sequence",
        tool_name="press_key_sequence",
        parameters={
            "keys": ["a", "b", "c"],
            "interval": 0.2
        }
    )
])

press_hotkey

Press multiple keys simultaneously (hotkey combination).

Parameters:

  • keys (List[str]): List of keys to press together

Example:

# Ctrl+C
await computer.run_actions([
    MCPToolCall(
        tool_key="action::press_hotkey",
        tool_name="press_hotkey",
        parameters={"keys": ["ctrl", "c"]}
    )
])

Mouse Control Tools

move_mouse

Move the mouse pointer.

Parameters:

  • x (int): X coordinate
  • y (int): Y coordinate
  • absolute (bool): Absolute (True) or relative (False) positioning (default: False)

click_mouse

Click mouse button.

Parameters:

  • button (str): "left", "right", or "middle" (default: "left")
  • count (int): Number of clicks (default: 1)
  • interval (float): Interval between clicks (default: 0.1)

press_mouse_button

Press and hold mouse button.

Parameters:

  • button (str): Mouse button (default: "left")

release_mouse_button

Release mouse button.

Parameters:

  • button (str): Mouse button (default: "left")

scroll_mouse

Scroll mouse wheel.

Parameters:

  • vertical (int): Vertical scroll amount (default: 0)
  • horizontal (int): Horizontal scroll amount (default: 0)

Example:

# Scroll down
await computer.run_actions([
    MCPToolCall(
        tool_key="action::scroll_mouse",
        tool_name="scroll_mouse",
        parameters={"vertical": -5, "horizontal": 0}
    )
])

drag_mouse

Drag mouse from start to end position.

Parameters:

  • start (Tuple[int, int]): Start (x, y) coordinates
  • end (Tuple[int, int]): End (x, y) coordinates
  • button (str): Mouse button (default: "left")
  • duration (float): Drag duration in seconds (default: 0.5)

Example:

await computer.run_actions([
    MCPToolCall(
        tool_key="action::drag_mouse",
        tool_name="drag_mouse",
        parameters={
            "start": [100, 100],
            "end": [300, 300],
            "duration": 1.0
        }
    )
])

double_click_mouse

Perform double-click.

Parameters:

  • button (str): Mouse button (default: "left")

right_click_mouse

Shortcut for right-click.

middle_click_mouse

Shortcut for middle-click.

BB-8 Test Fixture Tools

Test fixture for Surface device testing.

bb8_status

Get BB-8 test fixture status.

bb8_connect / bb8_disconnect

Connect/disconnect to BB-8.

bb8_usb_port_plug / bb8_usb_port_unplug

Plug/unplug USB device.

Parameters:

  • port_name (str): USB port name

bb8_psu_charger_plug / bb8_psu_charger_unplug

Plug/unplug PSU charger.

bb8_blade_attach / bb8_blade_detach

Attach/detach blade.

bb8_lid_open / bb8_lid_close

Open/close lid.

bb8_button_press

Press a physical button.

Parameters:

  • button_name (str): Button name

bb8_button_long_press

Long press a button.

Parameters:

  • button_name (str): Button name

Robot Arm Tools

Physical robot arm for touchscreen interaction.

robot_arm_status

Get robot arm status (position, connection).

robot_arm_connect / robot_arm_disconnect

Connect/disconnect robot arm.

touch_screen

Simulate touch at specific screen location.

Parameters:

  • location (Tuple[int, int]): (x, y) coordinates

Example:

await computer.run_actions([
    MCPToolCall(
        tool_key="action::touch_screen",
        tool_name="touch_screen",
        parameters={"location": [500, 300]}
    )
])

draw_on_screen

Draw on screen by following coordinate path.

Parameters:

  • path (List[Tuple[int, int]]): List of (x, y) coordinates

tap_screen

Simulate tap(s) on screen.

Parameters:

  • location (Tuple[int, int]): Tap location
  • count (int): Number of taps (default: 1)
  • interval (float): Interval between taps (default: 0.1)

swipe_screen

Simulate swipe gesture.

Parameters:

  • start_location (Tuple[int, int]): Start position
  • end_location (Tuple[int, int]): End position
  • duration (float): Swipe duration (default: 0.5)

long_press_screen

Simulate long press.

Parameters:

  • location (Tuple[int, int]): Press location
  • duration (float): Press duration (default: 1.0)

double_tap_screen

Simulate double tap.

Parameters:

  • location (Tuple[int, int]): Tap location

press_key

Simulate keyboard key press via robot arm.

Parameters:

  • key (str): Key to press
  • modifiers (List[str]): Modifier keys (e.g., ["ctrl", "shift"])
  • duration (float): Press duration (default: 0.1)

tap_trackpad / swipe_trackpad

Simulate trackpad interactions.

Screenshot Tool

take_screenshot

Capture a screenshot.

Returns: str - Base64-encoded image data

Example:

result = await computer.run_actions([
    MCPToolCall(
        tool_key="action::take_screenshot",
        tool_name="take_screenshot",
        parameters={}
    )
])
# result[0].data = "iVBORw0KGgoAAAANSUhEUgAA..."

Configuration

# Client configuration (Windows agent)
HostAgent:
  default:
    action:
      - namespace: HardwareExecutor
        type: http
        host: "192.168.1.100"  # Hardware server IP
        port: 8006
        path: "/mcp"

Deployment

Starting the Server

# Start hardware MCP server
python -m ufo.client.mcp.http_servers.hardware_mcp_server --host 0.0.0.0 --port 8006

# Output:
# ==================================================
# UFO Hardware MCP Server
# Hardware automation via Model Context Protocol
# Running on 0.0.0.0:8006
# ==================================================

Configuration

Default Values:

  • Host: localhost
  • Port: 8006
  • Path: /mcp

Best Practices

1. Network Configuration

# Use IP address for remote hardware
action:
  - namespace: HardwareExecutor
    type: http
    host: "192.168.1.100"  # Hardware server
    port: 8006

2. Error Handling

All tools return dict with success key:

result = await computer.run_actions([
    MCPToolCall(tool_key="action::touch_screen", parameters={"location": [100, 100]})
])

if not result[0].data.get("success"):
    logger.error(f"Touch failed: {result[0].data.get('error')}")

3. Physical Hardware Requirements

  • Arduino HID: Requires Arduino board with HID firmware
  • BB-8: Microsoft Surface test fixture
  • Robot Arm: Physical robot arm setup
  • Network: Stable network connection for HTTP communication

Use Cases

Automated Testing

# 1. Connect to hardware
await computer.run_actions([
    MCPToolCall(tool_key="action::robot_arm_connect", parameters={})
])

# 2. Touch screen at login button
await computer.run_actions([
    MCPToolCall(tool_key="action::touch_screen", parameters={"location": [500, 700]})
])

# 3. Take screenshot to verify
screenshot = await computer.run_actions([
    MCPToolCall(tool_key="action::take_screenshot", parameters={})
])

Related Documentation