You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/embedded-and-microcontrollers/device-connect-strands/background.md
+18-36Lines changed: 18 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,57 +6,39 @@ weight: 2
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Why connect AI agents to edge devices?
9
+
## Physical AI starts with connectivity
10
10
11
-
Arm processors are at the heart of a remarkable range of systems - from Cortex-M microcontrollers in industrial sensors to Neoverse servers running in the cloud. That breadth of hardware is one of Arm's greatest strengths, but it raises a practical question for AI developers: how do you give an agent structured, safe access to devices that are physically distributed and built on different software stacks?
11
+
Arm processors are at the heart of a remarkable range of systems, from Cortex-M microcontrollers in industrial sensors to Neoverse servers running in the cloud. That breadth of hardware is one of Arm’s greatest strengths, but it raises a practical question for AI developers: how do you give an agent structured, safe access to devices that are physically distributed and built on different software stacks?
12
12
13
-
Device Connect is Arm's answer to that question. It's a platform layer that handles device registration, discovery, and remote procedure calls across a network of devices, with no bespoke networking code required. Strands is an open-source agent SDK from AWS that takes a model-driven approach to building AI agents - an LLM calls Python tools in a structured reasoning loop, and the SDK handles the rest. When you combine them, an agent can ask "which devices are online and what can they do?" and then invoke a function on a specific device, turning natural language intent into physical action.
13
+
Natural language is becoming more than a software interface. Physical AI systems can now sense, decide, and act based on instructions from an LLM agent. But for that to work, the devices need to be reachable. They need a shared infrastructure for discovery, communication, and coordination.
14
14
15
-
This Learning Path puts both tools through their paces. It starts with a single machine, for example a laptop, where a simulated robot and an agent discover each other automatically, then extends to a two-machine setup where a Raspberry Pi joins the same device mesh over the network.
15
+
[Device Connect](https://github.com/arm/device-connect) is Arm's device-aware framework for exactly that. Once devices register through a shared mesh, agents can discover and command any of them without caring where they run. A fleet of robot arms, a network of sensors, or a mix of physical and simulated devices all become equally reachable.
16
16
17
-
## Device Connect architecture layers
17
+
[Strands Robots](https://github.com/strands-labs/robots) is a robot SDK that integrates Device Connect with the [AWS Strands Agents SDK](https://strandsagents.com/). Using Strands, an LLM can query the device mesh ("who's available?"), understand what each device can do, and dynamically invoke actions - turning natural language intent into real-world outcomes.
18
18
19
-
**Device layer**
19
+
This Learning Path starts on a single machine, where a simulated robot and an agent discover each other automatically, then optionally extends to a Raspberry Pi joining the same device mesh over the network.
20
20
21
-
A device is any process that registers itself on the mesh and exposes callable functions. In this Learning Path you'll create a simulated robot arm, namely the simulated robotic arm SO-100 from Hugging Face, from the `strands-robots` SDK. The moment this object is created, it registers on the local network under a unique device ID (for example, `so100_sim-abc23`) and begins publishing a presence heartbeat. No explicit registration call is required. Device Connect uses Zenoh as its underlying messaging transport, which handles low-level connectivity and routing automatically.
21
+
## How the pieces fit together
22
22
23
-
**Agent layer**
23
+
Two packages make this work.
24
24
25
-
Two interfaces sit at this layer. The `device-connect-agent-tools` package exposes `discover_devices()` and `invoke_device()` as plain Python functions you can call directly from a script or REPL, with no LLM involved. The `robot_mesh` tool from `robots` wraps the same capabilities as a Strands agent tool, which means an LLM can also call them during a reasoning loop. Both share the same underlying Device Connect transport, so anything you can do with one you can do with the other.
25
+
**`device-connect-sdk`** is the device-side runtime. Any process that wraps a robot in a `DeviceDriver` and starts a `DeviceRuntime` joins the mesh: it registers itself, announces what it can do, and starts publishing state events. Zenoh handles low-level connectivity; in device-to-device mode no broker or environment configuration is needed.
26
26
27
-
The diagram below shows how these layers communicate at runtime:
27
+
**`device-connect-agent-tools`** is the agent-side runtime. It exposes `discover_devices()` and `invoke_device()` as plain Python functions. The `robot_mesh` tool in Strands Robots wraps the same interface as a Strands tool, so an LLM can call it too. Both use the same underlying transport, so the same calls work whether the caller is a script or an agent.
28
28
29
-
```
30
-
┌──────────────────────────────────────┐
31
-
│ Agent layer │
32
-
│ discover_devices · invoke_device │
33
-
│ robot_mesh Strands tool │
34
-
└──────────────┬───────────────────────┘
35
-
│ Device Connect
36
-
│ (device-to-device discovery & RPC)
37
-
┌──────────────▼───────────────────────┐
38
-
│ Device layer │
39
-
│ Simulated SO-100 arm |
40
-
| - so100-abc123 │
41
-
│ heartbeat · execute · getStatus │
42
-
└──────────────────────────────────────┘
43
-
```
29
+
![Architecture diagram showing two tiers: the agent layer containing discover_devices, invoke_device, and the robot_mesh Strands tool, connected via a Device Connect Zenoh D2D arrow to the device layer containing the SO-100 robot process with its peer ID and RPC functions#center](./images/visual1.png"Device Connect architecture: agent layer and device layer connected over Zenoh D2D")
44
30
45
-
## How device discovery works
31
+
## What a device exposes
46
32
47
-
When the `SO-100 arm` instance starts, Device Connect automatically announces the device on the local network. Any process running `discover_devices()`or `robot_mesh(action='peers')`on the same network will hear the announcement and add the device to its live table of available hardware.
33
+
This Learning Path uses the SO-100 arm, an open-source robot arm. When `Robot('so100').run()` starts, it registers on the mesh and exposes three callable functions. These are what `invoke_device()`on the agent side targets — calling `invoke_device("so100-abc123", "execute", {...})`routes a request over Zenoh to the robot process and executes the function there, returning the result back to the caller:
48
34
49
-
## What the simulated robot provides
35
+
-`execute` — send a natural language instruction and a policy provider to the robot
36
+
-`getStatus` — query what the robot is currently doing
37
+
-`stop` — halt the current task, or `emergency_stop` to halt every device on the mesh at once
50
38
51
-
When you run `Robot('so100')`, the SDK downloads the MuJoCo physics model for the SO-100 arm (this happens once on first run) and starts a local simulation. The robot exposes three functions that any agent can call via RPC:
39
+
A motion policy is the component that translates a high-level instruction like "pick up the cube" into a sequence of joint movements. Different policy providers connect to different backends — from local model inference to remote policy servers. For this Learning Path, `policy_provider='mock'` is used, so `execute` accepts the task and returns immediately without running real motion. Replacing `'mock'` with a real provider like `'lerobot_local'` or `'groot'` is a one-line change once you have the connectivity working.
52
40
53
-
-`execute` - start a task with a given instruction and policy provider
54
-
-`getStatus` - query the current task state
55
-
-`stop` - halt the current task
56
-
57
-
For this Learning Path, the `policy_provider='mock'` argument is used, which means `execute` accepts the call and returns `{'status': 'accepted'}` without actually running a motion policy. This keeps the focus on the connectivity and invocation patterns rather than robotics.
58
-
59
-
Once you have the flow working end to end, replacing `'mock'` with a real policy is a one-line change.
41
+
Beyond discrete RPC calls, devices can also publish a continuous stream of sensor data over the same mesh. A camera publishes image frames, a depth sensor publishes point clouds, and an IMU reports pose updates — all as Device Connect events that any subscriber on the network receives in real time. The simulated robot in this Learning Path publishes joint state and observation updates at 10 Hz.
Copy file name to clipboardExpand all lines: content/learning-paths/embedded-and-microcontrollers/device-connect-strands/run-example.md
+32-3Lines changed: 32 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,7 +12,7 @@ This section runs Device Connect's device-to-device discovery. There are two way
12
12
13
13
### Option 1: run on a single machine
14
14
15
-
For a proof-of-concept, follow the steps using two terminal windows on your machine with the virtual environment set up.
15
+
For a conceptual implementation, follow the steps using two terminal windows on your machine with the virtual environment set up.
16
16
17
17
### Option 2: run with real hardware
18
18
@@ -45,12 +45,24 @@ python <<'PY'
45
45
import logging
46
46
logging.basicConfig(level=logging.INFO)
47
47
from strands_robots import Robot
48
-
r = Robot('so100')
48
+
r = Robot('so100', peer_id='so100-abc123')
49
49
r.run()
50
50
PY
51
51
```
52
52
53
-
When the `Robot('so100')` object is created, the SDK downloads the MuJoCo physics model for the SO-100 arm. This download happens only on the first run and takes a minute or two. After that, it starts the simulation and registers the robot on the Device Connect device mesh. The robot publishes a presence heartbeat every 0.5 seconds under a unique device ID, for example `so100-abc123`.
53
+
Two things happen when this script runs.
54
+
55
+
**`Robot('so100')`** calls the factory function, which checks for USB servo hardware and, finding none, creates a MuJoCo `Simulation` instance. On the first run it downloads the SO-100 MJCF physics model from Hugging Face — this can take up to 20 minutes depending on your connection. Subsequent runs use the local cache and start immediately.
56
+
57
+
**`r.run()`** calls `init_device_connect_sync()`, which does the following:
58
+
59
+
1. Creates a `SimulationDeviceDriver` — a Device Connect `DeviceDriver` adapter that wraps the simulation and maps its methods to structured RPCs.
60
+
2. Starts a `DeviceRuntime` with the Zenoh D2D backend. No broker or environment variables are needed; devices discover each other on the LAN via Zenoh multicast scouting.
61
+
3. Subscribes the device to its command topic: `device-connect.default.<PEER_ID>.cmd`.
5. Starts a 10Hz background loop that emits `stateUpdate` and `observationUpdate` events to any listener on the mesh.
64
+
65
+
![Sequence diagram showing the call flow when r.run() executes: the script calls Robot() and DeviceRuntime, which announces presence to the Zenoh mesh, subscribes to the command topic, registers RPC handlers, and starts emitting stateUpdate events at 10 Hz#center](./images/visual2.png"Call flow inside r.run(): device registration and event publishing over Zenoh")
54
66
55
67
You should see INFO-level log output similar to:
56
68
@@ -64,6 +76,17 @@ device_connect_sdk.device.so100-abc123 - INFO - Subscribed to commands on device
64
76
65
77
Leave this process running. The simulated robot is only discoverable as long as this process is alive.
66
78
79
+
## Open a second terminal
80
+
81
+
Leave terminal 1 running with the robot process. Open a new terminal window, navigate to the repository, and activate the virtual environment:
82
+
83
+
```bash
84
+
cd~/strands-device-connect/robots
85
+
source .venv/bin/activate
86
+
```
87
+
88
+
Run all remaining commands in this section from this second terminal.
89
+
67
90
## Control the robot using the robot_mesh Strands tool
68
91
69
92
The `robot_mesh` tool wraps the same discovery and invocation primitives as a Strands agent tool. You can call it directly from a Python script or attach it to an LLM agent; the API is identical either way.
When this runs, `robot_mesh` calls `_ensure_connected()`, which sets `MESSAGING_BACKEND=zenoh` (if the environment variable is not already set) and opens the agent-side Zenoh connection via `device_connect_agent_tools`. It then calls `conn.list_devices()`, which queries the Zenoh network for all registered Device Connect devices. Each device registers its `device_id`, `device_type`, availability from its `DeviceStatus`, and the list of RPC functions it exposes. The tool formats this into the human-readable summary below.
106
+
82
107
The output is similar to:
83
108
84
109
```output
@@ -105,6 +130,8 @@ print(robot_mesh(
105
130
PY
106
131
```
107
132
133
+
Under the hood, `robot_mesh` calls `conn.invoke(target, "execute", params)`, which serializes the arguments and routes them over Zenoh to the device's command topic: `device-connect.default.<PEER_ID>.cmd`. The `SimulationDeviceDriver.execute()` RPC handler on the robot side receives the call, resolves the robot name inside the simulation world, and calls `sim.start_policy(instruction=..., policy_provider='mock', ...)`. With the mock policy provider, the handler returns immediately with a success result without running real motion — the call is a connectivity and RPC round-trip test. The `stateUpdate` events you'll see in terminal 1 are published by the separate 10Hz background loop that was started by `r.run()`.
This doesn't send a single broadcast message. Instead, `robot_mesh` first calls `conn.list_devices()` to enumerate every device currently on the mesh, then calls `conn.invoke(device_id, "stop", timeout=3.0)` on each one in sequence. On the device side, `SimulationDeviceDriver.stop()` sets `policy_running = False` for every robot in the simulation world and returns immediately. Failures per device are swallowed so that a single unresponsive device doesn't block the rest from stopping.
0 commit comments