Skip to content

Commit 95dffe4

Browse files
pareenavermapareenaverma
authored
Merge pull request #2983 from annietllnd/gtc-strands
New LP with AI agents using Strands and Device Connect
2 parents 0c1f50d + 4ff17dd commit 95dffe4

6 files changed

Lines changed: 604 additions & 0 deletions

File tree

‎content/learning-paths/embedded-and-microcontrollers/device-connect-strands/_index.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
title: Connect AI agents to edge devices using Device Connect and Strands
3+
4+
minutes_to_complete: 30
5+
6+
who_is_this_for: This Learning Path is for software developers who want to connect AI agents to physical or simulated edge devices using Device Connect — Arm's platform for structured device access — and Strands, the open-source agent SDK from AWS.
7+
8+
learning_objectives:
9+
- Understand how Device Connect and Strands work together to give AI agents structured access to Arm-based edge devices.
10+
- Set up a Python environment with the Device Connect SDK and agent tools installed from source.
11+
- Start a simulated robot that registers itself on the local network and is discovered automatically by an agent.
12+
- Discover and invoke the robot using the Device Connect agent tools and the robot_mesh Strands tool.
13+
14+
prerequisites:
15+
- An development machine with Python 3.12 installed.
16+
- Git installed.
17+
- Basic familiarity with Python virtual environments and command-line tools.
18+
- (Optional) A Raspberry Pi for testing a full device-to-device (D2D) setup.
19+
20+
author:
21+
- Annie Tallund
22+
- Kavya Sri Chennoju
23+
24+
25+
26+
### Tags
27+
skilllevels: Introductory
28+
subjects: ML
29+
armips:
30+
- Cortex-A
31+
- Neoverse
32+
operatingsystems:
33+
- Linux
34+
- macOS
35+
tools_software_languages:
36+
- Python
37+
- Docker
38+
- strands-agents
39+
40+
further_reading:
41+
- resource:
42+
title: Strands Agents SDK documentation
43+
link: https://strandsagents.com/
44+
type: website
45+
- resource:
46+
title: Strands robots repository
47+
link: https://github.com/strands-labs/robots/tree/dev
48+
type: website
49+
- resource:
50+
title: Device Connect integration guide
51+
link: https://github.com/atsyplikhin/robots/blob/feat/device-connect-integration-draft/strands_robots/device_connect/GUIDE.md
52+
type: website
53+
54+
### FIXED, DO NOT MODIFY
55+
# ================================================================================
56+
weight: 1 # _index.md always has weight of 1 to order correctly
57+
layout: "learningpathall" # All files under learning paths have this same wrapper
58+
learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
59+
---

‎content/learning-paths/embedded-and-microcontrollers/device-connect-strands/_next-steps.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
# ================================================================================
3+
# FIXED, DO NOT MODIFY THIS FILE
4+
# ================================================================================
5+
weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation.
6+
title: "Next Steps" # Always the same, html page title.
7+
layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing.
8+
---

‎content/learning-paths/embedded-and-microcontrollers/device-connect-strands/background.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: Background and architecture
3+
weight: 2
4+
5+
# FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Why connect AI agents to edge devices?
10+
11+
Arm processors are at the heart of a remarkable range of systems — from Cortex-M microcontrollers in industrial sensors to Neoverse servers running in the cloud. That breadth of hardware is one of Arm's greatest strengths, but it raises a practical question for AI developers: how do you give an agent structured, safe access to devices that are physically distributed and built on different software stacks?
12+
13+
Device Connect is Arm's answer to that question. It is a platform layer that handles device registration, discovery, and remote procedure calls across a network of devices, with no bespoke networking code required. Strands is an open-source agent SDK from AWS that takes a model-driven approach to building AI agents — an LLM calls Python tools in a structured reasoning loop, and the SDK handles the rest. When you combine them, an agent can ask "which devices are online and what can they do?" and then invoke a function on a specific device, turning natural language intent into physical action.
14+
15+
This Learning Path puts both tools through their paces. It starts with a single machine, for example a laptop, where a simulated robot and an agent discover each other automatically, then extends to a two-machine setup where a Raspberry Pi joins the same device mesh over the network.
16+
17+
## The two layers
18+
19+
**Device layer**
20+
21+
A device is any process that registers itself on the mesh and exposes callable functions. In this Learning Path you will create a simulated robot arm, namely the simulated robotic arm SO-100 from Hugging Face, from the `strands-robots` SDK. The moment this object is created, it registers on the local network under a unique device ID (for example, `so100_sim-abc23`) and begins publishing a presence heartbeat. No explicit registration call is required. Device Connect uses Zenoh as its underlying messaging transport, which handles low-level connectivity and routing automatically.
22+
23+
**Agent layer**
24+
25+
Two interfaces sit at this layer. The `device-connect-agent-tools` package exposes `discover_devices()` and `invoke_device()` as plain Python functions you can call directly from a script or REPL, with no LLM involved. The `robot_mesh` tool from `robots` wraps the same capabilities as a Strands agent tool, which means an LLM can also call them during a reasoning loop. Both share the same underlying Device Connect transport, so anything you can do with one you can do with the other.
26+
27+
The diagram below shows how these layers communicate at runtime:
28+
29+
```
30+
┌──────────────────────────────────────┐
31+
│ Agent layer │
32+
│ discover_devices · invoke_device │
33+
│ robot_mesh Strands tool │
34+
└──────────────┬───────────────────────┘
35+
│ Device Connect
36+
│ (device-to-device discovery & RPC)
37+
┌──────────────▼───────────────────────┐
38+
│ Device layer │
39+
│ Simulated SO-100 arm |
40+
| — so100-abc123 │
41+
│ heartbeat · execute · getStatus │
42+
└──────────────────────────────────────┘
43+
```
44+
45+
## How device discovery works
46+
47+
When the `SO-100 arm` instance starts, Device Connect automatically announces the device on the local network. Any process running `discover_devices()` or `robot_mesh(action='peers')` on the same network will hear the announcement and add the device to its live table of available hardware.
48+
49+
## What the simulated robot provides
50+
51+
When you run `Robot('so100')`, the SDK downloads the MuJoCo physics model for the SO-100 arm (this happens once on first run) and starts a local simulation. The robot exposes three functions that any agent can call via RPC:
52+
53+
- `execute` — start a task with a given instruction and policy provider
54+
- `getStatus` — query the current task state
55+
- `stop` — halt the current task
56+
57+
For this Learning Path, the `policy_provider='mock'` argument is used, which means `execute` accepts the call and returns `{'status': 'accepted'}` without actually running a motion policy. This keeps the focus on the connectivity and invocation patterns rather than robotics.
58+
59+
Once you have the flow working end to end, replacing `'mock'` with a real policy is a one-line change.
60+
61+
## What you'll accomplish in this Learning Path
62+
63+
By working through the remaining sections you will:
64+
65+
- Clone the sample repository and install the Device Connect SDK, agent tools, and Strands robot runtime from source into a single virtual environment.
66+
- Start a simulated robot that registers itself on the local device mesh.
67+
- Discover and invoke the robot using `device-connect-agent-tools` directly.
68+
- Discover and command the robot through the `robot_mesh` Strands tool, including an emergency stop.
69+
- Optionally extend the setup to a Raspberry Pi connected over the network, discovering and commanding it from your laptop through the Device Connect infrastructure.
70+
71+
The next section covers the environment setup.
72+
73+
TODO: replace robots and feat/device-connect-integration-draft. Update furhter reading links. Clarify device support.

‎content/learning-paths/embedded-and-microcontrollers/device-connect-strands/run-example.md‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
---
2+
title: Run the example end to end
3+
weight: 4
4+
5+
# FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Overview
10+
11+
This section runs Device Connect's device-to-device discovery. There are two ways to walk through this setup. Optionally, you can connect an external device.
12+
13+
### Option 1: Run on a single machine
14+
15+
For a proof-of-conect, follow the steps using two terminal windows on your machine with the virtual environment set up.
16+
17+
### Option 2: Run with real hardware
18+
19+
If you have access to an external device, you can use it with this setup as well. You will need:
20+
21+
- Your machine with the virtual environment set up. This machine will be referred to as the host.
22+
- A Raspberry Pi (or any similar device) connected to the same network as your host machine. This machine is your target.
23+
- An SSH connection or keyboard and monitor attached to the target.
24+
25+
You will need two terminal windows open at the same time: one to keep the simulated robot running, and one to invoke it from the agent side. Both terminals need the virtual environment activated.
26+
27+
| Machine | Terminal | Purpose |
28+
|---------|----------|---------|
29+
| Host or Target | 1 | Simulated robot — keep running throughout |
30+
| Host | 2 | Agent tool invocations |
31+
32+
Make sure you are in the repository directory and that your virtual environment is activated:
33+
34+
```bash
35+
cd ~/strands-device-connect/robots
36+
source .venv/bin/activate
37+
```
38+
39+
## Start the simulated robot
40+
41+
In terminal 1, run the following command to create and start the simulated SO-100 robot arm:
42+
43+
```python
44+
python <<'PY'
45+
import logging
46+
logging.basicConfig(level=logging.INFO)
47+
from strands_robots import Robot
48+
r = Robot('so100')
49+
r.run()
50+
PY
51+
```
52+
53+
When the `Robot('so100')` object is created, the SDK downloads the MuJoCo physics model for the SO-100 arm (this download happens only on the first run and takes a minute or two), then starts the simulation and registers the robot on the Device Connect device mesh. The robot publishes a presence heartbeat every 0.5 seconds under a unique device ID, for example `so100-abc123`.
54+
55+
You should see INFO-level log output similar to:
56+
57+
```output
58+
device_connect_sdk.device.so100-abc123 - INFO - Using ZENOH messaging backend
59+
device_connect_sdk.device.so100-abc123 - INFO - Connected to ZENOH broker: []
60+
device_connect_sdk.device.so100-abc123 - INFO - Driver connected: strands_sim
61+
device_connect_sdk.device.so100-abc123 - INFO - Subscribed to commands on device-connect.default.so100-abc123.cmd
62+
🤖 so100-abc123 is online. Ctrl+C to stop.
63+
```
64+
65+
Leave this process running. The simulated robot is only discoverable as long as this process is alive.
66+
67+
## Control the robot using the robot_mesh Strands tool
68+
69+
The `robot_mesh` tool wraps the same discovery and invocation primitives as a Strands agent tool. You can call it directly from a Python script or attach it to an LLM agent; the API is identical either way.
70+
71+
### List peers
72+
73+
Start by confirming which robots are currently visible on the mesh:
74+
75+
```python
76+
python <<'PY'
77+
from strands_robots.tools.robot_mesh import robot_mesh
78+
print(robot_mesh(action='peers'))
79+
PY
80+
```
81+
82+
The output is similar to:
83+
84+
```output
85+
Discovered 1 device(s):
86+
[robot] so100-abc123 — idle
87+
Functions: execute, getFeatures, getStatus, reset, step, stop
88+
```
89+
90+
The peer ID (for example `so100-abc123`) is assigned at startup and changes each run. Note the actual ID shown in your terminal - you will need it in the next step.
91+
92+
### Execute an instruction
93+
94+
Send a task to one of the discovered robots. Replace `so100-abc123` with the peer ID shown in your `peers` output:
95+
96+
```python
97+
python <<'PY'
98+
from strands_robots.tools.robot_mesh import robot_mesh
99+
print(robot_mesh(
100+
action='tell',
101+
target='so100-abc123',
102+
instruction='pick up the cube',
103+
policy_provider='mock',
104+
))
105+
PY
106+
```
107+
108+
You will see the following output:
109+
110+
```output
111+
-> so100-abc123: pick up the cube
112+
{"status": "success", "content": [...]}
113+
```
114+
115+
{{% notice Robot output in terminal 1 %}}
116+
The robot also logs event updates as it processes the task. If you switch back to terminal 1, it logs the execution of the task, similar to:
117+
118+
```output
119+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
120+
*** EVENT so100-abc123::stateUpdate [111aaabb]
121+
payload: sim_time=4.34, step_count=2070, running_policies={'so100': {'steps': 814, 'instruction': 'pick up the cube'}}
122+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
123+
```
124+
{{% /notice %}}
125+
126+
127+
### Emergency stop
128+
129+
The emergency stop broadcasts a halt command to every device on the mesh simultaneously. This is useful when you want to stop all robots without knowing their individual peer IDs:
130+
131+
```python
132+
python <<'PY'
133+
from strands_robots.tools.robot_mesh import robot_mesh
134+
print(robot_mesh(action='emergency_stop'))
135+
PY
136+
```
137+
138+
The output is similar to:
139+
140+
```output
141+
E-STOP: 1/1 devices stopped
142+
```
143+
144+
## Optional: Discover and invoke the robot using the agent tools
145+
146+
The `device-connect-agent-tools` package gives you direct programmatic access to the mesh, without involving an LLM. This is useful for testing, scripting, or validating the stack before wiring it up to an agent. Open terminal 2, activate the virtual environment, then run:
147+
148+
```python
149+
python <<'PY'
150+
from device_connect_agent_tools import connect, discover_devices, invoke_device
151+
152+
connect()
153+
154+
devices = discover_devices(device_type='')
155+
print(f'Found {len(devices)} robot(s):')
156+
for d in devices:
157+
print(f' {d["device_id"]}')
158+
159+
if devices:
160+
result = invoke_device(
161+
devices[0]['device_id'],
162+
'execute',
163+
{'instruction': 'pick up the cube', 'policy_provider': 'mock'},
164+
)
165+
print(f'Execute result: {result}')
166+
167+
status = invoke_device(devices[0]['device_id'], 'getStatus')
168+
print(f'Status: {status}')
169+
PY
170+
```
171+
172+
{{% notice About the snippet %}}
173+
When an agent calls `execute(instruction="pick up the cup", policy_provider="groot")`, Device Connect handles the RPC delivery, and the policy handles the actual arm movement.
174+
175+
`discover_devices(device_type='')` returns all devices on the mesh regardless of type. If you pass `device_type='strands_robot'` you can filter to only `Robot()` instances. `invoke_device` sends an RPC to the named device; here `policy_provider='mock'` tells the robot to accept the task without executing real motion, which is appropriate for this connectivity test.
176+
{{% /notice %}}
177+
178+
179+
The output is similar to:
180+
181+
```output
182+
Found 1 robot(s):
183+
so100-abc123 — idle
184+
Execute result: {'success': True, 'result': {'status': 'success', 'content': [...]}}
185+
Status: {'success': True, 'result': {...}} # full sim state dict
186+
```
187+
188+
## What you've accomplished and what's next
189+
190+
In this section you:
191+
192+
- Started a simulated SO-100 robot that registered itself on the Device Connect device mesh.
193+
- Used `device-connect-agent-tools` to discover the robot and invoke an RPC call against it.
194+
- Used the `robot_mesh` Strands tool to list peers, send an instruction, and trigger an emergency stop.
195+
196+
This showcases the ease of setting up a mesh on a local network. In the next section, you can extend the configuration to a Docker-based approach, opening up a new category of possibilites with agent integration.

0 commit comments

Comments
 (0)