You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| Backend — Windows installer | Any Windows machine | Windows 11 |
14
+
| Backend — Franka Emika | Dedicated Linux PC| Ubuntu 20.04 (RT kernel) |
15
15
16
16
The Windows backend installer supports all adapters that do not require a Linux environment — currently Mock and Universal Robots. The Franka adapter requires a dedicated Linux setup described in Part 3.
17
17
@@ -37,21 +37,21 @@ Open PowerShell as Administrator (right-click → *Run as Administrator*) and ru
The script installs `ffmpeg` and `uv` via winget if not present, downloads the frontend to `C:\Program Files\Speech-to-Cobot`, creates a Desktop shortcut, and pre-downloads the Whisper base model (~140 MB).
40
+
The script installs `ffmpeg` and `uv` via winget if not present, downloads the frontend to `C:\Program Files\Speech-to-Cobot`, creates a Desktop shortcut, and pre-downloads the Whisper small model (~466 MB).
41
41
42
42
### Launch
43
43
44
44
Double-click **Speech-to-Cobot** on the Desktop.
45
45
46
46
### Uninstall
47
47
48
-
Run the following in Administrator PowerShell, or run `uninstall_frontend.ps1` directly from the install directory if internet is not available:
48
+
Run the following in Administrator PowerShell, or run `uninstall_frontend.ps1` directly from the installation directory if internet is not available:
Removes the install directory, Desktop shortcut, and Whisper model cache. Optionally removes `ffmpeg` and `uv` — you will be prompted for each. LM Studio must be removed manually via **Settings → Apps**.
54
+
Removes the installation directory, Desktop shortcut, and Whisper model cache. Optionally removes `ffmpeg` and `uv` — you will be prompted for each. LM Studio must be removed manually via **Settings → Apps**.
55
55
56
56
---
57
57
@@ -78,13 +78,13 @@ Double-click **Speech-to-Cobot Backend** on the Desktop. A terminal window opens
78
78
Before running with a real UR robot, verify the following in `Backend\config_backend.py`:
@@ -113,27 +113,69 @@ Refer to the official documentation for each:
113
113
114
114
### Backend installation
115
115
116
-
Once the ROS stack is confirmed working, install the backend Python dependencies into the ROS Python environment:
116
+
Once the ROS stack is confirmed working, install the backend Python dependencies into the ROS Python environment.
117
117
118
-
```bash
119
-
pip install pyzmq numpy scipy
120
-
```
118
+
### Launch
121
119
122
-
Download or clone the repository and navigate to the repo root:
120
+
Start the backend server (from the right directory):
123
121
124
122
```bash
125
-
cd~/path/to/repo
123
+
python -m Backend.main
126
124
```
127
125
128
-
### Launch
126
+
The backend connects to the robot at `192.168.1.100` by default. Verify the robot IP in `Backend/robot_controllers/franka_controller.py` (`ROBOT_IP`).
129
127
130
-
Start the backend server:
128
+
---
131
129
132
-
```bash
133
-
python -m Backend.main
134
-
```
130
+
## Part 4 — Robot Preparation (Per-Session)
131
+
132
+
These steps are required before each evaluation or usage session.
133
+
They cover physical robot preparation and are distinct from the software
134
+
installation steps in Parts 1–3.
135
135
136
-
The backend connects to the robot at `192.168.1.100` by default. Verify the robot IP in `Backend/robot_controllers/franka_controller.py` (`ROBOT_IP`) and the backend machine IP in `Backend/config_backend.py` (`PC_IP`).
136
+
---
137
+
138
+
### UR10e
139
+
140
+
1. Power on the UR10e via the Start Button on the Teach Pendant
141
+
2. After Startup, On the UR pendant, set the robot mode to **Remote Mode** (top right corner)
142
+
3.*(First-time only)* Verify the robot's IP address:
143
+
`Menu → About → IP address`. Robot IPs are typically pre-configured
144
+
to a consistent address (and can be modified) — confirm it matches `DEFAULT_ROBOT_IP` in
145
+
`Backend/robot_controllers/ur_controller.py`.
146
+
4. Plug the robot Ethernet cable into the robot and the backend machine.
147
+
5.*(First-time only)* Configure the laptop Ethernet adapter with a static
148
+
IP on the same subnet as the robot
149
+
6. Launch the frontend and the backend server on the machine. When all IPs are correct and the network connection is working, the frontend's **Ping Backend** button will show a green indicator and the system is ready.
150
+
151
+
> **Note:** Always set Remote Mode before a session. If the robot is left in Local Mode, motion commands will silently fail — the backend sends the command, the robot acknowledges it, but does not move. Gripper and save-position commands continue to work normally in Local Mode, which can make the root cause hard to identify. Always confirm Remote Mode is active before proceeding.
152
+
153
+
---
154
+
155
+
### Franka Emika Panda
156
+
157
+
1. Power on the Franka controller using the controller switch
158
+
2. Start the Linux PC (running the real-time kernel)
159
+
3. Connect the robot controller to the Linux PC via Ethernet
160
+
4.*(First-time only)* Configure the Linux PC Ethernet adapter with a static IP
161
+
on the same subnet as the robot
162
+
5. Connect the frontend machine to the Linux PC via Ethernet
163
+
6.*(First-time only)* Configure both Ethernet adapters (Linux PC and operator machine)
164
+
with static IPs on the same subnet. Update `BACKEND_IPS` in `Frontend/config_frontend.py`
165
+
with the resulting addresses
166
+
7. Open a browser and navigate to `https://<franka-robot-ip>`. Accept the certificate
167
+
warning — Desk uses a self-signed certificate
168
+
8. Unlock the joints using the Desk interface
169
+
9. Engage the latching button (connected to X4 on the Arm base) once — the robot
170
+
indicator changes from purple *(after startup only)* to white (hand-guide mode)
171
+
10. Release the latching button to exit hand-guide mode — the indicator turns blue (Ready)
172
+
11. In Desk, click **Activate FCI** — the indicator stays blue, FCI window appears
173
+
12. Start the backend on the Linux PC
174
+
13. Start the frontend on the operator machine. When all IPs are correct and the backend
175
+
is reachable, the **Ping Backend** button shows a green indicator and the system is ready
176
+
177
+
> **Note:** The Franka must be in Ready (blue) state for motion commands to execute.
178
+
> If left in hand-guide mode (white), the robot accepts commands but will not execute them.
137
179
138
180
---
139
181
@@ -143,27 +185,32 @@ Both components have a single configuration file that covers all runtime paramet
143
185
144
186
**Frontend — `Frontend/config_frontend.py`**
145
187
146
-
| Parameter | Description |
147
-
|---|---|
148
-
149
-
|`ASR_MODEL_SIZE`| Whisper model size: `tiny`, `base`, `small`, `medium`, `large`|
150
-
|`LLM_API_BASE`| LM Studio server URL (default: `http://localhost:1234/v1`) |
151
-
|`LLM_MODEL_NAME`| Model name as shown in LM Studio |
152
-
|`BACKEND_IPS`| ZeroMQ addresses for each robot backend |
|`ZMQ_TIMEOUT_MS`| ZeroMQ socket timeout in milliseconds (default: 1000) |
210
+
|`AVAILABLE_ROBOTS`| List of adapters to load: `mock`, `ur`, `franka`|
211
+
|`PC_IP`| IP of the backend machine as seen by the UR robot (required for UR motion callback only)|
212
+
|`LOGGING_LEVEL`| Console log verbosity |
213
+
|`LOGGING_LEVEL_FILE`| File log verbosity |
167
214
168
215
The frontend connects to the backend via ZeroMQ on port `5555`. How you configure the network depends on your physical setup.
169
216
@@ -175,10 +222,10 @@ Once the network is configured, update `Frontend/config_frontend.py` with the co
175
222
176
223
```python
177
224
BACKEND_IPS= {
178
-
"franka": "tcp://<franka-pc-ip>:5555",
179
-
"ur": "tcp://<ur-backend-ip>:5555",
225
+
"franka": "tcp://192.168.2.20:5555",# Linux PC running Franka backend
226
+
"ur": "tcp://localhost:5555",# UR backend runs on operator machine
180
227
"mock": "tcp://localhost:5555",
181
228
}
182
229
```
183
230
184
-
Use the **Ping Backend** button in the frontend GUI to verify the connection before issuing commands. A green indicator confirms the backend is reachable. If it stays red, check that the backend is running, the IPs are correct, and port `5555` is not blocked by the firewall.
231
+
Use the **Ping Backend** button in the frontend GUI to verify the connection before issuing commands. A green indicator confirms the backend is reachable. If it stays red, check that the backend is running, the IPs are correct, and port `5555` is not blocked by the firewall.
0 commit comments