Skip to content

Commit 1e333cd

Browse files
committed
feat(oob): Adds ADB automation handling
1 parent 7e3bdae commit 1e333cd

11 files changed

Lines changed: 843 additions & 46 deletions

File tree

docs/source/references/oob_teleop_control.rst

Lines changed: 58 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,17 +24,34 @@ on first run):
2424
2525
python -m isaacteleop.cloudxr --accept-eula --setup-oob
2626
27+
This will:
28+
29+
1. Verify a USB-connected headset is available via ``adb devices``
30+
2. Start the WSS proxy with the OOB control hub
31+
3. Open the teleop page on the headset via ``adb shell am start``
32+
2733
You should see output confirming the hub is running:
2834

2935
.. code-block:: text
3036
3137
CloudXR WSS proxy: running, log file: /home/<user>/.cloudxr/logs/wss.2026-04-13T202133Z.log
32-
oob: enabled (hub running in WSS proxy)
38+
oob: enabled (hub + USB adb automation — see OOB TELEOP block)
39+
40+
.. note::
3341

34-
**Step 2 — Open the web client on the headset**
42+
The headset must be:
3543

36-
On the XR headset browser, navigate to the client URL with **all three**
37-
required query parameters — ``oobEnable``, ``serverIP``, and ``port``:
44+
- **Connected via USB cable** for adb commands (opening the teleop URL)
45+
- **Connected to WiFi** on the same network as the streaming host (for web
46+
page access and CloudXR streaming)
47+
48+
No ``adb reverse`` or USB tethering is used.
49+
50+
**Step 2 — (Manual fallback) Open the web client on the headset**
51+
52+
If the adb automation fails (e.g. headset not paired), you can manually open
53+
the client URL on the headset browser with **all three** required query
54+
parameters — ``oobEnable``, ``serverIP``, and ``port``:
3855

3956
.. code-block:: text
4057
@@ -106,6 +123,27 @@ state endpoint from a PC to collect them:
106123
107124
The ``metricsByCadence`` field on each headset entry will now contain live streaming metrics.
108125

126+
ADB automation
127+
--------------
128+
129+
The ``--setup-oob`` flag automates headset setup via USB ``adb``:
130+
131+
1. **adb devices** — verifies exactly one device is connected
132+
2. **am start** — opens the teleop bookmark URL in the headset browser with
133+
the correct ``oobEnable=1``, ``serverIP``, and ``port`` parameters
134+
135+
No ``adb reverse`` ports or USB tethering is used. The headset reaches the
136+
streaming host directly over WiFi.
137+
138+
Prerequisites:
139+
140+
- ``adb`` must be on ``PATH`` (Android SDK Platform Tools)
141+
- The headset must be connected via USB with USB debugging enabled
142+
- The headset must be on the same WiFi network as the streaming host
143+
144+
If adb automation fails, the hub still starts and you can open the URL
145+
on the headset manually.
146+
109147
Architecture
110148
------------
111149

@@ -123,6 +161,7 @@ Architecture
123161
* - **Streaming host**
124162
- ``python -m isaacteleop.cloudxr --setup-oob``
125163
- Runs CloudXR runtime + WSS proxy + OOB hub on a single TLS port.
164+
Opens the teleop page on the headset via USB adb.
126165
* - **Operator / scripts**
127166
- ``curl``, browser, or custom tooling
128167
- Reads state via HTTP, optionally pushes config via HTTP.
@@ -264,7 +303,7 @@ The client builds ``wss://{serverIP}:{port}/oob/v1/ws`` and:
264303

265304
1. Registers as role ``"headset"``
266305
2. Reports ``clientMetrics`` periodically (default every 500 ms)
267-
3. Receives ``config`` pushes (phase 2)
306+
3. Receives ``config`` pushes from operator
268307

269308
URL query parameter overrides
270309
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -293,3 +332,17 @@ Environment variables
293332
- Optional auth token for hub access
294333
* - ``TELEOP_STREAM_SERVER_IP``
295334
- Override the auto-detected LAN IP in hub initial config
335+
* - ``TELEOP_PROXY_HOST``
336+
- Override the LAN IP used for headset bookmark URLs
337+
* - ``TELEOP_WEB_CLIENT_BASE``
338+
- Override the WebXR client origin URL
339+
* - ``TELEOP_STREAM_PORT``
340+
- Override the signaling port (default same as proxy port)
341+
* - ``TELEOP_CLIENT_CODEC``
342+
- Default video codec for headset bookmarks
343+
* - ``TELEOP_CLIENT_PANEL_HIDDEN_AT_START``
344+
- Hide control panel on load (``true`` / ``false``)
345+
* - ``TELEOP_CLIENT_PER_EYE_WIDTH``
346+
- Per-eye render width override
347+
* - ``TELEOP_CLIENT_PER_EYE_HEIGHT``
348+
- Per-eye render height override

src/core/cloudxr/python/CMakeLists.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,8 @@ add_custom_target(cloudxr_python ALL
111111
"${CMAKE_CURRENT_SOURCE_DIR}/launcher.py"
112112
"${CMAKE_CURRENT_SOURCE_DIR}/runtime.py"
113113
"${CMAKE_CURRENT_SOURCE_DIR}/wss.py"
114+
"${CMAKE_CURRENT_SOURCE_DIR}/oob_teleop_adb.py"
115+
"${CMAKE_CURRENT_SOURCE_DIR}/oob_teleop_env.py"
114116
"${CMAKE_CURRENT_SOURCE_DIR}/oob_teleop_hub.py"
115117
"${CLOUDXR_PYTHON_DIR}/"
116118
COMMAND ${CMAKE_COMMAND} -E rm -rf "${CLOUDXR_PYTHON_DIR}/__pycache__"

src/core/cloudxr/python/__main__.py

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,22 @@
66
import argparse
77
import os
88
import signal
9+
import sys
910
import time
1011

1112
from isaacteleop import __version__ as isaacteleop_version
1213
from isaacteleop.cloudxr.env_config import get_env_config
1314
from isaacteleop.cloudxr.launcher import CloudXRLauncher
1415
from isaacteleop.cloudxr.runtime import latest_runtime_log, runtime_version
16+
from isaacteleop.cloudxr.oob_teleop_adb import (
17+
OobAdbError,
18+
assert_exactly_one_adb_device,
19+
require_adb_on_path,
20+
)
21+
from isaacteleop.cloudxr.oob_teleop_env import (
22+
print_oob_hub_startup_banner,
23+
resolve_lan_host_for_oob,
24+
)
1525

1626

1727
def _parse_args() -> argparse.Namespace:
@@ -41,8 +51,9 @@ def _parse_args() -> argparse.Namespace:
4151
action="store_true",
4252
default=False,
4353
help=(
44-
"Enable OOB teleop control hub in the WSS proxy. "
45-
"Exposes WebSocket and HTTP API for headset metrics and remote config."
54+
"Enable OOB teleop control hub and open the teleop page on the headset via USB adb. "
55+
"The headset must be connected via USB cable (for adb) and on WiFi (for streaming). "
56+
'See docs: "Out-of-band teleop control".'
4657
),
4758
)
4859
return parser.parse_args()
@@ -52,6 +63,11 @@ def main() -> None:
5263
"""Launch the CloudXR runtime and WSS proxy, then block until interrupted."""
5364
args = _parse_args()
5465

66+
if args.setup_oob:
67+
require_adb_on_path()
68+
resolve_lan_host_for_oob()
69+
assert_exactly_one_adb_device()
70+
5571
with CloudXRLauncher(
5672
install_dir=args.cloudxr_install_dir,
5773
env_config=args.cloudxr_env_config,
@@ -75,8 +91,9 @@ def main() -> None:
7591
)
7692
if args.setup_oob:
7793
print(
78-
" oob: \033[32menabled\033[0m (hub running in WSS proxy)"
94+
" oob: \033[32menabled\033[0m (hub + USB adb automation — see OOB TELEOP block)"
7995
)
96+
print_oob_hub_startup_banner(lan_host=resolve_lan_host_for_oob())
8097
print(
8198
f"Activate CloudXR environment in another terminal: \033[1;32msource {env_cfg.env_filepath()}\033[0m"
8299
)
@@ -99,4 +116,10 @@ def on_signal(sig, frame):
99116

100117

101118
if __name__ == "__main__":
102-
main()
119+
try:
120+
main()
121+
except OobAdbError as e:
122+
print("", file=sys.stderr)
123+
print(str(e), file=sys.stderr)
124+
print("", file=sys.stderr)
125+
raise SystemExit(1) from None

src/core/cloudxr/python/launcher.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,8 +90,8 @@ def __init__(
9090
accept_eula: Accept the NVIDIA CloudXR EULA
9191
non-interactively. When ``False`` and the EULA marker
9292
does not exist, the user is prompted on stdin.
93-
setup_oob: Enable the OOB teleop control hub in the WSS
94-
proxy.
93+
setup_oob: Enable the OOB teleop control hub and USB
94+
adb automation in the WSS proxy.
9595
9696
Raises:
9797
RuntimeError: If the EULA is not accepted or the runtime
Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
"""ADB automation for OOB teleop (``--setup-oob``): open the headset bookmark URL via USB adb.
5+
6+
The headset is connected via USB cable for adb commands only. Streaming and
7+
web-page access use WiFi — no ``adb reverse`` or USB tethering.
8+
"""
9+
10+
from __future__ import annotations
11+
12+
import logging
13+
import os
14+
import shlex
15+
import shutil
16+
import subprocess
17+
18+
from .oob_teleop_env import (
19+
DEFAULT_WEB_CLIENT_ORIGIN,
20+
build_headset_bookmark_url,
21+
client_ui_fields_from_env,
22+
resolve_lan_host_for_oob,
23+
web_client_base_override_from_env,
24+
)
25+
26+
log = logging.getLogger("oob-teleop-adb")
27+
28+
29+
class OobAdbError(Exception):
30+
"""``--setup-oob`` adb step failed; ``str(exception)`` is formatted for users (print without traceback)."""
31+
32+
33+
def _adb_output_text(proc: subprocess.CompletedProcess[str]) -> str:
34+
return (proc.stderr or proc.stdout or "").strip()
35+
36+
37+
def adb_automation_failure_hint(diagnostic: str) -> str:
38+
"""Human-readable next steps for common ``adb`` failures."""
39+
d = diagnostic.lower()
40+
if "unauthorized" in d:
41+
return (
42+
"Device is unauthorized: unlock the headset, confirm the USB debugging (RSA) prompt, "
43+
"and run `adb devices` until the device shows `device` not `unauthorized`. "
44+
"If this persists, try `adb kill-server` and reconnect the cable."
45+
)
46+
if (
47+
"no devices/emulators" in d
48+
or "no devices found" in d
49+
or "device not found" in d
50+
):
51+
return (
52+
"No adb device: plug in the USB cable, enable USB debugging on the headset, "
53+
"and check `adb devices`."
54+
)
55+
if "more than one device" in d:
56+
return "Multiple adb devices: unplug extras so only one headset shows in `adb devices`."
57+
if "offline" in d:
58+
return "Device offline: reconnect the USB cable and confirm USB debugging on the headset."
59+
return ""
60+
61+
62+
def oob_adb_automation_message(rc: int, detail: str, hint: str) -> str:
63+
d = detail.strip() if detail else "(no output from adb)"
64+
lines = [
65+
f"OOB adb automation failed (adb exit code {rc}).",
66+
"",
67+
d,
68+
]
69+
if hint.strip():
70+
lines.extend(["", hint])
71+
lines.extend(
72+
[
73+
"",
74+
"To run the WSS proxy and OOB hub without adb, omit --setup-oob and open the teleop URL on the headset yourself.",
75+
]
76+
)
77+
return "\n".join(lines)
78+
79+
80+
def require_adb_on_path() -> None:
81+
"""Raise :exc:`OobAdbError` if ``adb`` is missing."""
82+
if shutil.which("adb"):
83+
return
84+
raise OobAdbError(
85+
"Cannot use --setup-oob: `adb` was not found on PATH.\n\n"
86+
"Install Android Platform Tools and ensure `adb` is available, or omit --setup-oob and open "
87+
"the teleop bookmark URL on the headset yourself."
88+
)
89+
90+
91+
def assert_exactly_one_adb_device() -> None:
92+
"""Fail unless exactly one device is in ``device`` state."""
93+
try:
94+
proc = subprocess.run(
95+
["adb", "devices"],
96+
capture_output=True,
97+
text=True,
98+
timeout=30,
99+
check=False,
100+
)
101+
except FileNotFoundError as e:
102+
raise OobAdbError(
103+
"Cannot use --setup-oob: `adb` was not found on PATH.\n\n"
104+
"Install Android Platform Tools and ensure `adb` is available, or omit --setup-oob."
105+
) from e
106+
if proc.returncode != 0:
107+
diag = _adb_output_text(proc)
108+
raise OobAdbError(
109+
f"adb devices failed (exit code {proc.returncode}).\n\n"
110+
f"{diag}\n\n"
111+
"Check your adb installation and USB connection."
112+
)
113+
text = (proc.stdout or "") + "\n" + (proc.stderr or "")
114+
ready: list[str] = []
115+
for line in text.strip().splitlines()[1:]:
116+
line = line.strip()
117+
if not line:
118+
continue
119+
parts = line.split()
120+
if len(parts) >= 2 and parts[-1] == "device":
121+
ready.append(parts[0])
122+
if len(ready) == 0:
123+
raise OobAdbError(
124+
"No adb device found for --setup-oob.\n\n"
125+
"Plug in the USB cable, enable USB debugging on the headset, and check `adb devices`. "
126+
"Or omit --setup-oob and open the teleop URL on the headset yourself."
127+
)
128+
if len(ready) > 1:
129+
listed = ", ".join(ready)
130+
raise OobAdbError(
131+
"Too many adb devices for --setup-oob.\n\n"
132+
f"Currently connected: {listed}\n\n"
133+
"Unplug extras so only one headset is connected, then retry. "
134+
"Or omit --setup-oob and open the teleop URL manually."
135+
)
136+
137+
138+
def run_adb_headset_bookmark(*, resolved_port: int) -> tuple[int, str]:
139+
"""Open the teleop bookmark URL on the headset via ``am start``.
140+
141+
Uses the PC's LAN address — the headset reaches the proxy over WiFi.
142+
``resolved_port`` is used as the stream port unless ``TELEOP_STREAM_PORT``
143+
is set explicitly. Returns ``(exit_code, diagnostic)``.
144+
"""
145+
env_port = os.environ.get("TELEOP_STREAM_PORT", "").strip()
146+
signaling_port = int(env_port) if env_port else resolved_port
147+
proxy_host = resolve_lan_host_for_oob()
148+
stream_cfg: dict = {
149+
"serverIP": proxy_host,
150+
"port": signaling_port,
151+
**client_ui_fields_from_env(),
152+
}
153+
154+
ovr = web_client_base_override_from_env()
155+
web_base = ovr if ovr else DEFAULT_WEB_CLIENT_ORIGIN
156+
token = os.environ.get("CONTROL_TOKEN") or None
157+
url = build_headset_bookmark_url(
158+
web_client_base=web_base,
159+
stream_config=stream_cfg,
160+
control_token=token,
161+
)
162+
163+
shell_cmd = "am start -a android.intent.action.VIEW -d " + shlex.quote(url)
164+
full = ["adb", "shell", shell_cmd]
165+
log.info("ADB automation: %s", " ".join(shlex.quote(c) for c in full))
166+
proc = subprocess.run(full, capture_output=True, text=True)
167+
if proc.returncode != 0:
168+
diag = _adb_output_text(proc)
169+
return proc.returncode, diag
170+
log.info("ADB automation: am start completed")
171+
return 0, ""

0 commit comments

Comments
 (0)