WebXR (Android/Quest) support for XR mode#515
Conversation
# Conflicts: # rolldown.config.mjs
|
Thanks, this looks very cool. I can take a look in more detail soon, but I think my primary concern is about the secure context requirement. The first version of AdvantageScope XR that I used for development was based on WebXR, but I couldn’t come up with a good solution to the secure context problem that “just works” for users. The best solution I came up with was using VSCode’s built-in tunneling, but that 1) requires an internet connection and 2) is still too confusing to set up. In order to be viable, I think any solution for Android/VR probably needs to be just as easy to set up as the existing system on iOS. |
|
I have managed to get hosting a self-signed HTTPS server to work, and at least on desktop it can load WebXR perfectly. I didn't manage to test on device yet probably because of my weird network config, still troubleshooting that. The only problem with a self-signed server is that it does have the big red scary "this connection is not secure" warning that users have to manually bypass from the more details -> proceed unsafe section. Is that acceptable, or should I look for a solution with a nicer UI? Using a tunneling service would work, Cloudflare for example lets you run it from the CLI interface, but I think it would be really slow and use a lot of bandwidth. Also, requiring both devices to be online would make using this with live data on robot networks very difficult. But the UI would absolutely be a lot nicer. I think the theoretical option with the best UI and best performance would be developing an Android app that proxies the AdvantageScope Desktop IP to localhost and then loads it in a browser, but that would also require by far the most development and maintenance, and users would still have to download an app. |
|
I confirmed that a self-signed server will work on mobile devices as well. Here's what the scare screen looks like: https.mp4 |
|
Works for the Apple Vision Pro, at least in a minimal way of viewing. AVP doesn't scan QR codes, so I typed in the URL manually. |
|
Several people in Discord were opposed to the self-signed scare screen for Android phone users, so I've gone ahead and moved forward with making an Android app proxy. I've gotten QR scanning, web embedding, and the proxy part working, I just need to integrate them properly and work on the UI. I'm currently working on development at https://github.com/j5155/AdvantageScopeXR-Android , let me know how best to integrate that back into the larger project. Also, what's the reasoning behind the qr code exposing every IP of the server at the moment? I'm thinking about having the QR generation ui guess at the correct one and adding a dropdown menu for selecting it. That way the QR can go directly to http://server-ip:56328 and I can make a GUI for choosing whether to open in app or trigger the scare screen, and VR users can see what IP to type into their browsers manually like Todd. |
…ldown # Conflicts: # package-lock.json # package.json
# Conflicts: # src/main/electron/XRServer.ts
|
Here's a video with a Quest 3s! |
@blaze-developer has been working on some huge improvements to the web UI as well
|
# Conflicts: # src/main/electron/XRServer.ts
5 participants
Since the XR view currently uses three.js, it's relatively easy to enable WebXR and allow for both Android phones and VR headsets to use it.
Current status: Complete except for Android app. Demo video here, though I'd love it if someone with a Quest 3 or newer could record a better one that includes the passthrough!
Current setup: download and run a copy of AdvantageScope from the Actions build, then open 3D Field and XR mode. Scan the QR code with an Android phone, and/or press the Send to Meta Quest button to open on a Quest headset. To open on another device, manually type the displayed URL into the browser. (Docs updated.)
Development/Debugging: I configured tsconfig to allow you to build with ENABLE_SOURCEMAP=true as an environment variable, which makes debugging much easier. A local copy of something Chromium-based with this extension installed can be used for testing/debugging without a headset with the controls added in the WebXR section of Developer Tools [edit: seemingly updating three.js broke this??]. If you want to debug from within WebStorm, which I personally highly recommend, create a manual JavaScript Debug configuration with URL
http://localhost:56328and your Chromium-based browser. You'll need to install the extension a second time within the profile opened by the debugger. For debugging on headset, with it connected in ADB, you can openchrome://inspect/#devicesto remotely access DevTools on the headset. I haven't yet found a solution to attach the WebStorm debugger to the headset!TODO:
Known issues I don't know how to fix/need input on:
Potential future work:
Tested by me on a Pixel 3a, a Quest 2, an iPhone 12 and a iPad Air, and by others on an Apple Vision Pro and Quest 3S. No AI was used for any part of this PR beyond JetBrains IDE single line completion.