Any interaction method feasible?

[mkeblx]

Is there any way using Extension API to click on page, etc?

[samuelcardillo]

I am quite sure the Chrome (nor Firefox) APIs doesn't allow controlling the cursor because of security measures.

I think a good work around would be to use the NodeJS server OR to build a little application (could still use NodeJS + Electron or something similar), link WebScreenVR & the app with Socket.IO and use that to control the mouse.

I am focusing a bit more on the HUD & UX right now but that might be the next step. If you want to work on it or any other suggestions about how this would be possible, would be cool!

[mkeblx]

Yes more extensive architecture would work but argh, wish there was a way for trusted extension to simulate clicks directly.

document.elementFromPoint(x, y).click() might work, in more restricted web content case of tab or window capture type.

https://developer.mozilla.org/en-US/docs/Web/API/Document/elementFromPoint
https://developer.chrome.com/extensions/desktopCapture#type-DesktopCaptureSourceType

[samuelcardillo]

What if we take the values of where the raycaster is intersecting, the dimension of a-video element and the screen resolution? Have not tried yet but maybe there is a way to get the real screen coords through these values and then dispatch a "mouseClick" event (through NodeJS then...) ?

Worth a shot! I'll have a look when I come back on my computer in few hours!

[mkeblx]

Something like this from three.js:

var intersections = raycaster.intersectObject(screenObj);
var uv  = intersections[0].uv;
var x = uv.x * screenResolutionWidth;
var y = (1 - uv.y) * screenResolutionHeight;

document.elementFromPoint(x, y).click();
// again, only would semi-work for clicks, & restricted to web content..

https://threejs.org/docs/index.html#api/core/Raycaster

I see can access same from Aframe intersection events.
https://aframe.io/docs/0.5.0/components/raycaster.html