diff --git a/src/_posts/hands-on-i.md b/src/_posts/hands-on-i.md new file mode 100644 index 00000000..f87f3109 --- /dev/null +++ b/src/_posts/hands-on-i.md @@ -0,0 +1,149 @@ +--- +title: Hands-On! Building Interactive Training With A-Frame +author: twitter|salvadelapuente|Salvador de la Puente +date: 2017-08-09 +layout: blog +image: + src: hands-on-i.png +--- + +In an industry where manipulation of complex mechanisms involves tens of tools, expensive equipment and even safety risks, virtual reality helps to increase operator safety and reduce material costs. In addition, WebVR puts everything just one link away. + +This is the first of a series of articles providing an in-depth walkthrough of a hand-based interaction and manipulation model for training purposes ([check the demo online](https://delapuente.github.io/aframe-interactive-training/step3/)). The series is not a step-by-step tutorial but a postmortem breakdown covering one major feature at a time. At the same time you build the experience, you will also learn about core A-Frame concepts, its ecosystem, the entity-component-system architecture, how to use the [A-Frame physics system](https://hacks.mozilla.org/2017/05/having-fun-with-physics-and-a-frame/), common errors, workarounds and good practices. All you need to confront a real project. + + + +This is an advanced virtual reality experience and so it requires a head-mounted display supporting room-scale VR and [6DoF](https://en.wikipedia.org/wiki/Six_degrees_of_freedom) controllers such as those included in the [HTC VIVE](https://www.vive.com/us/product/#controller-intro) or the [Oculus Touch](https://www.oculus.com/rift). Personally, I used a VIVE. + +All articles in the series: + + 1. Hands-on! Building interactive training with A-Frame + 2. [Hands-on! Simple physical behaviours in A-Frame](/blog/hands-on-ii/) + 3. [Hands-on! Hand-based interactions in A-Frame](/blog/hands-on-iii/) + +## The project repository + +Use git to clone the [repository of the simulation](https://github.com/delapuente/aframe-interactive-training). Enter the repository and run `npm install` then `npm start` to install the dependencies and start the development server at port `8080`. + +At the root of the project, you'll find [`step1/`](https://github.com/delapuente/aframe-interactive-training/tree/master/step1), [`step2/`](https://github.com/delapuente/aframe-interactive-training/tree/master/step2) and [`step3/`](https://github.com/delapuente/aframe-interactive-training/tree/master/step3) with the **finished HTML** for each step. Look at the same folders inside the [`js/`](https://github.com/delapuente/aframe-interactive-training/tree/master/js) directory to browse the JavaScript code. + +Open the browser and enter the URL `localhost:8080` to find the index of steps. Append `step1` at the end of the URL path and click on the VR button on the bottom-right corner of the display to experience the project after step 1 is completed (or select step 1 in the list). + +The project is based on the [A-Frame SPA skeleton](https://github.com/delapuente/aframe-spa-skeleton) that quickly bootstraps a virtual reality single-page application with auto reload and publishing features. Read the [a-frame-skeleton README file](https://github.com/delapuente/aframe-spa-skeleton#a-frame-spa-skeleton) for further information. + +## Planning the demo + +The app’s initial aim was to provide a simulation of the process of backing up the contents of a chip from an in-car key reader device, which includes loosening the solder and removing the chip from a circuit board. As a reference, I used a YouTube video showing the [complete in-real-life procedure](https://youtu.be/qFGX3AHVLqA?t=90). The video comprises several steps involving around eight different tools. For this series, I'm going to focus on the specific step of separating the chip from the circuit board. See a detailed sequence here: + + + +The required steps are as follow: + + 1. With a hot air gun, soften the soldering around the chip. + 2. With the help of a suction pad, separate the chip from the circuit board. + 3. Carefully place the chip on the table, to avoid ruining the chip pins. + +I started prototyping to implement one step at a time without realistic models in the beginning. Once I am able to recreate the complete process, I can think about adding different levels of detail to the demo. + +But before providing specific simulations and behaviors, we need some basic hand-based interaction. + +## Setting the scene up + +The first thing I needed was a scene containing all the elements I wanted to work with: the air gun, the suction pad, the chip, the soldering around the chip, and the circuit board. Of course I needed a representation of my own hands and some environmental elements such as a sky, a floor and of course a table! + +This is the initial code for the scene: + +```html + + + + + + + + + + + + + + + + + + + + + + + +``` + +When working with A-Frame, all the scene objects are contained inside the `a-scene` element. A-Frame provides some special tags to achieve common tasks such as adding a sky dome or a camera into the virtual space and it also has primitives available such as boxes, planes, or cylinders. + +This is how I set up the environment with a sky dome, and a plane decorated with a couple of sad grey tones. + +```html + + + +``` + +Length units in A-Frame are expressed in meters and angles are described in sexagesimal degrees (0º to 360º). Thus, the floor has a surface of _15m x 15m_, the workbench is _2cm_ thick and has a surface of _2m x 1m_, and the circuit board is _3mm thick_ and measures _5cm x 5cm_. Real measures are important for two reasons: they **improve the feeling of immersion** and provide **realistic physics**. + +When placing elements, take into account that they are placed according to their central point in the coordinate system of the parent element. If they are direct children of the `a-scene` element, then they are positioned according to the world coordinates system. In the case of the chip, **it is positioned relative to the central point of the circuit board** because the circuit board is the parent node of the chip. + +```html + + + + +``` + +A-Frame provides a way of adding empty nodes to the scene or, in A-Frame nomenclature, an **entity**. [Entities](/docs/0.6.0/core/entity.html) are represented by the `a-entity` element and they have nothing associated with them: no geometry, model, or material. They do have implicit `rotation`, `position`, `scale` and `visible` attributes, which are automatically injected by the framework for all the elements within the scene. + +Adding the [`hand-controls`](/docs/0.6.0/components/hand-controls.html) attribute to an entity displays a hand model following the position and orientation of one of the tracked controllers of an HTC Vive or Oculus. + +```html + + +``` + +This is the resulting scene: + +[![Initial scene](/images/blog/hands-on/step1-initial-scene.png)](/images/blog/hands-on/step1-initial-scene.png) + +### One workspace to rule them all + +To increase the feeling of immersion, I placed the elements as if they were resting on my real desktop only 5cm higher up. This allowed me to avoid my hands colliding with my physical keyboard when I’m in VR mode. Adjusting the whole workspace at the same time is achieved by changing the `position` attribute of the `workspace` entity. Thanks to the real and virtual table heights matching, the demo has a great feeling of presence. + +A simpler alternative would have been to make the circuit board and tools children of the table. Unfortunately this configuration does not play well when adding physics during the next article. The physics engine would consider the nested elements to be compounding parts of the root element and I didn’t want my tools and circuit board to be considered part of the table but separated elements. + +Also notice that if you are experiencing room-scale VR, you will also need to manually position the workspace and camera relative to the center of your configured stage, making them match with your real-life table and yourself respectively. + +Finally, while developing I spend most of my the time sitting in front of my computer. So as a last touch I changed the height correction of the camera, the `user-height` attribute, to match the distance of my eyes from the floor when I'm sitting down: + +```html + +``` + +This correction is [automatically disabled when entering VR](/docs/0.6.0/components/camera.html#vr-behavior) mode since the position of the camera is provided by the VR head-mounted display (HMD). + +## Conclusion + +The very first thing you need to start implementing a simulation experience is a clear reference of the real procedure. Some clues? Watch videos, take descriptive notes and diagrams, and maintain a fluid conversation with an expert. + +The next step is to figure out a basic depiction of what the environment looks like. You don't need complex models to do it, just some basic shapes to provide placeholders for the realistic versions that you will add in the future. This is a common practice when prototyping game environments called [**greyboxing**](http://jackw-gamedesign.tumblr.com/post/139960850160/what-is-greyboxing). + +In the [next article](/blog/hands-on-ii/), I will add some physics in preparation for grabbing things, something imperative in almost any simulation experience. diff --git a/src/_posts/hands-on-ii.md b/src/_posts/hands-on-ii.md new file mode 100644 index 00000000..26fa1415 --- /dev/null +++ b/src/_posts/hands-on-ii.md @@ -0,0 +1,175 @@ +--- +title: Hands-On! Simple Physical Behaviours In A-Frame +author: twitter|salvadelapuente|Salvador de la Puente +date: 2017-08-09 +layout: blog +image: + src: hands-on-ii.png +--- + +This article is the second in a series showing how to build a hand-based, interactive training experience in VR for a circuit manipulation procedure. In the [first part](/blog/hands-on-i/), I chose a [specific step in the procedure](https://www.youtube.com/embed/tHP2kX6aAZM?rel=0&showinfo=0) and prototyped the workspace, tools and materials used in that step with simple geometrical primitives. This time around I will add basic physics to the elements of the scene and you will learn about A-Frame’s entity-component-system architecture. + +All articles in the series: + + 1. [Hands-on! Building interactive training with A-Frame](/blog/hands-on-i/) + 2. Hands-on! Simple physical behaviours in A-Frame + 3. [Hands-on! Hand-based interactions in A-Frame](/blog/hands-on-iii/) + + + +## The project repository + +If you’ve not done so already, clone the [repository of the simulation](https://github.com/delapuente/aframe-simulation-demo). Enter the repository and run `npm install` and `npm start` to install the dependencies and start the development server at port `8080`. + +Look at [`step2/index.html`](https://github.com/delapuente/aframe-interactive-training/blob/master/step2/index.html) and [`js/step2`](https://github.com/delapuente/aframe-interactive-training/tree/master/js/step2) to browse the finished HTML and JavaScript code of this step. Open a browser and enter the following URL `localhost:8080` to find the index of steps. Append `step2` at the end of the URL path to play the scene after step 2 is completed (or select step 2 in the list). + +## The holding problem + +An accurate correspondence between virtual and real objects is vital to increase the sense of presence. The problem with the hands models in the scene is that they are not oriented in the same way than your real hands hold the controllers. + +Take a look at the following video, focus on how the real fingertips and the virtual hand's don't match. The grab axis, around which the fingers close, is also misaligned. + + + +The simulation is more effective when using the 3D models of the VIVE controllers since the tools you see in VR will match the real tools hold in your hands. Using the `vive-controls` instead of `hand-controls` attribute also [allow us to change the model](/docs/0.6.0/components/vive-controls.html#value_model) in the future so we can provide a better representation of our hands. + +```html + + + + + +``` + +## The physics system +The [`aframe-physics-system`](https://github.com/donmccurdy/aframe-physics-system) extension is a wrapper around the [Cannonjs physics engine](http://www.cannonjs.org/) developed by [Don McCurdy](https://twitter.com/donrmccurdy). Mozilla Hacks blog has a good introduction to this extension in [Having fun with physics and A-Frame](https://hacks.mozilla.org/2017/05/having-fun-with-physics-and-a-frame/). + +Enabling physics in A-Frame consists on importing the proper module after importing A-Frame (see `/js/step2/index.js`): + +```js +import * as AFRAME from 'aframe'; +import * as physics from 'aframe-physics-system'; +``` + +Next, we need to set some attributes in the elements of the scene to define how they will be affected by the physics engine. I turned the floor, the workbench, and the operator's hands into _static bodies_ by adding the `static-body` attribute to these elements. A _static body_ is still a physical object but its properties are not controlled by the physics engine —they are not affected by gravity or other forces, which prevents the floor and table from falling indefinitely. + +```html + + + + + + + + + + + +``` + +On the contrary, a _dynamic body_ is an object fully controlled by the physics engine. I turned the circuit board containing the chip into a _dynamic body_ by setting the attribute `dynamic-body` on it. The value of the property establishes the mass of the object —I have set it to weigh 10g (which equates to 0.01, since Cannon.js uses [S.I.](https://en.wikipedia.org/wiki/International_System_of_Units) units, so kilograms for mass). + +```html + + + + +``` + +Dynamic bodies collide with other dynamic bodies **and with static bodies**. This is what the scene looks like after enabling physics: + +[![Red boxes around the floor and table. The circuit board is missing.](/images/blog/hands-on/step2-initial-scene.png)](/images/blog/hands-on/step2-initial-scene.png) + +Red boxes are a result of the physics engine debug mode being enabled. You can enable it by setting the `physics` attribute of the `scene` tag to `debug: true`. You'll see a red box around the workbench, floor and circuit board... wait a moment! Where the heck is the circuit board!? + +[![The circuit board is under the table. It fell through the table somehow.](/images/blog/hands-on/step2-where-is-the-chip.gif)](/images/blog/hands-on/step2-where-is-the-chip.gif) + +Oh! There it is. + +### Limits of the physics system + +It turns out that the chip is not resting on the table but on the floor. The chip fell through the table because of the thickness of their physical bodies. + +Indeed, Cannon.js [lacks from continuous collision detection](https://github.com/schteppe/cannon.js/issues/50#issuecomment-13730383): it only checks if an object is colliding another from frame to frame. With the Earth gravity (which is the default in Cannon.js), an [object falls around 3cm after the first 80ms](https://en.wikipedia.org/wiki/Free_fall#Uniform_gravitational_field_without_air_resistance) (~5 frames). Assuming I was probably missing some frames during the scene setup, a quick workardound was to increas the thickness (the `height` attribute of a box) of the workbench by a couple of centimetres to prevent the circuit board from passing through the table. This forced me to relocate the elements resting on the table, but this was preferable to changing the gravity. + +```html + + +``` + +Another problem regarding the physics engine is related to the collision bounding shapes used for rotated models such as the VIVE controllers. They are miscalculated in terms of dimensions and positioned out of place for certain initial configurations. + +[![Miscalculated and misplaced bounding box.](/images/blog/hands-on/step2-miscalculated-collision-box.png)](/images/blog/hands-on/step2-miscalculated-collision-box.png) + +Nevertheless, this is something I’ll fix during the next article. At this point, you can experiment with the scene using the bounding box to move the circuit board around. Since the controllers can pass through the table, a funny interaction is trying to raise the chip placing your controller below it: + +[![The VIVE controls interact with the circuit board realistically.](/images/blog/hands-on/step2-raising-the-circuit-board.gif)](/images/blog/hands-on/step2-raising-the-circuit-board.gif) + +## The entity-component-system architecture + +The [entity-component-system pattern](https://en.wikipedia.org/wiki/Entity%E2%80%93component%E2%80%93system) (ECS) is an architectural pattern in which objects or entities are composed of multiple aspects. A component captures an aspect of an entity and a system orchestrates the global interactions among the entities possessing a component of the same aspect. The A-Frame documentation includes a [complete chapter dedicated to ECS](/docs/0.6.0/introduction/entity-component-system.html). + +In A-Frame, the entities are the `a-entity` elements and all the primitives (`a-box`, `a-plane`, `a-cylinder`, `a-camera`...) included in the scene. Components are represented by the elements' attributes so, hereinafter, we will stop using the word _attribute_ and start using _component_ when applicable. For instance, consider the operator's hands: + +```html + + + + +``` + +Each hand is actually an entity which has two components `static-body` and `vive-controls`. The first is required by the physics engine to detect when it is colliding with other bodies. The second shows the specified controller making it automatically match the position and rotation of the tracked control. Components can be one-value such as the former `hand-controls` or multi-value like `static-body` or `vive-controls` (i.e., comprised of several named values like the _mass_ or the colliding _shape_). The different values of a multi-value component are called properties. + +Finally, systems are pure behaviour extensions and therefore, they are usually invisible. Take the physics engine for instance. It is a system and it orchestrates the interactions between those entities with `static-body` and `dynamic-body` components (among others). + +### A note on primitives + +[Primitives are entities under the hood](/docs/0.6.0/introduction/html-and-primitives.html#primitives). Primitives have specific attributes mapped to certain properties of components implied by the primitive. For instance: + +```html + + +``` + +This `a-cylinder` primitive implies [`geometry`](/docs/0.6.0/components/geometry.html) and [`material`](/docs/0.6.0/components/material.html) components. The `primitive` property of the `geometry` component is automatically set to `cylinder`. Then, `height`, `radius`, and `color` are attributes mapped to `geometry.height`, `geometry.radius`, and `material.color` respectively, while `position` and `rotation` are components on their own. + +The equivalent entity looks like: + +```html + + +``` + +## The A-Frame registry + +The [A-Frame registry](/aframe-registry/) is a centralized source of components. There you can find a collection of components, download the libraries and visit their home pages. Look at the [`aframe-auto-detect-controllers`](https://www.npmjs.com/package/aframe-auto-detect-controllers-component) component for instance. Given that I replaced the hand controls with the HTC VIVE controllers, I could have used this component to automatically detect which particular tracked controller to use, e.g. HTC Vive or Oculus Touch. + +## Conclusion + +All the bits we need are now in place to start implementing behaviours. Despite the limitations of the Cannon.js physics engine, [Don McCurdy’s](https://github.com/donmccurdy) A-Frame physics offers a simple and intuitive component-oriented API, and exposes enough features for covering our specific needs. Nevertheless, for simulation experiences to succeed in WebVR, we need more accurate and faster physics implementations. My hopes are with [Web Assembly](https://hacks.mozilla.org/2017/02/a-cartoon-intro-to-webassembly/), a new language for the Web that promises huge performance gains over JavaScript. + +In the [next chapter](/blog/hands-on-iii/) I will go deeper in the ECS pattern, implementing _grab and drop_ and fixing the collision bounding box problem by using some custom components and systems. diff --git a/src/_posts/hands-on-iii.md b/src/_posts/hands-on-iii.md new file mode 100644 index 00000000..514448f5 --- /dev/null +++ b/src/_posts/hands-on-iii.md @@ -0,0 +1,330 @@ +--- +title: Hands-On! Grab & Drop Interactions In A-Frame +author: twitter|salvadelapuente|Salvador de la Puente +date: 2017-08-09 +layout: blog +image: + src: hands-on-iii.png +--- + +This is the third article in the Hands-on! series, showing how to build an interactive training experience in VR. In the previous parts, I showed you [how to prototype a scene](/blog/hands-on-i/) and [enable physics allowing for basic interactions](/blog/hands-on-ii/). You also learned about A-Frame’s Entity-Component-System pattern (ECS), which I’ll use in this part to implement _grab and drop_. + +All articles in the series: + + 1. [Hands-on! Building interactive training with A-Frame](/blog/hands-on-i/) + 2. [Hands-on! Simple physical behaviours in A-Frame](/blog/hands-on-ii/) + 3. Hands-on! Hand-based interactions in A-Frame + + + +## The project repository + +Clone and setup the [repository of the simulation](https://github.com/delapuente/aframe-simulation-demo). Run `npm start` to start the development server at port `8080` and enter the following URL `localhost:8080` to find the index of steps. Append `step3` at the end of the URL path to play the scene after step 3 is completed (or select step 3 in the list). + +Look at [`step3/index.html`](https://github.com/delapuente/aframe-interactive-training/blob/master/step3/index.html) and [`js/step3`](https://github.com/delapuente/aframe-interactive-training/tree/master/js/step3) to browse the finished HTML and JavaScript code of this step. Open a browser and + +## Grabbing things + +There are several ways of grabbing things. The most important thing after grabbing something is to ensure that the grabbed item follows the position and orientation of the tracked hand. I accomplished this in two different ways: + + 1. For tools like the hot-air gun and the suction pad, I didn’t rely on physics; I instead performed a quick proximity test and attached the tools directly to the tracked controls. + + 2. For other entities such as the circuit board or the chip (once it is separated), I relied on the physics system, setting a [_constraint_](https://github.com/donmccurdy/aframe-physics-system#components--constraint) between the circuit board and the tracked controller. A constraint is a way of binding the physical behavior of a body to a different one. + +### The operator system + +According to ECS theory, coordinating interactions between the operator’s hands and other elements in the scene sounds like something a system should be in charge of. This system will be the `operator` system. + +In A-Frame, a system can be configured using the component notation. For instance, when setting `physics` component to `debug: true`, I’m telling the physics system to render a representation of the collision bounding boxes. The following code will configure the `operator` system that I’ll implement in a while: + +```html + + +``` + +The `operator` system is parametrized by two properties: `hands` and `items`. The `items` property represents all the items that the hands can grab. I’ve used CSS attribute selectors and they will be translated into lists of elements as if they were looked up using the [`querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll) method. The barebones of the `operator` system looks like this: + +```js +AFRAME.registerSystem('operator', { + schema: { + hands: { type: 'selectorAll' }, + items: { type: 'selectorAll' } + }, + _grabbedItems: {}, + init() { + const hands = this.data.hands; + hands.forEach(hand => { + hand.addEventListener('gripdown', () => console.log('Grab!')); + hand.addEventListener('gripup', () => console.log('Drop!')); + }); + } +}); +``` + +The A-Frame documentation has a chapter explaining in detail — see [how to create and register new systems](/docs/0.6.0/core/systems.html). + +The [`gripdown` and `gripup` events](https://aframe.io/docs/0.5.0/components/vive-controls.html#events) are emitted by the entities with the `vive-controls` component when buttons on the grip are pressed or released respectively. + +### Grabbing physical bodies + +Grabbing a physical body will establish a constraint between the item being grabbed and the hand. The code looks like: + +```js +_grab(hand, item) { + if (item) { + const isGrabbed = item.hasAttribute('constraint'); + if (!isGrabbed) { + item.setAttribute('constraint', `target: #${hand.id}`); + this._grabbedItems[hand.id] = item; + } + } +} +``` + +We keep track of which item is grabbed by each hand to simplify dropping: + +```js +_drop(hand) { + const grabbedItem = this._grabbedItems[hand.id]; + if (grabbedItem) { + grabbedItem.removeAttribute('constraint'); + this._grabbedItems[hand.id] = null; + } +} +``` + +With those in place, the final `init` method looks like this: + +```js +init() { + const hands = this.data.hands; + const items = this.data.items; + hands.forEach(hand => { + hand.addEventListener('gripdown', () => { + const item = this._findNearby(hand, items); + this._grab(hand, item); + }); + hand.addEventListener('gripup', () => { + this._dropBody(hand); + }); + }); +} +``` + +The proximity test used to find which of the items is reachable by the operator’s hand can be implemented in several ways. For simplicity, I decided to compute the distance between the controller origin and item origins. If this distance is less than _20cm_, then it is reachable: + +```js +_findNearby(hand, items) { + for (let i = 0, l = items.length; i < l; i++) { + const item = items[i]; + if (distance(item) < 0.2) { + return item; + } + } + return null; + + function distance(item) { + const handPosition = hand.getObject3D('mesh').getWorldPosition(); + const itemPosition = item.getObject3D('mesh').getWorldPosition(); + return handPosition.distanceTo(itemPosition); + } +} +``` + +With A-Frame you can use well known HTML APIs to create VR interactions. The only thing new for developers to learn is the API for computing distances, which will be covered later in this post. + +[![Grabbing physical objects](/images/blog/hands-on/step3-grabbing-physical-bodies-thumb.png)](/images/blog/hands-on/step3-grabbing-physical-bodies.gif) + +### Grabbing tools + +Grabbing tools is a different thing. It usually involves complex handling not only to hold the tool but to hold it properly. For this simulation, I chose to automatically correct the position and rotation of the tool in the hand once the operator grabbed it. + +When dropping, instead of letting the tool fall from its current position, I opted for returning it to its original position. To store the proper position and rotation of the tool while being held or resting, I created the `tool` component: + +```js +AFRAME.registerComponent('tool', { + dependencies: [‘tool-original-location’], + schema: { + position: { type: 'vec3' }, + rotation: { type: 'vec3' } + }, + grab() { + this._rememberLocation(); + this.setAttribute('position', this.data.position); + this.setAttribute('rotation', this.data.rotation); + this.isGrabbed = true; + }, + drop() { + this._restoreLocation(); + this.isGrabbed = false; + }, + _rememberLocation() { + this._originalParent = this.el.parentNode; + this._originalPosition = this.el.getAttribute('position'); + this._originalRotation = this.el.getAttribute('rotation'); + }, + _restoreLocation() { + this.el._originalParent.appendChild(this.el); + this.el.setAttribute('position', this._originalPosition); + this.el.setAttribute('rotation', this._originalRotation); + } +}); +``` + +The [`schema`](/docs/0.6.0/core/component.html#schema) key allows us to define multiple properties describing the `component`. A [complete list of all property types](/docs/0.6.0/core/component.html#property-types) can be found in the A-Frame documentation, which also includes a chapter explaining [how to create and register new components](/docs/0.6.0/core/component.html) with additional detail. + +I applied the component `tool` to the tools in the scene with the correct values for position and rotation while being held: + +```html + + + + +``` + +The following functions implement the code for grabbing and dropping tools. The return value indicates the result of grabbing or dropping and it is used by the callee of the functions to mark the items as grabbed or released: + +```js +function _grabTool(element) { + const tool = element.components.tool; + if (!tool.isGrabbed) { + grabbedItem = element; + tool.grab(); + hand.appendChild(element); + return true; // signal the success + } + return false; +} + +function _dropTool() { + const tool = grabbedItem.components.tool; + tool.drop(); + return true; +} +``` + +Before continuing you should be aware that the above code does not function correctly. There is an aspect in which **A-Frame and normal HTML differ**, and it is subtle and counterintuitive. + +As you can see, grabbing a tool implies making it a direct child of the operator’s hand. This way the tool will respond to the hand’s movements in a realistic way. But detaching and re-attaching the tool from the DOM **causes the components to be reset**. More precisely: when re-attaching, the components are completely new. By the time `_dropTool()` is called, the component has been detached from the workspace node and re-attached as a child of the hand so it won’t preserve the values in `_originalParent`, `_originalPosition` and `_originalRotation` set during the `tool.grab()` call. + +A solution would have been to declare those values as part of the schema but it does not work out of the box. It requires a call to the entity’s [`flushToDOM()`](https://aframe.io/docs/0.5.0/core/entity.html#flushtodom-recursive) method because A-Frame avoids serializing data for performance reasons. Furthermore, it is currently impossible to properly serialize the parent node if that node doesn’t have an `id`. + +One way of preserving this data, without changing the way the element is used, is to store it inside the HTML node. The important piece is in the `init` method where the component creates or reuses a storage object to hold the state. + +```js +AFRAME.registerComponent('tool', { + schema: { + position: { type: 'vec3' }, + rotation: { type: 'vec3' } + }, + init() { + this.el._tool = this.el._tool || {}; + }, + grab() { + this._rememberLocation(); + this.el.setAttribute('position', this.data.position); + this.el.setAttribute('rotation', this.data.rotation); + this.el._tool._isGrabbed = true; + }, + drop() { + this._restoreLocation(); + this.el._tool._isGrabbed = false; + }, + _rememberLocation() { + this.el._tool._originalParent = this.el.parentNode; + this.el._tool._originalPosition = this.el.getAttribute('position'); + this.el._tool._originalRotation = this.el.getAttribute('rotation'); + }, + _restoreLocation() { + this.el._tool._originalParent.appendChild(this.el); + this.el.setAttribute('position', this.el._tool._originalPosition); + this.el.setAttribute('rotation', this.el._tool._originalRotation); + }, + isGrabbed() { + return this.el._tool._isGrabbed; + } +}); +``` + +Now we can grab every element in the scene: + +[![Grabbing tools](/images/blog/hands-on/step3-grabbing-tools-thumb.png)](/images/blog/hands-on/step3-grabbing-tools.gif) + +### Fixing the collision bounding shapes + +In the [previous article](/blog/hands-on-ii.html), I mentioned that the physic engine was miscalculating the collision bounding box of the VIVE controllers. Making the bounding box to match the controller model is especially important when trying to grab physical objects, after disabling the debug mode of physics. If the bounding box, and the model position and dimensions don’t match, the user real interactions won't match what the user sees in VR resulting in a frustrating experience. + +The problem lies in the communication between A-Frame and the pyhsics component: on one side, the physics component is not considering the current rotation of the controller properly so, unless the controller is resting in its default position, the bounding box dimensions would be miscalculated. On the A-Frame side, the `vive-controls` component is [applying a correction in the position of the model after loading the model](https://github.com/aframevr/aframe/blob/master/src/components/vive-controls.js#L202). This correction is not advertised outside the component so the physics component can not realize when the model is fully placed. Since the bounding box calculation is done before applying the correction, the bounding box will be misplaced. + +To fix this problem, I’ve created a `delayed-static-body` component which solves both problems. To deal with the positioning problem, I delayed setting the real `static-component` until the next frame, once I know the browser has recovered the control of the execution and so, the correction must to be applied: + +```js +import * as AFRAME from 'aframe'; +import * as physics from 'aframe-physics-system'; + +const Component = AFRAME.registerComponent('delayed-static-body', { + schema: AFRAME.components['static-body'].schema, + init() { + this.el.addEventListener('model-loaded', () => { + window.requestAnimationFrame(this._resetBoundingBox.bind(this)); + }); + }, + _resetBoundingBox() { /* ... */ } +}); +``` + +Notice how I borrow the schema from the `static-body` component to be the same. + +To fix the miscalculation of dimensions, I store the current rotation of the entity, reset the rotation to 0, set the `static-body` component to the value of this component and restore the original rotation so the bounding box calculations happen when the entity lies in its default orientation. + +```js +_resetBoundingBox() { + var currentRotation = this.el.getAttribute('rotation'); + this.el.setAttribute('rotation', { x: 0, y: 0, z: 0 }); + this.el.removeAttribute('static-body'); + this.el.setAttribute('static-body', this.data); + this.el.setAttribute('rotation', currentRotation); +} +``` + +The complete component is in the [`js/step3/components/delayed-static-body.js`]() file. To use it, I replaced the name part of the `static-body` component of the hands with `delayed-static-body` leaving the value parts untouched. + +```html + + + + +``` + +## A-Frame and Three.js + +In the same way the `aframe-physics-system` is a wrapper around the Cannon.js library, which exposes part of the API in the form of convenient `components` for the A-Frame nodes, A-Frame itself is a wrapper for the [Three.js 3D library](https://threejs.org/). + +Three.js is a powerful and popular 3D library built on top of WebGL, which exposes a high-level API focused on manipulating a scene graph instead of working directly with 3D primitives. + +One final function that I needed for the demo was to be able to calculate distances between objects. To do this I used some of the native Three.js functionality in A-Frame with the help of the [`getObject3d`](/docs/0.6.0/core/entity.html#getobject3d-type) entities’ method. I used this API to access the Three.js [Object3D API](https://threejs.org/docs/index.html) and compute the [distance](https://threejs.org/docs/index.html#api/math/Vector3.distanceTo) between two Three.js [Vector3](https://threejs.org/docs/index.html#api/math/Vector3) objects according to [their world coordinates](https://threejs.org/docs/index.html#api/core/Object3D.getWorldPosition) in the `distance` auxiliary function: + +```js +function distance(item) { + const handPosition = hand.getObject3D('mesh').getWorldPosition(); + const itemPosition = item.getObject3D('mesh').getWorldPosition(); + return handPosition.distanceTo(itemPosition); +} +``` + +## Conclusion + +So far, you have learned how to prototype a complex and semantic VR scene, enable physics, workaround some limitations, and add basic interactions. You’ve discovered the [A-Frame registry](https://aframe.io/aframe-registry/) and now you know how to make the most of the [ECS pattern](/docs/0.6.0/introduction/entity-component-system.html) and create your own [customised components](/docs/0.6.0/core/component.html) and [systems](/docs/0.6.0/core/systems.html). + +Stay tuned for new articles of the Hands-on! series and you will learn how to enrich the experience by adding textures, effects, 3D models, sound and UI. If you liked these articles, [mention us on Twitter](https://twitter.com/aframevr) and don't hesitate asking your questions in the [A-Frame slack channel](https://aframevr.slack.com/). diff --git a/src/images/blog/hands-on-i.png b/src/images/blog/hands-on-i.png new file mode 100644 index 00000000..89a90f66 Binary files /dev/null and b/src/images/blog/hands-on-i.png differ diff --git a/src/images/blog/hands-on-ii.png b/src/images/blog/hands-on-ii.png new file mode 100644 index 00000000..c578e83e Binary files /dev/null and b/src/images/blog/hands-on-ii.png differ diff --git a/src/images/blog/hands-on-iii.png b/src/images/blog/hands-on-iii.png new file mode 100644 index 00000000..1958264c Binary files /dev/null and b/src/images/blog/hands-on-iii.png differ diff --git a/src/images/blog/hands-on/step1-initial-scene.png b/src/images/blog/hands-on/step1-initial-scene.png new file mode 100644 index 00000000..912e797d Binary files /dev/null and b/src/images/blog/hands-on/step1-initial-scene.png differ diff --git a/src/images/blog/hands-on/step2-initial-scene.png b/src/images/blog/hands-on/step2-initial-scene.png new file mode 100644 index 00000000..0eaf41e2 Binary files /dev/null and b/src/images/blog/hands-on/step2-initial-scene.png differ diff --git a/src/images/blog/hands-on/step2-miscalculated-collision-box.png b/src/images/blog/hands-on/step2-miscalculated-collision-box.png new file mode 100644 index 00000000..4c24a6b0 Binary files /dev/null and b/src/images/blog/hands-on/step2-miscalculated-collision-box.png differ diff --git a/src/images/blog/hands-on/step2-raising-the-circuit-board.gif b/src/images/blog/hands-on/step2-raising-the-circuit-board.gif new file mode 100644 index 00000000..bdf08003 Binary files /dev/null and b/src/images/blog/hands-on/step2-raising-the-circuit-board.gif differ diff --git a/src/images/blog/hands-on/step2-the-holding-problem.mp4 b/src/images/blog/hands-on/step2-the-holding-problem.mp4 new file mode 100644 index 00000000..a248ba5c Binary files /dev/null and b/src/images/blog/hands-on/step2-the-holding-problem.mp4 differ diff --git a/src/images/blog/hands-on/step2-where-is-the-chip.gif b/src/images/blog/hands-on/step2-where-is-the-chip.gif new file mode 100644 index 00000000..611a4804 Binary files /dev/null and b/src/images/blog/hands-on/step2-where-is-the-chip.gif differ diff --git a/src/images/blog/hands-on/step3-grabbing-physical-bodies-thumb.png b/src/images/blog/hands-on/step3-grabbing-physical-bodies-thumb.png new file mode 100644 index 00000000..d7d9f8db Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-physical-bodies-thumb.png differ diff --git a/src/images/blog/hands-on/step3-grabbing-physical-bodies.gif b/src/images/blog/hands-on/step3-grabbing-physical-bodies.gif new file mode 100644 index 00000000..88f6bff9 Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-physical-bodies.gif differ diff --git a/src/images/blog/hands-on/step3-grabbing-tools-thumb.png b/src/images/blog/hands-on/step3-grabbing-tools-thumb.png new file mode 100644 index 00000000..a240a9dc Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-tools-thumb.png differ diff --git a/src/images/blog/hands-on/step3-grabbing-tools.gif b/src/images/blog/hands-on/step3-grabbing-tools.gif new file mode 100644 index 00000000..95a3162c Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-tools.gif differ