Skip to content

Commit 5dd3011

Browse files
marwiemarwie
committed
update image tracking docs
1 parent 876a55d commit 5dd3011

4 files changed

Lines changed: 194 additions & 0 deletions

File tree

‎documentation/how-to-guides/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ Head to [Getting Started](/docs/getting-started/) to install Needle Engine for [
4848
- [Use Physics](/docs/how-to-guides/scripting/use-physics) - Rigidbodies, forces, collisions, triggers — powered by [Rapier](https://rapier.rs/)
4949
- [Accessibility](/docs/how-to-guides/accessibility) - Make 3D scenes accessible to screen readers and assistive technology
5050
- [Detect Mobile Devices](/docs/how-to-guides/scripting/detect-mobile-devices) - Platform detection
51+
- [Image Tracking Scripting](/docs/how-to-guides/scripting/image-tracking) - Access tracked images and objects from code
5152

5253
---
5354

‎documentation/how-to-guides/scripting/image-tracking.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
---
2+
title: Scripting Image Tracking
3+
description: Access tracked images, assigned objects, and tracking data from code using the image-tracking event
4+
---
5+
6+
# Scripting Image Tracking
7+
8+
Learn how to access tracked image data, assigned 3D objects, and tracking state from TypeScript using the `image-tracking` event.
9+
10+
:::warning Advanced
11+
This guide covers scripting image tracking from code. For most use cases, the built-in `WebXRImageTracking` component handles everything automatically — just set it up in Unity, Blender, or [add it from code](#setting-up-image-tracking-from-code). See the [Image Tracking setup guide](/docs/how-to-guides/xr/image-tracking) first.
12+
:::
13+
14+
:::tip Prerequisites
15+
Make sure you have [WebXR Image Tracking](/docs/how-to-guides/xr/image-tracking) set up in your scene first.
16+
:::
17+
18+
---
19+
20+
## Setting Up Image Tracking from Code
21+
22+
You can configure image tracking entirely from TypeScript without the Unity/Blender editor:
23+
24+
```ts
25+
import { Behaviour, WebXRImageTracking, WebXRImageTrackingModel } from "@needle-tools/engine";
26+
import { Object3D } from "three";
27+
28+
export class CodeImageTracking extends Behaviour {
29+
start() {
30+
const tracker = this.gameObject.getOrAddComponent(WebXRImageTracking);
31+
32+
const model = new WebXRImageTrackingModel({
33+
url: "https://engine.needle.tools/samples/image-tracking/assets/imagetracking-01-marker.jpg",
34+
widthInMeters: 0.3,
35+
object: this.gameObject,
36+
hideWhenTrackingIsLost: true,
37+
imageDoesNotMove: false,
38+
});
39+
40+
tracker.trackedImages.push(model);
41+
}
42+
}
43+
```
44+
45+
---
46+
47+
## Reacting to Tracking on the Tracked Object
48+
49+
Normally, the simplest approach is to add a component directly to the object assigned to the image tracker. You can use `onEnterXR` and `onLeaveXR` to react to the XR session starting and ending:
50+
51+
```ts
52+
import { Behaviour, NeedleXREventArgs } from "@needle-tools/engine";
53+
54+
// Add this component to the object assigned to WebXRImageTrackingModel
55+
export class OnImageFound extends Behaviour {
56+
onEnterXR(args: NeedleXREventArgs) {
57+
console.log("XR session started");
58+
}
59+
60+
onLeaveXR(args: NeedleXREventArgs) {
61+
console.log("XR session ended");
62+
}
63+
}
64+
```
65+
66+
---
67+
68+
## Listening to the Image Tracking Event
69+
70+
The `WebXRImageTracking` component dispatches an `image-tracking` event every frame when images are being tracked. The event's `detail` contains an array of `WebXRTrackedImage` objects.
71+
72+
```ts
73+
import { Behaviour, WebXRImageTracking, WebXRTrackedImage } from "@needle-tools/engine";
74+
75+
export class ImageTrackingHandler extends Behaviour {
76+
onEnable() {
77+
// Get the WebXRImageTracking component in the scene
78+
const tracker = this.gameObject.getComponent(WebXRImageTracking);
79+
tracker?.addEventListener("image-tracking", this.onImageTracking);
80+
}
81+
82+
onDisable() {
83+
const tracker = this.gameObject.getComponent(WebXRImageTracking);
84+
tracker?.removeEventListener("image-tracking", this.onImageTracking);
85+
}
86+
87+
private onImageTracking = (event: CustomEvent) => {
88+
const trackedImages: WebXRTrackedImage[] = event.detail;
89+
for (const img of trackedImages) {
90+
console.log(img.url, img.state);
91+
}
92+
}
93+
}
94+
```
95+
96+
---
97+
98+
## Accessing the Assigned Object
99+
100+
Each `WebXRTrackedImage` gives you access to the 3D object assigned in the `WebXRImageTrackingModel` configuration:
101+
102+
```ts
103+
private onImageTracking = (event: CustomEvent) => {
104+
const trackedImages: WebXRTrackedImage[] = event.detail;
105+
for (const img of trackedImages) {
106+
// Access the assigned AssetReference via the model configuration
107+
const obj = img.model.object;
108+
109+
// From Needle Engine 5.1+ you can also use the shorthand:
110+
// const obj = img.trackedModel;
111+
}
112+
}
113+
```
114+
115+
| Property | Type | Description |
116+
| --- | --- | --- |
117+
| `img.model` | `WebXRImageTrackingModel` | Full configuration including `object`, `widthInMeters`, `hideWhenTrackingIsLost`, etc. |
118+
| `img.model.object` | `AssetReference` | The 3D object assigned to this tracked image marker |
119+
| `img.trackedModel` | `AssetReference` | Shorthand for `img.model.object` *(available from Needle Engine 5.1+)* |
120+
121+
---
122+
123+
## Reading Position and Rotation
124+
125+
Get the local position and rotation of the tracked image (relative to the XR rig):
126+
127+
```ts
128+
import { Vector3, Quaternion } from "three";
129+
130+
const position = new Vector3();
131+
const rotation = new Quaternion();
132+
133+
private onImageTracking = (event: CustomEvent) => {
134+
const trackedImages: WebXRTrackedImage[] = event.detail;
135+
for (const img of trackedImages) {
136+
img.getPosition(position);
137+
img.getQuaternion(rotation);
138+
console.log("Position:", position, "Rotation:", rotation);
139+
}
140+
}
141+
```
142+
143+
---
144+
145+
## Checking Tracking State
146+
147+
Each tracked image reports its current state:
148+
149+
```ts
150+
for (const img of trackedImages) {
151+
if (img.state === "tracked") {
152+
// Image is actively being tracked by the device
153+
} else if (img.state === "emulated") {
154+
// Tracking is emulated (less accurate)
155+
}
156+
}
157+
```
158+
159+
| State | Description |
160+
| --- | --- |
161+
| `"tracked"` | Image is actively tracked by the device camera |
162+
| `"emulated"` | Tracking is estimated/emulated (less accurate) |
163+
164+
---
165+
166+
## Next Steps
167+
168+
**Learn more:**
169+
- [WebXR Image Tracking](/docs/how-to-guides/xr/image-tracking) - Setup, platform support, and best practices
170+
- [XR Development](/docs/how-to-guides/xr/) - Full XR capabilities
171+
- [iOS WebXR App Clip](/docs/how-to-guides/xr/ios-webxr-app-clip) - Native iOS support
172+
173+
**Reference:**
174+
- [Scripting Examples](/docs/reference/scripting-examples) - More code patterns
175+
- [Component Lifecycle](/docs/how-to-guides/scripting/use-lifecycle-hooks) - awake, start, update methods

‎documentation/how-to-guides/xr/image-tracking.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,7 @@ Image tracking opens up many creative possibilities:
206206
- [iOS WebXR Guide](/docs/how-to-guides/xr/ios-webxr-app-clip) - Enable native iOS support
207207

208208
**Learn More:**
209+
- [Scripting Image Tracking](/docs/how-to-guides/scripting/image-tracking) - Access tracked images and objects from TypeScript
209210
- [XR Documentation](/docs/how-to-guides/xr/) - Full XR capabilities
210211
- [Everywhere Actions](/docs/how-to-guides/everywhere-actions/) - Alternative AR approaches
211212
- [Deployment Guide](/docs/how-to-guides/deployment/) - Publish your AR experience

‎documentation/reference/faq.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -770,6 +770,23 @@ If you see "WebXR not found" or simply can't enter AR, check the following:
770770

771771
For AR on iOS, Needle Engine supports WebXR via [App Clips (Needle Go)](/docs/explanation/core-concepts/ios-webxr-app-clip). See also the [Platform Support](#does-it-work-on-ios) section.
772772

773+
## How do I access the tracked object from an image tracking event?
774+
775+
Each `WebXRTrackedImage` received from the `image-tracking` event gives you access to the assigned 3D object via `img.model.object`:
776+
777+
```ts
778+
tracker.addEventListener("image-tracking", (event: CustomEvent) => {
779+
const trackedImages: WebXRTrackedImage[] = event.detail;
780+
for (const img of trackedImages) {
781+
// Access the assigned AssetReference
782+
const obj = img.model.object;
783+
// From Needle Engine 5.1+ you can also use: img.trackedModel
784+
}
785+
});
786+
```
787+
788+
See the [Scripting Image Tracking](/docs/how-to-guides/scripting/image-tracking) guide for more details.
789+
773790
# AI
774791

775792
## How do I use AI to help me build with Needle Engine?

0 commit comments

Comments
 (0)