Skip to content

Commit 22d72a5

Browse files
reineckereinecke
committed
Added/updated docs and version map
Signed-off-by: Eric Reinecke <ereinecke@netflix.com>
1 parent b1c3792 commit 22d72a5

5 files changed

Lines changed: 439 additions & 9 deletions

File tree

‎docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ Tutorials
4242
tutorials/feature-matrix
4343
tutorials/otio-timeline-structure
4444
tutorials/time-ranges
45+
tutorials/channel-mapping
4546
tutorials/otio-filebundles
4647
tutorials/write-an-adapter
4748
tutorials/write-a-media-linker

‎docs/tutorials/channel-mapping.md‎

Lines changed: 373 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,373 @@
1+
# Stream and Channel Mapping
2+
3+
## Overview
4+
5+
OTIO's stream and channel mapping support allows media references to describe the individual streams they contain (video eyes, audio channels, etc.) and for timeline items to select or remix those streams via effects. The system has three layers: **addressing** a stream within a container, **describing** a stream's semantic role, and **selecting or mixing** streams on a clip.
6+
7+
---
8+
9+
## Layer 1: Addressing a Stream Within a Container
10+
11+
A `StreamAddress` identifies where a specific stream lives inside a media file. The appropriate `StreamAddress` subclass depends on how the container format identifies its streams. The `StreamAddress` subclass should be chosen based on how closely it matches the container's native identification scheme.
12+
13+
| Class | Use when |
14+
|---|---|
15+
| `IndexStreamAddress` | The container uses a single integer index per stream (e.g. WAV channel index, ffmpeg stream index) |
16+
| `StreamChannelIndexStreamAddress` | The container organises media into tracks, each with one or more channels (e.g. MP4/MOV, MXF) |
17+
| `StringStreamAddress` | The container uses string identifiers (e.g. codec name, channel label, USD scene path) |
18+
19+
### Format reference
20+
21+
Below is a chart of well-known media formats and the recommended `StreamAddress` subclass usage for each.
22+
23+
| Format | StreamAddress class | Notes |
24+
|---|---|--------------------------------------------------------------------------------------------|
25+
| WAV | `IndexStreamAddress` | |
26+
| AIFF | `IndexStreamAddress` | |
27+
| MOV / MP4 | `StreamChannelIndexStreamAddress` | When there is only one channel in a track (e.g. single-view video), use `channel_index=0`. |
28+
| USD | `StringStreamAddress` | Use the USD path. |
29+
| PNG, JPEG, TIFF, GIF | *(single-image formats — no stream address needed)* | |
30+
31+
The above is not an exhaustive list. If you are working with a format not listed here, consider how its streams are identified and choose the `StreamAddress` subclass that best matches that scheme.
32+
33+
### StreamAddress
34+
35+
`StreamAddress` is the base class for all stream address types. You will not instantiate it directly; use one of the concrete subclasses below.
36+
37+
### IndexStreamAddress
38+
39+
Addresses a stream by a single integer index. Use for formats where each stream is identified by a flat index (e.g. a WAV file's channel index).
40+
41+
```python
42+
# Stream 2 of a WAV file
43+
addr = otio.schema.IndexStreamAddress(index=2)
44+
```
45+
46+
### StreamChannelIndexStreamAddress
47+
48+
Addresses a stream by a track index and a channel index within that track. Use for container formats that organise media into named or numbered tracks, each potentially containing multiple channels (MOV, MP4).
49+
50+
```python
51+
# Stream 1, channel 2 of an MP4
52+
addr = otio.schema.StreamChannelIndexStreamAddress(stream_index=1, channel_index=2)
53+
```
54+
55+
When a track contains only one channel (e.g. a single-view video track), use `channel_index=0`.
56+
57+
### StringStreamAddress
58+
59+
Addresses a stream by a string label or path. Use for formats where streams are identified by name rather than numeric index (e.g. a USD scene path, a codec channel label).
60+
61+
```python
62+
# A camera in a USD scene
63+
addr = otio.schema.StringStreamAddress(address="/World/set/cam_left")
64+
```
65+
66+
---
67+
68+
## Layer 2: Describing a Stream's Semantic Role
69+
70+
Once you can locate a stream, the next layer describes what that stream *is* — its media kind, human-readable name, and semantic role. This information lives on the media reference and is keyed by well-known identifier strings.
71+
72+
### StreamInfo
73+
74+
`StreamInfo` describes a single media stream within a source. It is the smallest unit of temporal media — one eye's video, one audio channel, one camera view.
75+
76+
**Fields:**
77+
78+
- **`name`** — Human-readable label for this stream. May be any descriptive string (e.g. `"Alice lavalier (boom backup)"`, `"left eye"`). Not a semantic identifier.
79+
- **`address`** — A `StreamAddress` subclass locating the stream in its container.
80+
- **`kind`** — The type of media. Favor constants from `otio.schema.TrackKind` wherever appropriate.
81+
- **`metadata`** — Arbitrary metadata dictionary for additional properties (e.g. ITU-R channel labels).
82+
83+
```python
84+
stream = otio.schema.StreamInfo(
85+
name="Low Frequency Effects",
86+
address=otio.schema.StreamChannelIndexStreamAddress(stream_index=1, channel_index=3),
87+
kind=otio.schema.TrackKind.Audio,
88+
metadata={
89+
"ITU_label": "LFE",
90+
},
91+
)
92+
```
93+
94+
### StreamInfo.Identifier
95+
96+
`StreamInfo.Identifier` provides well-known strings that identify the **semantic role** of a stream — what presentation device it targets. These constants are the canonical keys for `available_streams`.
97+
98+
`otio.schema.StreamIdentifier` is an alias for `StreamInfo.Identifier`.
99+
100+
#### Visual streams
101+
102+
| Constant | Meaning |
103+
|---|---|
104+
| `monocular` | A traditional 2D video stream (single-view) |
105+
| `left_eye` | Left eye in a stereo 3D pair |
106+
| `right_eye` | Right eye in a stereo 3D pair |
107+
108+
#### Simple aural streams
109+
110+
| Constant | Meaning |
111+
|---|---|
112+
| `monaural` | Single mono audio channel |
113+
| `stereo_left` | Left channel of a stereo pair |
114+
| `stereo_right` | Right channel of a stereo pair |
115+
116+
> **Note:** Stereo and surround channels are kept separate because presenting the left/right channels of a surround mix as stereo is generally not a desirable result.
117+
118+
#### Surround aural streams (loosely based on ITU-R BS.2051-3)
119+
120+
| Constant | ITU label |
121+
|---|---|
122+
| `surround_left_front` | L |
123+
| `surround_right_front` | R |
124+
| `surround_center_front` | C |
125+
| `surround_low_frequency_effects` | LFE |
126+
| `surround_left_rear` | Ls |
127+
| `surround_right_rear` | Rs |
128+
129+
### MediaReference.available_streams
130+
131+
`available_streams()` returns a dict keyed by stream identifier string mapping to `StreamInfo` objects. Use `set_available_streams()` to populate it.
132+
133+
#### Key conventions
134+
135+
1. **Presentation-ready streams** SHOULD use an `Identifier` constant directly as the key. This signals to consumers which stream serves a well-known role.
136+
137+
```python
138+
ref.set_available_streams({
139+
StreamInfo.Identifier.monocular: otio.schema.StreamInfo(
140+
name="main camera",
141+
address=otio.schema.IndexStreamAddress(index=0),
142+
kind=otio.schema.TrackKind.Video,
143+
),
144+
StreamInfo.Identifier.stereo_left: otio.schema.StreamInfo(
145+
name="mix L",
146+
address=otio.schema.IndexStreamAddress(index=1),
147+
kind=otio.schema.TrackKind.Audio,
148+
),
149+
})
150+
```
151+
152+
2. **Additional streams** SHOULD prefix an `Identifier` value to form a unique key within the media reference.
153+
154+
```python
155+
# A music stem alongside the audio mix
156+
"music_stereo_left": otio.schema.StreamInfo(...)
157+
```
158+
159+
3. **Arbitrary streams** MAY use any unique string when no well-known identifier applies — for example, spatial audio object IDs, or production audio tracks identified by mic placement.
160+
161+
```python
162+
"alice_lavalier": otio.schema.StreamInfo(...)
163+
```
164+
165+
When generating these keys, try to avoid names that might collide with future `Identifier` constants unless you think your use case is in line with the precedent in that constant. In these cases, consider submitting a new constant to OTIO to include.
166+
167+
---
168+
169+
## Layer 3: Selecting or Mixing Streams on a Clip
170+
171+
Stream effects are `Effect` subclasses that attach to a clip's `effects` list and transform which streams are visible downstream and under what names. They operate on the stream identifiers declared in `available_streams`.
172+
173+
### StreamSelector
174+
175+
`StreamSelector` selects a subset of named streams from a clip and exposes them downstream with the **same names**. Use it to filter away streams you do not need without renaming them.
176+
177+
- **`output_streams`** — List of stream identifier strings to pass through. Values SHOULD be `Identifier` constants.
178+
179+
```python
180+
# Select only the left eye from a stereo 3D clip
181+
clip.effects.append(
182+
otio.schema.StreamSelector(
183+
output_streams=[StreamInfo.Identifier.left_eye]
184+
)
185+
)
186+
187+
# Select a stereo pair from an 8 channel (5.1 + Stereo Mixdown) source
188+
clip.effects.append(
189+
otio.schema.StreamSelector(
190+
output_streams=[
191+
StreamInfo.Identifier.stereo_left,
192+
StreamInfo.Identifier.stereo_right,
193+
]
194+
)
195+
)
196+
```
197+
198+
### StreamMapper
199+
200+
`StreamMapper` renames stream identifiers as they pass through a clip. Each entry maps an **output** stream name (as it will appear downstream) to an **input** stream name (the key in the upstream `available_streams` map).
201+
202+
Use `StreamMapper` when a source uses one identifier but a downstream consumer expects a different one.
203+
204+
- **`stream_map`** — `{output_name: input_name}` dict. Keys and values SHOULD be `Identifier` constants where applicable.
205+
206+
```python
207+
# Expose the left eye of a stereo source as the conventional monocular stream
208+
clip.effects.append(
209+
otio.schema.StreamMapper(
210+
stream_map={
211+
StreamInfo.Identifier.monocular: StreamInfo.Identifier.left_eye
212+
}
213+
)
214+
)
215+
```
216+
217+
After this effect, downstream consumers that look up `StreamInfo.Identifier.monocular` would use the content addressed by the upstream `StreamInfo.Identifier.left_eye` entry.
218+
219+
### AudioMixMatrix
220+
221+
`AudioMixMatrix` mixes audio streams using a coefficient matrix. The structure is:
222+
223+
```
224+
{ output_stream_name: { input_stream_name: coefficient, ... }, ... }
225+
```
226+
227+
- **Output keys** SHOULD use `Identifier` constants where applicable. They correspond to the stream names that will appear in the downstream `available_streams` map after mixing.
228+
- **Input keys** SHOULD match keys in the upstream `MediaReference.available_streams`.
229+
230+
```python
231+
# Downmix 5.1 surround to Lo/Ro stereo
232+
clip.effects.append(
233+
otio.schema.AudioMixMatrix(
234+
name="5.1_to_stereo",
235+
matrix={
236+
StreamInfo.Identifier.stereo_left: {
237+
StreamInfo.Identifier.surround_left_front: 1.0,
238+
StreamInfo.Identifier.surround_center_front: 0.707,
239+
StreamInfo.Identifier.surround_left_rear: 0.707,
240+
},
241+
StreamInfo.Identifier.stereo_right: {
242+
StreamInfo.Identifier.surround_right_front: 1.0,
243+
StreamInfo.Identifier.surround_center_front: 0.707,
244+
StreamInfo.Identifier.surround_right_rear: 0.707,
245+
},
246+
},
247+
)
248+
)
249+
```
250+
251+
### Comparison of Stream Effects
252+
253+
| Effect | Input | Output | Use when |
254+
|---|---|---|---|
255+
| `StreamSelector` | List of stream names to keep | Same streams, same names | You want to filter down to a subset of streams without renaming them |
256+
| `StreamMapper` | `{output_name: input_name}` mapping | Renamed streams | A source uses one identifier but downstream expects a different name |
257+
| `AudioMixMatrix` | `{output_name: {input_name: coefficient}}` matrix | New mixed streams | You need to combine or downmix audio channels with weighting |
258+
259+
These effects can be combined: for example, use `StreamSelector` to isolate a stereo pair from a source with 8-channel audio, then `AudioMixMatrix` on a downstream clip to downmix it further.
260+
261+
---
262+
263+
## Putting It Together
264+
265+
### Stereo 3D video clip + 5.1 audio downmix to stereo
266+
267+
This example models a more complex scenario: a single MOV file
268+
containing stereo 3D video in stream 0 and a 5.1 surround audio mix in
269+
stream 1. Two clips are built from the same reference — one that remaps the
270+
left eye to monocular, and one that downmixes the surround channels to stereo.
271+
272+
```python
273+
import opentimelineio as otio
274+
from opentimelineio.schema import StreamInfo
275+
276+
# Single MOV reference.
277+
# Stream 0: stereo 3D video (both eyes packed into one track, e.g. MV-HEVC or scalable AV1).
278+
# Stream 1: 5.1 audio, six channels addressed by channel index.
279+
ref = otio.schema.ExternalReference(target_url="/show/shot.mov")
280+
ref.set_available_streams({
281+
# Video — stream 0. The Identifier distinguishes which eye to extract.
282+
StreamInfo.Identifier.left_eye: otio.schema.StreamInfo(
283+
name="left eye",
284+
address=otio.schema.StreamChannelIndexStreamAddress(
285+
stream_index=0, channel_index=0
286+
),
287+
kind=otio.schema.TrackKind.Video,
288+
),
289+
StreamInfo.Identifier.right_eye: otio.schema.StreamInfo(
290+
name="right eye",
291+
address=otio.schema.StreamChannelIndexStreamAddress(
292+
stream_index=0, channel_index=1
293+
),
294+
kind=otio.schema.TrackKind.Video,
295+
),
296+
# Audio — stream 1, one entry per surround channel.
297+
StreamInfo.Identifier.surround_left_front: otio.schema.StreamInfo(
298+
name="L",
299+
address=otio.schema.StreamChannelIndexStreamAddress(
300+
stream_index=1, channel_index=0
301+
),
302+
kind=otio.schema.TrackKind.Audio,
303+
),
304+
StreamInfo.Identifier.surround_right_front: otio.schema.StreamInfo(
305+
name="R",
306+
address=otio.schema.StreamChannelIndexStreamAddress(
307+
stream_index=1, channel_index=1
308+
),
309+
kind=otio.schema.TrackKind.Audio,
310+
),
311+
StreamInfo.Identifier.surround_center_front: otio.schema.StreamInfo(
312+
name="C",
313+
address=otio.schema.StreamChannelIndexStreamAddress(
314+
stream_index=1, channel_index=2
315+
),
316+
kind=otio.schema.TrackKind.Audio,
317+
),
318+
StreamInfo.Identifier.surround_low_frequency_effects: otio.schema.StreamInfo(
319+
name="LFE",
320+
address=otio.schema.StreamChannelIndexStreamAddress(
321+
stream_index=1, channel_index=3
322+
),
323+
kind=otio.schema.TrackKind.Audio,
324+
),
325+
StreamInfo.Identifier.surround_left_rear: otio.schema.StreamInfo(
326+
name="Ls",
327+
address=otio.schema.StreamChannelIndexStreamAddress(
328+
stream_index=1, channel_index=4
329+
),
330+
kind=otio.schema.TrackKind.Audio,
331+
),
332+
StreamInfo.Identifier.surround_right_rear: otio.schema.StreamInfo(
333+
name="Rs",
334+
address=otio.schema.StreamChannelIndexStreamAddress(
335+
stream_index=1, channel_index=5
336+
),
337+
kind=otio.schema.TrackKind.Audio,
338+
),
339+
})
340+
341+
# Video clip — remap left_eye to monocular for downstream consumers.
342+
video_clip = otio.schema.Clip(name="shot_001_video", media_reference=ref)
343+
video_clip.effects.append(
344+
otio.schema.StreamMapper(
345+
name="expose_left_eye_as_monocular",
346+
stream_map={
347+
StreamInfo.Identifier.monocular: StreamInfo.Identifier.left_eye,
348+
}
349+
)
350+
)
351+
352+
# Audio clip — downmix 5.1 surround to Lo/Ro stereo.
353+
audio_clip = otio.schema.Clip(name="shot_001_audio", media_reference=ref)
354+
audio_clip.effects.append(
355+
otio.schema.AudioMixMatrix(
356+
name="5.1_to_stereo",
357+
matrix={
358+
StreamInfo.Identifier.stereo_left: {
359+
StreamInfo.Identifier.surround_left_front: 1.0,
360+
StreamInfo.Identifier.surround_center_front: 0.707,
361+
StreamInfo.Identifier.surround_left_rear: 0.707,
362+
StreamInfo.Identifier.surround_low_frequency_effects: 0.707,
363+
},
364+
StreamInfo.Identifier.stereo_right: {
365+
StreamInfo.Identifier.surround_right_front: 1.0,
366+
StreamInfo.Identifier.surround_center_front: 0.707,
367+
StreamInfo.Identifier.surround_right_rear: 0.707,
368+
StreamInfo.Identifier.surround_low_frequency_effects: 0.707,
369+
},
370+
},
371+
)
372+
)
373+
```

0 commit comments

Comments
 (0)