Codec profile optimisation for Aether (AVPlayer)

[DrHurt]

Summary of codecs, containers, protocols supported by Apple's AVPlayer

**Video Containers:**

MP4 (including via HLS), MOV, M4V codecs supported:
Video: H264/AVC, H265/HEVC, AV1
Audio: AAC, AC3, EAC3, FLAC, ALAC
*direct playback - codecs restricted by container so most media in above containers will direct play.

MPEG-TS:
h264 direct play (used for IPTV apps, live streams)
HEVC require MP4 HLS

MKV & WEBM
HLS pathway for codecs supported via HLS

AVI & others:
contain old, uncommon codecs & mostly unsupported
need software support

**Video codec details:**
HEVC:
Main & Main 10
SDR, HLG, HDR10(+), DV 5, 8.1, 8.4

AVC:
Main & High
SDR, HDR10

AV1:
Main and High
SDR, HDR10(+), DV 10, 10.1, 10.4

VP9: (not worth it)
SDR, HDR10
Only direct play in webm containers on Safari (YouTube) so not worth it
Not supported via HLS, but supported via videotoolbox HW decode

MPEG4: (not worth it)
only SP & limited to 640x480

**Audio (without video) containers & codecs:**
Compressed: mp3 (mpeg-1 III), m4a (aac, alac),
Lossless: wav, aiff, flac -- note container (file extension) & codec have same name.

All above will audio containers will direct play as codecs are limited by container and Apple supports them all.

Opus & Vorbis audio will only direct play in CAF container (theoretical, not used in practice). Opus also supported in WEBM videos.

In my testing, both are NOT supported in HLS so if found in other containers (with or without video), will need to fall back to HLS FLAC bridge

[superuser404notfound]

Picking up your comment from #2 here, since this is the right thread for it.

You were right about AV1, and the way I learned it was the obvious way: an AV1 file went black on Apple TV mid-test. The comment in VTCapabilityProbe.swift that read "Apple ships AV1 via integrated dav1d on tvOS 17+ / iOS 17+ / macOS 14+" over-extended the WWDC 2023 announcement. Apple ships dav1d on iOS and macOS, not on tvOS. VTIsHardwareDecodeSupported(av01) returns false on every Apple TV chip in service, and VTRegisterSupplementalVideoDecoderIfAvailable doesn't light up a SW path on tvOS either. The probe is now gated strictly on VTIsHardwareDecodeSupported (1893327).

That alone would have left AV1 sources without a playback path on Apple TV, which conflicts with the engine's reason for existing (no server-side transcode). So the ffmpeg + dav1d layer is back, scoped to AV1, and routed via codec dispatch at load(url:) rather than via a separately-installable extension:

                                                  ┌─► HEVC / H.264 / VP9 → HLSVideoEngine → loopback HLS → AVPlayer
load(url:) ── probe → codec_id → dispatch
                                                  └─► AV1 → SoftwareVideoDecoder (dav1d) → AVSampleBufferDisplayLayer
                                                            AudioDecoder → AudioOutput → AVSampleBufferRenderSynchronizer

Both pipelines coexist in the engine and publish the same @Published state surface (with PlaybackBackend.native / .software so hosts can tell which served a session). AV1 sources never carry DV in practice (DV is HEVC-profile-driven) and effectively never carry Atmos (mastering runs in HEVC), so losing those capabilities on the SW pipeline is a theoretical limitation rather than a real one.

I went for in-engine dispatch over the extension model for two reasons. One binary, one published state surface, no plugin lifecycle to design; hosts see the same state object whichever pipeline ran. And the SW codec footprint we want today is small (AV1 only). If it grew to cover several codecs AVPlayer can't take, the cost-benefit would flip toward your extension boundary, and that would be the right architecture at that point.

On VP9: agreed it's effectively a YouTube/WebM codec in practice. AVPlayer does decode VP9 on A12+ chips after VTRegisterSupplementalVideoDecoderIfAvailable registration, which covers Apple TV 4K Gen 2 and up, so the native path handles VP9 without engine intervention on modern silicon. Pre-A12 Apple TV (Gen 1) loses it; accepting that, the hardware is fading.

Full state of the dual-pipeline architecture lives in the README and the 1.0.0 release notes. The 1.0.0 tag was force-moved forward to include this work; the prior state had a real AV1 hole that put it short of a complete 1.0.

Thanks for catching this. The probe-was-too-optimistic assumption would have hit users eventually; your note got there first.

[superuser404notfound]

Correction to my earlier comment in this thread: I claimed VP9 would direct-play via the native AVPlayer path because VTRegisterSupplementalVideoDecoderIfAvailable + VTIsHardwareDecodeSupported(vp09) returns true on A12+ chips. That's only half right — VT can HW-decode VP9, but AVPlayer's HLS-fMP4 pipeline silently rejects the vp09 CODECS attribute regardless. You were correct in the original writeup.

Verified empirically via aetherctl on macOS 26 against a libvpx-vp9 + libopus WebM source: AVPlayer GETs master.m3u8 + media.m3u8, parses, then stops fetching. AVPlayerItem.status never leaves .unknown. No error surfaced. Same silent-rejection behavior as you described.

Engine now routes VP9 to the SW pipeline alongside AV1, via libavcodec's native VP9 decoder + AVSampleBufferDisplayLayer. Shipped in de86335 along with the Opus → FLAC bridge move (Opus has the same HLS rejection behavior per your audio table).

A related correction landed in 9ce8908: the AV1 dispatch is now per-platform. Devices with HW AV1 (M3+ Mac, iPhone 15 Pro+, future Apple TV chips with HW AV1) route AV1 sources to the native AVPlayer path so Atmos / DV / system HDR engage. Everything else routes to the SW dav1d pipeline. I also wired the AV1+DV codec-tag handling (P10.0, P10.1 → dav1.10.LL; P10.4 → av01.0.LL.10 primary + SUPPLEMENTAL dav1.10.LL/db4h), though I couldn't find publicly available AV1+DV samples to verify against. If you happen to know a working test source, I'd love a pointer.

The vp9Available / av1Available probes stay in VTCapabilityProbe as informational + future-routing hooks. If Apple ever opens the HLS-fMP4 pipeline to either codec (or ships HW AV1 on Apple TV), the gates light up automatically.

[DrHurt]

Massive progress here! Can’t wait to try it out.

One random comment I have is that when playing dvh1 on non-DV display, you tag it as hvc1 but I feel that it should remain as dvh1 (true to source) so that the tone mapper knows exactly what metadata the file contains and works from there.

On HDR10 display with DV master playlist, AVPlayer will tone map to HDR10 automatically

On non-HDR displays with the media playlist directly, it’ll force tone map to SDR.

Probably a non-issue but thinking out loud.

[DrHurt]

Regarding subtitles, and again thinking out loud, you could ask ffmpeg to convert text subtitles to webvtt on the fly while remuxing (-c:s webvtt) and then AVPlayer should be able to render them natively.

This could be a user option: use native subtitles.

Personally, I’ll use your custom renderer as that would allow more control (eventually) like position, font, colour, outline etc.

But the native text subtitles renderer gives more platform integration (consistency between apps and global tvOS settings will apply to them) and will potentially be more efficient as they’ll be extracted & converted by the same instance of ffmpeg during remux.

You could possibly even use the webvtt fragments with your custom renderer! (Which I suspect will be more efficient that the current implementation)

[superuser404notfound]

Update on both threads.

DV codec tag (dvh1 on non-DV displays): shipped on main as of a9eef1d, with LoadOptions.keepDvh1TagWithoutDV default flipped to true. Verified end-to-end against the Dolby browser test kit (24fps HD bundle) on macOS Tahoe / Apple Silicon, HDR-eligible non-DV display:

Profile	Mode A (legacy hvc1 strip)	Mode B (keep dvh1)	Codecs emitted
P5 (DV-only, IPT-PQ-c2)	plays, but as raw PQ HDR10 with the wrong color volume (no DV processing)	plays cleanly	dvh1.05.01,ec-3
P8.1 (HDR10-base)	plays, readyToPlay ~2.6s	plays, readyToPlay ~0.4s	dvh1.08.01,ec-3
P8.4 (HLG-base)	plays	plays	hvc1.2.4.L93.b0,ec-3 + supplemental dvh1.08.LL/db4h

P5 is what swung the call. Stripping the tag wasn't just a tone-mapping quality regression, it was a latent correctness bug (P5 has no HDR10 fallback in the bitstream, so AVPlayer was rendering raw PQ with the wrong color volume). Master playlist with full DV signalling also gets AVPlayer to readyToPlay ~6x faster on P8.1 because the codec hint comes upfront. The legacy hvc1 downgrade is now only reachable via aetherctl --no-dv for debug; the host path keeps the tag unconditionally.

Companion changes that landed in the same series: published videoFormat clamps to actual panel capabilities (a DV source on an HDR10-only panel now publishes .hdr10, not .dolbyVision), and HDR10+ T.35 detection is wired in both backends so the badge upgrades .hdr10 -> .hdr10Plus mid-stream when dynamic metadata appears. Sodalite picks all of this up at a9eef1d.

Thinking out loud paid off; thanks for pushing on it.

Subtitle WebVTT conversion: parking it for now, with the door open for later as an opt-in. Three things weighing against doing it as the default:

1. ASS / SubStation-Alpha fidelity loss. -c:s webvtt flattens \an8 positioning, karaoke timings, font and colour overrides, and typesetting to plain text. Fansub-heavy / anime libraries would feel that immediately, and there isn't really a way back inside WebVTT.
2. PGS and DVB bitmap subs stay on the custom renderer regardless (native HLS subs are text-only). So the WebVTT path would land alongside the existing pipeline rather than replacing it, doubling the surface area.
3. The HLS rendition-group plumbing isn't trivial: a separate WebVTT segmenter, #EXT-X-MEDIA:TYPE=SUBTITLES in the master, SUBTITLES="group" on every #EXT-X-STREAM-INF, segment alignment with video. Meaningful engine work for what's mostly a UX polish gain.

The tvOS-accessibility-settings angle (system text size, contrast, dyslexic font) is the genuine win and I want to come back to it. If user feedback later surfaces it as a real ask, it goes in as an opt-in "Use native subtitle renderer (loses advanced typesetting)" with the custom path as default. Until then, keeping the custom path keeps ASS fidelity for the users who care most.

Both threads have been useful, keep them coming.

[DrHurt]

I think WebVTT subtitle can be side loaded directly bypassing HLS but like I said I prefer the custom engine anyway because Apple's native subtitles are not customisable enough. Might be helpful adding this to your notes in case you want to revisit in the future.

ANYWAY, tested 0.6 and it is truly excellent for a first release. I haven't tested everything yet but those are the results of the few files I tested.

What works:
MP4, MKV, WebM
HEVC SDR, HDR10+, DV8.1
AVC SDR
AV1 HDR10
SRT, ASS, VOB/DVD subs
AAC, DTS audio

Doesn't work correctly:
DV 8.4: TV switches to HDR mode and sodalite show says "HLG"
DV 5: says playback stopped
VP9 WebM fast forwarded itself and froze (but that might be a bad file as it's video only, no audio track)

Non-codec issues:
Match frame rate only engages when changing dynamic range. So if watching SDR 24fps, TV stays at SDR 60Hz. If watch HDR 24fps, TV switches to HDR 24Hz. Should be a very straight forward fix splitting the logic.

Seeking works but feels a bit temperamental but I have broken remote so I'll test again with the other remote.

I have a few feature requests / bug reports for Sodalite so I'll open a few issues on the Sodalite page if that's okay.

Ps. I see you’re going with AVPlayerViewController which is the best option in my opinion due to integration with the wider ecosystem! (Reduce loud sounds, dialogue enhancement, HomePods etc) & UI consistency!

You may find this link useful:
https://developer.apple.com/documentation/avkit/customizing-the-tvos-playback-experience

[DrHurt]

More good news:

• Dolby Atmos is working and the TV displays the Atmos logo
• DV P5 works when I manually force the Apple TV into Dolby Vision mode so problem is related to the mode switching code.
• I suspect DV 8.4 is also simply a mode switch issue
• TrueHD 7.1 also working (outs as PCM 7.1 & channels are taped correctly)
• H264 Hi10 works (anime)

So the fixes currently needed are just the frame rate & dynamic range switch code.

I wish I could buy you a beer man :D (can't see a donate button)

[DrHurt]

More data points:

TV forced to HDR10 mode: (match off)
DV5 plays correctly (AVPlayer tone maps to HDR10)
DV8.1 plays correctly (presumably decoding the HDR10 base)
DV8.4 plays correctly (presumably decoding the HLG base)

presumabley all above is using master playlist? DV to HDR10 tone mapping is working as expected.

TV forced to SDR mode: (match off)
DV5, 8.1, HDR10+ all fail to play
DV 8.4 plays

are we not using media playlist? DV & HDR tone mapping to SDR is not working.

PS. I do agree with DV8.1 using codecs = hvc1 + supplemental codecs = dvh1 for consistency now

[DrHurt]

Did some more testing to try to address the codec tags confusion.
Did my best to stick to Dolby reference & Apple HLS specs.

DV8.1 MKV manually remuxed to HLS on my MacBook using ffmpeg to see what exactly tvOS 26 needs to work correctly.

I can summarise it into THREE points.

1- ffmpeg -vtag:hvc1: init.mp4 tagged as hvc1 (dvh1 works too, but Dolby reference files use dvh1 for P5 only)

2- Master playlist: #EXT-X-STREAM-INF:BANDWIDTH=6441150,AVERAGE-BANDWIDTH=6441150,VIDEO-RANGE=PQ,CODECS="hvc1.2.4.L120.B0,mp4a.40.2",SUPPLEMENTAL-CODECS="dvh1.08.03/db1p",RESOLUTION=1920x1034,FRAME-RATE=23.976

3- PreferredDisplayCriteria needs to be set dovi otherwise avplayer stalls (may need to be set independently).

With the above combination, DV 8.1 on DV-capable TV plays correctly,
DV 8.1 on HDR display correctly plays the HDR10 base layer (tested by disabling TV DV mode, forcing HDR10 only - it worked)

DV 8.4 should be the same but with different supplemental-codecs

DV 5 init.mp4 MUST be tagged dvh1, master MUST use CODEC=dvh1 (no supplemental), and display criteria MUST be forced into dovi (avplayer will tone map automatically is display is HDR only)

Test this on your HDR10-only setup and see if 8.1 & 8.4 fallback and 5 tone maps. It should as per my testing. Biggest pitfall is making sure PreferredDisplayCriteria is set correctly. Otherwise, we run into unexpected problems which we were incorrectly attributing to "codec".

I'm just trying to keep Aether specs compliant and let Apple's frameworks do their stuff without Aether needing workarounds for different display types.

[DrHurt]

Thinking out loud again.

Curiously, the Jellyfin server uses tag:v dvh1 for DV 8.1

My question therefore is where PreferredDisplayCriteria gets its cue from: init.mp4 vs master playlist.

And whether you need to manually override value handed over to PreferredDisplayCriteria? (to allow using specs compliant media segments & master manifests)