Native subtitle rendition (AVMediaSelection / WebVTT) so subtitles survive PiP, AirPlay & external display

[edde746]

What should the engine do?

Optionally emit the active subtitle as a native HLS subtitle rendition (#EXT-X-MEDIA:TYPE=SUBTITLES → WebVTT, served by HLSLocalServer from the already-decoded cues) and expose it through the legible AVMediaSelectionGroup, so AVPlayer renders the subtitle itself. That is the only way subtitles can survive Picture-in-Picture, AirPlay, and external-display playback, where a host-drawn overlay never appears.

Motivating media or use case

Scenario: a tvOS/iOS client playing any title with an embedded or sidecar text subtitle (e.g. an MKV with an embedded ASS dialogue track, or a .srt/.vtt sidecar) where the user turns on Picture-in-Picture, AirPlay, or connects an external display.

Today: subtitles are surfaced to the host as decoded cues ($subtitleCues + the preserveASSMarkup/assHeader host-render contract from #30/#48) and the host paints them with swift-ass renderer. That works inline, but the overlay is a sibling layer outside the AVPlayerItem's video output, so subtitles disappear the moment playback moves to PiP / AirPlay / external display.

Why a host can't fix this itself (so this has to live in the engine):

• A host overlay view (or AVPlayerViewController.contentOverlayView) is not shown in PiP by AVKit,
  and isn't sent over AirPlay / to an external display.
• The AssVideoComposition CoreImage videoComposition route can't be used either — CIFilter video
  compositions are unsupported on HLS assets, and the engine plays the loopback media.m3u8 /
  master.m3u8. So there is no host-side path to get rendered subtitles into the frames AVPlayer emits.

The only thing the OS renders in PiP/AirPlay is what lives inside the AVPlayerItem — i.e. a native
legible selection. This also lines up with the AVMediaSelection wiring is a tracked follow-up note in
AetherEngine.subtitleCues's doc comment; VideoSegmentProvider.masterClosedCaptions is currently
"NONE" and there's no EXT-X-MEDIA:TYPE=SUBTITLES rendition.

Area

Public API surface

Host app / integration context

No response

Would you be willing to open a PR?

No

[superuser404notfound]

Thanks for the detailed writeup. The diagnosis is exactly right: a host-drawn overlay lives outside the AVPlayerItem, so PiP / AirPlay / external display only ever show what is inside the item, which means a native legible selection. We want this, and it has to live in the engine.

We prototyped the WebVTT-rendition path you outlined (EXT-X-MEDIA:TYPE=SUBTITLES plus a /subs.m3u8 of WebVTT segments served from the decoded cues, exposed through the legible group) and hit a structural tension with how the engine routes playback. Sharing the reasoning, because it points at a cleaner option.

Why the WebVTT rendition is awkward for this engine specifically

1. A WebVTT rendition only exists in a multivariant (master) playlist. EXT-X-MEDIA cannot live in a media playlist. But the loopback server serves the media playlist directly (no master) for a large, deliberate set of cases.

2. That media-direct routing is not incidental, it protects SDR/HDR/DV playback. A master that advertises VIDEO-RANGE=PQ against an SDR-locked panel gets rejected by AVPlayer (CoreMediaErrorDomain -11848 / -11868), so the engine deliberately bypasses the master on SDR and non-DV-ready panels (and on Dolby Vision Profile 5). Forcing a master back onto those paths just to carry a subtitle rendition re-enters exactly the picture-routing logic we worked hard to stabilize.

3. AVPlayer fixes its media selection groups at asset load. The subtitle's presence therefore has to be declared before the user ever turns subtitles on, which means routing the master from the start for essentially every subtitled title, broadening the master path well past where it is needed today.

4. Dolby Vision Profile 5 cannot be wrapped in a master at all: its CODECS is dvh1.05 with no non-DV fallback, and AVPlayer's master-level codec filter drops it on a non-DV panel (it fetches the master repeatedly and never advances to the media playlist). So a WebVTT rendition is structurally impossible for native P5 on a non-DV display, the picture would not even start.

Why tx3g (mov_text) muxed into the fMP4 looks like the better fit

Instead of a separate rendition, declare a text track (tx3g / mov_text) as a third trak inside the existing fragmented-MP4 media stream the engine already produces, alongside video and audio.

• It rides the media.m3u8 path that is already served, so it needs no master. That sidesteps the entire SDR/HDR/DV routing question: which playlist AVPlayer loads and what video signaling it sees are byte-identical to today. The subtitle is just another track in the same fMP4.
• It works uniformly across SDR / HDR10 / HLG / Dolby Vision, including P5, because it never depends on master-level variant selection.
• AVPlayer still surfaces it as a legible AVMediaSelectionOption, so PiP / AirPlay / external display render it natively, which is the whole goal.
• The "declare at load" constraint still applies (the text trak has to be in the init moov), but that is a self-contained change in the MP4 segment muxer, not a change to playback routing.

The cost is real muxer work (the segment muxer currently writes video plus optional audio; this adds an empty tx3g trak to the init moov and appends mov_text samples as cues are decoded), but it is isolated and stays out of the sensitive routing code.

Both tx3g and WebVTT are plain-text only natively, so rich ASS styling would keep being host-rendered inline for the non-PiP case in either approach. Bitmap subtitles (PGS/DVB) cannot become a native text track either way and stay host-only (or would need burn-in, which is out of scope here).

Before we commit: is there a constraint on your side that specifically needs the WebVTT rendition (a client that already consumes EXT-X-MEDIA subtitle groups, tooling, and so on), or would the tx3g-in-fMP4 route work for your use case? Given how it interacts with the SDR/HDR/DV routing, tx3g is our lean, but we would rather hear your take before building.

[edde746]

No constraint on my side.

[DrHurt]

Funny I was thinking about that before opening an issue on AetherPlayer (Mac) to request AirPlay support.

Adding an opt in “include simple text subtitles in fMP4 segments” and maybe “include all text subtitles” to additionally include flattened ASS.

Optional to allow players to use custom renderers for customisation of wanted.

If forced on, AVPlayerViewController will always show a subtitles menu and custom renderers will end up having a duplicate menu.

Ffmpeg already has a converter via the -c:s mov_text command

[superuser404notfound]

Thanks, this lines up with where we landed. We went with tx3g/mov_text muxed into the existing fMP4 stream (no master, so the SDR/HDR/DV routing is untouched), and yes, the engine uses libavformat's mov_text path.

Your duplicate-menu point is exactly why it is strictly opt-in. The native track is only declared when the host sets a prepareNativeSubtitles load option; with it off the muxer output is byte-identical to today and AVPlayerViewController shows no subtitle menu. So a host that draws its own subtitles either leaves it off, or turns it on and suppresses the stock legible menu in its own player UI. The engine never forces the track on. The track is also declared disposition:default=0 / defaultOption == nil, so even when present it is not auto-displayed: the host selects it explicitly (we expose setNativeSubtitleSelected(_:)), which is what makes it appear in PiP/AirPlay/external while inline stays host-rendered.

On the two tiers: v1 is the "simple text" one, a single active text track (embedded or sidecar) carried as plain mov_text, with rich ASS still host-rendered inline. "Flatten ASS styling into the track" and "carry all text tracks, not just the active one" are both sensible follow-ups (mov_text can hold basic style boxes), but we are keeping v1 plain and opt-in to avoid the duplicate-menu and styling-mismatch surprises. Worth noting both are plain-text only natively, so a flattened-ASS track in PiP would still differ from the inline swift-ass render.

(Separately, AirPlay on AetherPlayer for Mac is a good shout, that belongs in the AetherPlayer repo.)

[DrHurt]

All tracks would need to be included for AVPlayerViewController to enumerate them in the menu so users can select the desired one.

[superuser404notfound]

Agreed, and that is the next round. To enumerate languages in the stock AVPlayerViewController menu, every text track has to be present as its own tx3g trak with a language tag, plus a track-selection API so the host (or the user, via the native menu) can pick one.

v1 just landed on main: the single active text track muxed as tx3g, so the currently selected subtitle survives PiP / AirPlay / external display. We kept v1 to one track to ship the core win and keep the opt-in surface minimal. The follow-up carries all text tracks (embedded + sidecar) with language metadata so the native menu lists them, which is exactly your point. A single side-demuxer pass can decode every text stream into per-track stores, so it stays one extra read pass, not N.

Thanks for pushing on this.

[DrHurt]

Why a subtitles API? I thought FFmpeg would just copy all tracks at once on the fly via the -c:s mov_text and -map 0