Skip to content

fix(mediaplayer): sync audio-only sources to remote players#976

Merged
dooly123 merged 2 commits into
BasisVR:developerfrom
towneh:fix/mediaplayer-audio-only-remote-sync
Jul 21, 2026
Merged

fix(mediaplayer): sync audio-only sources to remote players#976
dooly123 merged 2 commits into
BasisVR:developerfrom
towneh:fix/mediaplayer-audio-only-remote-sync

Conversation

@towneh

@towneh towneh commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Loading an audio-only URL (.wav, and equally .mp3 / .m4a / .opus) played for whoever loaded
it but never reached anyone else. Remote clients did nothing while a video was playing, and once
their current VOD ended they replayed that video rather than the new source. Late joiners got the
audio fine, which is what isolates it to the live broadcast rather than the URL itself.

Two things combined.

Readiness was video-only. In BasisNativeVideoSource.Pump, the whole readiness block sits
inside if (fc != lastFrameCounter), the video frame counter, and is further gated on
OutputTexture != null. An audio-only source never ticks that counter and never produces a
texture, so OnReady never fired, on any client. The native engine handles audio-only correctly,
reaching PLAYING once audio frames are flowing on a source that announced no video track, but
that state was never plumbed through to the C# event. IsPrepared and Status were left wrong for
audio-only too, so the damage went wider than the sync.

The URL broadcast hung off that event. SetUrl skipped its up-front BroadcastFullState() for
directly-playable URLs and deferred to the OnReady path, which never ran. FullState is the only
message carrying a URL, so currentSyncedUrl was set locally and never transmitted. Peers received
only the bare Play command and played whatever source they still held.

The changes:

  • Fire OnReady when the engine reaches Playing with an audio format and no video size. The
    engine only reports that pairing once the source has announced no video track at all, and
    split-stream (whose video leg announces later) is excluded, so it's a sound audio-only signal
    rather than a guess about timing.
  • Broadcast FullState unconditionally from SetUrl.
  • Mark that broadcast as a fresh load. It goes out before LoadUrl, while the player still holds
    the outgoing media, so SerializeFullState describes the source being replaced rather than the
    one being loaded. The direct-URL apply path feeds that state and playhead straight into
    StartPosition, so peers need the broadcast to describe the load being started. The page-URL
    path stashes state instead of applying it that way, so it never took that route.
  • Carry that intent through the pre-network-ready deferral queue, and retire it once the load
    reaches OnReady, after which the player's own state and position are the truth and later local
    commands must not be re-serialised as a pending load at position zero.
  • Honour the owner's advertised state when applying a resolved page URL. ApplyPendingRemoteState
    only ever stopped or paused; starting playback was left to the peer's own
    AutoPlayOnSourceAssigned, so a peer with that unticked sat stopped while the owner played. The
    direct-URL path already forces this around LoadSource, so the two now agree. IsPlaying and
    IsPaused are independent here, so a source that arrived paused is resumed rather than started.

Required checks

All boxes below must be ticked before this PR can merge. If a check is genuinely N/A, tick it anyway and explain under Notes.

  • Tested — I built and ran this locally. The change works in the editor and (where relevant) in a built player.
  • Transform access is combined and limited — In hot paths, transform reads/writes go through TransformAccessArray or are otherwise batched. I have not added per-frame transform.position / transform.rotation / transform.localPosition calls inside loops. Whenever I need both position and rotation, I use the combined APIs — SetPositionAndRotation / SetLocalPositionAndRotation for writes, GetPositionAndRotation / GetLocalPositionAndRotation for reads — instead of two separate property accesses; the combined call does one local-to-world matrix traversal instead of two.
  • Addressables used for asset/memory loading — Any new asset loads go through Addressables. No new Resources.Load, no direct asset references that pull large content into memory on scene load.
  • No new GetComponent / AddComponent where avoidable — Where unavoidable, the result is cached on a field, and any GetComponent<T> is replaced with TryGetComponent<T>(out var x) — bare GetComponent will be denied. TryGetComponent is the modern API (Unity 2019.2+) and skips the Editor-only GC allocation GetComponent causes when a component is missing: Unity wraps the null return in a managed "fake null" object so its overloaded == operator can still detect destroyed C++ objects, and constructing that wrapper allocates; TryGetComponent returns a bool plus out parameter and never builds the wrapper. None of these calls run inside Update, LateUpdate, FixedUpdate, jobs, or other per-frame code paths.
  • Per-frame work is scheduled through BasisEventDriver — Any new per-frame work hooks into BasisEventDriver rather than adding standalone Update / LateUpdate / FixedUpdate callbacks on a MonoBehaviour.
  • Anything added to BasisEventDriver is bulletproof, or guarded by try/catchBasisEventDriver runs the single per-frame tick that drives the whole framework (network apply, local player sim, blendshapes, JigglePhysics, nameplates, and more) as one sequential chain. An unhandled exception anywhere in that chain aborts the rest of the tick, so every step after the throwing one is silently skipped for that frame. New work added to the driver must either be guaranteed not to throw, or be wrapped in a try/catch that contains the failure and surfaces it through BasisDebug — logged once / rate-limited, never every frame (see the existing HVRBasisBuiltInAddresses.Simulate() guard for the pattern). Expect this to be scrutinized closely in review.
  • Considered jobification — I asked whether this work can be moved to a Unity Job (Burst-compiled where possible). If it can, it is. If it cannot, the reason is in Notes.
  • No needless { get; set; } properties or access lockdowns — Public fields are fine; Basis is meant to be read and modified freely, so don't wall things off private/internal without a real reason. Don't wrap a field in { get; set; } when the accessors do nothing — property accessors have a real performance cost vs direct field access, and the lead maintainer prefers plain fields (or a method / setter-only property when only the setter needs logic) over a noop-getter pair. For .Instance singletons, callers reassigning Type.Instance is allowed; if that would break your code, log a warning or throw — don't block the assignment. Locking down access is not your call.
  • Camera access goes through BasisLocalCameraDriver — Code that needs the local camera (transform, projection, rig data, etc.) pulls it from BasisLocalCameraDriver rather than looking one up itself. Don't roll a separate camera discovery path.
  • Logging uses BasisDebug — All new logging calls go through BasisDebug.Log / BasisDebug.LogWarning / BasisDebug.LogError (with an appropriate LogTag) instead of UnityEngine.Debug.Log / Debug.LogWarning / Debug.LogError. BasisDebug routes through Basis's tagged, color-coded logger and respects the project-wide LoggingDisabled toggle so logging can be killed at runtime; bare Debug.Log calls bypass that and will be denied.
  • No scene-wide discovery for dependencies — New code is architected so it does not need FindObjectOfType / FindObjectsOfType / GameObject.Find / FindGameObjectsWithTag to locate what it depends on. References are wired in — registered through an existing manager/driver, injected at init, or passed in by the caller — rather than discovered by scanning the scene at runtime. If a scene scan is genuinely unavoidable, justify it under Notes.
  • No allocations in hot paths — Per-frame code (Update / LateUpdate / FixedUpdate, simulation loops, jobs, anything called once per frame or more) does not allocate. No new on reference types, no LINQ, no string concatenation/interpolation, no boxing, no foreach over interface-typed collections. Allocate once at init and reuse the buffer.
  • No debugging in hot paths — No log calls of any kind on per-frame paths, including BasisDebug. Hot-path logging floods the console and incurs cost on every frame regardless of whether the message is filtered out downstream. If a hot-path log is needed while iterating, gate it behind #if UNITY_EDITOR and remove (or leave gated) before merge.
  • Hot-path collection access is optimized — Cache .Count (lists) / .Length (arrays) into a local int before the loop instead of re-reading the property each iteration. Prefer T[] (with a separate length int when the array is over-sized) over List<T> where the data is hot — Unity's mono BCL doesn't expose CollectionsMarshal.AsSpan(List<T>), so a list can't be fed into Span<T> / unsafe paths cleanly. Where the perf justifies it, drop into Span<T> / ref locals / Unsafe.As / unsafe pointer code to skip bounds checks and copies, and call out the invariants you're relying on under Notes so reviewers can sanity-check them.

Testing details

Tick the platforms you actually tested on. Leave the rest unticked — these are informational and do not block merge.

  • Windows
  • Linux
  • Android
  • iOS
  • macOS

Input / control mode coverage:

  • Tested in VR (note headset under Notes)
  • Tested in desktop / non-VR mode
  • Tested with phone controls (mobile touch input)
  • N/A — change does not touch player/XR/input code

Where applicable, confirm these flows still work after your changes:

  • Hot-switching (desktop ↔ VR mode swap at runtime)
  • Avatar swapping
  • Server swapping (joining / leaving / changing servers)
  • N/A — change does not touch any of the above

Notes

Verified on Windows x64 standalone and Quest Pro. Both platforms were exercised against a
two-client setup with the Quest as the peer rather than the owner, since the owner's own
playback worked throughout and is not what this fixes.

Covered on both: an audio-only source loaded by the owner now plays on remote clients; loading it
while a peer is mid-playback of a video switches correctly and starts near the beginning rather
than at the outgoing video's playhead; late join still works and lands on the owner's current
position; and video playback is unaffected, which is the regression that mattered most here since
the readiness path is shared. On Quest the video lane also covers the Vulkan AccessTexture path,
which is where readiness differs from desktop. Quest logcat was clean of media player errors.

Fixtures were .wav (LPCM) plus the other audio-only extensions on the directly-playable list.

Verification did turn up a separate defect, which this PR does not cause and does not fix. When the
owner reaches the end of a source it broadcasts an unconditional stop, and remote clients apply it
wherever their own playhead happens to be. A late joiner is inherently a little behind the owner,
so it loses that much off the end. Measured at 96% on a 1:40 clip, roughly 4 seconds short.

It is not specific to this change or to audio: HandleLocalEnded is pre-existing and untouched
here, and a video late joiner is affected the same way. This change only makes it visible for
audio-only sources, which never reached remote clients at all before. Loading a URL with the remote
client already present is unaffected, since both clients stay in sync and finish together. Tracked
separately.

A behaviour change worth a second opinion: a world author who deliberately unticks
AutoPlayOnSourceAssigned on a networked player will now see peers start playing when the owner
does, on the resolver path as well as the direct one. The reasoning is that in a networked instance
the owner's state should win, and that's what the direct-URL path has always done. It is still a
change for that configuration rather than a pure fix, so happy to gate it if you'd rather it stayed
opt-out.

On the perf checks: the added readiness test rides the existing Pump tick rather than introducing
any per-frame work of its own, and is latched behind readyFired so it stops evaluating once a
load is ready. It calls two native getters with out int parameters, so no allocation, no logging,
no collection access. The rest of the diff is event and message plumbing outside any hot path.

towneh added 2 commits July 20, 2026 10:00
Loading an audio-only URL (.wav, and equally .mp3 / .m4a / .opus) played for
whoever loaded it but never reached anyone else. Peers did nothing while a
video was playing, and once their current VOD ended they replayed that video
rather than the new source. Late joiners received the audio correctly, which
isolates it to the live broadcast rather than the URL itself.

Readiness was video-only. In BasisNativeVideoSource.Pump the whole readiness
block sits behind the video frame counter and a non-null OutputTexture, neither
of which an audio-only source ever produces, so OnReady never fired on any
client. The engine handles audio-only correctly - it reaches Playing once audio
frames are flowing on a source that announced no video track - but that state
was never plumbed through to the C# event, which also left IsPrepared and
Status wrong for audio-only.

The URL broadcast then hung off that event. SetUrl skipped its up-front
BroadcastFullState for directly-playable URLs and deferred to OnReady, which
never ran, and FullState is the only message carrying a URL. currentSyncedUrl
was set locally and never transmitted, so peers kept the source they already
held and played that on the next bare Play command.

Fire OnReady when the engine reaches Playing with an audio format and no video
size - a pairing the engine only reports once the source has announced no video
track, with split-stream excluded since its video leg announces later - and
broadcast unconditionally from SetUrl.

Broadcasting there happens before LoadUrl, while the player still holds the
outgoing media, so the serialized state and playhead would describe the source
being replaced and land on peers as the incoming source's start position. Mark
that broadcast as a fresh load so it describes the load being started instead.
The intent has to survive the pre-network-ready deferral queue as well, and be
retired once the load reaches OnReady, after which the player's own state is
the truth and later local commands must not be re-serialised as a pending load
at position zero.

Applying a resolved page URL also ignored the owner's advertised state:
ApplyPendingRemoteState only ever stopped or paused, leaving playback to the
peer's own AutoPlayOnSourceAssigned, so a peer with that unticked sat stopped
while the owner played. The direct-URL path already forces this around
LoadSource, so both paths now agree. IsPlaying and IsPaused are independent
here, so a source that arrived paused is resumed rather than started.
Audio-only sources carry no video track, so a readiness regression on that
path is invisible from the owner's client and only shows up on peers. Name
the two-client scenario, including the load-while-peers-are-playing case and
the peer with autoplay disabled.
@towneh
towneh marked this pull request as ready for review July 20, 2026 10:50
@towneh
towneh requested a review from dooly123 July 20, 2026 11:07
@towneh towneh added the bug Something isn't working label Jul 20, 2026
@dooly123
dooly123 merged commit 87f18b3 into BasisVR:developer Jul 21, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants