Expressions update! by dan2wik · Pull Request #256 · Project-Babble/Baballonia

dan2wik · 2026-06-27T21:22:04Z

Sounds like we're ready to move this into main.

Store a cloned orderedKeys array and use it for Keys and enumeration. This makes sure enumeration follows the provided key order (rather than relying on FrozenDictionary iteration) and protects against external mutation of the input array by cloning it in the constructor.

- ONNX: cap IntraOpNumThreads=2 to eliminate intra-op thread fan-out CPU. Eye-only CPU inference on a Ryzen 9 9900X: ~225% -> ~67% of one core - Add always-on ThreadProfiler + Debug/Performance page: per-thread CPU, pipeline stage timings, throughput counters - Settings: add "Split Eye Video Swap" (default unswapped) and opt-in "Show Debug menu"; move Advanced options to the bottom of the list - Bundle in-progress expr-dev capture/pipeline work (V4L2 capture, video sources, OSC recv/send, parameter sender, home/calibration view models)

OpenCV's DSHOW backend can't stream Y800 (raw greyscale) - which some trackers expose as their default pin - and in fact can't open such devices at all. Result was an endless "No data received" with the camera never producing frames on Windows (it works on Linux via GStreamer/MJPEG). - Add DirectShowModeSelector: enumerate a device's DirectShow modes and pick the best non-Y800 mode (prefer MJPG, then larger frame, then higher fps). Generic - no per-camera assumptions. - OpenCvCapture: apply the selected mode as size -> fps -> FourCC (last), since the frame rate is what drives the backend onto the MJPG pin when a camera exposes the same resolution as both raw and compressed. Fall back to Media Foundation (MSMF) when DSHOW yields no frames; additive, so cameras already working on DSHOW are unaffected. - Hand off a fresh Mat per frame in the capture loop. Reusing a single Mat raced the consumer (concurrent write / use-after-dispose) and crashed natively (0xC0000005) once frames actually started flowing. - Log opened backend, negotiated format and a frame probe for diagnostics; sleep when idle to avoid busy-spinning a CPU core.

The update loop reused one Mat that SetRawMat hands to the consumer, so the next Read overwrote the buffer the consumer was still using and the feed stalled. Allocate a fresh Mat per frame and dispose it on a miss.

- Add a 'UI Render thread' row showing Avalonia's compositor frame rate, counted via a per-frame RequestCompositionUpdate hook in the view. - Rename 'UI thread' to 'UI data thread' and the header to 'Performance metrics'; drop the (responsiveness) and (NNN% = fully saturated) notes. - Relabel Face -> Mouth (inference rows, pipeline header, worker thread). - Rename 'CPU hotspots (per thread)' to 'Thread CPU Usage' and '% of one core' to '% x 1 core'.

- Sample every rate against a monotonic Stopwatch clock and read the camera frame count live, then EWMA-smooth, so delivered fps stops jittering - Count genuinely lost frames at the capture (overwritten before the pipeline acquired them) and show the total over a rolling 60s window - Tidy the Debug page header spacing

This reverts commit 5a2a7b1.

- Replace the single eye-camera card with a dynamic Cameras list: one card per active capture, with its own delivered fps, 60s drop count, negotiated fps, resolution and format - Handle both a single split eye feed (Eye + Mouth) and two independent left/right eye feeds (Eye left/right + Mouth) - Publish the active capture sources from the workers; the sampler reads each one's counters live, keyed per camera

The mini OpenCV runtime has no V4L2 backend, so opening a "/dev/videoN" path via the string ctor falls back to GStreamer's file source and tries to read the char device as a media file ("unable to start pipeline"). Resolve the path to an index and open via FromCamera so GStreamer builds a proper v4l2src pipeline instead.

The bundled mini OpenCV runtime captures only through GStreamer, which needs the v4l2src plugin (gst-plugins-good) to read /dev/video*. That plugin isn't bundled, so on systems without it the OpenCV "Normal Camera" backend silently fails. Which backend got used was decided by arbitrary Directory.GetFiles order, so /dev/video* devices could default to the broken GStreamer path instead of the dependency-free LibV4L2 one. - DesktopConnector: deterministically prefer "V4L2 Camera" on Linux. - OpenCvCapture: when the device fails to open, log an actionable message (install gst-plugins-good or use V4L2 Camera) and bail instead of starting a dead capture loop. - SingleCameraSource.Start: return the backend's real start result so a failed open fails fast rather than waiting out the 13s frame timeout.

…time The udev P/Invokes named "libudev.so", the dev-package symlink. Minimal runtimes like the Steam Linux Runtime (sniper) ship only the versioned "libudev.so.1", so the load failed there, camera enumeration returned nothing, friendly names never mapped to /dev/videoN, and starting a camera threw "No provider ... not found". Register a DllImportResolver that maps "libudev.so" to whichever soname exists (libudev.so.1, then .so, then .so.0), so enumeration works under sniper, scout, and bare distros without the -dev package.

Preferring the V4L2 backend only fixes fresh setups; users who already have "Normal Camera" persisted still get the OpenCV/GStreamer backend, which can't open /dev/video* without the unbundled v4l2src plugin. CreateStart now tries every compatible backend in preference order (the requested one first, then the rest) until one actually produces a frame, so a dead GStreamer open automatically falls back to LibV4L2. Also drops the ArgumentNullException-on-no-provider path in favour of a logged null, and waits on the frame signal instead of busy-polling.

A model hot-swap (training/calibration) replaces the inference service but reuses the filter, whose per-element state was sized to the first frame. A model with a different output count then overran those arrays (IndexOutOfRangeException). Detect a shape change and re-initialize.

Steam Linux Runtime 4 deployment: - Bundle app-local ICU so the app boots under steamrt4 (no system libicui18n) - Use the GStreamer-free OpenCV mini-runtime (4.11.0.2) on Linux - Access V4L2 through libc instead of the unversioned libv4l2 soname (LibV4L2 + VFT) - Build Linux CI artifacts inside the steamrt4 SDK container - Exclude the local CUDA trainer bundle (_internal) and fix the MSB3024 copy error - Fall back across capture backends without an unhandled exception aborting startup - Add steamrt4 build/run scripts; gitignore .vscode and the trainer _internal bundle Firmware / serial: - Surface live espflash progress and a clear failure state instead of always "Done!" - Stop only serial camera feeds on Refresh (never on tab navigation); release all firmware serial handles when leaving the tab - Skip version-unsupported requests (SetPausedRequest on v2) and drop dead-port sessions - Fix the SerialCameraCapture dispose crash on device unplug

…arp fixup The arm64 PatchSkiaSharp target runs `patchelf --add-needed` on libSkiaSharp.so, but the steamrt4 SDK container has no patchelf (the old ubuntu-arm runner did). Add it to the CI build prerequisites and the local build image.

At exit, Avalonia's Linux DBus integration (tray / platform) disconnects on a background thread and marshals a notification to the already-stopped UI dispatcher, surfacing as an unhandled TaskCanceledException that aborted the process. - Add global AppDomain.UnhandledException + TaskScheduler.UnobservedTaskException handlers (there were none). - During our shutdown, a benign cancellation now Environment.Exit(0)s instead of aborting; all other unhandled/unobserved exceptions are logged, not swallowed. - Complements 76516a9 (Windows exit path); this covers the Linux path.

- Clear NU1510/CA2017/CS0105/CS0649/MSB3568 warnings - Wire up OpenVRService so the autostart toggle works

- Apply user toggles reliably instead of swallowing them during load - Use a short-lived OpenVR session so toggling after SteamVR closes can't crash

Left/right start paths build source but never assign it to VideoSource, and are unreachable: TryStart{Left,Right}IfNotRunning no-op on a SingleCameraSource.

BabbleCalibration is a separate .NET (Godot Mono) binary with its own runtime and no app-local ICU, so under steamrt4 (no libicui18n) its globalization init FailFast/SIGABRTs at launch, which blocked calibration and training. Launch the overlay child process with DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1.

Identify the tracker during enumeration by USB VID 0BB4/PID 0321 or the "HTC Boot" name, and force the VFT capture backend over the generic "Normal Camera"/"V4L2 Camera" ones, which otherwise open it un-activated and mis-decode the image.

CLAassistant · 2026-06-27T21:22:13Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 4 committers have signed the CLA.

✅ RamesTheGeneric
✅ SummerSigh
✅ dan2wik
❌ AeroScripts
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

dfgHiatus · 2026-06-27T21:33:56Z

-    private float[] dxPrev;
-    private DateTime tPrev;
-    public OneEuroFilter(float[] x0, float minCutoff = 1.0f, float beta = 0.0f)
+    private const float TwoPi = 2.0f * MathF.PI;


Use Math.Tau here!

dfgHiatus · 2026-06-27T21:34:52Z

    private string[] _outputExpressionNames;
-    private bool _isOldEyeModel;
+    private bool _hasModelMetadata;
+    private OrderedFloatMap _outputs;


Two things:

Instead of a list of lists, let's make the outer collection a Dictionary<enum, <List<string>> so we can more explicitly index into it. IE _knownMappings[0] -> List becomes _knownMappings[ExpressionMappings.Eye] -> List. Yes we don't really index into the list anywhere but I don't want this sneaking up on us.

For (reused) strings, let's move this to a static class full of const strings.

I also see there's some string parsing in the VRCFT module, this would help us re-use/update strings in one "shared" file .

dfgHiatus · 2026-06-27T21:35:30Z

+            }
        }
+
+        _logger.LogInformation("Initialized model that predicts {Expressions}", string.Join(", ", _outputExpressionNames));


Change this to LogDebug, this is more for us and less for the end user.

Suggested change

_logger.LogInformation("Initialized model that predicts {Expressions}", string.Join(", ", _outputExpressionNames));

_logger.LogDebug("Initialized model that predicts {Expressions}", string.Join(", ", _outputExpressionNames));

dfgHiatus · 2026-06-27T21:35:53Z

+        } else {
+            // determine expression mapping from model output size
+            var outputSize = _session.OutputMetadata.Values.First().Dimensions[1];
+            foreach(List<string> mapping in _knownMappings)


Shouldn't we throw a warning/error here if we're not able to load a face model at all? IE if model input size doesn't match what were are expecting

dfgHiatus · 2026-06-27T21:36:37Z


    public override void Update()
    {
-        if (needsEye)


We need to keep the use of needsEye and needsExpression. I forget the exact reason why but maybe @benaclejames can chime in on this.

Looks like the module is isolated on VRCFT's side of things. I don't think anything breaks, we'd just be sending data uselessly into the void.
I'll see if I can come up with a graceful handler for it in a bit.

dfgHiatus · 2026-06-27T21:37:06Z

Don't forget to bump the version number (we're on 3.2.0 now IIRC) as well the the module.json!

dfgHiatus · 2026-06-27T21:37:57Z

+        int length = _xPrev.Length;

-        if (elapsedTime == 0.0f)
+        if (elapsedTime <= 0.0f)


If all we're doing here is copying a span, wouldn't it make more sense to use Span.CopyTo or Span.TryCopyTo?

dfgHiatus · 2026-06-27T21:38:46Z

+    }
+
+    // Expose the raw array/span directly so the ONNX runner can write to it
+    public Span<float> ValuesSpan => _values;


I noticed in https://github.com/Project-Babble/Baballonia/pull/253/changes#diff-5431c54da43664d697c115e0d9478a7e3a1f380cd37ff0a176042848e2d82685R49 we're doing something like this:

OrderedFloatMap x; ReadOnlySpan<float> xSpan = x.ValuesSpan; int length = _xPrev.Length;

Should we return a ReadOnlySpan here instead of a Span? Why/why not?

Also, is there any difference between:

public Span<float> ValuesSpan => _values;

And

public Span<float> ValuesSpan => _values.ToSpan();

I think there's an implicit conversion happening here, but I'm not too certain 🤔

RamesTheGeneric

LGTM

- OpenCvCapture: scope IsOpened() fail-fast to local devices so network streams aren't aborted before the read loop pumps frames - IpCameraCaptureFactory: accept https MJPEG streams, not just http - SingleCameraSourceFactory: prefix scheme-less hosts (e.g. openiristracker.local) with http:// so bare endpoints reach a backend - OpenCVCaptureFactory: claim only an explicit streaming-scheme allowlist instead of any absolute URI

- MainWindow: read AssemblyInformationalVersion (keeps -prerelease suffix), strip +<git-sha> - Baballonia.csproj: add Version default, disable source-revision suffix - build.yml: pass -p:Version from the workflow version-number input when set

…gging - Prefer IPv4 (A-record) connect with a 4s ConnectTimeout so a .local name opens as fast/deterministically as a raw IP, instead of stalling on an IPv6 link-local past the frame window - StopCapture disposes the HttpClient (not just cancels) to abort a stuck socket read and close it, freeing the single-connection camera's slot so failover/retry can connect; non-blocking - Log HTTP status, non-MJPEG content-type, connection failures, and first-frame delivery so silent 'no data' failures become diagnosable

…nfig wipes - Write via temp file + atomic rename, keeping the prior contents as .bak; a kill/crash mid-write can no longer truncate the live settings file - On load, recover from .bak when the main file is corrupt; set an unrecoverable file aside as .bad instead of silently resetting to empty; refuse to overwrite a file that couldn't be read at all - SaveSetting skips unchanged values to stop the redundant write churn (VM init re-saves, per-retry camera-state saves) - App.axaml.cs: final atomic flush right before the Windows force-kill so the last state lands and no debounced write is left mid-flight

… reason - Resolve DNS unfiltered (AF_UNSPEC) like browsers/curl, then order results IPv4-first instead of querying IPv4-only; the family-filtered lookup failed to resolve .local mDNS names on many systems (direct IP worked because it skips resolution) - Log the resolved address(es) for the host - Flatten the exception chain into the failure log so the underlying cause is visible even when the log sink doesn't render exceptions - Add a 'connecting...' debug line; drop the HttpClient overall timeout (stream is open-ended, ConnectTimeout bounds the open)

…enComposite crash

…ver can't The app runs inside the Steam Linux Runtime (pressure-vessel) sandbox, whose minimal nsswitch has no nss-mdns/Avahi, so getaddrinfo returns 'Name or service not known' for .local hosts even though the host shell resolves them. Add a one-shot multicast-DNS A-record query (RFC 6762) as a fallback when the OS resolver fails for a .local name; it only needs multicast UDP, which the sandbox has. Direct IPs and normal DNS are unaffected (OS resolver still tried first).

AeroScripts and others added 30 commits May 21, 2026 19:11

Push inference stack update

a0b41e3

Optimizations+fixes

823c40e

Prepend parameter names with "/" to match standard

78c6b96

Module rework

10d2894

Fix parameter mapping, optimizations

9385e2b

Fix calibration tab, enable eye expr sliders

0cf5390

Fix incorrect case

7fbd5b5

Fix calibration loading

e1fd379

Update calibration sequences and trainer

0deb8d3

Revert sequence times

fa73914

version number ig

b34a16d

Add Alpha warning banner

4a69127

Fix Stabilize Eyes not enabling

e37b1f6

Fix camera freeze from Mat reuse in capture loop

f6d1154

The update loop reused one Mat that SetRawMat hands to the consumer, so the next Read overwrote the buffer the consumer was still using and the feed stalled. Allocate a fresh Mat per frame and dispose it on a miss.

Add checkpoints for Linux Trainer

7ea358a

Revert "Fix Windows capture for cameras whose default DShow pin is Y800"

21764b4

This reverts commit 5a2a7b1.

Show Linux CI jobs as steamrt4-x64 / steamrt4-arm

09532fe

dan2wik added 10 commits June 26, 2026 10:52

Bump Tmds.DBus.Protocol to 0.21.3 to fix GHSA-xrw6-gwf8-vvr9

d8803b3

Fix build warnings and the broken SteamVR autostart toggle

3478327

- Clear NU1510/CA2017/CS0105/CS0649/MSB3568 warnings - Wire up OpenVRService so the autostart toggle works

Fix SteamVR autostart toggle off path and crash

39c06a2

- Apply user toggles reliably instead of swallowing them during load - Use a short-lived OpenVR session so toggling after SteamVR closes can't crash

Mount the BabbleTrainer symlink target into the steamrt4 build container

a2945aa

Flag flawed/unused SingleCameraSource branches in EyePipelineManager

a08ba5a

Left/right start paths build source but never assign it to VideoSource, and are unreachable: TryStart{Left,Right}IfNotRunning no-op on a SingleCameraSource.

Updated and improved linux workaround instructions.

8fdd7d3

Renaming linux workaround key to avoid stale translations

88334f1

dan2wik requested review from RamesTheGeneric and dfgHiatus June 27, 2026 21:22

dfgHiatus mentioned this pull request Jun 27, 2026

feat: Add mutable infer to input graphs, eye expressions #253

Closed

dfgHiatus reviewed Jun 27, 2026

View reviewed changes

dfgHiatus added the enhancement New feature or request label Jun 27, 2026

dfgHiatus self-assigned this Jun 27, 2026

RamesTheGeneric approved these changes Jun 27, 2026

View reviewed changes

dan2wik added 10 commits June 28, 2026 08:15

OneEuroFilter: use MathF.Tau instead of local TwoPi const

ebc8425

OneEuroFilter: copy spans via CopyTo in zero-dt branch

37fdf5f

DefaultInferenceRunner: demote model-expr init log to Debug

ef36032

DefaultInferenceRunner: throw when output size matches no known layout

c71581f

ActivationService: comment out boot-time OpenVR autostart to avoid Op…

1134922

…enComposite crash

dan2wik mentioned this pull request Jun 30, 2026

[Bug]: OneEuroFilter applied to DFR output #232

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Expressions update!#256

Expressions update!#256
dan2wik wants to merge 57 commits into
mainfrom
expr-opt

dan2wik commented Jun 27, 2026

Uh oh!

CLAassistant commented Jun 27, 2026 •

edited

Loading

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

dan2wik Jun 27, 2026

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

dfgHiatus Jun 27, 2026

Uh oh!

Uh oh!

RamesTheGeneric left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	_logger.LogInformation("Initialized model that predicts {Expressions}", string.Join(", ", _outputExpressionNames));
	_logger.LogDebug("Initialized model that predicts {Expressions}", string.Join(", ", _outputExpressionNames));

Uh oh!

Conversation

dan2wik commented Jun 27, 2026

Uh oh!

CLAassistant commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RamesTheGeneric left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CLAassistant commented Jun 27, 2026 •

edited

Loading