Skip to content

Add optional Face Expression OSC helper output#359

Open
0suu wants to merge 3 commits into
benaclejames:masterfrom
0suu:suu/fer-face-experssion-osc-helper
Open

Add optional Face Expression OSC helper output#359
0suu wants to merge 3 commits into
benaclejames:masterfrom
0suu:suu/fer-face-experssion-osc-helper

Conversation

@0suu

@0suu 0suu commented Jun 15, 2026

Copy link
Copy Markdown
2026-06-15.22-38-34_1.mp4

Summary

This PR adds an optional Core-side OSC helper that classifies calibrated facial expression patterns from Unified Expressions and exposes the result as avatar parameters.

It is intended as an opt-in avatar helper output built from the user's calibrated facial expression templates.

Why calibration is required

This helper uses calibration instead of a fixed rule-based mapping because face tracking behavior and intended avatar expressions can vary by user and avatar. Calibration lets the user define which facial pattern should map to each helper expression, instead of assuming one universal expression mapping.

What changed

  • Added a Face Expression OSC Output postprocessor mutation.
  • Added helper OSC parameters:
    • /avatar/parameters/v2/FaceExpressionIndex
    • /avatar/parameters/v2/FaceExpressionPower
  • Added calibration for:
    • Neutral
    • Joy
    • Angry
    • Sad
    • Surprise
  • Added per-expression activation thresholds for Joy, Angry, Sad, and Surprise.
  • Added saved calibration data, status rows, calibration buttons, and reset controls in the mutation UI.
  • Added template-based scoring from a reduced Unified Expressions feature vector.
  • Added hold/margin switching rules to reduce rapid expression flicker.
  • Kept the helper disabled until Neutral, Joy, Angry, Sad, and Surprise are all calibrated.

Behavior

Expression index values are:

  • 0 = Neutral
  • 1 = Joy
  • 2 = Angry
  • 3 = Sad
  • 4 = Surprise

Supporting changes

This PR also includes mutation UI infrastructure changes needed by the new calibration controls.

Mutation UI components are now created and refreshed through the UI dispatcher so stateful mutation controls, such as calibration status and button enabled/text updates, are marshalled back to the UI thread.

Mutation loading is now awaitable during startup, and each mutation is initialized before it is added to the active mutation collection. This avoids OSC updates observing partially loaded or uninitialized mutations during startup.

Notes

The helper parameters currently use v2/FaceExpressionIndex and v2/FaceExpressionPower. I would appreciate feedback on whether these names fit the existing v2 parameter naming scheme, or if a different naming convention would be preferred.

This PR extends the mutation UI support layer because the calibration workflow needs to display current calibration state, button text, and enabled/disabled state dynamically. I kept the new components generic (MutationInfo, MutationStatus, MutationStatusAction) instead of making them face-expression-specific, and would appreciate feedback on whether this is the right level of abstraction for the project.

Related to #358.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant