Skip to content

skill.mdx

Latest commit

 

History

History
149 lines (114 loc) · 5.33 KB

skill.mdx

File metadata and controls

149 lines (114 loc) · 5.33 KB
Raw
title UIFY Skill — Build Anything
description Self-contained source of truth for building with UIFY. Read this first.
slug skill
author rahulmnavneeth
date 23 APR 2026
tags
skill
guide
reference
source-of-truth

This page is the single entry point for building with UIFY. It is self-contained for the common path — an agent or human can complete ~90% of typical tasks without leaving this document. Links go out only for deep material (EKF-on-manifold derivation, CLAP internals, per-platform camera backend quirks).

If something in this document contradicts a reference doc, the code wins; file an issue or fix the doc.

What UIFY is

A Rust engine that turns camera input into a time-indexed stream of geometric primitives, with optional real-time transport to other programs. Five built-in trackers, all implementing one trait:

Tracker Geometry G Status Typical consumer
point (ball) Vec2 (2D) implemented audio automation, cursor control
bounding box (cx, cy, w, h) AABB implemented detection gating, VFX masks
plane SE(3) (smoother only) partial AR, virtual camera body, compositing
face landmarks + SE(3) pose stub expression capture, retargeting
rotoscope contour / mask stub matte extraction, VFX

Every tracker produces Sample<G, C> values — value, tangent-space covariance, confidence, host-monotonic timestamp.

60-second quick start

# Development shell (Nix). Installs Rust toolchain, pre-commit hooks.
nix develop

# Build the workspace.
make

# Run the full test suite (unit + property + synthetic GT).
make test

# Lint (clippy warnings = errors).
make lint

# Build the CLAP plugin bundle.
make clap

# Format everything.
treefmt

The one abstraction

Every tracker is this:

pub trait Tracker {
    type Geometry;
    type Covariance;
    type Measurement<'a>;

    fn step(&mut self, measurement: Self::Measurement<'_>)
        -> Result<Option<Sample<Self::Geometry, Self::Covariance>>, TrackerError>;

    fn reset(&mut self);
}

Different trackers differ only in Geometry and Covariance. The pipeline around them — detector, associator, filter, smoother, sink — is the same shape.

Minimal pipeline (point tracker)

use nalgebra::{Matrix2, Matrix4, Vector2, Vector4};
use uify_core::{Sink, Tracker};
use uify_point::{PointMeasurement, PointTracker2D};

let mut tracker = PointTracker2D::new(
    Vector4::zeros(),               // initial state (px, py, vx, vy)
    Matrix4::identity(),            // initial covariance
    Matrix4::identity() * 1e-3,     // process noise (per second)
    Matrix2::identity() * 1e-3,     // measurement noise
);

loop {
    let frame = camera.next_frame()?;                  // runtime::camera::Camera
    let detection: Vector2<f64> = inference.infer(&frame)?;
    let measurement = PointMeasurement { t: frame.t, position: detection };
    if let Some(sample) = tracker.step(measurement)? { // Sample<Vec2, Mat2>
        sink.write(&sample)?;                          // any Sink<Vec2, Mat2>
    }
}

End-to-end example

examples/pipeline/ ties it all together: synthetic camera → constant inference → PointTracker2D → RingbufSink → RingbufSource → OscPoint2DSink → UDP. Run it with:

cargo run -p uify-pipeline-example

A DAW-targeted gesture pipeline (HandLandmarker → GestureFSM → ParameterMap → CLAP plugin params) is a future addition; the gesture FSM will live as a consumer of tracker samples, not part of any tracker, so the same hand tracker can drive non-DAW consumers too.

Latency + accuracy rules of thumb

  • Audio thread never allocates, never locks, never blocks on I/O. It reads rtrb and interpolates in the tangent space to the block time.
  • Every sample has a host-monotonic timestamp. Latency is measured, not guessed.
  • Every filter lives on the correct manifold. Do not average quaternions componentwise. See architecture-manifolds.
  • Every sample carries covariance. Consumers should gate on uncertainty.

Glass-to-parameter target: ~15–25 ms baseline, ~8–12 ms with distilled models + zero-copy capture. Below ~8 ms requires a 240 Hz camera.

Gotchas

  • CoreML / DirectML / NNAPI have different f16 / f32 tolerances. The synthetic GT suite runs per-backend; do not compare numeric outputs across backends with ==.
  • CLAP hosts reload plugins during scan. Global state in static mut is a bug waiting to happen — keep all mutable state in the Plugin instance.
  • The GHC RTS is not involved. If you find yourself writing FFI glue for a garbage-collected runtime on the audio thread, stop.

Extending

  • Add a tracker: new crate under crates/uify-<name>/, implement Tracker, register in uify-clap-plugin if it should drive parameters.
  • Add a transport: new crate under crates/uify-transport-<name>/. Consume Sample<G, C> via a Sink impl.
  • Add a platform: implement the camera::Backend trait and feature-gate in uify-runtime.

See architecture-extension for the full guide.