Standardized skeletons

[CodyCBakerPhD]

@rhingo @talmo

Everything below is a starting point for discussion, not a decision; feel free to push back on anything

Motivation

For an initial version of per-species foundation models, it has been speculated that a common skeleton when labeling training data would be invaluable for stabilizing model training

While we may eventually want separate skeletons per species, the value of a cross-species skeleton for generalization should not be understated

Separately, recent interactions with Linda Wilbrecht suggest it may be valuable to support both coarse and fine granularity for node definitions

Proposal: naming convention

Shorthand for referring to a specific skeletal configuration:

{species}{granularity}{version} (e.g. RC0, NHPF2, HC0)

{species}

• R: Rodent (assuming one skeleton can apply to all mice/rats/gerbils/etc.)
• NHP: Non-Human Primates (assuming a common skeleton across that diversity)
• H: Human
• Main focus for now; more complex species (flies/birds/zebrafish) may come later

{granularity}

• C: coarse (major body motions)
• F: fine (finger joints, hand movements)
• U: ultra-fine (twitch/reflex/whisking; out of current scope IMO, practically a separate class of problems)

{version}: positive integer, starting at 0 for the prototype

We could use full semantic versioning, but virtually any change to node classifications breaks backward compatibility in some manner, so I think a running counter is sufficient

Proposal: v0 skeleton (RC0)

Head: nose, left eye, right eye, neck
Forelegs: left shoulder, left front paw, right shoulder, right front paw
Hindlegs: left knee, left back paw, right knee, right back paw
Spine: tail base

Note this is a strict subset of the old AP-10k project

I'm proposing it purely as a starting point for the group and the workshop; we can and should debate every node, and document the decision in each anatomical case

Contested nodes to resolve (defaults shown):

Node(s)	Rationale to debate
Ear points (base→tip)	multiple points per ear; uncertain payoff
Tail points (beyond base)	high deformation and occasional low contrast, labeler disagreement, fast movements
Spine intermediate points	only tail base anchors the spine currently

Note:HC0 is effectively already standardized as the well-known COCO-17 skeleton (17 keypoints, human-only)

The challenge of cross-labeler differences

Different labelers applying the same nodes to the same subject will produce differences in labeled positions

In the context of the collaborative Pozu labeler, I believe we can attack this from two directions:

a. Certification via training sequences (requires logged-in, known authorship)
Using a stable series of agreed-upon labeled frames, a new author earns certification for a given node by labeling those frames and staying below an error threshold. Real-time feedback can flag common problems such as mirroring (left vs. right).

b. Drift detection via pseudo-plurality consensus
Track drift from an individual's original style by comparing the same node labelings months apart against a consensus reference.

• To define: how consensus position is computed (mean / median / inter-labeler agreement metric) and what drift magnitude triggers re-certification.

[talmo]

I think this is a solid starting skeleton. I think including the tail is worth discussing but should be downstream of initial data curation (it is valuable biologically, but much harder to annotate and track).

Also, I think a more detailed annotation protocol is worth formalizing, at a minimum for the "expert" annotators that can create a honeypot (gold standard subset, carefully annotated, used to evaluate future annotators). Here's ours that we've used for crowdsourced annotation in the past. Note how it grounds the protocol in both anatomy and heuristics, as well as includes defined criteria for edge cases.

[CodyCBakerPhD]

That is great, thank you! May I use some of those illustrations as side-by-side 'references' as well as instructions during the user training protocol?

[talmo]

Go for it!