roadmap.md

Lightning Pose development roadmap

General enhancements

• support for training/inference with mixed precision
• introduce visibility column in labeled data to match COCO format; treat occluded and unlabeled points differently (Issue)
• read COCO JSON label files (need to address visibility handling first)
• introduce jaxtyping (#407)
• multi-GPU training for unsupervised models (#207)
• multi-GPU training for supervised models (#206)

Video reading enhancements

• look into using pynvvideocodec for accelerated inference and Windows support
• add OpenCV video reader option for inference for native Windows compatability

Losses and backbones

• compute non-temporal unsupervised losses on labeled data
• incorporate transformer backbones (#84, #106)

Multi-view support for non-mirrored setups

• unsupervised losses for multi-view (#187)
• context frames for multi-view (#126)
• implement supervised datasets/dataloaders that work with multiple views (#115)

Single-view dynamic crop (small animals in large frames)

• split CLI crop command into create_bbox and smooth_bbox commands for modularity (Issue)
• context frames for dynamic crop (#250)
• unsupervised losses for dynamic crop (#250)
• implement dynamic cropping pipeline with detector model and pose estimator (#250)

Multi-view dynamic crop

• perform view-specific dynamic cropping, re-assemble views after pose estimation stage
• context frames for multi-view dynamic crop
• unsupervised losses for multi-view dynamic crop

Multi-animal

• single-view, supervised
• single-view, context
• single-view, unsupervised losses
• multi-view, supervised
• multi-view, context
• multi-view, unsupervised losses