v0.3.1.0: Automated annotation with CoTracker

napari-deeplabcut v0.3.1.0

What's New

Automated annotation with point tracking

We're happy to introduce automated annotation with point tracking to help accelerate DeepLabCut annotation workflows in napari.

With the new Tracking Controls widget, you can now:

• Manually annotate keypoints on a reference frame
• Use a point tracking model to propagate those keypoints forward and/or backward in time
• Review, refine, delete, and merge tracked results before exporting them back into DeepLabCut

We currently support CoTracker3, please let us know if you would like to see more models added.

In more details:

• New Tracking Controls widget in napari
• Support for forward, backward, and bidirectional tracking
• Tracking results are created as separate Points layers, leaving the original annotation layer unchanged
• New refinement workflow for:
  • Deleting selected tracked points in future frames
  • Merging tracked results into annotation layers
  • Merge modes include:
    • Fill missing only
    • Overwrite existing target points
• Extensible point tracking model registry for future additions

Tracking is intended to accelerate annotation and does not replace manual review. The current workflow is still new, we welcome all feedback, improvement suggestions and feature requests.

Bug Fixes

Trajectory plotting robustness and tracking-layer support

Refactored trajectory plotting to use a more explicit internal render state and fixed issues affecting tracking layers.

Trajectory plot boundary handling

Fixed an issue where the trajectory plot window would compress to half the window size near the start or end of a video. The slider value is now more faithful to what is shown.

Config placeholder layer handling

Refactored and clarified the logic for handling config placeholder keypoint layers.

Changes include:

• Improved user prompts when new keypoints are detected in a config file
• Clearer choices for resolving placeholder layers:
  • Apply to current
  • Keep both
  • Cancel
• Fixed a segfault caused by premature layer deletion (see below)

Premature layer removal fix

Fixed a bug caused by placeholder layers being removed too early.

Color scheme legend text synchronization

Fixed an issue where legend color boxes updated when changing colormaps, but legend text did not.

This caused:

• Mismatch between legend text and color boxes
• Incorrect colors for newly added bodyparts
• Incorrect reset behavior after hover/unhover

Legend rendering is now more consistent with the selected colormap behavior.

Tracking widget and viewer synchronization

Fixed several issues in the tracking workflow:

• Fixed the viewer resetting to the first frame after adding tracking results
• Fixed the tracking widget not syncing to the current frame on startup
• Updated tracking progress behavior to use an indeterminate progress bar while preserving CoTracker-native batching
• Fixed the info button color so it follows napari settings instead of system theme settings (also affects layer stats)

We hope these fixes and improvements make the plugin more comfortable to use. Please let us know if you have any feedback!

v0.3.0.1

Bug fixes

• Fixed an incorrect saving issue with machine labels causing likelihood columns to be retained
  • Affected projects will currently have to be saved again in napari-deeplabcut in order to allow shuffle creation to succeed in DLC.
  • Affected project will once more work as expected after DeepLabCut/DeepLabCut#3323 is merged.
  • Please open an issue if your project is affected and you cannot recover!

Full changelog

See v0.3.0.0...v0.3.0.1

v0.3.0.0: Major plugin refactor

napari-deeplabcut v0.3.0.0

This release brings a nearly-complete rewrite of the plugin, keeping a familiar UI and workflow but adding many new safety improvements around data loading/saving, improved user experience including more informative dialogs and issue reporting tools, and many quality-of-life improvements.

Important

As much of the basic workflow has been modified and updated, experienced users may have to adapt to the newest changes in the plugin, as we simplified data loading and saving. Expect a slight adjustment period as you get used to the new workflow, but we hope you will appreciate the streamlining and new features!

New features

UI additions

New status panel with folder, progress estimate, and point size slider

• New tooltip showing currently opened folder
• New "progress estimate" widget, showing the estimated % frames labeled in the current folder.

Note

The estimate should be considered a relative progress metric, rather than a requirement to reach 100% on every folder, as the estimate cannot account for non-visible keypoints.

• Global point size slider:
  • Updates the size of all keypoints in the viewer
  • Saved to config.yaml automatically, and loaded on startup
  • One can still use the native napari point size slider for individual points, but this will not be persisted.

New color scheme display interactions

• Color scheme display improvements:

  • Now only shows entries for keypoints on the current frame
  • Added a toggle to show all keypoints from config.yaml, even if they are not visible on the current frame
  • Clicking an entry now selects all matching keypoints in the frame
  • Clicking an entry in "Show all keypoints" mode will jump to the first frame where that keypoint is visible, and select all matching keypoints in that frame, if any.
  • Also works for multi-animal data:
    • Clicking an individual's entry will select all keypoints for that individual
    • Clicking on a bodypart entry will select all keypoints for that bodypart across all individuals

• Fixed & improved trails:

  • Trails now work properly in multi-animal data, showing trails for all individuals of the proper color
  • Basic visualization settings for trails are persisted across sessions

New trajectory plot interactions

• Fixed and improved trajectory plot:
  • Colors are now dynamically updated to match individuals/bodyparts in the viewer
  • Selecting points automatically filters the plot to the selected individuals/bodyparts
  • Fixed Y coordinates to match napari viewer coordinates (top-left origin)

New tutorial

New keyboard shortcuts

• Improved tutorial and keyboard shortcuts dialogs
  • Keybinds are now dynamically shown if available, and document why they are enabled/disabled based on context
    • e.g. No Points layer -> Points navigation shortcuts are disabled
  • Updated tutorial visuals

New video frame extraction and cropping menu

• Improved video frame extraction and cropping
  • Loading a video opens the usual Video Controls panel, now with additional contextual information
  • Checking "Crop with rectangle" immediately creates a Shapes layer and selects the rectangle drawing tool
  • Safer method for selecting which rectangle to use for cropping when multiple are present, and better handling of edge cases
  • Shows the crop coordinates as they will appear in config.yaml

New debug log generation utility

• Debug log generation utility
  • Opening a Keypoint Controls dialog now adds a Help > Generate napari-dlc log option
  • Generates a copy-pastable log of relevant information about the plugin and napari environment, as well as latest debug messages, to help with debugging and issue reporting

Data loading/saving fixes and improvements

• Drag & drop loading is fixed!
  • Now auto-opens Keypoint Controls on folder load
  • When saving "project-less" annotations, the plugin may ask for a config.yaml if relevant, which will write annotations relative to the specified project. Users should then move the resulting annotations folder back to the specified project.

Important

Users should not always add the config.yaml to the viewer when loading labeled data folders. The plugin now checks parent folders as per the canonical DLC project structure, and will automatically find and load the config.yaml if it is present in a parent folder.

Note that in order to start annotating from scratch or to refresh body parts after modifying the config.yaml, one still must drag and drop the updated config.yaml into the viewer.
This is currently required to re-populate the metadata for the project. Other use cases, like resuming annotation or loading existing annotations, do not require dragging and dropping the config.yaml.

• Copy-paste is fixed!
• Any save that would overwrite data now shows a confirmation dialog
  • Can be disabled with the "Warn on overwrite" option in the Keypoint Controls, if one is labeling from scratch and requires faster saves
  • Saving machine labels now clearly shows that the CollectedData file will be modified, and warns about overwriting existing data if there are conflicts with existing labels
• Improved data loading checks and remapping of frames
  • Moving data between folders or machines should cause less issues with frame remapping, and the plugin should be able to recover from more types of data loading issues without crashing or incorrectly loading data.
• Loading a video and adding the config.yaml for the placeholder Points layer, and attempting to save, now warns about the missing frame extraction step, and offers a procedure to follow the DLC workflow more closely.
• Fixed several issues around config updates when refreshing by loading the config.yaml
• Attempting to open several labeled data folders at once now warns about only supporting one folder at a time, attempts to load annotations, and skips images/video frames from the new project, rather than ending up in an inconsistent state.

Bug fixes

• Several bugs, edge cases and unsafe data handling have been fixed, and the plugin should be much more stable and less likely to crash or lose data.
• Reduced reliance on specific napari versions, and introduced a compatibility layer for any code that is closer to napari internals
  • Greatly reduced napari internals monkeypatching and API-specific code
• Better handling of moved paths, missing config, broken or partial HDF5 files, and other common issues that may arise when moving data around or working with multiple machines

Code quality improvements

• Nearly complete rewrite of the plugin codebase, with a focus on modularity, readability, and maintainability
• Better separation of operations, vastly expanded testing, better documentation of plugin invariants and assumptions
  • Introduce pydantic models for various data structures, with validation and type checking to ensure data integrity and catch issues earlier
  • Better boundaries around UI updates, I/O operations and layer lifecycle management.
• Introduced a minimal deprecation system
• New debug contexts for monitoring performance and automated log generation

Updated documentation

The newly updated documentation can be found here.

Future directions

The codebase remains in a transitory state, with many improvements but still several areas that could use more consolidated logic, centralization around validated data models, and better abstraction of I/O formats to account for future changes in DLC projects.

As always, we welcome contributions and suggestions for improvements, and will continue to work on improving the plugin in future releases!

Feel free to add to #184 if you have any refactor-specific thoughts or suggestions.

Expect a focus on fixes and improvements shortly after this major update, followed by more features and improvements, such a automated point tracking for faster and more robust labeling.

Version 0.2.1.8

What's Changed

• Fix _extract_single_frame() hdf key by @maximpavliv in #149
• Upgrade & refactor plugin by @C-Achard in #153
• Improve CI/CD and Compatibility Across Python Versions, Suppress RuntimeWarnings by @maximpavliv in #150
• Expand test coverage (writer and widgets) by @C-Achard in #156
• Add lazy_imread function to read list of images with mismatched extensions by @C-Achard in #154
• Fix & improve frame matching behavior by @C-Achard in #157
• Migrate to pyproject.toml (PEP 518/621) by @C-Achard in #165

New Contributors

• @maximpavliv made their first contribution in #149

Full Changelog: v0.2.1.7...v0.2.1.8