Can't add body parts to labeling GUI after extracting outlier frames

[Sofia-Neira]

Hi,

I extracted outlier frames and opened the labeling GUI but now I cannot add a body part when I try to fix a missing label.

I saw another user had the same issue, so I tried to update it using both methods provided in the issue (#2552) and was not successful. When I check what napari I have in the conda environment, it says napari: 0.4.18

Appreciate the help!

[MMathisLab]

Hi @Sofia-Neira ! Thanks for reporting; to clarify, are most points shown?

If so, sometimes the point can be off the frame- can you check there by zooming around? As all points always get plotted for adjustment

[Sofia-Neira]

Thank you for the quick reply!

Yes, most points are shown. It's just that the machine labels are sometimes (infrequently) missing a body part that is clearly visible.

I tried to look around off frame and did not find the missing points.

[gminami]

Hi @Sofia-Neira, @MMathisLab, I bumped into exactly the same issue with DLC 2.3.9, napari 0.4.18 and napari deeplabcut 0.2.1.7. My version of way around is to hide 'machinelabels' files in a folder and overwrite 'CollectedData' files as if I'm working on the originally extracted frames. However, if I do so, merge data function no longer works and the project iteration always stays zero (i.e., iteration-0 is overwritten, instead of creating iteration-1, 2, 3,...).
Any suggestions are appreciated.

[gminami]

Hi @C-Achard , thanks for flagging this issue elsewhere. In my case, cropping was explicitly disabled:
cropping: false
However, earlier in the config.yaml it says the following.

video_sets:
  Path\to\my\video.avi:
    crop: 0, 1280, 0, 1024

Could this setting still have an effect, even though cropping is set to false? Thanks, and apologize for my ignorance in advance!

[C-Achard]

Hello @gminami,

Apologies for the late reply; and thanks for describing your workaround.

What you mentioned is very much in line with the kind of confusing file-handling behavior we are trying to clean up in the current refactor of the plugin, e.g. #164.
If this is still relevant / reproducible on your side, it would be very helpful if you could open a dedicated issue for it. In particular, the following details would help a lot:

• Your current DLC / napari / napari-deeplabcut versions, if they changed since
• Whether this happens in standalone napari, the DLC GUI, or both
• The exact folder contents before/after the workaround and what gets loaded/ignored in the plugin

That would make it much easier to identify whether this is a pure plugin issue, a project layout problem, or an interaction with the DLC GUI.
Thanks again, and sorry this got lost earlier.

[gminami]

Hi @C-Achard ,

Thank you for the response. Please see my answers to your questions below:

1. Software Versions
   My environment has remained consistent:
   DLC: 2.3.9
   napari: 0.4.18
   napari-deeplabcut: 0.2.1.7

2. Observations
   The issue (unable to add occluded labels) persists in the DeepLabCut GUI. Per your suggestion, I also tested the standalone napari for the first time, and the same problem occurred. Additionally, I encountered a separate issue in standalone napari, which I managed to resolve with a specific workaround (see below).

3. Folder Contents & Attachments
   Please refer to the attached images for the exact folder structure:

• "Content image1": Shows the main folder with machinelabels files hidden; the remaining files are the images for labeling (e.g., imgxxxxx.png).

• "Content image2": Shows the specific contents of the temp folder.

• The following is links to machinelabels and CollectedData files.
  https://drive.google.com/file/d/16LO5rBsx3Fm6JYZPCvh5jmeIs-P_jNSF/view?usp=sharing
  https://drive.google.com/file/d/1Dc-U5g7zCotQu7bqUirKU3dZMTLZgYXa/view?usp=sharing

Issue with standalone napari (Label Mismatch)
When loading a folder containing images and machinelabels_xxxx.h5 in napari, the label-to-image alignment is incorrect.

• The Problem: The images follow the file name order (img00001, img00002, etc.), but the machinelabels_xxxx.h5 data is organized differently. For example, labels intended for img00100 are overlaid on img00001.
• Comparison: This issue does not occur when loading a folder with CollectedData_xxxx.h5; in that case, the labels and images match perfectly.
• Workaround: I found that loading the "Keypoint controls" plugin prior to loading the folder prevents this mismatch.

Thank you for your support!
Best,
Genki

[C-Achard]

Hello @gminami, thank you very much for the detailed follow-up.

Since this also occurs in standalone napari-DLC, it looks like there are two separate issues involved here:

• The original issue of not being able to add a missing keypoint during refinement
  • This has been fixed in the ongoing refactor at Plugin refactor : full version #175, which I edited just now thanks to your feedback
• The image/label remapping mismatch issue, which was previously fixed in napari-deeplabcut 0.2.1.8, specifically in Fix & improve frame matching behavior #157

If you would like to test these fixes (and hopefully avoid needing the workaround), you can install the plugin directly from this branch in the same conda environment/venv you use for DeepLabCut:
pip install --upgrade "git+https://github.com/DeepLabCut/napari-deeplabcut.git@cy/fix-multi-trails-refactor"
After installing, restart napari / the DLC GUI before testing again.
Please note that this branch is still development code and is not considered fully stable yet, so I would strongly recommend backing up your project/data folder first, just in case (if not done already). This won't affect DLC and the GUI itself, only the napari plugin.

Once you are done testing, you can reinstall your preferred released version with:
pip install napari-deeplabcut==0.2.1.8
(or 0.2.1.7 if preferred, but the latest version should fix the remapping).

If you do try the branch, it would be helpful to know:

• Whether the two issues you listed in your latest message still occur
• Whether opening the Keypoint controls first still changes the behavior (this should be fixed as well)

Hope this helps, and thanks again for your report.

Best,
Cyril