Skip to content

Seb/display skeleton #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
hausmanns wants to merge 3 commits into
base: main
Choose a base branch
from
+51 −2
Draft

Seb/display skeleton #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fe36a98
Added keypoint matplotlib viewer
hausmanns Aug 21, 2023
f5e3bd3
Added skeleton display, still hardcoded
hausmanns Aug 21, 2023
f1757fe
Removed functions from matplotlib branch
hausmanns Aug 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions src/napari_deeplabcut/_reader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,34 @@ def read_hdf(filename: str) -> List[LayerData]:
metadata["metadata"]["root"] = os.path.split(filename)[0]
# Store file name in case the layer's name is edited by the user
metadata["metadata"]["name"] = metadata["name"]

limb_pairs = [['nose','LeftForelimb'],
['nose', 'RightForelimb'],
['LeftForelimb', 'RightForelimb'],
['LeftHindlimb', 'RightHindlimb'],
['TailBase', 'LeftHindlimb'],
['TailBase', 'RightHindlimb'],
['TailBase', 'Tail1'],
['Tail1', 'Tail2'],
['Tail2', 'Tail3']
]
n = temp.shape[0]
vectors = np.zeros((n*len(limb_pairs), 2, 3))
for i, (kpt1, kpt2) in enumerate(limb_pairs):
origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:,:2]
end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:,:2]
vec = end-origin
vectors[i*n:(i+1)*n, 0, [2,1]] = origin
vectors[i*n:(i+1)*n, 1, [2,1]] = vec
vectors[i*n:(i+1)*n, :, 0] = np.arange(temp.shape[0])[:, None]
Comment on lines +241 to +249
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skeleton vectors are computed from temp.xs(bodypart, level='bodyparts') and then truncated with to_numpy()[:, :2]. For multi-animal data this xs will include all individuals in the remaining column levels; slicing :2 effectively uses only the first individual’s x/y and silently ignores the rest. Please make the behavior explicit (e.g., generate skeleton per individual or clearly select which individual) to avoid incorrect visualization for multi-animal projects.

Suggested change
n = temp.shape[0]
vectors = np.zeros((n*len(limb_pairs), 2, 3))
for i, (kpt1, kpt2) in enumerate(limb_pairs):
origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:,:2]
end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:,:2]
vec = end-origin
vectors[i*n:(i+1)*n, 0, [2,1]] = origin
vectors[i*n:(i+1)*n, 1, [2,1]] = vec
vectors[i*n:(i+1)*n, :, 0] = np.arange(temp.shape[0])[:, None]
# number of frames/rows
n = temp.shape[0]
# number of individuals (single-animal data has a single empty-string individual)
n_individuals = len(header.individuals)
n_pairs = len(limb_pairs)
# allocate vectors for each (individual, limb_pair) combination
vectors = np.zeros((n * n_pairs * n_individuals, 2, 3))
frame_indices = np.arange(n)[:, None]
for ind_idx, individual in enumerate(header.individuals):
for pair_idx, (kpt1, kpt2) in enumerate(limb_pairs):
block_idx = ind_idx * n_pairs + pair_idx
start = block_idx * n
stop = (block_idx + 1) * n
# select coordinates for this individual and bodypart
origin_df = temp.xs(
(individual, kpt1),
level=("individuals", "bodyparts"),
axis=1,
)
end_df = temp.xs(
(individual, kpt2),
level=("individuals", "bodyparts"),
axis=1,
)
origin = origin_df.to_numpy()[:, :2]
end = end_df.to_numpy()[:, :2]
vec = end - origin
vectors[start:stop, 0, [2, 1]] = origin
vectors[start:stop, 1, [2, 1]] = vec
vectors[start:stop, :, 0] = frame_indices

Copilot uses AI. Check for mistakes.

layers.append((data, metadata, "points"))
# Make a dictionary of the colors of the bodyparts based on the colormap
# and the bodyparts

Comment on lines +252 to +254
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These comments describe creating a bodypart→color dictionary, but no code follows and the vectors layer is appended immediately after. Please either implement the intended logic or remove/update the comments to avoid confusion.

Suggested change
# Make a dictionary of the colors of the bodyparts based on the colormap
# and the bodyparts
# Add a vectors layer for limb segments, currently using a uniform edge color

Copilot uses AI. Check for mistakes.
layers.append((vectors, {'edge_width':1, 'edge_color':'yellow'}, "vectors"))
Comment on lines +231 to +255
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skeleton construction is hardcoded to specific bodypart names (e.g., "nose", "LeftForelimb"), but read_hdf supports arbitrary DLC bodyparts. For most projects (and even this repo’s test fixture bodyparts like kpt_0), temp.xs(...) will raise a KeyError and prevent loading the file. Please derive limb pairs from the header/config and/or skip missing bodyparts gracefully so read_hdf never crashes on unknown bodyparts.

Suggested change
limb_pairs = [['nose','LeftForelimb'],
['nose', 'RightForelimb'],
['LeftForelimb', 'RightForelimb'],
['LeftHindlimb', 'RightHindlimb'],
['TailBase', 'LeftHindlimb'],
['TailBase', 'RightHindlimb'],
['TailBase', 'Tail1'],
['Tail1', 'Tail2'],
['Tail2', 'Tail3']
]
n = temp.shape[0]
vectors = np.zeros((n*len(limb_pairs), 2, 3))
for i, (kpt1, kpt2) in enumerate(limb_pairs):
origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:,:2]
end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:,:2]
vec = end-origin
vectors[i*n:(i+1)*n, 0, [2,1]] = origin
vectors[i*n:(i+1)*n, 1, [2,1]] = vec
vectors[i*n:(i+1)*n, :, 0] = np.arange(temp.shape[0])[:, None]
layers.append((data, metadata, "points"))
# Make a dictionary of the colors of the bodyparts based on the colormap
# and the bodyparts
layers.append((vectors, {'edge_width':1, 'edge_color':'yellow'}, "vectors"))
limb_pairs = [['nose', 'LeftForelimb'],
['nose', 'RightForelimb'],
['LeftForelimb', 'RightForelimb'],
['LeftHindlimb', 'RightHindlimb'],
['TailBase', 'LeftHindlimb'],
['TailBase', 'RightHindlimb'],
['TailBase', 'Tail1'],
['Tail1', 'Tail2'],
['Tail2', 'Tail3']
]
# Only keep limb pairs for which both bodyparts are present in this dataset
valid_limb_pairs = [
(kpt1, kpt2)
for (kpt1, kpt2) in limb_pairs
if (kpt1 in header.bodyparts and kpt2 in header.bodyparts)
]
n = temp.shape[0]
if valid_limb_pairs:
vectors = np.zeros((n * len(valid_limb_pairs), 2, 3))
for i, (kpt1, kpt2) in enumerate(valid_limb_pairs):
origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:, :2]
end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:, :2]
vec = end - origin
vectors[i * n:(i + 1) * n, 0, [2, 1]] = origin
vectors[i * n:(i + 1) * n, 1, [2, 1]] = vec
vectors[i * n:(i + 1) * n, :, 0] = np.arange(temp.shape[0])[:, None]
layers.append((data, metadata, "points"))
# Make a dictionary of the colors of the bodyparts based on the colormap
# and the bodyparts
if 'vectors' in locals():
layers.append((vectors, {'edge_width': 1, 'edge_color': 'yellow'}, "vectors"))

Copilot uses AI. Check for mistakes.

Comment on lines +230 to +256
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

read_hdf previously returned a single Points layer, and the test suite asserts len(layers) == 1 for HDF inputs (and expects only one extra layer when parsing folders). Adding an unconditional Vectors layer here is an API/behavior change that will break existing callers and current tests. Consider making skeleton output opt-in (e.g., controlled by config/metadata/UI toggle) and update the tests accordingly.

Suggested change
limb_pairs = [['nose','LeftForelimb'],
['nose', 'RightForelimb'],
['LeftForelimb', 'RightForelimb'],
['LeftHindlimb', 'RightHindlimb'],
['TailBase', 'LeftHindlimb'],
['TailBase', 'RightHindlimb'],
['TailBase', 'Tail1'],
['Tail1', 'Tail2'],
['Tail2', 'Tail3']
]
n = temp.shape[0]
vectors = np.zeros((n*len(limb_pairs), 2, 3))
for i, (kpt1, kpt2) in enumerate(limb_pairs):
origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:,:2]
end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:,:2]
vec = end-origin
vectors[i*n:(i+1)*n, 0, [2,1]] = origin
vectors[i*n:(i+1)*n, 1, [2,1]] = vec
vectors[i*n:(i+1)*n, :, 0] = np.arange(temp.shape[0])[:, None]
layers.append((data, metadata, "points"))
# Make a dictionary of the colors of the bodyparts based on the colormap
# and the bodyparts
layers.append((vectors, {'edge_width':1, 'edge_color':'yellow'}, "vectors"))
layers.append((data, metadata, "points"))
# Make a dictionary of the colors of the bodyparts based on the colormap
# and the bodyparts
# Optional: add a skeleton as a Vectors layer if explicitly enabled.
# This keeps the default behavior (a single Points layer) unchanged.
enable_skeleton = os.getenv("NAPARI_DEEPLABCUT_ENABLE_SKELETON", "").lower()
if enable_skeleton in {"1", "true", "yes", "on"}:
limb_pairs = [['nose', 'LeftForelimb'],
['nose', 'RightForelimb'],
['LeftForelimb', 'RightForelimb'],
['LeftHindlimb', 'RightHindlimb'],
['TailBase', 'LeftHindlimb'],
['TailBase', 'RightHindlimb'],
['TailBase', 'Tail1'],
['Tail1', 'Tail2'],
['Tail2', 'Tail3']
]
n = temp.shape[0]
vectors = np.zeros((n * len(limb_pairs), 2, 3))
for i, (kpt1, kpt2) in enumerate(limb_pairs):
origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:, :2]
end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:, :2]
vec = end - origin
vectors[i * n:(i + 1) * n, 0, [2, 1]] = origin
vectors[i * n:(i + 1) * n, 1, [2, 1]] = vec
vectors[i * n:(i + 1) * n, :, 0] = np.arange(temp.shape[0])[:, None]
layers.append(
(vectors, {'edge_width': 1, 'edge_color': 'yellow'}, "vectors")
)

Copilot uses AI. Check for mistakes.

Comment on lines +255 to +257
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The vectors layer is created without a name and without passing through the same metadata (e.g., paths, root, etc.) that other layers carry. This makes it harder to manage from the widget layer (visibility toggle, cleanup) and prevents _remap_frame_indices from remapping frame indices for the skeleton. Consider giving it a stable name (e.g., "skeleton") and copying the relevant metadata (especially paths).

Suggested change
layers.append((vectors, {'edge_width':1, 'edge_color':'yellow'}, "vectors"))
# Create metadata for the skeleton (vectors) layer, reusing paths/root/etc.
vectors_metadata = metadata.copy()
# Copy nested metadata dict so changes on one layer don't affect the other
if "metadata" in metadata and isinstance(metadata["metadata"], dict):
vectors_metadata["metadata"] = metadata["metadata"].copy()
# Give the vectors layer a stable name and visual properties
vectors_metadata.setdefault("name", "skeleton")
vectors_metadata.setdefault("edge_width", 1)
vectors_metadata.setdefault("edge_color", "yellow")
layers.append((vectors, vectors_metadata, "vectors"))

Copilot uses AI. Check for mistakes.
return layers


Expand Down
26 changes: 24 additions & 2 deletions src/napari_deeplabcut/_widgets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,6 @@ def on_close(self, event, widget):
else:
event.accept()


class KeypointControls(QWidget):
def __init__(self, napari_viewer):
super().__init__()
Expand Down Expand Up @@ -354,10 +353,21 @@ def __init__(self, napari_viewer):
self._trail_cb.stateChanged.connect(self._show_trails)
self._trails = None

# Add checkbox to show skeleton
skeleton_label = QLabel("Show skeleton")
self._skeleton_cb = QCheckBox()
self._skeleton_cb.setToolTip("toggle skeleton visibility")
self._skeleton_cb.setChecked(False)
self._skeleton_cb.setEnabled(False)
self._skeleton_cb.stateChanged.connect(self._show_skeleton)
self._skeleton = None

self._view_scheme_cb = QCheckBox("Show color scheme", parent=self)

hlayout.addWidget(trail_label)
hlayout.addWidget(self._skeleton_cb)
hlayout.addWidget(skeleton_label)
hlayout.addWidget(self._trail_cb)
hlayout.addWidget(trail_label)
hlayout.addWidget(self._view_scheme_cb)

self._layout.addLayout(hlayout)
Expand Down Expand Up @@ -394,6 +404,8 @@ def __init__(self, napari_viewer):
QTimer.singleShot(10, self.start_tutorial)
self.settings.setValue("first_launch", False)



Comment on lines +407 to +408
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra blank lines were introduced here; please remove them to keep formatting consistent with the rest of the file.

Suggested change

Copilot uses AI. Check for mistakes.
@cached_property
def settings(self):
return QSettings()
Expand Down Expand Up @@ -427,6 +439,13 @@ def _show_trails(self, state):
self._trails.visible = True
elif self._trails is not None:
self._trails.visible = False


def _show_skeleton(self, state):
if state == Qt.Checked:
# Check if "skeleton" and "skeleton_color" are in the config.yaml metadata
return True
Comment on lines +445 to +447
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_show_skeleton is wired to the checkbox but currently doesn’t toggle anything (it just returns True when checked, and does nothing when unchecked). Since the checkbox is enabled in on_insert, users can interact with it but nothing happens. Please implement visibility toggling for the actual skeleton layer (and handle the unchecked state), or keep the checkbox disabled/hidden until it’s functional.

Suggested change
if state == Qt.Checked:
# Check if "skeleton" and "skeleton_color" are in the config.yaml metadata
return True
"""Toggle visibility of an existing 'skeleton' layer based on checkbox state.
If a layer named 'skeleton' exists in the viewer, its visibility will be
toggled according to the checkbox state. If no such layer exists, this
function is a no-op.
"""
skeleton_layer = None
for layer in getattr(self.viewer, "layers", []):
if getattr(layer, "name", None) == "skeleton":
skeleton_layer = layer
break
if skeleton_layer is None:
# No skeleton layer to toggle; nothing to do.
return
if state == Qt.Checked:
skeleton_layer.visible = True
else:
skeleton_layer.visible = False

Copilot uses AI. Check for mistakes.


def _form_video_action_menu(self):
group_box = QGroupBox("Video")
Expand Down Expand Up @@ -681,6 +700,7 @@ def on_insert(self, event):
}
)
self._trail_cb.setEnabled(True)
self._skeleton_cb.setEnabled(True)

# Hide the color pickers, as colormaps are strictly defined by users
controls = self.viewer.window.qt_viewer.dockLayerControls
Expand Down Expand Up @@ -710,6 +730,7 @@ def on_remove(self, event):
menu.deleteLater()
menu.destroy()
self._trail_cb.setEnabled(False)
self._skeleton_cb.setEnabled(False)
self.last_saved_label.hide()
elif isinstance(layer, Image):
self._images_meta = dict()
Expand All @@ -718,6 +739,7 @@ def on_remove(self, event):
self.video_widget.setVisible(False)
elif isinstance(layer, Tracks):
self._trail_cb.setChecked(False)
self._skeleton_cb.setChecked(False)
self._trails = None
Comment on lines 740 to 743
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on_remove only resets the skeleton checkbox when a Tracks layer is removed, but the skeleton layer being added by the reader is a Vectors layer. This will leave the checkbox state out of sync and won’t clear any stored reference to the skeleton layer. Please update removal handling to target the actual layer type/name used for the skeleton.

Copilot uses AI. Check for mistakes.

@register_points_action("Change labeling mode")
Expand Down
Loading