Seb/display skeleton

[hausmanns]

Skeleton visualisation of keypoints.

demo.mov

For now hardcoded with my keypoints but should be easily adjustable to use config bodyparts.
For now, the show skeleton tick box is not connected to skeleton, as the skeleton is displayed when read_hdf is called.

Left Todos:

• Adjust skeleton to config skeleton
• Make use of the "Show Skeleton" tick box.

[jeylau]

Awesome! Let's talk on Monday to finalize it 😊

[jeylau]

I'd even add a more ambitious todo: display the skeleton edges dynamically, as a user is annotating keypoints.

[deprecated-napari-hub-preview-bot]

Preview page for your plugin is ready here:
https://preview.napari-hub.org/DeepLabCut/napari-deeplabcut/92
Created: 2023-08-30T12:38:25.604699

[MMathisLab]

I would say lets please finish or kill cc @hausmanns @jeylau

[hausmanns]

I think this was a very useful addition! We'll polish it!

[Copilot]

Pull request overview

Adds an initial “skeleton” visualization for DeepLabCut keypoints by generating a napari Vectors layer from HDF keypoints, and introduces a (currently non-functional) UI checkbox intended to control skeleton visibility.

Changes:

• Generate a skeleton Vectors layer in read_hdf from hardcoded limb pairs.
• Add a “Show skeleton” checkbox to the keypoint controls UI and enable/disable it alongside the trails toggle.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

File	Description
src/napari_deeplabcut/_widgets.py	Adds skeleton UI toggle wiring (currently stubbed) and enables/disables it with keypoint layers.
src/napari_deeplabcut/_reader.py	Adds skeleton vector generation and returns an additional Vectors layer from read_hdf.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

read_hdf previously returned a single Points layer, and the test suite asserts len(layers) == 1 for HDF inputs (and expects only one extra layer when parsing folders). Adding an unconditional Vectors layer here is an API/behavior change that will break existing callers and current tests. Consider making skeleton output opt-in (e.g., controlled by config/metadata/UI toggle) and update the tests accordingly.

Suggested change

	limb_pairs = [['nose','LeftForelimb'],
	['nose', 'RightForelimb'],
	['LeftForelimb', 'RightForelimb'],
	['LeftHindlimb', 'RightHindlimb'],
	['TailBase', 'LeftHindlimb'],
	['TailBase', 'RightHindlimb'],
	['TailBase', 'Tail1'],
	['Tail1', 'Tail2'],
	['Tail2', 'Tail3']
	]
	n = temp.shape[0]
	vectors = np.zeros((n*len(limb_pairs), 2, 3))
	for i, (kpt1, kpt2) in enumerate(limb_pairs):
	origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:,:2]
	end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:,:2]
	vec = end-origin
	vectors[i*n:(i+1)*n, 0, [2,1]] = origin
	vectors[i*n:(i+1)*n, 1, [2,1]] = vec
	vectors[i*n:(i+1)*n, :, 0] = np.arange(temp.shape[0])[:, None]
	layers.append((data, metadata, "points"))
	# Make a dictionary of the colors of the bodyparts based on the colormap
	# and the bodyparts
	layers.append((vectors, {'edge_width':1, 'edge_color':'yellow'}, "vectors"))
	layers.append((data, metadata, "points"))
	# Make a dictionary of the colors of the bodyparts based on the colormap
	# and the bodyparts
	# Optional: add a skeleton as a Vectors layer if explicitly enabled.
	# This keeps the default behavior (a single Points layer) unchanged.
	enable_skeleton = os.getenv("NAPARI_DEEPLABCUT_ENABLE_SKELETON", "").lower()
	if enable_skeleton in {"1", "true", "yes", "on"}:
	limb_pairs = [['nose', 'LeftForelimb'],
	['nose', 'RightForelimb'],
	['LeftForelimb', 'RightForelimb'],
	['LeftHindlimb', 'RightHindlimb'],
	['TailBase', 'LeftHindlimb'],
	['TailBase', 'RightHindlimb'],
	['TailBase', 'Tail1'],
	['Tail1', 'Tail2'],
	['Tail2', 'Tail3']
	]
	n = temp.shape[0]
	vectors = np.zeros((n * len(limb_pairs), 2, 3))
	for i, (kpt1, kpt2) in enumerate(limb_pairs):
	origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:, :2]
	end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:, :2]
	vec = end - origin
	vectors[i * n:(i + 1) * n, 0, [2, 1]] = origin
	vectors[i * n:(i + 1) * n, 1, [2, 1]] = vec
	vectors[i * n:(i + 1) * n, :, 0] = np.arange(temp.shape[0])[:, None]
	layers.append(
	(vectors, {'edge_width': 1, 'edge_color': 'yellow'}, "vectors")
	)

[Copilot]

The vectors layer is created without a name and without passing through the same metadata (e.g., paths, root, etc.) that other layers carry. This makes it harder to manage from the widget layer (visibility toggle, cleanup) and prevents _remap_frame_indices from remapping frame indices for the skeleton. Consider giving it a stable name (e.g., "skeleton") and copying the relevant metadata (especially paths).

Suggested change

	layers.append((vectors, {'edge_width':1, 'edge_color':'yellow'}, "vectors"))
	# Create metadata for the skeleton (vectors) layer, reusing paths/root/etc.
	vectors_metadata = metadata.copy()
	# Copy nested metadata dict so changes on one layer don't affect the other
	if "metadata" in metadata and isinstance(metadata["metadata"], dict):
	vectors_metadata["metadata"] = metadata["metadata"].copy()
	# Give the vectors layer a stable name and visual properties
	vectors_metadata.setdefault("name", "skeleton")
	vectors_metadata.setdefault("edge_width", 1)
	vectors_metadata.setdefault("edge_color", "yellow")
	layers.append((vectors, vectors_metadata, "vectors"))

[Copilot]

The skeleton vectors are computed from temp.xs(bodypart, level='bodyparts') and then truncated with to_numpy()[:, :2]. For multi-animal data this xs will include all individuals in the remaining column levels; slicing :2 effectively uses only the first individual’s x/y and silently ignores the rest. Please make the behavior explicit (e.g., generate skeleton per individual or clearly select which individual) to avoid incorrect visualization for multi-animal projects.

Suggested change

	n = temp.shape[0]
	vectors = np.zeros((n*len(limb_pairs), 2, 3))
	for i, (kpt1, kpt2) in enumerate(limb_pairs):
	origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:,:2]
	end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:,:2]
	vec = end-origin
	vectors[i*n:(i+1)*n, 0, [2,1]] = origin
	vectors[i*n:(i+1)*n, 1, [2,1]] = vec
	vectors[i*n:(i+1)*n, :, 0] = np.arange(temp.shape[0])[:, None]
	# number of frames/rows
	n = temp.shape[0]
	# number of individuals (single-animal data has a single empty-string individual)
	n_individuals = len(header.individuals)
	n_pairs = len(limb_pairs)
	# allocate vectors for each (individual, limb_pair) combination
	vectors = np.zeros((n * n_pairs * n_individuals, 2, 3))
	frame_indices = np.arange(n)[:, None]
	for ind_idx, individual in enumerate(header.individuals):
	for pair_idx, (kpt1, kpt2) in enumerate(limb_pairs):
	block_idx = ind_idx * n_pairs + pair_idx
	start = block_idx * n
	stop = (block_idx + 1) * n
	# select coordinates for this individual and bodypart
	origin_df = temp.xs(
	(individual, kpt1),
	level=("individuals", "bodyparts"),
	axis=1,
	)
	end_df = temp.xs(
	(individual, kpt2),
	level=("individuals", "bodyparts"),
	axis=1,
	)
	origin = origin_df.to_numpy()[:, :2]
	end = end_df.to_numpy()[:, :2]
	vec = end - origin
	vectors[start:stop, 0, [2, 1]] = origin
	vectors[start:stop, 1, [2, 1]] = vec
	vectors[start:stop, :, 0] = frame_indices

[Copilot]

_show_skeleton is wired to the checkbox but currently doesn’t toggle anything (it just returns True when checked, and does nothing when unchecked). Since the checkbox is enabled in on_insert, users can interact with it but nothing happens. Please implement visibility toggling for the actual skeleton layer (and handle the unchecked state), or keep the checkbox disabled/hidden until it’s functional.

Suggested change

	if state == Qt.Checked:
	# Check if "skeleton" and "skeleton_color" are in the config.yaml metadata
	return True
	"""Toggle visibility of an existing 'skeleton' layer based on checkbox state.
	If a layer named 'skeleton' exists in the viewer, its visibility will be
	toggled according to the checkbox state. If no such layer exists, this
	function is a no-op.
	"""
	skeleton_layer = None
	for layer in getattr(self.viewer, "layers", []):
	if getattr(layer, "name", None) == "skeleton":
	skeleton_layer = layer
	break
	if skeleton_layer is None:
	# No skeleton layer to toggle; nothing to do.
	return
	if state == Qt.Checked:
	skeleton_layer.visible = True
	else:
	skeleton_layer.visible = False

[Copilot]

on_remove only resets the skeleton checkbox when a Tracks layer is removed, but the skeleton layer being added by the reader is a Vectors layer. This will leave the checkbox state out of sync and won’t clear any stored reference to the skeleton layer. Please update removal handling to target the actual layer type/name used for the skeleton.

[Copilot]

Extra blank lines were introduced here; please remove them to keep formatting consistent with the rest of the file.

Suggested change

[Copilot]

These comments describe creating a bodypart→color dictionary, but no code follows and the vectors layer is appended immediately after. Please either implement the intended logic or remove/update the comments to avoid confusion.

Suggested change

	# Make a dictionary of the colors of the bodyparts based on the colormap
	# and the bodyparts
	# Add a vectors layer for limb segments, currently using a uniform edge color

[Copilot]

The skeleton construction is hardcoded to specific bodypart names (e.g., "nose", "LeftForelimb"), but read_hdf supports arbitrary DLC bodyparts. For most projects (and even this repo’s test fixture bodyparts like kpt_0), temp.xs(...) will raise a KeyError and prevent loading the file. Please derive limb pairs from the header/config and/or skip missing bodyparts gracefully so read_hdf never crashes on unknown bodyparts.

Suggested change

	limb_pairs = [['nose','LeftForelimb'],
	['nose', 'RightForelimb'],
	['LeftForelimb', 'RightForelimb'],
	['LeftHindlimb', 'RightHindlimb'],
	['TailBase', 'LeftHindlimb'],
	['TailBase', 'RightHindlimb'],
	['TailBase', 'Tail1'],
	['Tail1', 'Tail2'],
	['Tail2', 'Tail3']
	]
	n = temp.shape[0]
	vectors = np.zeros((n*len(limb_pairs), 2, 3))
	for i, (kpt1, kpt2) in enumerate(limb_pairs):
	origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:,:2]
	end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:,:2]
	vec = end-origin
	vectors[i*n:(i+1)*n, 0, [2,1]] = origin
	vectors[i*n:(i+1)*n, 1, [2,1]] = vec
	vectors[i*n:(i+1)*n, :, 0] = np.arange(temp.shape[0])[:, None]
	layers.append((data, metadata, "points"))
	# Make a dictionary of the colors of the bodyparts based on the colormap
	# and the bodyparts
	layers.append((vectors, {'edge_width':1, 'edge_color':'yellow'}, "vectors"))
	limb_pairs = [['nose', 'LeftForelimb'],
	['nose', 'RightForelimb'],
	['LeftForelimb', 'RightForelimb'],
	['LeftHindlimb', 'RightHindlimb'],
	['TailBase', 'LeftHindlimb'],
	['TailBase', 'RightHindlimb'],
	['TailBase', 'Tail1'],
	['Tail1', 'Tail2'],
	['Tail2', 'Tail3']
	]
	# Only keep limb pairs for which both bodyparts are present in this dataset
	valid_limb_pairs = [
	(kpt1, kpt2)
	for (kpt1, kpt2) in limb_pairs
	if (kpt1 in header.bodyparts and kpt2 in header.bodyparts)
	]
	n = temp.shape[0]
	if valid_limb_pairs:
	vectors = np.zeros((n * len(valid_limb_pairs), 2, 3))
	for i, (kpt1, kpt2) in enumerate(valid_limb_pairs):
	origin = temp.xs(kpt1, level='bodyparts', axis=1).to_numpy()[:, :2]
	end = temp.xs(kpt2, level='bodyparts', axis=1).to_numpy()[:, :2]
	vec = end - origin
	vectors[i * n:(i + 1) * n, 0, [2, 1]] = origin
	vectors[i * n:(i + 1) * n, 1, [2, 1]] = vec
	vectors[i * n:(i + 1) * n, :, 0] = np.arange(temp.shape[0])[:, None]
	layers.append((data, metadata, "points"))
	# Make a dictionary of the colors of the bodyparts based on the colormap
	# and the bodyparts
	if 'vectors' in locals():
	layers.append((vectors, {'edge_width': 1, 'edge_color': 'yellow'}, "vectors"))