Skip to content

WIP: MCAP image-annotation decoder + camera-tile overlay#7635

Draft
mozabes wants to merge 1 commit into
feat/mmTileTitlesAndDropdownfrom
feat/mmCameraLabels
Draft

WIP: MCAP image-annotation decoder + camera-tile overlay#7635
mozabes wants to merge 1 commit into
feat/mmTileTitlesAndDropdownfrom
feat/mmCameraLabels

Conversation

@mozabes
Copy link
Copy Markdown
Contributor

@mozabes mozabes commented May 22, 2026

🔗 Related Issues

WIP. First slice of the Foxglove annotations ticket: 2D image annotations end-to-end with auto-bind by topic prefix. 3D `SceneUpdate`, sidebar overlay UX, calibration projection, and dynamic channel discovery are follow-ups.

Stacks on #7633.

📋 What changes are proposed in this pull request?

Decoder layer

  • New `IMAGE_ANNOTATIONS` visualization kind and renderer-neutral primitive types in `decoders/types.ts`: `ImageAnnotationCircle`, `ImageAnnotationPoints` (points / line-strip / line-loop / line-list), `ImageAnnotationText`, normalized `RgbaColor`.
  • New `FOXGLOVE_IMAGE_ANNOTATIONS_PAYLOAD` descriptor.
  • New `foxglove.image-annotations` decoder. Reads via the existing protobufjs schema cache; tolerates either enum-number or enum-string `type` and either camelCase or snake_case keys.
  • Registered in the Foxglove built-in decoder list.

Scene inventory

  • Hardcoded NuScenes inventory gains six `image-annotation` source entries (one per camera, `/CAM_*/annotations`) plus matching sync policies. Still POC-stubbed — dynamic channel discovery from the MCAP will replace this.

Camera-tile overlay

  • New `ImageAnnotationsOverlay` SVG renderer for circles, polylines, points, and texts. Computes the image's displayed rect explicitly (mirroring `ImageTexturePlane`'s fit math) and positions the SVG over that rect via a `ResizeObserver` so overlays stay locked to the letterboxed image regardless of tile size.
  • New `McapCameraAnnotationOverlay` (in the MCAP adapter): subscribes to one annotation topic via `useMcapTopicStream` and renders `ImageAnnotationsOverlay`.
  • `ImagePanel` exposes `onImageLoaded(width, height)` so callers can size their own overlays without coupling to the panel internals.
  • `McapCameraTile` derives the annotation topic from the selected camera by prefix (`/CAM_FRONT/image_rect_compressed` → `/CAM_FRONT/annotations`) and always renders an overlay alongside the image. Settings sidebar is just the source picker.

🧪 How is this patch tested? If it is not, please explain why.

  • `yarn check` clean across multimodal (lint, deps, types).
  • 100/100 multimodal vitest tests pass.
  • Manually verified with a NuScenes `.mcap`: bounding boxes / labels stick to the rendered image as the playhead advances and as the tile is resized.

📝 Release Notes

Is this a user-facing change?

  • No.
  • Yes.

What areas of FiftyOne does this PR affect?

  • App
  • Build
  • Core
  • Documentation
  • Other

…camera tile

Adds the data path for 2D image annotations end-to-end through the
decoder layer, with the camera tile's settings sidebar listing the
annotation streams that match the selected source. No rendering on
the image panel yet — that lands in a follow-up alongside the 3D
SceneUpdate path.

### Decoder

- New `IMAGE_ANNOTATIONS` visualization kind and renderer-neutral
  primitive types in `decoders/types.ts`: `ImageAnnotationCircle`,
  `ImageAnnotationPoints` (points / line-strip / line-loop /
  line-list), `ImageAnnotationText`, normalized `RgbaColor`.
- New `FOXGLOVE_IMAGE_ANNOTATIONS_PAYLOAD` descriptor.
- New `foxglove.image-annotations` decoder in
  `adapters/mcap/decoders/foxglove/image-annotations.ts`. Reads via
  the existing protobufjs schema cache; tolerates either enum-number
  or enum-string `type` and either camelCase or snake_case keys.
- Registered in the Foxglove built-in decoder list.

### Scene inventory + sidebar

- New `"image-annotation"` source type. The NuScenes POC inventory
  now lists per-camera annotation topics
  (`/CAM_*/image_annotations`) and their sync policies mirror the
  matching cameras.
- `McapCameraTile`'s settings sidebar replaces the placeholder
  Show-overlays / Show-bounding-boxes / Show-track-ids checkboxes
  with a real Annotations section. Annotation streams are filtered
  by topic prefix against the selected camera; each row is a
  checkbox the user can toggle. Empty state shown when no
  annotations match.
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 22, 2026

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (5)
  • WIP
  • DO NOT MERGE
  • DRAFT
  • MERGE OSS
  • OSS MERGE

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a9cdb5d8-de90-4aab-bed8-39a90b246de8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/mmCameraLabels

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant