SAM2 interactive preannotation UI does not call interactive-annotating

[thunder-007]

Describe the bug

Interactive SAM2 image segmentation does not work from the Label Studio UI even when the ML backend is connected, healthy, and returns valid masks when called directly.

The backend is not the failing part. The Label Studio frontend does not reliably enter interactive preannotation mode and does not fire the expected interactive request from the smart keypoint/rectangle tools.

Reproduction

1. Run Label Studio 1.23.0.
2. Run the segment_anything_2_image ML backend from label-studio-ml-backend.
3. Configure the project with BrushLabels, KeyPointLabels smart="true", and RectangleLabels smart="true".
4. Connect the ML backend and confirm it is marked is_interactive=true.
5. Open a task in the normal Label Studio UI.
6. Use the smart keypoint or smart rectangle tool on the image.

Expected behavior

After drawing a smart point/rectangle, the frontend should call:

POST /api/ml/{id}/interactive-annotating

with a body containing:

{
  "task": 1,
  "context": {
    "result": [...]
  }
}

Then the returned brushlabels RLE mask should be shown as a suggestion/annotation.

Actual behavior

The normal UI path sends only the non-interactive prediction/test request with context: null, for example:

{
  "params": {
    "login": null,
    "password": null,
    "context": null
  }
}

The SAM2 backend correctly returns empty predictions for that request:

{"results":[]}

Label Studio then logs:

ML backend returns an incorrect response, results field must be a list with at least one item

This error is misleading in this case because the backend was called without interactive context.

Backend verification

The same backend works when called through the interactive endpoint or directly with params.context.result.

Verified manually:

• POST /api/ml/{id}/interactive-annotating returns 200
• response contains one brushlabels result
• result contains valid value.format = "rle"
• result contains a non-empty value.rle

A minimal standalone frontend that directly calls /api/ml/{id}/interactive-annotating works with the same project, same task, same image, same backend, and same label config.

Suspected frontend cause

From the packaged source maps in Label Studio 1.23.0:

• main.js.map
  • ./src/pages/DataManager/DataManager.jsx
• 167.js.map
  • ../../libs/datamanager/src/sdk/dm-sdk.js
  • ../../libs/datamanager/src/sdk/lsf-sdk.js
• 358.js.map
  • editor smart tool/event gating

The Data Manager finds an interactive backend and enables the autoAnnotation interface, but the LSF wrapper is created without isInteractivePreannotations.

In ../../libs/datamanager/src/sdk/dm-sdk.js, initLSF() creates:

new LSFWrapper(this, element, {
  ...this.labelStudioOptions,
  task: this.store.taskStore.selected,
  preload: this.preload,
  isLabelStream: this.mode === "labelstream",
});

But ../../libs/datamanager/src/sdk/lsf-sdk.js only enables forced auto annotation if it receives:

isInteractivePreannotations: true

which then becomes:

forceAutoAnnotation: this.isInteractivePreannotations,
forceAutoAcceptSuggestions: this.isInteractivePreannotations,

Without that flag, the smart tool path does not consistently emit regionFinishedDrawing, and the Data Manager listener never calls /interactive-annotating.

Local workaround that fixed it

Patching the packaged 167.js so initLSF() passes the flag fixes the issue:

isInteractivePreannotations: this.hasInterface("autoAnnotation")

I also had to cache-bust lazy chunks because the browser can keep stale 167.js.

Version checked

Environment:

• Label Studio 1.23.0
• frontend commit reported by /api/version: 2a9bfb
• Community edition
• Browser: Chromium/Brave on Linux
• ML backend: segment_anything_2_image SAM2 image backend

I inspected the published wheels from 1.13.0 through 1.23.0 and saw the same missing bridge shape in the bundled frontend code, so downgrading one version to 1.22.0 does not appear to fix this.

Related

Possibly related to reports where SAM2 interactive masks do not appear and only empty prediction responses are seen.