Skip to content

Commit dda78fb

Browse files
Feature/segmentation overlap box (#2362)
* workflows: add Overlap Analysis block + detections_overlaps kind New roboflow_core/overlap_analysis@v1 (also known as OverlapAnalysis) under core_steps/fusion/overlap_analysis/. Takes two sv.Detections inputs at the same dimensionality (reference_predictions, candidate_predictions) and returns, per workflow row, a list of per-pair overlap records of the new DETECTIONS_OVERLAPS_KIND (internal_data_type List[Dict[str, Any]]). Geometry mirrors the original "compute intersection / reference.area" flow: vectorised sv.box_iou_batch fast-prefilter for non-touching pairs, then shapely Polygon intersection for surviving pairs. Polygons are built from sv.mask_to_polygons (longest contour, validated) when the source detection has a mask, otherwise from the 4-corner bbox. sv.Detections always exposes a `mask` attribute, but it can be None for bbox-only sources; the helper handles both. Per-record schema documented on DETECTIONS_OVERLAPS_KIND. Optional reference_detection_id / candidate_detection_id keys appear only when the source sv.Detections.data carries detection_id. The relation is intentionally non-symmetric: the denominator is always the reference detection's area. Wiring: - entities/types.py: declare DETECTIONS_OVERLAPS_KIND with formal per-record schema in docs. - core_steps/loader.py: import the new kind + block; register the kind in load_kinds(), KINDS_SERIALIZERS (via serialize_wildcard_kind, which dispatches List[dict] recursively), and KINDS_DESERIALIZERS (via deserialize_list_of_values_kind for baseline list validation); append the block to the blocks list. * workflows: unit tests for OverlapAnalysisBlockV1 17 deterministic tests covering manifest validation (positional + alias type literals, range check), every behavioural branch of run(): - bbox-only path: full containment, partial overlap, below threshold, threshold boundary just above and just below, disjoint inputs. - mask path: rectangular masks inside identical bboxes verify the shapely / sv.mask_to_polygons branch (bbox IoU would be 1.0, mask overlap is ~0.5). - mask path fallback: a degenerate mask (<3 vertices in the longest contour) is rejected and the helper falls back to the 4-corner bbox polygon. - detection_id propagation: present-on-both, present-on-one-only. - empty inputs (left and right). - N x M cardinality with mixed overlaps and disjoint pairs. - Parity test: bbox-only fixtures cross-checked against a verbatim port of the original Python-block code, asserting equal (class-pair, ratio) record sets. No whole-output snapshots or change-detector assertions. * workflows: integration test for OverlapAnalysisBlockV1 End-to-end ExecutionEngine workflow JSON that runs two RoboflowObjectDetectionModel steps at different confidence thresholds and feeds their predictions into the OverlapAnalysis block. Assertions are shape-based (record schema, ratio range, detection_id propagation) rather than numeric so the test stays stable across model weight nudges. Registered with the workflows gallery decorator to match the convention used by every other fusion-block integration test in the suite. A second test pins min_overlap=1.0 and asserts that every emitted record has ratio == 1.0 (the only way to clear the threshold). Local env in this commit cannot compile the full loader chain because of a pre-existing missing `trackers` package import in core_steps/trackers/botsort/v1.py:5 (unrelated to this change). The test follows the same pattern as the working test_workflow_with_detections_consensus_block.py and runs in CI. * Make linters happy
1 parent 6fb9105 commit dda78fb

6 files changed

Lines changed: 998 additions & 0 deletions

File tree

inference/core/workflows/core_steps/fusion/overlap_analysis/__init__.py

Whitespace-only changes.
Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
from typing import Any, Dict, List, Literal, Optional, Tuple, Type, Union
2+
3+
import numpy as np
4+
import supervision as sv
5+
from pydantic import ConfigDict, Field
6+
from shapely.geometry import Polygon, box
7+
from supervision.config import CLASS_NAME_DATA_FIELD
8+
9+
from inference.core.workflows.execution_engine.constants import DETECTION_ID_KEY
10+
from inference.core.workflows.execution_engine.entities.base import OutputDefinition
11+
from inference.core.workflows.execution_engine.entities.types import (
12+
DETECTIONS_OVERLAPS_KIND,
13+
FLOAT_ZERO_TO_ONE_KIND,
14+
INSTANCE_SEGMENTATION_PREDICTION_KIND,
15+
OBJECT_DETECTION_PREDICTION_KIND,
16+
FloatZeroToOne,
17+
Selector,
18+
)
19+
from inference.core.workflows.prototypes.block import (
20+
BlockResult,
21+
WorkflowBlock,
22+
WorkflowBlockManifest,
23+
)
24+
25+
LONG_DESCRIPTION = """
26+
Compute pairwise geometric overlap between two sets of detections.
27+
28+
For each pair (reference, candidate) drawn from `reference_predictions` x
29+
`candidate_predictions`, the block computes
30+
`intersection_area / reference_polygon.area` and emits one record per
31+
pair whose ratio reaches `min_overlap`.
32+
33+
When a detection carries a mask, the precise polygon is the longest
34+
contour of that mask (validated via shapely); otherwise the bounding
35+
box polygon is used. A vectorised bbox-IoU prefilter
36+
(`supervision.box_iou_batch`) eliminates non-touching pairs before any
37+
shapely intersection is computed.
38+
39+
The relation is intentionally **not symmetric** across the two inputs:
40+
the denominator is always the reference detection's area. Swap the two
41+
selectors if you want overlap reported relative to the other set.
42+
43+
The output is a flat list of dicts (one per accepted pair) attached to
44+
the same dimensionality as the inputs — the block does not increase
45+
dimensionality. See the `detections_overlaps` kind docs for the
46+
per-record schema.
47+
"""
48+
49+
50+
class BlockManifest(WorkflowBlockManifest):
51+
model_config = ConfigDict(
52+
json_schema_extra={
53+
"name": "Overlap Analysis",
54+
"version": "v1",
55+
"short_description": (
56+
"Compute pairwise overlap between two sets of detections."
57+
),
58+
"long_description": LONG_DESCRIPTION,
59+
"license": "Apache-2.0",
60+
"block_type": "fusion",
61+
},
62+
protected_namespaces=(),
63+
)
64+
type: Literal["roboflow_core/overlap_analysis@v1", "OverlapAnalysis"]
65+
reference_predictions: Selector(
66+
kind=[
67+
OBJECT_DETECTION_PREDICTION_KIND,
68+
INSTANCE_SEGMENTATION_PREDICTION_KIND,
69+
]
70+
) = Field(
71+
description=(
72+
"Detections whose area is the denominator of the overlap ratio. "
73+
"For each reference detection, overlap with every candidate is "
74+
"computed; pairs above `min_overlap` appear in the output."
75+
),
76+
examples=["$steps.model_a.predictions"],
77+
)
78+
candidate_predictions: Selector(
79+
kind=[
80+
OBJECT_DETECTION_PREDICTION_KIND,
81+
INSTANCE_SEGMENTATION_PREDICTION_KIND,
82+
]
83+
) = Field(
84+
description="Detections checked against each reference.",
85+
examples=["$steps.model_b.predictions"],
86+
)
87+
min_overlap: Union[FloatZeroToOne, Selector(kind=[FLOAT_ZERO_TO_ONE_KIND])] = Field(
88+
default=0.1,
89+
description=(
90+
"Minimum (intersection / reference_area) ratio for a pair to be "
91+
"included in the output."
92+
),
93+
examples=[0.1, "$inputs.min_overlap"],
94+
)
95+
96+
@classmethod
97+
def describe_outputs(cls) -> List[OutputDefinition]:
98+
return [
99+
OutputDefinition(name="overlaps", kind=[DETECTIONS_OVERLAPS_KIND]),
100+
]
101+
102+
@classmethod
103+
def get_execution_engine_compatibility(cls) -> Optional[str]:
104+
return ">=1.3.0,<2.0.0"
105+
106+
107+
class OverlapAnalysisBlockV1(WorkflowBlock):
108+
109+
@classmethod
110+
def get_manifest(cls) -> Type[WorkflowBlockManifest]:
111+
return BlockManifest
112+
113+
def run(
114+
self,
115+
reference_predictions: sv.Detections,
116+
candidate_predictions: sv.Detections,
117+
min_overlap: float,
118+
) -> BlockResult:
119+
if len(reference_predictions) == 0 or len(candidate_predictions) == 0:
120+
return {"overlaps": []}
121+
122+
iou_matrix = sv.box_iou_batch(
123+
reference_predictions.xyxy, candidate_predictions.xyxy
124+
)
125+
126+
ref_ids = reference_predictions.data.get(DETECTION_ID_KEY)
127+
cand_ids = candidate_predictions.data.get(DETECTION_ID_KEY)
128+
ref_classes = reference_predictions.data.get(CLASS_NAME_DATA_FIELD)
129+
cand_classes = candidate_predictions.data.get(CLASS_NAME_DATA_FIELD)
130+
131+
results: List[Dict[str, Any]] = []
132+
ref_polys: Dict[int, Polygon] = {}
133+
cand_polys: Dict[int, Polygon] = {}
134+
135+
for i in range(len(reference_predictions)):
136+
for j in range(len(candidate_predictions)):
137+
if iou_matrix[i, j] <= 0.0:
138+
continue
139+
if i not in ref_polys:
140+
ref_polys[i] = _detection_to_shapely(reference_predictions, i)
141+
if j not in cand_polys:
142+
cand_polys[j] = _detection_to_shapely(candidate_predictions, j)
143+
ref_poly = ref_polys[i]
144+
cand_poly = cand_polys[j]
145+
if ref_poly.area <= 0:
146+
continue
147+
intersection_area = ref_poly.intersection(cand_poly).area
148+
overlap_ratio = intersection_area / ref_poly.area
149+
if overlap_ratio < min_overlap:
150+
continue
151+
record: Dict[str, Any] = {
152+
"reference_class": _safe_get(ref_classes, i),
153+
"reference_confidence": (
154+
float(reference_predictions.confidence[i])
155+
if reference_predictions.confidence is not None
156+
else None
157+
),
158+
"candidate_class": _safe_get(cand_classes, j),
159+
"candidate_confidence": (
160+
float(candidate_predictions.confidence[j])
161+
if candidate_predictions.confidence is not None
162+
else None
163+
),
164+
"overlap_ratio": float(overlap_ratio),
165+
}
166+
if ref_ids is not None:
167+
record["reference_detection_id"] = _safe_get(ref_ids, i)
168+
if cand_ids is not None:
169+
record["candidate_detection_id"] = _safe_get(cand_ids, j)
170+
results.append(record)
171+
return {"overlaps": results}
172+
173+
174+
def _detection_to_shapely(detections: sv.Detections, idx: int) -> Polygon:
175+
"""Return the precise polygon for the detection at `idx`.
176+
177+
When `detections.mask` is set and non-empty for this row, the polygon is
178+
the longest contour of the mask (validated via shapely). Otherwise — and
179+
on any invalidity / emptiness — the 4-corner bbox polygon is returned.
180+
`sv.Detections` always exposes a `mask` attribute, but it may be `None`
181+
for bbox-only sources (e.g. plain object detection).
182+
"""
183+
x1, y1, x2, y2 = detections.xyxy[idx]
184+
bbox_poly = box(float(x1), float(y1), float(x2), float(y2))
185+
if detections.mask is not None and np.any(detections.mask[idx]):
186+
polygons = sv.mask_to_polygons(mask=detections.mask[idx].astype(np.uint8))
187+
if polygons:
188+
longest = max(polygons, key=len)
189+
if len(longest) >= 3:
190+
candidate = Polygon([(float(pt[0]), float(pt[1])) for pt in longest])
191+
if candidate.is_valid and not candidate.is_empty:
192+
return candidate
193+
return bbox_poly
194+
195+
196+
def _safe_get(arr: Any, idx: int) -> Optional[Any]:
197+
"""Return `arr[idx]` when `arr` is not None and `idx` is within range,
198+
else `None`. Works on both numpy arrays and plain Python sequences."""
199+
if arr is None:
200+
return None
201+
try:
202+
if len(arr) <= idx:
203+
return None
204+
value = arr[idx]
205+
except (TypeError, IndexError):
206+
return None
207+
# numpy scalar -> Python scalar where applicable; keep strings as-is.
208+
if hasattr(value, "item") and not isinstance(value, (bytes, str)):
209+
try:
210+
return value.item()
211+
except (ValueError, TypeError):
212+
return value
213+
return value

inference/core/workflows/core_steps/loader.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,9 @@
194194
DimensionCollapseBlockV1,
195195
)
196196
from inference.core.workflows.core_steps.fusion.image_stack.v1 import ImageStackBlockV1
197+
from inference.core.workflows.core_steps.fusion.overlap_analysis.v1 import (
198+
OverlapAnalysisBlockV1,
199+
)
197200
from inference.core.workflows.core_steps.math.cosine_similarity.v1 import (
198201
CosineSimilarityBlockV1,
199202
)
@@ -627,6 +630,7 @@
627630
CLASSIFICATION_PREDICTION_KIND,
628631
CONTOURS_KIND,
629632
DETECTION_KIND,
633+
DETECTIONS_OVERLAPS_KIND,
630634
DICTIONARY_KIND,
631635
EMBEDDING_KIND,
632636
FLOAT_KIND,
@@ -831,6 +835,7 @@ def load_blocks() -> List[Type[WorkflowBlock]]:
831835
CropVisualizationBlockV1,
832836
DetectionsConsensusBlockV1,
833837
DetectionsStitchBlockV1,
838+
OverlapAnalysisBlockV1,
834839
DistanceMeasurementBlockV1,
835840
DominantColorBlockV1,
836841
DotVisualizationBlockV1,
@@ -1009,6 +1014,7 @@ def load_kinds() -> List[Kind]:
10091014
DICTIONARY_KIND,
10101015
DETECTION_KIND,
10111016
CLASSIFICATION_PREDICTION_KIND,
1017+
DETECTIONS_OVERLAPS_KIND,
10121018
POINT_KIND,
10131019
ZONE_KIND,
10141020
OBJECT_DETECTION_PREDICTION_KIND,

inference/core/workflows/execution_engine/entities/types.py

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -467,6 +467,55 @@ def __hash__(self) -> int:
467467
)
468468

469469

470+
DETECTIONS_OVERLAPS_KIND_DOCS = """
471+
List of per-pair overlap records computed between two sets of detections at
472+
the same dimensionality. Internally and on the wire: `List[Dict[str, Any]]`.
473+
474+
Each record is a dict with the following keys.
475+
476+
**Always present:**
477+
478+
* `reference_class: Optional[str]` —
479+
`class_name` of the reference-side detection. `None` when the source
480+
`sv.Detections` does not carry `class_name`.
481+
482+
* `reference_confidence: Optional[float]` —
483+
Confidence of the reference-side detection. `None` when the source
484+
`sv.Detections.confidence` is `None`.
485+
486+
* `candidate_class: Optional[str]` —
487+
`class_name` of the candidate-side detection (same nullability rules).
488+
489+
* `candidate_confidence: Optional[float]` —
490+
Confidence of the candidate-side detection (same nullability rules).
491+
492+
* `overlap_ratio: float` —
493+
`intersection_area(reference_polygon, candidate_polygon) / reference_polygon.area`.
494+
Range `[0.0, 1.0]`. The denominator is *always* the reference detection's
495+
area, so the relation is not symmetric across the two inputs. Polygons
496+
come from masks when available (longest contour, validated), otherwise
497+
from the bounding-box rectangle.
498+
499+
**Conditionally present** (only when the source `sv.Detections.data` carries
500+
`detection_id` for that side):
501+
502+
* `reference_detection_id: Optional[str]`
503+
* `candidate_detection_id: Optional[str]`
504+
505+
Records below the configured `min_overlap` threshold are not emitted. The
506+
top-level list is unordered with respect to pair identity; do not rely on
507+
positional indexing to cross-reference back into the input batches — use
508+
`reference_detection_id` / `candidate_detection_id` instead.
509+
"""
510+
DETECTIONS_OVERLAPS_KIND = Kind(
511+
name="detections_overlaps",
512+
description="List of per-pair detection overlap records",
513+
docs=DETECTIONS_OVERLAPS_KIND_DOCS,
514+
serialised_data_type="List[Dict[str, Any]]",
515+
internal_data_type="List[Dict[str, Any]]",
516+
)
517+
518+
470519
DETECTION_KIND_DOCS = """
471520
This kind represents single detection in prediction from a model that detects multiple elements
472521
(like object detection or instance segmentation model). It is represented as a tuple

0 commit comments

Comments
 (0)