Skip to content

Commit 9909a69

Browse files
committed
Add tests and docs
1 parent ed09813 commit 9909a69

30 files changed

Lines changed: 960 additions & 383 deletions

README.md

Lines changed: 69 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -2,122 +2,118 @@
22

33
`ImageVision` is a simple, opinionated image vision library for Elixir. It sits alongside the [`image`](https://hex.pm/packages/image) library and answers three questions about any image — **what's in it**, **where are the objects**, and **which pixels belong to which object** — with strong defaults and no ML expertise required.
44

5-
## Tasks
6-
7-
| Task | Module | What it does |
8-
|---|---|---|
9-
| Classification | [`Image.Classification`](https://hexdocs.pm/image_vision/Image.Classification.html) | Labels like `"sports car"` or `"Blenheim spaniel"` |
10-
| Embedding | [`Image.Classification`](https://hexdocs.pm/image_vision/Image.Classification.html) | 768-dim feature vector for similarity search |
11-
| Segmentation | [`Image.Segmentation`](https://hexdocs.pm/image_vision/Image.Segmentation.html) | Pixel masks — promptable ("cut out this object") or class-labeled ("every region in the image") |
12-
| Detection | [`Image.Detection`](https://hexdocs.pm/image_vision/Image.Detection.html) | Bounding boxes with class labels |
13-
14-
## Installation
5+
## Quick start
156

167
```elixir
17-
def deps do
18-
[
19-
{:image_vision, "~> 0.2"},
20-
# Nx backend for classification (pick one)
21-
{:exla, "~> 0.9"},
22-
# Required for classification
23-
{:bumblebee, "~> 0.6"},
24-
# Required for segmentation and detection
25-
{:ortex, "~> 0.1"}
26-
]
27-
end
28-
```
29-
30-
## Quick examples
31-
32-
```elixir
33-
# Classification — what's in this image?
8+
# Classification — what is in this image?
349
iex> puppy = Image.open!("puppy.jpg")
3510
iex> Image.Classification.labels(puppy)
3611
["Blenheim spaniel"]
3712

38-
# Embedding — feature vector for similarity search
39-
iex> Image.Classification.embed(puppy)
40-
#Nx.Tensor<f32[768]>
41-
42-
# Promptable segmentation — mask the object at the centre
43-
iex> %{mask: mask} = Image.Segmentation.segment(puppy)
44-
iex> cutout = Image.Segmentation.apply_mask!(puppy, mask)
13+
# Detection — where are the objects and what are they?
14+
iex> street = Image.open!("street.jpg")
15+
iex> detections = Image.Detection.detect(street)
16+
iex> hd(detections)
17+
%{label: "person", score: 0.94, box: {120, 45, 60, 180}}
4518

46-
# Promptable segmentation — mask the object at a specific point
47-
iex> %{mask: mask} = Image.Segmentation.segment(puppy, prompt: {:point, 320, 240})
19+
# Draw bounding boxes on the image
20+
iex> Image.Detection.draw_bbox_with_labels(detections, street)
4821

49-
# Class-labeled segmentation — every region
50-
iex> street = Image.open!("street.jpg")
22+
# Segmentation — which pixels belong to which object?
5123
iex> segments = Image.Segmentation.segment_panoptic(street)
5224
iex> Enum.map(segments, & &1.label)
5325
["person", "car", "road", "sky"]
5426

55-
# Overlay coloured segments on the original image
56-
iex> overlay = Image.Segmentation.compose_overlay(street, segments)
27+
# Colour-coded overlay of all segments
28+
iex> Image.Segmentation.compose_overlay(street, segments)
5729

58-
# Object detection — bounding boxes with labels
59-
iex> detections = Image.Detection.detect(street)
60-
iex> hd(detections)
61-
%{label: "person", score: 0.94, box: {120, 45, 60, 180}}
30+
# Promptable segmentation — isolate the object at a specific point
31+
iex> %{mask: mask} = Image.Segmentation.segment(puppy, prompt: {:point, 320, 240})
32+
iex> {:ok, cutout} = Image.Segmentation.apply_mask(puppy, mask)
6233

63-
# Draw bounding boxes on the image
64-
iex> annotated = Image.Detection.draw_bbox_with_labels(detections, street)
34+
# Embedding — 768-dim feature vector for similarity search
35+
iex> Image.Classification.embed(puppy)
36+
#Nx.Tensor<f32[768]>
37+
```
38+
39+
## Installation
40+
41+
Add `:image_vision` to `mix.exs` along with whichever optional ML backends you need:
42+
43+
```elixir
44+
def deps do
45+
[
46+
{:image_vision, "~> 0.2"},
47+
48+
# Required for Image.Classification and Image.Classification.embed/2
49+
{:bumblebee, "~> 0.6"},
50+
{:nx, "~> 0.11"},
51+
{:exla, "~> 0.9"}, # or {:torchx, "~> 0.9"} for Torch backend
52+
53+
# Required for Image.Detection and Image.Segmentation
54+
{:ortex, "~> 0.1"}
55+
]
56+
end
6557
```
6658

59+
All ML deps are optional — omit any you do not use. The library compiles cleanly without them.
60+
6761
## Default models
6862

69-
All defaults are permissively licensed (Apache 2.0 / MIT). Models are downloaded automatically on first call and cached to disk — no manual setup needed.
63+
All models are permissively licensed. Weights are downloaded automatically on first call and cached on disk — no manual setup required.
7064

7165
| Task | Model | License | Size |
7266
|---|---|---|---|
73-
| Classification | `facebook/convnext-tiny-224` | Apache 2.0 | ~110 MB |
74-
| Embedding | `facebook/dinov2-base` | Apache 2.0 | ~340 MB |
75-
| Promptable segmentation | `SharpAI/sam2-hiera-tiny-onnx` (SAM 2) | Apache 2.0 | ~150 MB |
76-
| Class-labeled segmentation | `Xenova/detr-resnet-50-panoptic` | Apache 2.0 | ~175 MB |
77-
| Object detection | `onnx-community/rtdetr_r50vd` (RT-DETR) | Apache 2.0 | ~175 MB |
67+
| Classification | [`facebook/convnext-tiny-224`](https://huggingface.co/facebook/convnext-tiny-224) | Apache 2.0 | ~110 MB |
68+
| Embedding | [`facebook/dinov2-base`](https://huggingface.co/facebook/dinov2-base) | Apache 2.0 | ~340 MB |
69+
| Object detection | [`onnx-community/rtdetr_r50vd`](https://huggingface.co/onnx-community/rtdetr_r50vd) | Apache 2.0 | ~175 MB |
70+
| Promptable segmentation | [`SharpAI/sam2-hiera-tiny-onnx`](https://huggingface.co/SharpAI/sam2-hiera-tiny-onnx) | Apache 2.0 | ~150 MB |
71+
| Panoptic segmentation | [`Xenova/detr-resnet-50-panoptic`](https://huggingface.co/Xenova/detr-resnet-50-panoptic) | Apache 2.0 | ~175 MB |
7872

7973
## Configuration
8074

8175
### Model cache
8276

83-
ONNX models are cached in a per-user directory by default. Override in `config/runtime.exs`:
77+
ONNX model weights are cached in a per-user directory by default (`~/.cache/image_vision` on Linux, `~/Library/Caches/image_vision` on macOS). Override in `config/runtime.exs`:
8478

8579
```elixir
8680
config :image_vision, :cache_dir, "/var/lib/my_app/models"
8781
```
8882

89-
The default is `~/Library/Caches/image_vision` on macOS and `~/.cache/image_vision` on Linux.
90-
9183
### Classification serving
9284

93-
`Image.Classification` runs a supervised Bumblebee serving. It autostarts by default:
85+
`Image.Classification` runs a supervised [Bumblebee](https://hex.pm/packages/bumblebee) serving. It does not autostart by default. Start it in your application's supervision tree:
86+
87+
```elixir
88+
# application.ex
89+
def start(_type, _args) do
90+
children = [
91+
Image.Classification.classifier(),
92+
Image.Classification.embedder() # omit if you do not need embeddings
93+
]
94+
95+
Supervisor.start_link(children, strategy: :one_for_one)
96+
end
97+
```
98+
99+
Or enable autostart via config so `ImageVision.Supervisor` handles it:
94100

95101
```elixir
96102
# config/runtime.exs
97-
config :image_vision, :classifier,
98-
model: {:hf, "facebook/convnext-large-224-22k-1k"}, # larger model
99-
featurizer: {:hf, "facebook/convnext-large-224-22k-1k"},
100-
autostart: true
103+
config :image_vision, :classifier, autostart: true
101104
```
102105

103-
Set `autostart: false` to manage the serving in your own supervision tree:
106+
To use a different model:
104107

105108
```elixir
106-
# application.ex
107-
children = [
108-
Image.Classification.classifier(),
109-
Image.Classification.embedder()
110-
]
109+
config :image_vision, :classifier,
110+
model: {:hf, "facebook/convnext-large-224-22k-1k"},
111+
featurizer: {:hf, "facebook/convnext-large-224-22k-1k"},
112+
autostart: true
111113
```
112114

113115
## Guides
114116

115117
- [Classification](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/classification.md) — classifying images and computing embeddings
116-
- [Segmentation](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/segmentation.md) — promptable and class-labeled segmentation
117118
- [Detection](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/detection.md) — bounding-box object detection
118-
119-
## Design
120-
121-
`ImageVision` is for developers who want answers, not ML configuration. The defaults are chosen for permissive licensing, broad applicability, and reasonable size — not raw benchmark ranking. Power users can override every default via options; most users never need to.
122-
123-
See [CLAUDE.md](https://github.com/elixir-image/image_vision/blob/v0.2.0/CLAUDE.md) for the full design rationale.
119+
- [Segmentation](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/segmentation.md) — promptable and panoptic segmentation

lib/detection.ex

Lines changed: 56 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,12 @@ if ImageVision.ortex_configured?() do
99
1010
## Quick start
1111
12-
iex> street = Image.open!("./test/support/images/street.jpg")
13-
iex> [%{label: _, score: _, box: _} | _] = Image.Detection.detect(street)
12+
iex> car = Image.open!("./test/support/images/lamborghini-forsennato-concept.jpg")
13+
iex> [%{label: _, score: _, box: _} | _] = Image.Detection.detect(car)
1414
1515
## Default model
1616
17-
The default is [RT-DETR](https://huggingface.co/PekingU/rtdetr_r50vd) —
17+
The default is [RT-DETR](https://huggingface.co/onnx-community/rtdetr_r50vd) —
1818
a real-time, transformer-based detector that beats YOLOv8 on
1919
COCO and is **Apache 2.0 licensed** (unlike YOLOv8/11 which are
2020
AGPL). The ONNX export is hosted at
@@ -117,9 +117,9 @@ if ImageVision.ortex_configured?() do
117117
118118
### Examples
119119
120-
iex> street = Image.open!("./test/support/images/street.jpg")
120+
iex> car = Image.open!("./test/support/images/lamborghini-forsennato-concept.jpg")
121121
iex> [%{label: _, score: _, box: _} | _] =
122-
...> Image.Detection.detect(street, min_score: 0.5)
122+
...> Image.Detection.detect(car, min_score: 0.5)
123123
124124
"""
125125
@spec detect(image :: Vimage.t(), options :: Keyword.t()) :: [detection()]
@@ -144,58 +144,77 @@ if ImageVision.ortex_configured?() do
144144
@doc """
145145
Draws bounding boxes with class labels onto an image.
146146
147+
Builds an SVG overlay — one box and label per detection — and
148+
composites it onto the image. Each distinct class label gets a
149+
consistent colour so multiple detections of the same class are
150+
easy to identify at a glance.
151+
147152
### Arguments
148153
149154
* `detections` is the list returned from `detect/2`.
150155
151156
* `image` is the image upon which detection was run.
152157
153-
### Options
154-
155-
* `:stroke_color` is the box outline color. The default is `:red`.
156-
157-
* `:stroke_width` is the box outline thickness in pixels. The
158-
default is `4`.
159-
160-
* `:font_size` is the label font size. The default is `20`.
158+
* `options` is a keyword list of options. Currently unused;
159+
accepted for forward compatibility.
161160
162161
### Returns
163162
164163
* The annotated `t:Vix.Vips.Image.t/0`.
165164
166165
### Examples
167166
168-
iex> street = Image.open!("./test/support/images/street.jpg")
167+
iex> car = Image.open!("./test/support/images/lamborghini-forsennato-concept.jpg")
169168
iex> annotated =
170-
...> street
169+
...> car
171170
...> |> Image.Detection.detect()
172-
...> |> Image.Detection.draw_bbox_with_labels(street)
171+
...> |> Image.Detection.draw_bbox_with_labels(car)
173172
iex> match?(%Vix.Vips.Image{}, annotated)
174173
true
175174
176175
"""
177176
@spec draw_bbox_with_labels([detection()], Vimage.t(), Keyword.t()) :: Vimage.t()
178-
def draw_bbox_with_labels(detections, %Vimage{} = image, options \\ []) do
179-
stroke_color = Keyword.get(options, :stroke_color, :red)
180-
stroke_width = Keyword.get(options, :stroke_width, 4)
181-
font_size = Keyword.get(options, :font_size, 20)
182-
183-
{width, height, _bands} = Image.shape(image)
184-
185-
Enum.reduce(detections, image, fn %{label: label, box: {x, y, w, h}}, acc ->
186-
{:ok, box_image} =
187-
Image.Shape.rect(w, h, stroke_color: stroke_color, stroke_width: stroke_width)
188-
189-
{:ok, text_image} =
190-
Image.Text.text(label, text_fill_color: stroke_color, font_size: font_size)
191-
192-
acc
193-
|> Image.compose!(box_image, x: x, y: y)
194-
|> Image.compose!(text_image,
195-
x: min(max(x, 0), width) + 5,
196-
y: min(max(y, 0), height) + 5
197-
)
198-
end)
177+
def draw_bbox_with_labels(detections, %Vimage{} = image, _options \\ []) do
178+
width = Image.width(image)
179+
height = Image.height(image)
180+
181+
palette = ~w(
182+
#e6194b #3cb44b #4363d8 #f58231 #911eb4
183+
#42d4f4 #f032e6 #bfef45 #469990 #9a6324
184+
)
185+
186+
label_colors =
187+
detections
188+
|> Enum.map(& &1.label)
189+
|> Enum.uniq()
190+
|> Enum.zip(Stream.cycle(palette))
191+
|> Map.new()
192+
193+
boxes_svg =
194+
Enum.map(detections, fn %{label: label, score: score, box: {x, y, w, h}} ->
195+
color = Map.fetch!(label_colors, label)
196+
text = "#{label} #{Float.round(score * 100, 1)}%"
197+
label_y = max(0, y - 20)
198+
label_w = String.length(text) * 8 + 8
199+
200+
"""
201+
<rect x="#{x}" y="#{y}" width="#{w}" height="#{h}"
202+
fill="none" stroke="#{color}" stroke-width="2"/>
203+
<rect x="#{x}" y="#{label_y}" width="#{label_w}" height="20" fill="#{color}"/>
204+
<text x="#{x + 4}" y="#{label_y + 14}"
205+
font-family="sans-serif" font-size="13" font-weight="bold" fill="white">#{text}</text>
206+
"""
207+
end)
208+
|> Enum.join()
209+
210+
svg = """
211+
<svg xmlns="http://www.w3.org/2000/svg" width="#{width}" height="#{height}">
212+
#{boxes_svg}
213+
</svg>
214+
"""
215+
216+
{:ok, overlay} = Image.open(svg, access: :sequential)
217+
Image.compose!(image, overlay)
199218
end
200219

201220
@doc """

lib/segmentation.ex

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -245,7 +245,7 @@ if ImageVision.ortex_configured?() do
245245
246246
### Examples
247247
248-
iex> image = Image.open!("./test/support/images/street.jpg")
248+
iex> image = Image.open!("./test/support/images/puppy.webp")
249249
iex> segments = Image.Segmentation.segment_panoptic(image)
250250
iex> is_list(segments)
251251
true
@@ -345,7 +345,7 @@ if ImageVision.ortex_configured?() do
345345
346346
### Examples
347347
348-
iex> image = Image.open!("./test/support/images/street.jpg")
348+
iex> image = Image.open!("./test/support/images/puppy.webp")
349349
iex> segments = Image.Segmentation.segment_panoptic(image)
350350
iex> overlay = Image.Segmentation.compose_overlay(image, segments)
351351
iex> match?(%Vix.Vips.Image{}, overlay)

0 commit comments

Comments
 (0)