|
2 | 2 |
|
3 | 3 | `ImageVision` is a simple, opinionated image vision library for Elixir. It sits alongside the [`image`](https://hex.pm/packages/image) library and answers three questions about any image — **what's in it**, **where are the objects**, and **which pixels belong to which object** — with strong defaults and no ML expertise required. |
4 | 4 |
|
5 | | -## Tasks |
6 | | - |
7 | | -| Task | Module | What it does | |
8 | | -|---|---|---| |
9 | | -| Classification | [`Image.Classification`](https://hexdocs.pm/image_vision/Image.Classification.html) | Labels like `"sports car"` or `"Blenheim spaniel"` | |
10 | | -| Embedding | [`Image.Classification`](https://hexdocs.pm/image_vision/Image.Classification.html) | 768-dim feature vector for similarity search | |
11 | | -| Segmentation | [`Image.Segmentation`](https://hexdocs.pm/image_vision/Image.Segmentation.html) | Pixel masks — promptable ("cut out this object") or class-labeled ("every region in the image") | |
12 | | -| Detection | [`Image.Detection`](https://hexdocs.pm/image_vision/Image.Detection.html) | Bounding boxes with class labels | |
13 | | - |
14 | | -## Installation |
| 5 | +## Quick start |
15 | 6 |
|
16 | 7 | ```elixir |
17 | | -def deps do |
18 | | - [ |
19 | | - {:image_vision, "~> 0.2"}, |
20 | | - # Nx backend for classification (pick one) |
21 | | - {:exla, "~> 0.9"}, |
22 | | - # Required for classification |
23 | | - {:bumblebee, "~> 0.6"}, |
24 | | - # Required for segmentation and detection |
25 | | - {:ortex, "~> 0.1"} |
26 | | - ] |
27 | | -end |
28 | | -``` |
29 | | - |
30 | | -## Quick examples |
31 | | - |
32 | | -```elixir |
33 | | -# Classification — what's in this image? |
| 8 | +# Classification — what is in this image? |
34 | 9 | iex> puppy = Image.open!("puppy.jpg") |
35 | 10 | iex> Image.Classification.labels(puppy) |
36 | 11 | ["Blenheim spaniel"] |
37 | 12 |
|
38 | | -# Embedding — feature vector for similarity search |
39 | | -iex> Image.Classification.embed(puppy) |
40 | | -#Nx.Tensor<f32[768]> |
41 | | - |
42 | | -# Promptable segmentation — mask the object at the centre |
43 | | -iex> %{mask: mask} = Image.Segmentation.segment(puppy) |
44 | | -iex> cutout = Image.Segmentation.apply_mask!(puppy, mask) |
| 13 | +# Detection — where are the objects and what are they? |
| 14 | +iex> street = Image.open!("street.jpg") |
| 15 | +iex> detections = Image.Detection.detect(street) |
| 16 | +iex> hd(detections) |
| 17 | +%{label: "person", score: 0.94, box: {120, 45, 60, 180}} |
45 | 18 |
|
46 | | -# Promptable segmentation — mask the object at a specific point |
47 | | -iex> %{mask: mask} = Image.Segmentation.segment(puppy, prompt: {:point, 320, 240}) |
| 19 | +# Draw bounding boxes on the image |
| 20 | +iex> Image.Detection.draw_bbox_with_labels(detections, street) |
48 | 21 |
|
49 | | -# Class-labeled segmentation — every region |
50 | | -iex> street = Image.open!("street.jpg") |
| 22 | +# Segmentation — which pixels belong to which object? |
51 | 23 | iex> segments = Image.Segmentation.segment_panoptic(street) |
52 | 24 | iex> Enum.map(segments, & &1.label) |
53 | 25 | ["person", "car", "road", "sky"] |
54 | 26 |
|
55 | | -# Overlay coloured segments on the original image |
56 | | -iex> overlay = Image.Segmentation.compose_overlay(street, segments) |
| 27 | +# Colour-coded overlay of all segments |
| 28 | +iex> Image.Segmentation.compose_overlay(street, segments) |
57 | 29 |
|
58 | | -# Object detection — bounding boxes with labels |
59 | | -iex> detections = Image.Detection.detect(street) |
60 | | -iex> hd(detections) |
61 | | -%{label: "person", score: 0.94, box: {120, 45, 60, 180}} |
| 30 | +# Promptable segmentation — isolate the object at a specific point |
| 31 | +iex> %{mask: mask} = Image.Segmentation.segment(puppy, prompt: {:point, 320, 240}) |
| 32 | +iex> {:ok, cutout} = Image.Segmentation.apply_mask(puppy, mask) |
62 | 33 |
|
63 | | -# Draw bounding boxes on the image |
64 | | -iex> annotated = Image.Detection.draw_bbox_with_labels(detections, street) |
| 34 | +# Embedding — 768-dim feature vector for similarity search |
| 35 | +iex> Image.Classification.embed(puppy) |
| 36 | +#Nx.Tensor<f32[768]> |
| 37 | +``` |
| 38 | + |
| 39 | +## Installation |
| 40 | + |
| 41 | +Add `:image_vision` to `mix.exs` along with whichever optional ML backends you need: |
| 42 | + |
| 43 | +```elixir |
| 44 | +def deps do |
| 45 | + [ |
| 46 | + {:image_vision, "~> 0.2"}, |
| 47 | + |
| 48 | + # Required for Image.Classification and Image.Classification.embed/2 |
| 49 | + {:bumblebee, "~> 0.6"}, |
| 50 | + {:nx, "~> 0.11"}, |
| 51 | + {:exla, "~> 0.9"}, # or {:torchx, "~> 0.9"} for Torch backend |
| 52 | + |
| 53 | + # Required for Image.Detection and Image.Segmentation |
| 54 | + {:ortex, "~> 0.1"} |
| 55 | + ] |
| 56 | +end |
65 | 57 | ``` |
66 | 58 |
|
| 59 | +All ML deps are optional — omit any you do not use. The library compiles cleanly without them. |
| 60 | + |
67 | 61 | ## Default models |
68 | 62 |
|
69 | | -All defaults are permissively licensed (Apache 2.0 / MIT). Models are downloaded automatically on first call and cached to disk — no manual setup needed. |
| 63 | +All models are permissively licensed. Weights are downloaded automatically on first call and cached on disk — no manual setup required. |
70 | 64 |
|
71 | 65 | | Task | Model | License | Size | |
72 | 66 | |---|---|---|---| |
73 | | -| Classification | `facebook/convnext-tiny-224` | Apache 2.0 | ~110 MB | |
74 | | -| Embedding | `facebook/dinov2-base` | Apache 2.0 | ~340 MB | |
75 | | -| Promptable segmentation | `SharpAI/sam2-hiera-tiny-onnx` (SAM 2) | Apache 2.0 | ~150 MB | |
76 | | -| Class-labeled segmentation | `Xenova/detr-resnet-50-panoptic` | Apache 2.0 | ~175 MB | |
77 | | -| Object detection | `onnx-community/rtdetr_r50vd` (RT-DETR) | Apache 2.0 | ~175 MB | |
| 67 | +| Classification | [`facebook/convnext-tiny-224`](https://huggingface.co/facebook/convnext-tiny-224) | Apache 2.0 | ~110 MB | |
| 68 | +| Embedding | [`facebook/dinov2-base`](https://huggingface.co/facebook/dinov2-base) | Apache 2.0 | ~340 MB | |
| 69 | +| Object detection | [`onnx-community/rtdetr_r50vd`](https://huggingface.co/onnx-community/rtdetr_r50vd) | Apache 2.0 | ~175 MB | |
| 70 | +| Promptable segmentation | [`SharpAI/sam2-hiera-tiny-onnx`](https://huggingface.co/SharpAI/sam2-hiera-tiny-onnx) | Apache 2.0 | ~150 MB | |
| 71 | +| Panoptic segmentation | [`Xenova/detr-resnet-50-panoptic`](https://huggingface.co/Xenova/detr-resnet-50-panoptic) | Apache 2.0 | ~175 MB | |
78 | 72 |
|
79 | 73 | ## Configuration |
80 | 74 |
|
81 | 75 | ### Model cache |
82 | 76 |
|
83 | | -ONNX models are cached in a per-user directory by default. Override in `config/runtime.exs`: |
| 77 | +ONNX model weights are cached in a per-user directory by default (`~/.cache/image_vision` on Linux, `~/Library/Caches/image_vision` on macOS). Override in `config/runtime.exs`: |
84 | 78 |
|
85 | 79 | ```elixir |
86 | 80 | config :image_vision, :cache_dir, "/var/lib/my_app/models" |
87 | 81 | ``` |
88 | 82 |
|
89 | | -The default is `~/Library/Caches/image_vision` on macOS and `~/.cache/image_vision` on Linux. |
90 | | - |
91 | 83 | ### Classification serving |
92 | 84 |
|
93 | | -`Image.Classification` runs a supervised Bumblebee serving. It autostarts by default: |
| 85 | +`Image.Classification` runs a supervised [Bumblebee](https://hex.pm/packages/bumblebee) serving. It does not autostart by default. Start it in your application's supervision tree: |
| 86 | + |
| 87 | +```elixir |
| 88 | +# application.ex |
| 89 | +def start(_type, _args) do |
| 90 | + children = [ |
| 91 | + Image.Classification.classifier(), |
| 92 | + Image.Classification.embedder() # omit if you do not need embeddings |
| 93 | + ] |
| 94 | + |
| 95 | + Supervisor.start_link(children, strategy: :one_for_one) |
| 96 | +end |
| 97 | +``` |
| 98 | + |
| 99 | +Or enable autostart via config so `ImageVision.Supervisor` handles it: |
94 | 100 |
|
95 | 101 | ```elixir |
96 | 102 | # config/runtime.exs |
97 | | -config :image_vision, :classifier, |
98 | | - model: {:hf, "facebook/convnext-large-224-22k-1k"}, # larger model |
99 | | - featurizer: {:hf, "facebook/convnext-large-224-22k-1k"}, |
100 | | - autostart: true |
| 103 | +config :image_vision, :classifier, autostart: true |
101 | 104 | ``` |
102 | 105 |
|
103 | | -Set `autostart: false` to manage the serving in your own supervision tree: |
| 106 | +To use a different model: |
104 | 107 |
|
105 | 108 | ```elixir |
106 | | -# application.ex |
107 | | -children = [ |
108 | | - Image.Classification.classifier(), |
109 | | - Image.Classification.embedder() |
110 | | -] |
| 109 | +config :image_vision, :classifier, |
| 110 | + model: {:hf, "facebook/convnext-large-224-22k-1k"}, |
| 111 | + featurizer: {:hf, "facebook/convnext-large-224-22k-1k"}, |
| 112 | + autostart: true |
111 | 113 | ``` |
112 | 114 |
|
113 | 115 | ## Guides |
114 | 116 |
|
115 | 117 | - [Classification](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/classification.md) — classifying images and computing embeddings |
116 | | -- [Segmentation](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/segmentation.md) — promptable and class-labeled segmentation |
117 | 118 | - [Detection](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/detection.md) — bounding-box object detection |
118 | | - |
119 | | -## Design |
120 | | - |
121 | | -`ImageVision` is for developers who want answers, not ML configuration. The defaults are chosen for permissive licensing, broad applicability, and reasonable size — not raw benchmark ranking. Power users can override every default via options; most users never need to. |
122 | | - |
123 | | -See [CLAUDE.md](https://github.com/elixir-image/image_vision/blob/v0.2.0/CLAUDE.md) for the full design rationale. |
| 119 | +- [Segmentation](https://github.com/elixir-image/image_vision/blob/v0.2.0/guides/segmentation.md) — promptable and panoptic segmentation |
0 commit comments