Skip to content

Commit 59379b0

Browse files
charlieshehjulian-rischclaudeanakin87
authored
Add ImageContent documentation page (#11331)
Co-authored-by: Julian Risch <julian.risch@deepset.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
1 parent 58f0578 commit 59379b0

4 files changed

Lines changed: 508 additions & 2 deletions

File tree

‎docs-website/docs/concepts/data-classes.mdx‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ description: "In Haystack, there are a handful of core classes that are regularl
99

1010
In Haystack, there are a handful of core classes that are regularly used in many different places. These are classes that carry data through the system and you are likely to interact with these as either the input or output of your pipeline.
1111

12-
Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem.
12+
Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, ImageContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem.
1313

1414
You can check out the detailed parameters in our [Data Classes](/reference/data-classes-api) API reference.
1515

@@ -126,6 +126,12 @@ Read the detailed documentation for the `ChatMessage` data class on a dedicated
126126

127127
Read the detailed documentation for the `FileContent` data class on a dedicated [FileContent](data-classes/filecontent.mdx) page.
128128

129+
### ImageContent
130+
131+
`ImageContent` represents image-based content used in multimodal chat messages and vision-language pipelines.
132+
133+
Read the detailed documentation for the `ImageContent` data class on a dedicated [ImageContent](data-classes/imagecontent.mdx) page.
134+
129135
### Document
130136

131137
#### Overview

‎docs-website/docs/concepts/data-classes/imagecontent.mdx‎

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
---
2+
title: "ImageContent"
3+
id: imagecontent
4+
slug: "/imagecontent"
5+
description: "`ImageContent` represents image-based content in Haystack chat messages and multimodal pipelines."
6+
---
7+
8+
# ImageContent
9+
10+
`ImageContent` is a Haystack data class used to represent image-based content in chat messages and multimodal AI pipelines.
11+
12+
It is commonly used with:
13+
14+
* multimodal LLMs
15+
* vision-language models
16+
* image-aware chat applications
17+
* document/image processing workflows
18+
19+
`ImageContent` stores images as base64-encoded strings together with metadata such as MIME type and image detail level.
20+
21+
If you are looking for the full API reference, see the [API documentation](/reference/data-classes-api#imagecontent).
22+
23+
---
24+
25+
# Creating ImageContent
26+
27+
You can create an `ImageContent` object directly from a base64 string:
28+
29+
```python
30+
from haystack.dataclasses import ImageContent
31+
32+
image = ImageContent(base64_image="your_base64_encoded_image", mime_type="image/png")
33+
34+
print(image)
35+
```
36+
37+
---
38+
39+
# Loading Images from a File Path
40+
41+
The `from_file_path()` class method provides a convenient way to load local image files.
42+
43+
```python
44+
from haystack.dataclasses import ImageContent
45+
46+
image = ImageContent.from_file_path("sample.png", detail="low")
47+
48+
print(image)
49+
```
50+
51+
The optional `detail` parameter is currently supported by OpenAI vision models and accepts:
52+
53+
* `"auto"`
54+
* `"high"`
55+
* `"low"`
56+
57+
You can also resize images while loading:
58+
59+
```python
60+
image = ImageContent.from_file_path("sample.png", size=(512, 512))
61+
```
62+
63+
This helps reduce:
64+
65+
* memory usage
66+
* processing time
67+
* payload size
68+
69+
when working with multimodal LLM APIs.
70+
71+
---
72+
73+
# Loading Images from a URL
74+
75+
You can also create an `ImageContent` object directly from an image URL:
76+
77+
```python
78+
from haystack.dataclasses import ImageContent
79+
80+
image = ImageContent.from_url(
81+
"https://images.unsplash.com/photo-1546182990-dffeafbe841d",
82+
detail="low",
83+
)
84+
85+
print(image)
86+
```
87+
88+
Internally, Haystack downloads the image and converts it into a base64 representation.
89+
90+
---
91+
92+
# Producing ImageContent with Converters
93+
94+
In a pipeline, you usually don't create `ImageContent` objects by hand. Instead, you use converter components that read files and produce `ImageContent` for you:
95+
96+
* [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) converts local image files (such as PNG or JPEG) into `ImageContent` objects.
97+
* [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) renders the pages of PDF files into `ImageContent` objects.
98+
99+
```python
100+
from haystack.components.converters.image import (
101+
ImageFileToImageContent,
102+
PDFToImageContent,
103+
)
104+
105+
image_converter = ImageFileToImageContent()
106+
image_contents = image_converter.run(sources=["image.jpg", "another_image.png"])[
107+
"image_contents"
108+
]
109+
110+
pdf_converter = PDFToImageContent()
111+
pdf_image_contents = pdf_converter.run(sources=["file.pdf"])["image_contents"]
112+
```
113+
114+
Both converters accept the optional `detail` and `size` parameters, which are forwarded to the `ImageContent` objects they create.
115+
116+
---
117+
118+
# Using ImageContent with ChatMessage
119+
120+
`ImageContent` is commonly used together with [`ChatMessage`](chatmessage.mdx) for multimodal conversations.
121+
122+
```python
123+
from haystack.dataclasses import ChatMessage, ImageContent
124+
125+
image = ImageContent.from_url(
126+
"https://images.unsplash.com/photo-1546182990-dffeafbe841d",
127+
detail="low",
128+
)
129+
130+
message = ChatMessage.from_user(content_parts=["What does this image show?", image])
131+
132+
print(message)
133+
```
134+
135+
This allows multimodal LLMs to process both:
136+
137+
* textual prompts
138+
* image inputs
139+
140+
within the same message.
141+
142+
For more dynamic prompts, you can build multimodal messages with [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) using Jinja2 string templates. The `| templatize_part` filter inserts an `ImageContent` object as a structured content part instead of plain text:
143+
144+
```python
145+
from haystack.components.builders import ChatPromptBuilder
146+
from haystack.dataclasses import ChatMessage, ImageContent
147+
148+
template = """
149+
{% message role="user" %}
150+
Hello! I am {{user_name}}. What's the difference between the following images?
151+
{% for image in images %}
152+
{{ image | templatize_part }}
153+
{% endfor %}
154+
{% endmessage %}
155+
"""
156+
157+
builder = ChatPromptBuilder(template=template)
158+
images = [
159+
ImageContent.from_file_path("apple.jpg"),
160+
ImageContent.from_file_path("kiwi.jpg"),
161+
]
162+
result = builder.run(user_name="John", images=images)
163+
164+
print(result["prompt"])
165+
```
166+
167+
---
168+
169+
# Metadata
170+
171+
The optional `meta` parameter allows you to attach custom metadata to the image.
172+
173+
```python
174+
image = ImageContent.from_url(
175+
"https://images.unsplash.com/photo-1546182990-dffeafbe841d",
176+
meta={"source": "example-dataset"},
177+
)
178+
```
179+
180+
This can be useful for:
181+
182+
* tracing
183+
* dataset tracking
184+
* workflow metadata
185+
* custom application logic
186+
187+
---
188+
189+
# Validation
190+
191+
By default, `ImageContent` validates:
192+
193+
* base64 encoding
194+
* MIME type correctness
195+
* image MIME compatibility
196+
197+
Validation can be disabled to improve performance:
198+
199+
```python
200+
image = ImageContent(
201+
base64_image="your_base64_encoded_image",
202+
mime_type="image/png",
203+
validation=False,
204+
)
205+
```
206+
207+
---
208+
209+
# Serialization
210+
211+
`ImageContent` supports dictionary serialization.
212+
213+
```python
214+
image_dict = image.to_dict()
215+
216+
restored_image = ImageContent.from_dict(image_dict)
217+
```
218+
219+
---
220+
221+
# Displaying Images
222+
223+
The `show()` method can display images directly in:
224+
225+
* Jupyter notebooks
226+
* local desktop environments
227+
228+
```python
229+
image.show()
230+
```
231+
232+
This requires the `Pillow` package:
233+
234+
```bash
235+
pip install pillow
236+
```
237+
238+
---
239+
240+
# Related Components
241+
242+
`ImageContent` is frequently used with:
243+
244+
* [`ChatMessage`](chatmessage.mdx) — to build multimodal messages
245+
* [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) — to template multimodal prompts
246+
* [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) — to convert image files into `ImageContent`
247+
* [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) — to convert PDF pages into `ImageContent`

‎docs-website/versioned_docs/version-2.30/concepts/data-classes.mdx‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ description: "In Haystack, there are a handful of core classes that are regularl
99

1010
In Haystack, there are a handful of core classes that are regularly used in many different places. These are classes that carry data through the system and you are likely to interact with these as either the input or output of your pipeline.
1111

12-
Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem.
12+
Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, ImageContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem.
1313

1414
You can check out the detailed parameters in our [Data Classes](/reference/data-classes-api) API reference.
1515

@@ -126,6 +126,12 @@ Read the detailed documentation for the `ChatMessage` data class on a dedicated
126126

127127
Read the detailed documentation for the `FileContent` data class on a dedicated [FileContent](data-classes/filecontent.mdx) page.
128128

129+
### ImageContent
130+
131+
`ImageContent` represents image-based content used in multimodal chat messages and vision-language pipelines.
132+
133+
Read the detailed documentation for the `ImageContent` data class on a dedicated [ImageContent](data-classes/imagecontent.mdx) page.
134+
129135
### Document
130136

131137
#### Overview

0 commit comments

Comments
 (0)