From 126c00d78e042218b986174b00492c331700ad3e Mon Sep 17 00:00:00 2001 From: charliesheh <59520443+charliesheh@users.noreply.github.com> Date: Sun, 17 May 2026 15:45:11 -0700 Subject: [PATCH 1/3] Add ImageContent documentation page --- docs-website/docs/concepts/data-classes.mdx | 6 + .../concepts/data-classes/imagecontent.mdx | 213 ++++++++++++++++++ 2 files changed, 219 insertions(+) create mode 100644 docs-website/docs/concepts/data-classes/imagecontent.mdx diff --git a/docs-website/docs/concepts/data-classes.mdx b/docs-website/docs/concepts/data-classes.mdx index 91efbc3674a..e07ec4b6f28 100644 --- a/docs-website/docs/concepts/data-classes.mdx +++ b/docs-website/docs/concepts/data-classes.mdx @@ -120,6 +120,12 @@ image = ByteStream.from_file_path("dog.jpg") Read the detailed documentation for the `ChatMessage` data class on a dedicated [ChatMessage](data-classes/chatmessage.mdx) page. +### ImageContent + +`ImageContent` represents image-based content used in multimodal chat messages and vision-language pipelines. + +Read the detailed documentation for the `ImageContent` data class on the dedicated [ImageContent](./data-classes/imagecontent) page. + ### Document #### Overview diff --git a/docs-website/docs/concepts/data-classes/imagecontent.mdx b/docs-website/docs/concepts/data-classes/imagecontent.mdx new file mode 100644 index 00000000000..6c2302751e4 --- /dev/null +++ b/docs-website/docs/concepts/data-classes/imagecontent.mdx @@ -0,0 +1,213 @@ +--- + +title: "ImageContent" +id: imagecontent +slug: "/imagecontent" +description: "`ImageContent` represents image-based content in Haystack chat messages and multimodal pipelines." +---------------------------------------------------------------------------------------------------------------- + +# ImageContent + +`ImageContent` is a Haystack data class used to represent image-based content in chat messages and multimodal AI pipelines. + +It is commonly used with: + +* multimodal LLMs +* vision-language models +* image-aware chat applications +* document/image processing workflows + +`ImageContent` stores images as base64-encoded strings together with metadata such as MIME type and image detail level. + +If you are looking for the full API reference, see the [API documentation](/reference/data-classes-api#imagecontent). + +--- + +# Creating ImageContent + +You can create an `ImageContent` object directly from a base64 string: + +```python +from haystack.dataclasses import ImageContent + +image = ImageContent( + base64_image="your_base64_encoded_image", + mime_type="image/png" +) + +print(image) +``` + +--- + +# Loading Images from a File Path + +The `from_file_path()` class method provides a convenient way to load local image files. + +```python +from haystack.dataclasses import ImageContent + +image = ImageContent.from_file_path( + "sample.png", + detail="low" +) + +print(image) +``` + +The optional `detail` parameter is currently supported by OpenAI vision models and accepts: + +* `"auto"` +* `"high"` +* `"low"` + +You can also resize images while loading: + +```python +image = ImageContent.from_file_path( + "sample.png", + size=(512, 512) +) +``` + +This helps reduce: + +* memory usage +* processing time +* payload size + +when working with multimodal LLM APIs. + +--- + +# Loading Images from a URL + +You can also create an `ImageContent` object directly from an image URL: + +```python +from haystack.dataclasses import ImageContent + +image = ImageContent.from_url( + "https://images.unsplash.com/photo-1546182990-dffeafbe841d", + detail="low" +) + +print(image) +``` + +Internally, Haystack downloads the image and converts it into a base64 representation. + +--- + +# Using ImageContent with ChatMessage + +`ImageContent` is commonly used together with `ChatMessage` for multimodal conversations. + +```python +from haystack.dataclasses import ChatMessage, ImageContent + +image = ImageContent.from_url( + "https://images.unsplash.com/photo-1546182990-dffeafbe841d", + detail="low" +) + +message = ChatMessage.from_user( + content_parts=[ + "What does this image show?", + image + ] +) + +print(message) +``` + +This allows multimodal LLMs to process both: + +* textual prompts +* image inputs + +within the same message. + +--- + +# Metadata + +The optional `meta` parameter allows you to attach custom metadata to the image. + +```python +image = ImageContent.from_url( + "https://example.com/image.png", + meta={"source": "example-dataset"} +) +``` + +This can be useful for: + +* tracing +* dataset tracking +* workflow metadata +* custom application logic + +--- + +# Validation + +By default, `ImageContent` validates: + +* base64 encoding +* MIME type correctness +* image MIME compatibility + +Validation can be disabled to improve performance: + +```python +image = ImageContent( + base64_image="your_base64_encoded_image", + mime_type="image/png", + validation=False +) +``` + +--- + +# Serialization + +`ImageContent` supports dictionary serialization. + +```python +image_dict = image.to_dict() + +restored_image = ImageContent.from_dict(image_dict) +``` + +--- + +# Displaying Images + +The `show()` method can display images directly in: + +* Jupyter notebooks +* local desktop environments + +```python +image.show() +``` + +This requires the `Pillow` package: + +```bash +pip install pillow +``` + +--- + +# Related Components + +`ImageContent` is frequently used with: + +* `ChatMessage` +* multimodal chat generators +* image converters +* vision-language pipelines +* `PDFToImageContent` +* `ImageFileToImageContent` From 1d3750d70d8f111c60a0d3571ce512f4e360ff56 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Fri, 26 Jun 2026 11:29:06 +0200 Subject: [PATCH 2/3] docs: address review feedback on ImageContent page Apply reviewer suggestions on PR #11331: - Link the ChatMessage documentation page - Add a "Producing ImageContent with Converters" section covering ImageFileToImageContent and PDFToImageContent, with links - Add a multimodal ChatPromptBuilder example using Jinja2 string templates and the | templatize_part filter - Turn the Related Components list into links to the relevant pages - Fix the broken data-classes.mdx link to the ImageContent page (use the .mdx target so Docusaurus resolves the slug) - Normalize the page frontmatter so no stray rule leaks into the body Mirror all changes into versioned_docs/version-2.30 (current latest stable docs version). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../concepts/data-classes/imagecontent.mdx | 96 ++++--- .../version-2.30/concepts/data-classes.mdx | 8 +- .../concepts/data-classes/imagecontent.mdx | 247 ++++++++++++++++++ 3 files changed, 319 insertions(+), 32 deletions(-) create mode 100644 docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx diff --git a/docs-website/docs/concepts/data-classes/imagecontent.mdx b/docs-website/docs/concepts/data-classes/imagecontent.mdx index 6c2302751e4..f5ef39911f4 100644 --- a/docs-website/docs/concepts/data-classes/imagecontent.mdx +++ b/docs-website/docs/concepts/data-classes/imagecontent.mdx @@ -1,10 +1,9 @@ --- - title: "ImageContent" id: imagecontent slug: "/imagecontent" description: "`ImageContent` represents image-based content in Haystack chat messages and multimodal pipelines." ----------------------------------------------------------------------------------------------------------------- +--- # ImageContent @@ -30,10 +29,7 @@ You can create an `ImageContent` object directly from a base64 string: ```python from haystack.dataclasses import ImageContent -image = ImageContent( - base64_image="your_base64_encoded_image", - mime_type="image/png" -) +image = ImageContent(base64_image="your_base64_encoded_image", mime_type="image/png") print(image) ``` @@ -47,10 +43,7 @@ The `from_file_path()` class method provides a convenient way to load local imag ```python from haystack.dataclasses import ImageContent -image = ImageContent.from_file_path( - "sample.png", - detail="low" -) +image = ImageContent.from_file_path("sample.png", detail="low") print(image) ``` @@ -64,10 +57,7 @@ The optional `detail` parameter is currently supported by OpenAI vision models a You can also resize images while loading: ```python -image = ImageContent.from_file_path( - "sample.png", - size=(512, 512) -) +image = ImageContent.from_file_path("sample.png", size=(512, 512)) ``` This helps reduce: @@ -89,7 +79,7 @@ from haystack.dataclasses import ImageContent image = ImageContent.from_url( "https://images.unsplash.com/photo-1546182990-dffeafbe841d", - detail="low" + detail="low", ) print(image) @@ -99,24 +89,45 @@ Internally, Haystack downloads the image and converts it into a base64 represent --- +# Producing ImageContent with Converters + +In a pipeline, you usually don't create `ImageContent` objects by hand. Instead, you use converter components that read files and produce `ImageContent` for you: + +* [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) converts local image files (such as PNG or JPEG) into `ImageContent` objects. +* [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) renders the pages of PDF files into `ImageContent` objects. + +```python +from haystack.components.converters.image import ( + ImageFileToImageContent, + PDFToImageContent, +) + +image_converter = ImageFileToImageContent() +image_contents = image_converter.run(sources=["image.jpg", "another_image.png"])[ + "image_contents" +] + +pdf_converter = PDFToImageContent() +pdf_image_contents = pdf_converter.run(sources=["file.pdf"])["image_contents"] +``` + +Both converters accept the optional `detail` and `size` parameters, which are forwarded to the `ImageContent` objects they create. + +--- + # Using ImageContent with ChatMessage -`ImageContent` is commonly used together with `ChatMessage` for multimodal conversations. +`ImageContent` is commonly used together with [`ChatMessage`](chatmessage.mdx) for multimodal conversations. ```python from haystack.dataclasses import ChatMessage, ImageContent image = ImageContent.from_url( "https://images.unsplash.com/photo-1546182990-dffeafbe841d", - detail="low" + detail="low", ) -message = ChatMessage.from_user( - content_parts=[ - "What does this image show?", - image - ] -) +message = ChatMessage.from_user(content_parts=["What does this image show?", image]) print(message) ``` @@ -128,6 +139,31 @@ This allows multimodal LLMs to process both: within the same message. +For more dynamic prompts, you can build multimodal messages with [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) using Jinja2 string templates. The `| templatize_part` filter inserts an `ImageContent` object as a structured content part instead of plain text: + +```python +from haystack.components.builders import ChatPromptBuilder +from haystack.dataclasses import ChatMessage, ImageContent + +template = """ +{% message role="user" %} +Hello! I am {{user_name}}. What's the difference between the following images? +{% for image in images %} +{{ image | templatize_part }} +{% endfor %} +{% endmessage %} +""" + +builder = ChatPromptBuilder(template=template) +images = [ + ImageContent.from_file_path("apple.jpg"), + ImageContent.from_file_path("kiwi.jpg"), +] +result = builder.run(user_name="John", images=images) + +print(result["prompt"]) +``` + --- # Metadata @@ -137,7 +173,7 @@ The optional `meta` parameter allows you to attach custom metadata to the image. ```python image = ImageContent.from_url( "https://example.com/image.png", - meta={"source": "example-dataset"} + meta={"source": "example-dataset"}, ) ``` @@ -164,7 +200,7 @@ Validation can be disabled to improve performance: image = ImageContent( base64_image="your_base64_encoded_image", mime_type="image/png", - validation=False + validation=False, ) ``` @@ -205,9 +241,7 @@ pip install pillow `ImageContent` is frequently used with: -* `ChatMessage` -* multimodal chat generators -* image converters -* vision-language pipelines -* `PDFToImageContent` -* `ImageFileToImageContent` +* [`ChatMessage`](chatmessage.mdx) — to build multimodal messages +* [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) — to template multimodal prompts +* [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) — to convert image files into `ImageContent` +* [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) — to convert PDF pages into `ImageContent` diff --git a/docs-website/versioned_docs/version-2.30/concepts/data-classes.mdx b/docs-website/versioned_docs/version-2.30/concepts/data-classes.mdx index 576c9f62214..636fa5fd34c 100644 --- a/docs-website/versioned_docs/version-2.30/concepts/data-classes.mdx +++ b/docs-website/versioned_docs/version-2.30/concepts/data-classes.mdx @@ -9,7 +9,7 @@ description: "In Haystack, there are a handful of core classes that are regularl In Haystack, there are a handful of core classes that are regularly used in many different places. These are classes that carry data through the system and you are likely to interact with these as either the input or output of your pipeline. -Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem. +Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, ImageContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem. You can check out the detailed parameters in our [Data Classes](/reference/data-classes-api) API reference. @@ -126,6 +126,12 @@ Read the detailed documentation for the `ChatMessage` data class on a dedicated Read the detailed documentation for the `FileContent` data class on a dedicated [FileContent](data-classes/filecontent.mdx) page. +### ImageContent + +`ImageContent` represents image-based content used in multimodal chat messages and vision-language pipelines. + +Read the detailed documentation for the `ImageContent` data class on a dedicated [ImageContent](data-classes/imagecontent.mdx) page. + ### Document #### Overview diff --git a/docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx b/docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx new file mode 100644 index 00000000000..f5ef39911f4 --- /dev/null +++ b/docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx @@ -0,0 +1,247 @@ +--- +title: "ImageContent" +id: imagecontent +slug: "/imagecontent" +description: "`ImageContent` represents image-based content in Haystack chat messages and multimodal pipelines." +--- + +# ImageContent + +`ImageContent` is a Haystack data class used to represent image-based content in chat messages and multimodal AI pipelines. + +It is commonly used with: + +* multimodal LLMs +* vision-language models +* image-aware chat applications +* document/image processing workflows + +`ImageContent` stores images as base64-encoded strings together with metadata such as MIME type and image detail level. + +If you are looking for the full API reference, see the [API documentation](/reference/data-classes-api#imagecontent). + +--- + +# Creating ImageContent + +You can create an `ImageContent` object directly from a base64 string: + +```python +from haystack.dataclasses import ImageContent + +image = ImageContent(base64_image="your_base64_encoded_image", mime_type="image/png") + +print(image) +``` + +--- + +# Loading Images from a File Path + +The `from_file_path()` class method provides a convenient way to load local image files. + +```python +from haystack.dataclasses import ImageContent + +image = ImageContent.from_file_path("sample.png", detail="low") + +print(image) +``` + +The optional `detail` parameter is currently supported by OpenAI vision models and accepts: + +* `"auto"` +* `"high"` +* `"low"` + +You can also resize images while loading: + +```python +image = ImageContent.from_file_path("sample.png", size=(512, 512)) +``` + +This helps reduce: + +* memory usage +* processing time +* payload size + +when working with multimodal LLM APIs. + +--- + +# Loading Images from a URL + +You can also create an `ImageContent` object directly from an image URL: + +```python +from haystack.dataclasses import ImageContent + +image = ImageContent.from_url( + "https://images.unsplash.com/photo-1546182990-dffeafbe841d", + detail="low", +) + +print(image) +``` + +Internally, Haystack downloads the image and converts it into a base64 representation. + +--- + +# Producing ImageContent with Converters + +In a pipeline, you usually don't create `ImageContent` objects by hand. Instead, you use converter components that read files and produce `ImageContent` for you: + +* [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) converts local image files (such as PNG or JPEG) into `ImageContent` objects. +* [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) renders the pages of PDF files into `ImageContent` objects. + +```python +from haystack.components.converters.image import ( + ImageFileToImageContent, + PDFToImageContent, +) + +image_converter = ImageFileToImageContent() +image_contents = image_converter.run(sources=["image.jpg", "another_image.png"])[ + "image_contents" +] + +pdf_converter = PDFToImageContent() +pdf_image_contents = pdf_converter.run(sources=["file.pdf"])["image_contents"] +``` + +Both converters accept the optional `detail` and `size` parameters, which are forwarded to the `ImageContent` objects they create. + +--- + +# Using ImageContent with ChatMessage + +`ImageContent` is commonly used together with [`ChatMessage`](chatmessage.mdx) for multimodal conversations. + +```python +from haystack.dataclasses import ChatMessage, ImageContent + +image = ImageContent.from_url( + "https://images.unsplash.com/photo-1546182990-dffeafbe841d", + detail="low", +) + +message = ChatMessage.from_user(content_parts=["What does this image show?", image]) + +print(message) +``` + +This allows multimodal LLMs to process both: + +* textual prompts +* image inputs + +within the same message. + +For more dynamic prompts, you can build multimodal messages with [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) using Jinja2 string templates. The `| templatize_part` filter inserts an `ImageContent` object as a structured content part instead of plain text: + +```python +from haystack.components.builders import ChatPromptBuilder +from haystack.dataclasses import ChatMessage, ImageContent + +template = """ +{% message role="user" %} +Hello! I am {{user_name}}. What's the difference between the following images? +{% for image in images %} +{{ image | templatize_part }} +{% endfor %} +{% endmessage %} +""" + +builder = ChatPromptBuilder(template=template) +images = [ + ImageContent.from_file_path("apple.jpg"), + ImageContent.from_file_path("kiwi.jpg"), +] +result = builder.run(user_name="John", images=images) + +print(result["prompt"]) +``` + +--- + +# Metadata + +The optional `meta` parameter allows you to attach custom metadata to the image. + +```python +image = ImageContent.from_url( + "https://example.com/image.png", + meta={"source": "example-dataset"}, +) +``` + +This can be useful for: + +* tracing +* dataset tracking +* workflow metadata +* custom application logic + +--- + +# Validation + +By default, `ImageContent` validates: + +* base64 encoding +* MIME type correctness +* image MIME compatibility + +Validation can be disabled to improve performance: + +```python +image = ImageContent( + base64_image="your_base64_encoded_image", + mime_type="image/png", + validation=False, +) +``` + +--- + +# Serialization + +`ImageContent` supports dictionary serialization. + +```python +image_dict = image.to_dict() + +restored_image = ImageContent.from_dict(image_dict) +``` + +--- + +# Displaying Images + +The `show()` method can display images directly in: + +* Jupyter notebooks +* local desktop environments + +```python +image.show() +``` + +This requires the `Pillow` package: + +```bash +pip install pillow +``` + +--- + +# Related Components + +`ImageContent` is frequently used with: + +* [`ChatMessage`](chatmessage.mdx) — to build multimodal messages +* [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) — to template multimodal prompts +* [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) — to convert image files into `ImageContent` +* [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) — to convert PDF pages into `ImageContent` From 289d43e06fb87813daf5eceb56f32c2cc0d6f7d9 Mon Sep 17 00:00:00 2001 From: Stefano Fiorucci Date: Fri, 26 Jun 2026 14:08:14 +0200 Subject: [PATCH 3/3] Apply suggestions from code review --- docs-website/docs/concepts/data-classes/imagecontent.mdx | 2 +- .../version-2.30/concepts/data-classes/imagecontent.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-website/docs/concepts/data-classes/imagecontent.mdx b/docs-website/docs/concepts/data-classes/imagecontent.mdx index f5ef39911f4..0690020a1d2 100644 --- a/docs-website/docs/concepts/data-classes/imagecontent.mdx +++ b/docs-website/docs/concepts/data-classes/imagecontent.mdx @@ -172,7 +172,7 @@ The optional `meta` parameter allows you to attach custom metadata to the image. ```python image = ImageContent.from_url( - "https://example.com/image.png", + "https://images.unsplash.com/photo-1546182990-dffeafbe841d", meta={"source": "example-dataset"}, ) ``` diff --git a/docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx b/docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx index f5ef39911f4..0690020a1d2 100644 --- a/docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx +++ b/docs-website/versioned_docs/version-2.30/concepts/data-classes/imagecontent.mdx @@ -172,7 +172,7 @@ The optional `meta` parameter allows you to attach custom metadata to the image. ```python image = ImageContent.from_url( - "https://example.com/image.png", + "https://images.unsplash.com/photo-1546182990-dffeafbe841d", meta={"source": "example-dataset"}, ) ```