-
Notifications
You must be signed in to change notification settings - Fork 3k
Add ImageContent documentation page #11331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
126c00d
Add ImageContent documentation page
charliesheh db37e23
Merge branch 'main' into docs-imagecontent-page
julian-risch 1d3750d
docs: address review feedback on ImageContent page
julian-risch 289d43e
Apply suggestions from code review
anakin87 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
247 changes: 247 additions & 0 deletions
247
docs-website/docs/concepts/data-classes/imagecontent.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,247 @@ | ||
| --- | ||
| title: "ImageContent" | ||
| id: imagecontent | ||
| slug: "/imagecontent" | ||
| description: "`ImageContent` represents image-based content in Haystack chat messages and multimodal pipelines." | ||
| --- | ||
|
|
||
| # ImageContent | ||
|
|
||
| `ImageContent` is a Haystack data class used to represent image-based content in chat messages and multimodal AI pipelines. | ||
|
|
||
| It is commonly used with: | ||
|
|
||
| * multimodal LLMs | ||
| * vision-language models | ||
| * image-aware chat applications | ||
| * document/image processing workflows | ||
|
|
||
| `ImageContent` stores images as base64-encoded strings together with metadata such as MIME type and image detail level. | ||
|
|
||
| If you are looking for the full API reference, see the [API documentation](/reference/data-classes-api#imagecontent). | ||
|
|
||
| --- | ||
|
|
||
| # Creating ImageContent | ||
|
|
||
| You can create an `ImageContent` object directly from a base64 string: | ||
|
|
||
| ```python | ||
| from haystack.dataclasses import ImageContent | ||
|
|
||
| image = ImageContent(base64_image="your_base64_encoded_image", mime_type="image/png") | ||
|
|
||
| print(image) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Loading Images from a File Path | ||
|
|
||
| The `from_file_path()` class method provides a convenient way to load local image files. | ||
|
|
||
| ```python | ||
| from haystack.dataclasses import ImageContent | ||
|
|
||
| image = ImageContent.from_file_path("sample.png", detail="low") | ||
|
|
||
| print(image) | ||
| ``` | ||
|
|
||
| The optional `detail` parameter is currently supported by OpenAI vision models and accepts: | ||
|
|
||
| * `"auto"` | ||
| * `"high"` | ||
| * `"low"` | ||
|
|
||
| You can also resize images while loading: | ||
|
|
||
| ```python | ||
| image = ImageContent.from_file_path("sample.png", size=(512, 512)) | ||
| ``` | ||
|
|
||
| This helps reduce: | ||
|
|
||
| * memory usage | ||
| * processing time | ||
| * payload size | ||
|
|
||
| when working with multimodal LLM APIs. | ||
|
|
||
| --- | ||
|
|
||
| # Loading Images from a URL | ||
|
|
||
| You can also create an `ImageContent` object directly from an image URL: | ||
|
|
||
| ```python | ||
| from haystack.dataclasses import ImageContent | ||
|
|
||
| image = ImageContent.from_url( | ||
| "https://images.unsplash.com/photo-1546182990-dffeafbe841d", | ||
| detail="low", | ||
| ) | ||
|
|
||
| print(image) | ||
| ``` | ||
|
|
||
| Internally, Haystack downloads the image and converts it into a base64 representation. | ||
|
|
||
| --- | ||
|
|
||
| # Producing ImageContent with Converters | ||
|
|
||
| In a pipeline, you usually don't create `ImageContent` objects by hand. Instead, you use converter components that read files and produce `ImageContent` for you: | ||
|
|
||
| * [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) converts local image files (such as PNG or JPEG) into `ImageContent` objects. | ||
| * [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) renders the pages of PDF files into `ImageContent` objects. | ||
|
|
||
| ```python | ||
| from haystack.components.converters.image import ( | ||
| ImageFileToImageContent, | ||
| PDFToImageContent, | ||
| ) | ||
|
|
||
| image_converter = ImageFileToImageContent() | ||
| image_contents = image_converter.run(sources=["image.jpg", "another_image.png"])[ | ||
| "image_contents" | ||
| ] | ||
|
|
||
| pdf_converter = PDFToImageContent() | ||
| pdf_image_contents = pdf_converter.run(sources=["file.pdf"])["image_contents"] | ||
| ``` | ||
|
|
||
| Both converters accept the optional `detail` and `size` parameters, which are forwarded to the `ImageContent` objects they create. | ||
|
|
||
| --- | ||
|
|
||
| # Using ImageContent with ChatMessage | ||
|
|
||
| `ImageContent` is commonly used together with [`ChatMessage`](chatmessage.mdx) for multimodal conversations. | ||
|
|
||
| ```python | ||
| from haystack.dataclasses import ChatMessage, ImageContent | ||
|
|
||
| image = ImageContent.from_url( | ||
| "https://images.unsplash.com/photo-1546182990-dffeafbe841d", | ||
| detail="low", | ||
| ) | ||
|
|
||
| message = ChatMessage.from_user(content_parts=["What does this image show?", image]) | ||
|
|
||
| print(message) | ||
| ``` | ||
|
|
||
| This allows multimodal LLMs to process both: | ||
|
|
||
| * textual prompts | ||
| * image inputs | ||
|
|
||
| within the same message. | ||
|
|
||
|
anakin87 marked this conversation as resolved.
|
||
| For more dynamic prompts, you can build multimodal messages with [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) using Jinja2 string templates. The `| templatize_part` filter inserts an `ImageContent` object as a structured content part instead of plain text: | ||
|
|
||
| ```python | ||
| from haystack.components.builders import ChatPromptBuilder | ||
| from haystack.dataclasses import ChatMessage, ImageContent | ||
|
|
||
| template = """ | ||
| {% message role="user" %} | ||
| Hello! I am {{user_name}}. What's the difference between the following images? | ||
| {% for image in images %} | ||
| {{ image | templatize_part }} | ||
| {% endfor %} | ||
| {% endmessage %} | ||
| """ | ||
|
|
||
| builder = ChatPromptBuilder(template=template) | ||
| images = [ | ||
| ImageContent.from_file_path("apple.jpg"), | ||
| ImageContent.from_file_path("kiwi.jpg"), | ||
| ] | ||
| result = builder.run(user_name="John", images=images) | ||
|
|
||
| print(result["prompt"]) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Metadata | ||
|
|
||
| The optional `meta` parameter allows you to attach custom metadata to the image. | ||
|
|
||
| ```python | ||
| image = ImageContent.from_url( | ||
| "https://images.unsplash.com/photo-1546182990-dffeafbe841d", | ||
| meta={"source": "example-dataset"}, | ||
| ) | ||
| ``` | ||
|
|
||
| This can be useful for: | ||
|
|
||
| * tracing | ||
| * dataset tracking | ||
| * workflow metadata | ||
| * custom application logic | ||
|
|
||
| --- | ||
|
|
||
| # Validation | ||
|
|
||
| By default, `ImageContent` validates: | ||
|
|
||
| * base64 encoding | ||
| * MIME type correctness | ||
| * image MIME compatibility | ||
|
|
||
| Validation can be disabled to improve performance: | ||
|
|
||
| ```python | ||
| image = ImageContent( | ||
| base64_image="your_base64_encoded_image", | ||
| mime_type="image/png", | ||
| validation=False, | ||
| ) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Serialization | ||
|
|
||
| `ImageContent` supports dictionary serialization. | ||
|
|
||
| ```python | ||
| image_dict = image.to_dict() | ||
|
|
||
| restored_image = ImageContent.from_dict(image_dict) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Displaying Images | ||
|
|
||
| The `show()` method can display images directly in: | ||
|
|
||
| * Jupyter notebooks | ||
| * local desktop environments | ||
|
|
||
| ```python | ||
| image.show() | ||
| ``` | ||
|
|
||
| This requires the `Pillow` package: | ||
|
|
||
| ```bash | ||
| pip install pillow | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Related Components | ||
|
|
||
| `ImageContent` is frequently used with: | ||
|
|
||
| * [`ChatMessage`](chatmessage.mdx) — to build multimodal messages | ||
| * [`ChatPromptBuilder`](../../pipeline-components/builders/chatpromptbuilder.mdx) — to template multimodal prompts | ||
| * [`ImageFileToImageContent`](../../pipeline-components/converters/imagefiletoimagecontent.mdx) — to convert image files into `ImageContent` | ||
| * [`PDFToImageContent`](../../pipeline-components/converters/pdftoimagecontent.mdx) — to convert PDF pages into `ImageContent` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.