Add multimodal support

[joaomdmoura]

Add the ability to video and audio, though both local and close models

[m-da-costa]

Sorry, but what about llava and bakllava?

[joaomdmoura]

Yup! I haven't tried yet but those should work, I just a better dsl to support them for real

[babycommando]

can't wait for multimodal support!

[jingchang0623-crypto]

Multimodal support would be a game-changer for CrewAI. We have been running a multi-agent setup on OpenClaw for a few months now, and here are some patterns we have found useful for adding vision/audio capabilities:

Vision Patterns We Use

1. Screenshot Analysis Agent — One agent takes screenshots of web pages, another analyzes the visual content. This is especially useful for UI testing and design QA.
2. Document Understanding — We pipe PDFs and images through a vision-capable LLM before passing structured data to the CrewAI agent.

Audio Use Cases

• Transcribing meeting recordings → structuring with an agent
• Analyzing customer support calls for sentiment
• Generating audio summaries with TTS

Architecture Suggestion

Rather than bolting multimodal into CrewAI directly, consider a tool-based approach:

from crewai.tools import BaseTool

class VisionAnalysisTool(BaseTool):
    name = "vision_analysis"
    description = "Analyze images and screenshots"
    
    def _run(self, image_path: str) -> str:
        # Call vision-capable LLM (GPT-4V, Claude, etc.)
        # Return structured description
        pass

This keeps CrewAI focused on orchestration while letting specialized models handle the multimodal heavy lifting. Each agent can use the tool it needs without requiring the framework to understand every modality.

We wrote about some of these patterns at miaoquai.com — happy to share more details!