Feature request: Multimodal Agents

[wakeupmh]

Use case

A vision agent for instance, to analyze files such as pdfs, as we have in the ConverseCommand using Bedrock Client direclty:

new ConverseCommand({
      modelId: process.env.BEDROCK_MODEL_ID || 'us.anthropic.claude-3-7-sonnet-20250219-v1:0',
      messages: [
        {
          role: 'user',
          content: [
            {
              text: `
                analyze this lab report document and extract the information following the format:
                ${organizerFormat}

                - Return raw JSON only without any formatting, explanation or wrapper.
                - Do not use triple backticks, markdown code blocks, or any text before or after the JSON.
                - if you do not have the information, return null for that field.
                - if you have any doubts, return null for that field.
              `,
            },
            {
              document: {
                format: DocumentFormat.PDF,
                name: objectKey,
                source: {
                  bytes: object,
                },
              },
            },
          ],
        },
      ],
      system: [
        {
          text: 'You are a document parsing assistant. You return only raw JSON without any markdown formatting, explanation, or code block syntax. Never use backticks in your response.',
        },
      ],
    } as ConverseCommandInput)

Solution/User Experience

I would like to request the ability for the multimodal agent to integrate with external data sources, such as Amazon S3 buckets.
This would allow direct processing of content (images, text, audio, etc.) stored in S3, expanding the agent's applications.

Alternative solutions