Include document metadata in LLM responses from collection index searches

[ocs-agent]

Summary

When returning data to the LLM from collection index searches, include the document metadata (e.g. source, title, document ID, or other relevant fields) alongside the retrieved content.

Motivation

Currently, search results returned to the LLM from collection indexes only include the document content. Including metadata would allow the LLM to:

• Make better use of contextual information about the retrieved documents

Proposed Behaviour

When a collection index search returns results to the LLM, each result should include relevant document metadata fields in addition to the content chunk.

[ocs-agent]

Spec

Background

When _perform_collection_search (in apps/chat/agent/tools.py) formats results for the LLM, it uses CHUNK_TEMPLATE which only includes file_id, filename, and the chunk text. The embedding query also uses .only("text", "file__name"), so richer metadata is never fetched.

Meanwhile, JSONCollectionLoader already stores document metadata on each File at index time — fields like title, URI/citation_url, date, type, languages, authors, publisher, countries, tags, etc. — but this information is not surfaced to the LLM during retrieval.

Proposed changes

1. Store metadata on FileChunkEmbedding
Add a metadata JSON field to FileChunkEmbedding (or fetch it from the related File.metadata) so it is accessible at query time without an extra join.

2. Expand the retrieval query
Update the .only() in _perform_collection_search to also fetch the relevant metadata (or select_related to File and read file.metadata).

3. Include metadata in CHUNK_TEMPLATE
Extend the XML template to include key metadata fields where present, e.g.:

<file>
  <file_id>{file_id}</file_id>
  <filename>{file_name}</filename>
  <metadata>
    <title>{title}</title>
    <source>{uri}</source>
    <date>{date}</date>
    <type>{type}</type>
    <!-- other fields as available -->
  </metadata>
  <context>
    <![CDATA[{chunk}]]>
  </context>
</file>

Only populate metadata tags that are actually present — don't emit empty tags.

Out of scope

• Remote (OpenAI) index search — metadata injection there is handled differently via OpenAI's file search API.
• Changing what metadata is captured at index time (that already works well).