JSON collection index: add document filtering options

[ocs-agent]

Summary

Update the JSON collection index to support filtering which documents get indexed, to give users more control over index quality and content.

Requested Filters

1. Filter by metadata fields

Allow users to specify conditions on document metadata fields to include or exclude documents from indexing (e.g. only index documents where status == "published").

2. Ignore unsupported document types

Automatically skip documents with unsupported file types (e.g. .mp3, audio/video files) rather than failing or indexing them with no useful content.

3. Ignore entries with no document link

Skip entries that have no associated document link/URL, since there is nothing to index.

[ocs-agent]

Spec

Background

JSONCollectionLoader._process_item (in apps/documents/source_loaders/json_collection.py) currently has minimal filtering — it only skips items with neither a title nor a URI. Three new filtering capabilities are requested, all configurable on JSONCollectionSourceConfig.

---

1. Filter by metadata fields

Add an optional metadata_filters list to JSONCollectionSourceConfig. Each filter specifies a field name and an expected value (and optionally an operator: eq, in, exists, etc.).

During _process_item, evaluate filters against the raw item dict before any fetching. Items that don't satisfy all filters are skipped (with a debug log).

Example config:

{
  "json_url": "https://...",
  "metadata_filters": [
    {"field": "status", "operator": "eq", "value": "published"},
    {"field": "languages", "operator": "in", "value": ["en", "fr"]}
  ]
}

Start with eq and in operators; others can be added later.

---

2. Ignore unsupported document types

Currently, _fetch_and_extract downloads the attachment first and only discovers the format is unsupported when markitdown_read raises UnsupportedFormatException. For known-bad types (e.g. mp3, mp4, wav) this wastes a network round-trip.

Short-term: In _yield_attachment_documents, check the attachment's file_type field against a configurable (or hardcoded) blocklist before attempting to fetch. Skip with a warning log if the type is unsupported.

Also: Ensure the existing UnsupportedFormatException catch in _fetch_and_extract logs at warning level (not exception) and continues gracefully — it already does, but worth confirming this is consistently handled.

Suggested default blocklist: mp3, mp4, wav, ogg, avi, mov, mkv (audio/video types markitdown cannot process).

Optionally expose unsupported_file_types in JSONCollectionSourceConfig to let operators extend the blocklist.

---

3. Ignore entries with no document link

Currently, if an item has a title but no fetchable attachments, _process_item falls back to indexing just the title string as the document content. Skip any item where attachments is empty or none of the attachments have a link field, rather than falling back to the title-only document.

---

Implementation notes

• All three filters should be evaluated in _process_item before any network I/O.
• Skipped items should be logged at DEBUG or WARNING level with enough context (item title/URI) to diagnose unexpected skips.
• JSONCollectionSourceConfig changes need corresponding UI in the collection source config form (apps/documents/forms.py).