feat(docs): add multi-document collection discovery guide

Author: rstrahan

docs/discovery.md

@@ -28,6 +28,7 @@ https://github.com/user-attachments/assets/9c3923fb-f4ff-43cd-a563-44c7c6132921
   - [Discovery Without Ground Truth](#discovery-without-ground-truth)
   - [Discovery With Ground Truth](#discovery-with-ground-truth)
   - [Multi-Section Package Discovery](#multi-section-package-discovery)
+  - [Multi-Document Collection Discovery](#multi-document-collection-discovery)
   - [Choosing the Right Method](#choosing-the-right-method)
 - [Configuration](#configuration)
   - [Model Configuration](#model-configuration)
@@ -84,6 +85,7 @@ This analysis produces structured configuration templates that can be used to co
 - **⚡ Real-Time Processing**: Provides immediate feedback through the web interface
 - **📊 PDF Page Thumbnails**: Visual page preview with color-coded range highlighting in the browser
 - **🔗 BDA Blueprint Automation**: Automatic BDA blueprint creation when running in BDA mode
+- **📦 Multi-Document Collection Discovery**: Discover document classes from a collection of documents using embedding-based clustering (local or S3)
 
 ### Use Cases
 
@@ -378,16 +380,154 @@ When a `page_range` is specified for a PDF, the system uses `pypdfium2` to extra
 - Each page range job is independent and can run in parallel
 - The original document is never modified
 
+### Multi-Document Collection Discovery
+
+While the other discovery methods analyze a single document at a time, **Multi-Document Collection Discovery** discovers document classes from a *collection* of documents using embedding-based clustering. Given a folder of mixed documents (e.g., invoices, W-2s, bank statements), it automatically groups similar documents together and generates a JSON Schema and classification for each group.
+
+> **Requires extra dependencies:** `pip install "idp_common[multi_document_discovery]"` or `make setup` from the project root. This installs scikit-learn, scipy, numpy, strands-agents, and pypdfium2.
+
+> **Minimum 2 documents per class:** Clusters with fewer than 2 documents are filtered as noise. Ensure you provide at least 2 documents for each expected document type.
+
+#### How It Works
+
+```mermaid
+graph LR
+    A[Document Collection] --> B[Embed]
+    B --> C[Cluster]
+    C --> D[Analyze]
+    D --> E[Reflect]
+    E --> F[Discovered Classes + Schemas]
+```
+
+1. **Embed** — Each document's first page is rendered to an image and embedded as a vector using Amazon Bedrock (`us.cohere.embed-v4:0` by default)
+2. **Cluster** — Embeddings are clustered using KMeans with automatic K selection via silhouette analysis (scikit-learn). Clusters with fewer than `min_cluster_size` (default: 2) documents are filtered as noise.
+3. **Analyze** — For each cluster, a Strands agent with Claude (`us.anthropic.claude-sonnet-4-6`) examines sample document images and generates a classification name + JSON Schema definition
+4. **Reflect** — The agent produces a Markdown reflection report reviewing all discovered classes, their relationships, and potential overlaps
+
+#### Supported File Types
+
+`.pdf`, `.png`, `.jpg`, `.jpeg`, `.tiff`, `.tif`, `.webp`
+
+#### Two Execution Modes
+
+| Mode | Documents Source | Use Case | Entry Point |
+|------|-----------------|----------|-------------|
+| **Local** | Local filesystem | CLI/SDK development, no AWS infra needed | `run_local_pipeline()` |
+| **S3** | Amazon S3 bucket | Lambda/Step Functions, production workloads | `run_full_pipeline()` |
+
+Both modes produce identical `MultiDocDiscoveryResult` output and support the same pipeline steps.
+
+#### Usage — IDP CLI
+
+The simplest way to run multi-document discovery:
+
+```bash
+# Discover classes from a directory of documents
+idp-cli discover-multidoc --dir /path/to/documents/
+
+# With explicit files
+idp-cli discover-multidoc -d invoice1.pdf -d invoice2.pdf -d w2_form.pdf -d w2_form2.pdf
+
+# Save results to a configuration version
+idp-cli discover-multidoc --dir /path/to/documents/ --save-to-config --config-version v1
+```
+
+See [IDP CLI Reference — `discover-multidoc`](idp-cli.md) for all options.
+
+#### Usage — IDP SDK
+
+```python
+from idp_sdk import IDPClient
+
+client = IDPClient()
+result = client.discovery.run_multi_doc(
+    document_dir="/path/to/documents/",
+    progress_callback=lambda step, data: print(f"{step}: {data}"),
+)
+
+print(f"Status: {result.status}")
+print(f"Found {result.total_clusters} clusters from {result.total_documents} documents")
+
+for cls in result.discovered_classes:
+    print(f"  {cls.classification} — {cls.document_count} docs")
+    if cls.json_schema:
+        print(f"    Fields: {list(cls.json_schema.get('properties', {}).keys())}")
+
+print(result.reflection_report)
+```
+
+See [IDP SDK Reference — `discovery.run_multi_doc()`](idp-sdk.md) for all parameters.
+
+#### Usage — idp_common Directly
+
+```python
+from idp_common.discovery.multi_document_discovery import MultiDocumentDiscovery
+
+discovery = MultiDocumentDiscovery(
+    region="us-east-1",
+    config={
+        "embedding_model_id": "us.cohere.embed-v4:0",
+        "analysis_model_id": "us.anthropic.claude-sonnet-4-6",
+        "min_cluster_size": 2,
+    },
+)
+
+# Local pipeline
+result = discovery.run_local_pipeline(
+    document_dir="/path/to/documents/",
+    config_version="v1",  # Optional: save to DynamoDB config
+)
+
+# Or S3 pipeline
+result = discovery.run_full_pipeline(
+    bucket="my-bucket",
+    prefix="documents/batch-001/",
+)
+```
+
+See [idp_common API Reference — MultiDocumentDiscovery](idpcommon-api-reference.md#multidocumentdiscovery--multi-document-collection-discovery) for full method-level documentation.
+
+#### Output
+
+The pipeline returns a `MultiDocDiscoveryResult` containing:
+
+| Field | Description |
+|-------|-------------|
+| `discovered_classes` | List of discovered classes, each with `classification`, `json_schema`, `document_count`, `sample_doc_ids` |
+| `reflection_report` | Markdown report analyzing all discovered classes |
+| `total_documents` | Total documents processed |
+| `num_clusters` | Number of clusters found |
+| `num_failed_embeddings` | Documents that failed embedding |
+| `num_successful_schemas` / `num_failed_schemas` | Schema generation success/failure counts |
+
+#### Web UI
+
+The Web UI includes a **Multi-Doc Discovery** tab in the Discovery panel. This tab allows you to:
+- Select a directory of documents or upload multiple files
+- Monitor pipeline progress with step-by-step status updates
+- View discovered classes and their generated schemas
+
+> **Note:** The Web UI multi-doc discovery feature requires the IDP stack to be deployed with the multi-document discovery nested stack enabled.
+
+#### Best For
+
+- **Bulk onboarding**: You have a folder of hundreds of mixed documents and want to automatically discover all document types
+- **No prior knowledge**: You don't know what types of documents are in the collection
+- **Classification + Schema in one step**: Generates both the document class names and extraction schemas simultaneously
+- **Local development**: Run from your workstation with just Bedrock model access — no AWS deployment needed
+
 ### Choosing the Right Method
 
-| Factor | Without Ground Truth | With Ground Truth |
-|--------|---------------------|-------------------|
-| **Use Case** | New document exploration | Configuration optimization |
-| **Accuracy** | Good for structure discovery | Higher accuracy for known patterns |
-| **Speed** | Fast, single-pass analysis | Optimized based on reference data |
-| **Consistency** | May vary between runs | Consistent with reference patterns |
-| **Setup Effort** | Minimal - just upload document | Requires ground truth preparation |
-| **Best For** | Unknown document types | Improving existing workflows |
+| Factor | Without Ground Truth | With Ground Truth | Multi-Section Package | Multi-Document Collection |
+|--------|---------------------|-------------------|-----------------------|--------------------------|
+| **Input** | Single document | Single document + ground truth | Single multi-page PDF | Collection of documents |
+| **Use Case** | New document exploration | Configuration optimization | Mixed document packages | Bulk class discovery |
+| **Accuracy** | Good for structure discovery | Higher for known patterns | Good per-section | Good for clustering |
+| **Speed** | Fast, single-pass | Optimized with reference | Parallel per range | Minutes for 100+ docs |
+| **Setup Effort** | Minimal | Requires ground truth | Define page ranges | Just point at a folder |
+| **Output** | 1 class + schema | 1 class + schema | N classes + schemas | N classes + schemas |
+| **Best For** | Unknown document types | Improving existing workflows | Known multi-doc packages | Unknown mixed collections |
+| **Extra Deps** | None | None | None | `multi_document_discovery` pip extra |
 
 ## Configuration