feat(testset): Add ConfBench dataset deployer

config_library/managed_config/confbench/config.yaml

@@ -0,0 +1,103 @@
+# Stack-managed configuration — overwritten on every stack update.
+# To customize, use "Save as Version" in the UI to create an editable copy.
+managed: true
+test_set: "ConfBench"
+description: "Managed config for ConfBench — an augmentation of FCC Invoices Verified with up to 18 distinct document degradation pipelines"
+
+use_bda: false
+
+notes: "ConfBench dataset settings for pipeline mode (Bedrock LLM). An augmentation of FCC Invoices Verified, each document degraded by up to 18 distinct pipelines to support confidence calibration research, OCR robustness evaluation, and KIE under realistic noise conditions. Same Invoice schema as RealKIE-FCC-Verified. Assessment and summarization disabled by default."
+
+summarization:
+  enabled: false
+
+classes:
+- $schema: https://json-schema.org/draft/2020-12/schema
+  $defs:
+    LineItem:
+      type: object
+      properties:
+        LineItemRate:
+          description: Rate for the line item. Output null if not shown.
+          x-aws-idp-confidence-threshold: '0.8'
+          x-aws-idp-evaluation-method: NUMERIC_EXACT
+          type: number
+        LineItemDays:
+          description: Days of the week as shown in the line item. Output null if
+            not shown.
+          x-aws-idp-confidence-threshold: '0.8'
+          x-aws-idp-evaluation-method: LEVENSHTEIN
+          x-aws-idp-evaluation-threshold: '0.7'
+          type: string
+        LineItemStartDate:
+          description: Start date for the line item, formatted as MM/DD/YY (2-digit year, e.g. 10/22/16). Output null if not shown.
+          x-aws-idp-confidence-threshold: '0.8'
+          x-aws-idp-evaluation-method: DATE
+          x-aws-idp-evaluation-threshold: '0.7'
+          type: string
+        LineItemEndDate:
+          description: End date for the line item, formatted as MM/DD/YY (2-digit year, e.g. 10/22/16). Output null if not shown.
+          x-aws-idp-confidence-threshold: '0.8'
+          x-aws-idp-evaluation-method: DATE
+          x-aws-idp-evaluation-threshold: '0.7'
+          type: string
+        LineItemDescription:
+          description: Description of the line item. Output null if not shown.
+          x-aws-idp-evaluation-method: LEVENSHTEIN
+          x-aws-idp-evaluation-threshold: '0.7'
+          type: string
+  description: Invoice document
+  type: object
+  x-aws-idp-document-type: Invoice
+  properties:
+    Agency:
+      description: The agency the invoice is addressed to
+      x-aws-idp-confidence-threshold: '0.8'
+      x-aws-idp-evaluation-weight: '2'
+      x-aws-idp-evaluation-method: LEVENSHTEIN
+      x-aws-idp-evaluation-threshold: '0.7'
+      type: string
+    Advertiser:
+      description: The name of the advertiser
+      x-aws-idp-confidence-threshold: '0.8'
+      x-aws-idp-evaluation-weight: '2'
+      x-aws-idp-evaluation-method: FUZZY
+      x-aws-idp-evaluation-threshold: '0.8'
+      type: string
+    GrossTotal:
+      description: The gross total amount. Output null if not shown.
+      x-aws-idp-evaluation-weight: '2'
+      x-aws-idp-confidence-threshold: '0.8'
+      x-aws-idp-evaluation-method: NUMERIC_EXACT
+      type: number
+    PaymentTerms:
+      description: Terms of payment. Output null if not shown.
+      x-aws-idp-evaluation-weight: '0.2'
+      x-aws-idp-evaluation-method: FUZZY
+      x-aws-idp-evaluation-threshold: '0.7'
+      type: string
+    AgencyCommission:
+      description: The agency commission amount. Output null if not shown.
+      x-aws-idp-evaluation-weight: '0.2'
+      x-aws-idp-confidence-threshold: '0.8'
+      x-aws-idp-evaluation-method: NUMERIC_EXACT
+      type: number
+    NetAmountDue:
+      description: The net amount due after commission. Output null if not shown.
+      x-aws-idp-evaluation-weight: '2'
+      x-aws-idp-confidence-threshold: '0.8'
+      x-aws-idp-evaluation-method: NUMERIC_EXACT
+      type: number
+    LineItems:
+      type: array
+      description: List of line item details on the invoice; each item has several
+        possible elements
+      items:
+        $ref: '#/$defs/LineItem'
+  required:
+  - Agency
+  - Advertiser
+  - LineItems
+  $id: Invoice
+rule_validation:
+  enabled: false

docs/test-studio.md

@@ -39,6 +39,7 @@ The accelerator automatically deploys **four benchmark datasets** from HuggingFa
 2. **OmniAI-OCR-Benchmark**: 293 diverse document images across 9 formats
 3. **DocSplit-Poly-Seq**: 500 multi-page packets with 13 document types
 4. **Fake-W2-Tax-Forms**: 2,000 synthetic US W-2 tax form images with 45-field ground truth
+5. **ConfBench**: 1,346 FCC invoice documents with noise augmentation (75 originals × up to 18 degradation variants)
 
 All datasets are deployed automatically with zero manual steps required. Each test set has a corresponding **managed configuration version** (e.g., `fake-w2`, `docsplit`) that is auto-selected in Test Studio when the test set is chosen. See [Configuration — Managed Configuration Versions](configuration.md#managed-configuration-versions) for details.
 
@@ -78,6 +79,62 @@ Use with: `config_library/unified/realkie-fcc-verified/config.yaml`
 
 ---
 
+### ConfBench
+
+**Source**: https://huggingface.co/datasets/amazon/ConfBench
+
+ConfBench extends the 75 verified FCC invoice documents from RealKIE-FCC-Verified with up to 18 noise-augmented variants per document, producing 1,346 (document, noise\_variant) pairs. It is designed for confidence calibration research, OCR robustness evaluation, and key information extraction (KIE) benchmarking under realistic document degradation conditions.
+
+#### Noise Variants
+
+Each original document is augmented using the [Augraphy](https://github.com/sparkfish/augraphy) library with OCR-safe parameter settings. Variants include:
+
+| Variant | Description |
+|---------|-------------|
+| `original` | Clean source document (no augmentation) |
+| `default` | Default Augraphy pipeline |
+| `archetype3`–`archetype11` | Named archetype degradation pipelines |
+| `custom12`–`custom23` | Custom noise configurations |
+
+#### Dataset Schema
+
+Each row in the Parquet file represents one `(document, noise_variant)` pair:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | string | `{doc_hash}__{noise_variant}.pdf` — unique key and PDF filename |
+| `noise_variant` | string | Noise pipeline name (e.g., `original`, `default`, `archetype3`) |
+| `page_count` | int | Number of pages in the document |
+| `json_response` | struct | Ground truth matching the RealKIE-FCC-Verified invoice schema |
+
+Ground truth fields: `Agency`, `Advertiser`, `GrossTotal`, `PaymentTerms`, `AgencyCommission`, `NetAmountDue`, `LineItems[]`.
+
+#### Deployment Details
+
+During stack deployment, the system automatically:
+
+1. **Downloads Dataset Metadata** from HuggingFace parquet file (1,346 rows)
+2. **Downloads PDFs** directly from HuggingFace's `pdfs/` directory
+3. **Uploads PDFs** to `s3://TestSetBucket/confbench/input/`
+4. **Extracts Ground Truth** from `json_response` field
+5. **Uploads Baselines** to `s3://TestSetBucket/confbench/baseline/`
+6. **Registers Test Set** in DynamoDB with metadata
+
+Due to the volume (1,346 files), deployment uses a chunked self-invocation pattern — the Lambda processes 100 files per invocation and chains itself asynchronously until all files are deployed. Transient CDN errors are retried up to 3 times with exponential backoff. A deployment report is written to `s3://TestSetBucket/confbench/_deploy_state/{version}/failed_files.json`.
+
+#### Key Features
+
+- **Scale**: 1,346 documents covering 75 unique invoices × up to 18 noise levels
+- **Robustness Testing**: Evaluate model performance across degradation severity
+- **Confidence Calibration**: Assess extraction confidence under realistic noise conditions
+- **Consistent Schema**: Same ground truth format as RealKIE-FCC-Verified for direct comparison
+
+#### Corresponding Config
+
+Use with: `config_library/managed_config/confbench/config.yaml`
+
+---
+
 ### OmniAI-OCR-Benchmark
 
 **Source**: https://huggingface.co/datasets/getomni-ai/ocr-benchmark
@@ -147,6 +204,7 @@ All test sets are immediately available after stack deployment:
    - "OmniAI-OCR-Benchmark" for multi-format document testing
    - "DocSplit-Poly-Seq" for document splitting and classification testing
    - "Fake-W2-Tax-Forms" for W-2 tax form extraction testing
+   - "ConfBench" for noise-augmented invoice robustness testing
 3. Enter a description in the **Context** field
 4. Click **Run Test** to start processing
 5. Monitor progress and view results when complete

lib/idp_sdk/idp_sdk/_core/publish.py

@@ -2528,6 +2528,9 @@ def build_main_template(
                     "<w2_dataset_deployer_HASH_TOKEN>": self.get_directory_checksum(
                         "src/lambda/w2_dataset_deployer"
                     )[:16],
+                    "<CONFBENCH_DEPLOYER_HASH_TOKEN>": self.get_directory_checksum(
+                        "src/lambda/confbench_deployer"
+                    )[:16],
                     # BuildHash is the ONLY meaningful property of the
                     # DockerBuildRun custom resource, so it is the sole thing
                     # that re-triggers the container build on a stack update: if

patterns/unified/template.yaml

@@ -3313,6 +3313,7 @@ Resources:
       docsplit: !Sub "s3://${ConfigurationBucket}/config_library/managed_config/docsplit/config.yaml"
       ocr-benchmark: !Sub "s3://${ConfigurationBucket}/config_library/managed_config/ocr-benchmark/config.yaml"
       realkie-fcc-verified: !Sub "s3://${ConfigurationBucket}/config_library/managed_config/realkie-fcc-verified/config.yaml"
+      confbench: !Sub "s3://${ConfigurationBucket}/config_library/managed_config/confbench/config.yaml"
 
 
   ##########################################################################