feat(testset): Add ConfBench dataset deployer

[sujimart]

Adds the ConfBench benchmark dataset as an automatically deployed test set, extending RealKIE-FCC-Verified with up to 18 noise-augmented variants per document for confidence calibration and OCR robustness research.

Changes:

• Add ConfBench deployer Lambda (src/lambda/confbench_deployer/)
  • Chunked self-invocation pattern to handle 1,346 files within Lambda timeout (100 files per chunk)
  • Streams PDFs directly from HuggingFace to S3 via multipart upload
  • Retry with exponential backoff for transient CDN errors (5xx)
  • Failed file report written to S3 at end of deployment
• Add managed config (config_library/managed_config/confbench/)
• Wire confbench into patterns/unified/template.yaml UpdateDefaultConfig
• Add ConfBench CloudFormation resources to template.yaml
• Document ConfBench in docs/test-studio.md

Dataset: https://huggingface.co/datasets/amazon/ConfBench
Source: 75 FCC invoices x up to 18 Augraphy noise variants = 1,346 docs

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[rstrahan]

Findings

🔴 1. Blocking — template.yaml:1008 (ConfbenchDeployment): the chained deployment can exceed CloudFormation's 1-hour custom-resource response timeout, failing the stack.

CFN's default (and maximum) ServiceTimeout is 3600 s; if no response arrives, the stack operation fails. Because only the final chunk calls cfnresponse.send, the whole 14-chunk chain
must finish inside that one hour. I measured the actual dataset:

• 1,346 PDFs, 32.71 GB total, mean 24.3 MB, p90 73 MB, max 615 MB (258 files exceed 35 MB)
• Sustained HuggingFace CDN throughput from this environment: 2–14 MB/s per stream
• Pure serial download of 32.7 GB at 13.8 MB/s ≈ 40 minutes, before counting S3 upload time (each byte is also PUT to S3, roughly doubling wall clock) and 14 cold starts

So the happy path lands somewhere around 60–90 minutes — straddling the deadline. Any CDN slowness or a couple of retries (each up to 5+10+20 s of time.sleep) pushes it over, and the
failure mode is a stack rollback, which is exactly what the graceful-degradation design in the other deployers (CHANGELOG line 1026) was built to avoid.

Two related problems compound it:

• The module docstring's premise is understated: it says "1,346 files × up to 35 MB each", but the real p90 is 73 MB and the max is 615 MB. CHUNK_SIZE = 100 was sized against the wrong
  distribution. Per-chunk bytes in true parquet row order range from 1.12 GB to 5.21 GB (offset 1000) — that worst chunk alone needs ~5.8 MB/s sustained download plus the same again in
  upload to fit 900 s, with zero margin.
• A single 615 MB file at the observed 2 MB/s low end is ~5 minutes on its own, and with MAX_PDF_RETRIES it could consume most of one invocation.

Suggested fix: decouple from the CFN wait entirely — send cfnresponse.SUCCESS immediately after chunk 0 is kicked off (the DynamoDB record already carries status, so Test Studio can
show IN_PROGRESS → COMPLETED), and let the chain finish in the background. That matches the existing non-blocking philosophy and removes the deadline. Failing that, raise
CHUNK_SIZE-per-byte budgeting (chunk by cumulative bytes, not file count) and parallelize downloads within a chunk.

🔴 2. Blocking — src/lambda/confbench_deployer/index.py:691-708: the continuation payload grows unboundedly and can exceed Lambda's 256 KB async invoke limit.

FailedIdsSoFar accumulates {"id", "error"} for every failure across all 14 chunks and is passed inline in the self-invoke payload, alongside the entire CfnEvent. InvocationType="Event"
caps the payload at 256 KB. Each failure entry is ~60 bytes of id plus an unbounded str(e) — a botocore/urllib exception string can easily be 200–500 bytes. A systemic failure (HF
outage mid-deploy, S3 permission issue) yields hundreds of entries, and the invoke raises, silently ending the chain with no CFN response at all — which then blocks until the 1-hour
timeout and fails the stack.

The design already writes state to S3 (_store_row_data / _load_row_data); failed_ids should ride along there too rather than in the payload. Also note the str(e) in the failed-files
report is untruncated (contrast create_failed_testset_record, which caps at 500 chars).

🟡 3. Should fix — ~32.7 GB is added to every single deployment with no opt-out.

Measured comparison of the auto-deployed sets:

┌──────────────────────┬──────────┐
│ Dataset │ Size │
├──────────────────────┼──────────┤
│ RealKIE-FCC-Verified │ 0.08 GB │
├──────────────────────┼──────────┤
│ OmniAI-OCR-Benchmark │ 0.39 GB │
├──────────────────────┼──────────┤
│ Fake-W2-Tax-Forms │ 0.31 GB │
├──────────────────────┼──────────┤
│ ConfBench (new) │ 32.71 GB │
└──────────────────────┴──────────┘

That's ~42× the other three combined, landing in a versioned, KMS-encrypted bucket with DataRetentionInDays defaulting to 365. Every customer of the accelerator pays that storage and
transfer cost on every deploy, whether or not they do calibration research — and adds ~30–60 min to first deploy. Given the dataset's stated purpose ("confidence calibration research,
OCR robustness evaluation"), this is a specialist set. Recommend gating it behind a CFN parameter (default off), or deploying only the original + a few representative variants and
documenting how to pull the full set on demand.

🟡 4. Should fix — src/lambda/confbench_deployer/index.py:672-673: document_id may be unbound in the exception handler.

except Exception as e:
logger.error(f"Error processing row {idx} ({document_id}): {e}")

document_id is assigned as the first statement inside the try. If data_dict["id"][idx] itself raises (malformed/short column), the handler raises NameError/UnboundLocalError — masking
the real error and aborting the chunk mid-way with no CFN response (see finding 2's consequence). Hoist document_id = data_dict["id"][idx] above the try, or initialize it to f"<row
{idx}>".

🟡 5. Should fix — no CHANGELOG entry. Every prior test-set deployer got one (see CHANGELOG lines 1008, 1382, 1516). CLAUDE.md and the documentation skill require user-facing changes
to update CHANGELOG.md. A 32.7 GB addition is decidedly user-facing.

🟡 6. Should fix — template.yaml:963 + src/lambda/confbench_deployer/index.py:634-637: abort_multipart_upload is not itself guarded.

On a transfer error the handler calls abort_multipart_upload before recording last_pdf_error. If the abort call also fails (throttling, transient S3 error), that exception replaces the
original and propagates to the outer per-row handler, losing the retry loop entirely and leaking the MPU. The bucket's AbortIncompleteMultipartUpload: DaysAfterInitiation: 1 limits
the cost, so this is about diagnosis, not spend — wrap the abort in try/except and log.

🟢 7. Nit — docs/test-studio.md:36 and docs/configuration.md:42 are now stale. Line 36 still says "four benchmark datasets" while the list below it has five entries.
docs/configuration.md:42 enumerates the managed config versions (fake-w2, docsplit, ocr-benchmark, realkie-fcc-verified) and needs confbench added.

🟢 8. Nit — ruff format would reformat index.py. One block near create_testset_record's default description differs. src/ is in ruff.toml's extend-exclude, so CI won't catch it, but
it's worth matching for consistency. (ruff check --select=F,E9,B is clean, and the file compiles.)

🟢 9. Nit — datetime.utcnow() (lines 852, 883) is deprecated in Python 3.12. Copied from the existing deployers, so it's consistent — but datetime.now(timezone.utc) is the modern form
for new code.

🟢 10. Nit — _store_row_data writes full json_response ground truth to S3 (line 447). The docstring says "Only IDs and json_response are needed; page_count and noise_variant are
lightweight" — but json_response is the heavy column. The rows.json blob is re-downloaded and fully deserialized on every one of the 14 continuations. Minor waste, and the cleanup
delete_object at line 747 swallows all exceptions with a bare pass, so a leftover blob is invisible.

What's good

• The template block is a faithful copy of the FccDatasetDeployer pattern: dedicated KMS-encrypted LogGroup, PermissionsBoundary conditional, cfn_nag/checkov suppressions with reasons,
  VpcConfig and S3_ENDPOINT_URL handling for private-network mode, and the self-invoke IAM statement correctly scoped to the one function ARN with arn:${AWS::Partition}: and no
  hardcoded amazonaws.com. No security findings.
• Streaming HF → S3 via multipart with bounded memory is a genuine improvement over the existing deployers' load-into-/tmp approach, and the right call given the file sizes.
• Baseline JSON shape (document_class / split_document.page_indices / inference_result) matches fcc_dataset_deployer/index.py:318-324 exactly, so evaluation will work.
• The managed config is byte-identical to realkie-fcc-verified's schema apart from the header/notes — correct, since ConfBench reuses that invoice schema, and it makes cross-set
  comparison valid.
• Hash-token registration in publish.py and the UpdateDefaultConfig wiring are both present and correct — easy things to forget.
• I verified the dataset exists, is public and ungated, and that the hardcoded parquet path data/test-00000-of-00001.parquet and pdfs/{id} layout match the live repo.

Recommendation

Request changes.

The pattern-following is genuinely good — the template block, IAM, and baseline format are all right, and streaming multipart was the correct instinct for large files. But the
chunked-invocation design rests on a size assumption that the actual dataset contradicts by ~2× on p90 and ~17× on max, and the failure mode is a CloudFormation rollback rather than
the graceful degradation the other deployers deliberately provide. Findings 1 and 2 are the blockers: decouple the CFN response from the chain's completion, and move failed_ids out of
the async payload. Finding 3 is a judgment call for the maintainers rather than a defect — but 32.7 GB unconditionally added to every customer deployment deserves an explicit decision,
not a side effect of adding a test set. Findings 4–6 are quick fixes. No tests accompany the 682-line Lambda; the existing deployers don't have them either, so I'd call that
consistent-but-unfortunate rather than blocking.