Merge branch 'develop' v0.5.0

Author: rstrahan

.clinerules

@@ -186,3 +186,75 @@ The format is flexible - focus on capturing valuable insights that help me work
 REMEMBER: After every memory reset, I begin completely fresh. The Memory Bank is my only link to previous work. It must be maintained with precision and clarity, as my effectiveness depends entirely on its accuracy.
 
 REMEMBER: I always use mermaid diagrams when I want to visualize any concepts.
+
+## Mandatory QA Review
+
+Before calling `attempt_completion` on ANY task that involves code changes, I MUST perform a QA review. This is not optional — it is a required step in every implementation workflow.
+
+```mermaid
+flowchart TD
+    Done[Implementation Complete] --> QA{QA Review Gate}
+    
+    QA --> C1[1. Code Review]
+    QA --> C2[2. Test Verification]
+    QA --> C3[3. Consistency Check]
+    QA --> C4[4. Side Effect Analysis]
+    
+    C1 --> Pass{All Checks Pass?}
+    C2 --> Pass
+    C3 --> Pass
+    C4 --> Pass
+    
+    Pass -->|Yes| Complete[attempt_completion]
+    Pass -->|No| Fix[Fix Issues]
+    Fix --> QA
+```
+
+### QA Review Checklist
+
+For every code change, I must review and verify:
+
+#### 1. Code Quality
+- [ ] No syntax errors or typos in changed files
+- [ ] Consistent code style with existing codebase (ruff/formatting conventions)
+- [ ] No hardcoded values that should be configurable
+- [ ] Error handling is appropriate (no bare excepts, meaningful error messages)
+- [ ] No commented-out code left behind unless intentional
+
+#### 2. Test Coverage
+- [ ] **MANDATORY**: Run `make test-cicd -C lib/idp_common_pkg` (or `make test` from project root) and verify ALL tests pass — do NOT skip this step
+- [ ] **MANDATORY**: Run `ruff check` (or `make lint`) on changed Python files and verify no new lint errors
+- [ ] New functionality has corresponding tests (or note why tests weren't added)
+- [ ] Test assertions are meaningful, not just "it doesn't crash"
+
+#### 3. Cross-Module Consistency
+- [ ] Changes to shared interfaces (Document model, config schemas) are reflected in all consumers
+- [ ] If config format changed, config_library examples are updated
+- [ ] If API changed, docs are updated
+- [ ] CHANGELOG.md is updated for user-facing changes
+
+#### 4. Side Effect Analysis
+- [ ] Review imports — no circular dependencies introduced
+- [ ] Check if changed functions/methods are called elsewhere (use `search_files` to verify)
+- [ ] Backward compatibility is maintained (or breaking changes are documented)
+- [ ] No unintended changes to files outside the scope of the task
+
+#### 5. Documentation
+- [ ] Code comments for complex logic
+- [ ] docstrings for new public functions/classes
+- [ ] Memory Bank updated if significant patterns or decisions were made
+
+### QA Review Output Format
+
+After completing the QA review, I will include a brief summary in my completion message:
+
+```
+## QA Review ✅
+- **Code Quality**: [pass/issues found and fixed]
+- **Tests**: [ran X tests, all passing / N tests added]
+- **Consistency**: [cross-module impacts checked]
+- **Side Effects**: [none found / details]
+- **Docs**: [updated / not needed]
+```
+
+If any issues are found during QA, I MUST fix them before completing the task. I do NOT present incomplete or unreviewed work.

CHANGELOG.md

@@ -5,6 +5,50 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+## [0.5.0]
+
+### Added
+
+- **Unified Pattern** — Merged Pattern-1 (BDA) and Pattern-2 (Pipeline) into a single deployment. Switch between BDA and Pipeline processing modes at runtime using the `use_bda` configuration toggle — no redeployment needed. Use [Test Studio](./docs/test-studio.md) to compare accuracy and cost across both modes to find the optimal approach for your documents. See the [Migration Guide](./docs/migration-v04-to-v05.md) for upgrade instructions.
+
+- **Rule Validation for BDA mode** — Rule validation (business rule checking) is now available in both BDA and Pipeline modes. Previously it was Pipeline-only.
+
+- **Fake W-2 Tax Form Test Set Auto-Deployment** — New pre-deployed benchmark test set with 2,000 synthetically generated US W-2 tax form images and structured ground truth, sourced from HuggingFace (`singhsays/fake-w2-us-tax-form-dataset`, originally from Kaggle under CC0: Public Domain license). Features 45 ground truth fields per document covering employer info (EIN, name, address), employee info (SSN, name, address), federal wages/taxes (boxes 1-8), compensation codes (boxes 12a-d), checkboxes (box 13), and state/local taxes (boxes 15-20). Includes both clean and noisy image variants for testing OCR robustness. Ideal for benchmarking W-2 extraction accuracy, evaluating image quality impact on processing, and testing structured form data extraction at scale.
+
+- **AWS Profile Support for CLI** — Added optional `--profile` parameter to specify AWS credentials profile. Can be placed anywhere in the command. Automatically applies to all AWS SDK calls.
+
+- **Enhanced `status` CLI/MCP Command with Advanced Search, Filtering, and Analytics** — Added PK substring search (`--batch-id` now matches partial batch identifiers across multiple batches), `--object-status` filter for searching by processing status (COMPLETED, FAILED, etc.), `--get-time` flag for timing statistics (processing, queue, total time with min/max outlier tracking), `--include-metering` flag for Lambda GB-seconds usage and cost estimates, and `--show-details` flag for detailed document information. Introduces `TrackingTableSearcher` class for flexible DynamoDB tracking table queries. Fully backward compatible with existing usage.
+
+- **Added Replace/Merge sync modes for BDA synchronization** — Both "Sync from BDA" and "Sync to BDA" now support two modes: **Replace** (default) aligns the target to match the source exactly, removing items not in the source; **Merge** adds source items to the target without removing existing items. The UI modal now always shows a mode selection and ARN input (pre-filled for linked projects).
+
+
+### Deprecated
+
+- **Pattern-1 (BDA) and Pattern-2 (Pipeline) separate deployments** — Replaced by the Unified Pattern. Existing stacks are automatically upgraded. See the [Migration Guide](./docs/migration-v04-to-v05.md) for details.
+
+- **Pattern-3 (UDOP + Bedrock)** — Pattern-3 is no longer available as a deployment option. If you are currently using Pattern-3 with a SageMaker UDOP endpoint, do not upgrade to v0.5.x without first testing in a non-production environment. You can use the [Lambda Inference Hooks](./docs/lambda-hook-inference.md) feature (introduced in v0.4.15) to call your existing SageMaker UDOP endpoint from the unified pattern's classification step via a custom Lambda function.
+
+### Changed
+
+- **Switched `idp_sdk` pyproject.toml to auto-discovery** — Replaced explicit subpackage listing with `setuptools.packages.find` using `include = ["idp_sdk*"]` so new subpackages are automatically included without manual pyproject.toml updates.
+
+- **Resilient Test Set Deployment — Graceful Degradation on Download Failures** — All test set deployer Lambdas (RealKIE-FCC, OmniAI-OCR-Benchmark, DocSplit-Poly-Seq) now handle download failures gracefully instead of causing CloudFormation stack rollbacks. When a dataset source (HuggingFace) is unreachable or a download fails, the deployer creates a FAILED test set record in DynamoDB with a descriptive error message visible in the Test Studio UI, and sends `cfnresponse.SUCCESS` to CloudFormation so the stack deployment continues. Previously failed deployments are automatically retried on the next stack update. This ensures transient third-party service outages never block IDP infrastructure deployment.
+
+- **Replaced PyMuPDF (AGPL-3.0) with pypdfium2 (Apache-2.0/BSD-3-Clause) for PDF rendering** — Resolves license incompatibility with the project's MIT-0 license. pypdfium2 provides equivalent PDF-to-image rendering using PDFium engine. Page rendering is now performed sequentially before parallel OCR processing to ensure thread-safety.
+
+### Fixed
+
+- **Fixed "Sync from BDA" not removing IDP classes absent from BDA project** — Previously, "Sync from BDA" only added new classes from the BDA project without removing classes that weren't in BDA. Now defaults to "Replace" mode which fully aligns the config version's classes with the BDA project, removing classes not present in BDA. A new "Merge" mode is also available to preserve the legacy additive behavior.
+
+- **Fixed insufficient Lambda memory for Extraction, Assessment, and Evaluation functions in unified pattern template** — Increased MemorySize from 512 MB (Extraction, Assessment) and 1024 MB (Evaluation) to 4096 MB to match all other document processing Lambda functions, preventing potential out-of-memory errors during document processing. ([#205](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/issues/205))
+
+- **Fixed DOCX processing to extract text from embedded images and correct page splitting** — DOCX files with embedded images (e.g., `<w:drawing>` elements) now have image content OCR'd and included in the extracted text instead of being silently skipped. Page splitting now uses DOCX metadata (explicit page breaks, image display dimensions from `wp:extent`, section properties) instead of inaccurate height estimates, producing correct page boundaries.
+
+### Templates
+   - us-west-2: `https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.5.0.yaml`
+   - us-east-1: `https://s3.us-east-1.amazonaws.com/aws-ml-blog-us-east-1/artifacts/genai-idp/idp-main_0.5.0.yaml`
+   - eu-central-1: `https://s3.eu-central-1.amazonaws.com/aws-ml-blog-eu-central-1/artifacts/genai-idp/idp-main_0.5.0.yaml`
+
 ## [0.4.16]
 
 ### Added
@@ -25,12 +69,14 @@ SPDX-License-Identifier: MIT-0
 - **Added support for Claude Sonnet 4.6 model and Long Context (1M) variant**
 - **Included MCP tools `process`, `reprocess`, `status`, `search` for document processing**
 - **Added `process` and `reprocess` CLI commands for batch operations via command line**
-  - **Maintained `run-inference` and `rerun-inference` CLI commands with deprecation notices**
+- **Added external mcp client example `examples/external-mcp-client`**
+- **Maintained `run-inference` and `rerun-inference` CLI commands with deprecation notices**
 
 ### Fixed
 
 - **Fixed DynamoDB 400KB item size limit blocking configs with 45+ document classes** — Configuration data is now gzip-compressed before storing to DynamoDB, achieving 37-95x compression ratios. Supports 3,000+ document classes within the 400KB limit. Fully backward compatible with existing deployments. ([#200](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/issues/200), [#201](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/pull/201))
 - **Fixed Processing Flow chart using active stack config instead of the document's actual config version** for determining disabled steps (assessment, summarization, etc.)
+- **Fixed `idp_sdk` pip install from GitHub missing subpackages** — Non-editable pip installs of `idp_sdk` from GitHub were missing `core/`, `models/`, and `operations/` subpackages, causing `ModuleNotFoundError`. Fixed by explicitly declaring all subpackages in `pyproject.toml`. ([#196](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/issues/196))
 
 ### Templates
    - us-west-2: `https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.4.16.yaml`

CLAUDE.md

@@ -177,7 +177,6 @@ The solution uses a modular architecture with the main template (`template.yaml`
    - OCR with Amazon Textract
    - Classification with UDOP model on SageMaker
    - Extraction with Bedrock
-   - Location: `patterns/pattern-3/`
 
 ### Document Processing Flow
 

Makefile

@@ -49,13 +49,26 @@ ui-start:
 	@echo "Starting UI development server..."
 	cd src/ui && npm run start
 
-# Run tests in idp_common_pkg, idp_cli, idp_sdk, and capacity planning Lambda
+# Run tests in idp_common_pkg, idp_cli, idp_sdk, capacity planning Lambda, and config library
 test:
 	$(MAKE) -C lib/idp_common_pkg test
 	cd lib/idp_cli_pkg && python -m pytest -v
 	cd lib/idp_sdk && python -m pytest -m "not integration" -v
 	@echo "Running capacity planning Lambda tests..."
 	cd src/lambda/calculate_capacity && python -m pytest -v
+	@echo "Validating config library files..."
+	python -m pytest config_library/test_config_library.py -v
+
+# Run only config library validation tests
+test-config-library:
+	@echo "Validating config library YAML/JSON files..."
+	python -m pytest config_library/test_config_library.py -v
+
+# Run only IDP CLI tests
+test-cli:
+	@echo "Running IDP CLI tests..."
+	cd lib/idp_cli_pkg && python -m pytest -v
+	@echo -e "$(GREEN)✅ All CLI tests passed!$(NC)"
 
 # Run only capacity planning tests
 test-capacity:
@@ -170,9 +183,9 @@ ui-lint:
 	STORED_HASH=$$(test -f src/ui/.checksum && cat src/ui/.checksum || echo ""); \
 	if [ "$$CURRENT_HASH" != "$$STORED_HASH" ]; then \
 		echo "UI code checksum changed - running lint..."; \
-		cd src/ui && npm ci --prefer-offline --no-audit && npm run lint -- --fix && \
+		cd src/ui && npm ci --prefer-offline --no-audit && npm run lint -- --fix && npm run typecheck && \
 		echo "$$CURRENT_HASH" > .checksum; \
-		echo -e "$(GREEN)✅ UI lint completed and checksum updated$(NC)"; \
+		echo -e "$(GREEN)✅ UI lint and typecheck completed and checksum updated$(NC)"; \
 	else \
 		echo -e "$(GREEN)✅ UI code checksum unchanged - skipping lint$(NC)"; \
 	fi

README.md

@@ -57,8 +57,7 @@ Concierge support for customization, deployment, and integration of production u
 - **Comprehensive Monitoring**: Rich CloudWatch dashboard with detailed metrics and logs
 - **Web User Interface**: Modern UI for inspecting document workflow status and results
 - **Configuration Versioning**: Support for multiple configuration versions with version-specific processing and test comparison
-- **Human-in-the-Loop (HITL)**: Built-in review system for human validation workflows (Pattern 1 & Pattern 2)
-  - **Note**: When deploying multiple patterns with HITL, reuse existing private workteam ARN due to AWS account limits
+- **Human-in-the-Loop (HITL)**: Built-in review system for human validation workflows
 - **AI-Powered Evaluation**: Framework to assess accuracy against baseline data
 - **Extraction Confidence Assessment**: LLM-powered assessment of extraction confidence with multimodal document analysis
 - **Document Knowledge Base Query**: Ask questions about your processed documents
@@ -67,14 +66,13 @@ Concierge support for customization, deployment, and integration of production u
 
 ## Architecture Overview
 
-![Architecture Diagram](./images/IDP.drawio.png)
+![Architecture Diagram](./images/IDP.UnifiedPatterns.drawio.png)
 
 The solution uses a modular architecture with nested CloudFormation stacks to support multiple document processing patterns while maintaining common infrastructure for queueing, tracking, and monitoring.
 
-Current patterns include:
-- Pattern 1: Packet or Media processing with Bedrock Data Automation (BDA)
-- Pattern 2: OCR → Bedrock Classification (page-level or holistic) → Bedrock Extraction
-- Pattern 3: OCR → UDOP Classification (SageMaker) → Bedrock Extraction
+The unified pattern supports two processing modes, controlled by the `use_bda` configuration flag:
+- **Pipeline mode** (default): OCR → Bedrock Classification (page-level or holistic) → Bedrock Extraction → Assessment → Rule Validation → Summarization
+- **BDA mode**: End-to-end processing with Bedrock Data Automation (BDA) → Rule Validation → Summarization
 
 ## Quick Start
 
@@ -101,8 +99,7 @@ After deployment, choose the processing method that fits your use case:
 1. Open the Web UI URL from CloudFormation stack Outputs
 2. Log in and click "Upload Document"
 3. Upload a sample document:
-   - For Patterns 1 & 2: [samples/lending_package.pdf](./samples/lending_package.pdf)
-   - For Pattern 3: [samples/rvl_cdip_package.pdf](./samples/rvl_cdip_package.pdf)
+   - [samples/lending_package.pdf](./samples/lending_package.pdf)
 4. Monitor processing and view results in the dashboard
 
 #### Method 2: Direct S3 Upload (Simple)
@@ -161,8 +158,7 @@ To update an existing GenAIIDP stack to a new version:
 7. For detailed instructions, see the [Deployment Guide](./docs/deployment.md#updating-an-existing-stack)
 
 For testing, use these sample files:
-   - For Patterns 1 (BDA) and Pattern 2: Use [samples/lending_package.pdf](./samples/lending_package.pdf)
-   - For Pattern 3 (UDOP): Use [samples/rvl_cdip_package.pdf](./samples/rvl_cdip_package.pdf)
+   - Use [samples/lending_package.pdf](./samples/lending_package.pdf) for both Pipeline and BDA modes
 
 For detailed deployment and testing instructions, see the [Deployment Guide](./docs/deployment.md).
 
@@ -194,11 +190,11 @@ For detailed deployment and testing instructions, see the [Deployment Guide](./d
 - [Reporting Database](./docs/reporting-database.md) - Analytics database for evaluation metrics and metering data
 - [Troubleshooting](./docs/troubleshooting.md) - Troubleshooting and performance guides
 
-### Processing Patterns
+### Processing Modes
 
-- [Pattern 1: BDA](./docs/pattern-1.md) - Packet or Media processing with Bedrock Data Automation (BDA)
-- [Pattern 2: Textract + Bedrock](./docs/pattern-2.md) - OCR with Textract and generative AI with Bedrock
-- [Pattern 3: Textract + UDOP + Bedrock](./docs/pattern-3.md) - OCR with Textract, UDOP Classification, and Bedrock extraction
+- [Architecture](./docs/architecture.md) - Unified pattern with BDA and Pipeline processing modes
+- [BDA Mode Reference](./docs/pattern-1.md) - Bedrock Data Automation (BDA) concepts and behavior
+- [Pipeline Mode Reference](./docs/pattern-2.md) - Textract + Bedrock classification and extraction
 - [Few-Shot Examples](./docs/few-shot-examples.md) - Implementing few-shot examples for improved accuracy
 
 ### Python Development

VERSION

@@ -1 +1 @@
-0.4.16
+0.5.0