Merge branch 'develop' v0.5.4

Author: Bob Strahan

.github/workflows/developer-tests.yml

@@ -28,7 +28,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v4
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4.3.1
         with:
           fetch-depth: 0 # Fetch all history for git diff in typecheck-pr
 
@@ -89,7 +89,7 @@ jobs:
         continue-on-error: false
 
       - name: Upload coverage reports
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4.6.2
         if: always() && steps.run-tests.outcome != 'skipped'
         with:
           name: test-reports
@@ -99,15 +99,15 @@ jobs:
           retention-days: 7
 
       - name: Publish test results
-        uses: EnricoMi/publish-unit-test-result-action@v2
+        uses: EnricoMi/publish-unit-test-result-action@c950f6fb443cb5af20a377fd0dfaa78838901040  # v2.23.0
         if: always() && hashFiles('lib/idp_common_pkg/test-reports/test-results.xml') != ''
         with:
           files: lib/idp_common_pkg/test-reports/test-results.xml
           check_name: Test Results
           comment_mode: off  # Disable PR comments to avoid permission issues on fork PRs
 
       - name: Code Coverage Report
-        uses: irongut/CodeCoverageSummary@v1.3.0
+        uses: irongut/CodeCoverageSummary@51cc3a756ddcd398d447c044c02cb6aa83fdae95  # v1.3.0
         if: always() && hashFiles('lib/idp_common_pkg/test-reports/coverage.xml') != ''
         with:
           filename: lib/idp_common_pkg/test-reports/coverage.xml

CHANGELOG.md

@@ -5,6 +5,39 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+## [0.5.4]
+
+### Added
+
+- **MLflow Experiment Tracking Integration** — Optional integration with Amazon SageMaker MLflow for automated test run logging. When enabled (`EnableMLflow=true`), every Test Studio run automatically logs metrics (accuracy, cost, field-level scores), configuration parameters (model IDs, temperatures, inference settings), and artifacts (full config snapshots, class definitions, cost breakdowns) to an MLflow tracking server. Fire-and-forget async invocation — never blocks or delays test results. Zero resources created when disabled. See `docs/mlflow-integration.md`.
+
+- **BDA Blueprint Optimization** — Automatically improves BDA extraction accuracy using the `InvokeBlueprintOptimizationAsync` API. When discovery includes a ground truth file and `enable_blueprint_optimization: true` is set, the system optimizes the BDA blueprint by comparing extraction results against ground truth, evaluates before/after metrics, and updates the blueprint schema if improved. Disabled by default. See `docs/discovery.md` — Blueprint Optimization section.
+
+- **idp_common API Reference & Documentation** — Added `docs/idpcommon-api-reference.md` covering all 22 modules, created 6 missing module READMEs (discovery, schema, image, s3, utils, metrics), updated core data model docs to match current code, fixed `IDPConfig` lazy-loading bug in `__init__.py`, and integrated into docs-site sidebar.
+
+- **Consolidated publish and headless deploy into `idp-cli`** — All build/publish/deploy functionality now available through the CLI, deprecating standalone scripts:
+  - `publish.py` and `publish.sh` are deprecated — use `idp-cli publish` instead. `publish.py` remains as a thin backward-compatibility wrapper. `publish.sh` has been removed.
+  - `scripts/generate_govcloud_template.py` is deprecated — use `idp-cli publish --headless` or `idp-cli deploy --headless` instead. The script remains as a thin wrapper.
+  - New `--template-file` option on `idp-cli deploy` for deploying from a local CloudFormation template file produced by a previous `idp-cli publish`.
+  - `idp-cli deploy --headless` (without `--from-code`) now downloads the published template, transforms to headless with GovCloud config defaults, uploads to S3, and deploys — all in one command.
+
+### Fixed
+
+- **HITL review start overwrites document sections** — Fixed the Start Review action to update only the Review Status and Review Owner fields, preserving all existing document sections and other fields.
+
+- **Evaluation schema error for free-form objects** — Stickler mapper now detects and skips unevaluable object schemas (e.g., objects with `additionalProperties` but no defined `properties`, and arrays of such objects) instead of raising validation errors.
+
+- **Full document reprocess not re-running OCR** — Fixed bug where clicking "Reprocess" in the UI reused stale OCR results from the previous run instead of re-executing OCR with the current configuration. The reprocess resolver now deletes previous output data from S3 before queuing, preventing the OCR function's retry-safe recovery from reinstalling old results.
+
+- **Agentic extraction timeout on long documents** — Fixed repeated Lambda timeouts when agentic extraction exceeds the 15-minute limit on large documents (e.g., 25-page brokerage statements with 600+ holdings). Added incremental S3 checkpointing that saves extraction state after each tool call — covers both the extraction tools path (`extraction_tool`, `apply_json_patches`, `make_buffer_data_final_extraction`) and the buffer tools path (`patch_buffer_data`) that the agent uses for very large batched extractions. The checkpoint format tracks which state was saved (`current_extraction` vs `intermediate_extraction` buffer) so the correct resume path is used. On Step Function retry, the Lambda loads the checkpoint and the agent resumes from where it left off rather than restarting from scratch. No CloudFormation or Step Function changes required — the existing `Sandbox.Timedout` retry mechanism now makes incremental progress. Only active when agentic extraction is enabled; standard extraction is unaffected.
+
+- **Agentic extraction fails on Bedrock InternalServerException without retrying** — Fixed `InternalServerException` errors (transient Bedrock server-side errors) causing immediate Lambda failure after only botocore's fast 7 retries, bypassing the application-level retry decorator (50 retries with 5s→1800s exponential backoff). Root cause: `InternalServerException` and `InternalServerError` were missing from all three retry layers — the `async_exponential_backoff_retry` decorator's `DEFAULT_RETRYABLE_ERRORS` set (`bedrock_utils.py`), the `BedrockClient._invoke_with_retry()` retryable errors list (`bedrock/client.py`), and the Step Functions ExtractionStep Retry `ErrorEquals` list (`workflow.asl.json`). All three layers now include these transient errors, providing proper exponential backoff retry at the application level and Lambda-level retry via Step Functions as a safety net.
+
+### Templates
+   - us-west-2: `https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.5.4.yaml`
+   - us-east-1: `https://s3.us-east-1.amazonaws.com/aws-ml-blog-us-east-1/artifacts/genai-idp/idp-main_0.5.4.yaml`
+   - eu-central-1: `https://s3.eu-central-1.amazonaws.com/aws-ml-blog-eu-central-1/artifacts/genai-idp/idp-main_0.5.4.yaml`
+
 ## [0.5.3]
 
 ### Added

CLAUDE.md

@@ -24,14 +24,12 @@ python3 publish.py idp-1234567890 idp us-east-1
 # With verbose output for debugging build failures
 python3 publish.py idp-1234567890 idp us-east-1 --verbose
 
-# Legacy build script
-./publish.sh <cfn_bucket_basename> <cfn_prefix> <region>
 ```
 
 The build process:
 - Checks system dependencies (AWS CLI, SAM CLI, Docker, Python 3.12+, Node.js 22.12+)
 - Builds CloudFormation templates and assets using SAM
-- Pattern-2 functions are built as container images; Pattern-1 and Pattern-3 use ZIP-based Lambdas
+- All pattern functions are built within the unified pattern directory
 - Uploads artifacts to S3 bucket named `<cfn_bucket_basename>-<region>`
 
 ### Code Quality & Linting
@@ -91,8 +89,10 @@ pytest -m "integration"
 The IDP CLI is used for programmatic deployment and batch processing:
 
 ```bash
-# Install CLI
+# Install all packages into current Python environment
 make setup
+# Or create an isolated .venv first
+make setup-venv
 
 # Deploy a new stack
 idp-cli deploy \
@@ -106,7 +106,7 @@ idp-cli deploy \
 idp-cli deploy \
     --stack-name my-idp-stack \
     --pattern pattern-2 \
-    --custom-config ./config_library/pattern-2/bank-statement-sample/config.yaml \
+    --custom-config ./config_library/unified/bank-statement-sample/config.yaml \
     --wait
 
 # Process documents in batch
@@ -125,7 +125,7 @@ idp-cli download-results \
 ### Local Lambda Testing
 
 ```bash
-cd patterns/pattern-2/
+cd patterns/unified/
 sam build
 sam local invoke OCRFunction -e ../../testing/OCRFunction-event.json --env-vars ../../testing/env.json
 ```
@@ -153,30 +153,26 @@ The solution uses a modular architecture with the main template (`template.yaml`
 - Authentication (Cognito User Pool, Identity Pool)
 - AppSync GraphQL API (for UI-backend communication)
 
-**Pattern Stacks** (`patterns/pattern-*/template.yaml`) - Pattern-specific resources:
-- Step Functions State Machine
-- Pattern-specific Lambda Functions (OCR, Classification, Extraction)
-- Pattern-specific CloudWatch Dashboard
-- Model Endpoints and Configurations
+**Unified Pattern Stack** (`patterns/unified/template.yaml`) - Processing resources:
+- Step Functions State Machine (BDA branch + Pipeline branch + shared tail)
+- Lambda Functions (OCR, Classification, Extraction, Assessment, Summarization, Evaluation, etc.)
+- CloudWatch Dashboard
 
-### Processing Patterns
+### Processing Modes
 
-1. **Pattern 1: Bedrock Data Automation (BDA)**
+The unified architecture supports two processing modes, controlled by the `use_bda` configuration flag:
+
+1. **BDA Mode** (formerly Pattern 1)
    - Uses AWS Bedrock Data Automation for end-to-end processing
    - Handles packet or media documents with integrated OCR, classification, and extraction
-   - Location: `patterns/pattern-1/`
 
-2. **Pattern 2: Textract + Bedrock**
+2. **Pipeline Mode** (formerly Pattern 2)
    - OCR with Amazon Textract
    - Classification with Bedrock (page-level or holistic)
    - Extraction with Bedrock
    - Supports few-shot examples
-   - Location: `patterns/pattern-2/`
 
-3. **Pattern 3: Textract + UDOP + Bedrock**
-   - OCR with Amazon Textract
-   - Classification with UDOP model on SageMaker
-   - Extraction with Bedrock
+> **Note**: The separate `patterns/pattern-1/`, `patterns/pattern-2/`, and `patterns/pattern-3/` directories have been removed. All processing is now in `patterns/unified/`. See [pattern-1.md](docs/pattern-1.md) and [pattern-2.md](docs/pattern-2.md) for historical reference.
 
 ### Document Processing Flow
 

CONTRIBUTING.md

@@ -127,7 +127,8 @@ The project uses `make` to simplify common development tasks. Run `make` or `mak
 
 | Command | Description |
 |---------|-------------|
-| `make setup` | Create virtual environment and install all packages in development mode |
+| `make setup` | Install all packages into your current Python environment (no venv) |
+| `make setup-venv` | Create `.venv` virtual environment and install all packages into it |
 
 ### Code Quality
 

Dockerfile.optimized

@@ -22,6 +22,7 @@ ENV UV_LINK_MODE=copy
 # Build argument for function path
 ARG FUNCTION_PATH
 ARG INSTALL_IDP_COMMON=true
+ARG INSTALL_GIT=false
 
 # Create working directory
 WORKDIR /build
@@ -44,6 +45,10 @@ RUN --mount=from=uv,source=/uv,target=/bin/uv \
 # Final stage - minimal runtime
 FROM public.ecr.aws/lambda/python:3.12-arm64
 
+# Conditionally install git (required for mlflow/gitpython)
+ARG INSTALL_GIT=false
+RUN if [ "$INSTALL_GIT" = "true" ]; then dnf install -y git && dnf clean all; fi
+
 # Copy the runtime dependencies from the builder stage
 COPY --from=builder ${LAMBDA_TASK_ROOT} ${LAMBDA_TASK_ROOT}
 

Makefile

@@ -43,7 +43,33 @@ help: ## Show this help message
 all: lint test ## Run lint + test (default)
 
 ##@ Setup
-setup: ## Create venv and install all packages in development mode
+setup: ## Install all packages into current Python environment (no venv)
+	@# Always use the current shell's pip, ignoring .venv even if it exists
+	@SETUP_PIP=$$(python3 -m pip --version >/dev/null 2>&1 && echo "python3 -m pip" || echo "pip3"); \
+	SETUP_PYTHON=$$(command -v python3 2>/dev/null || echo python); \
+	echo "Installing packages into current Python environment..."; \
+	echo "Python: $$($$SETUP_PYTHON --version) at $$(which $$SETUP_PYTHON)"; \
+	echo "Pip: $$SETUP_PIP"; \
+	echo ""; \
+	echo "Upgrading pip..."; \
+	$$SETUP_PIP install --upgrade pip && \
+	echo "Installing idp_common package with all dependencies (including test)..." && \
+	$$SETUP_PIP install -e "lib/idp_common_pkg[all,dev,test]" && \
+	echo "Installing idp-cli package..." && \
+	$$SETUP_PIP install -e lib/idp_cli_pkg && \
+	echo "Installing idp_sdk package..." && \
+	$$SETUP_PIP install -e lib/idp_sdk && \
+	echo "Installing idp_mcp_connector package..." && \
+	$$SETUP_PIP install -e lib/idp_mcp_connector_pkg && \
+	echo "Installing capacity planning test dependencies..." && \
+	$$SETUP_PIP install -r src/lambda/calculate_capacity/requirements-test.txt && \
+	echo "Installing cfn-lint for CloudFormation template validation..." && \
+	$$SETUP_PIP install cfn-lint && \
+	echo "" && \
+	echo -e "$(GREEN)✅ Setup complete! idp_common, idp-cli, idp_sdk, idp_mcp_connector, and test dependencies are now installed.$(NC)" && \
+	echo -e "$(YELLOW)   Tip: Use 'make setup-venv' instead to install into an isolated virtual environment.$(NC)"
+
+setup-venv: ## Create .venv and install all packages into it
 	@echo "Creating virtual environment in $(VENV_DIR)..."
 	@PYENV_PYTHON=$$(pyenv which python 2>/dev/null); \
 	SYS_PYTHON=$$(command -v python3 2>/dev/null); \
@@ -66,6 +92,8 @@ setup: ## Create venv and install all packages in development mode
 	$(VENV_DIR)/bin/pip install -e lib/idp_mcp_connector_pkg
 	@echo "Installing capacity planning test dependencies..."
 	$(VENV_DIR)/bin/pip install -r src/lambda/calculate_capacity/requirements-test.txt
+	@echo "Installing cfn-lint for CloudFormation template validation..."
+	$(VENV_DIR)/bin/pip install cfn-lint
 	@echo ""
 	@echo -e "$(GREEN)✅ Setup complete! Virtual environment created at $(VENV_DIR)$(NC)"
 	@echo -e "$(GREEN)   idp_common, idp-cli, idp_sdk, idp_mcp_connector, and test dependencies are now installed.$(NC)"
@@ -343,10 +371,7 @@ docs-deploy: docs-build ## Deploy docs to GitHub Pages (from local build)
 
 ##@ Security (DSR)
 dsr: ## Run full DSR workflow (setup → scan → optional fix)
-	@if [ ! -f .dsr/dsr ]; then \
-		echo "DSR not found, running setup..."; \
-		$(MAKE) dsr-setup; \
-	fi
+	@$(MAKE) dsr-setup
 	@$(MAKE) dsr-scan
 	@echo ""
 	@echo "Do you want to run DSR fix? (y/N):"

README.md

@@ -187,6 +187,7 @@ For detailed deployment and testing instructions, see the [Deployment Guide](./d
 - [Assessment](./docs/assessment.md) - Extraction confidence evaluation using LLMs
 - [Rule Validation](./docs/rule-validation.md) - Business rule validation and compliance checking
 - [Evaluation Framework](./docs/evaluation.md) - Accuracy assessment system with analytics database and reporting
+- [MLflow Experiment Tracking](./docs/mlflow-integration.md) - Optional MLflow integration for tracking metrics, model parameters, and prompts across test runs
 - [Knowledge Base](./docs/knowledge-base.md) - Document knowledge base query feature
 - [Monitoring](./docs/monitoring.md) - Monitoring and logging capabilities
 - [IDP Accelerator Help Chat Bot](./docs/code-intelligence.md) - Chat bot for asking question about the IDP code base and features

VERSION

@@ -1 +1 @@
-0.5.3
+0.5.4

docs-site/astro.config.mjs

@@ -54,6 +54,7 @@ export default defineConfig({
             { label: "Web UI", slug: "web-ui" },
             { label: "IDP CLI", slug: "idp-cli" },
             { label: "IDP SDK", slug: "idp-sdk" },
+            { label: "idp_common API Reference", slug: "idpcommon-api-reference" },
             { label: "Demo Videos", slug: "demo-videos" },
             { label: "Troubleshooting", slug: "troubleshooting" },
             { label: "Error Analyzer", slug: "error-analyzer" },
@@ -97,6 +98,7 @@ export default defineConfig({
               slug: "evaluation-enhanced-reporting",
             },
             { label: "Test Studio", slug: "test-studio" },
+            { label: "MLflow Experiment Tracking", slug: "mlflow-integration" },
           ],
         },
         {

docs/assessment-bounding-boxes.md

@@ -20,7 +20,7 @@ The Assessment Service now supports **optional bounding box extraction** as part
 ### Core Capabilities
 
 - **Optional Feature**: Disabled by default, enabled via configuration
-- **UI Compatible**: Outputs geometry format compatible with existing pattern-1 UI
+- **UI Compatible**: Outputs geometry format compatible with existing BDA mode UI
 - **Multi-page Support**: Handles bounding boxes across multiple document pages
 - **Error Resilient**: Gracefully handles invalid or incomplete bounding box data
 - **Coordinate Normalization**: Converts from 0-1000 scale to 0-1 normalized coordinates
@@ -325,7 +325,7 @@ if explainability_info:
 
 ## Integration with UI
 
-The geometry format is fully compatible with the existing pattern-1 UI:
+The geometry format is fully compatible with the existing BDA mode UI:
 
 - **Coordinate System**: Normalized 0-1 coordinates
 - **Bounding Box Format**: `{top, left, width, height}`