Merge branch 'feature/refactor-publish-deploy' into 'develop'

Consolidated publish and headless deploy into idp-cli

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!592

Author: rstrahan

CHANGELOG.md

@@ -5,6 +5,14 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+### Added
+
+- **Consolidated publish and headless deploy into `idp-cli`** — All build/publish/deploy functionality now available through the CLI, deprecating standalone scripts:
+  - `publish.py` and `publish.sh` are deprecated — use `idp-cli publish` instead. `publish.py` remains as a thin backward-compatibility wrapper. `publish.sh` has been removed.
+  - `scripts/generate_govcloud_template.py` is deprecated — use `idp-cli publish --headless` or `idp-cli deploy --headless` instead. The script remains as a thin wrapper.
+  - New `--template-file` option on `idp-cli deploy` for deploying from a local CloudFormation template file produced by a previous `idp-cli publish`.
+  - `idp-cli deploy --headless` (without `--from-code`) now downloads the published template, transforms to headless with GovCloud config defaults, uploads to S3, and deploys — all in one command.
+
 ## [0.5.3]
 
 ### Added

CLAUDE.md

@@ -24,8 +24,6 @@ python3 publish.py idp-1234567890 idp us-east-1
 # With verbose output for debugging build failures
 python3 publish.py idp-1234567890 idp us-east-1 --verbose
 
-# Legacy build script
-./publish.sh <cfn_bucket_basename> <cfn_prefix> <region>
 ```
 
 The build process:
@@ -91,8 +89,10 @@ pytest -m "integration"
 The IDP CLI is used for programmatic deployment and batch processing:
 
 ```bash
-# Install CLI
+# Install all packages into current Python environment
 make setup
+# Or create an isolated .venv first
+make setup-venv
 
 # Deploy a new stack
 idp-cli deploy \

CONTRIBUTING.md

@@ -127,7 +127,8 @@ The project uses `make` to simplify common development tasks. Run `make` or `mak
 
 | Command | Description |
 |---------|-------------|
-| `make setup` | Create virtual environment and install all packages in development mode |
+| `make setup` | Install all packages into your current Python environment (no venv) |
+| `make setup-venv` | Create `.venv` virtual environment and install all packages into it |
 
 ### Code Quality
 

Makefile

@@ -43,7 +43,33 @@ help: ## Show this help message
 all: lint test ## Run lint + test (default)
 
 ##@ Setup
-setup: ## Create venv and install all packages in development mode
+setup: ## Install all packages into current Python environment (no venv)
+	@# Always use the current shell's pip, ignoring .venv even if it exists
+	@SETUP_PIP=$$(python3 -m pip --version >/dev/null 2>&1 && echo "python3 -m pip" || echo "pip3"); \
+	SETUP_PYTHON=$$(command -v python3 2>/dev/null || echo python); \
+	echo "Installing packages into current Python environment..."; \
+	echo "Python: $$($$SETUP_PYTHON --version) at $$(which $$SETUP_PYTHON)"; \
+	echo "Pip: $$SETUP_PIP"; \
+	echo ""; \
+	echo "Upgrading pip..."; \
+	$$SETUP_PIP install --upgrade pip && \
+	echo "Installing idp_common package with all dependencies (including test)..." && \
+	$$SETUP_PIP install -e "lib/idp_common_pkg[all,dev,test]" && \
+	echo "Installing idp-cli package..." && \
+	$$SETUP_PIP install -e lib/idp_cli_pkg && \
+	echo "Installing idp_sdk package..." && \
+	$$SETUP_PIP install -e lib/idp_sdk && \
+	echo "Installing idp_mcp_connector package..." && \
+	$$SETUP_PIP install -e lib/idp_mcp_connector_pkg && \
+	echo "Installing capacity planning test dependencies..." && \
+	$$SETUP_PIP install -r src/lambda/calculate_capacity/requirements-test.txt && \
+	echo "Installing cfn-lint for CloudFormation template validation..." && \
+	$$SETUP_PIP install cfn-lint && \
+	echo "" && \
+	echo -e "$(GREEN)✅ Setup complete! idp_common, idp-cli, idp_sdk, idp_mcp_connector, and test dependencies are now installed.$(NC)" && \
+	echo -e "$(YELLOW)   Tip: Use 'make setup-venv' instead to install into an isolated virtual environment.$(NC)"
+
+setup-venv: ## Create .venv and install all packages into it
 	@echo "Creating virtual environment in $(VENV_DIR)..."
 	@PYENV_PYTHON=$$(pyenv which python 2>/dev/null); \
 	SYS_PYTHON=$$(command -v python3 2>/dev/null); \
@@ -66,6 +92,8 @@ setup: ## Create venv and install all packages in development mode
 	$(VENV_DIR)/bin/pip install -e lib/idp_mcp_connector_pkg
 	@echo "Installing capacity planning test dependencies..."
 	$(VENV_DIR)/bin/pip install -r src/lambda/calculate_capacity/requirements-test.txt
+	@echo "Installing cfn-lint for CloudFormation template validation..."
+	$(VENV_DIR)/bin/pip install cfn-lint
 	@echo ""
 	@echo -e "$(GREEN)✅ Setup complete! Virtual environment created at $(VENV_DIR)$(NC)"
 	@echo -e "$(GREEN)   idp_common, idp-cli, idp_sdk, idp_mcp_connector, and test dependencies are now installed.$(NC)"

docs/deployment.md

@@ -45,8 +45,8 @@ For programmatic deployment, updates, and batch processing, use the IDP CLI.
 #### Install the CLI
 
 ```bash
-cd lib/idp_cli_pkg
-pip install -e .
+make setup-venv
+source .venv/bin/activate
 ```
 
 #### Deploy a New Stack
@@ -132,13 +132,13 @@ Copy the repo to your computer. Either:
 
 ### Option A: IDP CLI `--from-code` (Recommended)
 
-The easiest way to build, publish, and deploy from source in a single command is using the IDP CLI with the `--from-code` option. This builds all artifacts using `publish.py`, publishes them to S3, and deploys the CloudFormation stack — all in one step.
+The easiest way to build, publish, and deploy from source in a single command is using the IDP CLI with the `--from-code` option. This builds all artifacts, publishes them to S3, and deploys the CloudFormation stack — all in one step.
 
 #### Install the CLI
 
 ```bash
-cd lib/idp_cli_pkg
-pip install -e .
+make setup-venv
+source .venv/bin/activate
 ```
 
 #### Deploy a New Stack from Source
@@ -161,7 +161,7 @@ idp-cli deploy \
 ```
 
 **What `--from-code` does:**
-- Runs `publish.py` to build SAM templates, Lambda layers, container images, and the UI
+- Builds SAM templates, Lambda layers, container images, and the UI
 - Publishes all artifacts to an S3 bucket in your account
 - Creates or updates the CloudFormation stack with the newly published template
 - With `--wait`, monitors the deployment until completion
@@ -174,37 +174,45 @@ idp-cli deploy \
 
 ### Option B: Publish Templates + Deploy Separately
 
-If you prefer to publish artifacts first and then deploy as a separate step, use `publish.py` to build and publish, then deploy using the AWS CloudFormation console or CLI.
+If you prefer to publish artifacts first and then deploy as a separate step, use `idp-cli publish` to build and publish, then deploy using the AWS CloudFormation console or CLI.
 
-#### Step 1: Build and Publish with publish.py
+#### Step 1: Build and Publish with `idp-cli publish`
 
 ```bash
-python3 publish.py <cfn_bucket_basename> <cfn_prefix> <region> [--verbose] [--no-validate] [--clean-build] [--max-workers N]
+idp-cli publish --source-dir . --region <region> [--bucket-basename <bucket>] [--prefix <prefix>] [--verbose] [--no-validate] [--clean-build] [--max-workers N]
 ```
 
 **Parameters:**
 
-- `cfn_bucket_basename`: A prefix for the S3 bucket name (e.g., `idp-1234567890` to ensure global uniqueness)
-- `cfn_prefix`: S3 prefix for artifacts (e.g., `idp` or `idp-dev`)
-- `region`: AWS region for deployment (e.g., `us-east-1`)
+- `--source-dir`: Path to the IDP project root directory (default: `.`)
+- `--region`: AWS region for deployment (e.g., `us-east-1`)
+- `--bucket-basename`: (Optional) S3 bucket basename for artifacts (auto-generated if not provided)
+- `--prefix`: (Optional) S3 key prefix for artifacts (default: `idp-cli`)
 - `--verbose` or `-v`: (Optional) Enable detailed error output for debugging build failures
 - `--clean-build`: (Optional) Force a clean rebuild of all artifacts
 - `--max-workers N`: (Optional) Number of parallel build workers
+- `--headless`: (Optional) Also generate a headless (no-UI) template variant for GovCloud
 
 **Example:**
 
 ```bash
-python3 publish.py idp-1234567890 idp us-east-1
+idp-cli publish --source-dir . --region us-east-1
 ```
 
-The publish script:
+With custom bucket and prefix:
+
+```bash
+idp-cli publish --source-dir . --bucket-basename idp-1234567890 --prefix idp --region us-east-1
+```
+
+The publish command:
 
 - Checks your system dependencies for required packages
 - Builds SAM templates, Lambda layers, and container images
 - Packages and uploads the UI
-- Publishes all templates and assets to an S3 bucket (`<cfn_bucket_basename>-<region>`, created if it doesn't exist)
+- Publishes all templates and assets to an S3 bucket (auto-created if it doesn't exist)
 
-When completed, the script displays:
+When completed, the command displays:
 
 - The CloudFormation template's S3 URL
 - A 1-click URL for launching the stack creation in the CloudFormation console
@@ -215,7 +223,7 @@ When completed, the script displays:
 If the build fails, use the `--verbose` flag to see detailed error messages:
 
 ```bash
-python3 publish.py idp-1234567890 idp us-east-1 --verbose
+idp-cli publish --source-dir . --region us-east-1 --verbose
 ```
 
 This will show:
@@ -257,13 +265,7 @@ aws cloudformation update-stack \
   --parameters ParameterKey=AdminEmail,ParameterValue="<your-email>"
 ```
 
-#### Using publish.sh (Legacy)
-
-`publish.sh` is a Bash wrapper around `publish.py`. Use `publish.py` directly for new deployments.
-
-```bash
-./publish.sh <cfn_bucket_basename> <cfn_prefix> <region>
-```
+> **Note**: The legacy `publish.py` script is deprecated. Use `idp-cli publish` for all new deployments.
 
 ---
 
@@ -282,7 +284,7 @@ The solution **automatically** deploys all Lambda functions as container images
 
 ### Prerequisites
 
-- **Docker** must be running on your build machine (for local builds via `publish.py` or `idp-cli --from-code`)
+- **Docker** must be running on your build machine (for local builds via `idp-cli publish` or `idp-cli deploy --from-code`)
 - Your AWS credentials must have **ECR permissions**
 
 ### How It Works

docs/govcloud-deployment.md

@@ -50,21 +50,24 @@ You need to have the following packages installed on your computer:
 5. Node.js >=22.12.0
 6. npm >=10.0.0
 7. A local Docker daemon
-8. Python packages for publish.py.  You are encouraged to configure a virtual environment for dependency management, ie. `python -m venv .venv`.  Activate the environment (`. .venv/bin/activate`) and then install dependencies via `pip install boto3 rich PyYAML botocore setuptools docker ruff build`
+8. Python packages for the IDP CLI and SDK. Run `make setup-venv` from the project root to create a `.venv` and install all required packages (idp-cli, idp-sdk, idp_common). Activate with `source .venv/bin/activate`.
 
-### Step 1: Generate GovCloud Template
+### Step 1: Build and Generate Headless Template
 
-First, generate the GovCloud-compatible template - this run the standard build process first to create all Lambda functions and artifacts, and then creates a stripped down version for GovCloud:
+First, build and generate the GovCloud-compatible headless template using `idp-cli publish --headless`. This runs the standard build process to create all Lambda functions and artifacts, and then creates a stripped-down version for GovCloud:
 
 ```bash
-# Note: The Python script will create an S3 bucket automatically by concatenating the provided bucket name and region, ie. my-govcloud-bucket-us-gov-west-1.  You can change the bucket base name as desired.  Files will be placed under [my-prefix] prefix within the generated bucket.
-# Build for GovCloud region
-python scripts/generate_govcloud_template.py my-bucket-govcloud my-prefix us-gov-west-1
+# Build for GovCloud region with headless template
+idp-cli publish --source-dir . --region us-gov-west-1 --headless
 
 # Or build for commercial region first (for testing)
-python scripts/generate_govcloud_template.py my-bucket my-prefix us-east-1
+idp-cli publish --source-dir . --region us-east-1 --headless
 ```
 
+> **Note**: The CLI will create an S3 bucket automatically. You can customize the bucket name with `--bucket-basename` and the prefix with `--prefix`. The `--headless` flag generates a template with UI, AppSync, Cognito, and WAF resources removed.
+
+> **Legacy**: The `scripts/generate_govcloud_template.py` script is deprecated. Use `idp-cli publish --headless` instead.
+
 ### Step 2: Deploy to GovCloud
 
 Deploy the generated template to GovCloud using the AWS CloudFormation console (recommended) or deploy using AWS CLI e.g:

docs/idp-cli.md

@@ -74,8 +74,8 @@ https://github.com/user-attachments/assets/3d448a74-ba5b-4a4a-96ad-ec03ac0b4d7d
 ### Install from source
 
 ```bash
-cd lib/idp_cli_pkg
-pip install -e .
+make setup-venv
+source .venv/bin/activate
 ```
 
 ### Install with test dependencies
@@ -156,7 +156,7 @@ idp-cli deploy [OPTIONS]
 - `--admin-email`: Admin user email
 
 **Optional Parameters:**
-- `--from-code`: Deploy from local code by building with publish.py (path to project root)
+- `--from-code`: Deploy from local code by building and publishing artifacts (path to project root)
 - `--template-url`: URL to CloudFormation template in S3 (optional, auto-selected based on region)
 - `--custom-config`: Path to local config file or S3 URI
 - `--max-concurrent`: Maximum concurrent workflows (default: 100)

docs/idp-sdk.md

@@ -10,9 +10,11 @@ The IDP SDK provides programmatic Python access to all IDP Accelerator capabilit
 
 ```bash
 # Install from local development
-pip install -e ./lib/idp_sdk
+# Recommended: install everything at once
+make setup-venv
+source .venv/bin/activate
 
-# Or with uv
+# Or install just the SDK with pip/uv
 uv pip install -e ./lib/idp_sdk
 ```
 

docs/setup-development-env-WSL.md

@@ -84,22 +84,29 @@ sam --version (Example: SAM CLI, version 1.143.0)
 node --version (Example: v22.12.0)
 npm --version (Example: 11.0.0)
 ```
-### 4.2 Test Build Process
+### 4.2 Install the IDP CLI
 ```
-cd accelerated-intelligent-document-processing-on-aws
+cd accelerated-intelligent-document-processing-on-aws/lib/idp_cli_pkg
+pip install -e .
+cd ../..
+```
+
+### 4.3 Test Build Process
 ```
+cd accelerated-intelligent-document-processing-on-aws
 ```
-# Test publish script help
-python3 publish.py --help
+```bash
+# Test CLI help
+idp-cli publish --help
 
-# Test build (replace with your S3 bucket name)
-python3 publish.py your-bucket-name build-test us-east-1
+# Test build
+idp-cli publish --source-dir . --region us-east-1
 ```
 
-### 4.3 Troubleshooting Build Issues
+### 4.4 Troubleshooting Build Issues
 If the build fails, use the `--verbose` flag:
 ```bash
-python3 publish.py your-bucket-name build-test us-east-1 --verbose
+idp-cli publish --source-dir . --region us-east-1 --verbose
 ```
 
 The verbose flag shows:
@@ -108,6 +115,8 @@ The verbose flag shows:
 - Python version compatibility issues
 - Missing dependencies or configuration problems
 
+> **Note**: The legacy `publish.py` script is deprecated. Use `idp-cli publish` for all new builds.
+
 ## Step 5: Install Visual Studio Code on Local Machine for using WSL as a terminal
 ### Visit the official website: Go to [https://code.visualstudio.com/](https://code.visualstudio.com/)
 Download: Click the "Download for Windows" button