GenAIIDP Deployment Guide

This guide covers how to deploy, build, publish, and test the GenAI Intelligent Document Processing solution.

Deployment Options

The GenAI IDP Accelerator can be deployed using either the AWS CloudFormation console (recommended for first-time users) or the IDP CLI (recommended for automation and programmatic deployments).

Administrator Access Requirements

Important: Deploying the GenAI IDP Accelerator requires administrator access to your AWS account. However, for organizations that want to enable non-administrator users to deploy and manage IDP stacks, we provide an optional CloudFormation service role approach:

For Administrators: Use the deployment options below with your existing administrator privileges
For Delegated Access: See iam-roles/cloudformation-management/README.md for instructions on provisioning a CloudFormation service role that allows non-administrator users to deploy and maintain IDP stacks without requiring administrator permissions

Option 1: One-Click CloudFormation Console Deployment (Recommended for First-Time Users)

Choose your region and click the Launch Stack button:

Region name	Region code	Launch
US West (Oregon)	us-west-2
US East (N.Virginia)	us-east-1
EU Central (Frankfurt)	eu-central-1

Review the template parameters and provide values as needed
Check the acknowledgment box and click Create stack
Wait for the stack to reach the CREATE_COMPLETE state (10-15 minutes)

Note: When the stack is deploying for the first time, it will send an email with a temporary password to the address specified in the AdminEmail parameter. You will need to use this temporary password to log into the UI and set a permanent password.

Option 2: CLI-Based Deployment (Recommended for Automation)

For programmatic deployment, updates, and batch processing, use the IDP CLI.

Install the CLI

make setup-venv
source .venv/bin/activate

Deploy a New Stack

idp-cli deploy \
    --stack-name my-idp-stack \
    --admin-email your.email@example.com \
    --max-concurrent 100 \
    --wait

What this does:

Creates all CloudFormation resources (~120 resources)
Waits for deployment to complete (10-15 minutes)
Sends email with temporary admin password
Returns stack outputs including Web UI URL and bucket names

Deploy with Custom Configuration

# Deploy with local config file (automatically uploaded to S3)
idp-cli deploy \
    --stack-name my-idp-stack \
    --admin-email your.email@example.com \
    --custom-config ./config_library/unified/bank-statement-sample/config.yaml \
    --wait

Update an Existing Stack

# Update configuration
idp-cli deploy \
    --stack-name my-idp-stack \
    --custom-config ./updated-config.yaml \
    --wait

# Update parameters
idp-cli deploy \
    --stack-name my-idp-stack \
    --max-concurrent 200 \
    --log-level DEBUG \
    --wait

Benefits of CLI deployment:

Scriptable and automatable
Version-controlled deployments
Rapid iteration for configuration testing
Integration with CI/CD pipelines
No manual console clicking required

For complete CLI documentation, see IDP CLI Documentation.

Option 3: Build and Deploy from Source Code

Dependencies

You need to have the following packages installed on your computer:

bash shell (Linux, MacOS, Windows-WSL)
aws (AWS CLI)
sam (AWS SAM)
python 3.12 or later
A local Docker daemon
Python packages: pip install boto3 rich typer PyYAML botocore setuptools ruff build cfn-lint
Node.js 22.12+ and npm (required for UI validation in publish script)

For guidance on setting up a development environment, see:

Copy the repo to your computer. Either:

Use the git command to clone the repo, if you have access
OR, download and expand the ZIP file for the repo, or use the ZIP file that has been shared with you

Option A: IDP CLI `--from-code` (Recommended)

The easiest way to build, publish, and deploy from source in a single command is using the IDP CLI with the --from-code option. This builds all artifacts, publishes them to S3, and deploys the CloudFormation stack — all in one step.

Install the CLI

make setup-venv
source .venv/bin/activate

Deploy a New Stack from Source

idp-cli deploy \
    --stack-name my-idp-dev \
    --from-code . \
    --admin-email your.email@example.com \
    --wait

Update an Existing Stack from Source

idp-cli deploy \
    --stack-name my-idp-dev \
    --from-code . \
    --wait

What --from-code does:

Builds SAM templates, Lambda layers, container images, and the UI
Publishes all artifacts to an S3 bucket in your account
Creates or updates the CloudFormation stack with the newly published template
With --wait, monitors the deployment until completion

Tip: Use --from-code for development and testing iterations. For production deployments from pre-published templates, use --template-url instead (see Option 2: CLI-Based Deployment).

For complete CLI documentation, see IDP CLI Documentation.

Option B: Publish Templates + Deploy Separately

If you prefer to publish artifacts first and then deploy as a separate step, use idp-cli publish to build and publish, then deploy using the AWS CloudFormation console or CLI.

Step 1: Build and Publish with `idp-cli publish`

idp-cli publish --source-dir . --region <region> [--bucket-basename <bucket>] [--prefix <prefix>] [--verbose] [--no-validate] [--clean-build] [--max-workers N]

Parameters:

--source-dir: Path to the IDP project root directory (default: .)
--region: AWS region for deployment (e.g., us-east-1)
--bucket-basename: (Optional) S3 bucket basename for artifacts (auto-generated if not provided)
--prefix: (Optional) S3 key prefix for artifacts (default: idp-cli)
--verbose or -v: (Optional) Enable detailed error output for debugging build failures
--clean-build: (Optional) Force a clean rebuild of all artifacts
--max-workers N: (Optional) Number of parallel build workers
--headless: (Optional) Also generate a headless (no-UI) template variant for GovCloud

Example:

idp-cli publish --source-dir . --region us-east-1

With custom bucket and prefix:

idp-cli publish --source-dir . --bucket-basename idp-1234567890 --prefix idp --region us-east-1

The publish command:

Checks your system dependencies for required packages
Builds SAM templates, Lambda layers, and container images
Packages and uploads the UI
Publishes all templates and assets to an S3 bucket (auto-created if it doesn’t exist)

When completed, the command displays:

The CloudFormation template’s S3 URL
A 1-click URL for launching the stack creation in the CloudFormation console

Note: Ensure Docker is running — Lambda functions are deployed as container images built during the publish process.

Troubleshooting Build Issues: If the build fails, use the --verbose flag to see detailed error messages:

idp-cli publish --source-dir . --region us-east-1 --verbose

This will show:

Exact SAM build commands being executed
Complete error output from failed builds
Python version compatibility issues
Missing dependencies or configuration problems

Step 2: Deploy using AWS CloudFormation

Recommended: Deploy using the CloudFormation console

For your first deployment, use the 1-Click Launch URL provided by the publish script. This lets you inspect the available parameter options in the console.

CLI Deployment:

For scripted/automated deployments, use the AWS CLI:

aws cloudformation deploy \
  --region <region> \
  --template-file <template-file> \
  --s3-bucket <bucket-name> \
  --s3-prefix <s3-prefix> \
  --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
  --parameter-overrides AdminEmail=<your-email> \
  --stack-name <your-stack-name>

Or to update an already-deployed stack:

aws cloudformation update-stack \
  --stack-name <your-stack-name> \
  --template-url <template URL output by publish script, e.g. https://s3.us-east-1.amazonaws.com/blahblah.yaml> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
  --region <region> \
  --parameters ParameterKey=AdminEmail,ParameterValue="<your-email>"

Note: The legacy publish.py script is deprecated. Use idp-cli publish for all new deployments.

Processing Mode

The solution deploys a Unified Pattern that supports both BDA and pipeline processing modes. The processing mode is controlled at runtime by the use_bda configuration flag (set via the UI), not at deployment time.

Pipeline mode (use_bda: false, default) — Textract OCR → Bedrock Classification → Bedrock Extraction → Assessment → Rule Validation → Summarization
BDA mode (use_bda: true) — Bedrock Data Automation for end-to-end processing → Rule Validation → Summarization

After deployment, check the Outputs tab in the CloudFormation console to find links to dashboards, buckets, workflows, and other solution resources.

Container-Based Lambda Deployment

The solution automatically deploys all Lambda functions as container images — no manual configuration is required. During stack deployment, CodeBuild builds Docker images for each Lambda function, pushes them to ECR, and configures the Lambda functions to use the container images.

Prerequisites

Docker must be running on your build machine (for local builds via idp-cli publish or idp-cli deploy --from-code)
Your AWS credentials must have ECR permissions

How It Works

When you deploy (or build from source), the following happens automatically:

Creates/verifies an ECR repository for Lambda images
Builds optimized multi-stage Docker images for each Lambda function
Pushes images to ECR with appropriate tags
Configures CloudFormation templates to reference the container images

Architecture: Lambda functions use ARM64 (Graviton2) by default for better price/performance.

Monitoring Container Deployments

# View ECR images
aws ecr list-images --repository-name idp-<stack-name>-lambda

# Check Lambda function configuration
aws lambda get-function --function-name <function-name>

# View container logs
aws logs tail /aws/lambda/<function-name> --follow

Updating an Existing Stack

To update an existing GenAIIDP deployment to a new version:

Log into the AWS console
Navigate to CloudFormation in the AWS Management Console
Select your existing GenAIIDP stack
Click on the “Update” button
Select “Replace current template”
Provide the new template URL:
- us-west-2: https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main.yaml
- us-east-1: https://s3.us-east-1.amazonaws.com/aws-ml-blog-us-east-1/artifacts/genai-idp/idp-main.yaml
- eu-central-1: https://s3.eu-central-1.amazonaws.com/aws-ml-blog-eu-central-1/artifacts/genai-idp/idp-main.yaml
Click “Next”
Review the parameters and make any necessary changes
- The update will preserve your existing parameter values
- Consider checking for new parameters that may be available in the updated template
Click “Next”, then “Next” again on the Configure stack options page
Review the changes that will be made to your stack
Check the acknowledgment box for IAM capabilities
Click “Update stack”
Monitor the update process in the CloudFormation console

Note: Updating the stack may cause some resources to be replaced, which could lead to brief service interruptions. Consider updating during a maintenance window if the solution is being used in production.

Testing the Solution

Method 1: CLI-Based Batch Testing (Recommended for Automation)

For batch processing, evaluation workflows, or automated testing:

Quick Batch Test

# Install CLI
cd lib/idp_cli_pkg && pip install -e .

# Process sample documents
idp-cli run-inference \
    --stack-name IDP \
    --dir ./samples/ \
    --batch-id sample-test \
    --monitor

What you’ll see:

✓ Uploaded 3 documents to InputBucket
✓ Sent 3 messages to processing queue

Monitoring Batch: sample-test
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Status Summary
 ─────────────────────────────────────
 ✓ Completed      3     100%
 ⏸ Queued         0       0%

Download Results

# Download all results
idp-cli download-results \
    --stack-name IDP \
    --batch-id sample-test \
    --output-dir ./test-results/

# View extraction data
cat ./test-results/sample-test/lending_package.pdf/sections/1/result.json | jq .

Evaluation Workflow Testing

Test accuracy with validated baselines:

# 1. Process documents initially
idp-cli run-inference \
    --stack-name IDP \
    --dir ./test-docs/ \
    --batch-id baseline-run \
    --monitor

# 2. Download and validate results
idp-cli download-results \
    --stack-name IDP \
    --batch-id baseline-run \
    --output-dir ./baselines/ \
    --file-types sections

# 3. Manually review and correct baselines
# (Edit ./baselines/baseline-run/*/sections/*/result.json as needed)

# 4. Create manifest with baselines
cat > eval-manifest.csv << EOF
document_path,baseline_source
./test-docs/invoice.pdf,./baselines/baseline-run/invoice.pdf/
./test-docs/w2.pdf,./baselines/baseline-run/w2.pdf/
EOF

# 5. Reprocess with evaluation
idp-cli run-inference \
    --stack-name IDP \
    --manifest eval-manifest.csv \
    --batch-id eval-test \
    --monitor

# 6. Download evaluation results
idp-cli download-results \
    --stack-name IDP \
    --batch-id eval-test \
    --output-dir ./eval-results/ \
    --file-types evaluation

# 7. Review accuracy metrics
cat ./eval-results/eval-test/invoice.pdf/evaluation/report.md

For complete evaluation workflow documentation, see IDP CLI - Complete Evaluation Workflow.

Method 2: Direct S3 Upload Testing (Simple)

Open the S3InputBucketConsoleURL and S3OutputBucketConsoleURL from the stack Outputs tab
Open the StateMachineConsoleURL from the stack Outputs tab
Upload a PDF form to the Input bucket (sample files are in the ./samples folder):
- Recommended: samples/lending_package.pdf
Monitor the Step Functions execution to observe the workflow
When complete, check the Output bucket for the structured JSON file with extracted fields

Upload Multiple Files for Volume Testing

To copy a sample file multiple times:

n=10
for i in `seq 1 $n`; do
  aws s3 cp ./samples/lending_package.pdf \
    s3://idp-inputbucket-kmsxxxxxxxxx/lending_package-$i.pdf
done

Method 3: Web UI Testing (Interactive)

Open the Web UI URL from the CloudFormation stack’s Outputs tab
Log in using your credentials (the temporary password from the email if this is your first login)
Navigate to the main dashboard
Click the “Upload Document” button
Select a sample PDF file (e.g., samples/lending_package.pdf)
Follow the upload process and observe the document processing in the UI
View the extraction results once processing is complete

This simulates incoming documents based on minute-by-minute rates in the schedule CSV file.

GenAIIDP Deployment Guide

GenAIIDP Deployment Guide

Deployment Options

Administrator Access Requirements

Option 1: One-Click CloudFormation Console Deployment (Recommended for First-Time Users)

Option 2: CLI-Based Deployment (Recommended for Automation)

Install the CLI

Deploy a New Stack

Deploy with Custom Configuration

Update an Existing Stack

Option 3: Build and Deploy from Source Code

Dependencies

Option A: IDP CLI --from-code (Recommended)

Install the CLI

Deploy a New Stack from Source

Update an Existing Stack from Source

Option B: Publish Templates + Deploy Separately

Step 1: Build and Publish with idp-cli publish

Step 2: Deploy using AWS CloudFormation

Processing Mode

Container-Based Lambda Deployment

Prerequisites

How It Works

Monitoring Container Deployments

Updating an Existing Stack

Testing the Solution

Method 1: CLI-Based Batch Testing (Recommended for Automation)

Quick Batch Test

Download Results

Evaluation Workflow Testing

Method 2: Direct S3 Upload Testing (Simple)

Upload Multiple Files for Volume Testing

Method 3: Web UI Testing (Interactive)

Option A: IDP CLI `--from-code` (Recommended)

Step 1: Build and Publish with `idp-cli publish`