This runbook deploys the GenAI IDP Accelerator in a fully private / air-gapped environment:
Web UI served via API Gateway — an S3 proxy on the same private REST API that backs the UI’s data operations (no CloudFront, no ALB, no public internet)
REST API (Web UI and data operations) reachable only from inside the VPC via an execute-api Interface VPC Endpoint (no public endpoint)
All Lambda → AWS service traffic routed through VPC Interface Endpoints
Browser → S3 presigned uploads routed through global S3 (default, via NAT) or optionally through a bring-your-own S3 Interface VPC Endpoint (opt-in via S3PresignedUrlViaVpcEndpoint=true)
Internet-facing features (MCP Gateway, Knowledge Base) disabled
For standard public deployments, see Deployment Guide. For the full
reference on this hosting mode (including public/regional variants), see
API Gateway Hosting Guide.
The Web UI and the data API are served from one origin and one stage on the private
REST API, so there is no CORS between them and VPC access is inherited from the
execute-api endpoint. No S3 VPC endpoint is created for UI hosting — API Gateway reaches
the Web UI bucket over AWS-internal networking via an IAM role (WebUIProxyRole), not through
a customer VPCE. Browser uploads use global S3 by default (via NAT); set
S3PresignedUrlViaVpcEndpoint=true to route them through a bring-your-own S3 Interface VPC
Endpoint instead (requires corporate network DNS/routing to VPCE hostnames).
API Gateway hosting does not create any S3 VPC endpoint — the UI is served by the private
REST API, not by an S3-fronting ALB. The two models below differ only in whether browser
presigned uploads route through a bring-your-own S3 Interface VPCE.
The private REST API + WebUI S3 proxy. Browser uploads go to global s3.amazonaws.com via NAT. scripts/deploy-vpc-endpoints.py adds the service endpoints.
B. Customer-managed S3 VPCE uploads
Everything in A plus a pre-existing S3 Interface VPCE in the VPC, its vpce-id, its full DNS name (with random suffix), and S3PresignedUrlViaVpcEndpoint=true.
Nothing extra — the stack reads the VPCE id and DNS name from parameters and rewrites presigned upload URLs to use it.
Mode B fits central-network-account topologies where one team manages all VPC endpoints and IDP is one of many tenants.
See Deployment Guide → Dependencies for AWS CLI, SAM CLI, Python 3.12+, Node.js 22+. Docker is not required locally — images are built in AWS by CodeBuild.
Note: When DeployInVPC=true, all CodeBuild projects (WebUI build, Docker image builds, SDLC pipeline) run inside the VPC. They require either a NAT Gateway for public registry access or an internal artifact repository (JFrog Artifactory, AWS CodeArtifact, etc.) for air-gapped builds. See Dependency Mirroring for generating the full dependency manifest.
At least 2 private subnets in different Availability Zones — for the Lambdas.
DNS resolution enabled on the VPC — enableDnsSupport=true and enableDnsHostnames=true.
An execute-api Interface VPC Endpoint with PrivateDnsEnabled: true, passed as ApiGatewayVpcEndpointId. This is what makes the regional execute-api.<region>.amazonaws.com hostname resolve to private IPs inside the VPC — it fronts both the Web UI and the data API.
(Mode B only) DNS resolution for the BYO S3 Interface VPCE — PrivateDnsEnabled is off for S3 VPCEs by AWS design; bucket-vhost addressing routes via the per-endpoint bucket.<vpce-id>-<random>.s3.<region>.vpce.amazonaws.com name. VPC DNS resolution is what makes that name resolvable for in-VPC clients (Lambdas). Browsers only need to resolve this name when S3PresignedUrlViaVpcEndpoint=true; with the default (false), browsers use s3.amazonaws.com via NAT.
A Lambda security group allowing outbound HTTPS, passed as LambdaSecurityGroupId (or let the stack create one).
No 0.0.0.0/0 → IGW or → NAT route on the Lambda subnet — proves no public-internet egress (verify after deployment).
Don’t have a VPC? A ready-made, self-contained test VPC (NAT egress + one execute-api
endpoint) is provided at scripts/sdlc/apigw-hosting-test-vpc.yaml and is used by the CI
deployment test:
Users must reach the private REST API via VPN, Direct Connect, AWS Client VPN, WorkSpaces, or SSM port forwarding. The regional execute-api hostname is only resolvable inside the VPC (via the execute-api endpoint’s private DNS) or via a VPC DNS forwarder over the VPN. No TLS certificate is required — API Gateway terminates HTTPS with an AWS-managed certificate for the regional execute-api domain.
Every private-mode knob is exposed as a CloudFormation parameter, so the stack can be deployed entirely via the AWS console “Create stack” form — no CLI required.
Pre-requisites (must complete before clicking Launch)
Publish artifacts to your S3 bucket once (Step 1 below). One-click reuses idp-main.yaml from the artifact bucket. The bucket can live in the same or a different account.
vpce-0123-abcdef12.s3.us-east-1.vpce.amazonaws.com (full DNS, random suffix included, no leading *.)
Both must be set together. A CloudFormation Rules assertion blocks change-set creation if only one is provided. Run aws ec2 describe-vpc-endpoints --vpc-endpoint-ids <vpce-id> --query 'VpcEndpoints[0].DnsEntries[0].DnsName' and strip the leading *. to get the DNS-name value.
CloudFormation provisions the main stack + nested stacks (API resolvers, Pattern, DocumentKB, MultiDocDiscovery). Expect ~25–35 min.
Then complete the post-stack operation the console can’t do:
Deploy service VPC endpoints (Step 3 below) — required so Lambdas can reach Bedrock, Textract, etc.
Once that post-stack step completes, the UI is reachable from any VPN/DC client at the URL in the ApplicationWebURL stack output (the execute-api /api/ URL).
Pre-create a compliant bucket (KMS, tags, bucket policy, access logging, block public access) and pass it via --bucket-basename. When deploying, pass ArtifactsBucketKmsKeyArn=<key-arn> so CodeBuild (DockerBuildRole) and ConfigurationCopyFunction get kms:Decrypt.
The stack deploys the private REST API + WebUI S3 proxy. Browser presigned uploads go to
global s3.amazonaws.com via NAT; no S3 VPC endpoint is involved.
Set S3PresignedUrlViaVpcEndpoint=true plus both S3VpcEndpointIdOverrideandS3VpcEndpointDnsNameOverride. The stack rewrites presigned upload URLs to use your S3 VPCE. The deploy fails fast at change-set creation if only one of the two override params is set (CFN Rules assertion).
Discover your VPCE’s full DNS name (the vpce-<id>-<random> suffix is AWS-assigned and cannot be derived from the id alone):
The stack itself creates no service VPC endpoints. You supply the execute-api interface
endpoint (fronts the private REST API — UI + data) as ApiGatewayVpcEndpointId, and the
remaining service endpoints (Bedrock, Textract, S3, SQS, States, KMS, Logs, monitoring, SSM,
Lambda, Events, Athena, STS, secretsmanager, plus DynamoDB Gateway) come from
scripts/vpc-endpoints.yaml via the deploy script:
Terminal window
pythonscripts/deploy-vpc-endpoints.py\
--vpc-id<vpc-id>\
--stack-nameIDP-PRIVATE\
--region<region>
The script:
Reads LambdaSubnetIds and LambdaVpcSecurityGroupId from the IDP stack outputs
Detects which endpoints already exist in the VPC and skips them (avoids conflicting DNS domain)
Creates a shared endpoint SG that allows 443 from the Lambda SG
Adds egress from the Lambda SG to the VPC endpoints SG — required when the Lambda SG has restricted egress (no default 0.0.0.0/0 rule)
Includes the monitoring (CloudWatch) endpoint required by the DashboardMerger custom resource that fires during stack create/update — without it the stack hangs
Pass --dry-run to preview without changes.
⚠️ Lambda SG egress requirement: If your Lambda SG does NOT have a default 0.0.0.0/0 egress rule (common in hardened environments), you must also add egress to the S3 and DynamoDB Gateway endpoint prefix lists. The check-vpc-endpoints.sh script detects this and prints the required aws ec2 authorize-security-group-egress commands. Without these rules, custom resource Lambdas (InitializeConcurrencyTable, ConfigurationCopy, dataset deployers) will time out because they cannot reach DynamoDB or send cfnresponse via S3.
Mode B note: if you bring your own VPCEs for some/all services, set the corresponding Create*Endpoint=false flags in the vpc-endpoints.yaml parameters. The script auto-detects existing ones.
Cross-account Bedrock note: if you use the Cross-Account Bedrock feature (BedrockHubRoleArn set), the STS interface VPC endpoint is mandatory — Lambdas need it for sts:AssumeRole against the hub-account role. The endpoint is already in scripts/vpc-endpoints.yaml; just verify it deployed.
API Gateway hosting creates no S3 VPC endpoint. The REST API serves the SPA by proxying
GET requests to the Web UI bucket over AWS-internal networking, authenticated with an IAM role
(WebUIProxyRole) rather than through a customer VPCE. The proxy integration and the private
endpoint live in the api-resolvers nested stack (nested/api-resolvers/template.yaml) and the
main template.yaml (WebUIProxyRole, and the APIGateway branch of WebUIBucketPolicy).
Authorization for which buckets can be read or written is enforced at three layers:
Layer
Mechanism
Where
API
Stage-level WAFv2 IP allow-list (WAFAllowedIPv4Ranges) + Cognito auth on POST /op/{field}
nested/api-resolvers/template.yaml
IAM
WebUIProxyRole (read-only, Web UI bucket) + per-Lambda execution-role policies (least-privilege)
template.yaml
Bucket
Web UI bucket policy restricts reads to the API Gateway proxy role
When S3PresignedUrlViaVpcEndpoint=true, browser presigned uploads route through your
bring-your-own S3 Interface VPCE. Your central network team owns that endpoint’s policy; IDP
just consumes it via S3VpcEndpointIdOverride + S3VpcEndpointDnsNameOverride. Apply your
own scoping (e.g. aws:PrincipalAccount, bucket ARNs) on that endpoint policy as needed. Each
app bucket is additionally pinned to the VPCE via an aws:sourceVpce condition in
template.yaml::*BucketPolicy.
The Lambdas attach to LambdaSecurityGroupId (BYO or stack-created). The service endpoints in
scripts/vpc-endpoints.yaml use a shared endpoint SG with ingress 443 from the Lambda SG. The
execute-api endpoint you supply must allow ingress 443 from both the Lambdas and the
browser’s source network:
VPN clients are source-NAT’d to the IP of the VPN-association subnet (AWS Client VPN, Site-to-Site VPN). Add ingress on the relevant SGs from the NAT subnet’s CIDR, not the VPN client CIDR.
Target SG
Add ingress
Port
Reason
execute-api endpoint SG
(e.g. 10.1.3.0/24)
443
Browser → private REST API (UI + data).
BYO S3 VPCE endpoint SG
443
Browser → S3 VPCE for presigned upload. Only needed when S3PresignedUrlViaVpcEndpoint=true.
AWS Client VPN example: client tunnel IPs are 172.16.0.0/22, but inbound packets to the endpoint are seen as coming from the VPN association subnet (e.g. 10.1.3.0/24). Authorize 10.1.3.0/24, not 172.16.0.0/22.
Site-to-Site VPN over VGW + corporate VPN: source-NAT happens at the customer firewall. Use the post-NAT IP range advertised into the VPC.
Production with Direct Connect / Transit Gateway: typically no NAT — clients reach VPC with on-prem IPs. Authorize those CIDRs on the execute-api endpoint SG as part of standard onboarding.
Note: When S3PresignedUrlViaVpcEndpoint=false (default), browser uploads go to s3.amazonaws.com via NAT/internet — no S3 endpoint SG ingress rule needed for browsers.
cognito-idp.<region>.amazonaws.com has no VPC Interface Endpoint. The browser must reach it from outside the VPC, which is normal: end-user browsers on VPN/DC use their own internet via the corporate network. Lambdas inside the VPC do not call Cognito IDP at runtime.
For testing scenarios where the browser runs inside the VPC (WorkSpaces, bastion EC2), provide internet egress via a NAT Gateway in a separate public subnet.
WorkSpaces runs a Windows desktop inside the VPC. Setup notes:
Register an AWS Directory Service (Simple AD) with WorkSpaces in the same VPC.
WorkSpaces directory: EnableInternetAccess=true. Set before launching — toggling it later requires rebuild (~25 min).
NAT Gateway in a separate public subnet (with 0.0.0.0/0 → IGW). It cannot egress traffic from the same subnet it lives in. Private subnet route table: 0.0.0.0/0 → NAT GW.
The execute-api endpoint (PrivateDnsEnabled: true) must be in the WorkSpaces VPC so the /api/ URL resolves to the endpoint’s private IPs.
Cost: WorkSpaces AutoStop bundle ~$7.25/mo + NAT GW ~$0.045/hr active. Tear down after testing.
Authorize the VPN association-subnet CIDR (e.g. 10.1.3.0/24) on the execute-api endpoint SG (and the BYO S3 VPCE endpoint SG when S3PresignedUrlViaVpcEndpoint=true) — see SG matrix above.
When WebUIHosting=APIGateway and ApiGatewayVisibility=PRIVATE:
REST API → deployed as PRIVATE with a resource policy pinning access to the
ApiGatewayVpcEndpointId execute-api endpoint; serves both the UI (S3 proxy) and data ops (POST /op/{field})
Cognito callback / logout URLs → the execute-api /api/ URL, registered post-deploy via the WebUIClientOAuthUrls custom resource; plus CustomDomainUrl (with and without trailing slash) when set
UI build → built with Vite base=/api/ (VITE_UI_BASE_PATH=/api/); VITE_CLOUDFRONT_DOMAIN set to the execute-api /api/ URL by default, or "" when CustomDomainUrl is set (the Web UI then uses window.location.origin so both URLs work side by side — see API Gateway Hosting Guide)
Web UI bucket policy → APIGateway branch restricts reads to the WebUIProxyRole (API Gateway S3 proxy)
App bucket policies → aws:sourceVpce condition restricts access to the BYO S3 VPCE when S3PresignedUrlViaVpcEndpoint=true
CodeBuild projects (WebUI build, Docker image builds, SDLC pipeline) → placed in VPC with LambdaSecurityGroup; requires NAT or internal artifact repository for dependency resolution
Lambda functions (~21) → placed in LambdaSubnetIds with LambdaSecurityGroup
S3_ENDPOINT_URL env injected on presigner Lambdas (only when S3PresignedUrlViaVpcEndpoint=true or BYO endpoint override is set):
UploadResolverFunction (browser presigned POST)
DiscoveryUploadResolverFunction
TestSetResolverFunction
ApiHandlerFunction (jobs API presigner)
All API resolver Lambdas via the api-resolvers nested-stack S3EndpointUrl parameter
Lambda S3 client uses signature_version=s3v4 + addressing_style=virtual when S3_ENDPOINT_URL is set; falls back to default path style when unset (global S3)
Private REST API DNS resolution in cross-VPC and hybrid networks
When ApiGatewayVisibility=PRIVATE, the IDP stack deploys the REST API (Web UI and data operations) as a private API Gateway API that is only reachable via the execute-api VPC Interface Endpoint (com.amazonaws.<region>.execute-api). The frontend (browser) is baked at build time — and the Cognito callback/logout URLs are registered — with the API’s regional hostname:
For the browser to reach the API (and load the SPA the API serves), DNS must resolve this hostname to the private IP addresses of the execute-api VPC Interface Endpoint. That endpoint also exposes its own VPCE-specific hostname:
<vpce-id>.execute-api.<region>.vpce.amazonaws.com
When does this “just work”?
If the execute-api VPC endpoint is created in the same VPC where the browser’s DNS queries are resolved (i.e., the VPC’s built-in Route 53 Resolver at <VPC-CIDR-base>+2), and PrivateDnsEnabled: true is set on that endpoint, then DNS resolution works automatically. PrivateDnsEnabled: true makes the regional execute-api.<region>.amazonaws.com name resolve to the endpoint’s private IPs inside that VPC. This is the case when:
The user is on a VPN whose DNS server is the VPC’s .2 resolver
The user is on a WorkSpace or EC2 instance in that VPC
Lambdas are in the same VPC as the endpoint
This is the happy path assumed by the rest of this runbook — Prereq §2 requires PrivateDnsEnabled: true on the execute-api endpoint.
When does it NOT work?
PrivateDnsEnabled only injects the DNS override into the VPC that owns the endpoint (AWS creates a managed Route 53 Private Hosted Zone for execute-api.<region>.amazonaws.com, scoped to that VPC). It does NOT propagate across:
VPC peering connections
Transit Gateway attachments
Cross-account VPC associations
On-premises networks connected via Direct Connect or Site-to-Site VPN (unless DNS is forwarded to the VPC resolver)
This is the central network account scenario (Mode B): a networking team manages all VPC endpoints in a shared-services VPC, and the IDP workload VPC connects via Transit Gateway or peering. A browser (on corporate VPN) and the Lambdas (in the workload VPC) cannot resolve <api-id>.execute-api.<region>.amazonaws.com because the private DNS override lives only in the endpoint’s VPC.
For S3, the IDP stack solves cross-VPC access by rewriting presigned URLs to use the VPCE-specific DNS name (bucket.vpce-xxx.s3.<region>.vpce.amazonaws.com). This works because:
The IDP stack controls the presigner Lambda and can inject S3_ENDPOINT_URL
S3 VPCE DNS names are resolvable from any VPC (they are public DNS names that happen to route to VPCE IPs when resolved from within the VPC)
No custom headers are needed — S3 uses the Host header that the browser sets naturally from the URL
The private REST API is harder to redirect at the URL layer because:
The SPA origin is fixed at build time (the Vite /api/ base and VITE_CLOUDFRONT_DOMAIN), and the Cognito OAuth callback/logout URLs are registered to the regional <api-id>.execute-api.<region>.amazonaws.com origin. Swapping in a VPCE hostname would break the single-origin model and the Cognito redirect URLs.
You can invoke a private API via the VPCE’s own hostname (<vpce-id>.execute-api.<region>.vpce.amazonaws.com), but only if the request also carries Host: <api-id>.execute-api.<region>.amazonaws.com (or an x-apigw-api-id: <api-id> header) so API Gateway can route to the correct private API. A browser sets Host from the URL and cannot override it for SPA asset loads and navigations, so this path is impractical for a baked SPA.
Note: Unlike AWS AppSync (which required an X-AppSync-Domain/Host header and could not be reached over browser WebSockets), the private REST API needs no custom-header workaround on the happy path — the DNS-layer fix below makes the ordinary regional hostname resolve to the endpoint and every request works unchanged. Document status is delivered by REST polling, and chat streaming uses a separate Lambda Function URL origin that is direct-to-browser and not behind the private API, so streaming is unaffected by this DNS problem.
Therefore, the solution operates at the DNS layer — making the standard regional REST API hostname resolve to the correct private IPs regardless of which VPC the client is in.
Solution A: Route 53 Private Hosted Zone (recommended)
This is the AWS-recommended approach for cross-VPC and cross-account private API Gateway access — associate a Route 53 Private Hosted Zone for the API’s execute-api domain with the VPCs that need to resolve it, pointing at the execute-api Interface VPC Endpoint. It is the direct analogue of the pattern AWS documents for reaching any Interface-VPC-Endpoint-fronted service across VPCs (see Centralized access to VPC private endpoints).
Create an API-ID-scoped Route 53 Private Hosted Zone (PHZ) named for the exact regional hostname (<api-id>.execute-api.<region>.amazonaws.com)
In that zone, create an apex A-record alias to the execute-api VPC Interface Endpoint (com.amazonaws.<region>.execute-api)
Associate the PHZ with every VPC that needs to resolve the API hostname (workload VPC, shared-services VPC, etc.)
For on-premises/hybrid browsers: set up a Route 53 Resolver Inbound Endpoint and configure the corporate DNS to conditionally forward only this FQDN to it
Why scope to the API ID instead of the whole execute-api.<region>.amazonaws.com domain? Route 53 resolves the most specific matching private hosted zone first. A PHZ named for the exact API hostname overrides DNS for only this API; every other private API in the region — yours or another team’s — keeps resolving normally. A PHZ created at the regional execute-api.<region>.amazonaws.com apex would instead intercept all API Gateway DNS in every associated VPC, which can break unrelated applications. The scoped approach below is the least-blast-radius default; see Important considerations for the regional-zone alternative and when it makes sense.
The execute-api VPC Interface Endpoint (com.amazonaws.<region>.execute-api) must exist in at least one VPC (the central/shared VPC)
PrivateDnsEnabled should be set to false on that endpoint. Enabling it makes AWS create a managed PHZ for the regionalexecute-api.<region>.amazonaws.com domain inside the endpoint’s VPC, which reintroduces the blanket interception the scoped zone is designed to avoid. The scoped PHZ below replaces its function for this API only. (This differs from the single-VPC happy path in Prereq §2, where PrivateDnsEnabled: true is exactly what you want.)
The VPC endpoint must have a security group allowing inbound HTTPS (443) from the source networks
The API’s resource policy must permit access via this endpoint. When ApiGatewayVisibility=PRIVATE, IDP pins the policy to the ApiGatewayVpcEndpointId you passed; a second execute-api endpoint in a shared VPC must be the same one, or the policy must be widened to include it.
Single hostname, single zone. Unlike AWS AppSync (which exposed a separate appsync-realtime-api WebSocket hostname and therefore needed a second zone), the REST API is a single regional hostname. One scoped PHZ covers the UI and all data operations. Document status is delivered by REST polling over this same hostname; chat streaming uses a separate Lambda Function URL origin that is not behind the private API and does not need a zone here.
Step 3: Create the API-ID-scoped Private Hosted Zone
Create a PHZ named for the exact regional hostname. Because the zone name is the full API FQDN, Route 53 overrides DNS for only this API — other private APIs in the region are unaffected.
If users access the IDP UI from on-premises (via Direct Connect or VPN) and their DNS does not use the VPC’s .2 resolver, you need a Route 53 Resolver Inbound Endpoint:
Terminal window
# Create resolver inbound endpoint in the VPC associated with the scoped PHZ
Then configure your on-premises DNS server (Active Directory, BIND, Unbound, etc.) to conditionally forward the API-specific FQDN to the resolver endpoint IPs:
<api-id>.execute-api.<region>.amazonaws.com
Scoping the forwarder to this exact name (rather than the regional execute-api.<region>.amazonaws.com parent) keeps on-premises resolution of every other API Gateway API on its normal public path. Windows DNS, BIND, and Unbound all support conditional forwarding keyed to a full FQDN.
Note: If your DNS appliance can only forward at a broader suffix, you may forward execute-api.<region>.amazonaws.com instead. The blast radius stays contained on the AWS side — the scoped PHZ only answers for this API’s hostname, so any other execute-api name forwarded in still resolves via public DNS through the VPC resolver.
Note: If you already have a Route 53 Resolver Inbound Endpoint for other AWS services (e.g., S3, STS), you can reuse it. Just add the API FQDN to your on-premises conditional forwarder configuration.
Because the PHZ is named for the exact API FQDN, Route 53’s most-specific-match means the zone overrides DNS for only this API. Other API Gateway APIs in the region (public or private, yours or another team’s) keep resolving through normal public DNS. This is the key advantage over a regional-domain PHZ.
One zone per private API
Each ApiGatewayVisibility=PRIVATE API needs its own scoped zone (<api-id>.execute-api.<region>.amazonaws.com). Deploying multiple private IDP stacks in the same region simply means another scoped zone per stack — there is no shared zone to maintain and no record-enumeration burden.
Regional-zone alternative
If you genuinely want a single PHZ to cover every private API in a VPC, you can instead create one PHZ at the regional apex execute-api.<region>.amazonaws.com and add an alias record per API ID inside it. This is simpler to reason about for a fleet of private APIs, but it intercepts all API Gateway DNS in associated VPCs — any private API you do not add a record for will fail to resolve, and any public API in those VPCs breaks. Use it only when every execute-api consumer in those VPCs is private and accounted for.
Endpoint in multiple AZs
Deploy the execute-api VPC endpoint in at least 2 AZs for high availability. The alias record automatically load-balances across all endpoint ENIs.
PrivateDnsEnabled conflict
If PrivateDnsEnabled=true is set on the execute-api VPC endpoint, AWS creates a managed PHZ for the regional execute-api.<region>.amazonaws.com domain in the endpoint’s VPC. For VPCs associated with both that managed zone and your scoped zone, the scoped zone wins (most-specific match) — but to avoid confusion and unintended blanket interception, set PrivateDnsEnabled=false when using the PHZ approach for cross-VPC access.
No IDP stack changes required
This is purely a DNS/networking configuration. The IDP stack and frontend code work unchanged — the browser calls the same regional hostname it was built with.
A VPC endpoint already exists for that service. Re-run deploy-vpc-endpoints.py — it auto-detects and skips existing ones.
Browser upload fails: ERR_NAME_NOT_RESOLVED on bucket.vpce-...vpce.amazonaws.com
(a) Browser is not on the VPN — VPCE host is VPC-DNS-only. (b) Mode B: S3VpcEndpointDnsNameOverride was set without the AWS-assigned random suffix. Re-deploy with the full DNS from aws ec2 describe-vpc-endpoints ... DnsEntries[0].DnsName minus the *. prefix.
Browser upload fails: Failed to fetch / connect timeout
(a) S3 VPCE endpoint SG missing ingress for VPN NAT-subnet CIDR (see SG matrix). (b) VPN association subnet differs from S3 VPCE subnets — return path is fine if SG has the NAT CIDR.
Backend processing fails with ConnectTimeoutError on an AWS service endpoint
Shared endpoint SG (scripts/vpc-endpoints.yaml) does not allow ingress from this stack’s Lambda SG. Re-run deploy-vpc-endpoints.py, or add ingress 443 from the Lambda SG manually. Common when reusing a shared endpoint SG across stacks.
ConfigurationCopyFunction times out (5 min) during stack create
Shared service-endpoint SG (scripts/vpc-endpoints.yaml) missing ingress from the Lambda SG, so the function can’t reach S3/DynamoDB. Confirm LambdaSecurityGroupId is passed and re-run deploy-vpc-endpoints.py. Manual unblock: aws ec2 authorize-security-group-ingress --group-id <endpoint-sg> --protocol tcp --port 443 --source-group <lambda-sg>.
DashboardMerger custom resource times out
Missing monitoring (CloudWatch) Interface VPCE. Re-run deploy-vpc-endpoints.py — script now includes it.
WAF is enabled and your client IP is not in WAFAllowedIPv4Ranges, or (PRIVATE mode) the request did not arrive via the ApiGatewayVpcEndpointId execute-api endpoint the API resource policy pins to.
UI unreachable in PRIVATE mode
The execute-api endpoint’s private DNS is off, or its SG blocks your client. Confirm PrivateDnsEnabled: true and 443 ingress from your source network.
UI assets 404
The UI was not built with VITE_UI_BASE_PATH=/api/. CodeBuild sets this automatically when WebUIHosting=APIGateway; asset URLs in index.html must start with /api/.
Login hangs on cognito-idp.amazonaws.com (browser inside VPC)
No VPCE for Cognito IDP. Browser needs internet egress: NAT GW in a public subnet + private route 0.0.0.0/0 → NAT GW. End-user browsers on VPN don’t need this.
CodeBuild fails: timeout on pip install / npm ci
CodeBuild is in the VPC but has no route to package registries. Add a NAT Gateway, or configure an internal artifact repository and set registry environment variables. See Dependency Mirroring.
CodeBuild fails: AccessDenied: kms:Decrypt
Artifact bucket is KMS-encrypted but ArtifactsBucketKmsKeyArn was not passed. Redeploy with the key ARN.
Same as above — ConfigurationCopyFunction silently skipped due to missing kms:Decrypt. Pass ArtifactsBucketKmsKeyArn.
Mode B: change-set fails with S3VpcEndpointDnsNameOverride is required when S3VpcEndpointIdOverride is set
Working as intended. Both override params must be supplied together. Set both, or clear both to use Mode A.
Browser UI/API fails: ERR_NAME_NOT_RESOLVED on <api-id>.execute-api.<region>.amazonaws.com
Browser cannot resolve the regional REST API hostname. In cross-VPC/hybrid deployments, the execute-api VPC endpoint’s PrivateDnsEnabled only works within the VPC that owns the endpoint. See “Private REST API DNS resolution” section above — create the API-ID-scoped Route 53 PHZ (<api-id>.execute-api.<region>.amazonaws.com), add an apex alias to the execute-api endpoint, and associate it with the VPC where DNS queries originate.
UI loads but data operations (POST /op/{field}) fail / time out
(a) The execute-api endpoint SG is missing ingress 443 from the browser’s source network (see SG matrix). (b) The API resource policy pins access to a different ApiGatewayVpcEndpointId than the endpoint the request arrived through. Confirm the request carries the pinned endpoint id in x-amzn-vpce-id.
nslookup resolves the API to public IPs (not VPC endpoint IPs)
The scoped PHZ is not associated with the VPC where the DNS query runs, or PrivateDnsEnabled interference. Verify the association: aws route53 get-hosted-zone --id <phz-id> → check the VPCs list.
