Release Validation: v0.6.6

Release Validation Record — v0.6.6

Release: v0.6.6 (git 7fb426b27, VERSION = 0.6.6) Validated: 2026-08-28, us-west-2, commercial partition Artifact under test: the published template https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.6.yaml (Description (v0.6.6)) — not a local build. Every stack below was created from that object, except the two transform tiers, which must publish from source to apply the transform.

Verdict

✅ Ship. 14 of 14 tiers pass. No regression was found in shipped product code, in any tier. Three findings are recorded below; all three are in test/tooling code, not in the product, and one of them is a false-confidence bug in a security test that should be fixed regardless of this release.

The headline positive result: the integrated-confidence list-truncation defect that has been open in this project’s benchmark record since v0.6.0 is fixed, verified with repeated measures (9 of 9 complete, down from 2-of-4 truncating on v0.6.5) at 55% lower cost. See the benchmark entry.

---

Results

Offline tiers (no AWS)

Tier	Gate	Result
Build + package (publish.py --clean-build)	✅ PASS	240s; template validation + cfn-lint clean on all 4 packaged templates
Unit / integration suites (make test)	✅ PASS	all 52 test roots passed, 0 failures
Lint (make lint-cicd)	✅ PASS	all code-quality checks passed (check-only mode; no files modified)
Dependency audit (make dep-audit)	✅ PASS	no Python or Node advisory at or above HIGH
Type check (make typecheck)	⚠️ 2 errors	pre-existing, not a v0.6.6 change — see Finding 1

The 52-root make test result is notable in itself: the repo’s own testing skill documents a baseline of ~26 expected failures in the raw per-suite invocation. Run through make test, which supplies the PYTHONPATH and conftest wiring those suites need, the suite is fully green.

Security tiers

All four gates pass, and every one is numerically identical to v0.6.5 — no security regression. Redacted per-test detail: security/test-results/0.6.6/.

Tier	Gate	Result	vs v0.6.5
SRT (SAST + deps + security-matrix)	✅ PASS	0 open/reopened HIGH of 12,009 CI-visible findings	0 of 11,161 — same gate
RBAC static	✅ PASS	0 FAIL · 2 WARN (both documented, accepted gaps: GAP-01 getStepFunctionExecution, GAP-02 queryKnowledgeBase)	identical
RBAC dynamic	✅ PASS	556 checks, 0 hard failures — 106 operations × 5 roles, plus unauthenticated, malformed/expired-token, IDOR, TLS and deleted-resource cases	identical count
ZAP DAST	✅ PASS	High=0. 118 rules exercised: 117 PASS · 1 WARN · 0 FAIL. One Medium (Cross-Domain Misconfiguration, 5 instances)	same single Medium

The SRT run needs a caveat that matters operationally — see Finding 2.

Deploy-variant tiers (each self-deploys a stack, validates, tears down)

Tier	Gate	What it proved
APIGateway hosting, GLOBAL	✅ PASS	REGIONAL REST API serves the Web UI, HTTP 200
WAF-enabled hosting	✅ PASS	regional WebACL created and associated to the API stage
APIGateway hosting, PRIVATE (VPC)	✅ PASS	PRIVATE-type REST API created with a resource policy, reachable only via the execute-api VPC endpoint
Jobs API (EnableJobsApi, VPC)	✅ PASS	Jobs REST API deployed and serving on its own stage
Seller Entitlement Service e2e	✅ PASS	deploys into the seller account; unsigned → 403, unentitled → 403 with no detail leaked, unknown product byte-identical (no existence oracle), malformed body → 400, oversized → 413, 15 hostile payloads all refused with no 5xx

The two VPC tiers reused an existing VPC in the account that a prior stack-test run had already wired for this purpose (<VPC_ID> 10.0.0.0/16, two private subnets with NAT egress, a self-referencing 443 security group, and an execute-api interface endpoint with private DNS across both subnets). No VPC was created.

The seller tier failed on its first run for a reason that turned out not to be about the product at all — see Finding 3, which is the most important finding in this record.

Template-transform tiers — the only tier that deploys a transformed template

Both variants deployed a real stack from a transformed template via the documented user path (idp-cli deploy --headless|--govcloud --from-code . --wait), processed samples/lending_package.pdf end to end, and tore down. 2 of 2 passed.

Variant	Gate	Checks that ran	Resources
--headless	✅ PASS	✓ UI/Cognito/CloudFront absent · ✓ processing core present · ✓ sample document processed	193 main-stack resources
--govcloud	✅ PASS	✓ no AWS::CloudFront::* / AWS::Lambda::Url · ✓ LWA chat-stream family removed (#677) · ✓ UI retained, hosted on API Gateway · ✓ processing core present · ✓ sample document processed	279 main-stack resources

Both ran with a real document, not SKIP_DOC_TEST. The --govcloud run used the runner’s default of Knowledge Base disabled, so the KB path was not exercised in that variant.

⚠️ The --govcloud result does not mean “GovCloud works”

This run was in a commercial account (us-west-2). It proves the CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves nothing about GovCloud-specific behaviour — partition-correct ARNs, GovCloud model availability, and the BDA project rejection are all invisible outside us-gov-*. Two of the three defects that motivated this tier (#676, #677) would not have been caught by a commercial run.

A true GovCloud run is make transform-deploy-test-govcloud REGION=us-gov-west-1 against a GovCloud account. It was not performed for v0.6.6.

In-place upgrade v0.6.5 → v0.6.6

The customer-equivalent path: deploy the published v0.6.5 template, process a document, update-stack to the published v0.6.6 template reusing every parameter, process the same document again, compare.

Step	Result
v0.6.5 base stack CREATE_COMPLETE	✅
Parameter-set diff v0.6.5 → v0.6.6	identical, 83 ↔ 83 — no parameter added, removed or newly required, so UsePreviousValue=true on all 83 is sufficient
Baseline: lending_package.pdf on v0.6.5	✅ 1/1 completed
update-stack to v0.6.6	✅ UPDATE_COMPLETE in ~25 min, no rollback; UpdateDefaultConfig applied cleanly
Stack Description after update	AWS GenAI IDP Accelerator … (v0.6.6) ✅
Post-upgrade: same document re-processed	✅ 1/1 completed
Output comparison vs baseline	✅ identical file set (63 files); all 6 sections classified the same; identical field counts and null counts; 0 value differences

Per-section pre/post comparison:

Section	Class	fields	nulls	value differences
1	Payslip	33 → 33	11 → 11	0
2	US-drivers-licenses	22 → 22	4 → 4	0
3	Bank-checks	10 → 10	1 → 1	0
4	Bank-Statement	13 → 13	6 → 6	0
5	W2	23 → 23	2 → 2	0
6	Homeowners-Insurance-Application	41 → 41	8 → 8	0

Zero value drift across 142 extracted fields is a stronger result than this tier normally produces — some LLM non-determinism is expected and would not have been a regression.

Release benchmark A/B (v0.6.5 → v0.6.6)

Full entry: Release Benchmark: v0.6.6 vs v0.6.5. Headlines:

• 0 cell-level regressions, 0 failures in 90 runs.
• The integrated-confidence truncation hazard is fixed — 9/9 complete lists (recall 1.000, stdev 0.000) where v0.6.5 truncated 2 of 4 repeats — at −55% cost and a third of the wall time. Its accuracy also rose 0.667 → 1.000 (the group-field fix).
• Cost −5.8%/run overall, and all of it is that one cell; the other 9 are +4.0%, inside their sampling spread.
• One apparent recall drop (llm-simple-sep) root-caused to the bedrock_llm OCR backend corrupting fixed-width identifiers, with no code change in that path between the tags. Also relevant to operators — see the guidance note below.

Operator note carried over from the benchmark. The bedrock_llm OCR backend was observed transcribing SEQ00000 as SEQ000000 — an inserted digit — for every row on one page of a document while transcribing neighbouring pages correctly. No row was lost; 38% of the identifiers on a 100-row document were silently wrong, scalar accuracy stayed 1.000, and the run reported COMPLETED. Textract and BDA read the same documents correctly. Prefer a deterministic OCR backend (textract / textract_tables) when documents carry fixed-width identifiers that must be exact.

---

Findings

None of the three block the release. They are ordered by how much they matter.

Finding 3: the seller stack-test never forwards its region, and its assertions pass vacuously when it matters

Severity: medium — a security test that can report success without testing anything. Test-code only; the service itself is fine.

feature-platform/seller-entitlement-service/tests/stacktest.sh defines REGION (default us-east-1), accepts --region, and uses it for the CloudFormation deploy — but the invocation of the live probe at the bottom of the script passes only --endpoint and --product-id. dynamic_activation_test.py therefore falls back to its own --region us-east-1 default and SigV4-signs every request for us-east-1.

Run in any other region, API Gateway rejects all signed requests before they reach the Lambda with 403 {"message":"Credential should be scoped to a valid region. "}. That alone would only be a nuisance — but three of the probe’s assertions expect a 403, so they passed on the wrong 403:

• “unentitled account refused (403) with no internal detail leaked”
• “unknown product is byte-identical to not-entitled — no existence oracle”
• “15 hostile payloads all refused cleanly (no 5xx)”

None of those requests reached the handler. The only reason the run failed at all is that check_malformed_body expects a 400, not a 403.

Confirmed by re-running in the probe’s default region, where the same build passes cleanly (malformed → 400, oversized → 413, and the refusal assertions become genuine). The v0.6.6 seller result in the table above is the us-east-1 run, which is a real pass.

Two fixes are warranted, independent of each other:

1. Forward the region: dynamic_activation_test.py … --region "$REGION".
2. Make the refusal assertions non-vacuous — reject a 403 whose body is an API Gateway SigV4/authorizer error rather than the service’s own refusal. Otherwise any future misconfiguration that blocks requests at the edge will read as a clean security pass.

Finding 2: the SRT gate counts build artifacts

Severity: low — CI robustness. No security impact.

make srt-scan reported 48 open/reopened HIGH findings, and every one of them was in a gitignored build artifact (.aws-sam/idp-main.yaml, .aws-sam/idp-govcloud.yaml, */.aws-sam/packaged.yaml) — duplicates of source-template findings that are already suppressed against their source paths. 0 were in tracked source, which is why the curated snapshot records a clean gate: scripts/security/curate_results.py excludes gitignored findings “to match the CI view”.

The gate in scripts/srt/run.py does not apply that exclusion. It only exited 0 because is_ci was false; sys.exit(1) is taken when is_ci is set. So a CI run that scans after a build would fail on 48 findings in generated files. The pipeline is safe today only because of stage ordering. Applying the curator’s gitignore filter to the gate would make that safety explicit rather than incidental.

This run scanned after publish.py, which is what surfaced it.

Finding 1: make typecheck has 2 pre-existing errors

Severity: low — pre-existing, not a v0.6.6 change.

make typecheck (basedpyright over the whole repo) exits 3 with 2 errors, 44 warnings. Both errors are on one line, src/lambda/calculate_capacity/index.py:1267 (reportOperatorIssue on + and / over a loosely-typed union). git log -L dates that line to commit cb295699e (2026-02-19, the Capacity Planning feature) — it predates v0.6.6 by two releases.

CI type-checks only the files a PR changes (scripts/sdlc/typecheck_pr_changes.py), so the whole-repo target has been red without blocking anything. Worth fixing or explicitly baselining so make typecheck is a usable signal.

---

What was NOT validated

Stated explicitly so this record is not read as broader than it is:

• Real GovCloud. The --govcloud tier ran in a commercial account. See the warning above.
• The Knowledge Base path in --govcloud — disabled by default in that runner to avoid standing up an OpenSearch Serverless collection.
• The seller service’s positive path (a token actually issued and verified). That needs a genuinely subscribed buyer account, which cannot be created on demand; the probe covers refusals only and says so.
• Documents above 100 rows, and real (non-synthetic) corpora, in the benchmark A/B — only the corefast grid was run. This bounds the “integrated confidence is fixed” claim to ≤100-row lists.
• A same-stack benchmark A/B. The v0.6.5 side is the committed release baseline rather than a fresh re-run; see the deviation note in the benchmark entry.

Reproduce

AWS_PROFILE=default aws sts get-caller-identity     # confirm the deployment account
URL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.6.yaml

# offline tiers
make test && make lint-cicd && make dep-audit && make typecheck

# a stack to test against
aws cloudformation create-stack --stack-name <STACK> --region us-west-2 --template-url "$URL" \
  --parameters ParameterKey=AdminEmail,ParameterValue=<you> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND

# security tiers + curated snapshot
AWS_PROFILE=default make security-results STACK_NAME=<STACK> REGION=us-west-2

# deploy variants (each self-deploys and tears down)
AWS_PROFILE=default make stacktest-hosting-global TEMPLATE_URL="$URL" REGION=us-west-2 ADMIN_EMAIL=<you>
AWS_PROFILE=default make stacktest-waf            TEMPLATE_URL="$URL" REGION=us-west-2 ADMIN_EMAIL=<you>
AWS_PROFILE=default make stacktest-hosting-private TEMPLATE_URL="$URL" REGION=us-west-2 ADMIN_EMAIL=<you> \
    VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>
AWS_PROFILE=default make stacktest-jobsapi         TEMPLATE_URL="$URL" REGION=us-west-2 ADMIN_EMAIL=<you> \
    VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>
AWS_PROFILE=default make stacktest-seller          # leave REGION unset — see Finding 3

# transform tiers (publish from source; ~1h each)
AWS_PROFILE=default make transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<you>

# upgrade tier — see .claude/skills/test-upgrade.md
# benchmark tier — see .claude/skills/run-benchmarks.md

idp-cli on PATH may resolve to a different checkout; the transform tiers shell out to it by name, so confirm idp-cli imports idp_cli from the repo under test before running them.