Release Validation: v0.6.8

Release Validation Record — v0.6.8

Release: v0.6.8 (tag v0.6.8 = 267de2868; validated from develop d584958e0, which differs from the tag only in docs, CHANGELOG and a CI-workflow file — no product code) Validated: 2026-09-11 → 2026-09-12, us-west-2, commercial partition Artifact under test: the published template https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.8.yaml (Description (v0.6.8)) — not a local build. Every stack below was created from that object (or from the published idp-main_0.6.7.yaml where the tier starts on the previous release), except the two transform tiers, which must publish from source to apply the transform.

Verdict

✅ Ship. 18 of 18 tiers pass — 5 offline (type check at its pre-existing baseline), 4 security, 5 deploy-variant, 2 transform, the in-place upgrade, and the release benchmark A/B. No regression was found in shipped product code in any tier. Four findings are recorded below; none is in the product. One is about the validation account (it hit its IAM role quota, which failed two tiers on their first attempt and is why this record shows re-runs), three are in test tooling — two of which produce a false red in a security or test gate, and one of which makes the seller stack-test unrepeatable.

Two results are worth reading even if nothing else is: the security snapshot shows the getStepFunctionExecution authorization gap carried since v0.6.5 is closed, and the benchmark shows the assessment phase 37% cheaper with completeness and accuracy unchanged. The upgrade tier is the one place the record is less clean than v0.6.6 — 14 cosmetic value differences on the same document, listed in full, none a changed number or classification.

---

Results

Offline tiers (no AWS)

Tier	Gate	Result
Build + package (publish.py --clean-build)	✅ PASS	4m10s; template validation + cfn-lint clean on all 5 packaged templates
Unit / integration suites (make test)	✅ PASS	all 59 test roots pass, 0 failures — after removing two stale git worktrees under scratch/, see Finding 4
Lint (make lint-cicd)	✅ PASS	all code-quality checks passed (check-only mode)
Dependency audit (make dep-audit)	✅ PASS	1,811 packages audited, 0 gating, 3 allowlisted, 7 below HIGH
Type check (make typecheck)	⚠️ 2 errors / 47 warnings	both errors on the same pre-existing line in src/lambda/calculate_capacity/index.py recorded for v0.6.6; fewer than the 4 errors / 51 warnings baseline the testing skill documents for develop on 2026-09-11

Security tiers

All four gates pass. Redacted per-test detail: security/test-results/0.6.8/. The live tests ran against the upgraded stack from the upgrade tier below (published 0.6.7 → published 0.6.8), so the surface tested is the one a customer who upgrades actually gets.

Tier	Gate	Result	vs v0.6.7
SRT (SAST + deps + security-matrix)	✅ PASS	0 open/reopened HIGH of 16,352 CI-visible findings; 111 HIGH dispositioned (59 resolved, 52 suppressed)	0 of 17,729; 104 dispositioned — 7 new HIGH findings, all dispositioned before release (the throwaway OIDC test-fixture template)
RBAC static	✅ PASS	0 FAIL · 1 WARN (GAP-02 queryKnowledgeBase)	GAP-01 (getStepFunctionExecution had no group or ownership check) is closed — 2 WARN → 1
RBAC dynamic	✅ PASS	604 checks, 0 hard failures — 115 operations × 5 roles, plus unauthenticated, malformed/expired-token, IDOR, TLS and deleted-resource cases; 1 known-gap warning (stateless-JWT logout)	596 checks / 114 ops — one operation added
ZAP DAST	✅ PASS	High=0. 118 rules: 116 PASS · 1 WARN · 0 FAIL · 1 IGNORE. One Medium (Cross-Domain Misconfiguration, 5 instances)	same single Medium; the IGNORE is rule 10096 Timestamp Disclosure, muted in zap-rules.conf since 2026-07-20 and simply not triggered on 0.6.7 — no gate change

The SRT tier needed two runs — the first reported 10 open HIGH findings that were not real. See Finding 2.

Deploy-variant tiers (each self-deploys a stack from the published template, validates, tears down)

Tier	Gate	What it proved
APIGateway hosting, GLOBAL	✅ PASS	REGIONAL REST API serves the Web UI, HTTP 200 (38 min incl. teardown)
WAF-enabled hosting	✅ PASS	regional WebACL created and associated to the API stage
APIGateway hosting, PRIVATE (VPC)	✅ PASS (2nd attempt)	PRIVATE-type REST API created with a resource policy, reachable only via the execute-api VPC endpoint. The first attempt failed on the account’s IAM role quota, not the template — see Finding 1
Jobs API (EnableJobsApi, VPC)	✅ PASS	Jobs REST API deployed and serving on its own stage
Seller Entitlement Service e2e	✅ PASS (3rd attempt)	deploys into the account; unsigned → 403, unentitled → 403 with no detail leaked, unknown product byte-identical (no existence oracle), malformed → 400, oversized → 413, 15 hostile payloads refused with no 5xx. Run in us-east-1 (the probe’s own default) so the refusal assertions are genuine — the region-forwarding bug from the v0.6.6 record is still open. The first two attempts failed before the stack existed — see Finding 3

The two VPC tiers reused the account’s purpose-built stack-test VPC (<VPC_ID> 10.0.0.0/16, two private subnets with NAT egress, a self-referencing 443 security group, and an execute-api interface endpoint with private DNS). No VPC was created.

Template-transform tiers — the only tier that deploys a transformed template

Both variants deployed a real stack from a transformed template via the documented user path (idp-cli deploy --headless|--govcloud --from-code . --wait), processed samples/lending_package.pdf end to end, and tore down. 2 of 2 passed.

Variant	Gate	Checks that ran	Resources
--headless	✅ PASS (2nd attempt)	✓ UI/Cognito/CloudFront absent · ✓ processing core present · ✓ sample document processed	219 main-stack resources
--govcloud	✅ PASS	✓ no AWS::CloudFront::* / AWS::Lambda::Url · ✓ LWA chat-stream family removed (#677) · ✓ UI retained, hosted on API Gateway · ✓ processing core present · ✓ sample document processed	304 main-stack resources

Both ran with a real document, not SKIP_DOC_TEST. The --govcloud run used the runner’s default of Knowledge Base disabled, so the KB path was not exercised in that variant. The first --headless attempt failed at DocumentProcessingStateMachineRole on the IAM role quota (Finding 1); the identical build deployed cleanly once headroom existed. Resource counts are up from v0.6.6’s 193 / 279; v0.6.7 was not recorded, so the increase spans two releases.

⚠️ The --govcloud result does not mean “GovCloud works”

This run was in a commercial account (us-west-2). It proves the CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves nothing about GovCloud-specific behaviour — partition-correct ARNs, GovCloud model availability, and the BDA project rejection are all invisible outside us-gov-*. Two of the three defects that motivated this tier (#676, #677) would not have been caught by a commercial run.

A true GovCloud run is make transform-deploy-test-govcloud REGION=us-gov-west-1 against a GovCloud account. It was not performed for v0.6.8.

In-place upgrade v0.6.7 → v0.6.8

The customer-equivalent path: deploy the published v0.6.7 template, process a document, update-stack to the published v0.6.8 template reusing every parameter, process the same document again, compare.

Step	Result
v0.6.7 base stack CREATE_COMPLETE	✅ (~33 min)
Parameter-set diff v0.6.7 → v0.6.8	86 → 88: two added, both defaulted (ExternalIdPEmailMutable, WorkflowExecutionTimeoutSeconds), none removed, none newly required. One default changed: LogLevel INFO → WARN — an upgraded stack keeps INFO via UsePreviousValue; only fresh stacks get WARN
Baseline: lending_package.pdf on v0.6.7	✅ 1/1 completed, 193 s
update-stack to v0.6.8	✅ UPDATE_COMPLETE in ~22 min, no rollback; UpdateDefaultConfig applied cleanly
Stack Description after update	AWS GenAI IDP Accelerator … (v0.6.8) ✅
Post-upgrade: same document re-processed	✅ 1/1 completed, 189 s
Output comparison vs baseline	✅ identical file set (63 files); all 6 sections classified the same; 14 value differences, all cosmetic (below)

Per-section pre/post comparison:

Section	Class	fields	nulls	value differences
1	Payslip	48 → 48	11 → 9	13
2	US-drivers-licenses	20 → 22	4 → 5	1
3	Bank-checks	10 → 10	2 → 2	0
4	Bank-Statement	15 → 15	4 → 4	0
5	W2	35 → 35	2 → 2	0
6	Homeowners-Insurance-Application	41 → 41	8 → 8	0

The 14 differences are, in full: seven Payslip name/address fields changed from the document’s upper case to title case (ANYTOWN → Anytown, JOHN → John, …); five Payslip tax Period values gained a space after the minus sign (-5.94 → - 5.94); two Payslip fields that were null on v0.6.7 were extracted on v0.6.8 (EmployeeNumber 126543, PayrollNumber 00000000) — the same two fields the prerelease benchmark audit found flipping between null and correct on both code versions and filed as #839; and one driver’s-license ENDORSEMENTS[0] went from null to the literal NONE, which is what the document prints. No numeric value, date or classification changed. This is LLM run-to-run variance plus the v0.6.8 default prompt’s casing normalisation, not a regression — but it is not the zero-drift result v0.6.6 produced, so it is recorded rather than rounded down.

Release benchmark A/B (v0.6.7 → v0.6.8)

Full entry: Release Benchmark: v0.6.8 vs v0.6.7. Both sides are the published templates, run the same night with byte-identical configs (corefast, 19 cells × 3 documents × 3 repeats per side). Headlines:

• 0 regressions, 0 improvements at cell level, 0 failures in 171 + 171 runs. Recall 1.000 on every cell on both sides — 11,685 of 11,685 truth rows, 23,370 cells compared, so it is a full-row count. Cell accuracy 1.000 everywhere except the Bedrock-LLM OCR cell, 0.889 → 0.944 (within spread).
• Assessment cost −37% across the grid ($1.12 → $0.70), the Nova Lite confidence-loop fix (#861) shipped in this release; it lands on the integrated-confidence simple cell (core-tt-simple-int $0.181 → $0.127, −30%). Grid total −3.2%; OCR and classification cost identical to the cent.
• Wall time −31% per run (115 s → 79 s mean over cells), consistent across the simple cells. This is the least controlled number in the record — the two grids ran on two stacks — so read it as “not slower”, not as a measured speed-up.
• The section-splitting fix (#726/#817) that the prerelease audit reported as 2–3 → 1 sections per document does not appear as a delta here, because it ships as default configuration and the A/B’s byte-identical configs put it on both sides. Sections per document were 1.00 on both. The benchmark entry explains this and keeps the prerelease comparison that shows the combined effect.

Procedure deviation, stated plainly: the standard A/B runs both sides on one stack. Here the PREV grid ran on a sibling stack created from the same published 0.6.7 template minutes earlier (IDPBench067), concurrently with the NEW grid on the upgraded stack, to recover time lost to Finding 1. Cost, recall and accuracy are unaffected by that; wall time may be.

---

Findings

None of the four blocks the release, and none is in shipped product code. They are ordered by how much they cost this validation; Findings 2 and 3 are the ones worth fixing before the next one.

Finding 1: the account hit its IAM roles-per-account quota mid-validation

Severity: environment — blocks validation, not the product.

Two tiers failed on their first attempt with the same CloudFormation error on the same resource, DocumentProcessingStateMachineRole in the nested PATTERNSTACK:

Cannot exceed quota for RolesPerAccount: 2000 (Service: Iam, Status Code: 409)

Each IDP stack creates roughly 121 roles (158 with every feature on). The validation account already carried four standing IDP stacks plus two other projects, sitting at ~1,740 roles; five IDP stacks deploying at once pushed it over 2,000. The two failures (--headless transform, APIGateway PRIVATE hosting) are therefore attributable to the account, not the template — the identical template deployed cleanly in the GLOBAL and WAF variants minutes earlier and the same resource created fine once headroom existed.

Two consequences for anyone running this battery:

1. The role quota, not API rate limits, is the binding constraint on parallel stack-tests in a shared account. Count aws iam get-account-summary before starting; budget ~160 roles per in-flight stack.
2. The quota is adjustable. A Service Quotas request to raise L-FE177D64 to 3,000 was auto-approved in under ten minutes during this run, after which the remaining deploys ran two at a time without incident. The release-validation skill now says to check headroom first.

Finding 2: make srt-scan without make srt-setup reports suppressed findings as Open

Severity: medium — a false red in a security gate, and a false green in the opposite direction is one step away. Tooling only; the product is unaffected.

The first make srt-scan of this validation reported 10 open/reopened HIGH findings in tracked source — nine in the throwaway OIDC test-fixture template scripts/security/live_checks/oidc_provider/template.yaml and one (LAMBDA-012) in nested/bedrockkb/template.yaml. Every one of them is dispositioned suppressed, with a written reason, in the committed register scripts/srt/issues.json, and CI’s SRT job had passed on develop the same afternoon.

The cause is how dispositions reach the scanner. SRT keeps its own state in gitignored .srt/issues.json; the committed register is copied over it by make srt-setup, not by make srt-scan. CI runs both. A local make srt-scan alone merges the new scan into whatever .srt/issues.json already holds — here a state from 2026-09-04 that predates the suppressions added on 2026-09-09 — so the scanner saw those findings for the first time (firstDetectedAt = this scan) and opened them. make srt-setup && make srt-scan on the same tree: 0 open, gate PASS, which is the number the snapshot records.

scripts/security/run_security_tests.sh (behind make security-results) has the same gap: it calls make srt-scan only. Run locally on a stale .srt/, it would stamp a false SRT failure into the published snapshot. Worth fixing in the script; until then the release-validation skill runs setup first.

Two smaller points from the same episode: make srt-setup prompts interactively for an AWS profile when not in CI (set CI=1 to get the non-interactive path), and interrupting that prompt left SRT with no config and no scanner venv — CI=1 python scripts/srt/setup.py rebuilt both.

Finding 3: the seller stack-test cannot be re-run in the same account

Severity: medium — the test fails before it tests anything, on every run after the first. Test-code only; the service itself passed.

make stacktest-seller failed twice at changeset creation:

The following hook(s)/validation failed: [AWS::EarlyValidation::ResourceExistenceCheck]

CloudFormation’s early-validation hook now rejects a changeset that would create a named resource that already exists. The seller template names its API Gateway access-log group (/aws/apigateway/${AWS::StackName}-activation); it does not retain it, yet the group was present after both teardowns — most likely re-created by API Gateway’s buffered access-log delivery once the stage was gone (unverified; the fix should confirm from the group’s creationTime). tests/teardown_test_stack.sh cleans up the retained KMS key and DynamoDB table but never looks for log groups. So every run leaves behind exactly the resource that blocks the next one. Confirmed: the third run passed after deleting the leftover group by hand, and its own teardown left a fresh copy behind. Fix: have the teardown delete the stack’s named log groups after stack-delete-complete, or stop giving a throwaway test stack a fixed-name access-log group.

The v0.6.6 finding that the script does not forward --region to the live probe is still open; this run used the probe’s default region so its refusal assertions are real.

Finding 4: the IAM privilege-escalation gate scans gitignored worktrees

Severity: low — a false red in make test.

scripts/tests/test_iam_privilege_escalation.py walks the whole repo for CloudFormation templates and prunes .aws-sam, node_modules, .venv, build, dist, .git, __pycache__ — but not scratch/ and not gitignored paths in general. Two stale git worktree checkouts left under scratch/ by earlier benchmark sessions made both privilege-escalation tests fail on templates that are already in the repo’s exemption list under their real paths. Removing the worktrees: 19/19 pass. Same class as the SRT build-artifact issue recorded for v0.6.6; the gate should honour git check-ignore or prune scratch.

---

What was NOT validated

Stated explicitly so this record is not read as broader than it is:

• Real GovCloud. The --govcloud tier ran in a commercial account. See the warning above.
• The Knowledge Base path in --govcloud — disabled by default in that runner.
• The seller service’s positive path (a token actually issued and verified). The probe covers refusals only and says so.
• Documents above 100 rows, and real (non-synthetic) corpora, in the release A/B itself — the A/B is corefast by design. Both are covered by the guidance-paper refresh below (scaling to 3,200 rows at two models; RealKIE and the OCR benchmark at two models).
• A one-stack benchmark A/B. The two grids ran on sibling stacks the same night; see the deviation note in the benchmark entry before citing a wall-time delta.
• The new v0.6.8 features themselves beyond deployability: the Cohere Parse hook and the WorkflowExecutionTimeoutSeconds alarm were deployed by every stack here but not driven with documents or events. That is feature testing, which this record does not claim.
• The premium and lightweight model studies. Done on 2026-09-12/13, after this record was first written. The astravalue and astracap suites, the extraction-model sweep (Nova Lite, Nova Pro, Sonnet 5, Sonnet 5 :1m, Opus 5, GPT-6 Astra), the classification and confidence model sweeps, and the full guidance-paper refresh (core with both reference corpora, scaling, cost, intconf, advverify, and every knob suite) were run on the reference stack — 3,455 runs, $2,114 — and docs/benchmarking/config-guidance.md is re-headed to v0.6.8 with a new §5 model-selection section (PR #888). That work found two further product defects: #894 (a multi-instance-wrapped class with a long list cannot finish confidence assessment) and #895 (Nova Lite cannot run the agentic path; every failure retried as transient), plus #892 (the TestRunner rejects any compressed configuration carrying a float — fix in PR #893), which had to be hot-patched onto the reference stack before the OCR-benchmark corpus would launch. None is a regression; all three are in the guidance paper’s backlog.

Reproduce

AWS_PROFILE=default aws sts get-caller-identity     # confirm the deployment account
aws iam get-account-summary --query 'SummaryMap.[Roles,RolesQuota]'   # Finding 1: budget ~160 roles per in-flight stack
FROM=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.7.yaml
TO=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.8.yaml

# offline tiers (SRT before the build; setup restores the committed dispositions — Finding 2)
CI=1 python scripts/srt/setup.py && make srt-scan
make test && make lint-cicd && make dep-audit && make typecheck
env -i HOME=$HOME PATH=/usr/local/bin:/usr/bin:/bin AWS_PROFILE=default bash -lc \
  'python3 publish.py <bucket-basename> idp us-west-2 --clean-build'

# upgrade tier (then the live security tests run against the upgraded stack)
aws cloudformation create-stack --stack-name IDPUpg067to068 --template-url $FROM \
  --parameters ParameterKey=AdminEmail,ParameterValue=<you> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND --disable-rollback
idp-cli run-inference --stack-name IDPUpg067to068 --dir samples/ --file-pattern lending_package.pdf --batch-prefix pre-upgrade --monitor
aws cloudformation update-stack --stack-name IDPUpg067to068 --template-url $TO \
  --parameters file://<every existing key with UsePreviousValue=true> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
idp-cli run-inference --stack-name IDPUpg067to068 --dir samples/ --file-pattern lending_package.pdf --batch-prefix post-upgrade --monitor
make security-results STACK_NAME=IDPUpg067to068 REGION=us-west-2 SKIP_SRT=1

# deploy variants (two at a time at most; the VPC ones need the four VPC params)
make stacktest-hosting-global  TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you>
make stacktest-waf             TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you>
make stacktest-hosting-private TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…
make stacktest-jobsapi         TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…
make stacktest-seller          # REGION unset; delete /aws/apigateway/idp-seller-entitlement-citest-activation first (Finding 3)

# transforms (publish from source, deploy, process a document, tear down)
make transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<you>

# benchmark A/B — see docs/benchmarking/releases/v0.6.8.md