Release Validation: v0.6.9

Release Validation Record — v0.6.9

Release: v0.6.9 (VERSION = 0.6.9, CHANGELOG.md section cut at 1c67405ab) Validated: 2026-09-18 → 2026-09-19, us-west-2 (seller tier us-east-1), commercial partition Artifact under test: a self-published build of the release commit — s3://<PRIVATE_BUCKET>/idp069/idp-main.yaml, Description (v0.6.9), built with publish.py … --clean-build from develop 08b1ed242.

⚠️ This is a pre-publish validation, and that is a real difference from every previous record in this directory. idp-main_0.6.9.yaml was not present in any of the three public aws-ml-blog-* buckets while this ran (HTTP 403 in all three; idp-main_0.6.8.yaml returns 200). Earlier records validate the published object. This one validates a build of the same commit, published to a private bucket. The nested-stack TemplateURLs therefore point at that private prefix, so what is proven is the template content, not the public objects’ presence or permissions. Re-run the tier-C and tier-D rows against the public URLs once the release is published, per docs/release-runbook.md.

The v0.6.8 side of every comparison below is the published idp-main_0.6.8.yaml.

Verdict

✅ Ship — with one qualification that is unusual enough to state before the tier table.

11 of 12 tiers pass. Tier G (browser UX review) is NOT RUN: this session had no browser MCP server (the chrome-devtools server the ux-test skill requires is not registered, and the browsermcp server that is registered reported CONNECTION_CLOSED). It is not reported as passing and it is not dropped.

The qualification: the release as first committed could not be deployed at all by anyone using the optional CloudFormation service role, and this validation is what found that. The #927 service-role hardening shipped with iam:PassRole scoped by an iam:PassedToService list that omitted cloudformation itself — so the first nested stack the parent creates (DOCUMENTKB) was denied and every deploy rolled back. Two independent stacks reproduced it. Fixed in b249ebf71, after which all five deploy variants and both transforms passed. The commit that fix landed on is the one validated here, and the fix is not in the document-processing path.

No regression was found in the document-processing path. The upgrade tier extracted more data after the upgrade than before (172 leaf fields vs 166, identical classifications, nothing lost). The one genuine accuracy movement in the benchmark A/B is confined to the bedrock_llm OCR backend on a single scalar field (Finding 2) and is a second instance of a hazard already documented for that backend.

---

Results

Offline tiers (no AWS)

Tier	Gate	Result
Build + package (publish.py --clean-build)	✅ PASS	3.5 min; cfn-lint clean on all packaged templates; Description (v0.6.9)
Unit / integration suites (make test)	✅ PASS	all 59 test roots pass, 0 failures
Lint (make lint-cicd)	✅ PASS	all code-quality checks passed (check-only)
Dependency audit (make dep-audit)	✅ PASS	1,813 packages audited, 0 gating, 3 allowlisted, 7 below HIGH
Type check (make typecheck)	✅ PASS	0 errors / 67 warnings against the documented develop baseline of 4 errors / 51 warnings — errors down 4, warnings up 16. Report the delta, not the raw count
SRT (SAST + deps + security matrix)	✅ PASS	0 open / 0 reopened HIGH in tracked source; 52 suppressed. Three “Open” entries are LOCAL-ONLY scratch/ probe files and explicitly non-gating

Security tiers

All four gates pass. Redacted per-test detail in security/test-results/0.6.9/ (git sha 08b1ed242). The live tests ran against a stack at v0.6.9.

Tier	Gate	Result	vs v0.6.8
SRT (SAST + deps)	✅ PASS	0 open/reopened HIGH of 18,204 tracked findings	0 of 16,352 — 1,852 more findings tracked, still 0 open
RBAC static	✅ PASS	0 fail · 2 known-gap warn	moved: 1 → 2 warn. The new one is GAP-07, added deliberately by this release; GAP-02 (queryKnowledgeBase) carries over
RBAC dynamic	✅ PASS	604 checks, 0 hard failures	604 checks — unchanged
ZAP DAST	✅ PASS	High = 0	unchanged

Per the skill’s rule that any movement from PREV is a finding: the RBAC-static warn count moved up by one, and the SRT tracked-finding count rose by 1,852. Both are explained — a deliberately-added known gap, and more code under scan — and neither is a gate failure, but they are stated rather than smoothed over.

Deploy-variant tiers (each self-deploys from the built template, validates, tears down)

Variant	Gate	Result
APIGateway hosting GLOBAL	✅ PASS	REGIONAL REST API served the UI, HTTP 200
WAF (IP allow-list)	✅ PASS	WebACL associated with 1 resource
APIGateway hosting PRIVATE (VPC)	✅ PASS	deployed and validated in the stack-test VPC
Jobs API (VPC)	✅ PASS	deployed and validated in the stack-test VPC
Seller Entitlement Service (us-east-1)	✅ PASS	”All checks passed”, 16 assertions — unsigned request 403, unentitled 403 with no detail leak, 15 hostile payloads refused with no 5xx

VPC used: <VPC_ID> — the account’s purpose-built stack-test VPC (security group idp-stacktest-vpce-sg, two private subnets, an execute-api interface endpoint). No VPC was created for this run.

The seller tier ran in us-east-1, where its refusal assertions exercise the service’s own authorization rather than API Gateway’s SigV4 rejection — so those three passes are real.

All five variants failed on their first attempt, at two different points, for the two halves of Finding 1. The results above are from the runs after fd4acecba and b249ebf71.

Template-transform tiers — the only tier that deploys a transformed template

Both variants publish from source, apply the transform, deploy, and process samples/lending_package.pdf. Neither ran with SKIP_DOC_TEST=1.

Variant	Gate	Checks
--headless	✅ PASS	✓ UI / Cognito / CloudFront absent · ✓ processing core present · ✓ document processed (130 s, 17/22 fields populated, classified Payslip)
--govcloud	✅ PASS	✓ no AWS::CloudFront::* / AWS::Lambda::Url · ✓ LWA chat-stream family removed (#677) · ✓ UI retained, hosted on API Gateway · ✓ processing core present · ✓ document processed (150 s, 17/22 fields, Payslip)

⚠️ The --govcloud result is a commercial-partition result. It proves the CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves nothing about partition ARNs, GovCloud model availability or the BDA project rejection (#676/#677 were invisible to a commercial run). Real GovCloud is listed under What was NOT validated. The Knowledge Base was disabled in the --govcloud run — the runner’s default; --with-knowledge-base was not passed.

Both stacks deployed and processed their document on the first attempt. The structural assertions above come from re-validating the same two deployed stacks in place — the path the script documents — rather than from a second deploy.

In-place upgrade v0.6.8 → v0.6.9

The customer path on a fresh throwaway stack IDPUpg068to069, created from the published idp-main_0.6.8.yaml, document processed, then update-stack to the v0.6.9 build with UsePreviousValue=true on every existing parameter.

• UPDATE_COMPLETE, no rollback, no FAILED events. Description now (v0.6.9).
• Parameter set: 88 → 89 — one added, none removed, none newly required.
• UpdateDefaultConfig / UpdateConfigurationFunction logged no error, no migration failure and no rollback path. (This is where upgrades deadlock, so it was tailed during the update.)
• Document processed 1/1 on both sides. S3 outputs were pulled before teardown.

Per-section diff of the same lending_package.pdf — 6 sections both sides, identical classifications (Payslip, US-drivers-licenses, Bank-checks, Bank-Statement, W2, Homeowners-Insurance-Application):

Measure	0.6.8	0.6.9
Leaf fields extracted	166	172
Null / empty leaves	30	31
Keys present on only one side	0 only-pre	6 only-post
Value differences on shared keys	—	2

The 6 new keys are FederalTaxes[1] and [2] on the Payslip section (ItemDescription, Period, YTD each). The payslip lists six statutory deductions, so a three-entry FederalTaxes list is more complete than the one-entry list 0.6.8 returned; that is also why the null count rose by one. Nothing was lost — every key present on 0.6.8 is present on 0.6.9.

The 2 value differences are both on the Bank-checks section, and the page text supports either reading, so both are recorded as behaviour changes rather than wins or losses:

1. check_number: 0000000000 → 0001379. The check carries both a labelled field (Payroll check number: 0000000000, a placeholder on this VOID sample) and a MICR line (⑈001379⑈ ⑆122000496⑆). 0.6.8 returned the labelled field; 0.6.9 returned the MICR value, zero-padded to seven characters. Neither is wrong on this document — but a customer keying off the labelled field would see the value change.
2. bank_name: "BANK NAME" → null. The page literally contains the text BANK NAME, the placeholder where a real check prints the bank. 0.6.8 returned the literal string; 0.6.9 abstained. This is consistent with the release’s nullable-scalar-leaf transport change (an agent that cannot answer returns null rather than echoing a placeholder), and it is the one difference that could hide a real loss on a document where the bank name is genuinely present. Not reproduced on any other field or section here.

No extracted number changed.

Release benchmark A/B (published v0.6.8 → the v0.6.9 build)

corefast, 19 cells × 3 docs × 3 repeats = 171 runs per side, 342 total, 0 failures on either side. Same stack IDPUpg068to069: the 0.6.8 side ran before update-stack, the 0.6.9 side after, so the only difference is the code. --native-upload, byte-identical configs, same pricing table (pricing.yaml sha256 a8897364… both sides), both sides scored by the same aggregate.py.

aggregate.py --compare: 2 flagged regressions, 0 improvements, 0 cell-level regressions, 8 inconclusive. One flagged regression is real, one is noise; full treatment in docs/benchmarking/releases/v0.6.9.md.

• Real — core-llm-simple-sep | small_narrow.pdf: scalar_accuracy 1.000 → 0.500, deterministic 3/3 each side. See Finding 2. Everything else on that cell is unchanged and perfect (completeness_recall 1.000, rows_extracted 100/100, cell_accuracy 1.000 over cells_compared = 200).
• Noise — core-tl-adv-sep | longdesc_100.pdf: cost +22% (n=3 each side) inside a cell whose own cost CV is 0.50–0.62, and −46% at the cell level. --compare reports 0 cell-level regressions.

⚠️ No cost claim should be drawn from this A/B, in either direction. 0.6.9 changes the cost instrument: it fixes long-context (:1m) models being reported at up to double their real cost, and fixes cache tokens being mispriced by substring match. The per-run cost comes from the stack’s own metering, so the 0.6.9 side measures with a corrected ruler. Eight cells show large cost decreases (−23% to −53%); part of every one of those is the fix rather than a behaviour change, and this grid cannot separate the two. Quote the single-release cost suites re-measured on 0.6.9 in the guidance paper instead.

The same caveat applies to accuracy in the other direction: stickler-eval 0.5.0 → 1.0.0 now infers evaluation comparators per field. Both sides here were scored by the same (new) scorer, so this A/B is internally consistent — but its 0.6.8 numbers are not the ones in the published v0.6.8 benchmark entry, which were scored with the old comparators.

baseline.json is deliberately not promoted, because the cost half of the comparison measures a changed instrument and would mislead the next release’s gate.

Guidance-paper refresh

See docs/benchmarking/config-guidance.md. Scope and which sections carry re-measured data are stated in that document’s header and Appendix A.

Browser UX review — NOT RUN

make ux-test needs a browser MCP server to drive scripts/ux_flows.yaml. The chrome-devtools server the skill documents is not registered in this session, and the browsermcp server that is registered reported CONNECTION_CLOSED. No flow was exercised in a browser, so nothing about the web UI’s usability or flow-level behaviour on this release is evidenced by this record. The UI was, however, built by every stack in tiers C–E, and the APIGateway-GLOBAL variant confirmed it is served (HTTP 200).

---

Findings

Two findings in the shipped solution, both fixed or fully characterised before this record was written, ordered by how much they matter. The first is the reason this validation was worth running: it would have stopped a customer deploying.

Defects found in the project’s own test and build tooling are tracked in the issue tracker rather than here. This record is about the release.

Finding 1: the hardened CloudFormation service role could not deploy the solution (PRODUCT, ship-blocking)

Two coupled defects from the #927 service-role hardening, both fixed.

1a — the required parameter was never passed. iam-roles/cloudformation-management/IDP-Cloudformation-Service-Role.yaml gained a required, no-default CreatedRolePermissionsBoundaryArn. create_iam_resources() in scripts/sdlc/codebuild_deployment.py created that stack with no Parameters at all, so every caller died before deploying anything:

ValidationError: Parameters: [CreatedRolePermissionsBoundaryArn] must have values

Callers: all four stacktest-* variants, both transform-deploy-test-* tiers, and the SDLC pipeline’s own step 0. Nothing in the repository ever deployed a stack through this role, so the hardening shipped with every deployer that uses it broken. Fixed in fd4acecba, which creates the boundary policy first, passes CreatedRolePermissionsBoundaryArn and ManagedStackNamePrefix explicitly, wires the same ARN into the IDP stack’s PermissionsBoundaryArn, and adds test_script_deployed_template_parameters.py.

1b — the service role could not pass itself to its own nested stacks. With 1a fixed, the deploy still rolled back:

✗ DOCUMENTKB (AWS::CloudFormation::Stack)
  User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/idp-…-CFServiceRole/AWSCloudFormation
  is not authorized to perform: iam:PassRole on resource:
  arn:aws:iam::<ACCOUNT_ID>:role/idp-…-iam-CFServiceRole
  because no identity-based policy allows the iam:PassRole action
(59 additional resource(s) cancelled due to the above failure(s))

The PassStackRolesToStackServices statement granted iam:PassRole under StringEqualsIfExists { iam:PassedToService: [...] }, and that list of 14 services did not include cloudformation. When the parent stack creates a nested stack, CloudFormation propagates the parent’s service role to the child — an iam:PassRole whose iam:PassedToService is cloudformation. The Resource matched (role/idp*); the condition did not, so no Allow applied. No explicit Deny was involved.

This is easy to miss because a PassServiceRoleToCloudFormationOnly statement with exactly this grant does exist — in a separate managed policy, <StackName>-PassRolePolicy, documented for an administrator to attach to the deploying human. It is not attached to the service role, so it does nothing for the role passing itself.

Impact before the fix: any customer using the optional CloudFormation service role could not deploy at all. The solution is nested-stack-based, so DOCUMENTKB is reached every time. Observed live on two independent stacks. Fixed in b249ebf71; adding cloudformation.${AWS::URLSuffix} to the list is what took both stacks to CREATE_COMPLETE, which is also what confirms the diagnosis.

Why nothing caught it: validate_service_role_permissions.py reasons about the policy’s shape — wildcards, boundary conditions, PassedToService presence — and a condition that is present but missing one required value passes every such check. Only deploying through the role finds this, and no tier did until this one.

Finding 2: the bedrock_llm OCR backend now emits markdown emphasis, and one extracted scalar changed with it (PRODUCT, narrow)

core-llm-simple-sep | small_narrow.pdf, deterministic on 3/3 runs each side:

	0.6.8 (3/3)	0.6.9 (3/3)
Account Number (truth 000123456789)	000123456789 ✓	000123456789 ✓
Statement Period (truth 01/01/2024 - 12/31/2024)	01/01/2024 - 12/31/2024 ✓	January 2024 - December 2024 ✗
OCR text for that line	Statement Period: 01/01/2024 - 12/31/2024	**Statement Period:** 01/01/2024 - 12/31/2024

The OCR still contains the literal date range on both sides. What changed is that 0.6.9’s bedrock_llm OCR wraps field labels in markdown bold; given that input the extraction returned a re-worded range instead of copying the value through.

Scope, measured: only the bedrock_llm backend. The same field on the same document reads 01/01/2024 - 12/31/2024 on both releases under textract_tables, textract_layout and bda. Inferred, not proven: that the emphasis markup caused the re-wording — it is the only input difference, but no controlled with/without-bold probe was run. Unverified: whether other scalar fields or documents are affected; this was the only cell/document pair in 171 runs whose scalar accuracy moved.

Also visible on both sides, so not a regression: bedrock_llm transcribes the account number as 00012345678 (11 digits) where the document has 12, and the extractor returned the correct 12-digit value anyway on both releases. That is the known LLM-OCR digit-fidelity hazard, confirmed still open — and the second time this cell has produced a scoring artefact that first looked like a code regression.

---

What was NOT validated

Stated explicitly so this record is not read as broader than it is.

• The published public artifact. See the warning at the top: every tier ran against a self-published build of the release commit, because idp-main_0.6.9.yaml was not in the public buckets. Tier C and tier D should be re-run against the public URLs after publish.
• The browser UX review (tier G) — NOT RUN, no browser MCP server available. Nothing here evidences the web UI’s usability or flow behaviour on this release.
• Real GovCloud. The --govcloud tier ran in a commercial account. See the warning above.
• The Knowledge Base path in --govcloud — disabled by default in that runner.
• The seller service’s positive path (a token actually issued and verified). The probe covers refusals only and says so.
• Cost as a release-over-release delta. 0.6.9 changed the cost instrument; see the A/B section. Single-release cost numbers in the guidance paper are the ones to quote.
• Accuracy against the published v0.6.8 benchmark numbers. stickler-eval 1.0.0 changed the comparators, so this A/B’s 0.6.8 column is not comparable with the v0.6.8 entry.
• Documents above 100 rows, and real (non-synthetic) corpora, in the A/B itself — the A/B is corefast by design. Both are covered by the guidance-paper refresh.
• The release’s new features beyond deployability. Every stack here deployed them; none was driven with feature-specific inputs. That is feature testing, which this record does not claim.

Reproduce

AWS_PROFILE=default aws sts get-caller-identity     # confirm the deployment account
aws iam get-account-summary --query 'SummaryMap.[Roles,RolesQuota]'   # budget ~160 roles per in-flight stack
FROM=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.8.yaml
TO=<the 0.6.9 template URL>          # public once published; a private --clean-build here

# offline tiers (SRT before the build; setup restores the committed dispositions)
CI=1 python scripts/srt/setup.py && make srt-scan
make test && make lint-cicd && make dep-audit && make typecheck
env -i HOME=$HOME PATH=/usr/local/bin:/usr/bin:/bin AWS_PROFILE=default bash -lc \
  'python3 publish.py <bucket-basename> idp us-west-2 --clean-build'

# upgrade tier (then the live security tests run against the upgraded stack)
aws cloudformation create-stack --stack-name IDPUpg068to069 --template-url $FROM \
  --parameters ParameterKey=AdminEmail,ParameterValue=<you> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND --disable-rollback
idp-cli run-inference --stack-name IDPUpg068to069 --dir samples/ --file-pattern lending_package.pdf --batch-prefix pre-upgrade --monitor
aws cloudformation update-stack --stack-name IDPUpg068to069 --template-url $TO \
  --parameters file://<every existing key with UsePreviousValue=true> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
idp-cli run-inference --stack-name IDPUpg068to069 --dir samples/ --file-pattern lending_package.pdf --batch-prefix post-upgrade --monitor
make security-results STACK_NAME=<the v0.6.9 stack> REGION=us-west-2

# deploy variants (at most two or three at a time; the VPC ones need the four VPC params)
make stacktest-hosting-global  TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you>
make stacktest-waf             TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you>
make stacktest-hosting-private TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…
make stacktest-jobsapi         TEMPLATE_URL=$TO REGION=us-west-2 ADMIN_EMAIL=<you> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…
make stacktest-seller          # REGION unset; delete /aws/apigateway/idp-seller-entitlement-citest-activation first

# transforms (publish from source, deploy, process a document, tear down)
make transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<you>

# benchmarks — see docs/benchmarking/releases/v0.6.9.md and config-guidance.md