Skip to content

Release Validation: v0.6.14

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

Release: v0.6.14 (VERSION = 0.6.14, newest CHANGELOG.md section) Validated: 2026-11-06 to 2026-11-10 (tiers A–E, G and H by 2026-11-07; the tier F refresh finished 2026-11-10), us-west-2 (seller tier us-east-1), commercial partition Commit measured: ada0a9bc5, the commit tag v0.6.14 points to. Every offline figure in this record was measured in a dedicated git worktree at that commit, so no other session’s work could move the tree under the gates. Artifact under test: the published object https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.14.yaml, Description AWS GenAI IDP Accelerator (uksb-r8evguc4p9) (SO9027) (v0.6.14). Every stack in tiers B, C and E was created from that object (or, for tier E, from the published v0.6.13 object and then updated to it). Tier D publishes from source at the tag to apply its transforms, and tier A’s build row is a local publish.py --clean-build. Account <ACCOUNT_ID>. PREV = v0.6.13.

✅ Ship. The published artifact deploys in all six shipped configurations, including the new TenantIsolationMode=Strict configuration, whose isolation harness passed 23 of 23 checks. Both template transforms deploy and process a real document. The upgrade from 0.6.13 completed without a version effect on extracted values, and the 342-run release A/B shows 0 regressions and 0 improvements, with recall and cell accuracy identical on both sides over the same 23,370 compared cells. All four security gates pass. The two RBAC counts that moved are exactly the four tenant-administration operations this release adds.

All 8 tiers have a result. Tiers A–E, G and H and F1 are recorded here. Tier F’s guidance refresh (F2–F4) is reported in the sibling benchmark pull request, #1823. Six of its thirty lanes were killed by a host restart on 2026-11-07 and re-run in full, with the fifteen lanes that had not started, from 2026-11-09 (see Tier F).

The finding that matters most for a customer is in the upgrade path, and it is environmental in trigger but not in effect. The first update-stack attempt failed because the container-image build in the multi-document-discovery nested stack did not finish within its custom resource’s time budget. The rollback then failed on the same resource, leaving the stack in UPDATE_ROLLBACK_FAILED until continue-update-rollback was run by hand. The second attempt, with CodeBuild dispatching normally, completed in 25 m 41 s. A customer whose CodeBuild builds queue during an upgrade would reach the same state. Finding 1, filed as #1824.

This release closes all five product defects the v0.6.13 record filed. Per-section Re-extract no longer rewrites every section’s class (#1672). It ran correctly in the browser review. The SDK’s minimal configuration now validates (#1673), and the SDK integration suite passes 13 of 13 where it failed 1 of 13 at v0.6.13. The packet-window (#1676), blank-page (#1677) and confidence-step memory (#1678) fixes are measured by the benchmark refresh. This record files two Test Studio defects: one introduced in this release (Finding 2, #1816), and one whose first appearance was not checked against v0.6.13 (Finding 3, #1817).


TierGateResultvs v0.6.13
A — SRT✅ PASS0 open/reopened HIGH counted against the gate; 57 Bandit B105/B106 identifier names in test code, scoped out of the gate by the documented rule; 53,423 CI-visible findings51 → 57 scoped out; 50,868 → 53,423 tracked (Finding 10)
A — make test✅ PASSall 75 test roots, 0 failures, in the worktree at the tag74 roots → 75
A — make lint-cicd✅ PASSexit 0; markdown link gate clean over 402 files399 → 402 files
A — make dep-audit✅ PASS1,757 packages, 0 gating, 4 allowlisted, 5 below HIGH, 0 stale allowlist entries, 4 unmapped advisories listed for review (Finding 9)1,752 → 1,757 packages; 6 → 4 allowlisted; 15 → 5 below HIGH
A — make typecheck✅ PASS0 errors / 47 warningsunchanged
A — UI tests (vitest)✅ PASS251 files, 2,805 tests231 / 2,626 → 251 / 2,805
A — build + package✅ PASSpublish.py --clean-build in a clean env, 5.7 min; template validation passed and cfn-lint clean on all 5 packaged templates (8 advisory warnings)7.0 → 5.7 min; warning count unchanged
A — coverage✅ PASS11 trees measured, ratchet green (550 files at or above recorded floor)528 → 550 files; two trees fell by more than a point (Finding 6)
B — security snapshot✅ 4 of 4 gatesSRT PASS · RBAC static PASS (0 fail / 4 known-gap warn, 168 operations) · RBAC dynamic PASS (1,204 checks, 0 hard fail, 0 not run, 17 skipped) · ZAP PASS (High 0, Medium 1)164 → 168 static operations; 1,176 → 1,204 dynamic checks (Finding 10)
C — deploy variants✅ 6/6hosting-global (40 min), WAF (41 min), hosting-private (68 min, second attempt), Jobs API (63 min), tenant isolation (49 min, 23/23), seller (3 min); each self-deployed from the published template and torn down5/5 → 6/6, tenant isolation new
D — transforms✅ 2/2--headless and --govcloud each deployed, processed lending_package.pdf (classified Payslip, 17/22 fields) and tore down18/22 → 17/22 fields
E — upgrade 0.6.13 → 0.6.14⚠️ PASS on the second attemptattempt 1 failed and its rollback failed (Finding 1); attempt 2 UPDATE_COMPLETE in 25 m 41 s, Description now (v0.6.14); three added parameters, all optional with defaults; nothing removed or newly required. No consistent change in extracted values (Finding 4)22 m 46 s → 25 m 41 s; first upgrade in these records to need a second attempt
F1 — release A/B✅ PASS171/171 and 171/171; 0 regressions, 0 improvements, 2 inconclusive rows, both on the bedrock_llm armthe v0.6.13 split asymmetry did not appear on v0.6.14 (Finding 5)
F2–F4 — guidance refresh✅ completecarried by #1823: 32 scored sets, 3,338 rows, $1,970.57; 5 rows did not complete, all the expected simple-mode failure on the 3,200-row document. 6 of 30 lanes were killed by a host restart and re-run in full0.45% → 0.15% of rows not completed; astracap and scaling ran at their default repeats rather than v0.6.13’s 3
G — browser UX⚠️ RAN, 1 flow FAIL10 flows or areas PASS, 1 PASS partial (the Missing Required column, hidden by default); creating a test set from S3 files with no baselines FAIL (Finding 3); 4 flows not run (fixtures or role missing)flow 6.2’s #1672 regression is fixed; a separate defect in the same flow is filed as #1816 (Finding 2)
H — integration tests⚠️ 2 of 3 suites pass, 1 emptyidp_common 19/19; idp_sdk 13/13; idp_cli selects 0 of 1,547 tests (Finding 8)idp_sdk 12/13 → 13/13

make coverage-all over all eleven registry trees, then make check-coverage-debt, in the dedicated worktree. The ratchet is green: 550 files across 11 trees are at or above their recorded coverage.

treemeasuredv0.6.13 recorddelta
api_resolvers72.28% (lowest)71.38%+0.90
seller_entitlement82.03%82.03%—
idp_common86.26%85.44%+0.82
pii_anonymizer_hook86.98%86.98%—
src_lambda91.18%90.95%+0.23
scripts92.00%91.56%+0.44
main_stack_extensions93.90%93.90%—
pii_anonymizer_api95.87%94.93%+0.94
idp_sdk98.01%99.47%−1.46
idp_cli98.07%99.68%−1.61
idp_feature_sdk99.94%99.94%—

api_resolvers is still the lowest tree, now at 72.28%. Two trees lost more than a point. Both losses come from this release’s new tenancy code, which is covered less than the code around it (Finding 6).

Tier B — security gate results compared with the previous release

Section titled “Tier B — security gate results compared with the previous release”

All four gates pass, at the tag’s SHA, against the upgraded reference stack. The snapshot is in security/test-results/0.6.14/. ZAP is unchanged from v0.6.13: High 0, and one Medium Cross-Domain Misconfiguration alert on five URLs. The counts that moved are explained in Finding 10.

Every variant self-deployed its own throwaway stack from the published template, ran its assertions, and tore down:

  • hosting-global: a regional REST API serves the web UI with HTTP 200.
  • WAF: the WebACL is associated with the API.
  • hosting-private: a private REST API with its resource policy, in the purpose-built stack-test VPC (<VPC_ID>, two private subnets, an execute-api interface endpoint). The first attempt failed for the same reason as tier E’s first attempt (Finding 1). The container-image build did not complete, and the stack reached ROLLBACK_FAILED with the reporting bucket undeleted. The runner’s cleanup removed it. The second attempt passed.
  • Jobs API: deployed into the same VPC.
  • tenant isolation: a TenantIsolationMode=Strict stack, with the verify-tenant-isolation harness run against it: 23 of 23 checks PASS. This is the first record with this tier, so there is no earlier count to compare. The harness’s two uploader roles and access-point stack were removed, and no idp-tiv-uploader-* role was left behind.
  • seller: the unsigned-request and unentitled-account refusals (both 403, with no internal detail leaked), and 15 hostile payloads refused with no 5xx, in us-east-1.

As in every earlier record, the seller tier asserts refusals only. No token was issued and verified. Its teardown leaves the signing KMS key scheduled for deletion on a 7-day window. After the VPC tiers, three orphaned Lambda network interfaces were swept from the stack-test VPC: two from the hosting-private stacks and one from the Jobs API stack.

Tier D — template transforms, and the limits of a commercial-region run

Section titled “Tier D — template transforms, and the limits of a commercial-region run”

Both transforms deployed and processed samples/lending_package.pdf end to end. The document was classified Payslip with 17 of 22 fields populated; v0.6.13 populated 18. One run per transform per release cannot separate a version effect from run-to-run variation on this document, which Finding 4 measures directly across seven runs and attributes to variation.

  • --headless: the transform removed 99 of 356 resources (UI, Cognito and CloudFront), removed the six policy statements that referenced them, and disabled the Knowledge Base.
  • --govcloud: the transform removed the AWS::Lambda::Url resource and the LWA chat-stream function (#677), collapsed 21 CloudFront-hosting conditions to the API Gateway branch, and forced WebUIHosting=APIGateway. Its own check confirmed the result contains no CloudFront or Lambda URL resource and no dangling reference.

⚠️ The --govcloud result is a commercial-partition result. It proves the CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves nothing about partition ARNs, GovCloud model availability or the BDA project rejection (#676/#677 were invisible to a commercial run). The Knowledge Base was disabled in the --govcloud run, which is the runner’s default.

Tier E — in-place upgrade, and what it did to extracted values

Section titled “Tier E — in-place upgrade, and what it did to extracted values”

A fresh stack, IDPUpg0613to0614, was created from the published v0.6.13 template and processed lending_package.pdf twice. It was then updated to the published v0.6.14 template, with UsePreviousValue=true on all 93 existing parameters.

  • Attempt 1 (started 23:08 UTC) failed, and its rollback failed too (Finding 1). It was recovered with continue-update-rollback, and the stack returned to v0.6.13 at 00:49 UTC.
  • Attempt 2 (started 00:49:42 UTC) reached UPDATE_COMPLETE at 01:15:23 UTC with no failed event. The UpdateConfiguration custom resource ran twice and logged no warning or error.

The parameter set gained three entries, each with a default that keeps the existing behaviour: TenantIsolationMode (Off), TenantGrantSource (Manual) and RefreshTokenValidityHours (720). Nothing was removed or newly required. The upgraded stack then processed the same document five times.

All seven runs found the same six sections with the same classes. Per-section field counts and empty-field counts:

sectionclassfields (pre ×2 / post ×5)empty fields, preempty fields, post
1Payslip48, 48 / 48, 48, 48, 48, 429, 99 in all 5
2US-drivers-licenses22 / 224, 44, 4, 5, 5, 4
3Bank-checks10 / 101, 11 in all 5
4Bank-Statement16 / 166, 56, 4, 5, 6, 6
5W235 / 352, 22 in all 5
6Homeowners-Insurance-Application41 / 418, 88 in all 5

Finding 4 covers the value differences.

The release A/B entry and the regenerated guidance are carried by the sibling benchmark pull request, #1823, which adds docs/benchmarking/releases/v0.6.14.md and brings docs/benchmarking/config-guidance.md to this release. This section holds the part that belongs to the validation record.

F1. Same published templates, byte-identical configurations, only the code version differs: corefast, repeats: 3, --native-upload, both sides on the same stack, the PREV side run before update-stack (21:47–22:49 UTC) and the NEW side after it (01:26–02:27 UTC).

v0.6.13v0.6.14delta
runs / successful171 / 171171 / 171—
grid cost$25.6317$24.7289−3.5%
grid wall8,936.9 s8,818.8 s−1.3%
mean completeness recall1.00001.0000—
mean cell accuracy1.00001.0000—
cells_compared23,37023,370—
processing issues1813see below

aggregate.py --compare reports 0 regressions, 0 improvements and 0 cell-level changes either way. Both of its inconclusive rows are on the bedrock_llm OCR arm, core-llm-simple-sep. Its cost fell 45% (0.170 ± 0.169 → 0.093 ± 0.063, n = 9) and its scalar accuracy rose from 0.889 to 1.000, with paired per-document spread ±0.333. Both moves are inside the cell’s own spread. Finding 5 explains both.

The processing-issue count fell from 18 to 13. On v0.6.13 the 18 were 14 extraction_row_count_unverified, 2 extraction_sparse and 2 extraction_validation_failed. On v0.6.14 the 13 are 12 extraction_row_count_unverified and one assessment_truncated. That one is a confidence call on longdesc_100 that truncated, was split and retried, and left 0 rows unscored. The wall-time figure is not a performance claim: both sides ran with nothing else on the stack, but wall time on a shared account is load-sensitive.

F2–F4. Thirty lanes ran six at a time on the upgraded reference stack from 02:33 UTC on 2026-11-07. A scheduled host patch restart at 06:03 UTC killed the six lanes then in flight. Nine lanes had already exited at their planned run counts. The harness has no resume mode, so a killed lane cannot be completed in place. Its partial run directories were moved out of the results tree unscored, and the six lanes were re-run in full from 2026-11-09 20:56 UTC, together with the fifteen lanes that had not started. The classification-variant benchmark runs after them, alone. The refresh produced 32 scored sets: 3,338 rows for $1,970.57. Five rows did not complete, all the simple-mode cells on the 3,200-row document, which exceed a single response by design. The model-selection verdict and every per-suite comparison are in #1823.


1. An upgrade whose container build does not finish in time fails, and so does its rollback (product robustness, environment-triggered)

Section titled “1. An upgrade whose container build does not finish in time fails, and so does its rollback (product robustness, environment-triggered)”

Tier E’s first update-stack started at 23:08 UTC. The DockerBuildRun custom resource in the multi-document-discovery nested stack failed twice with Build … did not complete within the Lambda budget. Its counterpart in the unified pattern stack was cancelled, then failed during rollback with build did not complete - status: [STOPPED]. The parent reached UPDATE_ROLLBACK_FAILED at 23:42 UTC. continue-update-rollback at 00:35 UTC returned it to UPDATE_ROLLBACK_COMPLETE at 00:49 UTC, and the second attempt completed normally. The hosting-private stack-test, created at the same time, failed on the same resource and reached ROLLBACK_FAILED.

The trigger was environmental: the builds those resources start did not complete in time, during a window in which the run’s notes record CodeBuild builds in the account as not dispatching. The logs here do not establish why, or whether the queueing was account-wide. Once builds were dispatching again, both stacks deployed. The effect on a customer is not environmental: an upgrade during a CodeBuild slowdown does not simply fail and roll back. It leaves the stack needing a manual continue-update-rollback. The repository’s test-upgrade skill gives that command for a rollback deadlock, but no customer-facing page described this case at this release. Filed as #1824 and fixed in #1825, which also adds the recovery to Troubleshooting. The evidence is two occurrences in this pass, both on DockerBuildRun (tier E’s first attempt and the hosting-private stack-test’s first attempt), and the cause of the stall is not established.

2. Test Studio’s Re-extract offer after a class change is lost on reload, and a second change names the wrong class (product defect, this release)

Section titled “2. Test Studio’s Re-extract offer after a class change is lost on reload, and a second change names the wrong class (product defect, this release)”

Tier G, flow 6.2. After a section’s class is changed with Edit Page Grouping, the editor offers per-section Re-extract with the note “These fields were extracted as class”. That offer is held only in React state (reclassified in test-set-editor-slots.tsx). It is lost on a page reload, although the section’s fields still belong to the old class. It also records the class from before the latest change rather than the class the fields came from. A second change therefore names the wrong one: the note said “extracted as W2” for fields extracted as Bank-checks. The per-section Re-extract that #1672 needed now works correctly when it is offered. Filed as #1816.

3. A test set created from S3 files with no baselines fails after being reported created (product defect)

Section titled “3. A test set created from S3 files with no baselines fails after being reported created (product defect)”

Tier G. The From files in S3 wizard, pointed at documents with no baseline files, says on one step that unlabeled documents will arrive and in a hint that they are skipped. It then reports the set as created, and the set ends in FAILED. Adding documents is then refused because the set is “not in COMPLETED status”, so the only remedy is to delete and recreate it. Whether v0.6.13 behaves the same way was not checked. Filed as #1817.

4. The upgrade changed no extracted value consistently; one post-upgrade run silently dropped two of three tax rows (measurement)

Section titled “4. The upgrade changed no extracted value consistently; one post-upgrade run silently dropped two of three tax rows (measurement)”

Across the two v0.6.13 runs and five v0.6.14 runs of lending_package.pdf, every value difference is inconsistent within one version or the other:

fieldv0.6.13 (2 runs)v0.6.14 (5 runs)
Payslip name and address fields, e.g. EmployeeName.FirstName"JOHN" in 1, "John" in 1"JOHN" in 5
Payslip FederalTaxes3 rows in both3 rows in 4, 1 row in 1
Bank-checks check_number"0000000000" in 1, "0001379" in 1"0000000000" in 4, "0001379" in 1
Bank-Statement account_holder_address.Statenull in bothnull in 3, "USA" in 2
driver’s licence ENDORSEMENTS / RESTRICTIONS[] in both[] in 3, null or ["NONE"] in 2

So there is no consistent version effect on this document. Two observations are worth naming:

  • One run returned a shortened list as a success. The fifth v0.6.14 run returned one of the Payslip’s three FederalTaxes rows (Social Security and Medicare are missing). Its section reported Status: SUCCESS with no processing issue, from a single traditional extraction pass on Claude Sonnet 5, the preset’s extraction model. One run in seven is too few to attribute to the release, but the list loss itself was silent. Not filed. It fits the standing silent list-loss class that the benchmark paper’s hazards section tracks, rather than a new defect.
  • The tax-amount signs are now stable. All seven runs report the tax amounts as negative, where the v0.6.13 record saw one positive in five.

There is no ground-truth baseline for this sample in the repository, so which value is correct was not scored.

5. The A/B asymmetry the v0.6.13 record left open recurred on v0.6.13 and not on v0.6.14 (improvement, inferred)

Section titled “5. The A/B asymmetry the v0.6.13 record left open recurred on v0.6.13 and not on v0.6.14 (improvement, inferred)”

The v0.6.13 record found that two of three v0.6.13 runs of the bedrock_llm arm split longdesc_100, a one-section document, into two sections, against none of three on v0.6.12. It left open whether this was a release effect or OCR run-to-run variation. In this pass’s PREV side, v0.6.13 split it in two of three runs again, and v0.6.14 split it in none of three. Both inconclusive rows in F1 follow from that: the arm’s cost fell 45% and its scalar accuracy rose to 1.000. Two independent v0.6.13 grids showing the same 2-of-3 split, against 0-of-3 on the releases either side, points to a v0.6.13 behaviour that v0.6.14 removed. That reading is inferred from n = 3 per side, and no single change in this release has been identified as the cause.

6. The tenancy modules this release adds pulled two coverage trees down by more than a point (coverage)

Section titled “6. The tenancy modules this release adds pulled two coverage trees down by more than a point (coverage)”

idp_sdk fell from 99.47% to 98.01% and idp_cli from 99.68% to 98.07%. The ratchet passes, because it is per file, and each new file is recorded at the coverage it had when it was added. The lowest-covered files in those trees are all new in this release:

  • idp_sdk/_core/ui_api.py, 60.0%
  • idp_cli/tenancy.py, 75.4%
  • idp_sdk/_core/tenancy_verify.py, 78.8%
  • idp_sdk/operations/tenancy.py, 83.2%

The tier C tenant-isolation run exercises this code against a live stack, but that run is not counted in coverage.

7. The browser review’s usability findings (tier G)

Section titled “7. The browser review’s usability findings (tier G)”

Beyond Findings 2 and 3, the review ranked these, most important first. Each was seen on a screenshot of this build. None is filed as an issue in this pass.

  1. Documents whose labels are an unreviewed machine draft still label their values “Ground truth:”.
  2. A document reports Review Reason (6) while about 15 fields are shown red, including empty fields at 0%. Since this release stops empty fields raising confidence alerts, their red rendering no longer matches the count.
  3. Table-row review chips are ambiguous: several read only “Period”.
  4. The Edit/View toggle stays on Edit for a read-only document.
  5. The new Missing Required column is hidden by default.
  6. The configuration version is visible only in a badge’s hover text.
  7. The Generate draft labels copy mentions a published version when there is none.
  8. Next alert after a chip jump restarts at the first alert.
  9. Re-extracting one section resets the whole document to Draft (machine).
  10. A Reviewer’s Tasks filter still offers Test Studio, which this release removes from that role.
  11. Smaller polish items:
    • “Upload (1 files)”
    • Inbox column truncation
    • unread Inbox messages are not marked with the text Unread, which the changelog says they are
    • the Prefix Mappings breadcrumb reads “Documents”
    • degraded sections are not explained

Four flows were not run: 6.1, 11.1, 4.1 and 12.1, which need a misclassified fixture or an Annotator user. One Inbox message the review generated (“default: documents waiting”) could not be retracted. The full ranked list, per-flow evidence and the screenshots are kept out of the repository, as the review’s privacy rule requires.

8. Tier H’s idp_cli suite is still empty (test tooling, unchanged)

Section titled “8. Tier H’s idp_cli suite is still empty (test tooling, unchanged)”

lib/idp_cli_pkg selects 0 of 1,547 tests under -m integration (1,506 at v0.6.13), so that command runs nothing. The SDK’s integration suite, which exercises deployment and batch behaviour against the reference stack, passed 13 of 13 (v0.6.13: 12 of 13). Its one failure there was #1673, now fixed.

9. make dep-audit lists four unmapped advisories, three of them HIGH, that it cannot gate (informational)

Section titled “9. make dep-audit lists four unmapped advisories, three of them HIGH, that it cannot gate (informational)”

The audit’s unmapped-advisory pass lists four GitHub advisories that name a pinned package whose version is in the stated range, but carry no machine-readable affected range:

  • picomatch 2.3.2 and 4.0.7: ReDoS, HIGH
  • uri-js 4.4.1: denial of service, HIGH
  • uri-js 4.4.1: improper decoding, MEDIUM

They are reported for review, never gated, by design. In src/ui/package-lock.json both packages are development dependencies (build tooling). The other lockfiles that name them were not checked.

10. SRT’s scoped-out count and both RBAC counts grew (informational)

Section titled “10. SRT’s scoped-out count and both RBAC counts grew (informational)”
  • SRT scopes 57 HIGH findings out of the gate, up from 51. All 57 are Bandit B105/B106 identifier names in test code from which no deployment artifact is built. None counts against the gate.
  • RBAC static operations rose from 164 to 168, and dynamic operations from 161 to 165. The four new operations are the tenant-administration API this release adds: listTenants, putTenant, deleteTenant and setUserTenant. The dynamic check count rose by exactly their share, from 1,176 to 1,204. That is 16 authorization-matrix checks, 4 unauthenticated-access checks and 8 other per-operation checks. There were still 0 hard failures, 0 not run and 17 skipped.

11. make version leaves the lockfile behind, so every version bump breaks the locked install (release tooling)

Section titled “11. make version leaves the lockfile behind, so every version bump breaks the locked install (release tooling)”

The fresh worktree at v0.6.14 would not install until lib/idp_common_pkg/uv.lock’s own project version was patched to 0.6.14. make setup runs uv export --locked, which refuses a lockfile out of date with its pyproject.toml. PR #1808 fixed the instance on develop the same day. The version target in the Makefile still does not touch the lockfile, so the next bump will reproduce it. Filed as #1818.

12. Environment events during the pass (environment)

Section titled “12. Environment events during the pass (environment)”
  • Two CodeBuild-stall failures (Finding 1): tier E’s first attempt and hosting-private’s first attempt. Both were re-run and passed.
  • Orphaned network interfaces: three in the stack-test VPC, deleted.
  • A host restart killed six benchmark lanes at 06:03 UTC on 2026-11-07. It was a scheduled patch reboot, and it interrupted the run as the procedure warns. The lanes were re-run in full; partial results were discarded, not scored.

IssueFindingStatus
#1816 — Re-extract offer lost on reload, wrong class after a second change2open
#1817 — S3 test set with no baselines fails after being reported created3open
#1818 — make version does not update uv.lock11open

The five product issues the v0.6.13 record filed (#1672, #1673, #1676, #1677, #1678) are all closed, with fixes in this release.


  • Real GovCloud. Tier D’s --govcloud run was REGION=us-west-2 with commercial credentials. Partition ARNs, GovCloud model availability and the BDA project rejection are unproven, and the Knowledge Base was disabled in that run.
  • The seller positive path. No token was issued or verified.
  • TenantGrantSource=IdpClaim and Api. The tenant-isolation tier deploys TenantIsolationMode=Strict with the default Manual grant source. Granting a tenant from a federated claim or through setUserTenant was exercised only by the RBAC authorization checks on that operation.
  • idp_cli integration behaviour. The suite is empty (Finding 8).
  • The cause of the CodeBuild stall in Finding 1, and whether an upgrade fails the same way under a slow but progressing build queue.
  • Which value of Finding 4’s varying fields is correct. There is no ground-truth baseline for samples/lending_package.pdf in the repository.
  • Whether Finding 3 predates this release.
  • Four browser flows (6.1, 11.1, 4.1, 12.1), for want of a misclassified fixture and an Annotator user. Production review queues end to end: HITL is off on the reference stack, as on any stack upgraded with its previous values.
  • Canada (ca-central-1) and eu-central-1. All stacks were us-west-2 (seller us-east-1).
  • Malware scanning. MalwareScanning defaulted to DISABLED in every stack.

Terminal window
# Tier A (offline), in a dedicated worktree at the tag — never in a shared checkout
git worktree add <path> v0.6.14 && cd <path>
CI=1 python3 scripts/srt/setup.py && make srt-scan # before anything writes build output
make test && make lint-cicd && make typecheck && make dep-audit && (cd src/ui && npx vitest run)
make coverage-all && make check-coverage-debt
python3 publish.py <bucket-base> <prefix> us-west-2 --clean-build
URL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.14.yaml
# Tier B, against the upgraded stack
make security-results STACK_NAME=<stack> REGION=us-west-2 SKIP_SRT=1
# Tier C (6 variants)
make stacktest-hosting-global TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>
make stacktest-waf TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>
make stacktest-hosting-private TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \
VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>
make stacktest-jobsapi TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \
VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>
make stacktest-tenancy TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>
make stacktest-seller # REGION deliberately unset -> us-east-1
# Tier D
make transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<email>
# Tier E — several documents per side, or run-to-run variation reads as a version effect
aws cloudformation create-stack --stack-name IDPUpg0613to0614 --template-url <0.6.13 URL> ...
aws cloudformation update-stack --stack-name IDPUpg0613to0614 --template-url $URL \
--parameters file://<every existing parameter with UsePreviousValue=true>.json
# if the update and its rollback both fail on DockerBuildRun:
aws cloudformation continue-update-rollback --stack-name IDPUpg0613to0614
# Tier F1, PREV side before update-stack, NEW side after, on the same stack
python benchmarks/harness/run_matrix.py --stack <stack> --suite corefast --repeats 3 \
--native-upload --validate-against-release v0.6.13 --max-inflight 4
python benchmarks/harness/aggregate.py --compare <v0.6.14 summary> --baseline <v0.6.13 summary>
# Tier G: headless Chromium via Playwright, throwaway users from scripts/ux_test_session.py
# Tier H, on the upgraded stack
cd lib/idp_common_pkg && make test-integration
IDP_STACK_NAME=<stack> python -m pytest -m integration lib/idp_sdk/tests/integration

Raw logs for every tier are in gitignored scratch/ and are not published.