Release Validation: v0.6.14
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
Release Validation Record — v0.6.14
Section titled “Release Validation Record — v0.6.14”Release: v0.6.14 (VERSION = 0.6.14, newest CHANGELOG.md section)
Validated: 2026-11-06 to 2026-11-10 (tiers A–E, G and H by 2026-11-07; the tier F refresh finished 2026-11-10), us-west-2 (seller tier us-east-1), commercial partition
Commit measured: ada0a9bc5, the commit tag v0.6.14 points to. Every offline figure in
this record was measured in a dedicated git worktree at that commit, so no other session’s
work could move the tree under the gates.
Artifact under test: the published object
https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.14.yaml,
Description AWS GenAI IDP Accelerator (uksb-r8evguc4p9) (SO9027) (v0.6.14). Every stack in
tiers B, C and E was created from that object (or, for tier E, from the published v0.6.13
object and then updated to it). Tier D publishes from source at the tag to apply its
transforms, and tier A’s build row is a local publish.py --clean-build.
Account <ACCOUNT_ID>. PREV = v0.6.13.
Verdict
Section titled “Verdict”✅ Ship. The published artifact deploys in all six shipped configurations, including the
new TenantIsolationMode=Strict configuration, whose isolation harness passed 23 of 23
checks. Both template transforms deploy and process a real document. The upgrade from 0.6.13
completed without a version effect on extracted values, and the 342-run release A/B shows
0 regressions and 0 improvements, with recall and cell accuracy identical on both sides
over the same 23,370 compared cells. All four security gates pass. The two RBAC counts that
moved are exactly the four tenant-administration operations this release adds.
All 8 tiers have a result. Tiers A–E, G and H and F1 are recorded here. Tier F’s guidance refresh (F2–F4) is reported in the sibling benchmark pull request, #1823. Six of its thirty lanes were killed by a host restart on 2026-11-07 and re-run in full, with the fifteen lanes that had not started, from 2026-11-09 (see Tier F).
The finding that matters most for a customer is in the upgrade path, and it is
environmental in trigger but not in effect. The first update-stack attempt failed because
the container-image build in the multi-document-discovery nested stack did not finish within
its custom resource’s time budget. The rollback then failed on the same resource, leaving the
stack in UPDATE_ROLLBACK_FAILED until continue-update-rollback was run by hand. The
second attempt, with CodeBuild dispatching normally, completed in 25 m 41 s. A customer whose
CodeBuild builds queue during an upgrade would reach the same state. Finding 1, filed as
#1824.
This release closes all five product defects the v0.6.13 record filed. Per-section Re-extract no longer rewrites every section’s class (#1672). It ran correctly in the browser review. The SDK’s minimal configuration now validates (#1673), and the SDK integration suite passes 13 of 13 where it failed 1 of 13 at v0.6.13. The packet-window (#1676), blank-page (#1677) and confidence-step memory (#1678) fixes are measured by the benchmark refresh. This record files two Test Studio defects: one introduced in this release (Finding 2, #1816), and one whose first appearance was not checked against v0.6.13 (Finding 3, #1817).
Results per tier group
Section titled “Results per tier group”| Tier | Gate | Result | vs v0.6.13 |
|---|---|---|---|
| A — SRT | ✅ PASS | 0 open/reopened HIGH counted against the gate; 57 Bandit B105/B106 identifier names in test code, scoped out of the gate by the documented rule; 53,423 CI-visible findings | 51 → 57 scoped out; 50,868 → 53,423 tracked (Finding 10) |
A — make test | ✅ PASS | all 75 test roots, 0 failures, in the worktree at the tag | 74 roots → 75 |
A — make lint-cicd | ✅ PASS | exit 0; markdown link gate clean over 402 files | 399 → 402 files |
A — make dep-audit | ✅ PASS | 1,757 packages, 0 gating, 4 allowlisted, 5 below HIGH, 0 stale allowlist entries, 4 unmapped advisories listed for review (Finding 9) | 1,752 → 1,757 packages; 6 → 4 allowlisted; 15 → 5 below HIGH |
A — make typecheck | ✅ PASS | 0 errors / 47 warnings | unchanged |
A — UI tests (vitest) | ✅ PASS | 251 files, 2,805 tests | 231 / 2,626 → 251 / 2,805 |
| A — build + package | ✅ PASS | publish.py --clean-build in a clean env, 5.7 min; template validation passed and cfn-lint clean on all 5 packaged templates (8 advisory warnings) | 7.0 → 5.7 min; warning count unchanged |
| A — coverage | ✅ PASS | 11 trees measured, ratchet green (550 files at or above recorded floor) | 528 → 550 files; two trees fell by more than a point (Finding 6) |
| B — security snapshot | ✅ 4 of 4 gates | SRT PASS · RBAC static PASS (0 fail / 4 known-gap warn, 168 operations) · RBAC dynamic PASS (1,204 checks, 0 hard fail, 0 not run, 17 skipped) · ZAP PASS (High 0, Medium 1) | 164 → 168 static operations; 1,176 → 1,204 dynamic checks (Finding 10) |
| C — deploy variants | ✅ 6/6 | hosting-global (40 min), WAF (41 min), hosting-private (68 min, second attempt), Jobs API (63 min), tenant isolation (49 min, 23/23), seller (3 min); each self-deployed from the published template and torn down | 5/5 → 6/6, tenant isolation new |
| D — transforms | ✅ 2/2 | --headless and --govcloud each deployed, processed lending_package.pdf (classified Payslip, 17/22 fields) and tore down | 18/22 → 17/22 fields |
| E — upgrade 0.6.13 → 0.6.14 | ⚠️ PASS on the second attempt | attempt 1 failed and its rollback failed (Finding 1); attempt 2 UPDATE_COMPLETE in 25 m 41 s, Description now (v0.6.14); three added parameters, all optional with defaults; nothing removed or newly required. No consistent change in extracted values (Finding 4) | 22 m 46 s → 25 m 41 s; first upgrade in these records to need a second attempt |
| F1 — release A/B | ✅ PASS | 171/171 and 171/171; 0 regressions, 0 improvements, 2 inconclusive rows, both on the bedrock_llm arm | the v0.6.13 split asymmetry did not appear on v0.6.14 (Finding 5) |
| F2–F4 — guidance refresh | ✅ complete | carried by #1823: 32 scored sets, 3,338 rows, $1,970.57; 5 rows did not complete, all the expected simple-mode failure on the 3,200-row document. 6 of 30 lanes were killed by a host restart and re-run in full | 0.45% → 0.15% of rows not completed; astracap and scaling ran at their default repeats rather than v0.6.13’s 3 |
| G — browser UX | ⚠️ RAN, 1 flow FAIL | 10 flows or areas PASS, 1 PASS partial (the Missing Required column, hidden by default); creating a test set from S3 files with no baselines FAIL (Finding 3); 4 flows not run (fixtures or role missing) | flow 6.2’s #1672 regression is fixed; a separate defect in the same flow is filed as #1816 (Finding 2) |
| H — integration tests | ⚠️ 2 of 3 suites pass, 1 empty | idp_common 19/19; idp_sdk 13/13; idp_cli selects 0 of 1,547 tests (Finding 8) | idp_sdk 12/13 → 13/13 |
Tier A — test coverage, as measured
Section titled “Tier A — test coverage, as measured”make coverage-all over all eleven registry trees, then make check-coverage-debt, in the
dedicated worktree. The ratchet is green: 550 files across 11 trees are at or above their
recorded coverage.
| tree | measured | v0.6.13 record | delta |
|---|---|---|---|
| api_resolvers | 72.28% (lowest) | 71.38% | +0.90 |
| seller_entitlement | 82.03% | 82.03% | — |
| idp_common | 86.26% | 85.44% | +0.82 |
| pii_anonymizer_hook | 86.98% | 86.98% | — |
| src_lambda | 91.18% | 90.95% | +0.23 |
| scripts | 92.00% | 91.56% | +0.44 |
| main_stack_extensions | 93.90% | 93.90% | — |
| pii_anonymizer_api | 95.87% | 94.93% | +0.94 |
| idp_sdk | 98.01% | 99.47% | −1.46 |
| idp_cli | 98.07% | 99.68% | −1.61 |
| idp_feature_sdk | 99.94% | 99.94% | — |
api_resolvers is still the lowest tree, now at 72.28%. Two trees lost more than a point.
Both losses come from this release’s new tenancy code, which is covered less than the code
around it (Finding 6).
Tier B — security gate results compared with the previous release
Section titled “Tier B — security gate results compared with the previous release”All four gates pass, at the tag’s SHA, against the upgraded reference stack. The snapshot is in
security/test-results/0.6.14/.
ZAP is unchanged from v0.6.13: High 0, and one Medium Cross-Domain Misconfiguration alert on
five URLs. The counts that moved are explained in Finding 10.
Tier C — the six deploy variants
Section titled “Tier C — the six deploy variants”Every variant self-deployed its own throwaway stack from the published template, ran its assertions, and tore down:
- hosting-global: a regional REST API serves the web UI with HTTP 200.
- WAF: the WebACL is associated with the API.
- hosting-private: a private REST API with its resource policy, in the purpose-built
stack-test VPC (
<VPC_ID>, two private subnets, anexecute-apiinterface endpoint). The first attempt failed for the same reason as tier E’s first attempt (Finding 1). The container-image build did not complete, and the stack reachedROLLBACK_FAILEDwith the reporting bucket undeleted. The runner’s cleanup removed it. The second attempt passed. - Jobs API: deployed into the same VPC.
- tenant isolation: a
TenantIsolationMode=Strictstack, with theverify-tenant-isolationharness run against it: 23 of 23 checks PASS. This is the first record with this tier, so there is no earlier count to compare. The harness’s two uploader roles and access-point stack were removed, and noidp-tiv-uploader-*role was left behind. - seller: the unsigned-request and unentitled-account refusals (both 403, with no internal
detail leaked), and 15 hostile payloads refused with no 5xx, in
us-east-1.
As in every earlier record, the seller tier asserts refusals only. No token was issued and verified. Its teardown leaves the signing KMS key scheduled for deletion on a 7-day window. After the VPC tiers, three orphaned Lambda network interfaces were swept from the stack-test VPC: two from the hosting-private stacks and one from the Jobs API stack.
Tier D — template transforms, and the limits of a commercial-region run
Section titled “Tier D — template transforms, and the limits of a commercial-region run”Both transforms deployed and processed samples/lending_package.pdf end to end. The document
was classified Payslip with 17 of 22 fields populated; v0.6.13 populated 18. One run per
transform per release cannot separate a version effect from run-to-run variation on this
document, which Finding 4 measures directly across seven runs and attributes to variation.
--headless: the transform removed 99 of 356 resources (UI, Cognito and CloudFront), removed the six policy statements that referenced them, and disabled the Knowledge Base.--govcloud: the transform removed theAWS::Lambda::Urlresource and the LWA chat-stream function (#677), collapsed 21 CloudFront-hosting conditions to the API Gateway branch, and forcedWebUIHosting=APIGateway. Its own check confirmed the result contains no CloudFront or Lambda URL resource and no dangling reference.
⚠️ The --govcloud result is a commercial-partition result. It proves the
CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves
nothing about partition ARNs, GovCloud model availability or the BDA project rejection
(#676/#677 were invisible to a commercial run). The Knowledge Base was disabled in the
--govcloud run, which is the runner’s default.
Tier E — in-place upgrade, and what it did to extracted values
Section titled “Tier E — in-place upgrade, and what it did to extracted values”A fresh stack, IDPUpg0613to0614, was created from the published v0.6.13 template and
processed lending_package.pdf twice. It was then updated to the published v0.6.14 template,
with UsePreviousValue=true on all 93 existing parameters.
- Attempt 1 (started 23:08 UTC) failed, and its rollback failed too (Finding 1). It was
recovered with
continue-update-rollback, and the stack returned to v0.6.13 at 00:49 UTC. - Attempt 2 (started 00:49:42 UTC) reached
UPDATE_COMPLETEat 01:15:23 UTC with no failed event. TheUpdateConfigurationcustom resource ran twice and logged no warning or error.
The parameter set gained three entries, each with a default that keeps the existing
behaviour: TenantIsolationMode (Off), TenantGrantSource (Manual) and
RefreshTokenValidityHours (720). Nothing was removed or newly required. The upgraded
stack then processed the same document five times.
All seven runs found the same six sections with the same classes. Per-section field counts and empty-field counts:
| section | class | fields (pre ×2 / post ×5) | empty fields, pre | empty fields, post |
|---|---|---|---|---|
| 1 | Payslip | 48, 48 / 48, 48, 48, 48, 42 | 9, 9 | 9 in all 5 |
| 2 | US-drivers-licenses | 22 / 22 | 4, 4 | 4, 4, 5, 5, 4 |
| 3 | Bank-checks | 10 / 10 | 1, 1 | 1 in all 5 |
| 4 | Bank-Statement | 16 / 16 | 6, 5 | 6, 4, 5, 6, 6 |
| 5 | W2 | 35 / 35 | 2, 2 | 2 in all 5 |
| 6 | Homeowners-Insurance-Application | 41 / 41 | 8, 8 | 8 in all 5 |
Finding 4 covers the value differences.
Tier F — benchmarks
Section titled “Tier F — benchmarks”The release A/B entry and the regenerated guidance are carried by the sibling benchmark pull
request, #1823, which adds
docs/benchmarking/releases/v0.6.14.md
and brings docs/benchmarking/config-guidance.md to this
release. This section holds the part that belongs to the validation record.
F1. Same published templates, byte-identical configurations, only the code version
differs: corefast, repeats: 3, --native-upload, both sides on the same stack, the PREV
side run before update-stack (21:47–22:49 UTC) and the NEW side after it (01:26–02:27 UTC).
| v0.6.13 | v0.6.14 | delta | |
|---|---|---|---|
| runs / successful | 171 / 171 | 171 / 171 | — |
| grid cost | $25.6317 | $24.7289 | −3.5% |
| grid wall | 8,936.9 s | 8,818.8 s | −1.3% |
| mean completeness recall | 1.0000 | 1.0000 | — |
| mean cell accuracy | 1.0000 | 1.0000 | — |
cells_compared | 23,370 | 23,370 | — |
| processing issues | 18 | 13 | see below |
aggregate.py --compare reports 0 regressions, 0 improvements and 0 cell-level changes
either way. Both of its inconclusive rows are on the bedrock_llm OCR arm,
core-llm-simple-sep. Its cost fell 45% (0.170 ± 0.169 → 0.093 ± 0.063, n = 9) and its scalar
accuracy rose from 0.889 to 1.000, with paired per-document spread ±0.333. Both moves are
inside the cell’s own spread. Finding 5 explains both.
The processing-issue count fell from 18 to 13. On v0.6.13 the 18 were 14
extraction_row_count_unverified, 2 extraction_sparse and 2
extraction_validation_failed. On v0.6.14 the 13 are 12 extraction_row_count_unverified and
one assessment_truncated. That one is a confidence call on longdesc_100 that truncated,
was split and retried, and left 0 rows unscored. The wall-time figure is not a performance
claim: both sides ran with nothing else on the stack, but wall time on a shared account is
load-sensitive.
F2–F4. Thirty lanes ran six at a time on the upgraded reference stack from 02:33 UTC on 2026-11-07. A scheduled host patch restart at 06:03 UTC killed the six lanes then in flight. Nine lanes had already exited at their planned run counts. The harness has no resume mode, so a killed lane cannot be completed in place. Its partial run directories were moved out of the results tree unscored, and the six lanes were re-run in full from 2026-11-09 20:56 UTC, together with the fifteen lanes that had not started. The classification-variant benchmark runs after them, alone. The refresh produced 32 scored sets: 3,338 rows for $1,970.57. Five rows did not complete, all the simple-mode cells on the 3,200-row document, which exceed a single response by design. The model-selection verdict and every per-suite comparison are in #1823.
Findings, most important first
Section titled “Findings, most important first”1. An upgrade whose container build does not finish in time fails, and so does its rollback (product robustness, environment-triggered)
Section titled “1. An upgrade whose container build does not finish in time fails, and so does its rollback (product robustness, environment-triggered)”Tier E’s first update-stack started at 23:08 UTC. The DockerBuildRun custom resource in the
multi-document-discovery nested stack failed twice with Build … did not complete within the Lambda budget. Its counterpart in the unified pattern stack was cancelled, then failed during
rollback with build did not complete - status: [STOPPED]. The parent reached
UPDATE_ROLLBACK_FAILED at 23:42 UTC. continue-update-rollback at 00:35 UTC returned it to
UPDATE_ROLLBACK_COMPLETE at 00:49 UTC, and the second attempt completed normally. The
hosting-private stack-test, created at the same time, failed on the same resource and reached
ROLLBACK_FAILED.
The trigger was environmental: the builds those resources start did not complete in time,
during a window in which the run’s notes record CodeBuild builds in the account as not
dispatching. The logs here do not establish why, or whether the queueing was account-wide.
Once builds were dispatching again, both stacks deployed. The effect on a customer is not
environmental: an upgrade during a CodeBuild slowdown does not simply fail and roll back. It
leaves the stack needing a manual continue-update-rollback. The repository’s
test-upgrade skill gives that command for a rollback deadlock, but no customer-facing page
described this case at this release. Filed as #1824 and fixed in
#1825, which also adds the recovery to
Troubleshooting. The
evidence is two occurrences in this pass, both on DockerBuildRun (tier E’s first attempt and
the hosting-private stack-test’s first attempt), and the cause of the stall is not established.
2. Test Studio’s Re-extract offer after a class change is lost on reload, and a second change names the wrong class (product defect, this release)
Section titled “2. Test Studio’s Re-extract offer after a class change is lost on reload, and a second change names the wrong class (product defect, this release)”Tier G, flow 6.2. After a section’s class is changed with Edit Page Grouping, the editor
offers per-section Re-extract with the note “These fields were extracted as class”. That
offer is held only in React state (reclassified in test-set-editor-slots.tsx). It is
lost on a page reload, although the section’s fields still belong to the old class. It also
records the class from before the latest change rather than the class the fields came from.
A second change therefore names the wrong one: the note said “extracted as W2” for fields
extracted as Bank-checks. The per-section Re-extract that #1672 needed now works correctly when
it is offered. Filed as
#1816.
3. A test set created from S3 files with no baselines fails after being reported created (product defect)
Section titled “3. A test set created from S3 files with no baselines fails after being reported created (product defect)”Tier G. The From files in S3 wizard, pointed at documents with no baseline files, says on
one step that unlabeled documents will arrive and in a hint that they are skipped. It then
reports the set as created, and the set ends in FAILED. Adding documents is then refused
because the set is “not in COMPLETED status”, so the only remedy is to delete and recreate it.
Whether v0.6.13 behaves the same way was not checked. Filed as
#1817.
4. The upgrade changed no extracted value consistently; one post-upgrade run silently dropped two of three tax rows (measurement)
Section titled “4. The upgrade changed no extracted value consistently; one post-upgrade run silently dropped two of three tax rows (measurement)”Across the two v0.6.13 runs and five v0.6.14 runs of lending_package.pdf, every value
difference is inconsistent within one version or the other:
| field | v0.6.13 (2 runs) | v0.6.14 (5 runs) |
|---|---|---|
Payslip name and address fields, e.g. EmployeeName.FirstName | "JOHN" in 1, "John" in 1 | "JOHN" in 5 |
Payslip FederalTaxes | 3 rows in both | 3 rows in 4, 1 row in 1 |
Bank-checks check_number | "0000000000" in 1, "0001379" in 1 | "0000000000" in 4, "0001379" in 1 |
Bank-Statement account_holder_address.State | null in both | null in 3, "USA" in 2 |
driver’s licence ENDORSEMENTS / RESTRICTIONS | [] in both | [] in 3, null or ["NONE"] in 2 |
So there is no consistent version effect on this document. Two observations are worth naming:
- One run returned a shortened list as a success. The fifth v0.6.14 run returned one of
the Payslip’s three
FederalTaxesrows (Social Security and Medicare are missing). Its section reportedStatus: SUCCESSwith no processing issue, from a single traditional extraction pass on Claude Sonnet 5, the preset’s extraction model. One run in seven is too few to attribute to the release, but the list loss itself was silent. Not filed. It fits the standing silent list-loss class that the benchmark paper’s hazards section tracks, rather than a new defect. - The tax-amount signs are now stable. All seven runs report the tax amounts as negative, where the v0.6.13 record saw one positive in five.
There is no ground-truth baseline for this sample in the repository, so which value is correct was not scored.
5. The A/B asymmetry the v0.6.13 record left open recurred on v0.6.13 and not on v0.6.14 (improvement, inferred)
Section titled “5. The A/B asymmetry the v0.6.13 record left open recurred on v0.6.13 and not on v0.6.14 (improvement, inferred)”The v0.6.13 record found that two of three v0.6.13 runs of the bedrock_llm arm split
longdesc_100, a one-section document, into two sections, against none of three on v0.6.12.
It left open whether this was a release effect or OCR run-to-run variation. In this pass’s
PREV side, v0.6.13 split it in two of three runs again, and v0.6.14 split it in none
of three. Both inconclusive rows in F1 follow from that: the arm’s cost fell 45% and its
scalar accuracy rose to 1.000. Two independent v0.6.13 grids showing the same 2-of-3 split,
against 0-of-3 on the releases either side, points to a v0.6.13 behaviour that v0.6.14
removed. That reading is inferred from n = 3 per side, and no single change in this release
has been identified as the cause.
6. The tenancy modules this release adds pulled two coverage trees down by more than a point (coverage)
Section titled “6. The tenancy modules this release adds pulled two coverage trees down by more than a point (coverage)”idp_sdk fell from 99.47% to 98.01% and idp_cli from 99.68% to 98.07%. The ratchet passes,
because it is per file, and each new file is recorded at the coverage it had when it was
added. The lowest-covered
files in those trees are all new in this release:
idp_sdk/_core/ui_api.py, 60.0%idp_cli/tenancy.py, 75.4%idp_sdk/_core/tenancy_verify.py, 78.8%idp_sdk/operations/tenancy.py, 83.2%
The tier C tenant-isolation run exercises this code against a live stack, but that run is not counted in coverage.
7. The browser review’s usability findings (tier G)
Section titled “7. The browser review’s usability findings (tier G)”Beyond Findings 2 and 3, the review ranked these, most important first. Each was seen on a screenshot of this build. None is filed as an issue in this pass.
- Documents whose labels are an unreviewed machine draft still label their values “Ground truth:”.
- A document reports Review Reason (6) while about 15 fields are shown red, including empty fields at 0%. Since this release stops empty fields raising confidence alerts, their red rendering no longer matches the count.
- Table-row review chips are ambiguous: several read only “Period”.
- The Edit/View toggle stays on Edit for a read-only document.
- The new Missing Required column is hidden by default.
- The configuration version is visible only in a badge’s hover text.
- The Generate draft labels copy mentions a published version when there is none.
- Next alert after a chip jump restarts at the first alert.
- Re-extracting one section resets the whole document to Draft (machine).
- A Reviewer’s Tasks filter still offers Test Studio, which this release removes from that role.
- Smaller polish items:
- “Upload (1 files)”
- Inbox column truncation
- unread Inbox messages are not marked with the text Unread, which the changelog says they are
- the Prefix Mappings breadcrumb reads “Documents”
- degraded sections are not explained
Four flows were not run: 6.1, 11.1, 4.1 and 12.1, which need a misclassified fixture or an Annotator user. One Inbox message the review generated (“default: documents waiting”) could not be retracted. The full ranked list, per-flow evidence and the screenshots are kept out of the repository, as the review’s privacy rule requires.
8. Tier H’s idp_cli suite is still empty (test tooling, unchanged)
Section titled “8. Tier H’s idp_cli suite is still empty (test tooling, unchanged)”lib/idp_cli_pkg selects 0 of 1,547 tests under -m integration (1,506 at v0.6.13), so that
command runs nothing. The SDK’s integration suite, which exercises deployment and batch
behaviour against the reference stack, passed 13 of 13 (v0.6.13: 12 of 13). Its one failure
there was #1673, now fixed.
9. make dep-audit lists four unmapped advisories, three of them HIGH, that it cannot gate (informational)
Section titled “9. make dep-audit lists four unmapped advisories, three of them HIGH, that it cannot gate (informational)”The audit’s unmapped-advisory pass lists four GitHub advisories that name a pinned package whose version is in the stated range, but carry no machine-readable affected range:
picomatch2.3.2 and 4.0.7: ReDoS, HIGHuri-js4.4.1: denial of service, HIGHuri-js4.4.1: improper decoding, MEDIUM
They are reported for review, never gated, by design. In src/ui/package-lock.json both
packages are development dependencies (build tooling). The other lockfiles that name them were
not checked.
10. SRT’s scoped-out count and both RBAC counts grew (informational)
Section titled “10. SRT’s scoped-out count and both RBAC counts grew (informational)”- SRT scopes 57 HIGH findings out of the gate, up from 51. All 57 are Bandit
B105/B106identifier names in test code from which no deployment artifact is built. None counts against the gate. - RBAC static operations rose from 164 to 168, and dynamic operations from 161 to 165. The
four new operations are the tenant-administration API this release adds:
listTenants,putTenant,deleteTenantandsetUserTenant. The dynamic check count rose by exactly their share, from 1,176 to 1,204. That is 16 authorization-matrix checks, 4 unauthenticated-access checks and 8 other per-operation checks. There were still 0 hard failures, 0 not run and 17 skipped.
11. make version leaves the lockfile behind, so every version bump breaks the locked install (release tooling)
Section titled “11. make version leaves the lockfile behind, so every version bump breaks the locked install (release tooling)”The fresh worktree at v0.6.14 would not install until lib/idp_common_pkg/uv.lock’s own
project version was patched to 0.6.14. make setup runs uv export --locked, which refuses a
lockfile out of date with its pyproject.toml. PR #1808 fixed the instance on develop the
same day. The version target in the Makefile still does not touch the lockfile, so the
next bump will reproduce it. Filed as
#1818.
12. Environment events during the pass (environment)
Section titled “12. Environment events during the pass (environment)”- Two CodeBuild-stall failures (Finding 1): tier E’s first attempt and hosting-private’s first attempt. Both were re-run and passed.
- Orphaned network interfaces: three in the stack-test VPC, deleted.
- A host restart killed six benchmark lanes at 06:03 UTC on 2026-11-07. It was a scheduled patch reboot, and it interrupted the run as the procedure warns. The lanes were re-run in full; partial results were discarded, not scored.
Issues filed and their status
Section titled “Issues filed and their status”| Issue | Finding | Status |
|---|---|---|
| #1816 — Re-extract offer lost on reload, wrong class after a second change | 2 | open |
| #1817 — S3 test set with no baselines fails after being reported created | 3 | open |
#1818 — make version does not update uv.lock | 11 | open |
The five product issues the v0.6.13 record filed (#1672, #1673, #1676, #1677, #1678) are all closed, with fixes in this release.
What was NOT validated
Section titled “What was NOT validated”- Real GovCloud. Tier D’s
--govcloudrun wasREGION=us-west-2with commercial credentials. Partition ARNs, GovCloud model availability and the BDA project rejection are unproven, and the Knowledge Base was disabled in that run. - The seller positive path. No token was issued or verified.
TenantGrantSource=IdpClaimandApi. The tenant-isolation tier deploysTenantIsolationMode=Strictwith the defaultManualgrant source. Granting a tenant from a federated claim or throughsetUserTenantwas exercised only by the RBAC authorization checks on that operation.idp_cliintegration behaviour. The suite is empty (Finding 8).- The cause of the CodeBuild stall in Finding 1, and whether an upgrade fails the same way under a slow but progressing build queue.
- Which value of Finding 4’s varying fields is correct. There is no ground-truth baseline
for
samples/lending_package.pdfin the repository. - Whether Finding 3 predates this release.
- Four browser flows (6.1, 11.1, 4.1, 12.1), for want of a misclassified fixture and an Annotator user. Production review queues end to end: HITL is off on the reference stack, as on any stack upgraded with its previous values.
- Canada (
ca-central-1) andeu-central-1. All stacks wereus-west-2(sellerus-east-1). - Malware scanning.
MalwareScanningdefaulted toDISABLEDin every stack.
Reproduce
Section titled “Reproduce”# Tier A (offline), in a dedicated worktree at the tag — never in a shared checkoutgit worktree add <path> v0.6.14 && cd <path>CI=1 python3 scripts/srt/setup.py && make srt-scan # before anything writes build outputmake test && make lint-cicd && make typecheck && make dep-audit && (cd src/ui && npx vitest run)make coverage-all && make check-coverage-debtpython3 publish.py <bucket-base> <prefix> us-west-2 --clean-build
URL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.14.yaml
# Tier B, against the upgraded stackmake security-results STACK_NAME=<stack> REGION=us-west-2 SKIP_SRT=1
# Tier C (6 variants)make stacktest-hosting-global TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-waf TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-hosting-private TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \ VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>make stacktest-jobsapi TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \ VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>make stacktest-tenancy TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-seller # REGION deliberately unset -> us-east-1
# Tier Dmake transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<email>
# Tier E — several documents per side, or run-to-run variation reads as a version effectaws cloudformation create-stack --stack-name IDPUpg0613to0614 --template-url <0.6.13 URL> ...aws cloudformation update-stack --stack-name IDPUpg0613to0614 --template-url $URL \ --parameters file://<every existing parameter with UsePreviousValue=true>.json# if the update and its rollback both fail on DockerBuildRun:aws cloudformation continue-update-rollback --stack-name IDPUpg0613to0614
# Tier F1, PREV side before update-stack, NEW side after, on the same stackpython benchmarks/harness/run_matrix.py --stack <stack> --suite corefast --repeats 3 \ --native-upload --validate-against-release v0.6.13 --max-inflight 4python benchmarks/harness/aggregate.py --compare <v0.6.14 summary> --baseline <v0.6.13 summary>
# Tier G: headless Chromium via Playwright, throwaway users from scripts/ux_test_session.py# Tier H, on the upgraded stackcd lib/idp_common_pkg && make test-integrationIDP_STACK_NAME=<stack> python -m pytest -m integration lib/idp_sdk/tests/integrationRaw logs for every tier are in gitignored scratch/ and are not published.