Release Validation: v0.6.13
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
Release Validation Record — v0.6.13
Section titled “Release Validation Record — v0.6.13”Release: v0.6.13 (VERSION = 0.6.13, newest CHANGELOG.md section)
Validated: 2026-10-31, us-west-2 (seller tier us-east-1), commercial partition
Commit measured: a0aa0bb76, the commit tag v0.6.13 points to. Every offline figure in
this record was measured in a dedicated git worktree at that commit. The primary checkout is
shared with other assistant sessions, and one of them pulled it forward to a later develop
commit while an earlier offline pass was running, so that pass was discarded and every offline
tier was re-run in the worktree (Finding 10).
Artifact under test: the published object
https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.13.yaml,
HTTP 200, Description AWS GenAI IDP Accelerator (uksb-r8evguc4p9) (SO9027) (v0.6.13),
published 2026-10-31 01:33 UTC. Every stack in tiers B, C and E was created from that object
(or, for tier E, from the published v0.6.12 object and then updated to it). Tier D publishes
from source at the tag to apply its transforms, and tier A’s build row is a local
publish.py --clean-build.
Account <ACCOUNT_ID>. PREV = v0.6.12.
Verdict
Section titled “Verdict”✅ Ship, with two product defects in features this release adds that a maintainer should fix in the next patch. The published artifact deploys in all five shipped configurations, both template transforms deploy and process a real document, the upgrade from 0.6.12 applies without rollback, and the 342-run release A/B shows 0 regressions and 0 improvements, with recall and cell accuracy identical on both sides over the same 23,370 compared cells.
All 8 tiers ran. Tier G, the browser review, ran for the first time in any release record: a headless Chromium driven through Playwright replaced the debug-Chrome connection whose absence left it NOT RUN at v0.6.11 and v0.6.12. Tier H ran three suites rather than two, because this pass added the SDK’s own integration suite, which no earlier record ran.
The two defects that matter most are both in this release’s new surface, and neither is visible in the A/B.
- Re-extracting one section in the Test Studio editor rewrites the class of every section in the document (#1672). v0.6.13’s edit screen offers Re-extract on the section whose class was changed, but the API it calls is document-scoped, so a six-section packet ends with all six sections relabelled and each keeping its own fields. That is silent damage to ground truth. Finding 1.
- Packet-granularity classification with page images fails 18–20-page documents on the shipped classification model (#1676). The packet window is a page count, and a 19-page window of full-resolution scans exceeds Nova 2 Lite’s input, so the document fails rather than being split into smaller windows: 3 of 20 DocSplit packets in every packet+images variant. The same measurement shows packet granularity reaching 0.71–0.82 boundary accuracy against the shipped page default’s 0.45, at 23–65% of its classification cost per page, so this is a defect in the configuration a reader of the new guidance is most likely to choose. Finding 2.
One fix this release shipped holds only partly. The separate confidence step still runs
out of memory after a successful extraction on the large single-table benchmark document:
2 of 190 runs, down from 15 of 190 at v0.6.12 (14 failed and one was left in EXTRACTING) (#1678,
following #1488). Finding 3.
Tier F is complete: F1, the release A/B, plus the F2–F4 guidance refresh, 31 scored pipeline result sets, 3,368 scored rows, $2,077.09, plus a 140-document classification-variant benchmark that measures this release’s headline classification changes for the first time. Its unsuccessful rows fell from 26 of 3,248 (0.80%) at v0.6.12 to 15 of 3,368 (0.45%), and every one of the 15 is attributed below.
Results per tier group
Section titled “Results per tier group”| Tier | Gate | Result | vs v0.6.12 |
|---|---|---|---|
| A — SRT | ✅ PASS | 0 open/reopened HIGH counted against the gate; 51 B105/B106 identifier names in test code, scoped out of the gate by the documented rule; 50,868 CI-visible findings | 10 scoped-out → 51 (Finding 11) |
A — make test | ✅ PASS | all 74 test roots, 0 failures, in the worktree at the tag | 69 roots → 74 |
A — make lint-cicd | ✅ PASS | exit 0; markdown link gate clean over 399 files | unchanged |
A — make dep-audit | ✅ PASS | 1,752 packages, 0 gating, 6 allowlisted, 15 below HIGH, 0 stale allowlist entries | 10 → 15 below HIGH |
A — make typecheck | ✅ PASS | 0 errors / 47 warnings | 0 / 44 → 0 / 47 |
A — UI tests (vitest) | ✅ PASS | 231 files, 2,626 tests | 124 / 1,507 → 231 / 2,626 |
| A — build + package | ✅ PASS | publish.py --clean-build in a clean env, 7.0 min; template validation passed and cfn-lint clean on all 5 packaged templates (8 advisory warnings) | 4.2 min → 7.0 min; warnings identical |
| A — coverage | ✅ PASS | 11 trees measured, ratchet green (528 files at or above recorded floor) | see the coverage table |
| B — security snapshot | ✅ 4 of 4 gates | SRT PASS · RBAC static PASS (0 fail / 4 known-gap warn, 164 operations) · RBAC dynamic PASS (1,176 checks, 0 hard fail, 0 not-run, 17 skipped) · ZAP PASS (High 0, Medium 1) | RBAC dynamic 876 → 1,176 checks (Finding 11) |
| C — deploy variants | ✅ 5/5 | hosting-global (40 min), WAF (34 min), hosting-private (66 min), Jobs API (60 min), seller (3 min), each self-deployed from the published template and torn down | 5/5, unchanged |
| D — transforms | ✅ 2/2 | --headless and --govcloud each deployed, processed lending_package.pdf (classified Payslip, 18/22 fields) and tore down | 2/2; 17/22 → 18/22 fields |
| E — upgrade 0.6.12 → 0.6.13 | ✅ PASS | UPDATE_COMPLETE in 22 m 46 s, 0 rollback or failed events, Description now (v0.6.13); one added parameter (KnowledgeBaseEmbeddingModel, defaulted); the configuration custom resource logged no warning or error. No consistent change in extracted values (Finding 4) | 0.6.12 reported a version effect on six fields |
| F1 — release A/B | ✅ PASS | 171/171 and 171/171; 0 regressions, 0 improvements, 1 inconclusive (cost on the bedrock_llm arm, inside its own spread). That row is the grid’s one asymmetry: 2 of 3 v0.6.13 runs split a one-section document, 0 of 3 on v0.6.12 — unattributed, see What was NOT validated | same shape |
| F2–F4 — guidance refresh | ✅ COMPLETE | 31 scored sets, 3,368 rows, $2,077.09; 15 unsuccessful rows (0.45%), all attributed; classification-variant benchmark 140 documents | 26/3,248 (0.80%) → 15/3,368 (0.45%) |
| G — browser UX | ⚠️ RAN, 1 flow FAIL | 9 flows or areas PASS, flow 6.2 FAIL (Finding 1), production-queue Reviewer work BLOCKED (HITL off on the reference stack) | NOT RUN → run |
| H — integration tests | ⚠️ 2 of 3 suites pass | idp_common 19/19; idp_sdk 12 passed / 1 failed (Finding 6, pre-existing); idp_cli selects 0 of 1,506 tests | idp_sdk suite not run before |
Tier A — test coverage, as measured
Section titled “Tier A — test coverage, as measured”scripts/coverage_all.py --jobs 1 over all eleven registry trees, one tree at a time, then
make check-coverage-debt. The ratchet is green: 528 files across 11 trees at or above their
recorded coverage. Running the trees one at a time is deliberate. An earlier pass that ran them
concurrently in the shared checkout produced two false falls, described in Finding 10.
| tree | measured | v0.6.12 record | delta |
|---|---|---|---|
| api_resolvers | 71.38% (lowest) | 71.01% | +0.37 |
| seller_entitlement | 82.03% | 82.03% | — |
| idp_common | 85.44% | 84.11% | +1.33 |
| pii_anonymizer_hook | 86.98% | 86.98% | — |
| src_lambda | 90.95% | 89.90% | +1.05 |
| scripts | 91.56% | 86.12% | +5.44 |
| main_stack_extensions | 93.90% | 93.86% | +0.04 |
| pii_anonymizer_api | 94.93% | 94.93% | — |
| idp_sdk | 99.47% | 99.65% | −0.18 |
| idp_cli | 99.68% | 99.68% | — |
| idp_feature_sdk | 99.94% | 99.94% | — |
No tree lost more than a fifth of a point. api_resolvers stays the lowest tree, now at 71.38%.
Tier B — security gate results compared with the previous release
Section titled “Tier B — security gate results compared with the previous release”All four gates pass, at the tag’s SHA, against the upgraded reference stack. The snapshot is in
security/test-results/0.6.13/.
ZAP is unchanged from v0.6.12 (High 0, one Medium Cross-Domain Misconfiguration alert on five
URLs). The two counts that moved are explained in Finding 11.
Tier C — the five deploy variants
Section titled “Tier C — the five deploy variants”Every variant self-deployed its own throwaway stack from the published template, ran its assertions, and tore down:
- hosting-global: a regional REST API serves the web UI with HTTP 200.
- WAF: the WebACL is associated with the API.
- hosting-private: a private REST API with its resource policy, in the purpose-built
stack-test VPC (
<VPC_ID>, two private subnets, anexecute-apiinterface endpoint). - Jobs API: deployed into the same VPC.
- seller: the malformed-body, oversized-body and 15 hostile-payload refusals, in
us-east-1.
As in every earlier record, the seller tier asserts refusals only; no token was issued and verified. After the VPC tiers, four orphaned Lambda network interfaces were swept from the stack-test VPC: two from this pass’s hosting-private stack and two from an older stack that no longer exists.
Tier D — template transforms, and the limits of a commercial-region run
Section titled “Tier D — template transforms, and the limits of a commercial-region run”Both transforms deployed and processed samples/lending_package.pdf end to end. The document
was classified Payslip with 18 of 22 fields populated; v0.6.12 populated 17.
--headless(250 resources): UI, Cognito and CloudFront are absent, and the processing core is present.--govcloud(345 resources): noAWS::CloudFront::*orAWS::Lambda::Url, the LWA chat-stream family is removed (#677), and the UI is retained and hosted on API Gateway.
⚠️ The --govcloud result is a commercial-partition result. It proves the
CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves
nothing about partition ARNs, GovCloud model availability or the BDA project rejection.
The Knowledge Base was disabled in that run, which is the runner’s default.
Tier E — in-place upgrade, and what it did to extracted values
Section titled “Tier E — in-place upgrade, and what it did to extracted values”A fresh stack, IDPUpg0612to0613, was created from the published v0.6.12 template and
processed lending_package.pdf twice. It was then updated to the published v0.6.13 template,
with UsePreviousValue=true on all 92 existing parameters, and processed the same document
five times. The update completed in 22 m 46 s with no rollback or failed event, and the
UpdateConfiguration custom resource ran twice and logged no warning or error. The parameter
set gained one entry, KnowledgeBaseEmbeddingModel, which defaults to the embedding model every
existing deployment already uses. Nothing was removed or newly required.
All seven runs found the same six sections with the same classes. Finding 4 covers the field values.
Tier F — benchmarks
Section titled “Tier F — benchmarks”The release A/B entry is
docs/benchmarking/releases/v0.6.13.md and the
regenerated guidance is docs/benchmarking/config-guidance.md,
both carried by the sibling benchmark pull request. This section holds the part that belongs to
the validation record.
F1. Same published templates, byte-identical configurations, only the code version differs:
corefast, repeats: 3, --native-upload, both sides on the same stack, the PREV side before
update-stack.
| v0.6.12 | v0.6.13 | delta | |
|---|---|---|---|
| runs / successful | 171 / 171 | 171 / 171 | — |
| grid cost | $24.5960 | $24.8415 | +1.0% |
| grid wall | 10,074.3 s | 8,464.7 s | −16.0% |
| mean completeness recall | 1.0000 | 1.0000 | — |
| mean cell accuracy | 1.0000 | 1.0000 | — |
cells_compared | 23,370 | 23,370 | — |
| processing issues | 9 | 22 | see below |
aggregate.py --compare reports 0 regressions, 0 improvements and 0 cell-level changes
either way. Its one inconclusive row is the bedrock_llm OCR arm core-llm-simple-sep,
whose cost rose 57% (0.092 ± 0.062 → 0.145 ± 0.150, n = 9), inside its own spread. The two
expensive runs are two of the v0.6.13 side’s three runs on longdesc_100. Both split that
one-section document into two sections, and they already differed from the third run at OCR,
which runs before classification: OCR cost $0.046 against $0.0115 (4×). Extraction then cost
$0.334 against $0.134 (2.5×), with classification and assessment close to unchanged. Recall and cell
accuracy stayed 1.000. No v0.6.12 run, and no other v0.6.13 run, split a document, so this is
the grid’s one asymmetry between the sides. Two explanations fit and n = 3 per side cannot
separate them: this arm’s OCR text is itself written by a model, and the 4× OCR cost shows
those runs’ OCR output already differed before any section was formed; and this release
changed how classification forms sections. It is listed under What was NOT validated,
with the measurement that would separate the two. Wall time fell 16%. Both sides
ran with nothing else on the stack, but wall time on a shared account is load-sensitive, so
the figure is not a performance claim.
The processing-issue count rose from 9 to 22 because of new measurement, not a
regression. Fourteen of the 22 are extraction_row_count_unverified, a code this release
adds (#1489’s fix) to report a list it had no OCR evidence to check. The nine
assessment_geometry_incomplete notes v0.6.12 carried are gone, because this release narrowed
that code (#1520).
F2–F4. The lanes ran six at a time on the upgraded reference stack between 05:22 and
11:58 UTC. Every lane exited at its planned run count. The classification-variant benchmark
ran afterwards, alone. One harness defect shaped how the lanes were launched:
--validate-against-release v0.6.13 refuses the flat keys --native-upload always adds
(#1674),
so every lane validated against v0.6.12, as the A/B did. That attempt failed before uploading
anything, so it cost nothing.
Model selection (config-guidance.md §5). Claude Opus 5.5 and Claude Opus 5 are the only
extraction models at recall and cell accuracy 1.0000 over all 83,790 compared cells of the
19-configuration grid, and Opus 5.5 costs 16% less than Opus 5 there ($107.67 against $128.52
for the grid). The shipped default, Claude Sonnet 5, reaches recall 0.9307 over 69,042 cells,
and all of its shortfall is simple mode on the 17-page document. So the guidance is unchanged:
keep the default, and use advanced mode above about 400 rows. Paying at least 27% more for
Opus 5.5 buys a complete list in either mode, and Opus 5.5 is at least 43% cheaper than GPT-6
Astra, which has the lowest cell accuracy in the grid. pricing.yaml prices Sonnet 5 and
Sonnet 5 :1m at 1.5× their published rates, so every Sonnet 5 dollar figure is an upper bound
and every ratio against it is one-sided. The guide states this in its header.
Two closed issues still reproduce in the refresh. Claude Sonnet 5.5 still loses most of a 400-row list on 16 runs, 6 of them with no processing issue at all (#1494). The reprinted running-header packet still splits into three sections under the generic boundary prompt on 5 of 5 runs (#750).
Why the 15 unsuccessful runs failed
Section titled “Why the 15 unsuccessful runs failed”| runs | where | cause | new? |
|---|---|---|---|
| 9 | scaling and astracap, simple mode, scale_3200.pdf (3,200 rows, 66 pages) | one extraction request too long for the model: ~157,000 and ~320,000 tokens against a 200,000-token window, and ~325,000 tokens refused by GPT-6 Astra although its window is recorded as 1,050,000 | no — the simple-mode capacity limit; astra-simple-sep also failed at v0.6.12 |
| 4 | core, ocr_bench corpus | SCORE_ERROR: a stored confidence is 99.9 rather than 0.999 | no — v0.6.12 also had four, two of them in the same cells |
| 2 | coresynth, simple + separate, large_narrow.pdf | Runtime.OutOfMemory in AssessmentStep after extraction succeeded | partly — Finding 3 |
Classification variants
Section titled “Classification variants”The seven combinations were run on 20 DocSplit packets with Nova 2 Lite, the shipped
classification model. This is the first measurement of this release’s headline classification
change; the full table and its caveats are in config-guidance.md §5.6. Over the 15 packets every variant
scored:
- Boundaries: packet granularity places section boundaries correctly 0.711–0.821 of the time, against 0.446 for the shipped page default.
- Cost: its classification cost per page is 23–65% of the default’s.
- Class accuracy:
packet-imagesis highest, at 0.909 against the default’s 0.851.
Every packet variant that sends page images fails the same three 18–20-page packets (Finding 2), and text-only page classification fails two packets (Finding 5). Twenty documents is a small sample, and the guide says so.
Findings, most important first
Section titled “Findings, most important first”1. Re-extracting one section in the Test Studio editor rewrites the class of every section (product defect, this release)
Section titled “1. Re-extracting one section in the Test Studio editor rewrites the class of every section (product defect, this release)”Tier G, flow 6.2, on a test set built from lending_package.pdf (six sections). The reviewer
changed section 3’s class in the Test Studio item editor and pressed that section’s
Re-extract. All six sections/*/result.json files in the set’s baseline were then rewritten,
within the same second, with the new class. Each section kept its own extracted fields, so
payslip fields ended up under Bank-checks.
The cause is confirmed in the code. reextractTestSetDocument takes testSetId,
objectKey, documentClass and configVersion, with no section id, and its schema comment
says documentClass is “written to the document’s baseline sections”. The v0.6.12 editor
called it the same way. What v0.6.13 changes is that the edit screen now offers Re-extract on
the individual section whose class was changed, so a control that reads as section-scoped
drives a document-wide write. The damage is silent: the editor shows the new labels as if
intended, and the only recovery is an earlier test-set version. Filed as
#1672.
2. Packet classification with page images fails long packets on the shipped model (product defect, this release)
Section titled “2. Packet classification with page images fails long packets on the shipped model (product defect, this release)”The classification-variant benchmark ran 20 DocSplit packets through each of seven granularity, modality and strategy combinations on Nova 2 Lite, the shipped classification model.
The three variants that send page images at packet granularity are packet-both,
packet-images and split-then-classify-packet. All three failed the same three packets,
packet_0011, 0012 and 0019, which are 18 to 20 pages of 2550×3301 scans. Each one failed
with ValidationException: Input is too long for requested model on its single packet window,
then Packet classification produced no segments, and the document failed. Every
page-granularity variant classified those three packets, and so did packet-text.
The window is packet.window_pages, a page count with a default of 20, and nothing sizes it
against the model’s input limit or retries an oversized window as smaller ones. Filed as
#1676.
This matters more than its 15% failure rate suggests. The same benchmark shows packet
granularity is the better choice on these documents, so it is the configuration the new
guidance points readers towards.
3. The separate confidence step still discards some large documents after the #1488 fix (product defect, pre-existing, much reduced)
Section titled “3. The separate confidence step still discards some large documents after the #1488 fix (product defect, pre-existing, much reduced)”In simple mode with assessment: separate on large_narrow.pdf, the configuration and document
#1488 was measured on, the confidence step ran out of memory after extraction succeeded in
2 of 190 runs, against 15 of 190 at v0.6.12 (14 failed, 1 left in EXTRACTING). Both failures ended in AssessmentStep with
Runtime.OutOfMemory: one with textract_tables OCR and GPT-6 Astra extraction, one with
textract_layout OCR and Claude Haiku 5.5 extraction. The 1,200- and 1,600-row failures v0.6.12
recorded did not recur. #1488 is closed, so the residual is filed separately as
#1678.
4. The upgrade changed no extracted value consistently; three fields vary from run to run on v0.6.13 (measurement)
Section titled “4. The upgrade changed no extracted value consistently; three fields vary from run to run on v0.6.13 (measurement)”Across the two v0.6.12 runs and five v0.6.13 runs of lending_package.pdf, every difference
the diff found is inconsistent within v0.6.13 itself:
| field | v0.6.12 (2 runs) | v0.6.13 (5 runs) |
|---|---|---|
six Payslip tax amounts, e.g. FederalTaxes[0].Period | negative in both | negative in 4, positive in 1 |
driver’s licence ENDORSEMENTS[0] | "NONE" in both | "NONE" in 2, null in 3 |
Bank-checks bank_name | null in both | "BANK NAME" in 3, null in 2 |
So there is no consistent version effect on this document, and the most likely reading is model run-to-run variation. Two runs are too few to call v0.6.12 deterministic, so a variation present in both releases cannot be ruled out. The tax-amount sign is worth naming: the v0.6.12 record attributed a sign change in these same six fields to the 0.6.11 → 0.6.12 upgrade, from three identical runs, and at v0.6.13 that sign is not stable from run to run. There is no ground-truth baseline for this sample in the repository, so which sign is correct was not scored.
5. Text-only page classification fails the whole document on one blank page (product defect, pre-existing)
Section titled “5. Text-only page classification fails the whole document on one blank page (product defect, pre-existing)”In the classification-variant benchmark, the page-text variant failed 2 of 20 packets with
Processing failed for 1 out of N sections: Error classifying page K: No content available for classification. In each, one page with no OCR text (page 10 in one, page 2 in the other) took
the whole document down. The same result-with-error exists at v0.6.12, but this release makes
text-only modality an ordinary picklist choice in the configuration editor rather than an
edited prompt. Filed as
#1677.
6. The minimal configuration the SDK and CLI generate fails their own validation (product defect, pre-existing)
Section titled “6. The minimal configuration the SDK and CLI generate fails their own validation (product defect, pre-existing)”Tier H’s SDK suite failed one test of 13, test_config_management.py::test_upload_config.
config.create(features="min"), and therefore idp-cli config-create --features min, writes
# Uses system default prompt into every prompt field. config.validate then refuses the file,
because the classification and extraction task_prompt carry no {DOCUMENT_TEXT} or
{DOCUMENT_IMAGE} placeholder, and config.upload refuses it too. Reproduced at v0.6.12
with the same two errors, so it predates this release. It was not seen earlier because no
earlier record ran the SDK’s integration suite. Filed as
#1673.
7. The browser review’s usability findings (tier G, first run)
Section titled “7. The browser review’s usability findings (tier G, first run)”Beyond Finding 1, the review ranked these, most important first. Each was seen on a screenshot of this build; none is filed as an issue in this pass.
- Machine output can become ground truth without a confirmation. Use As Evaluation Baseline runs immediately and overwrites an existing baseline, and a test set then shows the unreviewed model output as green Ground truth. The Reviewer’s Queues list also counts fully labelled sets as entirely pending, while a set’s own queue page reports 0 pending.
- The same document reports two alert counts. It shows 37 confidence alerts on its own page and “7 of 142 fields flagged” in the test set, and empty fields render as a red “0.0%” confidence.
- A human correction looks like model output. An edited field still reads “Predicted … 100%”, no banner marks a saved-but-unprocessed draft, and reviewed queue rows look unreviewed.
- Annotator work is harder to reach than it should be. An item routed for one alert opens on all 361 fields. An Annotator’s queue page defaults to a filter that shows 0 documents while two are assigned to them.
- Copy points at names that do not exist. For example, Start annotating (the button reads Start editing) and My Work › Tasks (the navigation reads Home › Tasks).
- Read-only configuration looks editable. The managed
defaultprofile’s fields accept typing; only Save is disabled.
The full ranked list, per-flow evidence and the screenshots are kept out of the repository, as the review’s privacy rule requires.
8. Tier H’s idp_cli suite is empty, and the SDK’s integration suite was not in the tier (test tooling)
Section titled “8. Tier H’s idp_cli suite is empty, and the SDK’s integration suite was not in the tier (test tooling)”lib/idp_cli_pkg selects 0 of 1,506 tests under -m integration, as it did at v0.6.12, so the
command the release-validation procedure lists runs nothing. The suites that exercise
deployment and batch behaviour against a stack are in lib/idp_sdk/tests/integration/. This
pass ran them with IDP_STACK_NAME set to the reference stack: 12 passed and 1 failed
(Finding 6).
9. A local dependency audit reads gitignored lockfiles (gate tooling)
Section titled “9. A local dependency audit reads gitignored lockfiles (gate tooling)”In the primary checkout, make dep-audit reported 8 gating findings: aiohttp 3.14.3,
brotli/brotlicffi 1.2.0 and pypdf 6.19.0. All came from untracked copies of
lib/idp_common_pkg/uv.lock, in a scratch worktree and in the IDP data generator’s staged build
copies. The tracked lockfile no longer contains aiohttp at all, and the same command in a clean
worktree of the tag reported 0 gating. The cause is that the manifest generator discovers
lockfiles with rglob rather than git ls-files. CI is unaffected. Filed as
#1675.
10. Offline gates measured in a shared checkout measured a moving tree (process)
Section titled “10. Offline gates measured in a shared checkout measured a moving tree (process)”The first offline pass ran in the primary checkout. Another session pulled it from the release
tree to a later develop commit at 02:09 UTC, while that pass was running. The pass therefore
measured two trees, and the first security snapshot recorded the later SHA. In the same
checkout, two coverage runs that measured the trees concurrently reported false falls:
idp_common/discovery/regional_defaults.pydropped from 100% to 0%; it measured 85.45% for the tree when re-run alone.- A packaging test failed on layer build output that appeared during the run; the writer could not be attributed.
Every offline figure in this record therefore comes from a dedicated worktree at the tag, with the coverage trees measured one at a time. The procedure already says to use a worktree. This pass shows why the instruction has to be followed from the first gate, not after a problem appears.
11. SRT’s scoped-out count and the RBAC dynamic check count both grew (informational)
Section titled “11. SRT’s scoped-out count and the RBAC dynamic check count both grew (informational)”- SRT has 51 open HIGH findings that the gate scopes out, up from 10. All 51 are Bandit
B105/B106identifier names in test code from which no deployment artifact is built. Most are in test files this cycle added for the UX-test harness (scripts/tests/test_ux_*). None counts against the gate. - RBAC dynamic grew from 876 to 1,176 checks, still with 0 hard failures, 0 not run and 17 skipped, following this release’s new API operations (review queues, Inbox, Test Studio queues).
12. Environment debris found and cleaned (environment)
Section titled “12. Environment debris found and cleaned (environment)”- Orphaned network interfaces: four in the stack-test VPC, deleted.
- A failed stack delete: a
DELETE_FAILEDheadless stack from a run on 2026-09-19, and its-iamcompanion, are still in the account. Its delete failed because the service role in the companion could no longer be assumed. It is unrelated to this release and was left in place. - A publish-time warning: applying the
EnforceSSLOnlybucket policy to the account’s private artifacts bucket was denied. The publish completed. The public release buckets were not involved.
13. Tier G ran for the first time, and the refresh’s failure rate halved (improvement)
Section titled “13. Tier G ran for the first time, and the refresh’s failure rate halved (improvement)”The browser review had never run for any release, for want of a browser connection. A headless Chromium driven by Playwright is enough. It needs no debug Chrome and no MCP server, so the tier should not be NOT RUN again. Separately, the refresh’s unsuccessful rows fell from 0.80% to 0.45%. Most of that comes from Finding 3’s reduction, and the 1,200- and 1,600-row confidence-step failures did not recur.
Issues filed and their status
Section titled “Issues filed and their status”| Issue | Finding | Status |
|---|---|---|
| #1672 — per-section Re-extract rewrites every section’s class | 1 | open |
| #1676 — packet+images window exceeds the model’s input | 2 | open |
| #1678 — separate confidence step OOM residual after #1488 | 3 | open |
| #1677 — text-only page classification fails on a blank page | 5 | open |
#1673 — config-create --features min output fails validation | 6 | open |
| #1675 — dep-audit reads gitignored lockfiles locally | 9 | open |
#1674 — --native-upload cannot validate against v0.6.13 | F2 | open |
What was NOT validated
Section titled “What was NOT validated”- Real GovCloud. Tier D’s
--govcloudrun wasREGION=us-west-2with commercial credentials. Partition ARNs, GovCloud model availability and the BDA project rejection are unproven, and the Knowledge Base was disabled in that run. - Review queues end to end in production. HITL and the Default queue are off on the
reference stack, which is the configuration a stack upgraded from v0.6.12 has. Turning them
on would have changed the stack the benchmarks were running on. So the production claim, Skip
with reason, Release, take-over, reminders, Reviewer Process Changes and
review/status.jsonwere not exercised in a browser. Test Studio queues were. - The seller positive path. No token was issued or verified.
idp_cliintegration behaviour. The suite is empty (Finding 8).- Which value of Finding 4’s varying fields is correct. There is no ground-truth baseline
for
samples/lending_package.pdfin the repository. - Whether the A/B’s one asymmetry is a release effect. Two of three v0.6.13 runs on the
bedrock_llmarm splitlongdesc_100into two sections, against none of three on v0.6.12. The runs’ OCR output already differed, so model-written OCR variation is one explanation and this release’s classification changes are the other. Replaying one stored OCR output through both releases’ classification, or more repeats on that arm, would separate them. - The classification variants on any model but Nova 2 Lite, or on more than 20 documents.
- Canada (
ca-central-1) andeu-central-1. All stacks wereus-west-2(sellerus-east-1); the other published regions were checked for HTTP 200 only. - Malware scanning.
MalwareScanningdefaulted toDISABLEDin every stack.
Reproduce
Section titled “Reproduce”# Tier A (offline), in a dedicated worktree at the tag — never in a shared checkoutgit worktree add <path> v0.6.13 && cd <path>CI=1 python3 scripts/srt/setup.py && make srt-scan # before anything writes build outputmake security-results STACK_NAME=<stack> REGION=us-west-2 SKIP_SRT=1make test && make lint-cicd && (cd src/ui && npx vitest run)make dep-audit && make typecheckpython3 scripts/coverage_all.py --jobs 1 && make check-coverage-debt # one tree at a timepython3 publish.py <bucket-base> <prefix> us-west-2 --clean-build
URL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.13.yaml
# Tier C (5 variants)make stacktest-hosting-global TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-waf TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-hosting-private TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \ VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>make stacktest-jobsapi TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \ VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>make stacktest-seller # REGION deliberately unset -> us-east-1
# Tier Dmake transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<email>
# Tier E — run several documents per side, or run-to-run variation reads as a version effectaws cloudformation create-stack --stack-name IDPUpg0612to0613 --template-url <0.6.12 URL> ...aws cloudformation update-stack --stack-name IDPUpg0612to0613 --template-url $URL \ --parameters file://<every existing parameter with UsePreviousValue=true>.json
# Tier F1, PREV side before update-stack, NEW side after, on the same stackpython benchmarks/harness/gen_corpus.py && python benchmarks/harness/make_configs.py --suite corefastpython benchmarks/harness/run_matrix.py --stack <stack> --suite corefast --repeats 3 \ --native-upload --validate-against-release v0.6.12 --max-inflight 4python benchmarks/harness/aggregate.py --compare <v0.6.13 summary> --baseline <v0.6.12 summary>
# Tier F2-F4: one lane per suite/override, 6 concurrent; regenerate configs per lane,# and for `core` per reference corpus too (make_configs --suite core --class realkie|ocr_bench)python benchmarks/harness/run_matrix.py --stack <stack> --suite <suite> [--set <axis>=<value>] \ --native-upload --validate-against-release v0.6.12 --max-inflight 4python benchmarks/harness/run_classification_bench.py --stack <stack> --testset docsplit \ --models nova2lite --variants page-both,page-text,packet-both,packet-text,packet-images,split-then-classify-packet,split-then-classify-page
# Tier G: headless Chromium via Playwright, throwaway users from scripts/ux_test_session.py# Tier H, on the upgraded stackcd lib/idp_common_pkg && make test-integrationIDP_STACK_NAME=<stack> python -m pytest -m integration lib/idp_sdk/tests/integrationRaw logs for every tier are in gitignored scratch/ and are not published.