Release Validation: v0.6.11
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
Release Validation Record — v0.6.11
Section titled “Release Validation Record — v0.6.11”Release: v0.6.11 (VERSION = 0.6.11, CHANGELOG.md section cut at 9b36aa3f1)
Validated: 2026-09-27 → 2026-09-28, us-west-2 (seller tier us-east-1), commercial partition
Commit measured: develop 3ca2dcc4d — every figure in this record was measured at that
commit and nowhere else.
Artifact under test: a self-published build of that commit —
s3://<PRIVATE_BUCKET>/<PRIVATE_PREFIX>/idp-main_0.6.11.yaml, Description (v0.6.11), built
with publish.py … --clean-build.
Validated twice: before publication against a private build, and after publication against the released object. The second validation is the current record; the first is retained below under Prerelease pass for continuity. The post-publication pass added the deploy-variant and template-transform tiers against the public template, and covers the agentic read-timeout fix that landed between the two.
Commit under test:
64e95c7d2, tree3a3c99416c455233e8ce57a6c33b2bd186937d3c— byte-identical to52372f8adondevelop, the commit the nightly GitLab SDLC pipeline built and passed 12/12 on. That pipeline’s result therefore applies to the same source, exercised from a different account and region.
Verdict
Section titled “Verdict”✅ Ship — the published artifact deploys in every shipped configuration, upgrades cleanly from 0.6.10 without changing extracted values, and shows no accuracy or cost regression. 7 of 8 tiers pass; tier G (browser UX) is NOT RUN, for want of a browser on the validation host. No finding is attributable to this release.
Tier G’s scope is the whole web UI, not one flow. git diff v0.6.10..64e95c7d2 -- src/ui/ is 28 files, +1,902 / −356: Test Studio’s sortable columns and
classification-errors panel, the Document panel’s cost calculation, the configuration
revision-history panel, and the issue-form links. The compensating coverage is the UI unit
suite — 105 files, 1,215 tests, all passing — which runs in neither make test nor
make lint-cicd and was invoked separately (make ui-test). Unit coverage is not a person
completing a flow.
The release’s own changes are error-path fixes, so the healthy-document benchmark
correctly shows no movement: the A/B over 342 runs reports 0 regressions and 0
improvements, grid cost +0.4% and wall −2.9%. A release that fixes what happens after a
transient Bedrock error should not move accuracy on documents that never hit one, and it does
not. Step 8 of the nightly SDLC suite — agentic extraction with large tables, the path
#1310/#1014 repair — passed on the byte-identical tree, which is the end-to-end
confirmation the A/B cannot give.
Eighteen findings, none introduced by 0.6.11. Four are in shipped product code and all four pre-date this release (Findings 1, 2, 11, 12). The rest are gate, test, harness or documentation issues — including three controls that exist but do not run where it matters (Finding 13), two reporting limits that make a published figure unsafe (Findings 15, 16), and a vacuous pass that a completion count hides (Finding 14). Findings 11–18 are from the published-artifact pass; 1–10 are the prerelease pass’s, retained unchanged.
Two findings from the first validation are fixed in the released code: the library
integration tier went from 15 failures of 23 selected to 1 of 19. Commit 5310684a3
added credential reconciliation to tests/conftest.py, corrected
the wrong assert "output" in response assertions rather than deleting them, and re-homed the
MagicMock test to tests/unit/. The one remaining failure reproduces identically at
v0.6.10.
Second pass — the published artifact (2026-10-09 → 2026-10-12)
Section titled “Second pass — the published artifact (2026-10-09 → 2026-10-12)”Artifact: https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.11.yaml,
LastModified 2026-10-09 21:56:34 GMT, Description (v0.6.11), HTTP 200 in all three
published regions. Commit 64e95c7d2. Account <ACCOUNT_ID>, region us-west-2
(seller tier us-east-1), commercial partition.
| Tier | Gate | Result |
|---|---|---|
| A — SRT | ✅ PASS | 0 reopened; 7 open, all B105/B106 identifier names in test files, non-blocking by the documented scope decision |
A — make test | ✅ PASS | all 68 test roots, 0 failures; the standing-failure baseline declares none |
A — make lint-cicd | ✅ PASS | all code-quality checks |
A — make dep-audit | ✅ PASS | 1,805 packages, 0 gating, 5 allowlisted, 12 below HIGH |
A — make typecheck | ✅ PASS | 0 errors / 44 warnings vs the documented develop baseline of 4 / 51 — errors down 4 |
A — make ui-test | ✅ PASS | 105 files, 1,215 tests — runs in neither make test nor make lint-cicd; see Finding 13 |
| A — build + package | ✅ PASS | publish.py --clean-build in a clean env, 4.1 min; template validation + cfn-lint clean on all 5 packaged templates (8 advisory warnings) |
| A — coverage | ⚠️ ratchet FAILS | all 9 trees measured; no tree lost coverage; make check-coverage-debt exits 1 on 3 unratcheted files — Finding 7 |
| B — security snapshot | ✅ 3 of 4 gates | RBAC static PASS (0 fail / 4 known-gap warn) · RBAC dynamic PASS (609 checks, 0 hard fail, 0 not-run, 15 skipped) · ZAP PASS (High=0) · SRT snapshot reads FAIL — Finding 6 |
| C — deploy variants | ✅ 5/5 | hosting-global, WAF, hosting-private, Jobs API, seller — each self-deployed from the published template and torn down |
| D — transforms | ✅ 2/2 | --headless and --govcloud, each processing a real document (151.7 s / 144.5 s, 17/22 fields, classified Payslip) |
| E — upgrade 0.6.10 → 0.6.11 | ✅ PASS | UPDATE_COMPLETE, 0 rollback events, Description now (v0.6.11); no change to extracted values (control-verified) |
| F1 — release A/B | ✅ PASS | 171/0 + 171/0; 0 regressions, 0 improvements, 1 inconclusive |
| F2–F4 — guidance refresh | ✅ COMPLETE | 2,628 runs, 23 scored sets, 4,794 rows — the first full refresh since v0.6.7 |
| G — browser UX | ❌ NOT RUN | no Chrome/Chromium on the host, nothing on 127.0.0.1:9222, no chrome-devtools MCP; browsermcp disconnected mid-run |
| H — integration tests | ⚠️ 1 failure | idp_common 18/19 pass; the failure reproduces at v0.6.10. idp_cli has 0 integration tests — an empty suite, not a pass |
Coverage, as measured
Section titled “Coverage, as measured”make coverage-all over all nine trees. No file lost coverage beyond the ratchet’s 1%
tolerance (two moved: config/merge_utils.py −0.41, error_analyzer/tools/stepfunction_tool.py
−0.72).
| tree | files | measured | recorded baseline |
|---|---|---|---|
| seller_entitlement | 6 | 82.03% (lowest) | 82.03% |
| scripts | 188 | 83.04% | 82.12% |
| idp_common | 228 | 83.46% | 83.25% |
| pii_anonymizer_hook | 4 | 86.98% | 86.98% |
| pii_anonymizer_api | 2 | 94.93% | 94.93% |
| main_stack_extensions | 24 | 95.64% | 95.28% |
| idp_sdk | 52 | 99.66% | 36.61% |
| idp_cli | 6 | 99.68% | 98.97% |
| idp_feature_sdk | 9 | 99.94% | 99.94% |
⚠️ idp_sdk is not the tree to watch any more, and the recorded baseline says it is.
Measured 99.66% against a recorded 36.61% — a genuine 63-point improvement, verified (52
classes in the report against 51 baselined files, 8,928 statements, 30 uncovered). The lowest
measured tree is now seller_entitlement at 82.03%. Quoting make coverage-summary here
would have reported the stale figure and cannot fail visibly.
⚠️ Two substantial trees are outside the gate entirely: src/lambda (84 files) and
nested/api-resolvers (61 files) are not among the nine in check_coverage_debt.TREES. Stale
reports dated 2026-09-24 put them at 88.14% and 59.86% — the latter would be the lowest of
any tree. Not re-measured here.
Effect of the upgrade on extracted values: none detected
Section titled “Effect of the upgrade on extracted values: none detected”Fresh stack from the published 0.6.10 template → document → update-stack with
UsePreviousValue on all 90 parameters → same document. UPDATE_COMPLETE, zero
ROLLBACK/FAILED events, and both UpdateSchemaConfig and UpdateDefaultConfig custom
resources returned SUCCESS — the step where upgrades deadlock.
Parameter-set diff: 90 both sides, 0 added, 0 removed, 0 newly required, 0 default
changes. One AllowedValues narrowing: KnowledgeBaseModelId drops the two Sonnet 4 entries.
The default is unchanged (us.amazon.nova-pro-v1:0) and this stack held it, so the rejection
the CHANGELOG warns about was not triggered; a stack explicitly pinned to Sonnet 4 is
refused before any resource changes.
Output diff: 1,934 leaf fields over 6 sections, 76 value differences — of which only 8
are extracted business data (inference_result); 33 are explainability_info, 29 metadata or
timings, 6 processing_report.
A same-version control settles those 8 as noise. Running the identical document a third
time on the already-upgraded stack gives 15 inference_result differences — more than
the upgrade produced — and flips the same Payslip tax Period fields back. So the upgrade
changed nothing detectable; the 8 sit below a measured noise floor of 15. Product observation,
present at 0.6.10 too: those Period fields hold a monetary amount whose sign is bistable
across repeat runs of identical code, and casing is unstable (JOHN / John).
Failure census over the benchmark refresh
Section titled “Failure census over the benchmark refresh”90 non-COMPLETED of 4,794 scored rows (1.88%), matching the Step Functions count exactly. The A/B itself had zero failures on both sides.
| group | n | attribution |
|---|---|---|
simple mode on scale_3200 (astracap 6, scaling 9) | 15 | By design — this is the measurement. All four astracap advanced cells completed 3/3 |
| large documents in simple mode | ~65 | Finding 11 (the 300 s read timeout) and Finding 12 |
core-tt-adv-int on med_narrow | 5 | Finding 12 |
Results
Section titled “Results”Offline tiers (no AWS) — tier A
Section titled “Offline tiers (no AWS) — tier A”All at commit 3ca2dcc4d.
| Tier | Gate | Result |
|---|---|---|
Build + package (publish.py --clean-build) | ✅ PASS | 3.7 min; template validation passed; cfn-lint clean on 5 packaged templates; Description (v0.6.11) |
Lint (make lint-cicd) | ✅ PASS | all code-quality checks passed; 351 Markdown files, every relative link resolves |
Dependency audit (make dep-audit) | ✅ PASS | 1,802 packages audited, 0 gating, 3 allowlisted, 7 below HIGH |
Type check (make typecheck) | ✅ PASS | 0 errors / 44 warnings against the documented develop baseline of 4 errors / 51 warnings — errors down 4, warnings down 7. Report the delta, not the raw count |
| SRT (SAST + deps + security matrix) | ✅ PASS | make srt-scan: “no high-priority security issues found”. See Finding 6 for why the curated snapshot disagrees |
Unit / integration suites (make test) | ⚠️ PASS with 2 host-environment failures | 2 failures, both in scripts/tests/test_ai_mr_review.py, both proven to be a property of this host and not of the tree — Finding 4. Every other test root green |
Test coverage (make coverage-all + make check-coverage-debt) | ⚠️ measured; ratchet reports 2 problems | all 9 trees measured; no file lost coverage; two new modules are unratcheted — Finding 7 |
Test coverage, as measured
Section titled “Test coverage, as measured”Measured with make coverage-all at 3ca2dcc4d. These are measurements, not the figures in
scripts/coverage_debt.json, which is a ratchet floor and always self-consistent.
| Tree | Coverage |
|---|---|
idp_feature_sdk | 99.94% |
idp_cli | 99.68% |
idp_sdk | 99.66% |
main_stack_extensions | 95.28% |
pii_anonymizer_api | 94.93% |
pii_anonymizer_hook | 86.98% |
idp_common | 83.37% |
scripts | 82.21% |
seller_entitlement | 82.03% |
seller_entitlement at 82.03% is the lowest-covered tree, with scripts at 82.21% and
idp_common at 83.37% immediately above it. There is no per-tree table in any previous record
to diff against — coverage was absent from the release-validation deliverable list until this
release — so this table is the first baseline rather than a comparison.
The one movement that can be stated: idp_sdk was 36.61% in the recorded baseline at
0.6.10 and measures 99.66% now, so the debt on that tree has been paid down since.
⚠️ The scripts figure required a second measurement run. The first reported 82.06% and the
ratchet refused to compare it, correctly, because the run behind it recorded 2 test
failures (Finding 4) and so the falls it would have reported were artefacts. Re-measured with
the cause neutralised, the tree reads 82.21% and the refusal clears. That refusal working as
designed is worth recording: a partially-failed pytest still writes a well-formed
coverage.xml, and nothing inside the report says so.
Security tiers — tier B
Section titled “Security tiers — tier B”Run against IDP0611Sec (v0.6.11, created from the artifact under test). Snapshot:
security/test-results/0.6.11/.
| Tier | Gate | Result | vs PREV |
|---|---|---|---|
| SRT (SAST + deps) | ✅ PASS | 0 open/reopened HIGH per make srt-scan | vs 0.6.9: 0 of 18,204 → 0 of 34,329 tracked. 16,125 more findings tracked, still 0 open |
| RBAC static | ✅ PASS | 0 fail · 4 known-gap warn | vs 0.6.9: 2 → 4 warn, and the movement is added granularity, not new exposure — see below |
| RBAC dynamic | ❌ FAIL per the harness | 603 checks · 552 pass · 8 hard fail · 25 could not be run · 17 skipped · 26 known-gap | identical to 0.6.10 in every set — see below |
| ZAP DAST | ✅ PASS | High = 0, Medium = 1, Low = 0 | vs 0.6.9: unchanged (Medium = 1, Cross-Domain Misconfiguration, 5 instances) |
The RBAC dynamic result is not a regression, and this is established rather than assumed.
A 0.6.10 run of the same harness was already on disk from 2026-09-25. Every outcome set
matches exactly: 552 pass, 17 skip, 8 fail, 26 warn, 25 inconclusive — and the precise
(op, principal) membership of each of those five sets is identical between the two releases.
0.6.11 introduced none of it. The apparent movement against 0.6.9’s snapshot (604 checks, 0 hard fail) is an instrumentation change: 0.6.9 reported 603 pass + 1 warn = 604 and had
no fail-or-inconclusive classification at all, so a 500 and a refusal were counted together.
RBAC static 2 → 4 warn is one new check, not two new gaps. GAP-02
(queryKnowledgeBase has no group check) and GAP-07 (the chat Function URL transport
carries no Cognito group claim) both carry over. The two extra lines are a new S9 check
reporting GAP-07 once per affected route — POST /chat/document and POST /chat/agent.
Operations covered rose 118 → 119. S2 through S5 all pass.
Deploy-variant tiers — tier C
Section titled “Deploy-variant tiers — tier C”Each self-deploys from the artifact under test, validates, and tears down.
| Variant | Gate | Result |
|---|---|---|
| APIGateway hosting GLOBAL (no VPC) | ✅ PASS | deployed, validated, torn down |
| WAF (IP allow-list, no VPC) | ✅ PASS | WebACL associated with 1 resource |
| APIGateway hosting PRIVATE (VPC) | ✅ PASS | PRIVATE REST API present with resource policy |
Seller Entitlement Service (us-east-1) | ✅ PASS | 8 assertions — unsigned request 403, unentitled 403 with no detail leaked, unknown product byte-identical to not-entitled, malformed body 400, oversized body 413 before parsing, 15 hostile payloads refused with no 5xx |
| Jobs API (VPC) | ⏸️ DEFERRED | blocked by an account service quota, not by the code — Finding 3 |
VPC used for both VPC variants: the account’s purpose-built stack-test VPC (<VPC_ID>, security
group named idp-stacktest-vpce-sg, two private subnets, an execute-api interface endpoint).
Two things the skill lists as open through v0.6.9 are confirmed fixed. The seller test’s
--region really reaches dynamic_activation_test.py — the run log prints region us-east-1
— so its three refusal assertions can no longer pass vacuously on API Gateway’s own SigV4
403 outside us-east-1. And teardown swept
/aws/apigateway/idp-seller-entitlement-citest-activation itself, so the pre-run and post-run
manual deletes are gone.
Residuals on the seller tier, both unchanged: the positive path (a token issued and verified)
still needs --entitled-profile and is not covered, and teardown leaves a KMS key
scheduled for deletion on a 7-day window that keeps billing.
Template-transform tiers — tier D
Section titled “Template-transform tiers — tier D”The only tier that deploys a transformed template.
| Variant | Gate | Checks |
|---|---|---|
--headless | ✅ PASS | ✓ UI / Cognito / CloudFront absent · ✓ processing core present · ✓ document processed (17/22 fields populated, classified Payslip, verification string found) |
--govcloud | ✅ PASS | ✓ no AWS::CloudFront::* / AWS::Lambda::Url · ✓ LWA chat-stream family removed (#677) · ✓ UI retained, hosted on API Gateway · ✓ processing core present · ✓ transformed template carries no dangling refs · ✓ document processed (143.9 s, 18/22 fields, Payslip) |
cfn-lint clean on 4 templates. Both stacks and their IAM stacks torn down.
⚠️ The
--govcloudresult is a commercial-partition result. It proves the CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves nothing about partition ARNs, GovCloud model availability or the BDA project rejection (#676/#677 were invisible to a commercial run). This run wasREGION=us-west-2with commercial credentials. The Knowledge Base was disabled in the--govcloudrun (runner default:DocumentKnowledgeBase=DISABLED,ConfigurationPreset=lending-package-sample).
In-place upgrade v0.6.10 → v0.6.11 — tier E
Section titled “In-place upgrade v0.6.10 → v0.6.11 — tier E”Fresh throwaway stack IDPUpg0610to0611: deploy published 0.6.10 → process
samples/lending_package.pdf → update-stack to the artifact under test → process the same
document → diff.
| Check | Result |
|---|---|
| Parameter-set diff | empty — 90 parameters both sides, only AdminEmail required, none added, removed or retyped |
update-stack outcome | ✅ UPDATE_COMPLETE, zero ROLLBACK or FAILED events |
| Nested stacks | all 5 UPDATE_COMPLETE (APIRESOLVERSTACK, PATTERNSTACK, MULTIDOCDISCOVERYSTACK, FeaturePlatformStack, DOCUMENTKB) |
Description after update | (v0.6.11) |
UpdateDefaultConfig custom resource | no error, validation or migration message in its Lambda log |
| Document, pre-upgrade | 1/1 success, 151.5 s |
| Document, post-upgrade | 1/1 success, 142.4 s |
⚠️ One deviation forced by validating an unpublished build: PublicArtifactsBucket and
PublicArtifactsPrefix were set explicitly on update-stack rather than reused with
UsePreviousValue=true. Reusing them would have pointed the 0.6.11 stack at
aws-ml-blog-us-west-2/artifacts/genai-idp, which does not hold this build’s assets. All 88
other parameters were reused. This does not arise for a published release.
What the upgrade changed in extracted values
Section titled “What the upgrade changed in extracted values”Baseline outputs were pulled before the update. 49 JSON files each side, none added or removed.
| Measure | Value |
|---|---|
| Leaf fields compared | 61,574 |
| Leaves differing | 5,450 |
…of which Textract Blocks[] / lines[] | 5,307 |
| Differences in extracted values and summaries | 20 |
| Null / empty leaves, pre → post | 441 → 467 (all outside inference_result) |
The 5,307 are block ids, geometry and per-block confidence, which differ between any two runs including two runs of one version. The 20 that remain fall in four groups, all consistent with model run-to-run variance rather than a version effect:
| Group | n | Example |
|---|---|---|
| Case style | 6 | Anytown → ANYTOWN, John → JOHN, 475 Any Avenue → 475 ANY AVENUE |
Sign on payslip tax Period | 6 | -40.60 → 40.60, -8.43 → 8.43 |
| Field placement | 2 | USA moved from account_holder_address.Line2 to .State — wrong in both places; that section’s null count is unchanged |
| Summary table formatting | 6 | | Field | Value | vs | Attribute | Value |, row order, bolding |
⚠️ One run per side cannot separate a version effect from run-to-run variance, and this diff is reported rather than leaned on. What supports “no regression” is the blast radius — a simple-mode, unthrottled run enters neither code path 0.6.11 changed — together with the benchmark A/B below, which is the same question at three repeats.
The sign flip on six tax-deduction fields is worth a look on its own merits, independent of this release: whether a deduction is emitted negative or positive is not something a consumer should have to guess per run.
Benchmarks — tier F
Section titled “Benchmarks — tier F”Reference stack: IDPUpg0610to0611 at v0.6.11 (the upgraded stack from tier E). Configs were
generated once and not regenerated between sides, and --native-upload bypassed
idp-cli’s migration on both, so the two sides ran byte-identical configuration.
F1 — release A/B, published 0.6.10 → the 0.6.11 build
Section titled “F1 — release A/B, published 0.6.10 → the 0.6.11 build”corefast, 19 cells × 3 documents × 3 repeats = 171 runs per side, 342 total.
| Result | |
|---|---|
| Regressions | 0 |
| Improvements | 0 |
| Cell-level regressions | 0 |
| Cell-level improvements | 0 |
| Inconclusive | 3 |
| Failure census | 171/171 COMPLETED on both sides, zero non-COMPLETED rows either side |
All 171 rows on both sides carry cells_compared, so the “a null column scores 1.000” blind
spot does not apply to this comparison.
All three inconclusive items are on core-llm-simple-sep — the bedrock_llm OCR cell — and
all sit inside run-to-run spread:
| Metric | Movement | Spread |
|---|---|---|
| cost | −35% (0.143 ± 0.145 n9 → 0.093 ± 0.063 n9) | within noise |
| accuracy | +0.278 (0.667 → 0.944) | paired per-doc spread ±0.441 |
| recall | +0.042 (0.958 → 1.000) | paired per-doc spread ±0.127 |
⚠️ Two reading caveats. The two sides ran on different stacks — the 0.6.10 side on an
existing v0.6.10 stack, reused rather than newly deployed to save ~122 IAM roles — so
wall-time deltas between the sides are not comparable; accuracy, completeness and cost
are, because the configuration was identical. And corefast cost CV is 0.50–1.02 across all
19 cells at n=9, which is variance across the three different documents inside each cell
rather than run-to-run noise, so cell-level cost comparison on this suite is not reliable.
F2 / F3 — guidance-paper refresh: ⚠️ NOT COMPLETED
Section titled “F2 / F3 — guidance-paper refresh: ⚠️ NOT COMPLETED”This is an incomplete tier F and the record says so. The full refresh was planned at 2,148 runs (plan retained in the run’s scratch directory), and the run was stopped after F1 and one F2 lane. What ran:
| Lane | Planned | Result |
|---|---|---|
intconf (§2.1/§6 standing hazard) | 8 | ✅ complete and scored |
core (§2 matrix + real-world) | 90 | ⏸️ 20 launched, stopped, not scored |
scaling (§3) | 28 | ⏸️ 14 launched, stopped, not scored |
cost (§4) | 25 | ⏸️ 14 launched, stopped, not scored |
advverify, the 7-model extraction sweep, astravalue, astracap, opus55value, the classification and confidence sweeps | 1,997 | ❌ not started |
The three partial lanes are deliberately not scored. aggregate.py will happily score a
partial run and write a summary that looks complete; leaving them unscored is the only way a
later reader can tell.
intconf reads clean, and that is the one positive benchmark finding here. 8/8
COMPLETED, completeness_recall 1.000 and cell_accuracy 1.000 on every row,
cells_compared = 200 per row, and — the measure that matters for this hazard — 0 of 100
confidence rows unscored in both the integrated and the separate arm. The
integrated-confidence truncation hazard reads closed at 0.6.11 on that document.
Every paper under docs/benchmarking/ therefore still carries data from an earlier
release, by name: config-guidance.md, index.md, classification-confidence.md,
prompt-caching.md and feature-multi-instance.md. None was regenerated.
One thing that would not have changed had the refresh completed: 0.6.11 changed no configuration knob. The §7 knob A/Bs had nothing to re-measure, and running them would have measured nothing about this release.
Library integration tests — tier H
Section titled “Library integration tests — tier H”Run against IDP0611Sec at v0.6.11. 15 failed, 8 passed, 0 skipped of 23 selected.
None is attributable to 0.6.11 — the blast radius touches no file any of them exercises. They have four distinct causes, and only one is a product defect:
| Cause | Tests | Finding |
|---|---|---|
base-rule-discovery.yaml defaults produce rules the production validator rejects | 1 | Finding 2 — product |
assert "output" in response contradicts invoke_model’s documented return shape | 6 | Finding 8 — test |
Integration-marked tests under tests/unit/ never get real credentials | 6 | Finding 9 — test |
A MagicMock config leaks into a real code path as model_id | 2 | Finding 9 — test |
Browser UX review — tier G: ❌ NOT RUN
Section titled “Browser UX review — tier G: ❌ NOT RUN”No browser exists on the validation host: no google-chrome or chromium binary, DISPLAY
unset, nothing listening on 127.0.0.1:9222, and the chrome-devtools MCP tools were not
available to the session. The skill also requires a one-time interactive sign-in in a
persistent Chrome profile, which cannot be done unattended.
It is not reported as passing and not dropped. Its scope would have been flow 1
(sign-in → upload → results) only, because git diff v0.6.10..3ca2dcc4d -- src/ui/ is
empty — 0.6.11 ships no UI change, so the web UI it delivers is the one 0.6.10 delivered.
That bounds the risk; it does not substitute for the review. Adjacent coverage that did run:
the UI vitest suite (tier A) and ZAP DAST against the UI host (tier B, High = 0).
Findings
Section titled “Findings”Ordered by how much they matter. None was introduced by 0.6.11.
1. The TestSet resolver returns 500 where 404 or 400 is correct (product defect, pre-existing)
Section titled “1. The TestSet resolver returns 500 where 404 or 400 is correct (product defect, pre-existing)”nested/api-resolvers/src/lambda/test_set_resolver/index.py raises bare Exception and
KeyError for not-found and missing-argument conditions, which the dispatcher renders as HTTP
500:
Exception: Test set '__nope__' not found → should be 404, is 500KeyError: 'testSetId' / 'id' / 'fileName' → should be 400, is 500The security consequence is not an authorization hole but a hole in what the authorization
tests can prove: 25 of the 460 positive authorization assertions are unproven rather than
passed, because a 500 establishes nothing about the authorization decision that would have
been reached. Affected operations: getTestSetDocuments, getAnnotationQueue,
getDraftLabelJob, estimateReviewEffort, openTestSetAnnotationDraft,
updateTestSetDocumentSections, reextractTestSetDocument, resetTestSetLabels,
clearDraftLabels, removeDocumentsFromTestSet, createEmptyTestSet,
addDocumentsToTestSet, getFeatureLaunchUrl, subscribeFeature, unsubscribeFeature.
This is the same class of change 0.6.10 already made deliberately elsewhere — moving
getFileContents/getFilePresignedUrl off 500 and onto 403/400/400 — applied to the
instance and not the class.
2. Default rule-discovery settings produce rules the validator rejects (product defect, pre-existing)
Section titled “2. Default rule-discovery settings produce rules the validator rejects (product defect, pre-existing)”test_rules_discovery_with_defaults_against_medicare_sample makes a real Bedrock call
(global.anthropic.claude-sonnet-4-6) against samples/rule-validation/medicare_respiratory_pa_packet.pdf,
gets status: SUCCESS with at least one rule class, and those rules then fail the
production validator RulesDiscovery._validate_rules_response:
Rule class 0: Missing required field: x-aws-idp-rule-typeSo config_library’s base-rule-discovery.yaml defaults, on a real document, produce output
the code that consumes it refuses. Reproduced on an independent second run (95.8 s), so it
is not model non-determinism. Nothing in discovery/ or config_library/ changed in 0.6.11.
A second observation from the same run, minor but in the same defaults: Failed to convert max_tokens value '' to int. Not using max_tokens. — the rule-discovery configuration carries
an empty max_tokens.
3. Orphaned CloudFront policies accumulate until they block deploys (product defect, pre-existing)
Section titled “3. Orphaned CloudFront policies accumulate until they block deploys (product defect, pre-existing)”Tier C’s Jobs API variant rolled back on:
Limit exceeded for resource of type 'AWS::CloudFront::ResponseHeadersPolicy'.Reason: Account's Response Headers Policies limit reached. … ServiceLimitExceededThe quota is L-CF0D4FC5, 20 per account, adjustable, and the account sits at exactly 20.
(The MaxItems: 100 in the list-response-headers-policies reply is the page size, not the
quota — easy to misread as headroom.) Thirteen of the twenty are IDP
<stack>-security-headers-policy entries, several belonging to stacks long deleted. The
variants that passed all set WebUIHosting=APIGateway and create no CloudFront distribution;
Jobs API uses the default CloudFront hosting and so hit the wall.
Why they accumulate — lib/idp_sdk/idp_sdk/_core/stack.py:2199 calls
cloudfront_client.delete_response_headers_policy(Id=policy_id)with no IfMatch=. CloudFront requires the current ETag there, so every call fails
InvalidIfMatchVersion and the failure is swallowed into a logger.warning. That teardown
path has therefore never deleted a single orphaned policy; one run logged twelve such
warnings. The sibling implementation is correct —
lib/idp_sdk/idp_sdk/_core/cleanup_orphaned.py:880-889 does
get_response_headers_policy → read ETag → delete_response_headers_policy(Id, IfMatch=etag).
And the test cannot catch it. The teardown suite’s fake is
tests/unit/test_stack_deployer_teardown.py:2110, "delete_response_headers_policy": lambda Id, **_k:
— a signature that accepts the missing IfMatch silently. The orphaned-cleanup suite’s fake
declares delete_response_headers_policy(self, Id: str, IfMatch: str) and would have failed.
A quota increase to 100 was requested rather than deleting policies, because three of the twenty belong to unrelated stacks in the same account.
4. Two tests fail on a host with a managed git hooks path (test tooling)
Section titled “4. Two tests fail on a host with a managed git hooks path (test tooling)”scripts/tests/test_ai_mr_review.py::test_the_instruction_file_set_is_derived_not_authored and
::test_the_review_criteria_come_from_the_target_branch_not_the_mr both build a throwaway
repository and commit to it as t@example.invalid. Where /etc/gitconfig sets a system-wide
core.hooksPath pointing at a corporate commit-scanning tool, that tool aborts any commit
whose author email is not the developer’s registered one, so git commit exits 1 and the
fixture raises CalledProcessError.
Proven rather than inferred: GIT_CONFIG_NOSYSTEM=1 pytest scripts/tests/test_ai_mr_review.py
gives 38 passed, 1 skipped. The fixture should commit with git -c core.hooksPath= commit.
This is the same hazard CLAUDE.md already documents for make install-git-hooks, which
deliberately writes to --git-common-dir rather than honouring core.hooksPath for exactly
this reason — again, handled at one site and not the other.
5. The coverage ratchet checks only one of nine source trees in CI (gate tooling)
Section titled “5. The coverage ratchet checks only one of nine source trees in CI (gate tooling)”make check-coverage-debt runs in both CI configurations, immediately after
make test-cicd -C lib/idp_common_pkg. That step writes only the idp_common coverage
report. The other eight trees have no report, are correctly reported as “not checked”, and
that is not a failure as long as something was checked — which is documented behaviour
(#1190). Neither CI passes --require-tree, and nothing in either CI runs coverage-all.
So the ratchet is green in CI while reading one ninth of what it is written to read. Finding 7 is what that permits in practice.
6. The curated SRT result disagrees with the SRT gate (gate tooling)
Section titled “6. The curated SRT result disagrees with the SRT gate (gate tooling)”The published snapshot reads SRT — FAIL ❌ (5 open/reopened HIGH) for a release whose
security gate is green. The five are exactly Bandit B105 ×2 and B106 ×3 — the checks that
match an identifier’s name against a password wordlist — in test code that no deployment
artifact is built from. make srt-scan carries that scope decision;
scripts/security/curate_results.py implements no equivalent.
This record reports the gate’s verdict (PASS) and notes the snapshot’s disagreement. This is a known open issue, already described in the release-validation skill; it recurred exactly as written.
7. Two new modules have no coverage floor (coverage)
Section titled “7. Two new modules have no coverage floor (coverage)”idp_cli/parameters.py and idp_feature_sdk/parameters.py are measured but absent from
scripts/coverage_debt.json, so their coverage is unratcheted. Both arrived in 9bcdc0d60 on
2026-09-24, three days before this run, and passed through seven green CI runs — because of
Finding 5, neither tree is read by the ratchet in CI.
No file in any of the nine trees lost coverage.
8. Six integration tests contradict the API’s documented return shape (test tooling)
Section titled “8. Six integration tests contradict the API’s documented return shape (test tooling)”Every test in tests/integration/test_bedrock_service_tier_integration.py asserts
"output" in response, but BedrockClient.invoke_model documents and returns
{"response": …, "metering": …} — output lives at response["response"]["output"]. The
Bedrock calls all succeeded (HTTP 200, tokens metered); only the assertion is wrong.
It became visible only in the 0.6.10 cycle: 4ae05dfab (#1129, 2026-09-23) moved the
assertion out of a try/except Exception that had been converting the AssertionError into a
pytest.skip — which reads as green. So #1129 did its job, and this is the residue it
exposed.
9. The integration tests cannot run as documented (test tooling)
Section titled “9. The integration tests cannot run as documented (test tooling)”make test-integration is pytest -m "integration" over the whole tree, but only
tests/integration/conftest.py reconciles the sentinel credentials that
tests/conftest.py injects (os.environ.setdefault("AWS_ACCESS_KEY_ID", "testing")).
Integration-marked tests living under tests/unit/ therefore run with the sentinel key and
cannot reach AWS — InvalidClientTokenId, InvalidAccessKeyId. Two of the seven files
carrying @pytest.mark.integration are under tests/unit/, accounting for 6 of the 15
failures.
Separately, tests/integration/test_discovery_config_integration.py (2 failures) passes a
MagicMock configuration into a real code path, so model_id arrives as the repr of a mock
object and the production guard correctly rejects it; it also names a nonexistent
test-bucket.
10. The release-validation skill lists outdated classification models (documentation)
Section titled “10. The release-validation skill lists outdated classification models (documentation)”The skill’s F3 table names three values for the classification_model axis (nova_2_lite,
sonnet5, haiku45). The live axis in benchmarks/matrices/config_matrix.yaml holds two
— haiku45 is not in it. The skill instructs the reader to read the axis rather than trust
the copy, which is what was done here, so the stale copy cost nothing this time; it is exactly
the failure mode the skill warns about for the extraction_model axis.
11. A whole-document request over about 300 seconds always fails (product defect, pre-existing)
Section titled “11. A whole-document request over about 300 seconds always fails (product defect, pre-existing)”Non-streaming converse runs with read_timeout = CONFIDENCE_READ_TIMEOUT_SECONDS = 300.0
and DEFAULT_MAX_RETRIES = 7, inside a 900 s Lambda, and nothing checks the remaining
invocation before starting an attempt — grep finds no pre-attempt budget check in
bedrock/client.py, and the deadline logic that does exist says in its own comment that it
“CLAMPS a sleep; it never converts one into a failure”. Measured: attempt 1 dies at ~300 s,
attempt 2 at ~606 s, attempt 3 starts at +610.8 s with 289 s left and is killed at 900 s;
Step Functions retries once and the ProcessSections map discards the document.
Consequence on the model axis: Claude Opus 5.5 cannot complete an 800-row document in
simple mode at all (0 of 13 cells), while Sonnet 5 and Sonnet 5 :1m completed all 13 of
the same cells in the same window on the same stack — so this is a per-request limit, not
contention. Advanced (sharded) mode is a complete remedy: all six swept models return
800/800 rows, 6/6.
Pre-existing, not 0.6.11. The constant is unchanged at v0.6.10; 0.6.11 raised the
streamed agentic read timeout to 600 s and added a ladder guard on that path only.
12. Integrated confidence in advanced mode fails with Sonnet 4.6 on a 10-page document (product defect, pre-existing)
Section titled “12. Integrated confidence in advanced mode fails with Sonnet 4.6 on a 10-page document (product defect, pre-existing)”cost / core-tt-adv-int (advanced + integrated, extraction sonnet46): 5 of 5
attempts failed on a 10-page document, wall 1014.8–1021.6 s (spread ±7 s — a fixed
structure, not model latency). The same cell with assessment: separate completes in
111–127 s; the same cell at extraction_model: sonnet5 completed 25/25.
From ShardRuntimeFunction:
tool_name=<provide_field_assessment> | replacing with error message due to max_tokens truncation.MaxTokensReachedException: Agent has reached an unrecoverable state due to max_tokens limitExtraction succeeds first — the same log records Extraction complete for shard 1 (pages 1-5): 239 transactions captured ... All data validated successfully. What exceeds the
budget is the per-field assessment tool output, emitted inline in the same agent loop, on
shards of ~240 rows. Sonnet 5 survives because its output budget is larger.
Related and separately measured: 26 Runtime.OutOfMemory failures in AssessmentFunction,
killed at its 4,096 MB ceiling where every successful assessment peaks at 267–318 MB. One
invocation issued 55 Bedrock calls at ~80k input tokens each after a single
12-row → 6+6 split.
13. Three gaps in where tests and coverage checks run (gate tooling)
Section titled “13. Three gaps in where tests and coverage checks run (gate tooling)”- The UI vitest suite runs in neither
make testnormake lint-cicd. It is a separatemake ui-testtarget (Makefile:1181). It passes — 105 files, 1,215 tests — but only because this pass invoked it by hand, and it is the only coverage in this battery for the 28 changed UI files. make test-integrationpins no first-partyPYTHONPATH. It is a bare$(PYTHON) -m pytest -m "integration", not$(PYTEST_HERMETIC)— correctly, sinceHERMETIC_AWSstrips the credentials integration tests need, but it loses theFIRST_PARTY_PYTHONPATHpin with it, so the run can measure another checkout. Still true ondevelopat the time of writing.- The coverage ratchet’s three unratcheted files are still unratcheted on
develop, 178 commits later. The CI half that would make it fail (#1292) is open.
14. The forced-tool test passes without testing anything for Opus 5.5 (test tooling)
Section titled “14. The forced-tool test passes without testing anything for Opus 5.5 (test tooling)”Its force-on cells reach COMPLETED, which is why a completion count must not be read as a
result: forced_tool_attempted = 0, forced_tool_honored_rate = None, and
forced_tool_skips carries Bedrock’s refusal verbatim — tool_choice: type "tool" and "any" are not supported for this model. The cells ran with toolChoice: auto. Every other arm
honoured forcing at rate 1.00 (core 224/224 across 138 cells; all eleven sweep arms 7/7 or
8/8), and that identity is what makes the axis’s neutrality finding meaningful at all. v0.6.11
upgrades this from a model-card claim to an observed API refusal.
15. Two cost figures in the guidance are lower bounds, not measurements (reporting)
Section titled “15. Two cost figures in the guidance are lower bounds, not measurements (reporting)”pricing.yaml holds one flat rate per model and cannot express a usage-dependent band, but
two swept models have one: Haiku 5.5 is billed 5× on every token type above 100,000 input
tokens, and GPT-6 Astra moves to a long-context band above 272,000. Both are crossed in
practice — 105 of Haiku 5.5’s 129 completed runs exceed 100,000 input tokens (median
543,675, max 10.96M). At the full 5× its $0.149/doc becomes ~$0.77 against Sonnet 5’s $0.571,
so the sign of the comparison is unresolved. No cost ratio for either model is published;
both are labelled in-table. Closing it needs per-request token capture in the harness or a
two-band pricing.yaml entry. Same class as the :1m premium that once inflated an arm from
$68.82 to $125.42 (#899).
16. Accuracy for the real-document corpora was recorded but not reported (benchmark harness)
Section titled “16. Accuracy for the real-document corpora was recorded but not reported (benchmark harness)”All 2,280 corpus rows (realkie 1,140, ocr_bench 1,140, 20 documents each at 3
repeats) reached COMPLETED with 0 failures, and carry cost ($0.1932 and $0.0395 per
document), latency and accuracy: weighted_accuracy is populated on 2,280 of 2,280
rows (realkie mean 0.796, ocr_bench 0.996) with class_accuracy 1.000, scored
by the product’s own evaluation against the corpus labels.
What is 0 and nan is the exact-key family — cells_compared, cell_accuracy,
scalar_accuracy, completeness_recall — and it does not apply to these rows at all. A
reference corpus is a labelled test set on the stack, so there is no local per-cell truth
file to compare against; aggregate.score_reference_run hands analyze.score_doc no
truth, and the row comes from score_reference rather than score_synthetic. The two
scorers emit disjoint families, and the absence is symmetric: on this same grid the 380
completed synthetic rows carry no weighted_accuracy either, by the identical mechanism.
So the nan is a property of which column was read, not of the runs.
The defect was that nothing said so. The row recorded neither which scorer produced it nor
which column held its accuracy, the four exact-key columns were simply absent from the CSV,
and the notice write_summary prints for exactly this case was gated on
docs_missing_truth — which run_matrix populates only with synthetic documents whose
local truth file is missing, so it never fired for a reference corpus. Closed by
#1364:
every scored row now carries scored_against and, where the exact-key family does not
apply, exact_truth_unavailable naming the substitute columns; cells_compared reaches
the CSV beside cell_accuracy; cell_stats carries the per-scorer denominators; and
aggregate.py --reference-tables renders the per-corpus per-cell table from a committed
summary.
config-guidance.md §2c keeps the v0.6.8 two-model pair, which is a deliberate
trade rather than a consequence of this: v0.6.11 ran the corpora at the control only, so
refreshing one arm would turn that section’s model comparison into a cross-release one. The
v0.6.11 control figures are quoted in §2c with the command that renders them.
17. This corpus cannot produce typed-field accuracy (benchmark harness)
Section titled “17. This corpus cannot produce typed-field accuracy (benchmark harness)”typed_accuracy is None on all 4,704 completed rows in every scored set.
analyze.py populates it “only for truth files that declare fields_typed”, and of the 24
truth files in benchmarks/corpus/docs exactly one does (kv_form.pdf, 8 fields) — every
other carries rows_typed, which feeds cell_accuracy. The F3 model table therefore omits
the typed-accuracy column with that reason stated, rather than printing blanks that read as
zero. cell_accuracy is the per-value metric here and is populated for all 23 sets (~62,000
compared cells per arm).
18. Four outdated claims in the benchmarking documentation (documentation)
Section titled “18. Four outdated claims in the benchmarking documentation (documentation)”None is a product defect. All four are corrected in the accompanying benchmark PR:
config-guidance.mdwas headed v0.6.9 and attributed §2–§4 and §7 to “Sonnet 5 (the shipped default)”. Those sections are measured at Sonnet 4.6, the cross-version control — so every number in them was attributed to the wrong model.index.mdlisted the classification axis as four models including Haiku 4.5; the live axis is three andconfig_matrix.yamlstates Haiku 4.5 is deliberately not swept.- Both Opus 5.5 and Haiku 5.5 were marked ❌ not yet measured.
- The release-validation skill’s own copy of
sweeps.extraction_modellists 7 models; the axis holds 8. A sweep driven from the copy runs one model short. Finding 10 is a separate instance of the same drift, on a different axis.
index.md now states the rule that prevents it: read the axis in config_matrix.yaml,
not a prose list of it.
What was NOT validated
Section titled “What was NOT validated”Six gaps the prerelease pass listed are now closed: the published artifact, the Jobs API
variant, the guidance-paper refresh (2,628 runs), the cost suite, the A/B wall-time
comparison (both sides ran alone on the same stack), and dd0b7ecdf.
The following remain unvalidated:
- The web UI, in a browser. Tier G did not run on either pass. No flow was driven by a person or an agent. This matters more than the prerelease pass concluded, because 28 UI files changed between the two passes — Test Studio sorting and its classification-errors panel, the Document panel’s cost calculation, the configuration revision-history panel, and the issue-form links. Adjacent coverage that did run: the UI vitest suite (1,215 tests), ZAP DAST against the UI host (High = 0), and the nightly SDLC suite’s Test Studio evaluation step. None of those is a person completing a flow.
- Real GovCloud. The
--govcloudtier ran inus-west-2on the commercial partition with commercial credentials. It proves the CloudFront-free, API-Gateway-hosted template deploys and processes a document. It proves nothing about partition ARNs, GovCloud model availability or the BDA project rejection. The Knowledge Base was disabled in that run (DocumentKnowledgeBase=DISABLED, the runner default). - Real-corpus accuracy at the shipped default. The 2,280 reference-corpus runs are
scored (
weighted_accuracy0.796 and 0.996, Finding 16), but they ran at the cross-version control model only.config-guidance.md§2c therefore keeps the v0.6.8 two-model pair, because refreshing one arm of it would compare releases rather than models. - Typed accuracy. Unobtainable from this corpus — one truth file of 24 declares
fields_typed(Finding 17). - Confidence discrimination. Not estimable at this corpus’s 0.08–0.10% error rate; the
harness marks
calibration_reliablefalse on 104 of 129 runs per arm. The AUROC column is omitted from §5.4 rather than quoted. idp_cliintegration tests. Zero exist — 1,347 collected, 1,347 deselected. An empty suite, not a pass.- The seller positive path. A token issued and verified needs
--entitled-profile. - Coverage of
src/lambdaandnested/api-resolvers(145 files between them). Outside the nine-tree coverage gate; not measured here. - An upgrade from a stack pinned to Sonnet 4. The
KnowledgeBaseModelIdrejection the CHANGELOG documents was not triggered, because the test stack held the default. Verified statically from the template diff, not by a failed update. - Whether advanced mode rescues the assessment OOM (Finding 12’s second half) and whether the integrated-confidence failure reproduces at other document sizes.
Reproduce
Section titled “Reproduce”# Tier A (offline), at the measured commitgit checkout 3ca2dcc4dCI=1 python scripts/srt/setup.py && make srt-scanmake lint-cicd dep-audit typecheckmake test # see Finding 4 re: GIT_CONFIG_NOSYSTEM=1make coverage-all && make check-coverage-debtpython3 publish.py <bucket-base> <prefix> us-west-2 --clean-build
# Tier B, against a stack at v0.6.11make security-results STACK_NAME=<stack> REGION=us-west-2 SKIP_SRT=1
# Tier C / Dmake stacktest-hosting-global TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-waf TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-hosting-private TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…make stacktest-jobsapi TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…make stacktest-seller # REGION deliberately unsetmake transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<email>
# Tier E — note the two artifact parameters must be set explicitly for a private buildaws cloudformation create-stack --stack-name IDPUpg0610to0611 --template-url <0.6.10 public url> …idp-cli run-inference --stack-name IDPUpg0610to0611 --dir samples/ --file-pattern lending_package.pdf --batch-prefix pre-upgrade --monitoraws cloudformation update-stack --stack-name IDPUpg0610to0611 --template-url <0.6.11 url> --parameters file://params.json …
# Tier F1cd benchmarkspython3 harness/make_configs.py --suite corefastpython3 harness/run_matrix.py --stack <0.6.10 stack> --suite corefast --native-upload --max-inflight 4python3 harness/run_matrix.py --stack <0.6.11 stack> --suite corefast --native-upload --max-inflight 4python3 harness/aggregate.py --run results/run-<prev> --out results/v0.6.11/corefast-prev0610python3 harness/aggregate.py --run results/run-<new> --out results/v0.6.11/corefastpython3 harness/aggregate.py --compare results/v0.6.11/corefast/summary.json \ --baseline results/v0.6.11/corefast-prev0610/summary.json
# Tier Hcd lib/idp_common_pkg && make test-integration # see Finding 9