Skip to content

Release Validation: v0.6.11

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

Release: v0.6.11 (VERSION = 0.6.11, CHANGELOG.md section cut at 9b36aa3f1) Validated: 2026-09-27 → 2026-09-28, us-west-2 (seller tier us-east-1), commercial partition Commit measured: develop 3ca2dcc4d — every figure in this record was measured at that commit and nowhere else. Artifact under test: a self-published build of that commit — s3://<PRIVATE_BUCKET>/<PRIVATE_PREFIX>/idp-main_0.6.11.yaml, Description (v0.6.11), built with publish.py … --clean-build.

Validated twice: before publication against a private build, and after publication against the released object. The second validation is the current record; the first is retained below under Prerelease pass for continuity. The post-publication pass added the deploy-variant and template-transform tiers against the public template, and covers the agentic read-timeout fix that landed between the two.

Commit under test: 64e95c7d2, tree 3a3c99416c455233e8ce57a6c33b2bd186937d3c — byte-identical to 52372f8ad on develop, the commit the nightly GitLab SDLC pipeline built and passed 12/12 on. That pipeline’s result therefore applies to the same source, exercised from a different account and region.

✅ Ship — the published artifact deploys in every shipped configuration, upgrades cleanly from 0.6.10 without changing extracted values, and shows no accuracy or cost regression. 7 of 8 tiers pass; tier G (browser UX) is NOT RUN, for want of a browser on the validation host. No finding is attributable to this release.

Tier G’s scope is the whole web UI, not one flow. git diff v0.6.10..64e95c7d2 -- src/ui/ is 28 files, +1,902 / −356: Test Studio’s sortable columns and classification-errors panel, the Document panel’s cost calculation, the configuration revision-history panel, and the issue-form links. The compensating coverage is the UI unit suite — 105 files, 1,215 tests, all passing — which runs in neither make test nor make lint-cicd and was invoked separately (make ui-test). Unit coverage is not a person completing a flow.

The release’s own changes are error-path fixes, so the healthy-document benchmark correctly shows no movement: the A/B over 342 runs reports 0 regressions and 0 improvements, grid cost +0.4% and wall −2.9%. A release that fixes what happens after a transient Bedrock error should not move accuracy on documents that never hit one, and it does not. Step 8 of the nightly SDLC suite — agentic extraction with large tables, the path #1310/#1014 repair — passed on the byte-identical tree, which is the end-to-end confirmation the A/B cannot give.

Eighteen findings, none introduced by 0.6.11. Four are in shipped product code and all four pre-date this release (Findings 1, 2, 11, 12). The rest are gate, test, harness or documentation issues — including three controls that exist but do not run where it matters (Finding 13), two reporting limits that make a published figure unsafe (Findings 15, 16), and a vacuous pass that a completion count hides (Finding 14). Findings 11–18 are from the published-artifact pass; 1–10 are the prerelease pass’s, retained unchanged.

Two findings from the first validation are fixed in the released code: the library integration tier went from 15 failures of 23 selected to 1 of 19. Commit 5310684a3 added credential reconciliation to tests/conftest.py, corrected the wrong assert "output" in response assertions rather than deleting them, and re-homed the MagicMock test to tests/unit/. The one remaining failure reproduces identically at v0.6.10.


Second pass — the published artifact (2026-10-09 → 2026-10-12)

Section titled “Second pass — the published artifact (2026-10-09 → 2026-10-12)”

Artifact: https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.11.yaml, LastModified 2026-10-09 21:56:34 GMT, Description (v0.6.11), HTTP 200 in all three published regions. Commit 64e95c7d2. Account <ACCOUNT_ID>, region us-west-2 (seller tier us-east-1), commercial partition.

TierGateResult
A — SRT✅ PASS0 reopened; 7 open, all B105/B106 identifier names in test files, non-blocking by the documented scope decision
A — make test✅ PASSall 68 test roots, 0 failures; the standing-failure baseline declares none
A — make lint-cicd✅ PASSall code-quality checks
A — make dep-audit✅ PASS1,805 packages, 0 gating, 5 allowlisted, 12 below HIGH
A — make typecheck✅ PASS0 errors / 44 warnings vs the documented develop baseline of 4 / 51 — errors down 4
A — make ui-test✅ PASS105 files, 1,215 tests — runs in neither make test nor make lint-cicd; see Finding 13
A — build + package✅ PASSpublish.py --clean-build in a clean env, 4.1 min; template validation + cfn-lint clean on all 5 packaged templates (8 advisory warnings)
A — coverage⚠️ ratchet FAILSall 9 trees measured; no tree lost coverage; make check-coverage-debt exits 1 on 3 unratcheted files — Finding 7
B — security snapshot✅ 3 of 4 gatesRBAC static PASS (0 fail / 4 known-gap warn) · RBAC dynamic PASS (609 checks, 0 hard fail, 0 not-run, 15 skipped) · ZAP PASS (High=0) · SRT snapshot reads FAIL — Finding 6
C — deploy variants✅ 5/5hosting-global, WAF, hosting-private, Jobs API, seller — each self-deployed from the published template and torn down
D — transforms✅ 2/2--headless and --govcloud, each processing a real document (151.7 s / 144.5 s, 17/22 fields, classified Payslip)
E — upgrade 0.6.10 → 0.6.11✅ PASSUPDATE_COMPLETE, 0 rollback events, Description now (v0.6.11); no change to extracted values (control-verified)
F1 — release A/B✅ PASS171/0 + 171/0; 0 regressions, 0 improvements, 1 inconclusive
F2–F4 — guidance refresh✅ COMPLETE2,628 runs, 23 scored sets, 4,794 rows — the first full refresh since v0.6.7
G — browser UX❌ NOT RUNno Chrome/Chromium on the host, nothing on 127.0.0.1:9222, no chrome-devtools MCP; browsermcp disconnected mid-run
H — integration tests⚠️ 1 failureidp_common 18/19 pass; the failure reproduces at v0.6.10. idp_cli has 0 integration tests — an empty suite, not a pass

make coverage-all over all nine trees. No file lost coverage beyond the ratchet’s 1% tolerance (two moved: config/merge_utils.py −0.41, error_analyzer/tools/stepfunction_tool.py −0.72).

treefilesmeasuredrecorded baseline
seller_entitlement682.03% (lowest)82.03%
scripts18883.04%82.12%
idp_common22883.46%83.25%
pii_anonymizer_hook486.98%86.98%
pii_anonymizer_api294.93%94.93%
main_stack_extensions2495.64%95.28%
idp_sdk5299.66%36.61%
idp_cli699.68%98.97%
idp_feature_sdk999.94%99.94%

⚠️ idp_sdk is not the tree to watch any more, and the recorded baseline says it is. Measured 99.66% against a recorded 36.61% — a genuine 63-point improvement, verified (52 classes in the report against 51 baselined files, 8,928 statements, 30 uncovered). The lowest measured tree is now seller_entitlement at 82.03%. Quoting make coverage-summary here would have reported the stale figure and cannot fail visibly.

⚠️ Two substantial trees are outside the gate entirely: src/lambda (84 files) and nested/api-resolvers (61 files) are not among the nine in check_coverage_debt.TREES. Stale reports dated 2026-09-24 put them at 88.14% and 59.86% — the latter would be the lowest of any tree. Not re-measured here.

Effect of the upgrade on extracted values: none detected

Section titled “Effect of the upgrade on extracted values: none detected”

Fresh stack from the published 0.6.10 template → document → update-stack with UsePreviousValue on all 90 parameters → same document. UPDATE_COMPLETE, zero ROLLBACK/FAILED events, and both UpdateSchemaConfig and UpdateDefaultConfig custom resources returned SUCCESS — the step where upgrades deadlock.

Parameter-set diff: 90 both sides, 0 added, 0 removed, 0 newly required, 0 default changes. One AllowedValues narrowing: KnowledgeBaseModelId drops the two Sonnet 4 entries. The default is unchanged (us.amazon.nova-pro-v1:0) and this stack held it, so the rejection the CHANGELOG warns about was not triggered; a stack explicitly pinned to Sonnet 4 is refused before any resource changes.

Output diff: 1,934 leaf fields over 6 sections, 76 value differences — of which only 8 are extracted business data (inference_result); 33 are explainability_info, 29 metadata or timings, 6 processing_report.

A same-version control settles those 8 as noise. Running the identical document a third time on the already-upgraded stack gives 15 inference_result differences — more than the upgrade produced — and flips the same Payslip tax Period fields back. So the upgrade changed nothing detectable; the 8 sit below a measured noise floor of 15. Product observation, present at 0.6.10 too: those Period fields hold a monetary amount whose sign is bistable across repeat runs of identical code, and casing is unstable (JOHN / John).

90 non-COMPLETED of 4,794 scored rows (1.88%), matching the Step Functions count exactly. The A/B itself had zero failures on both sides.

groupnattribution
simple mode on scale_3200 (astracap 6, scaling 9)15By design — this is the measurement. All four astracap advanced cells completed 3/3
large documents in simple mode~65Finding 11 (the 300 s read timeout) and Finding 12
core-tt-adv-int on med_narrow5Finding 12

All at commit 3ca2dcc4d.

TierGateResult
Build + package (publish.py --clean-build)✅ PASS3.7 min; template validation passed; cfn-lint clean on 5 packaged templates; Description (v0.6.11)
Lint (make lint-cicd)✅ PASSall code-quality checks passed; 351 Markdown files, every relative link resolves
Dependency audit (make dep-audit)✅ PASS1,802 packages audited, 0 gating, 3 allowlisted, 7 below HIGH
Type check (make typecheck)✅ PASS0 errors / 44 warnings against the documented develop baseline of 4 errors / 51 warnings — errors down 4, warnings down 7. Report the delta, not the raw count
SRT (SAST + deps + security matrix)✅ PASSmake srt-scan: “no high-priority security issues found”. See Finding 6 for why the curated snapshot disagrees
Unit / integration suites (make test)⚠️ PASS with 2 host-environment failures2 failures, both in scripts/tests/test_ai_mr_review.py, both proven to be a property of this host and not of the tree — Finding 4. Every other test root green
Test coverage (make coverage-all + make check-coverage-debt)⚠️ measured; ratchet reports 2 problemsall 9 trees measured; no file lost coverage; two new modules are unratcheted — Finding 7

Measured with make coverage-all at 3ca2dcc4d. These are measurements, not the figures in scripts/coverage_debt.json, which is a ratchet floor and always self-consistent.

TreeCoverage
idp_feature_sdk99.94%
idp_cli99.68%
idp_sdk99.66%
main_stack_extensions95.28%
pii_anonymizer_api94.93%
pii_anonymizer_hook86.98%
idp_common83.37%
scripts82.21%
seller_entitlement82.03%

seller_entitlement at 82.03% is the lowest-covered tree, with scripts at 82.21% and idp_common at 83.37% immediately above it. There is no per-tree table in any previous record to diff against — coverage was absent from the release-validation deliverable list until this release — so this table is the first baseline rather than a comparison.

The one movement that can be stated: idp_sdk was 36.61% in the recorded baseline at 0.6.10 and measures 99.66% now, so the debt on that tree has been paid down since.

⚠️ The scripts figure required a second measurement run. The first reported 82.06% and the ratchet refused to compare it, correctly, because the run behind it recorded 2 test failures (Finding 4) and so the falls it would have reported were artefacts. Re-measured with the cause neutralised, the tree reads 82.21% and the refusal clears. That refusal working as designed is worth recording: a partially-failed pytest still writes a well-formed coverage.xml, and nothing inside the report says so.

Run against IDP0611Sec (v0.6.11, created from the artifact under test). Snapshot: security/test-results/0.6.11/.

TierGateResultvs PREV
SRT (SAST + deps)✅ PASS0 open/reopened HIGH per make srt-scanvs 0.6.9: 0 of 18,204 → 0 of 34,329 tracked. 16,125 more findings tracked, still 0 open
RBAC static✅ PASS0 fail · 4 known-gap warnvs 0.6.9: 2 → 4 warn, and the movement is added granularity, not new exposure — see below
RBAC dynamic❌ FAIL per the harness603 checks · 552 pass · 8 hard fail · 25 could not be run · 17 skipped · 26 known-gapidentical to 0.6.10 in every set — see below
ZAP DAST✅ PASSHigh = 0, Medium = 1, Low = 0vs 0.6.9: unchanged (Medium = 1, Cross-Domain Misconfiguration, 5 instances)

The RBAC dynamic result is not a regression, and this is established rather than assumed. A 0.6.10 run of the same harness was already on disk from 2026-09-25. Every outcome set matches exactly: 552 pass, 17 skip, 8 fail, 26 warn, 25 inconclusive — and the precise (op, principal) membership of each of those five sets is identical between the two releases. 0.6.11 introduced none of it. The apparent movement against 0.6.9’s snapshot (604 checks, 0 hard fail) is an instrumentation change: 0.6.9 reported 603 pass + 1 warn = 604 and had no fail-or-inconclusive classification at all, so a 500 and a refusal were counted together.

RBAC static 2 → 4 warn is one new check, not two new gaps. GAP-02 (queryKnowledgeBase has no group check) and GAP-07 (the chat Function URL transport carries no Cognito group claim) both carry over. The two extra lines are a new S9 check reporting GAP-07 once per affected route — POST /chat/document and POST /chat/agent. Operations covered rose 118 → 119. S2 through S5 all pass.

Each self-deploys from the artifact under test, validates, and tears down.

VariantGateResult
APIGateway hosting GLOBAL (no VPC)✅ PASSdeployed, validated, torn down
WAF (IP allow-list, no VPC)✅ PASSWebACL associated with 1 resource
APIGateway hosting PRIVATE (VPC)✅ PASSPRIVATE REST API present with resource policy
Seller Entitlement Service (us-east-1)✅ PASS8 assertions — unsigned request 403, unentitled 403 with no detail leaked, unknown product byte-identical to not-entitled, malformed body 400, oversized body 413 before parsing, 15 hostile payloads refused with no 5xx
Jobs API (VPC)⏸️ DEFERREDblocked by an account service quota, not by the code — Finding 3

VPC used for both VPC variants: the account’s purpose-built stack-test VPC (<VPC_ID>, security group named idp-stacktest-vpce-sg, two private subnets, an execute-api interface endpoint).

Two things the skill lists as open through v0.6.9 are confirmed fixed. The seller test’s --region really reaches dynamic_activation_test.py — the run log prints region us-east-1 — so its three refusal assertions can no longer pass vacuously on API Gateway’s own SigV4 403 outside us-east-1. And teardown swept /aws/apigateway/idp-seller-entitlement-citest-activation itself, so the pre-run and post-run manual deletes are gone.

Residuals on the seller tier, both unchanged: the positive path (a token issued and verified) still needs --entitled-profile and is not covered, and teardown leaves a KMS key scheduled for deletion on a 7-day window that keeps billing.

The only tier that deploys a transformed template.

VariantGateChecks
--headless✅ PASS✓ UI / Cognito / CloudFront absent · ✓ processing core present · ✓ document processed (17/22 fields populated, classified Payslip, verification string found)
--govcloud✅ PASS✓ no AWS::CloudFront::* / AWS::Lambda::Url · ✓ LWA chat-stream family removed (#677) · ✓ UI retained, hosted on API Gateway · ✓ processing core present · ✓ transformed template carries no dangling refs · ✓ document processed (143.9 s, 18/22 fields, Payslip)

cfn-lint clean on 4 templates. Both stacks and their IAM stacks torn down.

⚠️ The --govcloud result is a commercial-partition result. It proves the CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves nothing about partition ARNs, GovCloud model availability or the BDA project rejection (#676/#677 were invisible to a commercial run). This run was REGION=us-west-2 with commercial credentials. The Knowledge Base was disabled in the --govcloud run (runner default: DocumentKnowledgeBase=DISABLED, ConfigurationPreset=lending-package-sample).

In-place upgrade v0.6.10 → v0.6.11 — tier E

Section titled “In-place upgrade v0.6.10 → v0.6.11 — tier E”

Fresh throwaway stack IDPUpg0610to0611: deploy published 0.6.10 → process samples/lending_package.pdf → update-stack to the artifact under test → process the same document → diff.

CheckResult
Parameter-set diffempty — 90 parameters both sides, only AdminEmail required, none added, removed or retyped
update-stack outcome✅ UPDATE_COMPLETE, zero ROLLBACK or FAILED events
Nested stacksall 5 UPDATE_COMPLETE (APIRESOLVERSTACK, PATTERNSTACK, MULTIDOCDISCOVERYSTACK, FeaturePlatformStack, DOCUMENTKB)
Description after update(v0.6.11)
UpdateDefaultConfig custom resourceno error, validation or migration message in its Lambda log
Document, pre-upgrade1/1 success, 151.5 s
Document, post-upgrade1/1 success, 142.4 s

⚠️ One deviation forced by validating an unpublished build: PublicArtifactsBucket and PublicArtifactsPrefix were set explicitly on update-stack rather than reused with UsePreviousValue=true. Reusing them would have pointed the 0.6.11 stack at aws-ml-blog-us-west-2/artifacts/genai-idp, which does not hold this build’s assets. All 88 other parameters were reused. This does not arise for a published release.

What the upgrade changed in extracted values

Section titled “What the upgrade changed in extracted values”

Baseline outputs were pulled before the update. 49 JSON files each side, none added or removed.

MeasureValue
Leaf fields compared61,574
Leaves differing5,450
…of which Textract Blocks[] / lines[]5,307
Differences in extracted values and summaries20
Null / empty leaves, pre → post441 → 467 (all outside inference_result)

The 5,307 are block ids, geometry and per-block confidence, which differ between any two runs including two runs of one version. The 20 that remain fall in four groups, all consistent with model run-to-run variance rather than a version effect:

GroupnExample
Case style6Anytown → ANYTOWN, John → JOHN, 475 Any Avenue → 475 ANY AVENUE
Sign on payslip tax Period6-40.60 → 40.60, -8.43 → 8.43
Field placement2USA moved from account_holder_address.Line2 to .State — wrong in both places; that section’s null count is unchanged
Summary table formatting6| Field | Value | vs | Attribute | Value |, row order, bolding

⚠️ One run per side cannot separate a version effect from run-to-run variance, and this diff is reported rather than leaned on. What supports “no regression” is the blast radius — a simple-mode, unthrottled run enters neither code path 0.6.11 changed — together with the benchmark A/B below, which is the same question at three repeats.

The sign flip on six tax-deduction fields is worth a look on its own merits, independent of this release: whether a deduction is emitted negative or positive is not something a consumer should have to guess per run.

Reference stack: IDPUpg0610to0611 at v0.6.11 (the upgraded stack from tier E). Configs were generated once and not regenerated between sides, and --native-upload bypassed idp-cli’s migration on both, so the two sides ran byte-identical configuration.

F1 — release A/B, published 0.6.10 → the 0.6.11 build

Section titled “F1 — release A/B, published 0.6.10 → the 0.6.11 build”

corefast, 19 cells × 3 documents × 3 repeats = 171 runs per side, 342 total.

Result
Regressions0
Improvements0
Cell-level regressions0
Cell-level improvements0
Inconclusive3
Failure census171/171 COMPLETED on both sides, zero non-COMPLETED rows either side

All 171 rows on both sides carry cells_compared, so the “a null column scores 1.000” blind spot does not apply to this comparison.

All three inconclusive items are on core-llm-simple-sep — the bedrock_llm OCR cell — and all sit inside run-to-run spread:

MetricMovementSpread
cost−35% (0.143 ± 0.145 n9 → 0.093 ± 0.063 n9)within noise
accuracy+0.278 (0.667 → 0.944)paired per-doc spread ±0.441
recall+0.042 (0.958 → 1.000)paired per-doc spread ±0.127

⚠️ Two reading caveats. The two sides ran on different stacks — the 0.6.10 side on an existing v0.6.10 stack, reused rather than newly deployed to save ~122 IAM roles — so wall-time deltas between the sides are not comparable; accuracy, completeness and cost are, because the configuration was identical. And corefast cost CV is 0.50–1.02 across all 19 cells at n=9, which is variance across the three different documents inside each cell rather than run-to-run noise, so cell-level cost comparison on this suite is not reliable.

F2 / F3 — guidance-paper refresh: ⚠️ NOT COMPLETED

Section titled “F2 / F3 — guidance-paper refresh: ⚠️ NOT COMPLETED”

This is an incomplete tier F and the record says so. The full refresh was planned at 2,148 runs (plan retained in the run’s scratch directory), and the run was stopped after F1 and one F2 lane. What ran:

LanePlannedResult
intconf (§2.1/§6 standing hazard)8✅ complete and scored
core (§2 matrix + real-world)90⏸️ 20 launched, stopped, not scored
scaling (§3)28⏸️ 14 launched, stopped, not scored
cost (§4)25⏸️ 14 launched, stopped, not scored
advverify, the 7-model extraction sweep, astravalue, astracap, opus55value, the classification and confidence sweeps1,997❌ not started

The three partial lanes are deliberately not scored. aggregate.py will happily score a partial run and write a summary that looks complete; leaving them unscored is the only way a later reader can tell.

intconf reads clean, and that is the one positive benchmark finding here. 8/8 COMPLETED, completeness_recall 1.000 and cell_accuracy 1.000 on every row, cells_compared = 200 per row, and — the measure that matters for this hazard — 0 of 100 confidence rows unscored in both the integrated and the separate arm. The integrated-confidence truncation hazard reads closed at 0.6.11 on that document.

Every paper under docs/benchmarking/ therefore still carries data from an earlier release, by name: config-guidance.md, index.md, classification-confidence.md, prompt-caching.md and feature-multi-instance.md. None was regenerated.

One thing that would not have changed had the refresh completed: 0.6.11 changed no configuration knob. The §7 knob A/Bs had nothing to re-measure, and running them would have measured nothing about this release.

Run against IDP0611Sec at v0.6.11. 15 failed, 8 passed, 0 skipped of 23 selected.

None is attributable to 0.6.11 — the blast radius touches no file any of them exercises. They have four distinct causes, and only one is a product defect:

CauseTestsFinding
base-rule-discovery.yaml defaults produce rules the production validator rejects1Finding 2 — product
assert "output" in response contradicts invoke_model’s documented return shape6Finding 8 — test
Integration-marked tests under tests/unit/ never get real credentials6Finding 9 — test
A MagicMock config leaks into a real code path as model_id2Finding 9 — test

No browser exists on the validation host: no google-chrome or chromium binary, DISPLAY unset, nothing listening on 127.0.0.1:9222, and the chrome-devtools MCP tools were not available to the session. The skill also requires a one-time interactive sign-in in a persistent Chrome profile, which cannot be done unattended.

It is not reported as passing and not dropped. Its scope would have been flow 1 (sign-in → upload → results) only, because git diff v0.6.10..3ca2dcc4d -- src/ui/ is empty — 0.6.11 ships no UI change, so the web UI it delivers is the one 0.6.10 delivered. That bounds the risk; it does not substitute for the review. Adjacent coverage that did run: the UI vitest suite (tier A) and ZAP DAST against the UI host (tier B, High = 0).


Ordered by how much they matter. None was introduced by 0.6.11.

1. The TestSet resolver returns 500 where 404 or 400 is correct (product defect, pre-existing)

Section titled “1. The TestSet resolver returns 500 where 404 or 400 is correct (product defect, pre-existing)”

nested/api-resolvers/src/lambda/test_set_resolver/index.py raises bare Exception and KeyError for not-found and missing-argument conditions, which the dispatcher renders as HTTP 500:

Exception: Test set '__nope__' not found → should be 404, is 500
KeyError: 'testSetId' / 'id' / 'fileName' → should be 400, is 500

The security consequence is not an authorization hole but a hole in what the authorization tests can prove: 25 of the 460 positive authorization assertions are unproven rather than passed, because a 500 establishes nothing about the authorization decision that would have been reached. Affected operations: getTestSetDocuments, getAnnotationQueue, getDraftLabelJob, estimateReviewEffort, openTestSetAnnotationDraft, updateTestSetDocumentSections, reextractTestSetDocument, resetTestSetLabels, clearDraftLabels, removeDocumentsFromTestSet, createEmptyTestSet, addDocumentsToTestSet, getFeatureLaunchUrl, subscribeFeature, unsubscribeFeature.

This is the same class of change 0.6.10 already made deliberately elsewhere — moving getFileContents/getFilePresignedUrl off 500 and onto 403/400/400 — applied to the instance and not the class.

2. Default rule-discovery settings produce rules the validator rejects (product defect, pre-existing)

Section titled “2. Default rule-discovery settings produce rules the validator rejects (product defect, pre-existing)”

test_rules_discovery_with_defaults_against_medicare_sample makes a real Bedrock call (global.anthropic.claude-sonnet-4-6) against samples/rule-validation/medicare_respiratory_pa_packet.pdf, gets status: SUCCESS with at least one rule class, and those rules then fail the production validator RulesDiscovery._validate_rules_response:

Rule class 0: Missing required field: x-aws-idp-rule-type

So config_library’s base-rule-discovery.yaml defaults, on a real document, produce output the code that consumes it refuses. Reproduced on an independent second run (95.8 s), so it is not model non-determinism. Nothing in discovery/ or config_library/ changed in 0.6.11.

A second observation from the same run, minor but in the same defaults: Failed to convert max_tokens value '' to int. Not using max_tokens. — the rule-discovery configuration carries an empty max_tokens.

3. Orphaned CloudFront policies accumulate until they block deploys (product defect, pre-existing)

Section titled “3. Orphaned CloudFront policies accumulate until they block deploys (product defect, pre-existing)”

Tier C’s Jobs API variant rolled back on:

Limit exceeded for resource of type 'AWS::CloudFront::ResponseHeadersPolicy'.
Reason: Account's Response Headers Policies limit reached. … ServiceLimitExceeded

The quota is L-CF0D4FC5, 20 per account, adjustable, and the account sits at exactly 20. (The MaxItems: 100 in the list-response-headers-policies reply is the page size, not the quota — easy to misread as headroom.) Thirteen of the twenty are IDP <stack>-security-headers-policy entries, several belonging to stacks long deleted. The variants that passed all set WebUIHosting=APIGateway and create no CloudFront distribution; Jobs API uses the default CloudFront hosting and so hit the wall.

Why they accumulate — lib/idp_sdk/idp_sdk/_core/stack.py:2199 calls

cloudfront_client.delete_response_headers_policy(Id=policy_id)

with no IfMatch=. CloudFront requires the current ETag there, so every call fails InvalidIfMatchVersion and the failure is swallowed into a logger.warning. That teardown path has therefore never deleted a single orphaned policy; one run logged twelve such warnings. The sibling implementation is correct — lib/idp_sdk/idp_sdk/_core/cleanup_orphaned.py:880-889 does get_response_headers_policy → read ETag → delete_response_headers_policy(Id, IfMatch=etag).

And the test cannot catch it. The teardown suite’s fake is tests/unit/test_stack_deployer_teardown.py:2110, "delete_response_headers_policy": lambda Id, **_k: — a signature that accepts the missing IfMatch silently. The orphaned-cleanup suite’s fake declares delete_response_headers_policy(self, Id: str, IfMatch: str) and would have failed.

A quota increase to 100 was requested rather than deleting policies, because three of the twenty belong to unrelated stacks in the same account.

4. Two tests fail on a host with a managed git hooks path (test tooling)

Section titled “4. Two tests fail on a host with a managed git hooks path (test tooling)”

scripts/tests/test_ai_mr_review.py::test_the_instruction_file_set_is_derived_not_authored and ::test_the_review_criteria_come_from_the_target_branch_not_the_mr both build a throwaway repository and commit to it as t@example.invalid. Where /etc/gitconfig sets a system-wide core.hooksPath pointing at a corporate commit-scanning tool, that tool aborts any commit whose author email is not the developer’s registered one, so git commit exits 1 and the fixture raises CalledProcessError.

Proven rather than inferred: GIT_CONFIG_NOSYSTEM=1 pytest scripts/tests/test_ai_mr_review.py gives 38 passed, 1 skipped. The fixture should commit with git -c core.hooksPath= commit.

This is the same hazard CLAUDE.md already documents for make install-git-hooks, which deliberately writes to --git-common-dir rather than honouring core.hooksPath for exactly this reason — again, handled at one site and not the other.

5. The coverage ratchet checks only one of nine source trees in CI (gate tooling)

Section titled “5. The coverage ratchet checks only one of nine source trees in CI (gate tooling)”

make check-coverage-debt runs in both CI configurations, immediately after make test-cicd -C lib/idp_common_pkg. That step writes only the idp_common coverage report. The other eight trees have no report, are correctly reported as “not checked”, and that is not a failure as long as something was checked — which is documented behaviour (#1190). Neither CI passes --require-tree, and nothing in either CI runs coverage-all.

So the ratchet is green in CI while reading one ninth of what it is written to read. Finding 7 is what that permits in practice.

6. The curated SRT result disagrees with the SRT gate (gate tooling)

Section titled “6. The curated SRT result disagrees with the SRT gate (gate tooling)”

The published snapshot reads SRT — FAIL ❌ (5 open/reopened HIGH) for a release whose security gate is green. The five are exactly Bandit B105 ×2 and B106 ×3 — the checks that match an identifier’s name against a password wordlist — in test code that no deployment artifact is built from. make srt-scan carries that scope decision; scripts/security/curate_results.py implements no equivalent.

This record reports the gate’s verdict (PASS) and notes the snapshot’s disagreement. This is a known open issue, already described in the release-validation skill; it recurred exactly as written.

7. Two new modules have no coverage floor (coverage)

Section titled “7. Two new modules have no coverage floor (coverage)”

idp_cli/parameters.py and idp_feature_sdk/parameters.py are measured but absent from scripts/coverage_debt.json, so their coverage is unratcheted. Both arrived in 9bcdc0d60 on 2026-09-24, three days before this run, and passed through seven green CI runs — because of Finding 5, neither tree is read by the ratchet in CI.

No file in any of the nine trees lost coverage.

8. Six integration tests contradict the API’s documented return shape (test tooling)

Section titled “8. Six integration tests contradict the API’s documented return shape (test tooling)”

Every test in tests/integration/test_bedrock_service_tier_integration.py asserts "output" in response, but BedrockClient.invoke_model documents and returns {"response": …, "metering": …} — output lives at response["response"]["output"]. The Bedrock calls all succeeded (HTTP 200, tokens metered); only the assertion is wrong.

It became visible only in the 0.6.10 cycle: 4ae05dfab (#1129, 2026-09-23) moved the assertion out of a try/except Exception that had been converting the AssertionError into a pytest.skip — which reads as green. So #1129 did its job, and this is the residue it exposed.

9. The integration tests cannot run as documented (test tooling)

Section titled “9. The integration tests cannot run as documented (test tooling)”

make test-integration is pytest -m "integration" over the whole tree, but only tests/integration/conftest.py reconciles the sentinel credentials that tests/conftest.py injects (os.environ.setdefault("AWS_ACCESS_KEY_ID", "testing")). Integration-marked tests living under tests/unit/ therefore run with the sentinel key and cannot reach AWS — InvalidClientTokenId, InvalidAccessKeyId. Two of the seven files carrying @pytest.mark.integration are under tests/unit/, accounting for 6 of the 15 failures.

Separately, tests/integration/test_discovery_config_integration.py (2 failures) passes a MagicMock configuration into a real code path, so model_id arrives as the repr of a mock object and the production guard correctly rejects it; it also names a nonexistent test-bucket.

10. The release-validation skill lists outdated classification models (documentation)

Section titled “10. The release-validation skill lists outdated classification models (documentation)”

The skill’s F3 table names three values for the classification_model axis (nova_2_lite, sonnet5, haiku45). The live axis in benchmarks/matrices/config_matrix.yaml holds two — haiku45 is not in it. The skill instructs the reader to read the axis rather than trust the copy, which is what was done here, so the stale copy cost nothing this time; it is exactly the failure mode the skill warns about for the extraction_model axis.


11. A whole-document request over about 300 seconds always fails (product defect, pre-existing)

Section titled “11. A whole-document request over about 300 seconds always fails (product defect, pre-existing)”

Non-streaming converse runs with read_timeout = CONFIDENCE_READ_TIMEOUT_SECONDS = 300.0 and DEFAULT_MAX_RETRIES = 7, inside a 900 s Lambda, and nothing checks the remaining invocation before starting an attempt — grep finds no pre-attempt budget check in bedrock/client.py, and the deadline logic that does exist says in its own comment that it “CLAMPS a sleep; it never converts one into a failure”. Measured: attempt 1 dies at ~300 s, attempt 2 at ~606 s, attempt 3 starts at +610.8 s with 289 s left and is killed at 900 s; Step Functions retries once and the ProcessSections map discards the document.

Consequence on the model axis: Claude Opus 5.5 cannot complete an 800-row document in simple mode at all (0 of 13 cells), while Sonnet 5 and Sonnet 5 :1m completed all 13 of the same cells in the same window on the same stack — so this is a per-request limit, not contention. Advanced (sharded) mode is a complete remedy: all six swept models return 800/800 rows, 6/6.

Pre-existing, not 0.6.11. The constant is unchanged at v0.6.10; 0.6.11 raised the streamed agentic read timeout to 600 s and added a ladder guard on that path only.

12. Integrated confidence in advanced mode fails with Sonnet 4.6 on a 10-page document (product defect, pre-existing)

Section titled “12. Integrated confidence in advanced mode fails with Sonnet 4.6 on a 10-page document (product defect, pre-existing)”

cost / core-tt-adv-int (advanced + integrated, extraction sonnet46): 5 of 5 attempts failed on a 10-page document, wall 1014.8–1021.6 s (spread ±7 s — a fixed structure, not model latency). The same cell with assessment: separate completes in 111–127 s; the same cell at extraction_model: sonnet5 completed 25/25.

From ShardRuntimeFunction:

tool_name=<provide_field_assessment> | replacing with error message due to max_tokens truncation.
MaxTokensReachedException: Agent has reached an unrecoverable state due to max_tokens limit

Extraction succeeds first — the same log records Extraction complete for shard 1 (pages 1-5): 239 transactions captured ... All data validated successfully. What exceeds the budget is the per-field assessment tool output, emitted inline in the same agent loop, on shards of ~240 rows. Sonnet 5 survives because its output budget is larger.

Related and separately measured: 26 Runtime.OutOfMemory failures in AssessmentFunction, killed at its 4,096 MB ceiling where every successful assessment peaks at 267–318 MB. One invocation issued 55 Bedrock calls at ~80k input tokens each after a single 12-row → 6+6 split.

13. Three gaps in where tests and coverage checks run (gate tooling)

Section titled “13. Three gaps in where tests and coverage checks run (gate tooling)”
  • The UI vitest suite runs in neither make test nor make lint-cicd. It is a separate make ui-test target (Makefile:1181). It passes — 105 files, 1,215 tests — but only because this pass invoked it by hand, and it is the only coverage in this battery for the 28 changed UI files.
  • make test-integration pins no first-party PYTHONPATH. It is a bare $(PYTHON) -m pytest -m "integration", not $(PYTEST_HERMETIC) — correctly, since HERMETIC_AWS strips the credentials integration tests need, but it loses the FIRST_PARTY_PYTHONPATH pin with it, so the run can measure another checkout. Still true on develop at the time of writing.
  • The coverage ratchet’s three unratcheted files are still unratcheted on develop, 178 commits later. The CI half that would make it fail (#1292) is open.

14. The forced-tool test passes without testing anything for Opus 5.5 (test tooling)

Section titled “14. The forced-tool test passes without testing anything for Opus 5.5 (test tooling)”

Its force-on cells reach COMPLETED, which is why a completion count must not be read as a result: forced_tool_attempted = 0, forced_tool_honored_rate = None, and forced_tool_skips carries Bedrock’s refusal verbatim — tool_choice: type "tool" and "any" are not supported for this model. The cells ran with toolChoice: auto. Every other arm honoured forcing at rate 1.00 (core 224/224 across 138 cells; all eleven sweep arms 7/7 or 8/8), and that identity is what makes the axis’s neutrality finding meaningful at all. v0.6.11 upgrades this from a model-card claim to an observed API refusal.

15. Two cost figures in the guidance are lower bounds, not measurements (reporting)

Section titled “15. Two cost figures in the guidance are lower bounds, not measurements (reporting)”

pricing.yaml holds one flat rate per model and cannot express a usage-dependent band, but two swept models have one: Haiku 5.5 is billed 5× on every token type above 100,000 input tokens, and GPT-6 Astra moves to a long-context band above 272,000. Both are crossed in practice — 105 of Haiku 5.5’s 129 completed runs exceed 100,000 input tokens (median 543,675, max 10.96M). At the full 5× its $0.149/doc becomes ~$0.77 against Sonnet 5’s $0.571, so the sign of the comparison is unresolved. No cost ratio for either model is published; both are labelled in-table. Closing it needs per-request token capture in the harness or a two-band pricing.yaml entry. Same class as the :1m premium that once inflated an arm from $68.82 to $125.42 (#899).

16. Accuracy for the real-document corpora was recorded but not reported (benchmark harness)

Section titled “16. Accuracy for the real-document corpora was recorded but not reported (benchmark harness)”

All 2,280 corpus rows (realkie 1,140, ocr_bench 1,140, 20 documents each at 3 repeats) reached COMPLETED with 0 failures, and carry cost ($0.1932 and $0.0395 per document), latency and accuracy: weighted_accuracy is populated on 2,280 of 2,280 rows (realkie mean 0.796, ocr_bench 0.996) with class_accuracy 1.000, scored by the product’s own evaluation against the corpus labels.

What is 0 and nan is the exact-key family — cells_compared, cell_accuracy, scalar_accuracy, completeness_recall — and it does not apply to these rows at all. A reference corpus is a labelled test set on the stack, so there is no local per-cell truth file to compare against; aggregate.score_reference_run hands analyze.score_doc no truth, and the row comes from score_reference rather than score_synthetic. The two scorers emit disjoint families, and the absence is symmetric: on this same grid the 380 completed synthetic rows carry no weighted_accuracy either, by the identical mechanism. So the nan is a property of which column was read, not of the runs.

The defect was that nothing said so. The row recorded neither which scorer produced it nor which column held its accuracy, the four exact-key columns were simply absent from the CSV, and the notice write_summary prints for exactly this case was gated on docs_missing_truth — which run_matrix populates only with synthetic documents whose local truth file is missing, so it never fired for a reference corpus. Closed by #1364: every scored row now carries scored_against and, where the exact-key family does not apply, exact_truth_unavailable naming the substitute columns; cells_compared reaches the CSV beside cell_accuracy; cell_stats carries the per-scorer denominators; and aggregate.py --reference-tables renders the per-corpus per-cell table from a committed summary.

config-guidance.md §2c keeps the v0.6.8 two-model pair, which is a deliberate trade rather than a consequence of this: v0.6.11 ran the corpora at the control only, so refreshing one arm would turn that section’s model comparison into a cross-release one. The v0.6.11 control figures are quoted in §2c with the command that renders them.

17. This corpus cannot produce typed-field accuracy (benchmark harness)

Section titled “17. This corpus cannot produce typed-field accuracy (benchmark harness)”

typed_accuracy is None on all 4,704 completed rows in every scored set. analyze.py populates it “only for truth files that declare fields_typed”, and of the 24 truth files in benchmarks/corpus/docs exactly one does (kv_form.pdf, 8 fields) — every other carries rows_typed, which feeds cell_accuracy. The F3 model table therefore omits the typed-accuracy column with that reason stated, rather than printing blanks that read as zero. cell_accuracy is the per-value metric here and is populated for all 23 sets (~62,000 compared cells per arm).

18. Four outdated claims in the benchmarking documentation (documentation)

Section titled “18. Four outdated claims in the benchmarking documentation (documentation)”

None is a product defect. All four are corrected in the accompanying benchmark PR:

  1. config-guidance.md was headed v0.6.9 and attributed §2–§4 and §7 to “Sonnet 5 (the shipped default)”. Those sections are measured at Sonnet 4.6, the cross-version control — so every number in them was attributed to the wrong model.
  2. index.md listed the classification axis as four models including Haiku 4.5; the live axis is three and config_matrix.yaml states Haiku 4.5 is deliberately not swept.
  3. Both Opus 5.5 and Haiku 5.5 were marked ❌ not yet measured.
  4. The release-validation skill’s own copy of sweeps.extraction_model lists 7 models; the axis holds 8. A sweep driven from the copy runs one model short. Finding 10 is a separate instance of the same drift, on a different axis.

index.md now states the rule that prevents it: read the axis in config_matrix.yaml, not a prose list of it.


Six gaps the prerelease pass listed are now closed: the published artifact, the Jobs API variant, the guidance-paper refresh (2,628 runs), the cost suite, the A/B wall-time comparison (both sides ran alone on the same stack), and dd0b7ecdf.

The following remain unvalidated:

  • The web UI, in a browser. Tier G did not run on either pass. No flow was driven by a person or an agent. This matters more than the prerelease pass concluded, because 28 UI files changed between the two passes — Test Studio sorting and its classification-errors panel, the Document panel’s cost calculation, the configuration revision-history panel, and the issue-form links. Adjacent coverage that did run: the UI vitest suite (1,215 tests), ZAP DAST against the UI host (High = 0), and the nightly SDLC suite’s Test Studio evaluation step. None of those is a person completing a flow.
  • Real GovCloud. The --govcloud tier ran in us-west-2 on the commercial partition with commercial credentials. It proves the CloudFront-free, API-Gateway-hosted template deploys and processes a document. It proves nothing about partition ARNs, GovCloud model availability or the BDA project rejection. The Knowledge Base was disabled in that run (DocumentKnowledgeBase=DISABLED, the runner default).
  • Real-corpus accuracy at the shipped default. The 2,280 reference-corpus runs are scored (weighted_accuracy 0.796 and 0.996, Finding 16), but they ran at the cross-version control model only. config-guidance.md §2c therefore keeps the v0.6.8 two-model pair, because refreshing one arm of it would compare releases rather than models.
  • Typed accuracy. Unobtainable from this corpus — one truth file of 24 declares fields_typed (Finding 17).
  • Confidence discrimination. Not estimable at this corpus’s 0.08–0.10% error rate; the harness marks calibration_reliable false on 104 of 129 runs per arm. The AUROC column is omitted from §5.4 rather than quoted.
  • idp_cli integration tests. Zero exist — 1,347 collected, 1,347 deselected. An empty suite, not a pass.
  • The seller positive path. A token issued and verified needs --entitled-profile.
  • Coverage of src/lambda and nested/api-resolvers (145 files between them). Outside the nine-tree coverage gate; not measured here.
  • An upgrade from a stack pinned to Sonnet 4. The KnowledgeBaseModelId rejection the CHANGELOG documents was not triggered, because the test stack held the default. Verified statically from the template diff, not by a failed update.
  • Whether advanced mode rescues the assessment OOM (Finding 12’s second half) and whether the integrated-confidence failure reproduces at other document sizes.
Terminal window
# Tier A (offline), at the measured commit
git checkout 3ca2dcc4d
CI=1 python scripts/srt/setup.py && make srt-scan
make lint-cicd dep-audit typecheck
make test # see Finding 4 re: GIT_CONFIG_NOSYSTEM=1
make coverage-all && make check-coverage-debt
python3 publish.py <bucket-base> <prefix> us-west-2 --clean-build
# Tier B, against a stack at v0.6.11
make security-results STACK_NAME=<stack> REGION=us-west-2 SKIP_SRT=1
# Tier C / D
make stacktest-hosting-global TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email>
make stacktest-waf TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email>
make stacktest-hosting-private TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…
make stacktest-jobsapi TEMPLATE_URL=<url> REGION=us-west-2 ADMIN_EMAIL=<email> VPC_ID=… SUBNET_IDS=… LAMBDA_SG_ID=… APIGW_VPCE_ID=…
make stacktest-seller # REGION deliberately unset
make transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<email>
# Tier E — note the two artifact parameters must be set explicitly for a private build
aws cloudformation create-stack --stack-name IDPUpg0610to0611 --template-url <0.6.10 public url> …
idp-cli run-inference --stack-name IDPUpg0610to0611 --dir samples/ --file-pattern lending_package.pdf --batch-prefix pre-upgrade --monitor
aws cloudformation update-stack --stack-name IDPUpg0610to0611 --template-url <0.6.11 url> --parameters file://params.json …
# Tier F1
cd benchmarks
python3 harness/make_configs.py --suite corefast
python3 harness/run_matrix.py --stack <0.6.10 stack> --suite corefast --native-upload --max-inflight 4
python3 harness/run_matrix.py --stack <0.6.11 stack> --suite corefast --native-upload --max-inflight 4
python3 harness/aggregate.py --run results/run-<prev> --out results/v0.6.11/corefast-prev0610
python3 harness/aggregate.py --run results/run-<new> --out results/v0.6.11/corefast
python3 harness/aggregate.py --compare results/v0.6.11/corefast/summary.json \
--baseline results/v0.6.11/corefast-prev0610/summary.json
# Tier H
cd lib/idp_common_pkg && make test-integration # see Finding 9