Release Validation: v0.6.12
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
Release Validation Record — v0.6.12
Section titled “Release Validation Record — v0.6.12”Release: v0.6.12 (VERSION = 0.6.12, newest CHANGELOG.md section)
Validated: 2026-10-16 → 2026-10-18, us-west-2 (seller tier us-east-1), commercial partition
Commit measured: c56e06f12 — every offline figure in this record was measured at that
commit and nowhere else. Its tree is byte-identical to tag v0.6.12 (205b8c3fb):
git diff v0.6.12 github/develop is empty, 205b8c3fb being the merge of c56e06f12 into
main. Measured in a dedicated git worktree, because another assistant session shares the
primary checkout and moved it during a previous pass.
Artifact under test: the published object
https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.12.yaml,
HTTP 200, Description AWS GenAI IDP Accelerator (uksb-r8evguc4p9) (SO9027) (v0.6.12).
Every stack in tiers B, C and E was created from that object. Tier D must publish from source
to apply its transforms, and tier A’s build row is a local publish.py --clean-build.
Account <ACCOUNT_ID>. PREV = v0.6.11.
Verdict
Section titled “Verdict”✅ Ship, with four product findings a maintainer should read before the next release — the published artifact deploys in all five shipped configurations, both template transforms deploy and process a real document, the upgrade from 0.6.11 applies without rollback, and the 342-run release A/B shows 0 regressions and 0 improvements.
7 of 8 tiers ran and passed. Tier G (browser UX) is the only tier NOT RUN — no Chrome or
Chromium on the validation host, nothing on 127.0.0.1:9222, and the browsermcp MCP server
disconnected mid-session rather than being absent. Tier H passed on one of its two suites:
idp_common is 19 of 19, while idp_cli selected 0 of 1,494 tests, which is an empty suite
rather than a pass (Finding 16).
Tier F is complete. F1, the release A/B, plus the F2–F4 guidance refresh: 29 scored result sets — 3,248 scored rows from 2,488 runs, $1,970.55 — measured 2026-10-17 23:37 → 2026-10-18 06:22 UTC in one session on the upgraded reference stack, with all 28 lanes exiting at their planned run count. The configuration matrix, both reference corpora, the scaling study, the cost-variance runs, the two standing list-loss hazards, all nine extraction-model arms, the classification and confidence sweeps and all three premium head-to-heads ran. Five of §7’s knob A/Bs and §2c’s second model arm did not, and are named in Finding 7.
The refresh produced the most consequential product finding in this record, and it is not a
regression in this release. 17 of the 2,488 runs (0.68%) lost a document whose extraction
had already succeeded: in simple mode with assessment: separate, the separate confidence
pass exhausts its Lambda’s memory or its 900 s wall clock after every row has been
extracted, validated, scored and persisted, and the whole document is then discarded
(#1488).
This is the same defect shape as #1363, which this release fixed for the integrated
confidence path only. It is a configuration hazard rather than a blocker — advanced mode
completed every size in the scaling grid, and the integrated path is clean at 8/8 — but it is
marginal rather than rare at the sizes a customer is most likely to hit. See Finding 3.
The tier-E finding is the one the aggregate numbers hide. Diffing the document processed before and after the upgrade gives 38 differing leaves of 1,852, while two same-version control runs differ from each other on 56 of 1,860 — so by totals alone the upgrade moved less than the model’s own run-to-run variation, and “no change” is the conclusion the numbers invite. Two groups of differences are nonetheless identical across all three 0.6.12 runs and differ from 0.6.11, which is a version effect rather than noise: six Payslip statutory-tax amounts changed sign, and the driver’s-licence section populates one field fewer. Neither is in the CHANGELOG. See Finding 1.
Four results are improvements on 0.6.11, three of them closing items that record raised.
Both standing list-loss hazards — intconf (integrated confidence with simple extraction) and
advverify — measure clean at 8/8 with recall 1.000 and 100 of 100 rows on every run,
which the v0.6.11 record could not say either way; the curated security snapshot now reads PASS
on all four gates where 0.6.11 published SRT — FAIL ❌; the idp_common integration suite
went from 1 failure of 19 to 19 of 19; and the coverage registry grew from 9 trees to 11,
which moved the lowest-covered tree.
Results per tier group
Section titled “Results per tier group”| Tier | Gate | Result | vs v0.6.11 |
|---|---|---|---|
| A — SRT | ✅ PASS | 0 reopened, 10 open, 52 suppressed, 38,891 CI-visible. All 10 are B105/B106 identifier names in test files, non-blocking by the documented scope decision | 7 open → 10 (Finding 11) |
A — make test | ✅ PASS | all 69 test roots, 0 failures; the standing-failure baseline declares none | 68 roots → 69 |
A — make lint-cicd | ✅ PASS | exit 0; re-confirmed under the pinned ruff 0.15.13 (Finding 14) | unchanged |
A — make dep-audit | ✅ PASS | 1,805 packages, 0 gating, 5 allowlisted, 10 below HIGH | 12 → 10 below HIGH |
A — make typecheck | ✅ PASS | 0 errors / 44 warnings against the documented develop baseline of 4 / 51 | identical |
A — make ui-test | ✅ PASS | 124 files, 1,507 tests — runs in neither make test nor make lint-cicd | 105 / 1,215 → 124 / 1,507 |
| A — build + package | ✅ PASS | publish.py --clean-build in a clean env, 4.2 min; template validation + cfn-lint clean on all 5 packaged templates (8 advisory warnings) | identical |
| A — coverage | ✅ PASS | 11 trees measured, ratchet green (434 files at or above recorded) | 9 trees → 11 (Finding 8) |
| B — security snapshot | ✅ 4 of 4 gates | SRT PASS · RBAC static PASS (0 fail / 4 known-gap warn) · RBAC dynamic PASS (876 checks, 0 hard fail, 0 not-run, 17 skipped) · ZAP PASS (High=0) | 3 of 4 → 4 of 4 (Finding 17) |
| C — deploy variants | ✅ 5/5 | hosting-global, WAF, hosting-private, Jobs API, seller — each self-deployed from the published template and torn down | 5/5, unchanged |
| D — transforms | ✅ 2/2 | --headless (130.0 s) and --govcloud (137.6 s), each processing a real document, 17/22 fields, classified Payslip | 2/2, unchanged |
| E — upgrade 0.6.11 → 0.6.12 | ⚠️ PASS with a finding | UPDATE_COMPLETE in 15 m 57 s, 0 rollback events, Description now (v0.6.12); UpdateSchemaConfig and UpdateDefaultConfig both SUCCESS. Two reproducible changes in extracted values — Finding 1 | 0.6.11 reported no change (control-verified) |
| F1 — release A/B | ✅ PASS | 171/171 and 171/171, 0 failures either side; 0 regressions, 0 improvements, 2 inconclusive. Both inconclusive rows are a harness measurement defect, not a weak signal; corrected, all 19 cells are identical on both metrics — Finding 2 | 171/0 + 171/0, same shape |
| F2–F4 — guidance refresh | ✅ COMPLETE | 29 scored result sets, 3,248 scored rows from 2,488 runs, $1,970.55, measured 2026-10-17 23:37 → 2026-10-18 06:22 UTC on the upgraded stack; 28 lanes, each exiting at its planned run count. 23 non-success runs (0.92%) with three different causes — Finding 3 (#1488, 17 runs), by-design refusals (5), and one harness artefact. Five §7 knob A/Bs and §2c’s second model arm did not re-run — Finding 7 | 0.6.11: 23 sets, 2,628 runs, 4,794 rows |
| G — browser UX | ❌ NOT RUN | no Chrome/Chromium on the host, nothing on 127.0.0.1:9222, browsermcp disconnected — Finding 12 | also NOT RUN at 0.6.11 |
| H — integration tests | ⚠️ 1 of 2 suites | idp_common 19 passed / 0 failed / 0 skipped; idp_cli selected 0 of 1,494 — an empty suite, not a pass | 18/19 → 19/19 (Finding 16); idp_cli unchanged |
Tier A — test coverage, as measured
Section titled “Tier A — test coverage, as measured”make coverage-all over all eleven registry trees, then make check-coverage-debt-cicd
(--require-all-trees, so a tree with no report is refused by name rather than passing as
“not checked”). The ratchet is green: 434 files across 11 trees at or above their recorded
coverage.
| tree | measured | v0.6.11 record | delta |
|---|---|---|---|
| api_resolvers | 71.01% (lowest) | not in registry | new |
| seller_entitlement | 82.03% | 82.03% | — |
| idp_common | 84.11% | 83.46% | +0.65 |
| scripts | 86.12% | 83.04% | +3.08 |
| pii_anonymizer_hook | 86.98% | 86.98% | — |
| src_lambda | 89.90% | not in registry | new |
| main_stack_extensions | 93.86% | 95.64% | −1.78 |
| pii_anonymizer_api | 94.93% | 94.93% | — |
| idp_sdk | 99.65% | 99.66% | −0.01 |
| idp_cli | 99.68% | 99.68% | — |
| idp_feature_sdk | 99.94% | 99.94% | — |
Tier B — security gate results compared with the previous release
Section titled “Tier B — security gate results compared with the previous release”| Gate | v0.6.11 | v0.6.12 | Movement |
|---|---|---|---|
| SRT open/reopened HIGH | FAIL ❌ 7 open | PASS ✅ 0 open, 10 scoped out | gate flipped — Finding 17 |
| RBAC static | PASS, 0 fail / 4 known-gap warn | PASS, 0 fail / 4 known-gap warn | none |
| RBAC dynamic | PASS, 609 checks, 0 hard fail, 15 skipped | PASS, 876 checks, 0 hard fail, 17 skipped | +267 checks — Finding 15 |
| ZAP High / Medium | High=0 | High=0 | none |
Tier C — the five deploy variants
Section titled “Tier C — the five deploy variants”Each self-deployed from the published 0.6.12 template, asserted, and tore itself down.
VPC tiers used the account’s purpose-built stack-test VPC <VPC_ID> (security group
idp-stacktest-vpce-sg, two private subnets, an execute-api interface endpoint).
| Variant | Key assertion |
|---|---|
| hosting-global | GLOBAL REST API serving the Web UI, types=['REGIONAL'], HTTP 200 |
| WAF | regional WebACL created and attached |
| hosting-private | PRIVATE REST API present with a resource policy, types=['PRIVATE'] |
| Jobs API | Jobs API deployed and reachable at its stage URL |
| seller | 6 refusals: unsigned 403 · unentitled 403 with no internal detail · unknown product byte-identical to not-entitled (no existence oracle) · malformed 400 · oversized 413 before parsing · 15 hostile payloads with no 5xx |
The seller assertions are non-vacuous. The run was in us-east-1 with REGION
deliberately unset, and the log confirms the region reached dynamic_activation_test.py
(Region: us-east-1, signed checks as the deployment account), so the three refusals cannot
have passed on API Gateway’s own SigV4 403 — the way they passed vacuously outside
us-east-1 before v0.6.10. Teardown swept
/aws/apigateway/idp-seller-entitlement-citest-activation itself.
Two standing residuals of that tier held exactly as documented: the positive path was not
tested (it needs --entitled-profile, so a token issued and verified remains unproven), and
teardown left a KMS key on a 7-day pending-deletion window that keeps billing.
Tier D — template transforms, and the limits of a commercial-region run
Section titled “Tier D — template transforms, and the limits of a commercial-region run”Both ran idp-cli deploy --headless|--govcloud --from-code . --wait, processed
samples/lending_package.pdf, and tore down. --headless 130.0 s, --govcloud 137.6 s;
each 17/22 fields populated, each classified Payslip, each finding the expected verification
string. The transform step reported all ARN references already partition-aware (headless) and
no CloudFront / Lambda-URL resources or dangling refs (govcloud).
⚠️ The
--govcloudresult is a commercial-partition result. It proves the CloudFront-free, API-Gateway-hosted template deploys and processes documents. It proves nothing about partition ARNs, GovCloud model availability or the BDA project rejection — #676 and #677 were both invisible to a commercial run. The run wasREGION=us-west-2with commercial credentials. The Knowledge Base was disabled (DocumentKnowledgeBase=DISABLED, the runner default;--with-knowledge-basewas not passed).
Tier F1 — release A/B comparison
Section titled “Tier F1 — release A/B comparison”Same published templates, byte-identical configs, only the code version differs. corefast,
repeats: 3, --native-upload, both sides. The PREV side ran on the same stack before
update-stack, so wall-time deltas are comparable. Corpus regenerated from this worktree’s
own generators rather than reusing the primary checkout’s copy, whose provenance could not be
attributed.
| v0.6.11 | v0.6.12 | delta | |
|---|---|---|---|
| runs | 171 | 171 | — |
| successful | 171 | 171 | 0 failures either side |
| grid cost | $24.9546 | $24.9908 | +0.1% |
| grid wall | 10,357.8 s | 10,310.4 s | −0.5% |
| mean completeness recall | 1.0000 | 0.9978 | −0.2% |
| mean cell accuracy | 1.0000 | 0.9996 | −0.04% |
cells_compared | 23,370 | 23,294 | −0.3% |
| processing issues | 10 | 10 | — |
aggregate.py --compare: 0 regressions, 0 improvements, 0 cell-level either way, 2
inconclusive. Both inconclusive rows are the bedrock_llm OCR arm core-llm-simple-sep:
accuracy −0.111 (0.889 → 0.778) against a paired per-document spread of ±0.417, and recall
−0.042 (1.000 → 0.958) against ±0.127. Each is inside its own run-to-run spread at n=9. One
cost row, core-tt-adv-sep|tiny_form.pdf at +20%, sits inside a 64% spread.
⚠️ Both inconclusive rows are a harness defect rather than a weak signal, and the stronger
statement is available. The movement is one run of nine (small_narrow, dups: 38,
truncation_prefix: 5); the other eight are exactly 1.000 and the whole 0.6.11 side is 1.000
with dups: 0. rows_extracted counts distinct recognisable SEQ identifiers, and this
backend perturbs a digit in them, so the count falls with no row missing — Finding 2.
Corrected, all 19 cells are identical to three decimal places on both metrics and the A/B
moved nothing at all, which is a cleaner result than “inside the spread”. The two mean figures
in the table above are left as the tooling reported them, since that is what --compare read;
the per-cell corrections are in the benchmark entry.
cells_compared is quoted deliberately: a null column scores 1.000, so an accuracy delta is
only meaningful beside the denominator it was computed over. It moved −0.3%, which is small
enough that the accuracy figures above are comparable.
Cost CV per cell runs 0.47–0.62 at n=9 on the agentic cells on both sides, so cost movement was read against each cell’s own spread rather than a flat threshold.
Processing issues totalled 10 on each side, but not on the same cells. Both sides carry
assessment_geometry_incomplete on 9 of 9 runs of core-llm-simple-sep (severity info —
standing behaviour of that arm, unchanged). The single assessment_truncated warning sat on
core-tt-simple-sep at 0.6.11 and on enforce-escalate at 0.6.12, which is the signature of
nondeterminism rather than of a change.
Figures were generated (aggregate.py --figures-compare) into gitignored
benchmarks/paper/figures/ and are not published: no .png is tracked anywhere under
docs/, and no previous benchmark release entry cites one, so copying them into
docs/benchmarking/images/ would have introduced a convention this repository does not have.
Tier F2–F4 — configuration-guidance refresh and its failures
Section titled “Tier F2–F4 — configuration-guidance refresh and its failures”29 scored result sets, 3,248 scored rows from 2,488 runs, $1,970.55, run as 28 lanes (6
concurrent, --max-inflight 4) between 2026-10-17 23:37 and 2026-10-18 06:22 UTC on the same
upgraded reference stack every other live tier used, at v0.6.12, priced from
config_library/pricing.yaml at sha256 44a79aa06c15…. Every lane exited at its planned run
count. The per-section tables are in docs/benchmarking/config-guidance.md and the A/B entry
is docs/benchmarking/releases/v0.6.12.md, both carried by the sibling benchmark pull request;
what follows is the part that belongs to this record.
sonnet55 had to be added to the matrix before the refresh would measure it. Claude Sonnet
5.5 is the model this release adds, and it was absent from both the extraction_model axis and
its sweep list in benchmarks/matrices/config_matrix.yaml — so a refresh run exactly as
shipped would have measured eight models, skipped the ninth, and reported nothing missing. It
was added to both, which is why the model-axis count assertion moved (Finding 10).
Why the 23 unsuccessful runs failed
Section titled “Why the 23 unsuccessful runs failed”0.92% of 2,488 runs did not succeed. A single percentage is the wrong summary, because the 23 behind it divide into a defect, a correct refusal, and an artefact of the measuring harness, and only the first is a product problem:
| runs | share | where | attribution | what it means |
|---|---|---|---|---|
| 17 | 0.68% | large_narrow.pdf ×15, scale_1200 ×1, scale_1600 ×1 | 15 Runtime.OutOfMemory, 2 Sandbox.Timedout, all in AssessmentStep after ExtractionStep had succeeded | the #1488 defect: a complete extraction, and the document discarded. Every one scores completeness_recall 1.000 with every row extracted — Finding 3 |
| 5 | 0.20% | scale_3200.pdf — 1 in scaling, 4 in astracap | ExtractionInputTooLarge | by design and correct: a named, up-front refusal with nothing attempted. In astracap, the 66-page ceiling study, this is the measured outcome rather than a failure of it |
| 1 | 0.04% | coresynth__extraction-model-opus5 / core-llm-simple-sep / large_narrow | not a product failure — a harness polling timeout | scored EXTRACTING, rows_extracted 0, wall_s None, while the tracking table says that document reached COMPLETED / SUCCEEDED. The harness stopped waiting and recorded a non-success for a document that then succeeded |
Separately, 4 of the 760 (cell, corpus, document) triples on the reference corpora produced
no score at all — SCORE_ERROR on ocr_bench in four different cells. That is a scoring
failure rather than a processing one: those documents processed and the scorer returned no
figure for them, which is why four cells read 19 of 20 scored rather than 20. It is counted in
neither the 23 above nor the accuracy figures below.
⚠️ The memory failure is marginal, not deterministic, which is what makes it hard to see.
large_narrow.pdf ran 318 executions across the refresh: 303 succeeded and 15 ran out of
memory, about 5%. So it is a memory-pressure boundary at that document’s size and shape, not
a size threshold — an 800-row document with a plainer layout completes every time, and a
customer hitting this sees intermittent loss rather than a reproducible limit.
Scaling by document size
Section titled “Scaling by document size”| rows | core-tt-simple-sep (simple mode) | core-tt-adv-sep (advanced mode) |
|---|---|---|
| 25, 100, 400, 800 | COMPLETED, recall 1.000 | COMPLETED, recall 1.000 |
| 1,200 | FAILED — Sandbox.Timedout; recall 1.000, 1,200/1,200 rows over 2,400 cells, all 1,200 confidence rows scored | COMPLETED, recall 1.000 |
| 1,600 | FAILED — Sandbox.Timedout; recall 1.000, 1,600/1,600 rows over 3,200 cells | COMPLETED, recall 1.000 |
| 3,200 | FAILED — ExtractionInputTooLarge; recall 0.000, 0 rows (correct refusal) | COMPLETED, recall 1.000, assessment_recovered_with_retries=1 |
⚠️ Any figure from this grid has to be read with status, never with completeness_recall
alone. The harness reads extraction output from S3, which is complete and correct, so a
document whose data was written and whose workflow then failed scores as perfect. A table
built from recall alone reports simple mode as flawless at 1,600 rows. Advanced mode completed
every size in this grid, including 3,200 rows.
Extraction-model comparison
Section titled “Extraction-model comparison”All nine swept models were measured. Seven ran the 19-cell × 7-document coresynth grid at 133
runs each; Nova Lite and Nova Pro cannot complete the agentic path (#895) and ran simplegrid
at 63 runs each, with simplegrid’s own control as their comparison, so their rows are never
averaged with the block above them.
| extraction model | runs | completed | mean recall | mean cell accuracy | cells_compared | grid cost |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 133 | 130 | 1.0000 | 1.0000 | 83,790 | $101.67 |
| GPT-6 Astra | 133 | 133 | 1.0000 | 0.9737 | 83,788 | $209.28 (lower bound) |
| Claude Opus 5 | 133 | 132 | 0.9925 | 1.0000 | 82,190 | $126.55 |
| Claude Sonnet 5 (shipped default) | 133 | 133 | 0.9563 | 0.9996 | 74,652 | $86.10 |
Claude Sonnet 5 :1m | 133 | 133 | 0.9484 | 0.9992 | 73,334 | $87.42 |
| Claude Sonnet 5.5 (new this release) | 133 | 132 | 0.9041 | 1.0000 ⚠️ | 73,570 | $62.85 ⚠️ artefact |
| Claude Haiku 5.5 | 133 | 133 | 0.9269 | 0.9990 | 70,406 | $26.30 (lower bound) |
simplegrid control (Sonnet 4.6) | 63 | 61 | 0.9962 | 0.9990 | 39,462 | $29.17 |
Amazon Nova Pro — simplegrid | 63 | 63 | 0.4809 | 0.9361 | 6,960 | $10.89 |
Amazon Nova Lite — simplegrid | 63 | 63 | 0.4690 | 0.7525 | 8,418 | $8.76 |
⚠️ The denominators span 19% across the seven coresynth arms (70,406 to 83,790), so an
accuracy figure without its cells_compared says almost nothing. A cell an arm never
returned is a cell it is not graded on, so an arm that returns fewer rows is scored on fewer
cells: 1.0000 over 73,570 cells (Sonnet 5.5) and 1.0000 over 83,790 (Opus 5.5) are not the same
achievement, and Nova Pro’s 0.9361 over 6,960 cells against the control’s 39,462 means “94%
accurate on the rows it chose to return”. Recall is the column to rank on; accuracy is a
statement about the rows an arm did return.
Claude Opus 5.5 is the only arm with both perfect recall and perfect cell accuracy, and it reaches them over the largest denominator of any arm, for $101.67 against GPT-6 Astra’s $209.28 on the same grid — at least 51% less, the bound rather than the figure because Astra’s cost is itself a lower bound (see below). Astra also returns every row but has the lowest cell accuracy of the premium set (0.9737) over an essentially identical denominator, so that is not an artefact of what it returned. Opus 5.5’s three incomplete runs are the #1488 memory failure on one 17-page document, not a model property; Opus 5 and Sonnet 5.5 carry the same cause for theirs.
⚠️ Three of the cost figures are lower bounds, and this record quotes no cost ratio for
any of the three models. pricing.yaml holds one flat rate per model and cannot express a
usage-dependent band: Haiku 5.5 is billed 5× above 100,000 input tokens and GPT-6 Astra moves
to a long-context band above 272,000, both crossed in practice here. Sonnet 5.5 is a different
problem with the same consequence — its published per-token prices are identical to Sonnet
5’s, while the table prices Sonnet 5 at Sonnet 4.6’s rates, so its $62.85 against Sonnet 5’s
$86.10 is a pricing-table artefact and not evidence that it is cheaper. Compare those two
from the AWS price list, never from these figures.
Classification and confidence models
Section titled “Classification and confidence models”| classification model | recall | cell accuracy | cells_compared | cost |
|---|---|---|---|---|
nova_2_lite (shipped default) | 0.9982 | 0.9995 | 83,562 | $88.15 |
haiku55 | 0.9974 | 0.9995 | 83,472 | $88.07 |
sonnet5 | 0.9982 | 0.9991 | 83,560 | $113.81 |
Upgrading the classification model buys nothing measurable and costs 29% more. All three compared about 83,500 cells, so the rows are directly comparable.
| confidence model | recall | cell accuracy | cells_compared | cost | mean confidence | leaves < 0.9 |
|---|---|---|---|---|---|---|
nova_lite (shipped default) | 0.9945 | 0.9991 | 83,396 | $91.20 | 0.9955 | 0.011% |
nova_2_lite | 0.9989 | 0.9992 | 83,636 | $96.50 | 0.9981 | 0.000% |
sonnet5 | 0.9982 | 0.9995 | 83,562 | $199.95 | 0.9754 | 0.254% |
Sonnet 5 as the confidence model is 2.2× the cost and not an accuracy upgrade — it is a
far more conservative scorer. ⚠️ The flagging rates are small in absolute terms and the
comparison is a ratio, not a flood: 0.254% of its confidence leaves fall below 0.9 against
the default’s 0.011%, which is roughly 35× as many flagged fields (about 283 of 119,980
leaves against 8 of 119,943) but still a fraction of a percent. The operational reading is
that on either Nova arm a 0.9 threshold is close to inert on this corpus, and Sonnet 5 is
the one setting that actually populates the confidence-alerts filter at all. nova_2_lite is
a marginal improvement on the default at +5.8% cost, flagging nothing on this corpus.
Premium model comparisons
Section titled “Premium model comparisons”| suite | runs | completed | recall | cell accuracy | cells_compared | cost |
|---|---|---|---|---|---|---|
opus55value (first run ever) | 40 | 40 | 1.0000 | 1.0000 | 20,000 | $24.18 |
astravalue | 100 | 99 | 0.8747 | 0.9998 | 68,448 | $141.63 (0.102% of leaves < 0.9) |
astracap | 12 | 8 | 0.5841 | 1.0000 | 44,858 | $92.64 |
astracap’s four non-completions are its result: it is the 66-page ceiling study, and the
ExtractionInputTooLarge refusals are the ceiling it exists to locate.
Both known list-loss defects are fixed
Section titled “Both known list-loss defects are fixed”intconf — integrated confidence with simple extraction, the hazard that silently dropped list
rows — ran 8 of 8 COMPLETED with recall 1.000 and 100 of 100 rows on every run, and
advverify --set extraction_model=sonnet5 the same: 8 of 8, recall 1.000, 100 of 100 every
run. No processing issue on either, at n=4 per cell. The v0.6.11 record could not say whether
either was still open; both now measure clean.
Results on the real-document corpora
Section titled “Results on the real-document corpora”Both labelled corpora ran again — 756 scored rows over 760 (cell, corpus, document) triples,
0 parse failures, at the control model only. These rows are scored by the product’s own
evaluation against the corpus labels (weighted_accuracy for fields, class_accuracy for
classification), not by the harness’s exact-key scorer, so cells_compared, cell_accuracy
and completeness_recall are absent on these rows by construction — absent, not zero — and
the denominator to read them against is the 756/760 above. Class accuracy is 1.000 in every
cell of both corpora; RealKIE-FCC-Verified spans 0.758–0.812 weighted accuracy across the 19
cells and OmniAI-OCR-Benchmark 0.965–0.998. The cross-configuration ranking this produced is
Finding 6.
Findings, most important first
Section titled “Findings, most important first”1. Two reproducible, unannounced changes in extracted values after upgrading (product defect, this release)
Section titled “1. Two reproducible, unannounced changes in extracted values after upgrading (product defect, this release)”samples/lending_package.pdf was processed four times: once on 0.6.11 before update-stack,
once on 0.6.12 after it, and twice more on 0.6.12 as a same-version control. Timings,
processing_report free text and confidence_reason strings were excluded as noise.
| comparison | common leaves | differ | only-A | only-B |
|---|---|---|---|---|
| 0.6.11 vs 0.6.12-post | 1,852 | 38 | 20 | 2 |
| 0.6.12-ctrl1 vs 0.6.12-ctrl2 | 1,860 | 56 | 0 | 7 |
| 0.6.12-post vs 0.6.12-ctrl1 | 1,853 | 16 | 1 | 7 |
| 0.6.12-post vs 0.6.12-ctrl2 | 1,853 | 62 | 1 | 14 |
⚠️ Read the totals and you get the wrong answer. The upgrade diff (38) is smaller than the same-version control diff (56), which says the upgrade moved less than the model’s own variation. Two groups of differences are nevertheless identical in all three 0.6.12 runs and differ from 0.6.11 — agreement across repeats on one side is what separates a version effect from noise, and a single run per side cannot see it.
(a) Six Payslip statutory-tax amounts changed sign.
inference_result.{City,Federal,State}Taxes[*].Period: 0.6.11 returns '40.60', '28.05',
'6.56', '8.43', '5.94', '0.60'; every 0.6.12 run returns exactly those negated. The
field is declared Period: "Current period amount for this federal tax item" and evaluated
NUMERIC_EXACT, so the sign is scored rather than cosmetic.
Textract reads those lines as - - 40.60, -6.56, - 8.43, - -5.94, - -0.60 under the
document’s Deductions / Statutory heading, so the negative is arguably the more faithful
reading of the page. ⚠️ That is inference from the OCR and the column heading, not a
measurement of correctness: samples/lending_package.pdf has no ground-truth baseline in
the repository, so this record cannot say whether 0.6.12 corrected a long-standing error or
introduced one. Either way an operator comparing extracted tax figures across this upgrade
sees a sign change with nothing announcing it.
(b) The driver’s-licence section populates one field fewer.
metadata.population_check.fields_populated 9 → 8 and population_ratio 0.818 → 0.727,
identically in all three 0.6.12 runs, because ENDORSEMENTS returns an empty list where
0.6.11 returned ['NONE']. This also rewrites the extraction_incomplete processing issue to
name ENDORSEMENTS, RESTRICTIONS instead of RESTRICTIONS alone, and removes the
metadata.coercion block — 0.6.11 recorded a type_family_mismatch refusal on a null
RESTRICTIONS that 0.6.12 does not produce.
A contrast that validates the method: section 3’s fields_populated also moves (0.6.11
and 0.6.12-post both 8, both controls 9). Same shape as the two above, but it fails the
agreement test, so that one is nondeterminism.
Mechanism not isolated. config_library/ is unchanged between the tags apart from
pricing.yaml, model_config_limits.yaml and extensions-marketplace.yaml — nothing in the
lending-package preset — so the configuration is identical on both sides and the cause is in
code. lib/idp_common_pkg/idp_common/extraction/service.py gained 367 lines this release and
nothing in the diff mentions sign, negation or absolute value. Neither change appears in the
0.6.12 CHANGELOG. Both deserve either a changelog entry or a fix.
2. Published completeness figures for the bedrock_llm OCR backend were wrong for nine releases (benchmark harness, pre-existing)
Section titled “2. Published completeness figures for the bedrock_llm OCR backend were wrong for nine releases (benchmark harness, pre-existing)”The benchmark harness does not count rows. benchmarks/harness/analyze.py sets
rows_extracted to len(set(int(m) for m in SEQ.findall(blob))) against
SEQ = re.compile(r"SEQ(\d{5})"), so it counts distinct recognisable five-digit SEQ
identifiers in the result. The bedrock_llm OCR backend is independently known to insert a
digit into fixed-width zero-padded identifiers, page-scoped rather than uniform — SEQ00042
becomes SEQ000042, which the pattern reads as int("00004") and collapses onto row 4. The
distinct count falls and no row is missing.
So every completeness figure this arm has published is a reading of identifier fidelity, not of
row loss. 105 runs across 16 (cell, document, shape) groups, from v0.5.16 to v0.6.12, all
correct to completeness_recall 1.000. The arm was published between 0.5644 and 0.9796
mean recall over that span, correcting to 1.0000 on 31 of the 40 affected (grid, cell)
means; the worst figure ever published for it, v0.6.6/corefast at 0.5644, was a complete
extraction.
It reached cell_accuracy as well, and worse. The value join keys on the same identifier,
so a perturbed id pairs one row’s values against another row’s truth and reports a value error
that is an artefact of the join. cells_compared is exactly 2 × rows_extracted on all 90
affected runs, so accuracy was computed over 92.3% of the rows actually returned while being
published as if over all of them.
Three independent confirmations rather than one, because a metric correcting itself upward is the direction that deserves the most scepticism:
- Reproduced by probe. A blob containing every row, with ids corrupted at the observed rate, recovers all seven committed figure-shapes byte for byte.
- An identifier-free field disagrees with it.
conf_rows_expectedis derived from the extracted list’s own length and uses no identifier at all. It reads the full truth row count on all 75 affected runs that record it —med_narrowcarriesrows_extracted: 362besideconf_rows_expected: 400, conf_rows_scored: 400. - The two readings are separable in data the analysis already had. Genuine tail truncation
gives
dups: 0and atruncation_prefixequal to the kept count; scattered id corruption gives a largedupsand atruncation_prefixnear zero. Every affected run has the second signature.
⚠️ The standing narrative that this backend is “the noisiest of the nineteen” was therefore an instrument reading, and every release entry that treated its movement as a product signal was reporting the same defect. The v0.6.6 entry had already root-caused the behaviour to identifier corruption — the metric was simply never corrected for it, which is how it resurfaced as a fresh finding in each release since, including as the one “moved” cell of this release’s own A/B. A defect that is diagnosed and not fixed in the instrument is one the next pass pays for again.
One residual, stated rather than smoothed: core-tt-adv-sep / scale_1600 returned 1,698
identifiers for 1,600 truth rows, so it carries genuine duplication as well and its corrected
figure is an upper bound. The other 104 affected runs returned exactly the truth count of
identifiers.
Tracked as
#1500;
the fix canonicalises each identifier against the truth id set and records rows_id_repaired,
so a future grid cannot report this as row loss.
3. The separate confidence step discards fully extracted large documents (product defect, pre-existing)
Section titled “3. The separate confidence step discards fully extracted large documents (product defect, pre-existing)”17 of the refresh’s 2,488 runs (0.68%) threw away a document whose extraction had already
succeeded. Fifteen were Runtime.OutOfMemory and two Sandbox.Timedout, and all seventeen
failed in the same place. Step Functions history for one of each — large_narrow for the
memory failure, scale_1200 for the timeout — shows an identical sequence: ExtractionStep
exits successfully, PostExtractionHook succeeds, AssessmentStep is entered, its Lambda
fails (once retried), then MapIterationFailed → MapStateFailed → ExecutionFailed. So in
simple mode with assessment: separate, the separate confidence pass is a single point of
failure on large documents, and it fails after every row has been extracted, validated,
confidence-scored and persisted to S3. Two different resource limits — the 900 s Lambda wall
clock and Lambda memory — in one step. Filed as
#1488.
This is the same defect shape as #1363, which this release fixed for the integrated confidence path only. The separate path still discards the document. The 25-row default confidence batch cap is part of the mechanism: 1,600 rows is roughly 64 serial scoring calls inside one 900 s invocation.
⚠️ The failure is invisible in the accuracy metrics, which is the part to carry forward.
Every one of the seventeen scores completeness_recall 1.000 with every row present,
because the harness reads extraction output from S3 and that output is complete. A table built
from recall rather than from status reports this configuration as flawless at 1,600 rows.
It is not a row threshold, and it is not deterministic. scale_800 (800 rows) completes in
simple mode, while large_narrow — also 800 rows, but 17 pages and about 203,000 estimated
input tokens — ran 318 executions across the refresh and lost 15 of them, about 5%, in
three different cells. So the boundary is a resource limit at a particular size and shape, and
a customer meeting it sees intermittent loss rather than a reproducible ceiling. It sits at
exactly the document sizes a customer is most likely to hit.
Two configurations measured clean against it on this build, which is the practical remedy
until the defect is fixed: advanced mode completed every size in the scaling grid including
3,200 rows, and the integrated confidence path (intconf) is 8/8 with recall 1.000 and 100 of
100 rows on every run.
4. Claude Sonnet 5.5, new in this release, loses most rows on 400-row documents (product defect, this release)
Section titled “4. Claude Sonnet 5.5, new in this release, loses most rows on 400-row documents (product defect, this release)”13 of its 133 coresynth runs return 2 or 10 rows of a 400-row list, and they are not
marginal losses:
| recall | rows returned | documents / cells | cells_compared | cell accuracy on those runs |
|---|---|---|---|---|
| 0.005 | 2 of 400 | med_narrow × 4 cells | none recorded | not scored |
| 0.025 | 10 of 400 | med_narrow, wide_400, manylists_400 × 9 cells | 20 | 1.000 |
⚠️ Its headline cell accuracy of 1.0000 is computed over the 20 cells it returned. The arm’s mean recall of 0.9041 over 73,570 compared cells — among the three smallest denominators in the grid — is a bimodal distribution read as an average: every one of the 13 is a 400-row document and the smaller documents in the grid are unaffected, which is what makes a mean of 0.9041 look survivable. On a 400-row list this is the worst arm in the grid, not the cheapest good one.
Detection is mostly present and fails at the extreme. 9 of the 13 carry
extraction_rows_below_ocr_estimate. The four worst — the 2-of-400 runs — carry only
extraction_validation_failed on three of them and no completeness issue at all on the
fourth, and the coverage walk recorded no denominator for any of the four, so the coverage rung
could not speak either. Those same four runs are what falsified a pinned test premise
(Finding 10).
Three further properties make the model’s other columns non-comparable, independent of the row
loss: it rejects a forced toolChoice, so Schema Enforcement falls back to the prose
schema and extraction.forced_tool is recorded as skipped; it rejects
temperature/top_p/top_k, which the accelerator strips for this model id; and 1M is its
default window with no :1m id, which makes the derived agentic shard budget 578,400 tokens
against Sonnet 5’s 18,400 — wide shards defeat prompt caching, so its cost per document is not
comparable on the agentic path unless extraction.agentic.shard_token_budget is pinned.
Filed as #1494 and parked pending a product decision — whether to keep the model selectable as is, pin its shard budget, or gate it by document size, is a product call rather than a validation one. The guidance half has shipped: the configuration paper’s model section carries the measurement and recommends against selecting it for list extraction on documents of a few hundred rows or more.
5. A list with no OCR evidence is reported as complete instead of unverifiable (product defect, pre-existing)
Section titled “5. A list with no OCR evidence is reported as complete instead of unverifiable (product defect, pre-existing)”The six completeness signals that exist to make a short extraction visible —
extraction_rows_below_ocr_estimate, extraction_partial_row_shortfall,
extraction_output_truncated, extraction_incomplete, extraction_validation_failed,
extraction_sparse — fire on nearly every lossy run on three OCR backends and on about half of
them on bedrock_llm. Taking a lossy run to be one whose extracted rows are under 95% of
ground truth, over 607 scored runs: 48 of 49 flagged on textract_tables, 9 of 9 on bda, 8 of
8 on textract_layout, and 9 of 17 on bedrock_llm.
⚠️ Seven of those eight silent runs lost no rows — they are Finding 2’s identifier
corruption, which is why that finding is ordered above this one. Of the eight, seven carry
dups of 38–45 with truncation_prefix of 5–86 (the corruption signature) and one,
longdesc_100 on the nova_lite arm, carries dups: 0 with prefix: 64 against 66 of 100
rows, which is genuine truncation that reported nothing. A detection rate is the wrong
instrument for this question and it is quoted here only because it is what first exposed the
mechanism; the finding that survives is structural rather than statistical.
The structural half is measured, on a 3-property item schema. _expected_rows_for_width
sums rows only from OCR tables whose column count equals the item schema’s property count, and
evidence reaches zero by two distinct routes, not one:
| Rendering | _ocr_tables | _expected_rows_for_width(3, …) |
|---|---|---|
textractor TABLES form, 3 columns | [{'rows': 41, 'cols': 3}] | 41 |
| GitHub-legal pipe table, no leading pipes | [] | 0 |
| same table with a 4th column the schema lacks | [{'rows': 41, 'cols': 4}] | 0 |
| same table with an added row-number column | [{'rows': 41, 'cols': 4}] | 0 |
| rendered as a bulleted list | [] | 0 |
For the two four-column rows the width match is what rejects it — a table is found, and
_expected_rows_for_width(4, …) returns 41 against the same text. For the no-leading-pipe and
bulleted renderings _ocr_tables recognises no table at all, upstream of any width question.
Both land in the same place, and at zero evidence expected < _OCR_ROW_ESTIMATE_MIN holds for
every extracted count and every document size — so the two OCR-grounded checks are not merely
quiet, they are unreachable. bedrock_llm renders its own Markdown rather than emitting
Textract table blocks, which is why it is the backend most exposed to this.
What is not established is that this is what happened on the one genuinely truncated run.
That needs _ocr_tables output captured from it; the stacks are gone and no OCR page artifact
for those runs survives, so the link between the measured mechanism and the observed silence is
unverified.
Tracked as
#1489.
The remedy taken is the one that is correct under either reading of the detection rate: report
explicitly when a list came back with no usable evidence, so “could not check” stops being
indistinguishable from “checked and found complete” — which is what a user currently sees. The
exact-width match is deliberately left alone, including as a fallback reachable only when the
exact-width evidence is empty: “empty” is the ordinary state for an item schema with a derived
property, so such a fallback would be reachable on textract_tables too and would let a
two-column key/value block report a shortfall against a three-column list.
6. Simple mode is more accurate than advanced mode on real documents, at half the cost (configuration guidance)
Section titled “6. Simple mode is more accurate than advanced mode on real documents, at half the cost (configuration guidance)”This is the result the synthetic grid cannot produce, and the reason running the reference corpora matters more than their row count suggests. The 19 synthetic cells score 1.000 on recall and cell accuracy almost everywhere, so that grid ranks nothing. RealKIE-FCC-Verified ranks them, and it inverts the intuition that the agentic path is the more capable one:
| RealKIE cell | weighted accuracy | cost/doc | wall |
|---|---|---|---|
split-disabled | 0.812 | $0.1251 | 44 s |
core-tt-simple-sep | 0.807 | $0.1423 | 38 s |
core-tt-simple-off | 0.805 | $0.1392 | 29 s |
core-tl-adv-sep | 0.791 | $0.2464 | 53 s |
core-tt-adv-int | 0.768 | $0.4870 | 63 s |
core-bda-adv-sep | 0.760 | $0.3142 | 53 s |
core-tt-adv-sep | 0.758 | $0.2757 | 52 s |
The three best cells are all simple mode and the three worst are all advanced, and cost runs
about 2× in the same direction (core-tt-simple-sep $0.1423 against core-tt-adv-sep $0.2757).
On this corpus advanced extraction pays more for less.
Three bounds on that. The figures are weighted_accuracy from the product’s own evaluation
against the corpus labels, so there is no cells_compared to quote — the metric is absent on
these rows by construction, and the denominator to read them against is the 756 scored rows
over 760 (cell, corpus, document) triples with 0 parse failures. Only the control arm ran,
so this is a cross-configuration result on this release and not a model comparison. And
RealKIE’s documents are not long lists, so this says nothing about completeness at scale, which
is the scaling study’s territory and where advanced mode is unambiguously better: it completed
every size in that grid, while simple mode is the configuration that loses whole documents
(Finding 3). So this is a recommendation about labelled real-world forms, not a general one
— on long lists the ranking reverses.
7. Some configuration-option A/B tests were not re-run for this release (process)
Section titled “7. Some configuration-option A/B tests were not re-run for this release (process)”Tier F is complete and the guidance papers are refreshed on this build; what is left unmeasured is narrow, and these are the figures not to cite as v0.6.12 results:
- Five of §7’s knob A/Bs did not run —
splitcost,advsplitcost,boundaryab,multiinstanceandsizerab. Those subsections keep figures from earlier editions and say so in their first sentence.enforcement,enforcecost,forcingand the schema-restatement arms insidecorewere re-measured. - §2c ran at the control model only, so the two-model comparison the previous editions published is not refreshed. Comparing a v0.6.12 control figure against a v0.6.8 shipped-default figure would be a cross-release difference wearing a model label, so §2c is a cross-configuration result on this release and the v0.6.8 pair remains the only like-for-like model comparison on those corpora.
One metric is missing from every table in the refresh, and it is a gap rather than a choice:
typed_accuracy is emitted as null on all 3,248 scored rows
(#1490),
so no per-model typed-accuracy column could be produced. It is the one metric that separates a
value extracted with the right type from one that merely matches as a string.
8. A new lowest-coverage source tree, and one tree’s coverage fell within tolerance (gate, pre-existing)
Section titled “8. A new lowest-coverage source tree, and one tree’s coverage fell within tolerance (gate, pre-existing)”The coverage registry grew from 9 trees to 11: src_lambda (89.90%) and api_resolvers
(71.01%) are new since the v0.6.11 record. api_resolvers is now the lowest-covered tree by
11 points, displacing seller_entitlement (82.03%), which the previous record named as the one
to watch. api_resolvers holds the UI API resolver Lambdas — the layer this release moved
every document read behind (#1033) — so it is the tree where low coverage costs most right
now.
Separately, main_stack_extensions measured 93.86% against the previous record’s 95.64%, a
loss of 1.78 points, which the per-file ratchet’s 1% tolerance absorbed, so
make check-coverage-debt stayed green. That is the ratchet behaving as designed and is
exactly why the release record carries the measured per-tree table and its delta: a tree can
lose coverage with every gate green.
9. The confidence-coverage warning has no absolute limit, so more rows go unscored on longer lists without a warning (gate, pre-existing)
Section titled “9. The confidence-coverage warning has no absolute limit, so more rows go unscored on longer lists without a warning (gate, pre-existing)”benchmarks/tests/test_coverage_threshold_evidence.py pins the largest number of unscored rows
recorded on a run the coverage warning rung said nothing about. Re-measured on the v0.6.12
grids it is 115 unscored rows of 3,200 (3.6%) on scale_3200.pdf / astra-adv-wide,
against 24 of 800 (3.0%) when it was last measured.
⚠️ The two measurements together are the finding, not the new number. The row count grew
4.8× while the share barely moved, because the document grew 4× — which is exactly what a
purely multiplicative rung does. The silent band is proportional to list length, so the
absolute number of rows that can go unreported has no bound at all; it is 115 here only because
3,200 rows is the longest list in the corpus, and a longer production list carries a
proportionally larger band. That is the same shape #1361 fixed for
extraction_rows_below_ocr_estimate, by replacing a multiplicative tolerance with a 29-row
additive one, and it is the argument for giving this warning rung an absolute ceiling the way
its error rung already carries an absolute floor. Tracked as
#1495.
The pin only stops the figure growing unnoticed again; a pin over a committed corpus tracks the
longest list anyone benchmarked rather than bounding production.
10. Committing new benchmark results changed three test assumptions, and one was disproved (test, this release)
Section titled “10. Committing new benchmark results changed three test assumptions, and one was disproved (test, this release)”Those suites assert over the committed result sets, so landing 29 of them re-measures their premises rather than merely adding data. Three moved, and the distinction between the first two and the third is the point:
MAX_SILENT_UNSCORED_ROWS24 → 115. A re-measurement of the same quantity on a longer document — Finding 9.- The exact model-axis count, 19 → 20, in
benchmarks/tests/test_model_axis_coverage.py, becausesonnet55was added to theextraction_modelaxis and its sweep list. The count is asserted exactly so that a model disappearing from an axis fails rather than passing quietly, which means it is meant to be updated deliberately in the same change. - ⚠️
test_an_undefined_coverage_run_extracted_nothingwas falsified, not moved. Its premise is that a run reporting no coverage denominator extracted nothing — the guard declines to fire when a section produced no list row. Fourcoresynth__extraction-model-sonnet55runs onmed_narrow.pdfextracted 2 rows of 400 while recording no denominator, so the coverage rung could not speak about the 398 that were lost. That is #1494’s coverage-walk half (Finding 4), not a reason to relax the check, so the four are carved out by name and by count — that grid and that document, exactly four — and a fifth instance, or the same shape on another model or document, fails there. If the count drops to zero the defect is fixed and the carve-out is deleted with it.
A premise that is false and merely relaxed is how a check becomes decoration; bounding the carve-out on both axes is what keeps the next instance visible.
11. The SRT open-finding count rose from 7 to 10, all in test files (informational)
Section titled “11. The SRT open-finding count rose from 7 to 10, all in test files (informational)”All ten are Bandit B105/B106: matches of an identifier’s name against a password
wordlist, in test files no deployment artifact is built from, reported at Bandit’s own severity
rather than blocking. The three new ones are test_delete_documents_selector_failure.py,
test_delete_documents_coverage.py and test_batch_delete_selector_failure.py — test files
this release added with the document-deletion fix (#1366). Nothing reopened and the
suppression register is non-vacuous at 52 entries. No action; recorded because any movement in
a gate value is a finding.
12. The browser UX review (tier G) could not run (environment)
Section titled “12. The browser UX review (tier G) could not run (environment)”No Chrome or Chromium on the validation host, nothing listening on 127.0.0.1:9222, no
chrome-devtools MCP, and browsermcp disconnected during the session rather than being
absent. This is the second consecutive release with tier G NOT RUN.
It matters more here than at 0.6.11 because this release’s UI surface is large: page rotation
in four editors, the “why this field has no bounding box” control on all four field-displaying
surfaces, Test Studio’s server-side document search, the annotation-queue advance, the
prefix-mapping admin screens, the malware-scan banner and detail field, and the cost panel’s
unpriced-unit handling. The compensating coverage is the UI unit suite — 124 files, 1,507
tests, all passing — which runs in neither make test nor make lint-cicd and was invoked
separately. Unit coverage is not a person completing a flow, and no flow in
scripts/ux_flows.yaml was exercised.
13. A test writes build output into the tree, which degrades the next SRT scan (test tooling, pre-existing)
Section titled “13. A test writes build output into the tree, which degrades the next SRT scan (test tooling, pre-existing)”lib/idp_sdk/tests/unit/test_multi_doc_discovery_source_zip.py writes a ~7 MB
.aws-sam/multi-doc-discovery-source.zip into the repository root while it runs. Any
make srt-scan ordered after make test therefore reports 1 .aws-sam build directory present and drops into its degraded mode, emitting phantom LOCAL-ONLY findings and risking a
checkov crash on the largest templates.
The scan segregated them correctly, so the gate verdict held — but the preflight rule in
.claude/skills/release-validation.md is “scan before publish.py”, and that is not
sufficient: it has to be before make test too. Either the test should write to a temporary
directory, or the preflight rule should say so.
Tracked as
#1491.
The remedy taken is the first option, plus a suite-wide control: the test writes to a temporary
checkout and asserts the real tree is unchanged, and a shared guard reuses the SRT scanner’s own
**/.aws-sam predicate so any suite that starts writing build output into the tree fails
rather than being discovered by the next scan’s degraded mode.
14. The ruff on PATH is not the pinned version (gate, environment)
Section titled “14. The ruff on PATH is not the pinned version (gate, environment)”make lint-cicd printed ⚠️ ruff 0.15.20 is first on PATH, but this repo pins ruff 0.15.13
and then passed. Findings and formatting are both properties of the linter version, so a
green run under the wrong one does not establish the CI result. Re-run under the pinned
version via a shim (uvx ruff@0.15.13): make ruff-lint check-lint-debt exits 0, ruff check --fix modified nothing, and the lint-debt gate reports 82 files excluded from ruff check
and 171 from ruff format, all carrying exactly their recorded findings. So the gate is
genuinely green — but it was green for the wrong reason until re-run, and the warning is one
line in a long log.
15. RBAC dynamic checks grew from 609 to 876, with 17 skipped (test, informational)
Section titled “15. RBAC dynamic checks grew from 609 to 876, with 17 skipped (test, informational)”Movement is in the expected direction and consistent with the release’s new API surface
(getFilePresignedUrls, the prefix-mapping operations, the malware-scan fields). The
operation × role matrix grew 580 → 605, so most of the growth is outside the matrix. Skips
moved 15 → 17 and sit in three suites: 11 in the positive authorization matrix, 3 in
config-version scope enforcement, 1 in token lifecycle (with its 1 known-gap warning), 2 in
Other. 0 hard failures and 0 checks that could not be run, which is the distinction that
matters — a check that could not run establishes nothing.
16. A defect open across two releases is fixed (product, improvement)
Section titled “16. A defect open across two releases is fixed (product, improvement)”idp_common’s integration suite is 19 passed, 0 failed, 0 skipped (11,859 deselected). At
v0.6.11 it was 1 failure of 19, and that failure reproduced identically at v0.6.10. This is the
only tier that exercises the library against real Bedrock, Textract, S3 and DynamoDB rather
than moto, so it is not covered by make test or by either CI.
idp_cli is unchanged and is not a pass: -m integration collected 1,494 tests and
selected 0. An empty suite asserts nothing.
17. The security snapshot’s SRT result now agrees with the live scan (gate, improvement)
Section titled “17. The security snapshot’s SRT result now agrees with the live scan (gate, improvement)”0.6.11’s curated snapshot published SRT — FAIL ❌ (7 open/reopened HIGH) for a release whose
live scan was green, because the curator counted the scoped-out identifier-name findings as
blocking while the scan did not. 0.6.12’s snapshot reads PASS ✅ (0 open/reopened HIGH of 38,891 tracked, 10 scoped out) — the curator now applies the same scope decision. The
previous record logged this as a finding; it is closed.
18. Four orphaned network interfaces in the stack-test VPC (environment)
Section titled “18. Four orphaned network interfaces in the stack-test VPC (environment)”After the two VPC tiers, describe-network-interfaces --filters status=available in the
stack-test VPC returned four. Two named this run’s deleted apigwpriv stack and were deleted.
Two name an idpmwjobs stack that is also gone but is not from this run, and were left in
place rather than deleting another session’s resources. They will eventually block a VPC or
security-group delete and should be cleared deliberately.
Issues filed and their status
Section titled “Issues filed and their status”Eight issues came out of this pass. They are listed together because a finding a reader cannot act on is half a finding, and because the mapping is not one-to-one: two issues came out of a single finding splitting under investigation, and one came out of fixing another.
| Issue | Finding | Status |
|---|---|---|
| #1488 — separate confidence pass discards a complete document | 3 | fix open, PR #1493 |
| #1489 — a list with no OCR evidence reads as checked-and-complete | 5 | fix open, PR #1499 |
#1490 — typed_accuracy null where inapplicable | 10 | fix open, PR #1496 |
#1491 — a test writes .aws-sam/ into the tree | 13 | fix open, PR #1492 |
| #1494 — Sonnet 5.5 returns 2–10 rows of 400 | 4 | parked pending a product decision; the guidance half has shipped |
| #1495 — coverage WARNING rung has no absolute ceiling | 9 | fix open, PR #1498 |
#1497 — cell_accuracy present-and-null on list-free documents | — | fix open, PR #1501 |
#1500 — rows_extracted counts identifiers, not rows | 2 | fix open, PR #1502 |
Two of those entries are worth reading as process results rather than as defects.
#1500 was extracted from #1489 after the fact. #1489 was filed on a detection-rate table
whose eight silent runs were taken to be eight lost documents; running the discriminating
measurement (dups against truncation_prefix) showed seven of the eight had lost nothing, and
that half became #1500. The original figures are corrected in #1489’s own thread rather than
left standing. The lesson is cheap to state and was not cheap to learn: a count derived from
identifiers cannot be read as a count of rows on a backend that perturbs identifiers, and the
data needed to tell the two apart was already in every summary.
#1497 was found by fixing #1490, not by the validation. The null-where-inapplicable shape #1490 names turned out to have a second instance in the same file, which is why it carries no finding number — it is in this table because it is open work from this pass, not because a tier reported it.
What was NOT validated
Section titled “What was NOT validated”- Real GovCloud. Tier D’s
--govcloudrun wasREGION=us-west-2with commercial credentials. Partition ARNs, GovCloud model availability and the BDA project rejection are unproven — the class of defect that shipped as #676/#677. The Knowledge Base was also disabled in that run. - The web UI as a person uses it. Tier G NOT RUN (Finding 12). No flow in
scripts/ux_flows.yamlwas exercised on this build. - Five §7 knob A/Bs and the second arm of §2c (Finding 7):
splitcost,advsplitcost,boundaryab,multiinstanceandsizerabwere not re-measured on this build, and the reference corpora ran at the control model only, so no two-model comparison on them exists for this release. Everything else tier F answers was measured. typed_accuracyfor any model. It isnullon all 3,248 scored rows (#1490), so the one metric that would separate a correctly typed value from a string match is missing from every table in the refresh.- The seller positive path. No token was issued or verified; the tier asserts refusals only.
idp_cliintegration behaviour. The suite is empty (Finding 16).- Whether Finding 1’s sign change is a correction or a regression. There is no
ground-truth baseline for
samples/lending_package.pdfin the repository, so correctness could not be scored — only the change, and its reproducibility, were measured. - Canada (
ca-central-1). This release adds it as a published region with its own model mapping. Nothing in this pass deployed there; all stacks wereus-west-2(sellerus-east-1). - Malware scanning. The new
MalwareScanningparameter defaulted toDISABLEDin every stack here, so no gate decision, quarantine path or GuardDuty verdict was exercised.
Reproduce
Section titled “Reproduce”# Tier A (offline), in a clean worktree at the release commitCI=1 python scripts/srt/setup.py && make srt-scan # BEFORE make test — see Finding 13make test && make coverage-all && make check-coverage-debt-cicdmake lint-cicd && make typecheck && make dep-audit && make ui-testpython3 publish.py <bucket-base> idp us-west-2 --clean-build
URL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.12.yaml
# Tier C (5 variants)make stacktest-hosting-global TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-waf TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email>make stacktest-hosting-private TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \ VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>make stacktest-jobsapi TEMPLATE_URL=$URL REGION=us-west-2 ADMIN_EMAIL=<email> \ VPC_ID=<VPC_ID> SUBNET_IDS=<SUBNET_A>,<SUBNET_B> LAMBDA_SG_ID=<SG_ID> APIGW_VPCE_ID=<VPCE_ID>make stacktest-seller # REGION deliberately unset -> us-east-1
# Tier Dmake transform-deploy-test-all REGION=us-west-2 ADMIN_EMAIL=<email>
# Tier E — and run FOUR documents, not two: one on PREV, one on VERSION,# plus TWO same-version controls, or Finding 1 is invisible.aws cloudformation create-stack --stack-name IDPUpg0611to0612 --template-url <0.6.11 URL> ...aws cloudformation update-stack --stack-name IDPUpg0611to0612 --template-url $URL \ --parameters <UsePreviousValue=true for all 90 existing parameters>
# Tier B / F1 / H, on the upgraded stackmake security-results STACK_NAME=IDPUpg0611to0612 REGION=us-west-2 SKIP_SRT=1python benchmarks/harness/gen_corpus.pypython benchmarks/harness/make_configs.py --suite corefastpython benchmarks/harness/run_matrix.py --stack IDPUpg0611to0612 --suite corefast \ --repeats 3 --native-upload --max-inflight 4# NOTE the argument direction: --compare is the CURRENT side, --baseline the reference.python benchmarks/harness/aggregate.py --compare <v0.6.12 summary> --baseline <v0.6.11 summary>cd lib/idp_common_pkg && make test-integration
# Tier F2-F4 on the same stack: one lane per suite/override, 6 concurrent.# Add any model the release adds to BOTH the axis and its sweep list in# benchmarks/matrices/config_matrix.yaml FIRST, or the sweep runs one model short# and reports nothing missing.python benchmarks/harness/run_matrix.py --estimate --suite <suite> [--set <axis>=<value>]python benchmarks/harness/run_matrix.py --stack IDPUpg0611to0612 --suite <suite> \ [--set extraction_model=<model>] --native-upload --max-inflight 4python benchmarks/harness/aggregate.py --run benchmarks/results/run-<stamp> \ --out benchmarks/results/v0.6.12/<suite>[__<override>]# Read every scaling and grid row with `status`, never with completeness_recall alone# (Finding 3), and read every accuracy figure with its cells_compared (Finding 4).Raw logs for every tier are in gitignored scratch/ and are not published.