Benchmark: v0.6.12 vs v0.6.11
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
v0.6.12 vs v0.6.11 (published) — release benchmark audit
Section titled “v0.6.12 vs v0.6.11 (published) — release benchmark audit”Tier F is complete for v0.6.12. The release A/B below is one of 29 scored result sets — 3,248 rows from 2,488 runs, $1,970.55 — measured in one session on one stack. The configuration matrix, the two reference corpora, the scaling study, the cost-variance runs, the standing-hazard suites, all nine extraction-model arms, the classification and confidence sweeps and all three premium head-to-heads ran. What did not re-run is named in the configuration guidance: five of §7’s knob A/Bs, and §2c’s second model arm.
Verdict: flat, with one arm inconclusive. The release A/B over 342 runs reports 0 regressions and 0 improvements. Grid cost moved +0.1% and wall −0.5%, both well inside run-to-run spread, and 18 of 19 cells are identical to three decimal places on both completeness recall and cell accuracy. Neither side had a single failed run.
⚠️ The one apparent change is a measurement defect
Section titled “⚠️ The one apparent change is a measurement defect”core-llm-simple-sep, the bedrock_llm OCR arm, is reported at recall 1.000 → 0.958 and
cell accuracy 1.000 → 0.993. The rows were never lost.
rows_extracted is the count of distinct recognisable SEQ identifiers, matched by a
five-digit pattern. That backend perturbs a digit in fixed-width ids — duplicating the
leading zero, so SEQ00042 becomes SEQ000042, which the pattern reads as int("00004")
and collapses onto row 4. The count falls; no row is missing. Reproduced by probe against a
blob containing every row, recovering all seven committed figure-shapes byte for byte,
and corroborated by a field that uses no identifier at all: conf_rows_expected, derived
from the extracted list’s own length, reads the full truth row count on all 75 affected
runs that record it — med_narrow carries rows_extracted: 362 beside
conf_rows_expected: 400, conf_rows_scored: 400.
In this A/B the movement is one run of nine (small_narrow, dups: 38,
truncation_prefix: 5); the other eight are exactly 1.000, and the whole 0.6.11 side is
1.000 with dups: 0. Corrected, all 19 cells are identical and the A/B moved nothing at
all.
This is not new to this release — it is nine releases old. 105 runs across 16
(cell, document, shape) groups from v0.5.16 to v0.6.12 all correct to recall 1.000. This arm
was published between 0.5644 and 0.9796 mean recall over that span, correcting to
1.0000 on 31 of the 40 affected (grid, cell) means; the worst published figure,
v0.6.6/corefast at 0.5644, was a complete extraction. It also reached cell_accuracy,
and worse: the join keys on the same identifier, so a perturbed id could pair one row’s
values against another row’s truth and report a value error that is an artefact of the
join. cells_compared is exactly 2 × rows_extracted on all 90 affected runs, so accuracy
was measured over 92.3% of the rows actually returned.
⚠️ So the standing narrative that this backend is “the noisiest of the nineteen” was an
instrument reading, and every earlier entry that treated its movement as a product signal
was reporting the same defect. The v0.6.6 entry had already root-caused the behaviour to
identifier corruption and the metric was never corrected for it, which is how it resurfaced
as a fresh finding in each release since. Tracked as
#1500;
the harness now canonicalises the identifier against the truth id set and records
rows_id_repaired, so a future grid cannot report this as row loss.
One residual, stated rather than smoothed: core-tt-adv-sep / scale_1600 returned 1,698
identifiers for 1,600 truth rows, so it carries genuine duplication as well, and its
corrected figure is an upper bound. The other 104 affected runs returned exactly the truth
count of identifiers.
Flat is a defensible result for this release, but less obviously so than for 0.6.11.
0.6.11’s product changes were error-path fixes that only engage after a transient Bedrock
error, which corefast documents never hit — so flatness there was close to a prediction.
v0.6.12 does change code on the healthy path: extraction/service.py grew by 367 lines, the
row-shortfall check was re-derived, and truncation handling was added. The A/B says those
changes did not move accuracy or cost on this corpus. It does not say they moved nothing
anywhere: the corpus is three small documents (tiny_form, small_narrow, longdesc_100,
the largest being 100 rows over 4 pages), and the new row-shortfall and truncation signals are
by construction about large documents. The suite that would exercise them is the scaling
study, which was not run.
⚠️ One change this release makes to extracted values is not visible in this A/B at all.
The release validation record’s Finding 1 reports that six Payslip statutory-tax amounts
change sign across the 0.6.11 → 0.6.12 upgrade, reproducibly over three runs. That document
(samples/lending_package.pdf) is not in this corpus, and this corpus has no payslip class,
so no cell here could have detected it. A flat A/B is not evidence that extracted values are
unchanged — it is evidence about the cells and documents in the grid.
Provenance
Section titled “Provenance”| v0.6.11 (PREV) | v0.6.12 (NEW) | |
|---|---|---|
| template | published idp-main_0.6.11.yaml | published idp-main_0.6.12.yaml, Description (v0.6.12) |
| stack | IDPUpg0611to0612 — created fresh from the published 0.6.11 template | the same stack, after update-stack to 0.6.12 (the customer upgrade path), so wall-time deltas are comparable |
| suite | corefast (19 cells × 3 docs × 3 repeats) | same, identical config files (--native-upload) |
| runs / failures | 171 / 0 | 171 / 0 |
| run window (UTC) | 2026-10-16 22:48 → 23:53 | 2026-10-17 18:46 → 19:50 |
| extraction model | sonnet46 on all 171 runs | sonnet46 on all 171 runs |
| region / partition | us-west-2, commercial | same |
| corpus | regenerated from this checkout’s own generators, not reused | same corpus, same generator commit |
The control model is sonnet46 deliberately: it is the cross-version control, callable by
every release being compared. The shipped default is Sonnet 5, so no figure here is a
statement about the model a new deployment runs by default.
⚠️ Configs are built from the current repository’s defaults, so a fix that ships as default
configuration is present on both sides and cancels out. config_library/ changed between
these two tags only in pricing.yaml, model_config_limits.yaml and
extensions-marketplace.yaml — nothing in the corefast presets — so for this pair the two
sides differ in code alone, which is the cleanest form this comparison takes.
Grid totals
Section titled “Grid totals”| v0.6.11 | v0.6.12 | delta | |
|---|---|---|---|
| runs | 171 | 171 | — |
| successful runs | 171 | 171 | 0 failures either side |
| grid cost | $24.9546 | $24.9908 | +0.1% |
| grid wall | 10,357.8 s | 10,310.4 s | −0.5% |
| mean completeness recall | 1.0000 | 0.9978 | −0.2% |
| mean cell accuracy | 1.0000 | 0.9996 | −0.04% |
cells_compared | 23,370 | 23,294 | −0.3% |
| processing issues | 10 | 10 | — |
cells_compared is quoted beside the accuracy figures on purpose: a null column scores 1.000,
so an accuracy delta means nothing without the denominator it was computed over. It moved
−0.3%, small enough that the two sides’ accuracy figures are comparable.
Per-cell comparison
Section titled “Per-cell comparison”Mean completeness recall and mean cell accuracy over 9 runs per cell (3 docs × 3 repeats); cost is the cell’s total across those 9 runs.
| cell | recall 0.6.11 | recall 0.6.12 | cell acc 0.6.11 | cell acc 0.6.12 | cost 0.6.11 | cost 0.6.12 |
|---|---|---|---|---|---|---|
| core-bda-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.527 | $1.594 |
| core-bda-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.141 | $1.141 |
| core-llm-simple-sep | 1.000 | 0.958 | 1.000 | 0.993 | $1.060 | $1.056 |
| core-tl-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $2.074 | $2.074 |
| core-tl-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $0.870 | $0.871 |
| core-tt-adv-int | 1.000 | 1.000 | 1.000 | 1.000 | $2.145 | $2.091 |
| core-tt-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.625 | $1.647 |
| core-tt-simple-int | 1.000 | 1.000 | 1.000 | 1.000 | $1.142 | $1.143 |
| core-tt-simple-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.096 | $1.097 |
| core-tt-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.148 | $1.143 |
| enforce-escalate | 1.000 | 1.000 | 1.000 | 1.000 | $1.141 | $1.148 |
| enforce-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.148 | $1.154 |
| enforce-warn | 1.000 | 1.000 | 1.000 | 1.000 | $1.143 | $1.142 |
| force-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.142 | $1.143 |
| force-on | 1.000 | 1.000 | 1.000 | 1.000 | $1.141 | $1.141 |
| restate-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.520 | $1.544 |
| restate-on | 1.000 | 1.000 | 1.000 | 1.000 | $1.619 | $1.593 |
| split-disabled | 1.000 | 1.000 | 1.000 | 1.000 | $1.123 | $1.123 |
| split-llm | 1.000 | 1.000 | 1.000 | 1.000 | $1.147 | $1.148 |
Every cell is at ceiling on both metrics except core-llm-simple-sep. ⚠️ A grid that is
uniformly 1.000 is a statement about this corpus, not about the product: corefast’s three
documents are small and have exact ground truth, which is what makes it a cheap regression
detector and also what stops it discriminating between configurations. Nothing here supports
a recommendation between the nineteen cells; that is what the configuration matrix is for, and
it was not run.
Cost variance
Section titled “Cost variance”Per-cell cost CV at n=9 runs 0.47–0.62 on the agentic cells on both sides, so cost movement
was read against each cell’s own spread rather than against a flat percentage. The widest are
split-disabled (0.616), core-tt-simple-off (0.613), split-llm (0.609) and enforce-off
(0.599) on the new side. One per-document cost row, core-tt-adv-sep|tiny_form.pdf at +20%,
sits inside a 64% spread and is reported inconclusive.
With CVs that wide, no cost delta below roughly 50% on a single cell is readable at n=9. The grid-total +0.1% is meaningful only because it pools 171 runs.
Processing issues
Section titled “Processing issues”Both sides recorded 10, and both carry assessment_geometry_incomplete on 9 of 9 runs of
core-llm-simple-sep (severity info) — standing behaviour of that arm, unchanged.
The remaining issue is one assessment_truncated warning, and it moved cell: it sat on
core-tt-simple-sep at 0.6.11 and on enforce-escalate at 0.6.12. Same severity, same code,
same count, different cell — the signature of nondeterminism rather than of a change.
Figures
Section titled “Figures”Generated with aggregate.py --figures-compare into benchmarks/paper/figures/
(version_cost_compare.png, version_accuracy_compare.png) and not published with this
entry. That directory is tracked rather than ignored (benchmarks/.gitignore un-ignores
paper/figures/*.png), and earlier entries (v0.6.0, v0.6.5, v0.6.6 and v0.6.8) embed their
figures from the repository’s images/ directory.
Other tier F results
Section titled “Other tier F results”The A/B above answers “did this release regress?” and the answer is no. The other 28 result sets answer “what should a customer configure on this release?”, and three of their findings change what the guidance says. All three are in the configuration guidance; they are summarised here because a reader who comes to this page for the A/B would otherwise take “flat” as the whole story.
- ⚠️ A flat A/B is not evidence that extracted values are unchanged. The upgrade reproducibly flips the sign of six Payslip statutory-tax amounts and drops one populated field in another section — confirmed against two same-version control runs. That document is not in this corpus and this corpus has no payslip class, so no cell here could have seen it. It is Finding 1 of the release-validation record.
- On a real labelled corpus, simple mode beats advanced — at half the cost. The
synthetic grid scores both at 1.000 and cannot discriminate; RealKIE puts
split-disabledat 0.812 andcore-tt-simple-sepat 0.807 againstcore-tt-adv-sep’s 0.758, for $0.1423 per document against $0.2757. That is the single most useful thing the reference corpora bought back by running this time. - ⚠️ Claude Sonnet 5.5, new and selectable in this release, loses nearly the whole list on 400-row documents — 13 of 133 runs return 2 or 10 rows of 400 (#1494). Its headline cell accuracy of 1.0000 is computed over the 20 cells it returned.
Two results worth recording as closed rather than open: both standing list-loss hazards
(intconf, advverify) are clean at 8/8, which the v0.6.11 record could not say either
way.
Reproduce
Section titled “Reproduce”python benchmarks/harness/gen_corpus.pypython benchmarks/harness/make_configs.py --suite corefast# PREV side FIRST, on the 0.6.11 stack, before update-stackpython benchmarks/harness/run_matrix.py --stack <stack> --suite corefast \ --repeats 3 --native-upload --max-inflight 4python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.11-rvprev/corefast# ... update-stack to 0.6.12, then repeat on the same stack ...python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.12/corefast# NOTE the argument direction: --compare is the CURRENT side, --baseline the reference.python benchmarks/harness/aggregate.py \ --compare benchmarks/results/v0.6.12/corefast/summary.json \ --baseline benchmarks/results/v0.6.11-rvprev/corefast/summary.json