Skip to content

Benchmark: v0.6.12 vs v0.6.11

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

v0.6.12 vs v0.6.11 (published) — release benchmark audit

Section titled “v0.6.12 vs v0.6.11 (published) — release benchmark audit”

Tier F is complete for v0.6.12. The release A/B below is one of 29 scored result sets — 3,248 rows from 2,488 runs, $1,970.55 — measured in one session on one stack. The configuration matrix, the two reference corpora, the scaling study, the cost-variance runs, the standing-hazard suites, all nine extraction-model arms, the classification and confidence sweeps and all three premium head-to-heads ran. What did not re-run is named in the configuration guidance: five of §7’s knob A/Bs, and §2c’s second model arm.

Verdict: flat, with one arm inconclusive. The release A/B over 342 runs reports 0 regressions and 0 improvements. Grid cost moved +0.1% and wall −0.5%, both well inside run-to-run spread, and 18 of 19 cells are identical to three decimal places on both completeness recall and cell accuracy. Neither side had a single failed run.

⚠️ The one apparent change is a measurement defect

Section titled “⚠️ The one apparent change is a measurement defect”

core-llm-simple-sep, the bedrock_llm OCR arm, is reported at recall 1.000 → 0.958 and cell accuracy 1.000 → 0.993. The rows were never lost.

rows_extracted is the count of distinct recognisable SEQ identifiers, matched by a five-digit pattern. That backend perturbs a digit in fixed-width ids — duplicating the leading zero, so SEQ00042 becomes SEQ000042, which the pattern reads as int("00004") and collapses onto row 4. The count falls; no row is missing. Reproduced by probe against a blob containing every row, recovering all seven committed figure-shapes byte for byte, and corroborated by a field that uses no identifier at all: conf_rows_expected, derived from the extracted list’s own length, reads the full truth row count on all 75 affected runs that record it — med_narrow carries rows_extracted: 362 beside conf_rows_expected: 400, conf_rows_scored: 400.

In this A/B the movement is one run of nine (small_narrow, dups: 38, truncation_prefix: 5); the other eight are exactly 1.000, and the whole 0.6.11 side is 1.000 with dups: 0. Corrected, all 19 cells are identical and the A/B moved nothing at all.

This is not new to this release — it is nine releases old. 105 runs across 16 (cell, document, shape) groups from v0.5.16 to v0.6.12 all correct to recall 1.000. This arm was published between 0.5644 and 0.9796 mean recall over that span, correcting to 1.0000 on 31 of the 40 affected (grid, cell) means; the worst published figure, v0.6.6/corefast at 0.5644, was a complete extraction. It also reached cell_accuracy, and worse: the join keys on the same identifier, so a perturbed id could pair one row’s values against another row’s truth and report a value error that is an artefact of the join. cells_compared is exactly 2 × rows_extracted on all 90 affected runs, so accuracy was measured over 92.3% of the rows actually returned.

⚠️ So the standing narrative that this backend is “the noisiest of the nineteen” was an instrument reading, and every earlier entry that treated its movement as a product signal was reporting the same defect. The v0.6.6 entry had already root-caused the behaviour to identifier corruption and the metric was never corrected for it, which is how it resurfaced as a fresh finding in each release since. Tracked as #1500; the harness now canonicalises the identifier against the truth id set and records rows_id_repaired, so a future grid cannot report this as row loss.

One residual, stated rather than smoothed: core-tt-adv-sep / scale_1600 returned 1,698 identifiers for 1,600 truth rows, so it carries genuine duplication as well, and its corrected figure is an upper bound. The other 104 affected runs returned exactly the truth count of identifiers.

Flat is a defensible result for this release, but less obviously so than for 0.6.11. 0.6.11’s product changes were error-path fixes that only engage after a transient Bedrock error, which corefast documents never hit — so flatness there was close to a prediction. v0.6.12 does change code on the healthy path: extraction/service.py grew by 367 lines, the row-shortfall check was re-derived, and truncation handling was added. The A/B says those changes did not move accuracy or cost on this corpus. It does not say they moved nothing anywhere: the corpus is three small documents (tiny_form, small_narrow, longdesc_100, the largest being 100 rows over 4 pages), and the new row-shortfall and truncation signals are by construction about large documents. The suite that would exercise them is the scaling study, which was not run.

⚠️ One change this release makes to extracted values is not visible in this A/B at all. The release validation record’s Finding 1 reports that six Payslip statutory-tax amounts change sign across the 0.6.11 → 0.6.12 upgrade, reproducibly over three runs. That document (samples/lending_package.pdf) is not in this corpus, and this corpus has no payslip class, so no cell here could have detected it. A flat A/B is not evidence that extracted values are unchanged — it is evidence about the cells and documents in the grid.

v0.6.11 (PREV)v0.6.12 (NEW)
templatepublished idp-main_0.6.11.yamlpublished idp-main_0.6.12.yaml, Description (v0.6.12)
stackIDPUpg0611to0612 — created fresh from the published 0.6.11 templatethe same stack, after update-stack to 0.6.12 (the customer upgrade path), so wall-time deltas are comparable
suitecorefast (19 cells × 3 docs × 3 repeats)same, identical config files (--native-upload)
runs / failures171 / 0171 / 0
run window (UTC)2026-10-16 22:48 → 23:532026-10-17 18:46 → 19:50
extraction modelsonnet46 on all 171 runssonnet46 on all 171 runs
region / partitionus-west-2, commercialsame
corpusregenerated from this checkout’s own generators, not reusedsame corpus, same generator commit

The control model is sonnet46 deliberately: it is the cross-version control, callable by every release being compared. The shipped default is Sonnet 5, so no figure here is a statement about the model a new deployment runs by default.

⚠️ Configs are built from the current repository’s defaults, so a fix that ships as default configuration is present on both sides and cancels out. config_library/ changed between these two tags only in pricing.yaml, model_config_limits.yaml and extensions-marketplace.yaml — nothing in the corefast presets — so for this pair the two sides differ in code alone, which is the cleanest form this comparison takes.

v0.6.11v0.6.12delta
runs171171—
successful runs1711710 failures either side
grid cost$24.9546$24.9908+0.1%
grid wall10,357.8 s10,310.4 s−0.5%
mean completeness recall1.00000.9978−0.2%
mean cell accuracy1.00000.9996−0.04%
cells_compared23,37023,294−0.3%
processing issues1010—

cells_compared is quoted beside the accuracy figures on purpose: a null column scores 1.000, so an accuracy delta means nothing without the denominator it was computed over. It moved −0.3%, small enough that the two sides’ accuracy figures are comparable.

Mean completeness recall and mean cell accuracy over 9 runs per cell (3 docs × 3 repeats); cost is the cell’s total across those 9 runs.

cellrecall 0.6.11recall 0.6.12cell acc 0.6.11cell acc 0.6.12cost 0.6.11cost 0.6.12
core-bda-adv-sep1.0001.0001.0001.000$1.527$1.594
core-bda-simple-sep1.0001.0001.0001.000$1.141$1.141
core-llm-simple-sep1.0000.9581.0000.993$1.060$1.056
core-tl-adv-sep1.0001.0001.0001.000$2.074$2.074
core-tl-simple-sep1.0001.0001.0001.000$0.870$0.871
core-tt-adv-int1.0001.0001.0001.000$2.145$2.091
core-tt-adv-sep1.0001.0001.0001.000$1.625$1.647
core-tt-simple-int1.0001.0001.0001.000$1.142$1.143
core-tt-simple-off1.0001.0001.0001.000$1.096$1.097
core-tt-simple-sep1.0001.0001.0001.000$1.148$1.143
enforce-escalate1.0001.0001.0001.000$1.141$1.148
enforce-off1.0001.0001.0001.000$1.148$1.154
enforce-warn1.0001.0001.0001.000$1.143$1.142
force-off1.0001.0001.0001.000$1.142$1.143
force-on1.0001.0001.0001.000$1.141$1.141
restate-off1.0001.0001.0001.000$1.520$1.544
restate-on1.0001.0001.0001.000$1.619$1.593
split-disabled1.0001.0001.0001.000$1.123$1.123
split-llm1.0001.0001.0001.000$1.147$1.148

Every cell is at ceiling on both metrics except core-llm-simple-sep. ⚠️ A grid that is uniformly 1.000 is a statement about this corpus, not about the product: corefast’s three documents are small and have exact ground truth, which is what makes it a cheap regression detector and also what stops it discriminating between configurations. Nothing here supports a recommendation between the nineteen cells; that is what the configuration matrix is for, and it was not run.

Per-cell cost CV at n=9 runs 0.47–0.62 on the agentic cells on both sides, so cost movement was read against each cell’s own spread rather than against a flat percentage. The widest are split-disabled (0.616), core-tt-simple-off (0.613), split-llm (0.609) and enforce-off (0.599) on the new side. One per-document cost row, core-tt-adv-sep|tiny_form.pdf at +20%, sits inside a 64% spread and is reported inconclusive.

With CVs that wide, no cost delta below roughly 50% on a single cell is readable at n=9. The grid-total +0.1% is meaningful only because it pools 171 runs.

Both sides recorded 10, and both carry assessment_geometry_incomplete on 9 of 9 runs of core-llm-simple-sep (severity info) — standing behaviour of that arm, unchanged.

The remaining issue is one assessment_truncated warning, and it moved cell: it sat on core-tt-simple-sep at 0.6.11 and on enforce-escalate at 0.6.12. Same severity, same code, same count, different cell — the signature of nondeterminism rather than of a change.

Generated with aggregate.py --figures-compare into benchmarks/paper/figures/ (version_cost_compare.png, version_accuracy_compare.png) and not published with this entry. That directory is tracked rather than ignored (benchmarks/.gitignore un-ignores paper/figures/*.png), and earlier entries (v0.6.0, v0.6.5, v0.6.6 and v0.6.8) embed their figures from the repository’s images/ directory.

The A/B above answers “did this release regress?” and the answer is no. The other 28 result sets answer “what should a customer configure on this release?”, and three of their findings change what the guidance says. All three are in the configuration guidance; they are summarised here because a reader who comes to this page for the A/B would otherwise take “flat” as the whole story.

  • ⚠️ A flat A/B is not evidence that extracted values are unchanged. The upgrade reproducibly flips the sign of six Payslip statutory-tax amounts and drops one populated field in another section — confirmed against two same-version control runs. That document is not in this corpus and this corpus has no payslip class, so no cell here could have seen it. It is Finding 1 of the release-validation record.
  • On a real labelled corpus, simple mode beats advanced — at half the cost. The synthetic grid scores both at 1.000 and cannot discriminate; RealKIE puts split-disabled at 0.812 and core-tt-simple-sep at 0.807 against core-tt-adv-sep’s 0.758, for $0.1423 per document against $0.2757. That is the single most useful thing the reference corpora bought back by running this time.
  • ⚠️ Claude Sonnet 5.5, new and selectable in this release, loses nearly the whole list on 400-row documents — 13 of 133 runs return 2 or 10 rows of 400 (#1494). Its headline cell accuracy of 1.0000 is computed over the 20 cells it returned.

Two results worth recording as closed rather than open: both standing list-loss hazards (intconf, advverify) are clean at 8/8, which the v0.6.11 record could not say either way.

Terminal window
python benchmarks/harness/gen_corpus.py
python benchmarks/harness/make_configs.py --suite corefast
# PREV side FIRST, on the 0.6.11 stack, before update-stack
python benchmarks/harness/run_matrix.py --stack <stack> --suite corefast \
--repeats 3 --native-upload --max-inflight 4
python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.11-rvprev/corefast
# ... update-stack to 0.6.12, then repeat on the same stack ...
python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.12/corefast
# NOTE the argument direction: --compare is the CURRENT side, --baseline the reference.
python benchmarks/harness/aggregate.py \
--compare benchmarks/results/v0.6.12/corefast/summary.json \
--baseline benchmarks/results/v0.6.11-rvprev/corefast/summary.json