Benchmark: v0.6.13 vs v0.6.12
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
v0.6.13 vs v0.6.12 (published) — release benchmark audit
Section titled “v0.6.13 vs v0.6.12 (published) — release benchmark audit”Tier F is complete for v0.6.13. The release A/B below is one of 32 scored result sets: 31 pipeline sets holding 3,368 scored rows — 2,608 synthetic runs and 760 reference-corpus documents — for $2,077.09, plus a classification-variant benchmark scored by a different driver. All of it was measured on one stack on 2026-10-31. Every section of the configuration guidance was re-measured except §2c’s second model arm, the confidence batch-size A/B, the multi-instance study and the two split-cost suites, which that page names.
Verdict: flat, with one arm inconclusive on cost. The release A/B over 342 runs reports
0 regressions and 0 improvements, at grid level and at cell level. Mean completeness
recall and mean cell accuracy are 1.0000 on both sides, over 23,370 cells_compared on
both sides, so the accuracy comparison is on an identical denominator. Grid cost moved
+1.0%, inside run-to-run spread. Neither side had a failed run: 171 of 171 completed
on each.
The one finding aggregate.py --compare reports is inconclusive: core-llm-simple-sep,
the bedrock_llm OCR arm, costs +57% per run (mean $0.092 ± $0.062 → $0.145 ± $0.150,
n=9 each side), and its longdesc_100 row is +101% against a 151% spread. Two of that
cell’s three longdesc_100 runs on the new side cost $0.393 against $0.157 for the third
and for all three on the old side. Both of the expensive runs carry assessment_truncated,
extraction_sparse and extraction_validation_failed, while still scoring recall 1.000
and cell accuracy 1.000 over 200 cells. ⚠️ The extra cost is not a retry; those two runs differed from the
start. Both split this one-section document into two sections (sections: 2,
sections_correct: 0.0 against sections_expected: 1), and they already differed from the
third run at OCR, which runs before classification: OCR cost $0.046 against $0.0115 (4×).
Extraction then cost $0.334 against $0.134 (2.5×), with classification ($0.004) and
assessment ($0.008 against $0.007) close to unchanged. No other run on either side split a document —
0 of 3 on v0.6.12, and every other v0.6.13 cell scores sections_correct 1.0 — so this is
the grid’s one asymmetry between the sides. Two explanations fit and n = 3 per side cannot
separate them: this arm’s OCR text is itself written by a model, and the 4× OCR cost shows
those runs’ OCR output already differed before any section was formed; and this release
changed how classification forms sections. Replaying one stored OCR output through both
releases’ classification would separate them. Recall is unaffected because both sections’
rows were scored; what a deployment would see is the same document as two sections at
about 2.5× the cost.
Flat is the expected result for this pair, for a reason worth stating. The corefast
corpus is three documents of at most 100 rows and 4 pages. The extraction changes in
v0.6.13 that touch the healthy path — the time-bounded confidence pass (#1488), the
row-count-unverified issue (#1489), the narrower assessment_geometry_incomplete rule
(#1520) and leaf-counted confidence batch sizing — are about large documents or about
what gets reported, not about what gets extracted. The one that moves this grid is a
reporting change (next section). The large-document effects are measured by the scaling
study and the extraction-model sweep, summarised under “Other tier F results” below.
Provenance
Section titled “Provenance”| v0.6.12 (PREV) | v0.6.13 (NEW) | |
|---|---|---|
| template | published idp-main_0.6.12.yaml | published idp-main_0.6.13.yaml |
| stack | IDPUpg0612to0613, created from the published 0.6.12 template | the same stack after update-stack to 0.6.13 (the customer upgrade path) |
| suite | corefast (19 cells × 3 docs × 3 repeats) | same, identical config files (--native-upload) |
| runs / failures | 171 / 0 | 171 / 0 |
| runs launched (UTC) | 2026-10-31 02:29 → 03:33 | 2026-10-31 04:17 → 05:11 |
| extraction model | sonnet46 on all 171 runs | sonnet46 on all 171 runs |
| region / partition | us-west-2, commercial | same |
| pricing | config_library/pricing.yaml, sha256 44a79aa06c15… | same file, same hash |
The control model is sonnet46 deliberately: it is the cross-version control, callable by
every release being compared. The shipped default is Sonnet 5, so no figure here is a
statement about the model a new deployment runs by default.
⚠️ Both summaries record stack_version: 0.6.13, and the PREV one is not mislabelled
data. The harness reads the stack’s CloudFormation Description when it scores a run
rather than when the run executes, and the PREV side was scored after the upgrade had
completed. Every PREV run was launched before update-stack began. The v0.6.12 entry’s
PREV summary carries the same artefact (0.6.12 on its v0.6.11 side).
Both sides were run with --validate-against-release v0.6.12, including the NEW side.
--validate-against-release v0.6.13 refuses the flat classification keys that
--native-upload writes for a pre-0.9 release
(#1674),
so validating against the older release is what lets one set of config bytes reach both
sides. The NEW stack converts those keys on read, so it ran the configuration the files
describe.
⚠️ Configs are built from the current repository’s defaults, so a fix that ships as default configuration is present on both sides and cancels out. That is the right property for isolating code, and it means this A/B says nothing about the default-config changes in this release (for example, the classification settings moving inside each model call, config format 0.9).
baseline.json is promoted by this release: it now holds the v0.6.13 corefast grid
(benchmarks/results/v0.6.13/corefast/summary.json), replacing the v0.6.8 grid it held
since v0.6.8. Both sides of this comparison were scored with the same comparators and carry
the calibration and confidence-coverage metrics natively, so the promoted file needs no
backfill. This comparison itself does not use baseline.json — the reference is the
same-stack PREV grid committed under benchmarks/results/v0.6.12-rvprev/corefast/, which is
the like-for-like side.
Grid totals
Section titled “Grid totals”| v0.6.12 | v0.6.13 | delta | |
|---|---|---|---|
| runs | 171 | 171 | — |
| successful runs | 171 | 171 | 0 failures either side |
| grid cost | $24.5960 | $24.8415 | +1.0% |
| grid wall (sum of runs) | 10,074.3 s | 8,464.7 s | −16.0% |
| mean completeness recall | 1.0000 | 1.0000 | — |
| mean cell accuracy | 1.0000 | 1.0000 | — |
cells_compared | 23,370 | 23,370 | — |
| processing issues | 9 | 22 | +13, see below |
cells_compared is quoted beside the accuracy figures because a null column scores 1.000,
so an accuracy figure means nothing without the denominator it was computed over. Here the
denominators are identical, so the two 1.0000 figures are the same achievement.
The wall-clock reduction is measured but not attributed. Both sides ran alone on the
stack, so it is not a load artefact, and it is broad: 17 of the 19 cells are 14–26%
faster, and by document it is concentrated on longdesc_100 (−24%) with tiny_form and
small_narrow at −6% and −4%. The two cells that did not speed up are the ones that say
where it came from. core-tt-simple-off, the only cell with the confidence pass switched
off, moved only −7%, which points at the confidence pass; and core-llm-simple-sep was
+92%, the two runs above that split the document into two sections. The assessment phase is also the one phase whose cost
fell ($0.681 → $0.617 across the grid, −9%), which fits the leaf-counted batch sizing in
this release, but the rows record no per-phase wall, so that attribution is an inference.
Median run wall moved only 53.8 s → 51.8 s, so read the −16% as a statement about the
longer runs.
Per-cell comparison
Section titled “Per-cell comparison”Mean completeness recall and mean cell accuracy over 9 runs per cell (3 docs × 3 repeats); cost is the cell’s total across those 9 runs.
| cell | recall 0.6.12 | recall 0.6.13 | cell acc 0.6.12 | cell acc 0.6.13 | cost 0.6.12 | cost 0.6.13 |
|---|---|---|---|---|---|---|
| core-bda-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.557 | $1.513 |
| core-bda-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.141 | $1.134 |
| core-llm-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $0.831 | $1.304 |
| core-tl-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $2.110 | $2.042 |
| core-tl-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $0.870 | $0.856 |
| core-tt-adv-int | 1.000 | 1.000 | 1.000 | 1.000 | $2.089 | $2.075 |
| core-tt-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.560 | $1.543 |
| core-tt-simple-int | 1.000 | 1.000 | 1.000 | 1.000 | $1.142 | $1.135 |
| core-tt-simple-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.097 | $1.096 |
| core-tt-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.141 | $1.134 |
| enforce-escalate | 1.000 | 1.000 | 1.000 | 1.000 | $1.142 | $1.140 |
| enforce-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.148 | $1.139 |
| enforce-warn | 1.000 | 1.000 | 1.000 | 1.000 | $1.143 | $1.139 |
| force-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.142 | $1.134 |
| force-on | 1.000 | 1.000 | 1.000 | 1.000 | $1.140 | $1.133 |
| restate-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.518 | $1.520 |
| restate-on | 1.000 | 1.000 | 1.000 | 1.000 | $1.556 | $1.550 |
| split-disabled | 1.000 | 1.000 | 1.000 | 1.000 | $1.123 | $1.115 |
| split-llm | 1.000 | 1.000 | 1.000 | 1.000 | $1.145 | $1.139 |
All 19 cells are at ceiling on both metrics on both sides, including
core-llm-simple-sep, which the v0.6.11 and v0.6.12 entries reported below 1.000. That
arm’s apparent row loss was the identifier-collapse measurement defect
(#1500),
and the harness now canonicalises the row identifier against the truth id set before
counting, so the grid no longer reports it. Both sides of this A/B were scored by the same
harness, so the fix is on both sides and moves nothing between them.
⚠️ A grid that is uniformly 1.000 is a statement about this corpus, not about the
product: corefast’s three documents are small and have exact ground truth, which makes
it a cheap regression detector and also stops it discriminating between configurations.
The configuration matrix is what separates them, and on this release it ran.
Cost variance
Section titled “Cost variance”Per-cell cost CV at n=9 is 0.50–0.68 on every cell of the v0.6.12 side and 0.50–0.65 on
the v0.6.13 side, apart from core-llm-simple-sep at 1.03 there, because the three
documents differ 20× in row count. Cost movement was therefore read against each
cell’s own spread rather than a flat percentage. With CVs that wide, no cost delta below
roughly 50% on a single cell is readable at n=9, which is why the +57% on
core-llm-simple-sep is reported as inconclusive rather than as a regression. The
grid-total +1.0% is meaningful only because it pools 171 runs, and it is almost entirely
that one cell: without it the other 18 cells moved −1.0% in total ($23.765 → $23.538).
Processing issues
Section titled “Processing issues”The count rose from 9 to 22, and none of the rise is a regression.
| issue code | v0.6.12 | v0.6.13 | where | why it moved |
|---|---|---|---|---|
extraction_row_count_unverified | — | 14 | 12 on the Textract LAYOUT cells (core-tl-*, small_narrow and longdesc_100, every repeat), 2 on core-llm-simple-sep | new code in this release (#1489). It reports that a list of 30+ rows had no exact-width OCR table to check its row count against — which is always true for LAYOUT, because LAYOUT emits no tables. Every one of the 14 runs is at recall 1.000. It is instrumentation, not a regression |
assessment_geometry_incomplete | 9 | 0 | core-llm-simple-sep on every run at v0.6.12 | this release narrowed when the rung fires (#1520) |
assessment_truncated | 0 | 4 | 2 on core-llm-simple-sep (longdesc_100), 1 each on enforce-warn and enforce-escalate (longdesc_100) | single draws on a 100-row confidence pass; all four runs at recall and cell accuracy 1.000 |
extraction_sparse | 0 | 2 | the two expensive core-llm-simple-sep runs | the bedrock_llm arm’s run-to-run noise, see the verdict |
extraction_validation_failed | 0 | 2 | the same two runs | same |
Figures
Section titled “Figures”Not generated for this entry. Earlier entries (v0.6.0, v0.6.5, v0.6.6 and v0.6.8) embed
their figures from the repository’s images/ directory. aggregate.py --figures-compare
writes into benchmarks/paper/figures/, which is tracked (benchmarks/.gitignore
un-ignores paper/figures/*.png), so a generated figure shows up as a working-tree change;
copy the one you cite to images/benchmark-v<VERSION>-<name>.png.
Other tier F results
Section titled “Other tier F results”The A/B above answers “did this release regress?”, and the answer is no. The other 31 result sets answer “what should a customer configure on this release?”. Five of their findings change what the guidance says; all are in the configuration guidance, and they are summarised here because a reader who comes to this page for the A/B would otherwise take “flat” as the whole story.
- The large-document assessment failure is mostly gone. At v0.6.12, 15 of the 190
simple-mode, separate-confidence runs on the 17-page
large_narrowdocument did not complete, almost all of them theAssessmentStepmemory or wall-clock kill (#1488); on v0.6.13 2 of the same 190 did, bothRuntime.OutOfMemoryafter a successful extraction (#1678). And in the scaling study the 1,200- and 1,600-row simple-mode documents, which failed outright at v0.6.12 with every row extracted, now complete with every row — but withassessment_incompleteand confidence on only 19–38% of the rows, because the confidence pass now stops at its time budget instead of being killed. That is the intended trade, and it means those rows reach a consumer withconfidence: null. - On the extraction-model grid both Opus models are now at ceiling, and Opus 5.5 is 16%
cheaper for the same result. Opus 5.5 and Opus 5 each completed 133 of 133 runs at
recall 1.0000 and cell accuracy 1.0000 over 83,790
cells_compared, for $107.67 against $128.52 over the same grid. The shipped default, Sonnet 5, is at recall 0.9307 over 69,042 cells, its shortfall entirely simple-mode truncation on the 17-page document. - ⚠️ Claude Sonnet 5.5 still loses most of the list on 400-row documents (#1494): 16 of its 133 runs return 0 to 162 of 400 rows, 15 of them in simple mode, and 6 of the 16 carry no completeness issue at all.
- Classification granularity, modality and strategy have their first measurement. On
20 DocSplit packets with Nova 2 Lite, packet granularity places section boundaries
correctly far more often than the shipped page-level default — 0.711–0.821 against 0.446
boundary accuracy over the 15 packets every variant scored — at 23–65% of its
classification cost per page. But the three packet variants that send page images failed
the same three 18–20-page packets with
Input is too long for requested model(#1676), and text-only page granularity failed two packets on a page with no OCR text (#1677). Twenty documents is a small sample. - ⚠️
sectionSplitting: disabledloses rows on a two-statement packet, not only the boundary. It merged the two statements into one section on 10 of 10 runs and extracted only the first statement’s 20 rows of 40 on 8 of them (recall 0.500), with no processing issue. This is not a regression: the committed v0.6.8 grid of the same suite shows 20 of 40 on 4 of 5.llm_determinedsplit the packet correctly on 10 of 10 runs and returned all 40.
Failure census over the whole refresh. 15 of the 3,368 scored rows did not complete
(0.45%), against 26 of 3,248 at v0.6.12 (0.80%). Nine are simple-mode requests refused as
too long on the 66-page scale_3200.pdf, which is the expected outcome at that size (three
each on the scaling study’s control-model cell and on astracap’s Sonnet 5 and GPT-6
Astra simple cells). Four are SCORE_ERROR on the OCR-benchmark corpus, where one grader
leaf reported a confidence on a 0–100 scale (99.8 or 99.9); the harness refuses such a
leaf rather than averaging it. v0.6.12 also had four, on the same corpus, two of them in
the same cells. The remaining two are the #1678 memory kills above.
Two results worth recording as closed rather than open: both standing list-loss hazards
(intconf, advverify) are clean at 6 of 6 runs, each suite at 3 repeats per arm on
this release.
Reproduce
Section titled “Reproduce”python benchmarks/harness/gen_corpus.pypython benchmarks/harness/make_configs.py --suite corefast# PREV side FIRST, on the 0.6.12 stack, before update-stackpython benchmarks/harness/run_matrix.py --stack <stack> --suite corefast \ --repeats 3 --native-upload --validate-against-release v0.6.12 --max-inflight 4python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.12-rvprev/corefast# ... update-stack to 0.6.13, then repeat on the same stack with the same flags ...python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.13/corefast# NOTE the argument direction: --compare is the CURRENT side, --baseline the reference.python benchmarks/harness/aggregate.py \ --compare benchmarks/results/v0.6.13/corefast/summary.json \ --baseline benchmarks/results/v0.6.12-rvprev/corefast/summary.json