Skip to content

Benchmark: v0.6.13 vs v0.6.12

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

v0.6.13 vs v0.6.12 (published) — release benchmark audit

Section titled “v0.6.13 vs v0.6.12 (published) — release benchmark audit”

Tier F is complete for v0.6.13. The release A/B below is one of 32 scored result sets: 31 pipeline sets holding 3,368 scored rows — 2,608 synthetic runs and 760 reference-corpus documents — for $2,077.09, plus a classification-variant benchmark scored by a different driver. All of it was measured on one stack on 2026-10-31. Every section of the configuration guidance was re-measured except §2c’s second model arm, the confidence batch-size A/B, the multi-instance study and the two split-cost suites, which that page names.

Verdict: flat, with one arm inconclusive on cost. The release A/B over 342 runs reports 0 regressions and 0 improvements, at grid level and at cell level. Mean completeness recall and mean cell accuracy are 1.0000 on both sides, over 23,370 cells_compared on both sides, so the accuracy comparison is on an identical denominator. Grid cost moved +1.0%, inside run-to-run spread. Neither side had a failed run: 171 of 171 completed on each.

The one finding aggregate.py --compare reports is inconclusive: core-llm-simple-sep, the bedrock_llm OCR arm, costs +57% per run (mean $0.092 ± $0.062 → $0.145 ± $0.150, n=9 each side), and its longdesc_100 row is +101% against a 151% spread. Two of that cell’s three longdesc_100 runs on the new side cost $0.393 against $0.157 for the third and for all three on the old side. Both of the expensive runs carry assessment_truncated, extraction_sparse and extraction_validation_failed, while still scoring recall 1.000 and cell accuracy 1.000 over 200 cells. ⚠️ The extra cost is not a retry; those two runs differed from the start. Both split this one-section document into two sections (sections: 2, sections_correct: 0.0 against sections_expected: 1), and they already differed from the third run at OCR, which runs before classification: OCR cost $0.046 against $0.0115 (4×). Extraction then cost $0.334 against $0.134 (2.5×), with classification ($0.004) and assessment ($0.008 against $0.007) close to unchanged. No other run on either side split a document — 0 of 3 on v0.6.12, and every other v0.6.13 cell scores sections_correct 1.0 — so this is the grid’s one asymmetry between the sides. Two explanations fit and n = 3 per side cannot separate them: this arm’s OCR text is itself written by a model, and the 4× OCR cost shows those runs’ OCR output already differed before any section was formed; and this release changed how classification forms sections. Replaying one stored OCR output through both releases’ classification would separate them. Recall is unaffected because both sections’ rows were scored; what a deployment would see is the same document as two sections at about 2.5× the cost.

Flat is the expected result for this pair, for a reason worth stating. The corefast corpus is three documents of at most 100 rows and 4 pages. The extraction changes in v0.6.13 that touch the healthy path — the time-bounded confidence pass (#1488), the row-count-unverified issue (#1489), the narrower assessment_geometry_incomplete rule (#1520) and leaf-counted confidence batch sizing — are about large documents or about what gets reported, not about what gets extracted. The one that moves this grid is a reporting change (next section). The large-document effects are measured by the scaling study and the extraction-model sweep, summarised under “Other tier F results” below.

v0.6.12 (PREV)v0.6.13 (NEW)
templatepublished idp-main_0.6.12.yamlpublished idp-main_0.6.13.yaml
stackIDPUpg0612to0613, created from the published 0.6.12 templatethe same stack after update-stack to 0.6.13 (the customer upgrade path)
suitecorefast (19 cells × 3 docs × 3 repeats)same, identical config files (--native-upload)
runs / failures171 / 0171 / 0
runs launched (UTC)2026-10-31 02:29 → 03:332026-10-31 04:17 → 05:11
extraction modelsonnet46 on all 171 runssonnet46 on all 171 runs
region / partitionus-west-2, commercialsame
pricingconfig_library/pricing.yaml, sha256 44a79aa06c15…same file, same hash

The control model is sonnet46 deliberately: it is the cross-version control, callable by every release being compared. The shipped default is Sonnet 5, so no figure here is a statement about the model a new deployment runs by default.

⚠️ Both summaries record stack_version: 0.6.13, and the PREV one is not mislabelled data. The harness reads the stack’s CloudFormation Description when it scores a run rather than when the run executes, and the PREV side was scored after the upgrade had completed. Every PREV run was launched before update-stack began. The v0.6.12 entry’s PREV summary carries the same artefact (0.6.12 on its v0.6.11 side).

Both sides were run with --validate-against-release v0.6.12, including the NEW side. --validate-against-release v0.6.13 refuses the flat classification keys that --native-upload writes for a pre-0.9 release (#1674), so validating against the older release is what lets one set of config bytes reach both sides. The NEW stack converts those keys on read, so it ran the configuration the files describe.

⚠️ Configs are built from the current repository’s defaults, so a fix that ships as default configuration is present on both sides and cancels out. That is the right property for isolating code, and it means this A/B says nothing about the default-config changes in this release (for example, the classification settings moving inside each model call, config format 0.9).

baseline.json is promoted by this release: it now holds the v0.6.13 corefast grid (benchmarks/results/v0.6.13/corefast/summary.json), replacing the v0.6.8 grid it held since v0.6.8. Both sides of this comparison were scored with the same comparators and carry the calibration and confidence-coverage metrics natively, so the promoted file needs no backfill. This comparison itself does not use baseline.json — the reference is the same-stack PREV grid committed under benchmarks/results/v0.6.12-rvprev/corefast/, which is the like-for-like side.

v0.6.12v0.6.13delta
runs171171—
successful runs1711710 failures either side
grid cost$24.5960$24.8415+1.0%
grid wall (sum of runs)10,074.3 s8,464.7 s−16.0%
mean completeness recall1.00001.0000—
mean cell accuracy1.00001.0000—
cells_compared23,37023,370—
processing issues922+13, see below

cells_compared is quoted beside the accuracy figures because a null column scores 1.000, so an accuracy figure means nothing without the denominator it was computed over. Here the denominators are identical, so the two 1.0000 figures are the same achievement.

The wall-clock reduction is measured but not attributed. Both sides ran alone on the stack, so it is not a load artefact, and it is broad: 17 of the 19 cells are 14–26% faster, and by document it is concentrated on longdesc_100 (−24%) with tiny_form and small_narrow at −6% and −4%. The two cells that did not speed up are the ones that say where it came from. core-tt-simple-off, the only cell with the confidence pass switched off, moved only −7%, which points at the confidence pass; and core-llm-simple-sep was +92%, the two runs above that split the document into two sections. The assessment phase is also the one phase whose cost fell ($0.681 → $0.617 across the grid, −9%), which fits the leaf-counted batch sizing in this release, but the rows record no per-phase wall, so that attribution is an inference. Median run wall moved only 53.8 s → 51.8 s, so read the −16% as a statement about the longer runs.

Mean completeness recall and mean cell accuracy over 9 runs per cell (3 docs × 3 repeats); cost is the cell’s total across those 9 runs.

cellrecall 0.6.12recall 0.6.13cell acc 0.6.12cell acc 0.6.13cost 0.6.12cost 0.6.13
core-bda-adv-sep1.0001.0001.0001.000$1.557$1.513
core-bda-simple-sep1.0001.0001.0001.000$1.141$1.134
core-llm-simple-sep1.0001.0001.0001.000$0.831$1.304
core-tl-adv-sep1.0001.0001.0001.000$2.110$2.042
core-tl-simple-sep1.0001.0001.0001.000$0.870$0.856
core-tt-adv-int1.0001.0001.0001.000$2.089$2.075
core-tt-adv-sep1.0001.0001.0001.000$1.560$1.543
core-tt-simple-int1.0001.0001.0001.000$1.142$1.135
core-tt-simple-off1.0001.0001.0001.000$1.097$1.096
core-tt-simple-sep1.0001.0001.0001.000$1.141$1.134
enforce-escalate1.0001.0001.0001.000$1.142$1.140
enforce-off1.0001.0001.0001.000$1.148$1.139
enforce-warn1.0001.0001.0001.000$1.143$1.139
force-off1.0001.0001.0001.000$1.142$1.134
force-on1.0001.0001.0001.000$1.140$1.133
restate-off1.0001.0001.0001.000$1.518$1.520
restate-on1.0001.0001.0001.000$1.556$1.550
split-disabled1.0001.0001.0001.000$1.123$1.115
split-llm1.0001.0001.0001.000$1.145$1.139

All 19 cells are at ceiling on both metrics on both sides, including core-llm-simple-sep, which the v0.6.11 and v0.6.12 entries reported below 1.000. That arm’s apparent row loss was the identifier-collapse measurement defect (#1500), and the harness now canonicalises the row identifier against the truth id set before counting, so the grid no longer reports it. Both sides of this A/B were scored by the same harness, so the fix is on both sides and moves nothing between them.

⚠️ A grid that is uniformly 1.000 is a statement about this corpus, not about the product: corefast’s three documents are small and have exact ground truth, which makes it a cheap regression detector and also stops it discriminating between configurations. The configuration matrix is what separates them, and on this release it ran.

Per-cell cost CV at n=9 is 0.50–0.68 on every cell of the v0.6.12 side and 0.50–0.65 on the v0.6.13 side, apart from core-llm-simple-sep at 1.03 there, because the three documents differ 20× in row count. Cost movement was therefore read against each cell’s own spread rather than a flat percentage. With CVs that wide, no cost delta below roughly 50% on a single cell is readable at n=9, which is why the +57% on core-llm-simple-sep is reported as inconclusive rather than as a regression. The grid-total +1.0% is meaningful only because it pools 171 runs, and it is almost entirely that one cell: without it the other 18 cells moved −1.0% in total ($23.765 → $23.538).

The count rose from 9 to 22, and none of the rise is a regression.

issue codev0.6.12v0.6.13wherewhy it moved
extraction_row_count_unverified—1412 on the Textract LAYOUT cells (core-tl-*, small_narrow and longdesc_100, every repeat), 2 on core-llm-simple-sepnew code in this release (#1489). It reports that a list of 30+ rows had no exact-width OCR table to check its row count against — which is always true for LAYOUT, because LAYOUT emits no tables. Every one of the 14 runs is at recall 1.000. It is instrumentation, not a regression
assessment_geometry_incomplete90core-llm-simple-sep on every run at v0.6.12this release narrowed when the rung fires (#1520)
assessment_truncated042 on core-llm-simple-sep (longdesc_100), 1 each on enforce-warn and enforce-escalate (longdesc_100)single draws on a 100-row confidence pass; all four runs at recall and cell accuracy 1.000
extraction_sparse02the two expensive core-llm-simple-sep runsthe bedrock_llm arm’s run-to-run noise, see the verdict
extraction_validation_failed02the same two runssame

Not generated for this entry. Earlier entries (v0.6.0, v0.6.5, v0.6.6 and v0.6.8) embed their figures from the repository’s images/ directory. aggregate.py --figures-compare writes into benchmarks/paper/figures/, which is tracked (benchmarks/.gitignore un-ignores paper/figures/*.png), so a generated figure shows up as a working-tree change; copy the one you cite to images/benchmark-v<VERSION>-<name>.png.

The A/B above answers “did this release regress?”, and the answer is no. The other 31 result sets answer “what should a customer configure on this release?”. Five of their findings change what the guidance says; all are in the configuration guidance, and they are summarised here because a reader who comes to this page for the A/B would otherwise take “flat” as the whole story.

  • The large-document assessment failure is mostly gone. At v0.6.12, 15 of the 190 simple-mode, separate-confidence runs on the 17-page large_narrow document did not complete, almost all of them the AssessmentStep memory or wall-clock kill (#1488); on v0.6.13 2 of the same 190 did, both Runtime.OutOfMemory after a successful extraction (#1678). And in the scaling study the 1,200- and 1,600-row simple-mode documents, which failed outright at v0.6.12 with every row extracted, now complete with every row — but with assessment_incomplete and confidence on only 19–38% of the rows, because the confidence pass now stops at its time budget instead of being killed. That is the intended trade, and it means those rows reach a consumer with confidence: null.
  • On the extraction-model grid both Opus models are now at ceiling, and Opus 5.5 is 16% cheaper for the same result. Opus 5.5 and Opus 5 each completed 133 of 133 runs at recall 1.0000 and cell accuracy 1.0000 over 83,790 cells_compared, for $107.67 against $128.52 over the same grid. The shipped default, Sonnet 5, is at recall 0.9307 over 69,042 cells, its shortfall entirely simple-mode truncation on the 17-page document.
  • ⚠️ Claude Sonnet 5.5 still loses most of the list on 400-row documents (#1494): 16 of its 133 runs return 0 to 162 of 400 rows, 15 of them in simple mode, and 6 of the 16 carry no completeness issue at all.
  • Classification granularity, modality and strategy have their first measurement. On 20 DocSplit packets with Nova 2 Lite, packet granularity places section boundaries correctly far more often than the shipped page-level default — 0.711–0.821 against 0.446 boundary accuracy over the 15 packets every variant scored — at 23–65% of its classification cost per page. But the three packet variants that send page images failed the same three 18–20-page packets with Input is too long for requested model (#1676), and text-only page granularity failed two packets on a page with no OCR text (#1677). Twenty documents is a small sample.
  • ⚠️ sectionSplitting: disabled loses rows on a two-statement packet, not only the boundary. It merged the two statements into one section on 10 of 10 runs and extracted only the first statement’s 20 rows of 40 on 8 of them (recall 0.500), with no processing issue. This is not a regression: the committed v0.6.8 grid of the same suite shows 20 of 40 on 4 of 5. llm_determined split the packet correctly on 10 of 10 runs and returned all 40.

Failure census over the whole refresh. 15 of the 3,368 scored rows did not complete (0.45%), against 26 of 3,248 at v0.6.12 (0.80%). Nine are simple-mode requests refused as too long on the 66-page scale_3200.pdf, which is the expected outcome at that size (three each on the scaling study’s control-model cell and on astracap’s Sonnet 5 and GPT-6 Astra simple cells). Four are SCORE_ERROR on the OCR-benchmark corpus, where one grader leaf reported a confidence on a 0–100 scale (99.8 or 99.9); the harness refuses such a leaf rather than averaging it. v0.6.12 also had four, on the same corpus, two of them in the same cells. The remaining two are the #1678 memory kills above.

Two results worth recording as closed rather than open: both standing list-loss hazards (intconf, advverify) are clean at 6 of 6 runs, each suite at 3 repeats per arm on this release.

Terminal window
python benchmarks/harness/gen_corpus.py
python benchmarks/harness/make_configs.py --suite corefast
# PREV side FIRST, on the 0.6.12 stack, before update-stack
python benchmarks/harness/run_matrix.py --stack <stack> --suite corefast \
--repeats 3 --native-upload --validate-against-release v0.6.12 --max-inflight 4
python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.12-rvprev/corefast
# ... update-stack to 0.6.13, then repeat on the same stack with the same flags ...
python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.13/corefast
# NOTE the argument direction: --compare is the CURRENT side, --baseline the reference.
python benchmarks/harness/aggregate.py \
--compare benchmarks/results/v0.6.13/corefast/summary.json \
--baseline benchmarks/results/v0.6.12-rvprev/corefast/summary.json