Skip to content

Benchmark: v0.6.14 vs v0.6.13

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

v0.6.14 vs v0.6.13 (published) — release benchmark audit

Section titled “v0.6.14 vs v0.6.13 (published) — release benchmark audit”

Tier F is complete for v0.6.14. The release A/B below is one of 32 scored result sets: 31 pipeline sets holding 3,338 scored rows — 2,578 synthetic runs and 760 reference-corpus documents — for $1,970.57, plus a classification-variant benchmark scored by a different driver. All of it was measured on one stack at one build, on 2026-11-07 and 2026-11-09/10: a host restart on 2026-11-07 killed six suites mid-run, and those were discarded and re-run in full. Every section of the configuration guidance was re-measured except §2c’s second model arm, the confidence batch-size A/B, the multi-instance study and the two split-cost suites, which that page names.

Verdict: flat. The release A/B over 342 runs reports 0 regressions and 0 improvements, at grid level and at cell level. Mean completeness recall and mean cell accuracy are 1.0000 on both sides, over 23,370 cells_compared on both sides, so the accuracy comparison is on an identical denominator. Grid cost moved −3.5% and grid wall −1.3%. Neither side had a failed run: 171 of 171 completed on each.

The two findings aggregate.py --compare reports are both inconclusive, and both are one cell on one document: core-llm-simple-sep, the bedrock_llm OCR arm, on longdesc_100. Its cost fell −45% per run (mean $0.170 ± $0.169 → $0.093 ± $0.063, n=9 each side) and its scalar accuracy rose 0.889 → 1.000 against a ±0.333 paired spread. ⚠️ The asymmetry is on the PREV side, and its mechanism is visible in the stored OCR output. On all three v0.6.13 runs of that document the OCR model’s output for page 3 was a table header followed by roughly 950,000 spaces — a runaway to the output-token limit — so OCR emitted 12,811 output tokens against 4,240 on all three v0.6.14 runs, with identical input tokens (1,316) and page images of identical size. That one page explains the rest: OCR cost $0.045–0.047 against $0.012 (4×), extraction $0.327–0.334 against $0.134 (2.5×), wall 248–270 s against 75–80 s, and on two of the three PREV runs the classifier split the one-section document in two (sections: 2, sections_correct: 0.0), with extraction_sparse and extraction_validation_failed. The third PREV run kept one section and scored scalar accuracy 0.0. Recall stayed 1.000 over 200 cells on every run of both sides.

Why it happened is not established. This release’s only OCR code changes are the removal of an unreachable per-page path and a page-rendering fix for annotation-held form values, which applies only to a PDF carrying form annotations; this synthetic document carries none, and its page images are byte-for-byte the same size on both builds. The v0.6.13 audit saw the same shape on two of three runs on 2026-10-31, and here it is three of three on 2026-11-06 and zero of three four hours later on 2026-11-07, which fits a model-side behaviour on one page better than a code change. The release A/B cannot separate the two; replaying one stored page image through the OCR call on both builds would.

Flat is the expected result for this pair. The corefast corpus is three documents of at most 100 rows and 4 pages. The extraction changes in v0.6.14 that touch the healthy path — the confidence step’s memory fix (#1678), the throttling and dropped-connection retries, the packet-window recovery — are about large documents or about failure paths corefast never reaches. The large-document effects are measured by the scaling study and the extraction-model sweep, summarised under “Other tier F results” below.

v0.6.13 (PREV)v0.6.14 (NEW)
templatepublished idp-main_0.6.13.yamlpublished idp-main_0.6.14.yaml
stackIDPUpg0613to0614, created from the published 0.6.13 templatethe same stack after update-stack to 0.6.14 (the customer upgrade path)
suitecorefast (19 cells × 3 docs × 3 repeats)same, identical config files (--native-upload)
runs / failures171 / 0171 / 0
runs launched (UTC)2026-11-06 21:48 → 22:432026-11-07 01:27 → 02:25
extraction modelsonnet46 on all 171 runssonnet46 on all 171 runs
region / partitionus-west-2, commercialsame
pricingconfig_library/pricing.yaml, sha256 44a79aa06c15…same file, same hash

The control model is sonnet46 deliberately: it is the cross-version control, callable by every release being compared. The shipped default is Sonnet 5, so no figure here is a statement about the model a new deployment runs by default.

⚠️ Both summaries record stack_version: 0.6.14, and the PREV one is not mislabelled data. The harness reads the stack’s CloudFormation Description when it scores a run rather than when the run executes, and the PREV side was scored at 23:10 UTC, after the first update-stack attempt had begun at 23:08. Every PREV run was launched before that. The previous entries’ PREV summaries carry the same artefact. The first update attempt failed and was rolled back; the NEW side ran only after the second attempt reached UPDATE_COMPLETE at 01:15 UTC (the v0.6.14 release-validation record has the detail).

Both sides were run with --validate-against-release v0.6.13. That works on this release because the harness now uploads a generated configuration unchanged when the target release reads the nested classification blocks (#1674), so the bytes both sides received are the generated files with no flat-key copy added.

⚠️ Configs are built from the current repository’s defaults, so a fix that ships as default configuration is present on both sides and cancels out. That is the right property for isolating code, and it means this A/B says nothing about default-configuration changes in this release.

baseline.json is promoted by this release: it now holds the v0.6.14 corefast grid (benchmarks/results/v0.6.14/corefast/summary.json, stack IDPUpg0613to0614, scored 2026-11-07T02:29:10Z), replacing the v0.6.13 grid. Both sides of this comparison were scored with the same comparators and carry the calibration and confidence-coverage metrics natively, so the promoted file needs no backfill. This comparison itself does not use baseline.json — the reference is the same-stack PREV grid committed under benchmarks/results/v0.6.13-rvprev/corefast/, which is the like-for-like side.

v0.6.13v0.6.14delta
runs171171—
successful runs1711710 failures either side
grid cost$25.6317$24.7289−3.5%
grid wall (sum of runs)8,936.9 s8,818.8 s−1.3%
mean completeness recall1.00001.0000—
mean cell accuracy1.00001.0000—
cells_compared23,37023,370—
processing issues1813−5, see below

cells_compared is quoted beside the accuracy figures because a null column scores 1.000, so an accuracy figure means nothing without the denominator it was computed over. Here the denominators are identical, so the two 1.0000 figures are the same achievement.

Both grid deltas are the one cell above. Without core-llm-simple-sep the other 18 cells moved −0.9% in cost ($24.104 → $23.894) and +5.0% in wall (7,933 s → 8,331 s): all 18 are 2–15% slower, and the median run moved 54.2 s → 56.4 s. No other benchmark lane was running on the stack during either side, so it is not lane load, but one side per release on different evenings does not separate a code effect from Bedrock-side latency, and the rows record no per-phase wall. Read it as a measured +5% on healthy-path wall that this A/B cannot attribute.

Mean completeness recall and mean cell accuracy over 9 runs per cell (3 docs × 3 repeats); cost is the cell’s total across those 9 runs.

cellrecall 0.6.13recall 0.6.14cell acc 0.6.13cell acc 0.6.14cost 0.6.13cost 0.6.14
core-bda-adv-sep1.0001.0001.0001.000$1.522$1.519
core-bda-simple-sep1.0001.0001.0001.000$1.135$1.136
core-llm-simple-sep1.0001.0001.0001.000$1.528$0.835
core-tl-adv-sep1.0001.0001.0001.000$2.353$2.254
core-tl-simple-sep1.0001.0001.0001.000$0.856$0.865
core-tt-adv-int1.0001.0001.0001.000$2.324$2.148
core-tt-adv-sep1.0001.0001.0001.000$1.550$1.595
core-tt-simple-int1.0001.0001.0001.000$1.136$1.136
core-tt-simple-off1.0001.0001.0001.000$1.096$1.097
core-tt-simple-sep1.0001.0001.0001.000$1.135$1.137
enforce-escalate1.0001.0001.0001.000$1.135$1.136
enforce-off1.0001.0001.0001.000$1.141$1.141
enforce-warn1.0001.0001.0001.000$1.135$1.136
force-off1.0001.0001.0001.000$1.135$1.136
force-on1.0001.0001.0001.000$1.134$1.135
restate-off1.0001.0001.0001.000$1.514$1.514
restate-on1.0001.0001.0001.000$1.549$1.548
split-disabled1.0001.0001.0001.000$1.115$1.121
split-llm1.0001.0001.0001.000$1.139$1.141

All 19 cells are at ceiling on both metrics on both sides. The one per-cell movement --compare flags at document level besides core-llm-simple-sep is cost on tiny_form in the two TABLES advanced cells (+19% and +25%, n=3 per side, against spreads of 63% and 65%), which it marks inconclusive; tiny_form costs about $0.01–0.03 a run.

⚠️ A grid that is uniformly 1.000 is a statement about this corpus, not about the product: corefast’s three documents are small and have exact ground truth, which makes it a cheap regression detector and also stops it discriminating between configurations. The configuration matrix is what separates them, and on this release it ran.

Per-cell cost CV at n=9 is 0.49–0.72 on every cell of the v0.6.13 side apart from core-llm-simple-sep at 0.99, and 0.45–0.71 on the v0.6.14 side with that cell at 0.67, because the three documents differ 20× in row count. Cost movement was therefore read against each cell’s own spread rather than a flat percentage. With CVs that wide, no cost delta below roughly 50% on a single cell is readable at n=9, which is why the −45% on core-llm-simple-sep is reported as inconclusive rather than as an improvement — even though, for that cell, the per-run OCR token counts above show exactly where the difference came from.

By phase the grid moved: extraction $18.513 → $17.681, OCR $6.150 → $6.048, assessment $0.611 → $0.615, classification $0.355 → $0.384. The extraction and OCR reductions are almost entirely the three longdesc_100 runs above.

The count fell from 18 to 13, and none of the movement is a regression.

issue codev0.6.13v0.6.14wherewhy it moved
extraction_row_count_unverified141212 on the Textract LAYOUT cells (core-tl-*, small_narrow and longdesc_100, every repeat) on both sides; 2 more on core-llm-simple-sep at v0.6.13LAYOUT emits no tables, so a 30+-row list has nothing to check its count against — instrumentation, every run at recall 1.000. The two extra at v0.6.13 are the split-document runs above
extraction_sparse20the two split core-llm-simple-sep runsgone with the split
extraction_validation_failed20the same two runssame
assessment_truncated01split-disabled on longdesc_100, one repeata single draw on a 100-row confidence pass; the run is at recall and cell accuracy 1.000

Not generated for this entry. Earlier entries (v0.6.0, v0.6.5, v0.6.6 and v0.6.8) embed their figures from the repository’s images/ directory. aggregate.py --figures-compare writes into benchmarks/paper/figures/, which is tracked (benchmarks/.gitignore un-ignores paper/figures/*.png), so a generated figure shows up as a working-tree change; copy the one you cite to images/benchmark-v<VERSION>-<name>.png.

The A/B above answers “did this release regress?”, and the answer is no. The other 31 result sets answer “what should a customer configure on this release?”. Five of their findings change what the guidance says; all are in the configuration guidance, and they are summarised here because a reader who comes to this page for the A/B would otherwise take “flat” as the whole story.

  • The confidence pass now covers most of a large simple-mode list, and no large-document run failed in it. In the scaling study the 1,200- and 1,600-row simple-mode documents return every row with 95% and 80% of them confidence-scored, against 33–38% and 19–23% at v0.6.13 (one draw per size here). At the 66-page ceiling Sonnet 5’s wide-shard agentic runs moved from 22–78% coverage to 97–99%. Every one of the 931 runs of the seven-arm extraction grid completed, where v0.6.13 lost two to the confidence step’s memory kill (#1678), which this release fixes.
  • Both classification-variant failure modes are closed. Every image-bearing packet variant now classifies the 18–20-page DocSplit packets it failed at v0.6.13 (#1676), and text-only page granularity no longer fails a blank page (#1677). Packet granularity still places boundaries better than the page default (0.600–0.710 against 0.509 over 19 packets), but by less than v0.6.13 measured, and on the 14 packets both releases scored two packet variants fell by about twice what one packet can move the mean. One packet failed on four variants because the extraction model’s answer on its first section was Output blocked by content filtering policy.
  • Both Opus models are at ceiling on the extraction grid again, and Opus 5.5 is 17% cheaper for the same result — 133 of 133 runs each at recall 1.0000 and cell accuracy 1.0000 over 83,790 cells_compared, $105.95 against $127.08. The shipped default, Sonnet 5, is at recall 0.9463 over 72,364 cells, its shortfall entirely simple-mode truncation on the 17-page document (all 800 rows on 5 of 13 runs).
  • ⚠️ Claude Sonnet 5.5 still loses most of the list on 400-row documents (#1494): 15 of its 133 runs return 2 to 20 of 400 rows, all in simple mode; 13 of the 15 now carry a completeness issue.
  • On the real RealKIE corpus the simple-versus-advanced accuracy gap is now inside the noise. The matched pair is 0.798 simple against 0.774 advanced at $0.1428 against $0.2825 per document, but four cells running the identical simple configuration span 0.771–0.799. The cost difference is the finding that has held across three releases.

Failure census over the whole refresh. 5 of the 3,338 scored rows did not complete (0.15%), against 15 of 3,368 at v0.6.13 (0.45%). All five are simple-mode requests refused as too long on the 66-page scale_3200.pdf, which is the expected outcome at that size: one on the scaling study’s control-model cell and two each on astracap’s Sonnet 5 and GPT-6 Astra simple cells. v0.6.13 had nine of the same refusal because it ran those two suites at three repeats where this release ran their defaults (one and two), so the refusal rate on those cells is 100% on both releases. v0.6.13’s other six — four reference-corpus SCORE_ERRORs and two #1678 memory kills — did not occur. Six partial lanes killed by the host restart are not in this census: they were discarded unscored and re-run in full.

Two results worth recording as closed rather than open: both standing list-loss hazards (intconf, advverify) are clean at 8 of 8 runs, each suite at its default 4 repeats per arm on this release.

Terminal window
python benchmarks/harness/gen_corpus.py
python benchmarks/harness/make_configs.py --suite corefast
# PREV side FIRST, on the 0.6.13 stack, before update-stack
python benchmarks/harness/run_matrix.py --stack <stack> --suite corefast \
--repeats 3 --native-upload --validate-against-release v0.6.13 --max-inflight 4
python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.13-rvprev/corefast
# ... update-stack to 0.6.14, then repeat on the same stack with the same flags ...
python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.14/corefast
# NOTE the argument direction: --compare is the CURRENT side, --baseline the reference.
python benchmarks/harness/aggregate.py \
--compare benchmarks/results/v0.6.14/corefast/summary.json \
--baseline benchmarks/results/v0.6.13-rvprev/corefast/summary.json