Benchmark: v0.6.14 vs v0.6.13
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
v0.6.14 vs v0.6.13 (published) — release benchmark audit
Section titled “v0.6.14 vs v0.6.13 (published) — release benchmark audit”Tier F is complete for v0.6.14. The release A/B below is one of 32 scored result sets: 31 pipeline sets holding 3,338 scored rows — 2,578 synthetic runs and 760 reference-corpus documents — for $1,970.57, plus a classification-variant benchmark scored by a different driver. All of it was measured on one stack at one build, on 2026-11-07 and 2026-11-09/10: a host restart on 2026-11-07 killed six suites mid-run, and those were discarded and re-run in full. Every section of the configuration guidance was re-measured except §2c’s second model arm, the confidence batch-size A/B, the multi-instance study and the two split-cost suites, which that page names.
Verdict: flat. The release A/B over 342 runs reports 0 regressions and 0
improvements, at grid level and at cell level. Mean completeness recall and mean cell
accuracy are 1.0000 on both sides, over 23,370 cells_compared on both sides, so the
accuracy comparison is on an identical denominator. Grid cost moved −3.5% and grid wall
−1.3%. Neither side had a failed run: 171 of 171 completed on each.
The two findings aggregate.py --compare reports are both inconclusive, and both are one
cell on one document: core-llm-simple-sep, the bedrock_llm OCR arm, on longdesc_100. Its
cost fell −45% per run (mean $0.170 ± $0.169 → $0.093 ± $0.063, n=9 each side) and its
scalar accuracy rose 0.889 → 1.000 against a ±0.333 paired spread. ⚠️ The asymmetry is on the
PREV side, and its mechanism is visible in the stored OCR output. On all three v0.6.13 runs of
that document the OCR model’s output for page 3 was a table header followed by roughly
950,000 spaces — a runaway to the output-token limit — so OCR emitted 12,811 output tokens
against 4,240 on all three v0.6.14 runs, with identical input tokens (1,316) and page images of
identical size. That one page explains the rest: OCR cost $0.045–0.047 against $0.012 (4×),
extraction $0.327–0.334 against $0.134 (2.5×), wall 248–270 s against 75–80 s, and on two of the
three PREV runs the classifier split the one-section document in two (sections: 2,
sections_correct: 0.0), with extraction_sparse and extraction_validation_failed. The third
PREV run kept one section and scored scalar accuracy 0.0. Recall stayed 1.000 over 200 cells on
every run of both sides.
Why it happened is not established. This release’s only OCR code changes are the removal of an unreachable per-page path and a page-rendering fix for annotation-held form values, which applies only to a PDF carrying form annotations; this synthetic document carries none, and its page images are byte-for-byte the same size on both builds. The v0.6.13 audit saw the same shape on two of three runs on 2026-10-31, and here it is three of three on 2026-11-06 and zero of three four hours later on 2026-11-07, which fits a model-side behaviour on one page better than a code change. The release A/B cannot separate the two; replaying one stored page image through the OCR call on both builds would.
Flat is the expected result for this pair. The corefast corpus is three documents of at
most 100 rows and 4 pages. The extraction changes in v0.6.14 that touch the healthy path —
the confidence step’s memory fix
(#1678),
the throttling and dropped-connection retries, the packet-window recovery — are about large
documents or about failure paths corefast never reaches. The large-document effects are
measured by the scaling study and the extraction-model sweep, summarised under “Other tier F
results” below.
Provenance
Section titled “Provenance”| v0.6.13 (PREV) | v0.6.14 (NEW) | |
|---|---|---|
| template | published idp-main_0.6.13.yaml | published idp-main_0.6.14.yaml |
| stack | IDPUpg0613to0614, created from the published 0.6.13 template | the same stack after update-stack to 0.6.14 (the customer upgrade path) |
| suite | corefast (19 cells × 3 docs × 3 repeats) | same, identical config files (--native-upload) |
| runs / failures | 171 / 0 | 171 / 0 |
| runs launched (UTC) | 2026-11-06 21:48 → 22:43 | 2026-11-07 01:27 → 02:25 |
| extraction model | sonnet46 on all 171 runs | sonnet46 on all 171 runs |
| region / partition | us-west-2, commercial | same |
| pricing | config_library/pricing.yaml, sha256 44a79aa06c15… | same file, same hash |
The control model is sonnet46 deliberately: it is the cross-version control, callable by
every release being compared. The shipped default is Sonnet 5, so no figure here is a
statement about the model a new deployment runs by default.
⚠️ Both summaries record stack_version: 0.6.14, and the PREV one is not mislabelled
data. The harness reads the stack’s CloudFormation Description when it scores a run
rather than when the run executes, and the PREV side was scored at 23:10 UTC, after the first
update-stack attempt had begun at 23:08. Every PREV run was launched before that. The previous
entries’ PREV summaries carry the same artefact. The first update attempt failed and was rolled
back; the NEW side ran only after the second attempt reached UPDATE_COMPLETE at 01:15 UTC (the
v0.6.14 release-validation record has the detail).
Both sides were run with --validate-against-release v0.6.13. That works on this release
because the harness now uploads a generated configuration unchanged when the target release
reads the nested classification blocks
(#1674),
so the bytes both sides received are the generated files with no flat-key copy added.
⚠️ Configs are built from the current repository’s defaults, so a fix that ships as default configuration is present on both sides and cancels out. That is the right property for isolating code, and it means this A/B says nothing about default-configuration changes in this release.
baseline.json is promoted by this release: it now holds the v0.6.14 corefast grid
(benchmarks/results/v0.6.14/corefast/summary.json, stack IDPUpg0613to0614, scored
2026-11-07T02:29:10Z), replacing the v0.6.13 grid. Both sides of this comparison were scored
with the same comparators and carry the calibration and confidence-coverage metrics natively, so
the promoted file needs no backfill. This comparison itself does not use baseline.json — the
reference is the same-stack PREV grid committed under
benchmarks/results/v0.6.13-rvprev/corefast/, which is the like-for-like side.
Grid totals
Section titled “Grid totals”| v0.6.13 | v0.6.14 | delta | |
|---|---|---|---|
| runs | 171 | 171 | — |
| successful runs | 171 | 171 | 0 failures either side |
| grid cost | $25.6317 | $24.7289 | −3.5% |
| grid wall (sum of runs) | 8,936.9 s | 8,818.8 s | −1.3% |
| mean completeness recall | 1.0000 | 1.0000 | — |
| mean cell accuracy | 1.0000 | 1.0000 | — |
cells_compared | 23,370 | 23,370 | — |
| processing issues | 18 | 13 | −5, see below |
cells_compared is quoted beside the accuracy figures because a null column scores 1.000,
so an accuracy figure means nothing without the denominator it was computed over. Here the
denominators are identical, so the two 1.0000 figures are the same achievement.
Both grid deltas are the one cell above. Without core-llm-simple-sep the other 18 cells
moved −0.9% in cost ($24.104 → $23.894) and +5.0% in wall (7,933 s → 8,331 s): all 18
are 2–15% slower, and the median run moved 54.2 s → 56.4 s. No other benchmark lane was running
on the stack during either side, so it is not lane load, but one side per release on different evenings does not separate a code
effect from Bedrock-side latency, and the rows record no per-phase wall. Read it as a measured
+5% on healthy-path wall that this A/B cannot attribute.
Per-cell comparison
Section titled “Per-cell comparison”Mean completeness recall and mean cell accuracy over 9 runs per cell (3 docs × 3 repeats); cost is the cell’s total across those 9 runs.
| cell | recall 0.6.13 | recall 0.6.14 | cell acc 0.6.13 | cell acc 0.6.14 | cost 0.6.13 | cost 0.6.14 |
|---|---|---|---|---|---|---|
| core-bda-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.522 | $1.519 |
| core-bda-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.135 | $1.136 |
| core-llm-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.528 | $0.835 |
| core-tl-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $2.353 | $2.254 |
| core-tl-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $0.856 | $0.865 |
| core-tt-adv-int | 1.000 | 1.000 | 1.000 | 1.000 | $2.324 | $2.148 |
| core-tt-adv-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.550 | $1.595 |
| core-tt-simple-int | 1.000 | 1.000 | 1.000 | 1.000 | $1.136 | $1.136 |
| core-tt-simple-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.096 | $1.097 |
| core-tt-simple-sep | 1.000 | 1.000 | 1.000 | 1.000 | $1.135 | $1.137 |
| enforce-escalate | 1.000 | 1.000 | 1.000 | 1.000 | $1.135 | $1.136 |
| enforce-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.141 | $1.141 |
| enforce-warn | 1.000 | 1.000 | 1.000 | 1.000 | $1.135 | $1.136 |
| force-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.135 | $1.136 |
| force-on | 1.000 | 1.000 | 1.000 | 1.000 | $1.134 | $1.135 |
| restate-off | 1.000 | 1.000 | 1.000 | 1.000 | $1.514 | $1.514 |
| restate-on | 1.000 | 1.000 | 1.000 | 1.000 | $1.549 | $1.548 |
| split-disabled | 1.000 | 1.000 | 1.000 | 1.000 | $1.115 | $1.121 |
| split-llm | 1.000 | 1.000 | 1.000 | 1.000 | $1.139 | $1.141 |
All 19 cells are at ceiling on both metrics on both sides. The one per-cell movement
--compare flags at document level besides core-llm-simple-sep is cost on tiny_form in the
two TABLES advanced cells (+19% and +25%, n=3 per side, against spreads of 63% and 65%), which it
marks inconclusive; tiny_form costs about $0.01–0.03 a run.
⚠️ A grid that is uniformly 1.000 is a statement about this corpus, not about the
product: corefast’s three documents are small and have exact ground truth, which makes
it a cheap regression detector and also stops it discriminating between configurations.
The configuration matrix is what separates them, and on this release it ran.
Cost variance
Section titled “Cost variance”Per-cell cost CV at n=9 is 0.49–0.72 on every cell of the v0.6.13 side apart from
core-llm-simple-sep at 0.99, and 0.45–0.71 on the v0.6.14 side with that cell at 0.67, because
the three documents differ 20× in row count. Cost movement was therefore read against each cell’s
own spread rather than a flat percentage. With CVs that wide, no cost delta below roughly 50%
on a single cell is readable at n=9, which is why the −45% on core-llm-simple-sep is reported
as inconclusive rather than as an improvement — even though, for that cell, the per-run OCR token
counts above show exactly where the difference came from.
By phase the grid moved: extraction $18.513 → $17.681, OCR $6.150 → $6.048, assessment $0.611 →
$0.615, classification $0.355 → $0.384. The extraction and OCR reductions are almost entirely the
three longdesc_100 runs above.
Processing issues
Section titled “Processing issues”The count fell from 18 to 13, and none of the movement is a regression.
| issue code | v0.6.13 | v0.6.14 | where | why it moved |
|---|---|---|---|---|
extraction_row_count_unverified | 14 | 12 | 12 on the Textract LAYOUT cells (core-tl-*, small_narrow and longdesc_100, every repeat) on both sides; 2 more on core-llm-simple-sep at v0.6.13 | LAYOUT emits no tables, so a 30+-row list has nothing to check its count against — instrumentation, every run at recall 1.000. The two extra at v0.6.13 are the split-document runs above |
extraction_sparse | 2 | 0 | the two split core-llm-simple-sep runs | gone with the split |
extraction_validation_failed | 2 | 0 | the same two runs | same |
assessment_truncated | 0 | 1 | split-disabled on longdesc_100, one repeat | a single draw on a 100-row confidence pass; the run is at recall and cell accuracy 1.000 |
Figures
Section titled “Figures”Not generated for this entry. Earlier entries (v0.6.0, v0.6.5, v0.6.6 and v0.6.8) embed
their figures from the repository’s images/ directory. aggregate.py --figures-compare
writes into benchmarks/paper/figures/, which is tracked (benchmarks/.gitignore
un-ignores paper/figures/*.png), so a generated figure shows up as a working-tree change;
copy the one you cite to images/benchmark-v<VERSION>-<name>.png.
Other tier F results
Section titled “Other tier F results”The A/B above answers “did this release regress?”, and the answer is no. The other 31 result sets answer “what should a customer configure on this release?”. Five of their findings change what the guidance says; all are in the configuration guidance, and they are summarised here because a reader who comes to this page for the A/B would otherwise take “flat” as the whole story.
- The confidence pass now covers most of a large simple-mode list, and no large-document run failed in it. In the scaling study the 1,200- and 1,600-row simple-mode documents return every row with 95% and 80% of them confidence-scored, against 33–38% and 19–23% at v0.6.13 (one draw per size here). At the 66-page ceiling Sonnet 5’s wide-shard agentic runs moved from 22–78% coverage to 97–99%. Every one of the 931 runs of the seven-arm extraction grid completed, where v0.6.13 lost two to the confidence step’s memory kill (#1678), which this release fixes.
- Both classification-variant failure modes are closed. Every image-bearing packet variant
now classifies the 18–20-page DocSplit packets it failed at v0.6.13
(#1676),
and text-only page granularity no longer fails a blank page
(#1677).
Packet granularity still places boundaries better than the page default (0.600–0.710 against
0.509 over 19 packets), but by less than v0.6.13 measured, and on the 14 packets both releases
scored two packet variants fell by about twice what one packet can move the mean. One packet
failed on four variants because the extraction model’s answer on its first section was
Output blocked by content filtering policy. - Both Opus models are at ceiling on the extraction grid again, and Opus 5.5 is 17% cheaper
for the same result — 133 of 133 runs each at recall 1.0000 and cell accuracy 1.0000 over
83,790
cells_compared, $105.95 against $127.08. The shipped default, Sonnet 5, is at recall 0.9463 over 72,364 cells, its shortfall entirely simple-mode truncation on the 17-page document (all 800 rows on 5 of 13 runs). - ⚠️ Claude Sonnet 5.5 still loses most of the list on 400-row documents (#1494): 15 of its 133 runs return 2 to 20 of 400 rows, all in simple mode; 13 of the 15 now carry a completeness issue.
- On the real RealKIE corpus the simple-versus-advanced accuracy gap is now inside the noise. The matched pair is 0.798 simple against 0.774 advanced at $0.1428 against $0.2825 per document, but four cells running the identical simple configuration span 0.771–0.799. The cost difference is the finding that has held across three releases.
Failure census over the whole refresh. 5 of the 3,338 scored rows did not complete (0.15%),
against 15 of 3,368 at v0.6.13 (0.45%). All five are simple-mode requests refused as too long on
the 66-page scale_3200.pdf, which is the expected outcome at that size: one on the scaling
study’s control-model cell and two each on astracap’s Sonnet 5 and GPT-6 Astra simple cells.
v0.6.13 had nine of the same refusal because it ran those two suites at three repeats where this
release ran their defaults (one and two), so the refusal rate on those cells is 100% on both
releases. v0.6.13’s other six — four reference-corpus SCORE_ERRORs and two #1678 memory kills —
did not occur. Six partial lanes killed by the host restart are not in this census: they were
discarded unscored and re-run in full.
Two results worth recording as closed rather than open: both standing list-loss hazards
(intconf, advverify) are clean at 8 of 8 runs, each suite at its default 4 repeats per arm
on this release.
Reproduce
Section titled “Reproduce”python benchmarks/harness/gen_corpus.pypython benchmarks/harness/make_configs.py --suite corefast# PREV side FIRST, on the 0.6.13 stack, before update-stackpython benchmarks/harness/run_matrix.py --stack <stack> --suite corefast \ --repeats 3 --native-upload --validate-against-release v0.6.13 --max-inflight 4python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.13-rvprev/corefast# ... update-stack to 0.6.14, then repeat on the same stack with the same flags ...python benchmarks/harness/aggregate.py --run <run dir> --out benchmarks/results/v0.6.14/corefast# NOTE the argument direction: --compare is the CURRENT side, --baseline the reference.python benchmarks/harness/aggregate.py \ --compare benchmarks/results/v0.6.14/corefast/summary.json \ --baseline benchmarks/results/v0.6.13-rvprev/corefast/summary.json