Release Benchmark Audit — v0.6.9
v0.6.9 vs v0.6.8 (published) — release benchmark audit
Section titled “v0.6.9 vs v0.6.8 (published) — release benchmark audit”Verdict: no completeness or throughput regression, and a real −21% cost reduction
concentrated entirely in Extraction. 171 of 171 corefast runs completed on each side,
completeness_recall 1.000 → 1.000 on every one of the 19 cells, 0 cell-level regressions.
Grid cost $31.75 → $25.06 per pass (−21.1%), of which all of it is Extraction
($24.58 → $17.96, −27%); OCR, Classification, Assessment and Summarization each moved ≤3%.
Wall time −21% (1,550 s → 1,225 s summed across cells). One real accuracy regression, on one
field of one document under one OCR backend: see § The bedrock_llm emphasis
change.
baseline.json is not promoted — see § Why the baseline is not
promoted.
⚠️ The v0.6.9 side is not the published template.
idp-main_0.6.9.yamlwas not present in the publicaws-ml-blog-*buckets when this ran, so the NEW side is apublish.py --clean-buildof the release commit deployed from a private bucket. The v0.6.8 side is the publishedidp-main_0.6.8.yaml. See the release-validation record.
| v0.6.8 (PREV) | v0.6.9 (NEW) | |
|---|---|---|
| template | published idp-main_0.6.8.yaml | a --clean-build of the release commit (not yet published) |
| stack | IDPUpg068to069 — created fresh from the published 0.6.8 template | the same stack, after update-stack to 0.6.9 (the customer upgrade path) |
| suite | corefast (19 cells × 3 docs × 3 repeats) | same, identical config files (--native-upload) |
| runs / failures | 171 / 0 | 171 / 0 |
| run window (UTC) | 03:27 → 04:00:29 (last run’s completion) | 04:31 → 05:51 |
update-stack window (UTC) | — | 04:09:47 → 04:31:44 |
| extraction model | sonnet46 on all 171 runs | sonnet46 on all 171 runs |
pricing.yaml sha256 | a8897364… | a8897364… (identical) |
| region | us-west-2 | us-west-2 |
| scored by | aggregate.py at 96b936a76 | the same |
One stack, strictly ordered, verified by run window. Both sides ran on one stack, so this
isolates the code. The ordering was checked rather than assumed: the latest of the 171 v0.6.8
runs completed at 04:00:29Z and update-stack did not begin until 04:09:47Z — a 9m18s
margin — and all 171 rows are COMPLETED. No v0.6.8 run overlapped the upgrade.
The comparison
Section titled “The comparison”Accuracy column is acc_mean — the same quantity aggregate.py --compare reports as acc.
Cost is per run, averaged over the 3 documents; cost CV is the 0.6.9 side’s own
run-to-run coefficient of variation, shown because it is what decides whether a
single-cell cost delta means anything.
| cell | runs / fail (0.6.8 → 0.6.9) | recall | accuracy | cost / run | cost CV (0.6.9) | wall / run |
|---|---|---|---|---|---|---|
core-bda-adv-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.319 → $0.166 (−48%) | 0.51 | 135 s → 68 s (−50%) |
core-bda-simple-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.128 → $0.127 (−1%) | 0.62 | 73 s → 62 s (−15%) |
core-llm-simple-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 0.833 | $0.120 → $0.093 (−23%) | 0.68 | 84 s → 52 s (−38%) |
core-tl-adv-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.235 → $0.262 (+12%) | 0.73 | 113 s → 112 s (−1%) |
core-tl-simple-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.097 → $0.097 (+0%) | 0.66 | 63 s → 63 s (+0%) |
core-tt-adv-int | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.375 → $0.261 (−30%) | 0.59 | 112 s → 75 s (−33%) |
core-tt-adv-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.321 → $0.173 (−46%) | 0.50 | 117 s → 66 s (−43%) |
core-tt-simple-int | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (+0%) | 0.62 | 63 s → 62 s (−2%) |
core-tt-simple-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.122 → $0.122 (+0%) | 0.61 | 36 s → 35 s (−3%) |
core-tt-simple-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (−0%) | 0.62 | 63 s → 62 s (−2%) |
enforce-escalate | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (−0%) | 0.62 | 64 s → 62 s (−3%) |
enforce-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.129 → $0.128 (−1%) | 0.62 | 66 s → 63 s (−5%) |
enforce-warn | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (−0%) | 0.62 | 65 s → 62 s (−4%) |
force-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (−0%) | 0.62 | 64 s → 62 s (−3%) |
force-on | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (−0%) | 0.60 | 62 s → 61 s (−2%) |
restate-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.355 → $0.168 (−53%) | 0.52 | 131 s → 68 s (−48%) |
restate-on | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.312 → $0.173 (−44%) | 0.50 | 112 s → 67 s (−40%) |
split-disabled | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.125 → $0.125 (−0%) | 0.62 | 62 s → 60 s (−3%) |
split-llm | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 0.944 | $0.128 → $0.127 (−0%) | 0.62 | 64 s → 61 s (−4%) |
| grid total | 171/0 → 171/0 | — | — | $3.528 → $2.785 (−21.1%) | — | 1,550 s → 1,225 s (−21%) |
aggregate.py --compare: 2 flagged regressions, 0 improvements, 0 cell-level regressions,
0 cell-level improvements, 8 inconclusive.
The cost reduction is real, and it is all in Extraction
Section titled “The cost reduction is real, and it is all in Extraction”By phase, summed over all 171 runs per side:
| phase | 0.6.8 | 0.6.9 | delta |
|---|---|---|---|
| Extraction | $24.583 | $17.960 | −27% |
| OCR | $6.093 | $6.056 | −1% |
| Assessment | $0.703 | $0.682 | −3% |
| Classification | $0.371 | $0.361 | −3% |
| Summarization | $0.002 | $0.002 | +2% |
| total | $31.751 | $25.061 | −21.1% |
Why this is not an artefact of the two cost-reporting fixes this release shipped. 0.6.9
fixes :1m models being reported at up to double their real cost
(#899)
and cache tokens being mispriced by substring match
(#926).
Neither can reach these numbers, for four independent reasons:
- The benchmark harness does not read the stack’s reported cost. It prices the raw
metering map itself in
benchmarks/harness/lib.py::price_metering, matching both the model key and the unit name exactly, never by substring — the reference implementation of the rule that #926 fixed in production. So the production pricing bug was never in these figures, which is what the CHANGELOG means by “nothing underbenchmarks/was affected”. - Both sides were priced from the same
config_library/pricing.yaml, sha256a8897364…, recorded in both summaries, and scored by the sameaggregate.pybuild. - #899 is
:1m-specific and every one of the 342 runs usedsonnet46, not a:1mvariant. - #926 affected only model ids with no pricing entry or an entry missing
cacheReadInputTokens— among shipped models, the EU Nova 2 Lite service tiers. This grid usedus.anthropic.claude-sonnet-4-6,us.amazon.nova-lite-v1:0andus.amazon.nova-2-lite-v1:0, all with complete entries.
Instrument-independent corroboration. Token counts are reported by the model API and are never priced, so no pricing change can move them. Summed across the grid, on the same documents:
| token class | 0.6.8 | 0.6.9 | delta |
|---|---|---|---|
cacheWriteInputTokens | 501,971 | 242,120 | −52% |
outputTokens | 1,683,442 | 1,448,936 | −14% |
inputTokens | 5,438,294 | 4,884,252 | −10% |
cacheReadInputTokens | 14,892,151 | 13,417,772 | −10% |
Less work is being done, not merely priced differently. These totals are across all models in the grid, so they corroborate the direction and magnitude but cannot be attributed to the extraction model alone — the per-phase table above is what localises it to Extraction.
What is not established: which 0.6.9 change produces the reduction. The candidates all ship in this release and this grid cannot separate them; the halving of cache writes alongside a proportional fall in cache reads points at fewer distinct request prefixes (fewer or better-batched extraction requests) rather than at cheaper requests. Cross-side prompt-cache carryover is not a plausible explanation: the two sides are ~30 minutes apart, far beyond the cache TTL, and cache reads fell rather than rose.
The one cell that got more expensive, core-tl-adv-sep at +12% on the mean and +22% on
longdesc_100.pdf specifically, is the second --compare regression and is noise: that
cell’s own cost CV is 0.73, --compare reports 0 cell-level regressions, and the same
configuration under core-tt-adv-sep is −46%. A +22% single-document move inside a cell with
73% run-to-run spread is not a signal.
The bedrock_llm OCR backend now emits markdown emphasis, and one extracted scalar changed with it
Section titled “The bedrock_llm OCR backend now emits markdown emphasis, and one extracted scalar changed with it”The one real accuracy regression. core-llm-simple-sep | small_narrow.pdf, scalar_accuracy
1.000 → 0.500, deterministic on 3/3 runs each side.
Everything else on that cell is unchanged and perfect: completeness_recall 1.000,
rows_extracted 100/100, per-row cell_accuracy 1.000 over cells_compared = 200,
sections_correct 1.000. No table data was lost — this is one of the document’s two
scalar fields, which is why a single field moves the cell’s scalar accuracy by 0.5.
Measured, from the S3 artefacts pulled before teardown:
| 0.6.8 (3/3 runs) | 0.6.9 (3/3 runs) | |
|---|---|---|
Account Number (truth 000123456789) | 000123456789 ✓ | 000123456789 ✓ |
Statement Period (truth 01/01/2024 - 12/31/2024) | 01/01/2024 - 12/31/2024 ✓ | January 2024 - December 2024 ✗ |
| OCR text for that line | Statement Period: 01/01/2024 - 12/31/2024 | **Statement Period:** 01/01/2024 - 12/31/2024 |
The OCR still contains the literal date range on both sides. What changed is that 0.6.9’s
bedrock_llm OCR wraps field labels in markdown bold; given that input, extraction returned a
re-worded range instead of copying the value through.
- Scope, measured: only the
bedrock_llmbackend. The same field on the same document reads01/01/2024 - 12/31/2024on both releases undertextract_tables,textract_layoutandbda. - Inferred, not proven: that the emphasis markup caused the re-wording. It is the only input difference, but no controlled with-and-without-bold probe was run.
- Unverified: whether other scalar fields or documents are affected. This was the only cell/document pair in 171 runs whose scalar accuracy moved.
split-llm’s 1.000 → 0.944 is the same backend on a different cell and is reported
inconclusive by --compare (paired per-doc deltas n=9, spread ±0.167).
Also present on both sides, so not a regression: bedrock_llm transcribes the account
number as 00012345678 (11 digits) where the document has 12, and the extractor returned the
correct 12-digit value anyway on both releases. That is the known LLM-OCR digit-fidelity
hazard, confirmed still open — and the second time this cell has produced a scoring artefact
that first looked like a code regression (v0.6.6’s entry records the first).
Accuracy numbers here are not comparable with the published v0.6.8 entry
Section titled “Accuracy numbers here are not comparable with the published v0.6.8 entry”0.6.9 upgrades stickler-eval 0.5.0 → 1.0.0, which infers evaluation comparators per field by
name as well as type, so an un-annotated schema no longer scores every string FUZZY@0.85.
Both sides of this A/B were scored by the same (new) scorer, so this table is internally
consistent. But its 0.6.8 column is not the 0.6.8 column in
v0.6.8.md, which was scored with the old comparators. Do not read across the
two documents; re-measure if you need that comparison.
This is also why the re-scored 0.6.8 corefast set is not committed under
benchmarks/results/v0.6.8/: that directory holds the set behind the published v0.6.8 entry
and is never overwritten. Per
RETENTION.md,
a re-run to answer a one-off question is not retained once its numbers are written into the
page — the table above is the durable record of that column.
Why the baseline is not promoted
Section titled “Why the baseline is not promoted”baseline.json still points at the v0.6.8 measurement. Promoting this run would make the next
release’s gate diff against numbers produced by a different scorer, so every un-annotated
string field would appear to move at v0.7.0 for reasons that have nothing to do with v0.7.0.
Promote it once a release is measured with stickler-eval 1.0.0 on both sides.
benchmarks/results/v0.6.9/corefast/ — summary.json, summary.csv, cell_stats.csv.