Skip to content

Benchmark: v0.6.11 vs v0.6.10

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

v0.6.11 vs v0.6.10 (published) — release benchmark audit

Section titled “v0.6.11 vs v0.6.10 (published) — release benchmark audit”

Verdict: flat, and flat is the correct result for this release. The release A/B over 342 runs reports 0 regressions and 0 improvements: grid cost moved +0.4% and wall −2.9%, both inside run-to-run spread, and 17 of 19 cells are identical to three decimal places on both recall and cell accuracy.

That is what 0.6.11 should look like. Its product changes are error-path fixes — the agentic read timeout, the retry ladder’s duplicated cachePoint, and the throttling backoff — and they only engage after a transient Bedrock error. The corefast documents never hit one, so a release that moved accuracy or cost here would be the surprising outcome, not this.

The expensive half of this refresh (the configuration matrix, the scaling study, the model sweeps) is reported below the A/B and is where this release’s new information is: Claude Haiku 5.5 now has a measured accuracy and completeness figure, which its CHANGELOG entry explicitly said it did not.

⚠️ Its cost, however, is a lower bound rather than a measurement. Anthropic prices this model in two prompt-length bands — 5× on every token type above 100,000 input tokens — while pricing.yaml holds one flat rate per model. 105 of its 129 completed runs exceed that threshold (median 543,675 input tokens, maximum 10.96M), so the reported $0.155/doc is a floor; at the full 5× it would be ~$0.77/doc against Sonnet 5’s $0.587. No cost ratio for Haiku 5.5 is quoted. Resolving it needs per-request token capture in the harness or a two-band entry in pricing.yaml.

v0.6.10 (PREV)v0.6.11 (NEW)
templatepublished idp-main_0.6.10.yamlpublished idp-main_0.6.11.yaml (LastModified 2026-10-09 21:56:34 GMT)
stackIDPUpg0610to0611R — created fresh from the published 0.6.10 templatethe same stack, after update-stack to 0.6.11 (the customer upgrade path)
suitecorefast (19 cells × 3 docs × 3 repeats)same, identical config files (--native-upload)
runs / failures171 / 0171 / 0
run window (UTC)22:54 → 23:5700:44 → 01:49
extraction modelsonnet46 on all 171 runssonnet46 on all 171 runs
pricing.yaml sha2567726240470f1…7726240470f1… (identical)
regionus-west-2us-west-2
scored byaggregate.py at 8f6560d81the same

Both sides ran alone, with no other suite in flight, so the wall-time column is comparable between them. Every other suite in this report ran later, at six concurrent lanes; their wall figures are not comparable with the A/B’s.

⚠️ Configs are built from the current repo’s defaults, so a fix that ships as default config is on both sides and cancels out. This A/B isolates code.

aggregate.py --compare: 0 regressions, 0 improvements, 0 cell-level regressions, 0 cell-level improvements, 1 inconclusive.

cellruns/failrecallcell acccost/runwall/run
core-bda-adv-sep9/0 → 9/01.000 → 1.0001.000 → 1.000$0.173 → $0.170 (−2%)63s → 62s (−2%)
core-bda-simple-sep9/0 → 9/01.000 → 1.0001.000 → 1.000$0.128 → $0.127 (−0%)62s → 59s (−5%)
core-llm-simple-sep9/0 → 9/01.000 → 0.9581.000 → 0.993$0.146 → $0.144 (−1%)117s → 91s (−22%)
core-tl-adv-sep9/0 → 9/01.000 → 1.0001.000 → 1.000$0.253 → $0.265 (+4%)95s → 94s (−1%)
core-tl-simple-sep9/0 → 9/01.000 → 1.0001.000 → 1.000$0.098 → $0.097 (−1%)77s → 60s (−21%)
core-tt-adv-int9/0 → 9/01.000 → 1.0001.000 → 1.000$0.239 → $0.242 (+1%)60s → 61s (+2%)
core-tt-adv-sep9/0 → 9/01.000 → 1.0001.000 → 1.000$0.173 → $0.173 (+0%)64s → 65s (+2%)
core-tt-simple-int9/0 → 9/01.000 → 1.0001.000 → 1.000$0.127 → $0.127 (+0%)61s → 61s (+0%)
core-tt-simple-off9/0 → 9/01.000 → 1.0001.000 → 1.000$0.122 → $0.122 (+0%)32s → 32s (+1%)
core-tt-simple-sep9/0 → 9/01.000 → 1.0001.000 → 1.000$0.128 → $0.127 (−0%)62s → 59s (−4%)
enforce-escalate9/0 → 9/01.000 → 1.0001.000 → 1.000$0.127 → $0.128 (+0%)59s → 62s (+4%)
enforce-off9/0 → 9/01.000 → 1.0001.000 → 1.000$0.129 → $0.129 (+0%)58s → 62s (+7%)
enforce-warn9/0 → 9/01.000 → 1.0001.000 → 1.000$0.127 → $0.127 (+0%)59s → 60s (+2%)
force-off9/0 → 9/01.000 → 1.0001.000 → 1.000$0.127 → $0.127 (+0%)58s → 60s (+3%)
force-on9/0 → 9/01.000 → 1.0001.000 → 1.000$0.127 → $0.127 (+0%)57s → 58s (+1%)
restate-off9/0 → 9/01.000 → 1.0001.000 → 1.000$0.169 → $0.168 (−0%)64s → 63s (−2%)
restate-on9/0 → 9/01.000 → 1.0001.000 → 1.000$0.173 → $0.177 (+2%)65s → 63s (−3%)
split-disabled9/0 → 9/01.000 → 1.0001.000 → 1.000$0.125 → $0.125 (+0%)56s → 59s (+6%)
split-llm9/0 → 9/01.000 → 1.0001.000 → 1.000$0.128 → $0.128 (+0%)58s → 60s (+3%)
grid total171/0 → 171/0——$2.818 → $2.829 (+0.4%)1,227s → 1,191s (−2.9%)

The bedrock_llm OCR backend result (inconclusive)

Section titled “The bedrock_llm OCR backend result (inconclusive)”

core-llm-simple-sep is the only cell outside noise on recall: 1.000 → 0.958, driven by one document (small_narrow) at −0.127. aggregate.py --compare reports it INCONCLUSIVE — the paired per-document spread for that cell is ±0.380, so a −0.042 mean sits well inside it.

Two things make it worth a line anyway rather than silence. Cell accuracy barely moves (1.000 → 0.993) while row recall drops, which is the signature of rows going missing rather than values coming back wrong. And this is the third release in which this one backend is the only cell to move: v0.6.6 root-caused an apparent recall drop here to the backend corrupting fixed-width identifiers, and v0.6.9 recorded a deterministic scalar regression on it. Its wall time also fell 22% in the same comparison.

This is a standing caveat on ocr.backend: bedrock_llm — the least stable OCR option in the matrix — rather than a 0.6.11 regression.

The grid total’s +0.4% is not hiding a composition change: no phase moved more than its own spread, and the two nominal cost rises (core-tl-adv-sep +4%, restate-on +2%) are inside the per-cell spread.

⚠️ The pooled cost-CV warnings aggregate.py prints for this suite are not instability. corefast pools three documents of very different size into one cell statistic, so a cell CV of ~0.6 measures document-size spread. Measured within a single (cell, document) pair across repeats, cost CV has a median of 0.005 and a maximum of 0.465 over n=3. Quote per-document cost, never the pooled cell CV.

90 non-COMPLETED of 4,794 scored rows (1.88%) across all 23 result sets, which matches the 90 Step Functions failures exactly. The A/B itself had zero failures on both sides; every failure below is in the F2/F3 suites.

groupnattribution
simple mode on scale_3200 (astracap 6, scaling 9)15By design — this is the measurement. All four astracap advanced cells, including both 25-page-shard -wide arms, completed 3/3; only the two simple cells failed
large_narrow and friends in simple mode~65A per-request size limit: the 300 s non-streaming converse read timeout. The identical documents complete in advanced (sharded) mode
core-tt-adv-int on med_narrow (cost)5Product finding — see below

Nothing in this census is attributable to 0.6.11: the 300 s timeout constant (CONFIDENCE_READ_TIMEOUT_SECONDS = 300.0) is present at v0.6.10 unchanged, and 0.6.11’s timeout work raised the streamed agentic read timeout only.

Integrated confidence in advanced mode fails with Sonnet 4.6

Section titled “Integrated confidence in advanced mode fails with Sonnet 4.6”

In the cost suite, cell core-tt-adv-int (extraction_mode: advanced, assessment: integrated, extraction_model: sonnet46) failed 5 of 5 attempts on med_narrow, at wall times of 1014.8–1021.6 s — a spread of ±7 s, i.e. a fixed structure rather than model latency. For contrast:

  • the same cell with assessment: separate completes in 111–127 s;
  • the same cell at extraction_model: sonnet5 completed 25/25.

Mechanism, from the shard runtime’s own log:

tool_name=<provide_field_assessment> | replacing with error message due to max_tokens truncation.
MaxTokensReachedException: Agent has reached an unrecoverable state due to max_tokens limit

Extraction succeeds first — the same log records Extraction complete for shard 1 (pages 1-5): 239 transactions captured ... All data validated successfully. What exceeds the budget is the per-field assessment tool output, emitted inline in the same agent loop, on shards holding ~240 rows. Sonnet 5 survives because its output budget is far larger.

Guidance: integrated confidence in advanced mode needs a large-output extraction model. At the shipped control model it fails deterministically on a mid-size document, which is not a capacity case. med_narrow is 10 pages.

Terminal window
# A/B (both sides alone, for comparable wall times)
python3 benchmarks/harness/make_configs.py --suite corefast
python3 benchmarks/harness/run_matrix.py --stack <STACK> --suite corefast --repeats 3 \
--native-upload --max-inflight 4
python3 benchmarks/harness/aggregate.py --run benchmarks/results/run-<ts>-corefast-<hash> \
--out benchmarks/results/v0.6.11/corefast
python3 benchmarks/harness/aggregate.py \
--compare benchmarks/results/v0.6.11/corefast/summary.json \
--baseline benchmarks/results/v0.6.10-rvprev/corefast/summary.json

Note the argument order: --compare takes the new summary and --baseline the older one. Reversed, the signs invert and the one inconclusive cell reads +0.042 instead of −0.042.

baseline.json is not promoted by this release. It still holds v0.6.8 (meta.stack_version), because neither 0.6.9 nor 0.6.10 promoted it; promoting it from here would skip two releases’ worth of drift in one step without a record in between.


Prerelease measurements (2026-09-27, a private build of 3ca2dcc4d)

Section titled “Prerelease measurements (2026-09-27, a private build of 3ca2dcc4d)”

The same corefast A/B was run before publication, against a private build of the release commit, and reached the same verdict: 0 regressions, 0 improvements. Those measurements are retained below for continuity. They cover the A/B only — the guidance refresh above is the first for this release — and the configuration-matrix figures they cite are superseded by it.

0 regressions, 0 improvements, 0 cell-level regressions, 0 cell-level improvements. Three inconclusive movements, all on one cell, all inside run-to-run spread.

That is the correct result for this release rather than a weak one, and the reason is the size of the change. git diff v0.6.10..3ca2dcc4d touches exactly two files of product code — idp_common/extraction/agentic_idp.py and idp_common/utils/bedrock_utils.py — and both changes are on error paths: a retried Advanced-mode extraction now resumes the conversation the failed attempt started instead of re-sending the prompt, and the Bedrock backoff ladder now stops when its allowance is spent instead of clamping every remaining sleep to zero. A document that never hits a transient Bedrock error enters neither. corefast runs healthy documents, so it should not move, and it does not.

The corollary matters for anyone reading this as evidence: this grid does not exercise what 0.6.11 fixed. Confirming the fixes themselves requires inducing throttling or a transient error, which no suite in benchmarks/ does.

SideRunsCOMPLETEDOther
published v0.6.101711710
v0.6.11 build1711710

Zero non-COMPLETED rows on either side. All 171 rows on both sides carry cells_compared, so the “a null column scores 1.000” blind spot (benchmarks/matrices/METHODOLOGY.md §4) does not apply to this comparison — the accuracy figures are computed against populated columns on both sides.

All on core-llm-simple-sep, the bedrock_llm OCR cell, which is the noisiest cell in this grid across every release entry in this directory.

MetricMovementWhy it is not a finding
cost−35% (0.143 ± 0.145 n9 → 0.093 ± 0.063 n9)the standard deviation exceeds the shift on both sides
scalar accuracy+0.278 (0.667 → 0.944)paired per-document deltas over n=9 have spread ±0.441
completeness recall+0.042 (0.958 → 1.000)paired per-document spread ±0.127

Two per-document notes from the same cell, also inconclusive: small_narrow recall +0.127 against a spread of 0.380, and longdesc_100 scalar accuracy +0.667 against a spread of 1.000.

The direction is favourable in all three, and a favourable inconclusive is still inconclusive. Resolving them needs repeats, not interpretation.

Cost is not comparable at cell level on this suite. corefast’s cost CV is 0.50 to 1.02 across all 19 cells at n=9 on both sides. That is variance across the three different documents inside each cell — tiny_form and longdesc_100 differ by more than an order of magnitude — not run-to-run noise. A cell’s cost mean here is an average over unlike documents. Cost claims for this release would have to come from the cost suite (n ≥ 5 on one 400-row document), and that suite did not complete in this pass.

Wall-time is not comparable between the sides. They ran on different stacks: the 0.6.10 side on an existing v0.6.10 stack, reused rather than newly deployed to conserve IAM role headroom, and the 0.6.11 side on the stack the release-validation upgrade tier had just upgraded in place. Accuracy, completeness and cost are comparable, because the configuration was byte-identical — generated once by make_configs.py and not regenerated between sides, with --native-upload bypassing idp-cli’s migration on both.

baseline.json is deliberately not promoted. Nothing moved, so there is nothing to promote, and promoting a grid whose cost axis is unusable at cell level would make the next release’s --compare diff against a baseline this entry has told the reader not to trust.

intconf — the suite that watches the integrated-confidence row-loss hazard — ran to completion on the v0.6.11 stack and reads clean:

ArmRunscompleteness_recallcell_accuracycells_comparedConfidence rows unscored
core-tt-simple-int (integrated)41.0001.0002000 of 100
core-tt-simple-sep (separate)41.0001.0002000 of 100

The measure that matters for this hazard is the last column, because the failure mode is rows silently arriving unscored rather than an accuracy drop. Zero unscored in both arms, over four repeats each, says the hazard is closed at 0.6.11 on longdesc_100. It was open from v0.6.0 and first reported fixed at v0.6.6; this is a confirmation, not a new result.

Results: benchmarks/results/v0.6.11/intconf/.

⚠️ Configuration-guidance refresh (tier F) not completed

Section titled “⚠️ Configuration-guidance refresh (tier F) not completed”

The release-validation battery plans a full guidance-paper refresh alongside the A/B, at 2,148 runs. This pass completed the A/B (342) plus intconf (8) and was then stopped.

LanePlannedOutcome
intconf8complete, scored, reported above
core (§2 configuration matrix and real-world accuracy)9020 launched, stopped, not scored
scaling (§3 completeness cliffs)2814 launched, stopped, not scored
cost (§4 cost level and variance)2514 launched, stopped, not scored
advverify, the 7-value extraction_model sweep, astravalue, astracap, opus55value, the classification_model and confidence_model sweeps1,997not started

The three partial lanes are deliberately left unscored. aggregate.py will score a partial run and write a summary that looks complete, so leaving them unscored is the only signal a later reader gets that they are partial.

Consequence, stated by name: every paper under docs/benchmarking/ still carries data from an earlier release. config-guidance.md, index.md, classification-confidence.md, prompt-caching.md and feature-multi-instance.md were not regenerated, and config-guidance.md’s header still names an earlier release. In particular there is no measured model-selection verdict for 0.6.11 — the premium and lightweight model sweeps did not run, so the guide cannot say on this release which cheap model is at ceiling for a given document profile or whether a premium model earns its price.

One part of the refresh would have been empty regardless: 0.6.11 changed no configuration knob. The §7 one-knob A/Bs had nothing to re-measure, and running them would have measured nothing about this release.

Terminal window
cd benchmarks
python3 harness/make_configs.py --suite corefast # once; do not regenerate between sides
python3 harness/run_matrix.py --stack <0.6.10 stack> --suite corefast --native-upload --max-inflight 4
python3 harness/run_matrix.py --stack <0.6.11 stack> --suite corefast --native-upload --max-inflight 4
python3 harness/aggregate.py --run results/run-<prev> --out results/v0.6.11/corefast-prev0610
python3 harness/aggregate.py --run results/run-<new> --out results/v0.6.11/corefast
python3 harness/aggregate.py --compare results/v0.6.11/corefast/summary.json \
--baseline results/v0.6.11/corefast-prev0610/summary.json

Note that --compare pairs with --baseline; passing --compare alongside --run scores the run and prints no comparison.