Benchmark: v0.6.11 vs v0.6.10
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
v0.6.11 vs v0.6.10 (published) — release benchmark audit
Section titled “v0.6.11 vs v0.6.10 (published) — release benchmark audit”Verdict: flat, and flat is the correct result for this release. The release A/B over 342 runs reports 0 regressions and 0 improvements: grid cost moved +0.4% and wall −2.9%, both inside run-to-run spread, and 17 of 19 cells are identical to three decimal places on both recall and cell accuracy.
That is what 0.6.11 should look like. Its product changes are error-path fixes — the
agentic read timeout, the retry ladder’s duplicated cachePoint, and the throttling
backoff — and they only engage after a transient Bedrock error. The corefast documents
never hit one, so a release that moved accuracy or cost here would be the surprising
outcome, not this.
The expensive half of this refresh (the configuration matrix, the scaling study, the model sweeps) is reported below the A/B and is where this release’s new information is: Claude Haiku 5.5 now has a measured accuracy and completeness figure, which its CHANGELOG entry explicitly said it did not.
⚠️ Its cost, however, is a lower bound rather than a measurement. Anthropic prices this
model in two prompt-length bands — 5× on every token type above 100,000 input tokens — while
pricing.yaml holds one flat rate per model. 105 of its 129 completed runs exceed that
threshold (median 543,675 input tokens, maximum 10.96M), so the reported $0.155/doc is a
floor; at the full 5× it would be ~$0.77/doc against Sonnet 5’s $0.587. No cost ratio for
Haiku 5.5 is quoted. Resolving it needs per-request token capture in the harness or a
two-band entry in pricing.yaml.
Provenance
Section titled “Provenance”| v0.6.10 (PREV) | v0.6.11 (NEW) | |
|---|---|---|
| template | published idp-main_0.6.10.yaml | published idp-main_0.6.11.yaml (LastModified 2026-10-09 21:56:34 GMT) |
| stack | IDPUpg0610to0611R — created fresh from the published 0.6.10 template | the same stack, after update-stack to 0.6.11 (the customer upgrade path) |
| suite | corefast (19 cells × 3 docs × 3 repeats) | same, identical config files (--native-upload) |
| runs / failures | 171 / 0 | 171 / 0 |
| run window (UTC) | 22:54 → 23:57 | 00:44 → 01:49 |
| extraction model | sonnet46 on all 171 runs | sonnet46 on all 171 runs |
pricing.yaml sha256 | 7726240470f1… | 7726240470f1… (identical) |
| region | us-west-2 | us-west-2 |
| scored by | aggregate.py at 8f6560d81 | the same |
Both sides ran alone, with no other suite in flight, so the wall-time column is comparable between them. Every other suite in this report ran later, at six concurrent lanes; their wall figures are not comparable with the A/B’s.
⚠️ Configs are built from the current repo’s defaults, so a fix that ships as default config is on both sides and cancels out. This A/B isolates code.
The comparison
Section titled “The comparison”aggregate.py --compare: 0 regressions, 0 improvements, 0 cell-level regressions, 0
cell-level improvements, 1 inconclusive.
| cell | runs/fail | recall | cell acc | cost/run | wall/run |
|---|---|---|---|---|---|
core-bda-adv-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.173 → $0.170 (−2%) | 63s → 62s (−2%) |
core-bda-simple-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.128 → $0.127 (−0%) | 62s → 59s (−5%) |
core-llm-simple-sep | 9/0 → 9/0 | 1.000 → 0.958 | 1.000 → 0.993 | $0.146 → $0.144 (−1%) | 117s → 91s (−22%) |
core-tl-adv-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.253 → $0.265 (+4%) | 95s → 94s (−1%) |
core-tl-simple-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.098 → $0.097 (−1%) | 77s → 60s (−21%) |
core-tt-adv-int | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.239 → $0.242 (+1%) | 60s → 61s (+2%) |
core-tt-adv-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.173 → $0.173 (+0%) | 64s → 65s (+2%) |
core-tt-simple-int | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (+0%) | 61s → 61s (+0%) |
core-tt-simple-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.122 → $0.122 (+0%) | 32s → 32s (+1%) |
core-tt-simple-sep | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.128 → $0.127 (−0%) | 62s → 59s (−4%) |
enforce-escalate | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.128 (+0%) | 59s → 62s (+4%) |
enforce-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.129 → $0.129 (+0%) | 58s → 62s (+7%) |
enforce-warn | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (+0%) | 59s → 60s (+2%) |
force-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (+0%) | 58s → 60s (+3%) |
force-on | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.127 → $0.127 (+0%) | 57s → 58s (+1%) |
restate-off | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.169 → $0.168 (−0%) | 64s → 63s (−2%) |
restate-on | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.173 → $0.177 (+2%) | 65s → 63s (−3%) |
split-disabled | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.125 → $0.125 (+0%) | 56s → 59s (+6%) |
split-llm | 9/0 → 9/0 | 1.000 → 1.000 | 1.000 → 1.000 | $0.128 → $0.128 (+0%) | 58s → 60s (+3%) |
| grid total | 171/0 → 171/0 | — | — | $2.818 → $2.829 (+0.4%) | 1,227s → 1,191s (−2.9%) |
The bedrock_llm OCR backend result (inconclusive)
Section titled “The bedrock_llm OCR backend result (inconclusive)”core-llm-simple-sep is the only cell outside noise on recall: 1.000 → 0.958, driven
by one document (small_narrow) at −0.127. aggregate.py --compare reports it
INCONCLUSIVE — the paired per-document spread for that cell is ±0.380, so a
−0.042 mean sits well inside it.
Two things make it worth a line anyway rather than silence. Cell accuracy barely moves (1.000 → 0.993) while row recall drops, which is the signature of rows going missing rather than values coming back wrong. And this is the third release in which this one backend is the only cell to move: v0.6.6 root-caused an apparent recall drop here to the backend corrupting fixed-width identifiers, and v0.6.9 recorded a deterministic scalar regression on it. Its wall time also fell 22% in the same comparison.
This is a standing caveat on ocr.backend: bedrock_llm — the least stable OCR option in the
matrix — rather than a 0.6.11 regression.
Cost by processing step
Section titled “Cost by processing step”The grid total’s +0.4% is not hiding a composition change: no phase moved more than its
own spread, and the two nominal cost rises (core-tl-adv-sep +4%, restate-on +2%) are
inside the per-cell spread.
⚠️ The pooled cost-CV warnings aggregate.py prints for this suite are not
instability. corefast pools three documents of very different size into one cell
statistic, so a cell CV of ~0.6 measures document-size spread. Measured within a
single (cell, document) pair across repeats, cost CV has a median of 0.005 and a
maximum of 0.465 over n=3. Quote per-document cost, never the pooled cell CV.
Failure census
Section titled “Failure census”90 non-COMPLETED of 4,794 scored rows (1.88%) across all 23 result sets, which matches the 90 Step Functions failures exactly. The A/B itself had zero failures on both sides; every failure below is in the F2/F3 suites.
| group | n | attribution |
|---|---|---|
simple mode on scale_3200 (astracap 6, scaling 9) | 15 | By design — this is the measurement. All four astracap advanced cells, including both 25-page-shard -wide arms, completed 3/3; only the two simple cells failed |
large_narrow and friends in simple mode | ~65 | A per-request size limit: the 300 s non-streaming converse read timeout. The identical documents complete in advanced (sharded) mode |
core-tt-adv-int on med_narrow (cost) | 5 | Product finding — see below |
Nothing in this census is attributable to 0.6.11: the 300 s timeout constant
(CONFIDENCE_READ_TIMEOUT_SECONDS = 300.0) is present at v0.6.10 unchanged, and
0.6.11’s timeout work raised the streamed agentic read timeout only.
Integrated confidence in advanced mode fails with Sonnet 4.6
Section titled “Integrated confidence in advanced mode fails with Sonnet 4.6”In the cost suite, cell core-tt-adv-int (extraction_mode: advanced,
assessment: integrated, extraction_model: sonnet46) failed 5 of 5 attempts on
med_narrow, at wall times of 1014.8–1021.6 s — a spread of ±7 s, i.e. a fixed
structure rather than model latency. For contrast:
- the same cell with
assessment: separatecompletes in 111–127 s; - the same cell at
extraction_model: sonnet5completed 25/25.
Mechanism, from the shard runtime’s own log:
tool_name=<provide_field_assessment> | replacing with error message due to max_tokens truncation.MaxTokensReachedException: Agent has reached an unrecoverable state due to max_tokens limitExtraction succeeds first — the same log records Extraction complete for shard 1 (pages 1-5): 239 transactions captured ... All data validated successfully. What exceeds
the budget is the per-field assessment tool output, emitted inline in the same agent
loop, on shards holding ~240 rows. Sonnet 5 survives because its output budget is far
larger.
Guidance: integrated confidence in advanced mode needs a large-output extraction model.
At the shipped control model it fails deterministically on a mid-size document, which is
not a capacity case. med_narrow is 10 pages.
Reproduce
Section titled “Reproduce”# A/B (both sides alone, for comparable wall times)python3 benchmarks/harness/make_configs.py --suite corefastpython3 benchmarks/harness/run_matrix.py --stack <STACK> --suite corefast --repeats 3 \ --native-upload --max-inflight 4python3 benchmarks/harness/aggregate.py --run benchmarks/results/run-<ts>-corefast-<hash> \ --out benchmarks/results/v0.6.11/corefastpython3 benchmarks/harness/aggregate.py \ --compare benchmarks/results/v0.6.11/corefast/summary.json \ --baseline benchmarks/results/v0.6.10-rvprev/corefast/summary.jsonNote the argument order: --compare takes the new summary and --baseline the older
one. Reversed, the signs invert and the one inconclusive cell reads +0.042 instead of
−0.042.
baseline.json is not promoted by this release. It still holds v0.6.8
(meta.stack_version), because neither 0.6.9 nor 0.6.10 promoted it; promoting it from
here would skip two releases’ worth of drift in one step without a record in between.
Prerelease measurements (2026-09-27, a private build of 3ca2dcc4d)
Section titled “Prerelease measurements (2026-09-27, a private build of 3ca2dcc4d)”The same corefast A/B was run before publication, against a private build of the release
commit, and reached the same verdict: 0 regressions, 0 improvements. Those measurements are
retained below for continuity. They cover the A/B only — the guidance refresh above is the
first for this release — and the configuration-matrix figures they cite are superseded by it.
Summary
Section titled “Summary”0 regressions, 0 improvements, 0 cell-level regressions, 0 cell-level improvements. Three inconclusive movements, all on one cell, all inside run-to-run spread.
That is the correct result for this release rather than a weak one, and the reason is the
size of the change. git diff v0.6.10..3ca2dcc4d touches exactly two files of product code
— idp_common/extraction/agentic_idp.py and idp_common/utils/bedrock_utils.py — and both
changes are on error paths: a retried Advanced-mode extraction now resumes the conversation
the failed attempt started instead of re-sending the prompt, and the Bedrock backoff ladder
now stops when its allowance is spent instead of clamping every remaining sleep to zero. A
document that never hits a transient Bedrock error enters neither. corefast runs healthy
documents, so it should not move, and it does not.
The corollary matters for anyone reading this as evidence: this grid does not exercise what
0.6.11 fixed. Confirming the fixes themselves requires inducing throttling or a transient
error, which no suite in benchmarks/ does.
Failure census
Section titled “Failure census”| Side | Runs | COMPLETED | Other |
|---|---|---|---|
| published v0.6.10 | 171 | 171 | 0 |
| v0.6.11 build | 171 | 171 | 0 |
Zero non-COMPLETED rows on either side. All 171 rows on both sides carry
cells_compared, so the “a null column scores 1.000” blind spot
(benchmarks/matrices/METHODOLOGY.md §4) does not apply to this comparison — the accuracy
figures are computed against populated columns on both sides.
Three inconclusive changes
Section titled “Three inconclusive changes”All on core-llm-simple-sep, the bedrock_llm OCR cell, which is the noisiest cell in this
grid across every release entry in this directory.
| Metric | Movement | Why it is not a finding |
|---|---|---|
| cost | −35% (0.143 ± 0.145 n9 → 0.093 ± 0.063 n9) | the standard deviation exceeds the shift on both sides |
| scalar accuracy | +0.278 (0.667 → 0.944) | paired per-document deltas over n=9 have spread ±0.441 |
| completeness recall | +0.042 (0.958 → 1.000) | paired per-document spread ±0.127 |
Two per-document notes from the same cell, also inconclusive:
small_narrow recall +0.127 against a spread of 0.380, and longdesc_100 scalar accuracy
+0.667 against a spread of 1.000.
The direction is favourable in all three, and a favourable inconclusive is still inconclusive. Resolving them needs repeats, not interpretation.
⚠️ Limitations of this entry
Section titled “⚠️ Limitations of this entry”Cost is not comparable at cell level on this suite. corefast’s cost CV is 0.50 to
1.02 across all 19 cells at n=9 on both sides. That is variance across the three different
documents inside each cell — tiny_form and longdesc_100 differ by more than an order of
magnitude — not run-to-run noise. A cell’s cost mean here is an average over unlike documents.
Cost claims for this release would have to come from the cost suite (n ≥ 5 on one 400-row
document), and that suite did not complete in this pass.
Wall-time is not comparable between the sides. They ran on different stacks: the
0.6.10 side on an existing v0.6.10 stack, reused rather than newly deployed to conserve IAM
role headroom, and the 0.6.11 side on the stack the release-validation upgrade tier had just
upgraded in place. Accuracy, completeness and cost are comparable, because the configuration
was byte-identical — generated once by make_configs.py and not regenerated between sides,
with --native-upload bypassing idp-cli’s migration on both.
baseline.json is deliberately not promoted. Nothing moved, so there is nothing to
promote, and promoting a grid whose cost axis is unusable at cell level would make the next
release’s --compare diff against a baseline this entry has told the reader not to trust.
Integrated confidence, re-measured
Section titled “Integrated confidence, re-measured”intconf — the suite that watches the integrated-confidence row-loss hazard — ran to
completion on the v0.6.11 stack and reads clean:
| Arm | Runs | completeness_recall | cell_accuracy | cells_compared | Confidence rows unscored |
|---|---|---|---|---|---|
core-tt-simple-int (integrated) | 4 | 1.000 | 1.000 | 200 | 0 of 100 |
core-tt-simple-sep (separate) | 4 | 1.000 | 1.000 | 200 | 0 of 100 |
The measure that matters for this hazard is the last column, because the failure mode is rows
silently arriving unscored rather than an accuracy drop. Zero unscored in both arms, over four
repeats each, says the hazard is closed at 0.6.11 on longdesc_100. It was open from v0.6.0
and first reported fixed at v0.6.6; this is a confirmation, not a new result.
Results: benchmarks/results/v0.6.11/intconf/.
⚠️ Configuration-guidance refresh (tier F) not completed
Section titled “⚠️ Configuration-guidance refresh (tier F) not completed”The release-validation battery plans a full guidance-paper refresh alongside the A/B, at
2,148 runs. This pass completed the A/B (342) plus intconf (8) and was then stopped.
| Lane | Planned | Outcome |
|---|---|---|
intconf | 8 | complete, scored, reported above |
core (§2 configuration matrix and real-world accuracy) | 90 | 20 launched, stopped, not scored |
scaling (§3 completeness cliffs) | 28 | 14 launched, stopped, not scored |
cost (§4 cost level and variance) | 25 | 14 launched, stopped, not scored |
advverify, the 7-value extraction_model sweep, astravalue, astracap, opus55value, the classification_model and confidence_model sweeps | 1,997 | not started |
The three partial lanes are deliberately left unscored. aggregate.py will score a
partial run and write a summary that looks complete, so leaving them unscored is the only
signal a later reader gets that they are partial.
Consequence, stated by name: every paper under docs/benchmarking/ still carries data from
an earlier release. config-guidance.md, index.md, classification-confidence.md,
prompt-caching.md and feature-multi-instance.md were not regenerated, and
config-guidance.md’s header still names an earlier release. In particular there is no
measured model-selection verdict for 0.6.11 — the premium and lightweight model sweeps did
not run, so the guide cannot say on this release which cheap model is at ceiling for a given
document profile or whether a premium model earns its price.
One part of the refresh would have been empty regardless: 0.6.11 changed no configuration knob. The §7 one-knob A/Bs had nothing to re-measure, and running them would have measured nothing about this release.
Reproduce
Section titled “Reproduce”cd benchmarkspython3 harness/make_configs.py --suite corefast # once; do not regenerate between sidespython3 harness/run_matrix.py --stack <0.6.10 stack> --suite corefast --native-upload --max-inflight 4python3 harness/run_matrix.py --stack <0.6.11 stack> --suite corefast --native-upload --max-inflight 4python3 harness/aggregate.py --run results/run-<prev> --out results/v0.6.11/corefast-prev0610python3 harness/aggregate.py --run results/run-<new> --out results/v0.6.11/corefastpython3 harness/aggregate.py --compare results/v0.6.11/corefast/summary.json \ --baseline results/v0.6.11/corefast-prev0610/summary.jsonNote that --compare pairs with --baseline; passing --compare alongside --run scores the
run and prints no comparison.