Release Benchmark Audit — v0.6.7
v0.6.7 vs v0.6.6 — release benchmark audit
Section titled “v0.6.7 vs v0.6.6 — release benchmark audit”Verdict: safe to upgrade, with one cost caveat that is worth acting on. Zero
failures on either side of the accuracy gate, completeness unchanged, field accuracy
improved on four cells — and advanced (agentic) extraction costs 22–25% more, which a
follow-up same-stack A/B settled (n=5, ~7 SE) and traced to a specific interaction: v0.6.7
correctly starts running classification on single-class configs (#686), and the
now-active classifier over-splits one document into 2–4 sections (#726), each of which
costs a whole agent loop. Nothing in the agentic path regressed — with the section count
corrected it is 7% cheaper than v0.6.6. sectionSplitting: disabled removes the
premium for single-class corpora (but is wrong for packets); Sonnet 5 classification also
removes it but costs simple-mode users +45%.
| baseline | this release | |
|---|---|---|
| version | v0.6.6 | v0.6.7.dev8 |
| stack | IDPRel066 | IDP1 |
| commit | 7fb426b27 | 15d3fd212 |
| suite | corefast (10 cells × 3 docs × 3 repeats) | corefast (19 cells × 3 docs × 3 repeats) |
| runs | 90 | 172 — 163 of the 171-run grid, plus core-llm-simple-sep re-run to full sample (see Completeness of this run) |
| failures | 0 | 0 |
| region | us-west-2 | us-west-2 |
Costs are estimates from config_library/pricing.yaml. The baseline’s pricing hash
differs from this run’s, but the only removal between the two files is the retired
claude-3-5-haiku and no rate changed for any model these cells use
(sonnet-4-6, nova-lite, nova-2-lite, Textract) — verified by diffing the file
across both commits. Cost deltas are therefore not a pricing artifact. Rates as of
2026-09-02.
The comparison
Section titled “The comparison”--compare reports 0 cell-level regressions.
| cell | recall | scalar accuracy | cost | failures |
|---|---|---|---|---|
| core-tt-simple-sep | 1.000 → 1.000 | 0.833 → 1.000 | 0.112 → 0.116 | 0 → 0 |
| core-tt-simple-int | 1.000 → 1.000 | 1.000 → 1.000 | 0.171 → 0.179 | 0 → 0 |
| core-tt-simple-off | 1.000 → 1.000 | 0.833 → 1.000 | 0.105 → 0.109 | 0 → 0 |
| core-tt-adv-sep | 1.000 → 1.000 | 1.000 → 1.000 | 0.360 → 0.452 | 0 → 0 |
| core-tt-adv-int | 1.000 → 1.000 | 1.000 → 1.000 | 0.509 → 0.562 | 0 → 0 |
| core-tl-simple-sep | 1.000 → 1.000 | 0.833 → 0.944 | 0.087 → 0.094 | 0 → 0 |
| core-tl-adv-sep | 1.000 → 1.000 | 1.000 → 1.000 | 0.217 → 0.304 | 0 → 0 |
| core-bda-simple-sep | 1.000 → 1.000 | 0.833 → 1.000 | 0.110 → 0.115 | 0 → 0 |
| core-bda-adv-sep | 1.000 → 1.000 | 1.000 → 1.000 | 0.360 → 0.449 | 0 → 0 |
| core-llm-simple-sep | 0.564 → 1.000 (sd 0) | 1.000 → 0.889 | 0.088 → 0.117 | 0 → 0 |
Field accuracy improved, and it reproduced
Section titled “Field accuracy improved, and it reproduced”scalar_accuracy rose on exactly the four simple / non-integrated cells, and on
none of the advanced or integrated ones. On longdesc_100 one of two scalar fields went
from wrong to right, consistently.
The per-cell delta (+0.167) is individually within run-to-run spread (±0.250 at n=9),
so --compare marks each one inconclusive. What raises it above noise is that four
independent cells move the same way, and that the same pattern appeared in an earlier,
separately-invalidated run of this grid. Two runs agreeing on four cells is not a
coincidence of sampling.
Most likely cause — untested. #740 raised ocr.image.dpi from 150 to 300.
base-ocr.yaml records the mechanism: below ~200 dpi Textract “silently omits small,
faint or skewed characters — page numbers, box numbers, hand-filled values”. An account
number is exactly that. The cells that improved are the ones whose scalar fields come
from Textract text; the advanced cells were already at 1.000 and had no room to move.
Confirming this needs a targeted ocr_dpi A/B, which the axis now supports.
Advanced-mode cost IS up — +22 to 25% — and the cause is over-splitting, not the agentic path
Section titled “Advanced-mode cost IS up — +22 to 25% — and the cause is over-splitting, not the agentic path”All three advanced cells rose in the same direction on the corefast gate:
| cell | cost | flagged |
|---|---|---|
| core-tt-adv-sep | +26% (0.360±0.287 → 0.452±0.380) | within noise |
| core-tl-adv-sep | +40% (0.217±0.138 → 0.304±0.236) | within noise |
| core-bda-adv-sep | +25% (0.360±0.292 → 0.449±0.391) | within noise |
Each was individually inconclusive there — cost CV runs 0.28–0.39 at n=9 — so the
question was reopened with the purpose-built cost suite, which is the shape
config_matrix.yaml requires for a dollar claim: one mid-size document
(med_narrow, 400 rows / 9 pages), repeats: 5, one stack upgraded in place, and
byte-identical config files on both sides. Both arms ran with no other grid in
flight, and both were priced from the same pricing.yaml hash.
It reproduces, decisively.
| cell | v0.6.6 | v0.6.7 | delta |
|---|---|---|---|
| core-tt-simple-sep (control) | $0.5356 ± 0.0009 | $0.5326 ± 0.0083 | −0.6% |
| core-tt-adv-sep | $1.4005 ± 0.1104 | $1.7119 ± 0.1520 | +22.2% |
| core-tt-adv-int | $1.4883 ± 0.0838 | $1.8177 ± 0.1420 (n=3) | +22.1% |
| core-tl-adv-sep | $0.9902 ± 0.0749 | $1.2209 ± 0.0624 | +23.3% |
| core-bda-adv-sep | $1.3562 ± 0.0764 | $1.6957 ± 0.2314 | +25.0% |
Cost CV on this document is 0.056–0.079, not 0.28–0.39 — the corefast spread came
from averaging three very differently-sized documents, not from agentic
non-determinism. At n=5 that is a standard error of ~3% on each mean, so +22% is
roughly seven standard errors. aggregate.py --compare flags all four advanced cells
as cell-level cost regressions.
The mechanism, from the raw metering
Section titled “The mechanism, from the raw metering”The extra spend is entirely Sonnet 4.6 input tokens in the Extraction phase, with output essentially unchanged (recall is 1.000 on both sides — the same rows come back):
| core-tt-adv-sep, Extraction phase | v0.6.6 | v0.6.7 | delta |
|---|---|---|---|
| inputTokens | 96,423 | 152,098 | +57.7% |
| outputTokens | 44,107 | 44,938 | +1.9% |
| cacheReadInputTokens | 239,252 | 343,679 | +43.6% |
| cacheWriteInputTokens | 26,584 | 39,084 | +47.0% |
Extraction/lambda/requests | 1 | 4 | — |
sections (truth: 1) | 1,1,1,1,1 | 4,4,2,4,4 | — |
med_narrow is one bank statement, and the truth says one section. v0.6.7 splits it
into two-to-four. Each section is a separate Extraction Lambda invocation running a
complete agentic loop — system prompt, schema restatement, tool schemas, pre-flight table
parse, multi-turn conversation — so the per-section fixed overhead is paid two-to-four
times while the extracted rows, and therefore the output tokens, stay the same.
Why v0.6.6 got 1 section — and why that was not “correct”
Section titled “Why v0.6.6 got 1 section — and why that was not “correct””The tempting reading is “v0.6.6 classified correctly and v0.6.7 broke it”. The metering
says otherwise: on the v0.6.6 arm there is no Classification/bedrock/* meter at all,
in any of the 5 runs. v0.6.6 made zero classification inference calls on this document.
That is #686/#705:
sectionSplitting was skipped entirely for a single-class configuration. Its
sections=1 was a short-circuit, not a decision — the same code would have returned one
section for a genuine two-statement packet, which is exactly the bug v0.6.7 fixes.
So the real chain is:
- v0.6.7 fixes #686 — classification now actually runs on single-class configs (9 Nova 2 Lite requests per document here, ~$0.002).
- The now-active classifier over-splits one homogeneous multi-page statement into 2–4 sections — #726, partially addressed by #750’s running-header fix but not resolved.
- On the agentic path each spurious section costs a whole agent loop, which is the +22–25%.
The +22–25% is therefore the price of fixing #686 while #726 is still open, and it
lands almost entirely on advanced mode. FINDINGS.md measured #726 in simple mode and
concluded “cost is unchanged”; that holds for simple extraction and does not generalize —
a simple extraction has almost no per-section fixed cost, an agentic one has a great deal.
Confirmed causally, within one release — and two mitigations, neither free
Section titled “Confirmed causally, within one release — and two mitigations, neither free”advsplitcost re-runs the same two cells on the same v0.6.7 stack, repeats: 5, varying
one knob at a time:
| arm | sections (truth: 1) | Extraction invocations | core-tt-adv-sep | core-tt-simple-sep |
|---|---|---|---|---|
v0.6.6, llm_determined (classification skipped) | 1,1,1,1,1 | 1 | $1.4005 | $0.5356 |
v0.6.7, llm_determined, cls Nova 2 Lite (default) | 4,4,2,4,4 | 4 | $1.7119 (+22.2%) | $0.5326 (−0.6%) |
v0.6.7, sectionSplitting: disabled | 1,1,1,1,1 | 1 | $1.2969 (−7.4%) | $0.5289 (−1.3%) |
v0.6.7, llm_determined, cls Sonnet 5 | 1,1,1,1,1 | 1 | $1.5521 (+10.8%) | $0.7764 (+45.0%) |
Read across those rows:
- Nothing regressed in the agentic extraction path. With the section count at 1, v0.6.7 advanced extraction is 7.4% cheaper than v0.6.6.
- The over-split is model-dependent. Sonnet 5 classification gets 1 section in 5 of 5
runs; the shipped default (Nova 2 Lite) gets 1 in 0 of 5. This reconciles this result
with config-guidance §7, which measured
llm_determinedatsections_correct1.00 — that measurement used Sonnet 5 as the classifier. - Neither mitigation is free.
sectionSplitting: disabledis the cheapest correct setting for a single-class corpus, and is measured wrong by construction on a packet (§7:sections_correct0.00 on a two-statement file) — do not apply it to mixed input. Sonnet 5 classification fixes the split but is charged per page, so it costs simple-mode users +45% and only claws back 9% of advanced mode’s premium. - #726 is now a cost defect, not only a shape defect. Its recorded symptoms were
cosmetic (3 result files, 3 UI sections,
Account Number: nullon continuations). On the agentic path it is also a ~22% bill increase, and becauseextraction.validation.enablednow defaults totrue, each spurious continuation section additionally emits a real'Account Number' is a required propertyissue.
Recall was 1.000 and scalar accuracy 1.000 on every completed run in every arm, so no data was lost anywhere in this table — it is purely cost and output shape.
One thing this run also surfaced: a retry ladder that cannot fit its Lambda
Section titled “One thing this run also surfaced: a retry ladder that cannot fit its Lambda”core-tt-adv-int completed 5/5 on v0.6.6 (wall 513–585 s, tight) but only 3/5 on
v0.6.7 (173 s, 175 s, 557 s). The two that did not finish hit the 900 s Extraction
Lambda timeout and were retried by Step Functions; they were stopped at the
harness deadline and are recorded as failures, which is why that row reads n=3.
The trigger in the CloudWatch log is not a code path — it is a Bedrock
Read timed out. The amplifier is:
@async_exponential_backoff_retry(max_retries=50, initial_delay=5, max_delay=1800, ...)async def invoke_agent_with_retry(input, agent): ...max_delay=1800 is 30 minutes of backoff inside a 900-second Lambda, so a single
transient Bedrock timeout cannot be recovered — the ladder outlives the function, the
invocation is lost, and Step Functions repeats the whole extraction. These values are
byte-identical in v0.6.6 and v0.6.7, so this is a pre-existing defect, not a
regression, and the 2/5-vs-0/5 difference at n=5 is not attributable to the release.
It is worth fixing on its own terms: the ladder should be bounded by the remaining
Lambda execution time.
LLM-OCR: the strongest single result, and it points at DPI
Section titled “LLM-OCR: the strongest single result, and it points at DPI”core-llm-simple-sep was re-run to full sample as its own llmocr suite
(benchmarks/results/v0.6.7/llmocr/), 9 runs per side:
| metric | v0.6.6 | v0.6.7 | spread (new) |
|---|---|---|---|
| completeness recall | 0.5644 | 1.0000 | sd 0.000, CV 0.000 |
| scalar accuracy | 1.0000 | 0.8889 | sd 0.333 |
| cost | $0.0877 | $0.1166 | CV 0.95 |
| failures | 0 | 0 |
Recall is 1.000 on every one of nine runs, with zero spread. Unlike the Textract cells’ +0.167 accuracy move, this is far outside sampling noise and needs no four-cells-agree argument.
It is also the result that most directly supports the DPI explanation. bedrock_llm OCR
reads the page image rather than consuming Textract output, so it is the cell most
sensitive to render resolution — and it is the cell that moved most. #740 raising
ocr.image.dpi 150 → 300 is a coherent mechanism for a 56% → 100% recall change in an
image-reading backend. Still formally untested; the ocr_dpi axis exists to test it.
The scalar dip is a known defect, not a new one. Eight of nine runs score 1.000; the
ninth (longdesc_100 rep1) scores 0.000 and cost $0.372 against ~$0.145 for its
siblings. That is the bedrock_llm fixed-width identifier corruption root-caused in
the v0.6.6 audit — the backend inserts a digit into an identifier like an
account number, page-scoped, with nothing flagging it. It is why this backend should not
be used where identifiers matter, and it is unrelated to this release.
Cost is not claimable at CV 0.95; the mean is dominated by that one outlier run.
Completeness of this run
Section titled “Completeness of this run”163 of the 171-run grid. It was interrupted during its last cell, and all eight missing
runs were core-llm-simple-sep — re-run to full sample as the llmocr suite on the same
unchanged stack, so all ten baseline-matched cells have their full 9 runs and the
effective total is 172. Every feature arm is complete.
The stack did not move during the run, verified: run_matrix.py now fingerprints the
stack and re-checks before every launch, and it reported zero changes across all 163
launches (LastUpdatedTime 2026-09-03T19:35:26Z throughout). That guard exists because
the first attempt at this grid was invalidated by two CloudFormation updates landing
mid-run — 22 of 171 runs launched into an active update — which nothing detected until
scoring. That run is retained, clearly marked void, at
benchmarks/results/v0.6.7/corefast/.
Feature arms in this grid
Section titled “Feature arms in this grid”The 9 non-core-* cells measure v0.7 configuration options. Their findings are written
up per feature rather than here:
| cells | finding |
|---|---|
force-off / force-on | forcing is honored 100% and changes nothing measurable — results/v0.6.7/forcing/ |
restate-on / restate-off | dropping the duplicated schema costs no completeness |
enforce-off / -warn / -escalate | see results/v0.6.7/enforcement-* |
split-disabled / split-llm | disabled is wrong by construction on packets — results/v0.6.7/boundary-factorial/ |
Not done here
Section titled “Not done here”baseline.jsonwas not updated. Promoting this run would make a 163-run grid the reference for a 171-run suite, andcore-llm-simple-sepis not in it at full sample.- The
corefastaccuracy gate is against a baseline measured on a different stack (IDPRel066), which is the weaker form of the procedure in.claude/skills/run-benchmarks.md. It is stated rather than implied. Thecostsuite above is the strong form — one stack (IDPAB066) deployed from the published v0.6.6 template andupdate-stacked in place to the published v0.6.7 template, with the same config files replayed. - The accuracy gate’s
+0.167 scalar on four cellsremains untested as to cause. The DPI hypothesis is unchanged; theocr_dpiaxis exists to test it and was not run. - The
core-tt-adv-intfailure rate (0/5 → 2/5) is not attributed to the release — see the retry-ladder note above. It needs a larger n on a quiet account to be a claim.
Reproducing
Section titled “Reproducing”python3 benchmarks/harness/gen_corpus.pypython3 benchmarks/harness/make_configs.py --suite corefast --class bank_statementAWS_PROFILE=default python3 benchmarks/harness/run_matrix.py --stack <STACK> \ --suite corefast --class bank_statement --max-inflight 20AWS_PROFILE=default python3 benchmarks/harness/aggregate.py \ --run benchmarks/results/run-<stamp> --out benchmarks/results/v0.6.7/corefast-cleanpython3 benchmarks/harness/aggregate.py \ --compare benchmarks/results/v0.6.7/corefast-clean/summary.json \ --baseline benchmarks/results/baseline.jsonThe run aborts if the stack changes underneath it, so a grid can no longer silently span two builds.
And the cost A/B — the strong, same-stack form. Deploy the published v0.6.6 template,
measure, update-stack in place to the published v0.6.7 template, replay the same
config files, measure again:
python3 benchmarks/harness/make_configs.py --suite cost --class bank_statement
idp-cli deploy --stack-name <S> --region us-west-2 --admin-email <you> --wait \ --template-url https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.6.yamlAWS_PROFILE=default python3 benchmarks/harness/run_matrix.py --stack <S> --suite cost \ --native-upload --max-inflight 20AWS_PROFILE=default python3 benchmarks/harness/aggregate.py \ --run benchmarks/results/run-<stamp> --out benchmarks/results/v0.6.6/cost
idp-cli deploy --stack-name <S> --region us-west-2 --wait \ --template-url https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.7.yamlAWS_PROFILE=default python3 benchmarks/harness/run_matrix.py --stack <S> --suite cost \ --native-upload --max-inflight 20AWS_PROFILE=default python3 benchmarks/harness/aggregate.py \ --run benchmarks/results/run-<stamp> --out benchmarks/results/v0.6.7/costpython3 benchmarks/harness/aggregate.py \ --compare benchmarks/results/v0.6.7/cost/summary.json \ --baseline benchmarks/results/v0.6.6/cost/summary.json--native-upload on both arms, not just the older one: idp-cli config-upload
migrates the config, so letting it run on one side and not the other stops the two arms
from being byte-identical.
Then the causal check, on the v0.6.7 stack only:
python3 benchmarks/harness/make_configs.py --suite advsplitcost --class bank_statement \ --set section_splitting=disabledAWS_PROFILE=default python3 benchmarks/harness/run_matrix.py --stack <S> \ --suite advsplitcost --set section_splitting=disabled --native-upload