Benchmark Studies
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
Benchmark studies — one question each
Section titled “Benchmark studies — one question each”Everything in this folder was measured with the same harness, on a deployed stack, under the same honesty rules as the Configuration Guidance — but each file answers one question that came up during development, once, and is not regenerated per release. The systematic outputs live one level up: the Benchmarking Guide, the Configuration Guidance (refreshed per release) and the Release Audit Trail. Where a study’s conclusion changed a default or a recommendation, the guidance paper cites it; the study keeps the full evidence, including any wrong turn taken on the way.
None of these studies was re-run for the v0.6.14 benchmark refresh except one column of the classification-confidence study, which the classification-variant benchmark re-measured as a side effect; the date in each row’s Measured column is the date of its data.
| Study | Question | Measured | Verdict | Where it landed |
|---|---|---|---|---|
| Prompt Caching — Measured Behaviour | Are the duplicated schema copies really “cache reads at a tenth of the price”? | 2026-09-10, v0.6.7 | Not below a 1,024-token prefix: two of nine shipped classes never cache; a one-off document pays +24.9% for a cache it never reuses; the end-to-end effect is smaller than the mechanism suggests | guidance §7 (forcing); the metadata.prompt_cache observability shipped in 0.6.8 |
| Should we pad prompts to clear the cache minimum? | Would padding short prompts up to the cache minimum pay for itself? | 2026-09-10, v0.6.7 | ~20% off extraction input cost for an affected class, but the sign flips with workload shape and the accuracy risk is unbounded — proposal: not automatically | a product decision, still open |
| Classification Confidence — Does the Score Carry Signal? | When classification reports a confidence, is it worth acting on, and does that depend on the classifier? | 2026-09-02, v0.6.7 (DocSplit-Poly-Seq, 20 documents, 298 pages per model); the Nova 2 Lite column re-measured 2026-10-31 on v0.6.13 (separation 0.033) and 2026-11-10 on v0.6.14 (0.034) | Calibration separation 0.044 on Nova 2 Lite vs 0.207 on Haiku 4.5: actionable only on the stronger classifier; ~17% of the classification step | guidance §5.3; docs/classification.md |
| Feature study: multi-instance sections | Does x-aws-idp-multi-instance recover the records a merged section loses, and what does detection cost on real corpora? | 2026-09-03 → 09-11, v0.6.7 | The wrapper recovers them (40 of 40 rows); detection is neutral on the OCR benchmark and −1.3 points on RealKIE, so it ships off; two documented wrong conclusions and how each was caught | guidance §7; issue #894 (the v0.6.8 refresh found the wrapper fails assessment on a long list) |
| Schema Restatement — What Turning It Off Actually Costs | Is dropping the duplicated system-prompt schema copy a free token saving? | 2026-09-20, v0.6.10 (Sonnet 4.6, 3 synthetic documents, 50 runs / 25 per arm) | No quality cost in 50 runs at 1.000; a real 5–7% input-token reduction on two documents of three (p ≤ 0.008), but not a dollar saving — output tokens rise 17.4% on the list-heavy document (p = 1.2 × 10⁻⁵), so extraction cost’s central estimate is +7.5% (not significant). Turn it off for shard headroom | guidance §7; issue #710 |
| Greedy Decoding and Invalid Tool-Use Sequences | Does AWS’s recommended topK: 1 (or the full greedy triple) fix Nova Lite’s invalid tool-use failures, and what triggers them? | 2026-09-20, v0.6.10 (Nova Lite; 160 direct calls per arm plus 64 pipeline runs) | Null: 0.787 vs 0.812 invalid-ToolUse, Fisher p = 0.68, with 80% power only to an 18% relative improvement — a large effect is excluded, a small one is not. topK is not a Converse field and each family carries it differently (routing table). The trigger is request shape, not schema size: a 1,696-char schema passes where a 401-char one fails | docs/extraction-and-confidence.md; the extraction library README; issues #956 and #895. No knob shipped |
| Confidence Calibration — Can the Score Route Human Review? | Is the confidence score calibrated, and does it rank errors well enough to order a worst-first review queue? | 2026-09-21 re-score of v0.6.8 (stack IDPUpg067to068, 466,322 cells over 7 synthetic documents) with a v0.6.9 replication | Split verdict: every arm passes calibration (largest mean-confidence ECE 0.024 against a 0.15 bar) and the shipped default grader cannot rank — separate on Nova Lite puts all 70,320 cells in ONE confidence bin, AUROC 0.417, reproduced at 0.419 in v0.6.9. Sonnet 5 as the grader spreads over 6 bins and reaches AUROC 0.859 on the same extraction and the same error count. The worst case is an arm with 1,413 wrong cells, all 54,484 of its cells in the top bin at a mean confidence of 0.995, and an ECE of 0.021 — passing | the harness gained per-document ECE/Brier/AUROC and a pooled per-cell calibration gate in compare_cells; issue #935. No default changed — Nova Lite is a ~130× cost decision and this only supplies the half of the trade that was assumed |
| Advanced extraction — seven candidate refinements | Of seven ideas for making agentic extraction cheaper or more complete, which are real? | 2026-09-12 → 09-15, v0.6.8 | None of the seven is a new default; the defects found while measuring them are worth far more — fixing two took the same 12-run arm from $68.82 to $31.60 (−54%) with recall 0.976 → 1.000. Four defects in all: the “images off” arm wrote a config key nothing read, the sharded path never ran the pre-flight parse, neither image knob worked on the single-pass path (which is why 25-page shards failed outright), and :1m pricing reports ~1.9× the tokens actually justify. Prompt caching already works across shards (70–97% cache reads); batch inference is ineligible and costs more; Textract [LAYOUT]-only is 23% cheaper but drops the densest table to 0.098 recall; and with the fixes in place 25-page shards no longer fail but cost 5× the 5-page default ($158.00 vs $31.60), because wide shards defeat the cache | the four fixes in extraction/service.py, agentic_idp.py, tools/table_parser.py and the harness axis validator (PR #898); the pricing.yaml :1m reporting bug (#899) fixed in v0.6.9 — metering keys no longer carry the suffix and the :1m rates now match the base model’s, because the 1M context window carries no price premium on these models at all (the study’s assumption that the premium was merely mistimed was itself too generous); the missing pre-parsed guidance on the sharded path (#900) also fixed in v0.6.9, with the token saving still unmeasured; one follow-up still open — in-shard assessment discarding a successful extraction when it overflows the confidence model (#901) |
Adding a study: put it here with a title: frontmatter, a one-paragraph “why this page
exists”, a ## Reproduce section naming the suite and the result directory, and one row in
this table. The docs-site sidebar picks the file up automatically.