Classification Confidence — Does the Score Carry Signal?
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
A focused study, not a release audit. It answers one question: when classification reports a confidence, is the number worth acting on — and does the answer depend on the classifier? For “which extraction config should I pick?” see Configuration Guidance; for release-over-release safety see the Release Audit Trail.
Classification Confidence — Does the Score Carry Signal?
Section titled “Classification Confidence — Does the Score Carry Signal?”Release: v0.6.7.dev (branch feature/issue-673-classification-confidence) ·
Region: us-west-2 · Stack: IDPBench066
Test set: docsplit (DocSplit-Poly-Seq — RVL-CDIP-derived packets, 13 classes)
· first 20 packets, 298 pages per model per mode, one run each, no failures
Config: config_library/unified/docsplit/config.yaml, classification.combined.llm.confidence.mode
∈ {topk (top_k_candidates: 3), off}, everything else at system defaults;
summarization disabled
Pricing: config_library/pricing.yaml (rates as of 2026-08)
The Nova 2 Lite
topkcolumn was re-measured on v0.6.13 and again on v0.6.14; the rest of this page was not. The classification-variant benchmark (configuration guidance §5.6) ran the same 20 packets with Nova 2 Lite,topkand 3 candidates, and itspage-bothvariant is this page’s configuration. On v0.6.14 it scored 19 of the 20 packets (286 pages; the twentieth failed in extraction on a content-filter refusal, not in classification) and reads page class accuracy 0.818 (v0.6.13 0.839, here 0.846), calibration separation +0.034 (v0.6.13 +0.033, here +0.044), mean confidence 0.947 on correct pages and 0.914 on wrong ones (here 0.947 and 0.903), a median of 0.95 on wrong pages (unchanged), 7.0% of pages below 0.9 (here 8.4%) and $0.00097 of classification per page (here $0.00090). So the conclusion below — the default classifies about as accurately as Haiku 4.5 and its confidence barely separates right from wrong — still describes the default on this corpus. Haiku 4.5 and theoffcontrol were not re-run, so their columns are v0.6.7’s. Data:benchmarks/results/v0.6.14/clsvariants/summary.json, keynova2lite-page-both.
Reproduce with:
AWS_PROFILE=default python3 benchmarks/harness/run_classification_bench.py \ --stack <STACK> --testset docsplit --n 20 --models nova2lite,haiku45 --mode topk# ... and again with --mode off for the controlRaw data: benchmarks/results/v0.6.7/clsconf/ (summary.json, per-page
pages_*.json, escalation.json).
Abstract
Section titled “Abstract”Classification confidence was added in #673
on the argument that a self-reported number is usually ~0.95 on everything and
therefore worse than no score. This study measures that argument on a genuinely
confusable 13-class corpus, for the default classification model (Amazon Nova 2
Lite) and a mid-tier alternative (Anthropic Claude Haiku 4.5), against an
off control.
It is also why topk now ships ON by default. The control run answers the two
questions that decision turned on: asking for ranked candidates costs ~0.5 % of
total document cost (+17 % of the classification step, which is only ~3 % of the
bill on the default model) and changes classification accuracy by nothing
consistent (+0.013 on Nova, −0.007 on Haiku — opposite signs, single runs). A
signal that catches 43 % of the default model’s own errors from 8 % of its pages is
worth half a percent.
The two models are equally accurate and not remotely equally calibrated. Accuracy differs by 0.7 points (84.6 % vs 85.2 %, inside noise at this sample size), while the calibration separation differs by 4.7× (+0.044 vs +0.207). Nova 2 Lite put 90 % of all pages at exactly 0.95 and emitted only 6 distinct confidence values across 298 pages; Haiku 4.5 produced 11 values spread from 0.50 to 0.98. At a usable threshold, Haiku catches 73 % of misclassifications by reviewing 11 % of pages at 94 % precision; Nova catches 43 % by reviewing 8 % at 80 % precision, and half its errors sit at 0.95 where nothing distinguishes them from correct pages.
Classification confidence therefore costs 6.4× more per page on the model where it
is actually informative. That is the trade to decide deliberately, rather than one
the default can make for you — which is why this page exists. The default gives you
the signal; whether to pay for a model that makes it actionable, or to set
mode: off and keep neither, is the decision left open.
1. What turning it on costs (vs. the off control)
Section titled “1. What turning it on costs (vs. the off control)”Same 20 packets, same 298 pages, same config but for the mode:
Nova 2 Lite off | Nova 2 Lite topk | Haiku 4.5 off | Haiku 4.5 topk | |
|---|---|---|---|---|
| Page class accuracy | 0.832 | 0.846 | 0.859 | 0.852 |
| Classification output tokens / page | 154 | 199 | 268 | 353 |
| Classification cost / page | $0.00077 | $0.00090 (+17.5 %) | $0.00499 | $0.00573 (+14.9 %) |
| Classification as share of total doc cost | — | 3 % | — | 16 % |
| Total document cost / page | $0.03072 | $0.03062 (−0.3 %) | $0.03426 | $0.03474 (+1.4 %) |
Two things to read off this:
- Accuracy is unaffected. The two deltas have opposite signs and are both within single-run noise, so asking for candidates does not systematically change which class the model picks — it neither helps nor hurts the decision.
- The cost is negligible on the default model. +17.5 % of a step that is 3 % of the bill is ~+0.5 % of total, which is why the measured total came out slightly negative: the difference is smaller than run-to-run variance in OCR/extraction. On a classifier that is itself 16 % of the bill the delta is visible (+1.4 %) but still small.
The honest asymmetry: this is where the money goes on these documents (~10 pages of scans per packet, with extraction and assessment dominating). A deployment with very cheap extraction and very long documents will see classification — and this setting — take a larger share.
2. Results
Section titled “2. Results”| Metric | Nova 2 Lite (default) | Claude Haiku 4.5 |
|---|---|---|
| Pages evaluated / scored | 298 / 298 | 298 / 298 |
| Page classification accuracy | 0.846 | 0.852 |
| Misclassified pages | 46 (15.4 %) | 44 (14.8 %) |
| Calibration separation (mean conf right − wrong) | +0.044 | +0.207 |
| Mean confidence, correct pages | 0.947 | 0.948 |
| Mean confidence, wrong pages | 0.903 | 0.741 |
| Median confidence, wrong pages | 0.95 | 0.735 |
| Distinct confidence values emitted | 6 | 11 |
| Pages below 0.9 confidence | 8.4 % | 20.8 % |
| Classification cost / page | $0.00090 | $0.00573 |
| Classification output tokens / page | 199 | 353 |
Accuracy being equal is itself worth noting: on this corpus the cheaper default classifies as well as the 6× model. The difference is entirely in knowing when it is wrong.
3. How confidence is distributed across pages
Section titled “3. How confidence is distributed across pages”Confidence distribution over all 298 pages:
| Confidence | Nova 2 Lite | Haiku 4.5 |
|---|---|---|
| 0.98 | — | 114 |
| 0.95 | 269 | 79 |
| 0.92 | — | 43 |
| 0.90 | 2 | — |
| 0.85 | 22 | 28 |
| 0.75 | — | 11 |
| 0.70–0.72 | 2 | 8 |
| ≤ 0.65 | — | 15 |
| 1.00 | 2 | — |
Nova 2 Lite answers 0.95 for 90 % of pages — including 23 of its 46 errors. This is the “just ask for a number” failure the design warned about, and top-K only partially mitigates it: the model did move 22 pages to 0.85 (18 of them genuinely wrong), so the signal is a coarse two-level flag rather than a probability.
Haiku 4.5’s distribution is graded, and its errors concentrate in the low bins.
4. Escalating low-confidence pages at a threshold
Section titled “4. Escalating low-confidence pages at a threshold”Flag pages below a confidence threshold for review. Error recall = share of the model’s misclassifications caught; precision = share of flagged pages that really were wrong.
Nova 2 Lite (46 errors in 298 pages)
| Threshold | Pages flagged | Errors caught | Error recall | Precision |
|---|---|---|---|---|
| 0.95 | 27 (9.1 %) | 21 | 0.46 | 0.78 |
| 0.90 | 25 (8.4 %) | 20 | 0.43 | 0.80 |
| 0.85 | 3 (1.0 %) | 2 | 0.04 | 0.67 |
Claude Haiku 4.5 (44 errors in 298 pages)
| Threshold | Pages flagged | Errors caught | Error recall | Precision |
|---|---|---|---|---|
| 0.95 | 105 (35.2 %) | 39 | 0.89 | 0.37 |
| 0.90 | 62 (20.8 %) | 37 | 0.84 | 0.60 |
| 0.85 | 34 (11.4 %) | 32 | 0.73 | 0.94 |
| 0.70 | 15 (5.0 %) | 14 | 0.32 | 0.93 |
Haiku dominates on both axes at its best operating point: fewer wasted reviews (94 % of flagged pages are genuine errors) for far more coverage (73 % vs 43 %). Nova’s ceiling is structural — 23 errors are indistinguishable at 0.95, so no threshold can reach them.
5. Prompt caching with Haiku
Section titled “5. Prompt caching with Haiku”Measured per page: Nova 2 Lite served ~934 input tokens/page from prompt cache; Haiku 4.5 reported zero cache reads and zero cache writes, paying its full ~3.3 K input tokens on every page.
Haiku 4.5 is in the accelerator’s CACHEPOINT_SUPPORTED_MODELS, and the
classification prompt does carry <<CACHEPOINT>> (with the confidence block spliced
before it, deliberately). The likely cause is prefix size: the cacheable prefix
measured on Nova is ~934 tokens, below Anthropic’s minimum cacheable prefix for
Haiku 4.5, so Bedrock silently declines to cache. This is an observation with a
probable cause, not a verified diagnosis — a config with a larger class
vocabulary or few-shot examples would push the prefix over the minimum and is the
obvious way to test it.
Practical consequence: on a small class vocabulary, do not assume caching softens the cost of an Anthropic classifier. The 6.4× figure above is what you actually pay.
6. Guidance
Section titled “6. Guidance”⚠️ Every path on this page is the combined one, and under split_then_classify the
block you must edit is a different one. This study ran
classification.strategy: combined, so mode and the threshold below mean
classification.combined.llm.confidence.…. Under classification.strategy: split_then_classify the call that assigns a class is section classification, so the same
settings live at
classification.split_then_classify.section_classification.llm.confidence.… and the
combined block is not read at all — editing it there changes nothing and
produces no error. The splitting call has no confidence block, because it
assigns no class. The cost figures also assume combined metering: a split-then-classify
run’s classification spend is reported under Splitting/ and
SectionClassification/ rather than Classification/.
topkis on by default, and that is the right default — ~0.5 % of the bill for a signal that catches 43 % of the default model’s errors from 8 % of pages. Setmode: offif you process very long documents where classification is a large share of your cost and you will not act on the score at all.- If you want an actionable score, change the classifier, not just the mode. On this corpus Haiku 4.5’s confidence supports review routing (73 % of errors from 11 % of pages); Nova 2 Lite’s supports a weak triage at best.
- Prefer
topkoververbalized. Even on Nova, ranked candidates pulled 22 pages off the 0.95 pile; a single self-reported number has no such pressure. (This study did not runverbalized— treat that as untested here, not equal.) - Set the threshold from your own measured table, not from a default. The right cut differs per model: 0.90 for Nova, 0.85 for Haiku on this corpus.
- Re-measure on your own documents. 298 pages of one corpus, one run per model.
The harness prints
class_calibration_separation,class_accuracyandn_class_scored_pagesfor exactly this purpose.
7. Limitations
Section titled “7. Limitations”- One run per model per mode, 20 packets, 298 pages. No repeats, so neither the 0.7-point between-model accuracy gap nor the ±1-point off-vs-topk deltas are claims of difference — the latter’s opposite signs are the point (no systematic effect); the calibration gap (4.7×, plus a 269-page spike at one value) is far larger than any plausible single-run noise, but it is still one sample of one corpus.
- One corpus. RVL-CDIP-derived scans over 13 classes — deliberately confusable, and not representative of a 2-class deployment where errors are rarer.
- Both models ran the identical config, identical prompt (the composed block is byte-identical), and the same first 20 documents (test-set file selection is sorted and deterministic). Runs were sequential so neither model’s throttling shows up as the other’s latency or cost.
- Cost covers the classification step only (
Classification/*metering), which is the step the setting changes. Accuracy is page-level class accuracy from the evaluation report’sdoc_split_metrics, against the test set’s curated baseline. - Summarization was disabled; extraction ran at its defaults and is not scored here.