Skip to content
GenAI IDP

Release Benchmark Audit Trail

Release Benchmark Audit Trail

Section titled “Release Benchmark Audit Trail”

One entry per release, comparing each develop prerelease against the previous publicly published release on the corefast grid (10 config cells × 3 ≤100-row docs, extraction model held at a shared control both versions can run). Each entry is written once and never overwritten — this table is the durable history.

Generate a new entry with make benchmark-release VERSION=<new> PREV=<published> (see the Benchmarking Guide).

Releasevs (published)AccuracyCostNotableReport
v0.6.9v0.6.8 (published 0.6.8 → a --clean-build of the release commit; 0.6.9 was not yet published)recall 1.000 → 1.000 on all 19 cells, 0 cell-level regressions. One real regression: bedrock_llm OCR now emits markdown bold and one of two scalar fields on one document came back re-worded (1.000 → 0.500 scalar, deterministic 3/3 each side); no table data lost (cell_accuracy 1.000 over 200 cells)−21.1% grid total, all of it Extraction (−27%); other phases ≤3%. Real, not an artefact of the two cost-reporting fixes — the harness prices metering itself with exact key matching, same pricing.yaml sha both sides, and all 342 runs used sonnet46 not a :1m model. Corroborated by token counts: cacheWrite −52%, output −14%, input −10%. Wall −21%0 failures in 171+171 runs. ⚠️ Accuracy is not comparable with the v0.6.8 entry — stickler-eval 0.5.0 → 1.0.0 changed the comparators (both sides here use the new one). baseline.json deliberately not promoted, for that reasonv0.6.9.md
v0.6.8v0.6.7 (both published, same night, byte-identical configs)flat (recall 1.000 = 1.000 on all 19 cells, 11,685/11,685 rows; cell accuracy 1.000 everywhere but Bedrock-LLM OCR 0.889→0.944, within spread)−3.2% grid total; assessment −37% (#861), all of it in core-tt-simple-int (−30%); wall −31%0 regressions, 0 failures in 171+171 runs. Same-night A/B isolates code; the #726/#817 section fix ships as default config so it is on both sides and cancels — the prerelease entry (kept in the same file) shows that combined effect. baseline.json promoted.v0.6.8.md
v0.6.7v0.6.6+0.167 scalar on 4 of 10 cells (0.833→1.000 ×3, →0.944 ×1), recall 1.000→1.000 everywhere; each cell individually within spread but 4 cells agree, and the pattern reproduced across two runsadvanced cells +22–25%, CONFIRMED by a same-stack in-place cost A/B (n=5, CV 0.06–0.08, ~7 SE). Cause: #686 fixed (classification now runs on single-class configs — v0.6.6 made zero classification calls) and the now-active classifier over-splits one statement into 2–4 sections (#726), each costing a full agentic Extraction invocation ⇒ input tokens +58% at flat output. With sectionSplitting: disabled v0.6.7 is 7.4% cheaper than v0.6.6, so the agentic path did not regress. The over-split is model-dependent: Sonnet 5 classification gets 1 section 5/5 but costs simple mode +45%. Simple mode −0.6%0 regressions, 0 failures in 172 runs. LLM-OCR recall 0.564→1.000 on 9/9 runs, sd 0 — Likely cause of the accuracy gain is #740’s ocr.image.dpi 150→300 (Textract drops faint characters below ~200 dpi) — untested. Also surfaced a pre-existing (identical in both releases) defect: the agentic retry ladder is max_delay=1800 inside a 900 s Lambda, so one transient Bedrock read-timeout loses the invocation. First attempt at the grid was voided by two CloudFormation updates mid-run; the harness now aborts if the stack moves. baseline.json deliberately not promotedv0.6.7.md
v0.6.6v0.6.5+0.033 mean (one cell 0.667→1.000, other 9 identical)−5.8%/run, all of it one cell; the other 9 are +4.0% (within spread)0 regressions, 0 failures in 90 runs. The integrated-confidence row-loss hazard open since v0.6.0 is fixed — 9/9 complete at −55% cost. One apparent recall drop root-caused to the bedrock_llm OCR backend corrupting fixed-width identifiers (not a code change), which also refutes v0.6.5’s unconfirmed improvement on that cell. Baseline reused, not re-run — see the deviation notev0.6.6.md
v0.6.5 fixesv0.6.5 (pre-fix)flat (1.000 = 1.000)−9% total (inconclusive)Post-fix verification, not a release A/B. recall 0.901→0.947, 0 regressions, 0 failures; integrated-confidence 400-row lists 1%→100% complete (800-row still truncates, now detectable); first attempt discarded for a config/model confoundv0.6.5-fix-verification.md
v0.6.5v0.6.4flat (0.900 = 0.900, all 30 pairs equal)+1.4% across the 24 runs in the 8 stable cells; +8.5% total, all of it in 2 non-deterministic cells0 regressions. Measured recall 0.931→0.987 but a 4× repeat run shows the integrated-confidence cell is bimodal (2/4 truncate) — the hazard is NOT fixed and the “improvement” is noise; alert rate 10.7%→6.4%; confidence pass now attaches page images (per-page cost note)v0.6.5.md
v0.6.0v0.5.16flat (0.917 = 0.917)−32.5% total (advanced −44–55%)cacheRead −95%; one integrated-confidence completeness regression on long lists; v0.6 fixes v0.5.16 advanced-assessment timeoutsv0.6.0.md

What each entry contains

Section titled “What each entry contains”
  • TL;DR — safe-to-upgrade verdict + the headline accuracy/cost/latency deltas.
  • Methodology notes — the control model, disabled steps, doc set, and any cross-version config-compatibility handling (so the A/B is apples-to-apples).
  • Per-cell table — cost/recall/latency per config cell, both versions.
  • Regressions & improvements — anything past the aggregate.py --compare thresholds, with root cause verified from S3 output + metering.
  • Reproduce — the exact commands, plus honesty caveats (n, pricing estimate date).

Raw scored data for each side lives (unpublished) under benchmarks/results/<release>/<suite>/summary.{json,csv} + meta.json — one release directory per release, per RETENTION.md.