Fix Verification: v0.6.5 correctness fixes

Post-fix verification — the v0.6.5 correctness fix set

What this is: a live before/after measurement of the fixes raised after the v0.6.5 release benchmark and Configuration Guidance surfaced several silent data-loss hazards. It is not a release A/B — both sides are v0.6.5; the only variable is the fix set.

Method. All fix branches were merged into one integration branch and deployed to the same stack (IDPBench065, us-west-2) with --clean-build — required because one fix lives in a bundled YAML (the extraction prompt), which the build’s checksum cache can otherwise skip. Configs are byte-identical to the pre-fix run, written verbatim via --native-upload.

Every number below was taken with the extraction model verified from each run’s own billing metadata (metadata.extraction_model in the section result.json), not from the harness’s recorded axes. An earlier attempt at this verification was discarded because the harness silently ran the post-fix grid on a different model than the baseline — see What went wrong the first time.

Supporting data: the coresynth 70-run grid and the intconf repeated-measures run, both on Sonnet 5, plus security/test-results/0.6.5-verify/. Per benchmarks/results/RETENTION.md only one complete set is retained per release, so those two slices are no longer in the working tree — recover them from git:

git checkout ec3eb05ae -- benchmarks/results/v0.6.5-fixed2-config-core/ \
                          benchmarks/results/v0.6.5-fixed2-intconf-sonnet5/

---

1. Headline: one confirmed large improvement, no regressions

aggregate.py --compare, 70 runs per side, both on Claude Sonnet 5:

CELL-LEVEL REGRESSIONS (0)
CELL-LEVEL IMPROVEMENTS (1)
  core-tt-simple-int: recall +0.606 (0.294->0.900); paired per-doc deltas n=7 spread±0.484
INCONCLUSIVE (6)   ← all within run-to-run spread; see §3

	pre-fix	post-fix
mean completeness recall	0.9011	0.9465
mean scalar accuracy	1.000	1.000
failures	0	0
total cost, 70 runs	$75.08	$68.47

2. The integrated-confidence hazard, per document

This was the most dangerous configuration in the matrix — it returned near-zero list rows while reporting COMPLETED with perfect scalar accuracy. tt-simple-int (Textract TABLES · simple · integrated), rows recovered:

document	rows	pre-fix	post-fix
tiny_form	5	5 / 5	5 / 5
small_narrow	100	100 / 100	100 / 100
longdesc_100	100	1 / 100	100 / 100
med_narrow	400	10 / 400	400 / 400
wide_400	400	5 / 400	400 / 400
manylists_400	400	5 / 400	400 / 400
large_narrow	800	0 / 800	239 / 800 ⚠️

Four documents go from ~1% to complete. The 800-row document improves from nothing to 30% but is still truncated — which is the expected shape of the remaining limit, not a surprise: values and confidence still have to fit one response. Two consequences worth stating plainly:

• separate confidence remains the recommendation for list-bearing schemas. In the repeated-measures run below it was complete 4/4 and cheaper.
• The residual truncation is now detectable. With minItems set on the list, the 239/800 case raises extraction_list_truncated instead of passing as a clean success. That detection is the point of the companion fix.

Repeated measures on the same document

Because the failure is non-deterministic, --suite intconf runs the cell 4× with a separate control on the same document (longdesc_100, Sonnet 5 verified per run):

cell	pre-fix	post-fix
simple / integrated	0 of 4 complete	3 of 4 complete
simple / separate (control)	4 of 4 complete	4 of 4 complete
cost	—	integrated ~$0.27 vs separate ~$0.22

Honest reading: a large improvement (0/4 → 3/4), not a cure. One run in four still returns zero rows. separate is both more reliable and cheaper here.

3. What the six “inconclusive” results are

Two cells scored lower post-fix. Neither is touched by any fix in this set (advanced-integrated uses the agentic prompt, not the TopK prompt; Bedrock-LLM OCR uses the plain prompt), and both sit inside their measured run-to-run spread:

cell	recall	paired spread	verdict
tt-adv-int	1.000 → 0.857	±0.378	inconclusive
llm-simple-sep	0.860 → 0.708	±0.488	inconclusive
tt-adv-sep	0.857 → 1.000	±0.378	inconclusive (not claimed as a win)

tt-adv-int’s single loss is longdesc_100 returning a null list — the long-standing advanced tool-decline hazard documented at v0.6.0 and in config-guidance §2, landing on a different cell than last time. llm-simple-sep swung in both directions across documents (tiny_form 5→1 rows, wide_400 161→400), consistent with its already-documented instability.

Note that the comparison declines to credit tt-adv-sep’s +0.143 as an improvement either. That symmetry is deliberate — it is the same variance-aware gate applied in both directions, and it is new: previously quality metrics were compared on the raw mean shift, which is how the v0.6.5 release report came to publish a noise-driven “improvement” that had to be retracted by hand.

4. Security

All four gates PASS on the fixed build (security/test-results/0.6.5-verify/):

Test	Gate	Detail
SRT — SAST & deps	PASS ✅	0 open/reopened HIGH on CI-visible paths
RBAC — static	PASS ✅	0 fail, 2 known gaps
RBAC — dynamic	PASS ✅	556 checks, 0 hard fail
ZAP DAST	PASS ✅	High=0

The SRT line is stricter than the one in the pre-fix snapshot: the gate now counts reopened findings, which it previously ignored.

5. Infrastructure fixes verified on a stack that had actually failed

The workflow-concurrency leak was reproduced on the stack that carried it (active_count stuck with zero executions running) and the recovery observed end to end:

RECONCILED leaked concurrency counter: 100 -> 0 (running executions: 0,
  previous sample: 0). 100 slot(s) had been held by workflows that already ended.
Capacity recovered after reconciliation; proceeding with <document>

…and the previously-blocked document then processed to COMPLETED.

Verifying it found two further bugs in the fix itself, both now fixed and regression-tested: the reconciler rewrote its own observation timestamp on every refused increment (so the grace window never elapsed and the correction never fired), and a stale observation could outlive its episode and skip the two-sample safeguard. Neither was visible in unit tests; both needed a live stack.

What went wrong the first time

The first attempt at this verification produced a plausible-looking 70-run grid that was wrong, and it is worth recording why.

Benchmark config files were named <cell>__<class>.yaml with no suite in the name, but suites share cell names. Building a second suite overwrote the first suite’s config files while leaving its index untouched, so the index advertised extraction_model: sonnet5 while the file it pointed at was pinned to sonnet-4-6. The run executed on sonnet-4-6, recorded “sonnet5” in its own metadata, and was compared against a sonnet5 baseline. Every number was attributed to the wrong configuration.

That confound alone produced two entirely spurious findings — a scalar-accuracy “regression” (1.000 → 0.857 on two documents, actually just a different model formatting a date differently) and a new “FAILURE” — neither of which reproduced once the comparison was run properly. It was caught only by reading metadata.extraction_model out of a run’s S3 output.

Config filenames are now namespaced per suite, and run_matrix.py refuses to launch unless every config file matches the axes its index claims. The lesson for anyone reading a benchmark comparison here: verify the model from the run’s own billing metadata before trusting a before/after, exactly as this page does.