[Nathan-D-R]

Description

Stacked on #26. Fixes the AP deobfuscation findings (silent-failure audit S1 1, 2, 3, 4, 5 and S3 17), and the review findings B1, B3, S1, S2, S3 and nits N1, N2, N5.

The decoder no longer picks a bv or step and hopes. Both are chosen deterministically (ties go to the lowest value, no hash-ordered container decides anything) and accepted only when something independent confirms them:

1. Trailer row count (Confidence::Verified): the row count in a slotted page's prelude must equal the plaintext trailer's count byte at 0xFF0. The trailer is not obfuscated, so this is the one check that confirms bv on pages whose padding is not zero. It needs a count of four or more (the pair (1, 0) / (2, 0) is among the commonest words in page data: a match was wrong on 5 of 887 pages with count 1). When two or more exact sectors, or a block anchor on the histogram top, point at another bv, the page is AmbiguousCalibration, not Verified.
2. Exact sectors (Confidence::Calibrated): two or more sectors are exact AP progressions and agree on bv, with no exact sector disagreeing. This assumes the constant plaintext is zero, which is why it is not Verified. Disagreeing exact sectors are AmbiguousCalibration; nothing, including a block anchor, breaks the tie in strict mode, and lenient mode takes the majority, never the minority.

A calibrated page elsewhere in the 16-page block with the same bv is reported (Evidence::BlockAnchor) but is not confirmation: about a third of blocks (32 % of 28,225, in 93 of 112 files) hold more than one bv. A page that only has a block anchor is Unverified.

3. Corroborated (Confidence::Corroborated, its own tier below Calibrated): exactly one exact sector, the histogram top and a block anchor all give the same bv, and no other candidate has any support. The re-review measured it against the trailer answer on 2,186 of 2,186 checkable pages (0 wrong; 730 of 730 with count 10 or more); my guard run found 0 disagreements in 724 pages at this PR's tip and 0 in 2,073 at the stack tip. It admits 5,972 pages nothing else confirms, including the 211 I pages OpenQBW's sysindex reads. The checkable pages exclude the dense ones, so the tier is separate and the code comment and SPECIFICATION.md cite the measurement as a bound for pages like the checked ones.

Anything else is Unverified: KeystreamUnverified unless Leniency::KEYSTREAM is set, in which case the result carries a warning. deobfuscate_with_bv applies the same rule to a supplied key. Each sector's step needs a peak of 24 and a lead of 12 over the runner-up (random sectors reach 9 to 13 on this corpus), else LowConfidenceKeystream or a per-sector warning. An all-zero page was never written: deobfuscate returns it as it is with Confidence::Unallocated and bv() == None (DeobfuscatedPage::unallocated()), not through the guesser. learn returns NoCalibration / NotObfuscated, counts lowercase pure pages, and the model is tied to its store (StoreMismatch, keyed by page count and the CRCs of the first and last page).

CRC does not help here: it covers the stored (obfuscated) bytes, so it checks page integrity, not a keystream guess.

Review findings

• B1: trailer match no longer outranks agreeing evidence; threshold 4; conflicts are AmbiguousCalibration. Regression tests build the reviewer's cases: a_count_of_one_does_not_verify_a_key_that_is_off_by_one (a 0x01-filler page with a 01 00 prelude at the wrong key, count 1; it fails with the old threshold) and a_trailer_match_that_exact_sectors_contradict_is_ambiguous (count 5, three exact sectors on the right key).
• B3: ApModel::deobfuscate_with_bv strict mode errors on an unsupported key (supplied_bv_reports_its_support).
• S1: block anchor is evidence, not confirmation (a_block_anchor_alone_is_not_confirmation).
• S2: dissenting_exact_sectors_are_never_resolved_by_an_anchor (five sectors vote 86, one votes 88, the block has 88).
• S3: a_zero_page_is_unallocated_not_guessed.
• N1: BvSource is TrailerRowCount, ExactSectors, Histogram, Supplied, Unallocated, so "Verified" has one meaning. N2: no made-up bv on the impossible branch. N5: the fingerprint includes the last page.

Corpus delta (112 QBW files, 456,521 pages, strict)

learn succeeds on all 112 files (4,510 of 4,649 pure pages calibrate). Strict deobfuscate of every page, now against my earlier numbers:

Type	Decodes (was)	of which Corroborated	AmbiguousCalibration	KeystreamUnverified	LowConfidenceKeystream
A (208,783)	172,703 (181,642 first version; 170,273 before Corroborated)	2,430	1,543	32,491	2,046
E (242,460)	66,693 (99,902; 63,362)	3,331	607	101,140	74,020
I (421)	211 (211; 0)	211	0	210	0
M, H, C, @, G (4,745)	4,586	0	27	112	20

The M, H, C, @, G row in the first version of this description (4,326 / 14 / 0 / 30) was wrong; this is the measured row, and its cells sum to the 4,745 pages of those types.

The shift from the first version is the review's findings applied: block-anchor-only pages (36,481 E, 11,397 A before) are Unverified unless the single-exact-sector tier applies, count-1/2 trailer matches no longer verify, and the ambiguous group fell because the anchor no longer creates conflicts. The sums of decodes plus errors are the page counts of the types.

Reading the groups:

• LowConfidence is right: dense sectors where no byte dominates, so no step can be told from its neighbours.
• Ambiguous is right: exact sectors of one page disagree, or a trailer match contradicts them.
• Unverified is right: nothing confirms the histogram guess. Leniency::KEYSTREAM returns these pages and lists every affected sector (sectors(), warnings()).

Not resolved: a global model of the per-sector steps. Within a block they follow a sliding window of 12 values, repeated in pairs and shifted one sector per page; leave-one-out on strong sectors predicts correctly when unanimous (99.5 %), but the window shifts phase in a third of blocks, so it is not used. Recorded in SPECIFICATION.md section 6.

API breaks

On top of #26: BvSource variants changed; learn can fail; deobfuscate can fail with the new errors; deobfuscate_with_bv can now fail in strict mode. DeobfuscatedPage::bv() returns Option<u8> (None for an unallocated page, which has Confidence::Unallocated; own commit). New public items: Confidence (Unverified, Corroborated, Calibrated, Verified, Unallocated), Evidence, SectorFit::{exact, confident}, DeobfuscatedPage::unallocated(), Page::is_unallocated().

Tests changed

deobfuscate_uses_per_page_bv_not_block_bv asserts Confidence::Calibrated. random_plaintext_is_low_confidence_not_garbage now sees KeystreamUnverified first in strict mode (nothing supports the key); a new test covers the dense-sector error with a supported key. trailer_row_count_rescues_a_space_padded_page has noise in the space sectors so no exact sector contradicts the trailer; the all-space version is space_padded_page_with_a_zero_region_is_ambiguous. The old private-helper tests went with the helpers they covered.

Docs-site edits needed (release PR)

docs/docs/ap-deobfuscation.md: replace "always succeeds" and "stays correct either way" with the evidence ladder, Confidence, Leniency::KEYSTREAM. docs/docs/specification.md: section 6 additions.

OpenQBW follow-up

Treat LowConfidenceKeystream / KeystreamUnverified / AmbiguousCalibration per page: skip the page, or opt in with ReadOptions::lenient() and drop rows that fall in a sector whose confident() is false. Its bv_recovery module (recover_bv_qb_data, recover_bv_any) guesses a key the same way; route it through ApModel::deobfuscate_with_bv and read confidence(). Its C.36 anchor is the same idea as Evidence::TrailerRowCount, and the same small-count caution applies.

Checklist

• My commits are signed off (DCO): git commit -s
• I have read CONTRIBUTING.md, including the vendor software and clean-room policy
• This PR was not written or verified with the help of proprietary vendor software

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	opensqlanywhere-docs	2f07e47	Oct 25 2026, 08:19 AM