Skip to content

fix: read slot directories from the page prelude - #28

Merged
Nathan-D-R merged 4 commits into
mainfrom
fix/slot-directory
Oct 25, 2026
Merged

Nathan-D-R merged 4 commits into
mainfrom
fix/slot-directory

Conversation

@Nathan-D-R

@Nathan-D-R Nathan-D-R commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Description

Stacked on #27. Fixes the slot directory findings (S1 6, 7, 8, 9 and S2 13, 14) and the review findings B2, S4 and S5.

SlottedPage::parse no longer scans for the longest descending run in the first 0x300 bytes. It reads the slot count the page prelude declares (a u16 four bytes before the array, or six with a zero word between), requires it to equal the plaintext trailer's count byte, and takes exactly that many entries. The old scan stays as an unconfirmed fallback.

  • S2 13: a page with 4 to 7 rows has a directory (small_confirmed_directory_is_read).
  • S1 6: a stray 0x0040 after a zero gap is not a slot (stray_value_after_a_zero_gap_is_not_a_slot).
  • S1 7: two confirmed candidates are AmbiguousSlotDirectory; slot-like data the prelude does not confirm is NoSlotDirectory.
  • S2 14: a declared count longer than the valid entries is SlotDirectoryTruncated (entries read, count declared, and the byte offset where the array broke); a 200-entry directory past the old search window is read whole.
  • S1 8: Row::exact() is false for every row; decoding row headers is out of scope.
  • S1 9: overflow_prefix was removed in feat!: API foundation (errors, options, warnings, non_exhaustive) #26.

Review findings

  • B2. The confirmation was near-vacuous for counts 1 and 2: asking for a directory of the wrong count 1 finds one on 63 % of genuine pages. A declared count below 4 is now unconfirmed: NoSlotDirectory (or UnknownPrelude when nothing fits) in strict mode, a directory with UnverifiedDirectory under Leniency::DIRECTORY only when exactly one candidate exists. Test small_declared_counts_are_not_confirmed, including a page full of 01 00 words. The reviewer's null test (a wrong-count probe over 15,835 genuine pages) is the evidence; I did not add it as a unit test because it needs corpus pages.
  • S4. I had the diagnosis wrong. The in-directory check was offset < end, which also rejected rows that legitimately sit below the array (page 116: catalog rows at 0x2a..0x190, directory at 0x1a7); it is now array_start <= offset < end (rows_below_the_directory_are_rows). Lenient mode no longer throws numbers away: SlotDirectoryTruncated { found, expected } and SlotOffsetsInDirectory { dropped } are typed warnings (lenient_reads_count_the_rows_they_leave_out, lenient_reads_flag_a_rejected_directory).
  • Truncation, what I found. I looked for the reviewer's multi-run layout (89 entries, then 77 continuing from the page start). Across the pages I sampled the break most often falls at 0x1ff..0x205, i.e. at the sector 1 boundary, which points at a mis-decoded sector rather than a second run; where a continuation exists it is not unique (candidate starts 1, 3, 5, ... are all suffixes of one run). I did not find a rule that is safe to accept without ambiguity, so multi-run directories are still reported as truncated, with stopped_at so the sector boundary is visible. This is not solved.
  • S5. Ok(None) now means the page declares no rows and nothing looks like a directory. A trailer count above zero with no matching layout is Error::UnknownPrelude (a warning under Leniency::DIRECTORY). UnknownPrelude is constructed.

Open question recorded

SPECIFICATION.md section 6 now records the two multi-run shapes the re-review found (a run that continues at the start of the page, page 284; a value wrap mid-array, page 292) and that strict mode cannot read pages with 1 to 3 rows. S4 stays deferred.

Corpus delta

Decoded E pages with a Verified or Calibrated bv (a quarter of the files; 28,900 pages, fewer than before because block anchors no longer confirm), parsed strictly:

Result Pages
confirmed directory, 8 or more live 17,454
confirmed directory, under 8 live 3,253
declares no rows, none found 2,277
unconfirmed / small count 2,818
unknown prelude 1,392
offset inside the directory 1,763
truncated 1,942
ambiguous 1

Earlier numbers (64,000 pages with block-anchor pages included): 34,436 confirmed, 7,560 unconfirmed, 4,434 truncated, 3,538 offset-in-directory, 1,245 ambiguous. The sets are different, so the rates are what compare: ambiguous fell from 1.9 % to 0.003 % (counts below 4 no longer create candidates), offset-in-directory from 5.5 % to 6.1 % of the sample but now only genuine overlaps, truncated from 6.9 % to 6.7 %.

API breaks

On top of #26: SlotDirectory::declared_count(), SlottedPage::warnings() added; parse can fail with five errors; Error::SlotDirectoryTruncated gains stopped_at. SlotDirectory::slots() no longer includes trailing zero entries.

Tests changed

None existed for slotted.rs; 15 added.

Docs-site edits needed (release PR)

docs/docs/specification.md section 4.2: the confirmed prelude layout and the count-below-4 caveat. docs/docs/quickstart-rust.md: parse takes options.

OpenQBW follow-up

SlottedPage::parse(page, &opts)?, .directory(), .rows() (a Row with offset(), bytes(), exact()) replace parse, .directory, row_bytes() in attribution_schema.rs, attribution_content.rs and transaction_header.rs. Row slices are inexact; its record decoders must not treat the slice length as the row length. overflow_prefix is gone: record stitching that used it needs the prelude decoded first. A page with a small row count now needs Leniency::DIRECTORY.

Checklist

  • My commits are signed off (DCO): git commit -s
  • I have read CONTRIBUTING.md, including the vendor software and clean-room policy
  • This PR was not written or verified with the help of proprietary vendor software

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
opensqlanywhere-docs a621366 Oct 25 2026, 12:17 PM

The directory is the declared number of entries, confirmed against the
plaintext trailer count, instead of the best run found by scanning the
first 0x300 bytes. Small directories are read, long ones are read whole,
and truncated, ambiguous, unconfirmed or self-overlapping directories are
errors, or warnings under Leniency::DIRECTORY.

Signed-off-by: Nathan Riley <git@nathanriley.com>
… a directory read left out

Counts below four are unconfirmed, a declared count with no matching
layout is UnknownPrelude, offsets below the array are rows, and
truncation, ambiguity and left-out rows are typed errors and warnings
that carry the entries read, the count declared and the number dropped.

Signed-off-by: Nathan Riley <git@nathanriley.com>
Signed-off-by: Nathan Riley <git@nathanriley.com>
@Nathan-D-R
Nathan-D-R changed the base branch from fix/ap-deobfuscation to main October 25, 2026 12:13
@Nathan-D-R
Nathan-D-R merged commit 06faa48 into main Oct 25, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant