Skip to content

fix(cpp): stop misreading pages after an empty value page - #976

Merged
ColinLeeo merged 2 commits into
apache:developfrom
ColinLeeo:colin/fix-empty-page-header
Sep 30, 2026
Merged

ColinLeeo merged 2 commits into
apache:developfrom
ColinLeeo:colin/fix-empty-page-header

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #971.

Bug

Java encodes an all-null aligned value page as a single zero varint. It writes no compressed size and no statistics, and the Java PageHeader.deserializeFrom returns as soon as it reads that 0. The C++ PageHeader::deserialize_from previously read another varint and, when requested, a statistic. After an empty page it therefore consumed bytes from the next page as the empty page's header, so every later page in that value chunk was parsed from the wrong offset.

For a column whose empty page is followed by a non-empty page this produces silently wrong results. The reported aligned table-model file has alternating filled and all-null tablets, so the failure repeats throughout the value chunk. A trailing empty page does not expose the offset error because there is no later page to consume.

Fix

Return early from PageHeader::deserialize_from when uncompressed_size_ == 0, setting compressed_size_ = 0 and leaving statistic_ null. This matches the Java empty-page encoding. The aligned readers already treat compressed_size_ == 0 as an all-null value page; the row count still comes from the time page, so the production change is limited to page-header parsing.

Tests

cpp/test/common/tsfile_common_test.cc

  • DeserializeEmptyPageConsumesSingleVarint checks that an empty header does not consume the following page header.
  • DeserializeEmptyPageWithStatisticConsumesSingleVarint covers the statistics-enabled path.

The existing Java/C++ compatibility fixture matrix now also generates null-page cases across UNCOMPRESSED, LZ4, ZSTD, and LZMA2 for all supported table value types. Each mixed case contains an empty first page, consecutive empty pages, an empty page followed by a non-empty page, an in-page null bitmap, and trailing empty pages. The Java generator asserts that empty pages use the one-varint wire form; both readers validate complete scans and ranges crossing page boundaries, with a non-null anchor column verifying aligned row preservation. The workflow continues to validate both Java-written fixtures with C++ and C++-written fixtures with Java.

Validation

  • C++ compatibility validation: Java-written fixtures pass.
  • Java compatibility validation: C++-written fixtures pass.
  • C++ full test binary: 938 passed, 3 skipped.
  • Java compatibility test: 2 executions passed with 0 failures.
  • Mutation check: restoring the old C++ header parser makes the Java-written mixed-null fixture fail during C++ query iteration.

…ge (apache#971)

Java writes an all-null value page as a single zero varint and omits the
compressed size and statistics (ValueChunkWriter.writeEmptyPageToPageBuffer).
The C++ PageHeader::deserialize_from always read a second varint and, when
requested, a statistic, so after an empty page it consumed the following
page's bytes as the empty page's header and misread every later page.

Return early when uncompressed_size_ == 0, matching the Java format. The
aligned readers already treat compressed_size_ == 0 as an all-null page, so
the fix is localized to PageHeader deserialization.

Add PageHeader unit tests covering an empty page followed by a non-empty
page, for both deserialize_stat values, asserting it consumes exactly one
varint and leaves the following page intact.
@ColinLeeo
ColinLeeo merged commit 05b5d93 into apache:develop Sep 30, 2026
48 checks passed
@ColinLeeo
ColinLeeo deleted the colin/fix-empty-page-header branch September 30, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CPP] Reader misreads every page after an empty value page (IoTDB aligned series with an all-null page)

1 participant