Skip to content

ORC-2238: Do not skip row groups for IS_NULL on fields nested in a struct - #2742

Open
mhatipog wants to merge 1 commit into
apache:mainfrom
mhatipog:nested-isnull-sarg
Open

mhatipog wants to merge 1 commit into
apache:mainfrom
mhatipog:nested-isnull-sarg

Conversation

@mhatipog

Copy link
Copy Markdown

What changes were proposed in this pull request?

SargApplier no longer uses row group statistics to answer IS_NULL for a column nested in a struct. The leaf evaluates to YES_NO_NULL instead, so those row groups are read. Other operators and top-level columns are unchanged.

Why are the changes needed?

ORC writes a struct field only in rows where all of its ancestors are present, so the field's statistics do not count rows where a parent struct is null. When the field itself is never null, hasNull is false and IS_NULL evaluates to NO, which skips row groups that contain matching rows:

// struct<id:int,s:struct<x:int>> with rows (1, NULL) and (2, {x: 5})
SearchArgument sarg = SearchArgumentFactory.newBuilder()
    .isNull("s.x", PredicateLeaf.Type.LONG)
    .build();
reader.rows(reader.options().searchArgument(sarg, null)); // 0 rows before this change

Reproduced on 1.9.8, 2.1.4 and 2.3.1. Spark (SPARK-60362) and Iceberg (apache/iceberg#19064) hit the same case through their ORC readers.

How was this patch tested?

Added TestRecordReaderImpl#testIsNullOnFieldUnderNullStruct, which fails without the change (reads 0 rows instead of 2). TestRecordReaderImpl, TestVectorOrcFile, TestReaderImpl, TestFilters, TestPluginFilters, TestOrcFilterContext, TestRowFilteringIOSkip, TestStringDictionary and TestEncryption pass.

Was this patch authored or co-authored using generative AI tooling?

Yes. Generated-by: an AI coding assistant, used to draft the change and the test; I reviewed and tested both.

mhatipog added a commit to mhatipog/spark that referenced this pull request Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant