Repository navigation
Inconsistencies between RecordBatch and DataFrame schemas cause to_arrow_table to fail #1314
Description
Describe the bug
When the nullability of a RecordBatch column does not match with the DataFrame's schema, the conversion to a pyarrow table fails.
To Reproduce
from datafusion import SessionContext
ctx = SessionContext()
ctx.sql("create table t_(a int not null)").collect()
ctx.sql("insert into t_ values (1), (2), (3)").collect()
ctx.sql(f"copy (select * from t_) to 't.parquet'").collect()
ctx.register_parquet("t", path)
pyarrow_table = ctx.sql("select max(a) as m from t").to_arrow_table()...
pyarrow.lib.ArrowInvalid: Schema at index 0 was different:
m: int32
vs
m: int32 not null
Expected behavior
Execute without crashing.
Additional context
Activity
IshaGudewar commentedon Dec 4, 2025
Thanks for opening this issue and the fix!
I reviewed the problem and the proposed solution in PR #1315, and I can confirm that the root cause is the schema mismatch between:
the DataFrame schema (which may mark fields as NOT NULL), and
the RecordBatch schema (which often marks aggregated columns as nullable).
PyArrow requires all RecordBatch schemas to match exactly, so this mismatch causes:
ArrowInvalid: Schema at index 0 was different
To help strengthen the PR, I will add:
A regression test for the zero-record-batch case, verifying that when no batches are returned, to_arrow_table() still uses the DataFrame schema.
A test for an aggregation that returns NULL, such as max(a) on an empty table, confirming the output is [None] and schema nullability is handled correctly.
I will contribute these tests to PR #1315 shortly.
nuno-faria commentedon Dec 4, 2025
Thanks @IshaGudewar, but those tests have already been added.