Skip to content

fix: keep nulls in nullable int32 and int64 decimal parquet columns - #233

Merged
mchav merged 1 commit into
DataHaskell:mainfrom
skymanbp:fix/parquet-decimal-nulls
Sep 25, 2026
Merged

mchav merged 1 commit into
DataHaskell:mainfrom
skymanbp:fix/parquet-decimal-nulls

Conversation

@skymanbp

Copy link
Copy Markdown
Contributor

The DECIMAL conversion for INT32 and INT64 columns took the raw values with toVector and rebuilt the column without its null bitmap, so nulls read as 0.0 and the column was no longer nullable. It now maps the column with mapColumn, which keeps the bitmap, as the timestamp branch does. The new fixture was written by pyarrow 24.0.0 with store_decimal_as_integer=True.

Before:

int32_dec | int64_dec
----------|----------
 Double   |  Double
----------|----------
1.0       | 0.0
0.0       | 1234.5
2.5       | 0.0
0.75      | -5.0

After:

 int32_dec   |  int64_dec
-------------|-------------
Maybe Double | Maybe Double
-------------|-------------
1.0          | null
null         | 1234.5
2.5          | null
0.75         | -5.0

@mchav

mchav commented Sep 25, 2026

Copy link
Copy Markdown
Member

Great catch.

@mchav
mchav merged commit 3e98619 into DataHaskell:main Sep 25, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants