Skip to content

[SPARK-59699][CONNECT] Make DataFrameReader options case-insensitive - #58951

Open
haoyangeng-db wants to merge 2 commits into
apache:masterfrom
haoyangeng-db:fix-connect-reader-option-case-insensitive
Open

haoyangeng-db wants to merge 2 commits into
apache:masterfrom
haoyangeng-db:fix-connect-reader-option-case-insensitive

Conversation

@haoyangeng-db

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR makes Spark Connect's Python DataFrameReader.option handling case-insensitive.
When a new option matches an existing key ignoring case, it removes the existing entry and stores
the newest spelling and value.

It also adds plan serialization coverage for both named-table and data-source reads.

Why are the changes needed?

Spark reader option keys are case-insensitive and later calls should override earlier values.
Spark Connect currently stores options in a regular Python dictionary, so differently cased forms
of the same logical key can both reach the protobuf map. The server then resolves those duplicates
from an unordered map, which can select a value that does not match the user's call order.

For example, this should deterministically produce one versionAsOf option with value 2:

spark.read.option("versionAsOf", 0).option("versionasof", 1).option("versionAsOf", 2)

The fix preserves the spelling of the most recent key instead of lowercasing keys, so vendor option
names such as oracle.jdbc.timezoneAsRegion remain unchanged.

Does this PR introduce any user-facing change?

Yes. Spark Connect Python reader options now follow case-insensitive, last-call-wins semantics,
matching the classic DataFrameReader behavior. Workloads that supplied the same logical option
using multiple casing variants now deterministically use the most recently supplied value.

How was this patch tested?

  • Added regression coverage in SparkConnectPlanTests for mixed-case named-table options and
    repeated data-source options.
  • Ran pyspark.sql.tests.connect.test_connect_plan: 83 tests passed before the clean rebase onto
    the latest upstream/master; the rebase did not change the patch.
  • Ran Python compilation checks for both modified files after rebasing.
  • Ran git diff --check and line-length checks after rebasing.
  • A post-rebase suite rerun was blocked because the fresh worktree lacks the optional Connect test
    dependencies, and installing them was blocked by DNS resolution to PyPI.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex 0.154.0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant