fix: read fs.comet.libhdfs.schemes for the scan rule from the Hadoop conf

[dwsmith1983]

Which issue does this PR close?

Closes #6837.

Rationale for this change

CometScanRule read fs.comet.libhdfs.schemes from the SQLConf entry spark.hadoop.fs.comet.libhdfs.schemes, while native routes a scheme through libhdfs from the Hadoop conf the scan forwards. Spark's SessionState.newHadoopConf copies SQLConf keys as they are, without dropping the spark.hadoop. prefix, so the two could disagree. A list set only in core-site.xml made the rule fall back on a scheme native could read, and a runtime SET spark.hadoop.fs.comet.libhdfs.schemes made it claim a scan native then failed to open ("Unable to recognise URL").

Reading the list from the scan's conf alone is not enough either. Native picks libhdfs from the list the scan forwards, which includes reader options, but libhdfs itself builds a fresh Hadoop Configuration from the executor classpath and never sees those options. A scheme added only through a reader option, with its fs.<scheme>.impl also set there, would be claimed and then fail natively with No FileSystem for scheme.

What changes are included in this PR?

• A new NativeConfig.resolveScanLibhdfsSchemes(sessionConf, scanConf) takes the session's Hadoop conf (newHadoopConf()) and the scan's (newHadoopConfWithOptions(relation.options), the conf the scan serializes for native). It returns the schemes listed by both, which native reads through libhdfs, and the schemes listed only by the scan. Each list defaults to hdfs when the key is unset or blank, as native is_hdfs_scheme does (NativeConfig.resolveLibhdfsSchemes). fs.comet.s3Compliant.schemes was already read from the Hadoop conf.
• CometScanRule and the Delta contrib's gate (DeltaScanSupport.declineReason) both use it, so they cannot drift:
  • A root path whose scheme is listed only through reader options falls back, even when object_store could read it, because native would still route it to libhdfs. The reasons are "Native Parquet scan does not support filesystem scheme(s) X listed in fs.comet.libhdfs.schemes only through reader options" and the same starting "Native Delta scan". The Delta gate also checks the selected data file and deletion vector paths.
  • A scheme listed by both is libhdfs, as before.
  • Anything else goes through the existing object_store check. A reader option that leaves out a session-listed scheme therefore makes it non-libhdfs, as it is natively.
• The spark.hadoop.fs.comet.libhdfs.schemes doc says it is read from the Hadoop configuration and must be set before the SparkSession is created, that a scheme listed only through a reader option is not read natively, and that libhdfs reads a listed scheme's settings, such as fs.<scheme>.impl, from the executor's core-site.xml.

What users see, only where the sources disagreed (query results do not change, only native or fallback):

• A list set only in core-site.xml or as the bare key fs.comet.libhdfs.schemes now gets the scan run natively instead of falling back.
• A runtime SET spark.hadoop.fs.comet.libhdfs.schemes=... no longer changes what the planner claims. A scheme it added now falls back instead of failing at execution.
• A runtime SET of that key that leaves out hdfs no longer makes hdfs:// scans fall back, since native never saw it. Set the list at startup to change it.
• A scheme listed only in a reader or table option falls back, since libhdfs cannot see the options it would need.

The list set in the Spark conf at startup reaches every place and behaves as before. Passing a scan's Hadoop options through to libhdfs is left to #6967.

#6753 adds the same resolveLibhdfsSchemes (public there) and the same unit test, so whichever of the two lands second resolves only a small conflict.

How are these changes tested?

CometScanSchemeFallbackSuite writes a Parquet table once and, in one table-driven test, applies CometScanRule to the Spark plan of a read of it in five cases:

• the list as the bare Hadoop key: a fake:// scan is claimed. On main it is declined.
• the list as a reader option: the fake:// scan falls back with the reader-options reason.
• the list set at runtime under the spark.hadoop. key: the fake:// scan falls back with "Unsupported filesystem schemes: fake". On main it is claimed.
• the session lists fake,hdfs and a reader option lists only hdfs: the fake:// scan falls back with "Unsupported filesystem schemes: fake".
• a reader option lists file and the same table is read through file:: the scan falls back with the reader-options reason, though object_store reads file natively.

DeltaScanContribSuite writes a Delta table once and plans a read of it in the same five cases, checking for the native Delta scan or the matching "Native Delta scan does not support" reason. Both tests run every case and fail once, naming each case that broke.

NativeConfigSuite covers both resolvers: unset, blank, and a list with spaces and mixed case for resolveLibhdfsSchemes, and the libhdfs and scan-only splits for resolveScanLibhdfsSchemes.

Each case is load-bearing. With the gates using only the scan's list, the reader-option and reader-only file cases fail. With only the session's list, the reader-option, removal, and reader-only file cases fail. With the scan-only check dropped, the reader-option and reader-only file cases fail. These hold in both suites.

CometScanSchemeFallbackSuite, NativeConfigSuite and DeltaScanContribSuite pass on Spark 3.5 and 4.1, and the build compiles on the Spark 3.4 profile and with strict warnings on 3.5.