Native Iceberg writes commit brotli files that no JVM reader can read without Hadoop's BrotliCodec #6960

Describe the bug

parquet-mr compresses and decompresses brotli pages with Hadoop's org.apache.hadoop.io.compress.BrotliCodec, which it loads by name. Neither Spark nor Iceberg ships that class. It is not in Spark 4.1.1's jars/, and the Iceberg Spark runtimes (checked for 1.5.2 and 1.11.0) bundle a copy of Parquet that looks up the same class. Without a brotli codec jar on the classpath, iceberg-java fails a brotli write with BadConfigurationException: Class org.apache.hadoop.io.compress.BrotliCodec was not found.

The eligibility gate admits any brotli table for the native writer, and parquet-rs compresses brotli without that class. So with native Iceberg writes enabled, a brotli write that fails without Comet succeeds with it, and it commits files that no JVM Parquet reader on the same cluster can read. Spark without Comet fails to read the table, and so does Comet whenever its scan falls back to Spark's reader.

Steps to reproduce

Spark 4.1 with iceberg-spark-runtime-4.1_2.13:1.11.0, a Hadoop catalog, spark.comet.write.iceberg.enabled=true and spark.comet.exec.localTableScan.enabled=true (so the VALUES input runs in Comet):

CREATE TABLE local.db.b (id bigint, s string) USING iceberg
  TBLPROPERTIES ('write.parquet.compression-codec'='brotli');
INSERT INTO local.db.b VALUES (1, 'a'), (2, 'b');  -- plans CometIcebergWrite and succeeds

SET spark.comet.enabled=false;
SELECT * FROM local.db.b;
-- BadConfigurationException: Class org.apache.hadoop.io.compress.BrotliCodec was not found

Without Comet, the INSERT itself fails with the same error.

Expected behavior

The native writer should decline a brotli write unless org.apache.hadoop.io.compress.BrotliCodec can be loaded, so that the write falls back and fails as it does without Comet. With a brotli codec jar on the classpath, iceberg-java can write and read brotli, and the write can stay native. The user guide's eligibility table should say so.

Additional context

Found while deciding #5643. The lz4, snappy and zstd codecs, and zstd levels 0 and -3, all write successfully on iceberg-java in the same setup.

This issue first also covered gzip with a numeric write.parquet.compression-level, which the native writer accepts and iceberg-java fails: Hadoop reads zlib.compress.level as the name of a ZlibCompressor.CompressionLevel constant, such as BEST_SPEED, so 5 throws. That happens on Hadoop 3.4, and on Hadoop 3.3 when Hadoop's native zlib library is loaded. The native files are ordinary gzip pages, which every reader decompresses whatever the level (Spark 4.1.1's parquet-mr reads a gzip level 5 file without trouble), so that difference is better listed among the accepted divergences in iceberg-writes.md than reproduced.