[andygrove]

Which issue does this PR close?

Closes #6934.

Rationale for this change

Since #6543, the native Iceberg scan falls back to Spark whenever a projected struct, list or map column has a nested field that some schema in the table's history lacks or names differently. That was needed because iceberg-rust matched a data file's nested fields to the table's by position (apache/iceberg-rust#2617). apache/iceberg-rust#3255 matches them by field id and fills absent optional fields with NULLs, so most of those fallbacks are no longer needed.

What changes are included in this PR?

• Moves the iceberg and iceberg-storage-opendal pin to iceberg-rust b154603a. Cargo.lock changes only those three crates, and no Comet source change was needed to compile.
• Replaces IcebergReflection.nestedFieldsAddedOrRenamed with structsWithNoFieldsInCommon. The scan now falls back only when a projected struct, at any depth, shares no field id with the same struct in some schema in the table's history. That is the one shape where the native read still differs from Spark: it returns a struct of NULLs or NULL depending on what else the scan reads, where Spark returns NULL for a top-level struct and a struct of NULLs for a nested one.
• Runs that check on the task schema the scan reads (CometIcebergNativeScanMetadata.taskSchema, from perf: read only the nested fields Spark keeps in the native Iceberg scan #6725), instead of the table and scan schemas. With nested schema pruning on, that schema holds only the fields Spark keeps, so a read of only s.b falls back when a file predates s.b, while a read of s.a stays native.
• Narrows the schema-evolution limitation bullet in docs/source/user-guide/latest/iceberg.md.
• Notes under Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping #6794 a new failure mode. For data files with neither field ids nor a name mapping, iceberg-rust matches a struct's fields by position only when the counts are equal. A pruned read of some of a struct's fields now fails with "cannot reconcile struct fields by id" where it used to return the file's values (Spark returns NULL). A read of the whole struct is unchanged.

The name-mapping fallback from #7085 (#6790) stays. Removing it with this pin makes a pruned read of a map value fail in iceberg-rust with "Map field must have exactly 2 fields", when the table was imported and the files' physical names differ from the mapping, so every projected leaf under the map value is missing. Other reads of such tables match Spark. #6790 remains open.

How are these changes tested?

In CometIcebergNativeSuite:

• The four "falls back" nested-evolution tests (added to a struct, added inside an array or map, dropped and re-added, renamed) now assert native reads that match Spark. The self-join query allows two native scans. The VERSION AS OF tests added by perf: read only the nested fields Spark keeps in the native Iceberg scan #6725 now expect native scans too, with nested schema pruning on and off.
• A new test covers the fallback shapes: top-level struct, list element, nested struct, and a pruned s.b read. A struct that kept one of its fields stays native.
• The test for data files without field ids now expects the pruned read to fail, as described above.

On Spark 4.1 with Iceberg 1.11, CometIcebergNativeSuite gives 134 passed, 0 failed, 2 canceled, on top of apache/main at 6fe73d9. I have not run the Spark matrix or the other Iceberg shards.

[rich7420]

A few compatibility questions inline.

[rich7420]

Please preserve fallback for evolved ID-less structs: adding s.b now throws, while renames can expose incorrectly matched values.

[andygrove]

You're right that this changes behavior. For files with neither field ids nor a name mapping, iceberg-rust now matches a struct's children by position only when the counts are equal, so a pruned read of such a struct fails with "cannot reconcile struct fields by id". Before the bump it returned the file's values. I don't think we can keep the old fallback for this case. Nothing at planning time tells us whether a file has ids, and the old guard only fired when the schema history showed a nested add or rename, so keeping it would put back most of what this PR removes. Spark reads every nested field of these files as NULL, so the native scan already differs there. That's tracked in #6794 and listed under "Reads that differ from Spark" in iceberg.md, and the upstream side is apache/iceberg-rust#3436. Does it work for you to leave this as a documented divergence?

[rich7420]

Could default validation use the task schema? With pruning disabled, unprojected nested defaults now reach unsupported native reads.

[andygrove]

Good catch. With nested pruning off the task schema holds nested fields the query's schema doesn't, and iceberg-rust errors on a nested default the file lacks. The default check now also covers the projected columns of the task schema, with a test, in bc89e2c.

[rich7420]

Could equality deletes bind case-sensitively? This pin rejects id/ID because Comet still sets case_sensitive=false.

[andygrove]

Agreed. Comet's native tasks were built case-insensitive, and the equality-delete predicate is the only thing that reads that flag. Its names come from the task schema, so they match exactly. Tasks are now case-sensitive in bc89e2c, with a test that fails without it.

[rich7420]

Please preserve FixedSizeList→List conversion; Parquet’s Arrow schema hint can expose this newly rejected input.

[andygrove]

This one is real. In the new pin, RecordBatchTransformer only converts List and LargeList file columns to a List target, so a FixedSizeList from a Parquet Arrow schema hint now errors with "cannot promote". It is tracked upstream as apache/iceberg-rust#3471, and a fix is in progress. It only affects files whose embedded Arrow schema uses FixedSizeList, which Spark and Iceberg writers don't produce, so I'd like to merge as is and pick up the fix with the next iceberg-rust bump. Does that work for you?

[comphead]

Thanks @andygrove 👍