feat(metadata): add format v4 table metadata foundations by manuzhang · Pull Request #1070 · apache/iceberg-cpp

manuzhang · 2026-10-29T07:55:36Z

C++ rejects format-version: 4 table metadata. This draft adds version 4 metadata creation, loading, upgrades, and JSON round trips, together with the fixed-schema building blocks for the developing v4 manifest tree. The default table version remains 2.

Consume the creation-time format-version property without persisting it as an ordinary table property, and support upgrades while preserving older snapshots' manifest-list references. Schema selection may follow the version upgrade.
Match Java's nullable UUID reading for every version and nullable v4 location reading. An absent table location stays unset until supplied by a catalog. Data-path derivation requires a table base; an explicit absolute data URI is sufficient for the default location provider, while object-store providers require the table location for their context.
Preserve v4 root-manifest snapshot references. Share snapshot validation between parsing, table metadata serialization, and new-snapshot addition. Metadata parsing passes its format version explicitly and validates each snapshot once. Reject null snapshot entries before serialization. New v4 snapshots require roots; legacy snapshots retain non-empty manifest lists after an upgrade. Stale sequence numbers are checked first and remain retryable; parent timestamp failures with a fresh sequence are non-retryable validation errors.
Expose optional root_manifest in the snapshots metadata table, with a null manifest_list only for root snapshots. Existing column positions and field IDs are preserved, and both manifest columns support the metadata scan projection API.
Add FileTracking, DeletionVectorMetadata, and ManifestInfo data carriers with the spec's exact schemas, reusing one cached schema instance per type. Keep v4 tracking statuses separate from legacy manifest statuses.
Preserve the existing int8_t format-version interfaces. Strict JSON integer decoding rejects out-of-range and noninteger version values before conversion to int8_t. Ignore legacy identifier fields when reading v4 schemas. Reject writing current or historical schemas that still contain identifier fields until primary-key migration is supported, rather than silently dropping those fields.
Guard snapshot-expiration file cleanup before commit when v4 roots prevent reachability checks. Recheck final committed metadata before deletion; if a root appears after Apply, retain files and emit a warning while preserving the successful commit. No-op expiration and expiration with CleanupLevel::kNone remain available when expired-schema/spec pruning is disabled. CleanExpiredMetadata(true) with v4 roots is explicitly rejected before commit, including with kNone, because pruning also requires manifest-tree traversal.

Absent or null v4 constraints, constraint IDs, and snapshot constraint statuses use their defaults. Incorrect JSON types return parse errors; non-empty constraint/status arrays and nonzero constraint IDs remain explicitly unsupported. Table metadata owns the shared root-manifest predicate and location-validation rule. A missing table base for metadata writes reports the explicit metadata-path property as an alternative.

The v4 spec is still under development. This is a metadata foundation: v4 manifest-tree reads/writes, constraints, and identifier-to-primary-key migration return explicit unsupported errors. Typed content statistics, the full TrackedFile schema, column/file types, relative-path resolution for file I/O, and tree planning/I/O remain follow-ups.

Validation: based on main at d8ba4f6d and built in an isolated macOS Debug build with AppleClang 21, ICEBERG_BUILD_BUNDLE=ON, and ICEBERG_CXX20_COMPAT=ON. Arrow/Parquet 25 dependencies came from the existing dependency build; PR core, bundle, SQL catalog, and affected test sources were rebuilt from this branch.

table_test (289), json_serde_test (171), v4_table_metadata_test (25), v4_types_test (12), metadata_table_test (29), catalog_test (14), and schema_test (578): passed. Regressions cover current/historical identifier preservation, nullable constraints, real-version snapshot parsing, stale sequence precedence, shared location validation, and explicit metadata paths. One existing JSON test remains disabled.
table_update_test: 893 passed and 15 existing version-specific cases skipped. Both cleanup levels warn and retain manifest lists, manifests, and data files if committed metadata introduces a v4 root. Schema/spec pruning through roots fails explicitly before commit; plain kNone expiration remains available.
cpp20_compatibility_test, pre-commit on the full PR diff, and git diff --check: passed. The SQL catalog static library compiled with database connectors disabled.

Total for the affected suites: 2,011 unit tests passed, 15 skipped, plus the C++20 public-header compatibility target.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot

🟡 Changes recommended

Version inputs are narrowed to int8_t before validation, allowing invalid values such as 260 and -252 to silently upgrade tables to v4, where snapshot writes are unsupported.

1 open issue

Validate upgrade versions before narrowing to int8_t · New

_{Review effort: Balanced}

+  if (new_format_version < 1 ||
+      new_format_version > TableMetadata::kSupportedTableFormatVersion) {


Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues

When root_manifest is absent, ValidateForFormatVersion returns success without validating the… · New
This nulls manifest_list based on snapshot.manifest_list.empty() rather than on whether the… · New
These checks use direct JSON access and type-agnostic comparisons (`json[\"last-constraint-id\"] !=… · New
Validate upgrade versions before narrowing to int8_t
The precondition message is overly generic given the branching logic. When location is empty, the… · New

_{Review effort: Lite}

Introduce file tracking, deletion-vector references, and manifest summaries with schemas from the developing v4 specification. Keep v4 tracking statuses separate from legacy manifest statuses and retain the existing table-format support gate. Co-authored-by: Codex <codex@openai.com>

Accept format-version 4 for table creation, property upgrades and metadata JSON. Preserve optional table locations and root-manifest snapshots, retain legacy snapshots on upgrade, and validate v4 snapshot timestamps. Return explicit unsupported errors for unfinished manifest-tree I/O, constraints and primary-key migration. Cover catalog create/load/upgrade and staged creation alongside schema and serialization regressions. Co-authored-by: Codex <codex@openai.com>

Reject physical expiration cleanup before commit when v4 root manifests prevent reachability checks, and recheck final committed metadata before deleting files. Keep no-op expiration and CleanupLevel::kNone available. Pin the schema default-value fixture to v3 and cover retained roots, both cleanup levels and strategy selections, metadata-only expiration, and roots introduced after Apply. Co-authored-by: Codex <codex@openai.com>

Reject data-path derivation without a table base, preserve nullable UUID and v4 location parsing, classify parent timestamp failures as non-retryable, and allow upgrades before current-schema selection. Share snapshot format validation across metadata paths, expose root manifests in the snapshots metadata table, and avoid schema JSON copies outside the v4 identifier-field compatibility path. Co-authored-by: Codex <codex@openai.com>

Keep creation and upgrade version inputs as int32_t through the builder, update objects, catalog creation, and strict JSON decoding. Validate the supported range before storing the int8_t metadata version. Cover 260 and -252, signed integer bounds, cloned/decoded updates, and noninteger or out-of-range JSON values. Co-authored-by: Codex <codex@openai.com>

Reject empty legacy manifest lists across shared snapshot validation paths. Use root-manifest presence for inspection nulls and preserve projection support after the metadata table scan refactor. Give builder fixtures valid manifest paths. Strictly parse present v4 constraint arrays and integer IDs, and report actionable missing-location errors for each location provider. Co-authored-by: Codex <codex@openai.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

9 open issues

This loop dereferences snapshot unconditionally. Elsewhere in this PR (HasV4RootManifest)… · New
Returning NotSupported from Finalize() is risky because Finalize runs after the metadata update… · New
When root_manifest is absent, ValidateForFormatVersion returns success without validating the…
constraint-statuses is not type-validated. For malformed JSON (e.g., boolean/number/object), this… · New
These checks use direct JSON access and type-agnostic comparisons (`json[\"last-constraint-id\"] !=…
This nulls manifest_list based on snapshot.manifest_list.empty() rather than on whether the…
Validate upgrade versions before narrowing to int8_t
Type() allocates a new StructType (and vector) every call. These schema objects are typically… · New
The precondition message is overly generic given the branching logic. When location is empty, the…

_{Review effort: Lite}

Restore the original builder, table update, and catalog version types. Check upgrade JSON integers directly against the int8_t range before constructing an update. Adjust API tests to use representable unsupported values and retain JSON regressions for 260, -252, wider integers, and noninteger values. Co-authored-by: Codex <codex@openai.com>

Reject null snapshots before metadata serialization and strictly decode optional snapshot constraint-status arrays. Cache the fixed v4 schemas without changing their public return types. Treat v4 roots encountered after a successful expiration commit as a successful cleanup no-op. Cover both cleanup levels and prevent false finalization failure warnings. Co-authored-by: Codex <codex@openai.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

11 open issues

Returning NotSupported from Finalize() is risky because Finalize runs after the metadata update…
This loop dereferences snapshot unconditionally. Elsewhere in this PR (HasV4RootManifest)…
When root_manifest is absent, ValidateForFormatVersion returns success without validating the…
This still decodes format-version for the upgrade action as int8_t, which narrows during JSON… · New
constraint-statuses is not type-validated. For malformed JSON (e.g., boolean/number/object), this…
These checks use direct JSON access and type-agnostic comparisons (`json[\"last-constraint-id\"] !=…
This nulls manifest_list based on snapshot.manifest_list.empty() rather than on whether the…
Validate upgrade versions before narrowing to int8_t
The new precondition error is actionable but doesn’t indicate how to resolve it. Consider expanding… · New
Type() allocates a new StructType (and vector) every call. These schema objects are typically…
The precondition message is overly generic given the branching logic. When location is empty, the…

_{Review effort: Lite}

  }
  if (action == kActionAddSchema) {
    ICEBERG_ASSIGN_OR_RAISE(auto schema_json,
                            GetJsonValue<nlohmann::json>(json, kSchema));
    ICEBERG_ASSIGN_OR_RAISE(auto parsed_schema, SchemaFromJson(schema_json));


Copilot

🟢 Approval recommended

No active review issues remain that require resolution before approval.

3 open issues

This still decodes format-version for the upgrade action as int8_t, which narrows during JSON…
Validate upgrade versions before narrowing to int8_t
The new precondition error is actionable but doesn’t indicate how to resolve it. Consider expanding…

8 resolved since last review

Returning NotSupported from Finalize() is risky because Finalize runs after the metadata update…
This loop dereferences snapshot unconditionally. Elsewhere in this PR (HasV4RootManifest)…
When root_manifest is absent, ValidateForFormatVersion returns success without validating the…
constraint-statuses is not type-validated. For malformed JSON (e.g., boolean/number/object), this…
These checks use direct JSON access and type-agnostic comparisons (`json[\"last-constraint-id\"] !=…
This nulls manifest_list based on snapshot.manifest_list.empty() rather than on whether the…
Type() allocates a new StructType (and vector) every call. These schema objects are typically…
The precondition message is overly generic given the branching logic. When location is empty, the…

_{Review effort: Balanced}

Reject lossy identifier writes, preserve nullable constraint defaults, and validate stale snapshot sequences before parent timestamps. Parse table snapshots once against their real format version. Warn when postcommit v4 cleanup is skipped and reject unsupported schema and spec pruning before commit. Share location validation and the root predicate on TableMetadata, use feature-version constants, and simplify tracking-status decoding. Co-authored-by: Codex <codex@openai.com>

Copilot

🔵 Needs a closer look

Human assessment is needed because concurrent v4 root commits can invalidate planned metadata pruning, potentially removing a required spec despite the finalization guard.

2 open issues

1 resolved since last review

The new precondition error is actionable but doesn’t indicate how to resolve it. Consider expanding…

_{Review effort: Balanced}

Copilot AI balanced review requested due to automatic review settings October 29, 2026 07:55

Copilot AI reviewed Oct 29, 2026

Copilot AI balanced review requested due to automatic review settings October 29, 2026 08:43

manuzhang changed the title ~~feat(manifest): add foundational v4 metadata types~~ feat(metadata): add format v4 table metadata foundations Oct 29, 2026

Copilot AI reviewed Oct 29, 2026

Copilot AI balanced review requested due to automatic review settings October 30, 2026 09:01

Copilot AI reviewed Oct 30, 2026

Copilot AI balanced review requested due to automatic review settings November 1, 2026 15:15

Copilot started reviewing on behalf of manuzhang November 1, 2026 15:15

View session

Copilot AI reviewed Nov 1, 2026

View reviewed changes

Comment thread src/iceberg/table_metadata.cc

Comment on lines +770 to +771

if (new_format_version < 1 ||

new_format_version > TableMetadata::kSupportedTableFormatVersion) {

Copilot AI balanced review requested due to automatic review settings November 2, 2026 14:30

Copilot started reviewing on behalf of manuzhang November 2, 2026 14:54

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Comment thread src/iceberg/snapshot.cc Resolved

Comment thread src/iceberg/inspect/snapshots_table.cc Resolved

Comment thread src/iceberg/json_serde.cc Resolved

Comment thread src/iceberg/location_provider.cc Resolved

manuzhang and others added 6 commits November 2, 2026 23:16

manuzhang force-pushed the feat/v4-metadata-foundation branch from ce26c99 to 60ecac1 Compare November 2, 2026 15:23

Copilot AI balanced review requested due to automatic review settings November 2, 2026 15:23

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Comment thread src/iceberg/json_serde.cc Resolved

Comment thread src/iceberg/update/expire_snapshots.cc Resolved

Comment thread src/iceberg/json_serde.cc Resolved

Comment thread src/iceberg/manifest/v4_types.cc Resolved

Copilot AI balanced review requested due to automatic review settings November 2, 2026 15:57

Copilot stopped reviewing on behalf of manuzhang due to an error November 2, 2026 16:00
The job was not acquired by a runner even after multiple attempts.

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings November 2, 2026 16:27

Copilot stopped reviewing on behalf of manuzhang due to an error November 2, 2026 16:34
The job was not acquired by a runner even after multiple attempts.

Copilot started reviewing on behalf of manuzhang November 2, 2026 16:42

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings November 3, 2026 00:43

Copilot started reviewing on behalf of manuzhang November 3, 2026 00:44

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

		if (new_format_version < 1 \|\|
		new_format_version > TableMetadata::kSupportedTableFormatVersion) {

Conversation

manuzhang commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

manuzhang commented Oct 29, 2026 •

edited

Loading