Skip to content

[AMORO-4273] Add support for Lance format on Hive metastore - #4450

Open
zhangwl9 wants to merge 1 commit into
apache:masterfrom
zhangwl9:AMORO-add-hive-lance-dev
Open

zhangwl9 wants to merge 1 commit into
apache:masterfrom
zhangwl9:AMORO-add-hive-lance-dev

Conversation

@zhangwl9

@zhangwl9 zhangwl9 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Close #4273 .

Brief change log

  • Add a LanceHiveCatalog backed by Hive3Namespace, sharing the list/exists/
  • create/drop implementation hoisted into AbstractLanceCatalog.
  • Table data is opened from the HMS location with catalog credentials mapped to Lance storage options;
  • AMS and the dashboard enable Hive + Lance for S3/OSS/Local storage, exclusive like the REST catalog.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate
    lance-hive metastore-local storage

image image image

lance-hive metastore-s3 storage
image
image
image

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@zhangwl9
zhangwl9 force-pushed the AMORO-add-hive-lance-dev branch from d49a707 to beec2f9 Compare October 19, 2026 11:26
@zhangwl9
zhangwl9 force-pushed the AMORO-add-hive-lance-dev branch 2 times, most recently from 413d69f to 4d8d095 Compare October 19, 2026 11:47
@zhangwl9
zhangwl9 force-pushed the AMORO-add-hive-lance-dev branch from 4d8d095 to 0a8cc0e Compare October 20, 2026 06:11
@Override
protected void dropInNamespace(String database, String table, boolean purge) {
if (purge) {
namespace.dropTable(new DropTableRequest().id(tableId(database, table)));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve storage options and surface failures when purging table data

When credentials or a custom endpoint are configured only in the Amoro catalog (for example, a private S3/OSS bucket or MinIO), reads work through storageOptions(catalogProperties, location), but this purge path does not use those options. In the pinned lance-namespace-hive3:0.4.1 implementation, doDropTable() removes the HMS entry first, then safeDropDataset() calls Dataset.drop(location, Collections.emptyMap()) and catches deletion exceptions, logging only a warning. Consequently, dropTable(..., true) can report success after removing the metadata while leaving the underlying data intact; retrying by table name then cannot clean it up because the HMS entry is gone.

Please use the same resolved storage options as the read path for data deletion and propagate deletion failures instead of reporting a successful purge. A regression test should cover catalog-only credentials/custom endpoints and a failed data deletion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: Integrate Lance to support Hive as a metastore

2 participants