Repository navigation
Allow disabling remote signing in REST Catalog #3888
Description
Feature Request / Improvement
When using a REST Catalog (e.g. Lakekeeper), pyiceberg gets remote signing details from the catalog.
However, sometimes it would be useful to not use remote signing, in the case where the process performing the operations has access to the S3 endpoint through secret access keys. This is because remote signing requires a network round trip (plus any authorisation on server side).
It would be great if we could disable remote signing by respecting the s3.signer catalog property if it is empty.
The FileIO object gets created in pyiceberg/catalog/rest/__init__.py:
return Table(
identifier=identifier_tuple,
metadata_location=table_response.metadata_location, # type: ignore
metadata=table_response.metadata,
io=self._load_file_io(
{**table_response.metadata.properties, **table_response.config}, table_response.metadata_location
),
catalog=self,
config=table_response.config,
)This doesn't use the catalog properties that have been passed to the REST catalog init as far as I can tell.
I might be wrong, but I don't think Support storage-credentials in REST catalog LoadTableResult- #3042
fixes this, because it still uses the response from the catalog.
Thanks!
Activity
emecii commentedon Sep 7, 2026
I traced the current flow:
RestCatalog._load_file_io()already mergesself.properties, but the per-table response config is merged later, so a client-provideds3.signer=""cannot currently override a serverS3V4RestSignervalue.\n\nWould a narrow rule be acceptable: when the client explicitly supplies an emptys3.signer, apply that value after the table response solely as an opt-out; otherwise preserve today’s response-wins precedence? I would cover bothTableandStagedTableconstruction paths and leave all other properties unchanged.BenCradock commentedon Sep 8, 2026
Yes that makes sense to me - happy for it to be only on an explicit empty
s3.signer. Thanks!Ankitraj-builds commentedon Oct 21, 2026
I'd like to work on this if no one else is.
Since #3956 the FileIO property order for RestCatalog is table metadata < catalog config < server config < vended credentials, so a client-side empty s3.signer is still overridden by the server's signer. I'd follow the approach suggested above: only an explicitly set empty s3.signer from the catalog config is applied after the server config, as an opt-out, for both the Table and StagedTable paths. Everything else keeps the current order. I'd add tests in tests/catalog/test_rest.py for both paths.
Could a maintainer confirm this behavior is wanted before I open a PR?
Ankitraj-builds commentedon Oct 21, 2026
I've opened #4134 for this. It follows the approach discussed above: setting s3.signer to an empty string in the catalog config turns off remote signing, even if the server config sends a signer. Nothing changes if s3.signer is not set.
Happy to change it if a maintainer prefers a different behavior.
Fokko commentedon Oct 22, 2026
Looping in @c-thiel WDYT?