Repository navigation
aws.XXX options are not recognized in SQL API for CREATE EXTERNAL TABLE with AWS S3 location #970
Description
Describe the bug
Hi
I was trying to create external table from AWS S3 using examples from Datafusion CLI docs and SQL DDL docs, see To Reproduce section below. But looks like AWS-specific options for external tables doesn't work with SQL API in the same way as it described in CLI docs
Exception: DataFusion error: Configuration("Could not find config namespace \"aws\"")
I can successfully register and read the same parquet with the same credentials via register_object_store + register_parquet functions, so there's no problem with the data or S3 connection
To Reproduce
import datafusion
ctx = datafusion.SessionContext()
raw_data_df = ctx.sql(f"""
create external table raw_data
stored as parquet
options
(
'aws.access_key_id' '***',
'aws.secret_access_key' '***',
'aws.region' 'eu-central-1'
)
location 's3://some-bucket/some-folder/'
""")
raw_data_df.show()Expected behavior
create external table command works with aws.xxx options
Additional context
v 42.0.0
Activity
timsaucer commentedon Sep 16, 2026
I know this is a very old issue, but I've looked into the root cause of it. The
datafusion-cliadds in a special configuration option for AWS credentials. Indatafusion-pythonthe more canonical way to add these credentials is documented at https://datafusion.apache.org/python/user-guide/data-sources.html#object-storeIf you have a specific need to be able to add credentials via configuration we could go the route of adding a configuration extension as shown here in datafusion-cli but for right now I suspect using the credentials from the OS is the better path forward.
If you need that configuration option you could either add it as a user defined configuration option and expose it to python or we could consider adding it to this project. For the latter, we'd probably want more justification since the canonical support method already exists. Please reopen this issue if you need further asssistance!