Repository navigation
fix: keep URL passwords and query strings out of object store errors - #7556
Draft
dwsmith1983 wants to merge 29 commits into
Draft
dwsmith1983 wants to merge 29 commits into
dwsmith1983 wants to merge 29 commits into
Conversation
… logs The native Azure store printed whole URLs in its errors and debug lines, and handed them to object_store, whose own errors quote the URL. A password in the userinfo or a SAS token in the query could end up in task errors and logs. object_store now gets a copy without the password, query and fragment, keeping the container it reads from the username. Messages name the URL by scheme, container and host only.
… driver messages object_store_authority kept an ABFS URL's password, so it reached the store cache key, the registration URL, debug lines and the deregister_store error. object_store_backend parsed the raw URL and normalize_object_store_url echoed input it could not parse. On the driver, AbfsAuthResolver put the URI into forwarded errors and a warning, and the mixed-store decline reason and objectStoreKey kept the raw authority. All of these now name a URL by scheme, container and host only. The URL helpers move to objectstore/url_redaction.rs, NativeConfig gains a matching displayAuthority, objectStoreKey matches the native key again, and the resolver scrubs the URI's password and query from Hadoop messages it forwards. Credentials only come from configuration, so URLs that differ only by password share a store.
The message no longer needs the host, so the redacted helper that built it is removed.
Drop comments that restate the code or a test's name, and shorten the rest to why each part of a URL is kept or removed.
Native now builds the key callback path from the authority without its password, so the retriever registered under the listed path was never found. Register and look up retrievers by one key: S3 aliases folded to s3a, and for abfs/abfss the scheme lowercased and the password dropped by the same rule as native. The missing-retriever error prints that key.
Secrets were replaced one after another, so a URI password inside a configured credential was masked first and the rest of the credential stayed in the message. Match every secret in the same text and replace each run of masked characters with one marker.
The object store key is built per file during fallback checks and again per split in partition packing. Return the raw authority as is when it has no `@`, or no `:` before the last `@`, instead of rebuilding the same string.
…layAuthority nativeFallbackReason stripped the URL password with its own copy of the logic in NativeConfig.authorityWithoutPassword. Use the shared helpers so the resolver renders a location one way in every message: lower-case scheme, no password, path or query.
The ABFS resolver skipped secrets under eight characters, so a short password in the URI reached the driver warning when Hadoop named the container-scoped keystore alias it could not recover. URI secrets are now hidden at any length; the floor applies only to configured values. The password is read from the raw authority, split as Hadoop splits the container, so a registry-based authority (an underscore in the host) no longer loses it.
object_store's builders and its HTTP store quote the URL they are given in their errors, and several Comet paths printed the URL themselves, so a password in the userinfo or a SAS token or signature in the query ended up in task failures and logs. The fallthrough to object_store's parse_url, the S3 store and its template now work from a URL without userinfo or query; an az URL keeps its username, which object_store reads as the container. The S3 "not S3" error, the blob alias parse and missing bucket errors, and the hostless HDFS error name only the scheme. None of these stores reads the password or the query, so only the messages change.
This was referenced Nov 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #7128.
This is the first of four pull requests for #7128, each covering one area. They build on #7127, which adds the URL redaction helpers they use, so #7127 merges first. Suggested order after it:
They target
main, so until #7127 merges each one also shows its commits.Rationale for this change
Several native object store paths print the whole URL in an error message or keep it in a stored value. A URL can carry a password in its userinfo or a SAS token or signature in its query string, so those errors put secrets in task failures and logs. object_store's own errors quote whatever URL they are handed, so passing it the raw URL is enough to leak.
What changes are included in this PR?
parse_urlhands it the URL without userinfo or query. AnazURL keeps its username, which object_store reads as the container.blobstyle aliases no longer quote the URL.hdfsURL with no host reports its scheme instead of the whole URL.None of these stores reads the password or the query from the URL, so only the messages change.
How are these changes tested?
Native tests build each URL with a password, a username and a signature query, then check that none of them appears in the error or stored URL while the host or scheme does. One covers the HTTP store's request error, and one checks that an
azURL keeps its container. Each test fails on the base branch.