Repository navigation
Add s3.allow-delayed-open to PyArrowFileIO - #4109
Open
David-Banquet wants to merge 1 commit into
Open
David-Banquet wants to merge 1 commit into
David-Banquet wants to merge 1 commit into
Conversation
pyarrow's S3FileSystem starts a multipart upload as soon as an output stream opens, so every small object costs CreateMultipartUpload, UploadPart and CompleteMultipartUpload (apache#4093). The new property passes allow_delayed_open to S3FileSystem (pyarrow >= 21), which sends objects under the part size as a single PutObject. It is off by default.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4093
Rationale for this change
pyarrow's
S3FileSystem.open_output_streamstarts a multipart upload as soon as the stream opens, soPyArrowFileIOwrites every S3 object, however small, with three requests:CreateMultipartUpload,UploadPartandCompleteMultipartUpload. pyarrow 21 addedallow_delayed_openfor this: a stream that closes before reaching the part size (10 MiB) is sent as a singlePutObject._initialize_s3_fsbuilds the filesystem from a fixed set of properties, so there was no way to turn it on.This adds
s3.allow-delayed-open, passed toS3FileSystemfors3://andoss://. It is off by default, and setting it on pyarrow < 21 raises the sameImportErrorass3.ssl-ca-cert.Are these changes tested?
Yes.
test_pyarrow_s3_session_properties_with_allow_delayed_openchecks the option reachesS3FileSystem, andtest_pyarrow_s3_allow_delayed_open_requires_supported_pyarrow_versionchecks the version guard.Against real S3 (eu-west-1, pyarrow 25.0.1), counting requests from the SDK debug log:
s3.allow-delayed-open=trueversion-hint.textPutObjectcreate_table+appendof 3 rows (SQL catalog: 1 data file, 1 manifest, 1 manifest list, 2metadata.json)Are there any user-facing changes?
Yes, a new FileIO property, documented in
configuration.md. Nothing changes unless it is set.With it on, an error such as a missing bucket or a denied write is raised when the file is closed instead of when it is opened. It is the same
OSError, and Java'sS3OutputStreamalready behaves this way below its multipart threshold. I left the default off so the existing behavior does not move; happy to flip it if you would rather match Java.