Skip to content

Add Google Cloud Storage support for external payloads - #1980

Open
diwu-sf wants to merge 6 commits into
temporalio:mainfrom
diwu-sf:codex/gcs-storage-driver
Open

diwu-sf wants to merge 6 commits into
temporalio:mainfrom
diwu-sf:codex/gcs-storage-driver

Conversation

@diwu-sf

@diwu-sf diwu-sf commented Oct 16, 2026 •

Copy link
Copy Markdown

Add an experimental Google Cloud Storage driver so Temporal clients and workers can store large payloads in GCS and pass references through workflow history.

The driver follows the Python S3 driver's API, bucket selection, and object key layout. It uses the optional gcloud-aio-storage library for async I/O. The application owns and configures the storage client.

Uploads use a single request, with resumable uploads disabled. Existing objects are left unchanged during concurrent uploads. Stored payloads include their metadata, and downloads verify SHA-256. The references can also be read by the Java GCS driver in temporalio/sdk-java#3151.

Includes setup documentation, Python tests, and a changelog entry.

Testing

  • The GCS and S3 suites passed, including GCS emulator and Temporal workflow tests.
  • poe lint passed.
  • Throwaway tests against real GCS passed: a full workflow/activity round trip, 6 MiB and 50 MiB payloads, integrity checks, and concurrent uploads. Confirmed single-request uploads and cleaned up all live test objects.
  • Local checks passed with Python 3.10 and gcloud-aio-storage 9.6.4, the minimum supported version.
  • A throwaway Java test confirmed that the Java GCS driver can read Python-generated references and payloads.

Emulator tests require STORAGE_EMULATOR_HOST. The real GCS and Java test harnesses are not committed.

Related issue: #1502

@diwu-sf
diwu-sf requested a review from a team as a code owner October 16, 2026 22:30
@CLAassistant

CLAassistant commented Oct 16, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants