Skip to content

Propagate request reason to Cloud SQL requests - #3283

Open
CydeWeys wants to merge 1 commit into
google:masterfrom
CydeWeys:cloud-sql-request-reason
Open

CydeWeys wants to merge 1 commit into
google:masterfrom
CydeWeys:cloud-sql-request-reason

Conversation

@CydeWeys

@CydeWeys CydeWeys commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Purpose

Environments with GCP access-justification / request-reason enforcement reject human calls to the Cloud SQL Admin API (GetConnectSettings / GenerateEphemeralCert) with 403 Forbidden (NOT_AUTHORIZED) unless requests include a valid X-Goog-Request-Reason HTTP header, even when the caller holds the required IAM permissions.

The Cloud SQL JDBC Socket Factory does not read the standard CLOUDSDK_CORE_REQUEST_REASON environment variable on its own, causing nomulus CLI commands that connect to Cloud SQL to fail under these restrictions. This change propagates CLOUDSDK_CORE_REQUEST_REASON into the X-Goog-Request-Reason header on outgoing Cloud SQL Admin API requests so justified nomulus CLI commands pass validation and succeed.

Usage

Run nomulus with CLOUDSDK_CORE_REQUEST_REASON set in the environment to a request-reason value accepted by your organization's policy (for example, exported by an access-justification session wrapper or set by a shell wrapper around nomulus):

CLOUDSDK_CORE_REQUEST_REASON="<encoded-request-reason>" \
  nomulus -e production list_cursors --type=RECURRING_BILLING

Note that environments enforcing structured justification tokens expect an encoded request-reason payload (typically minted by a wrapper CLI) rather than raw free-form text. When CLOUDSDK_CORE_REQUEST_REASON is unset or whitespace-only, no X-Goog-Request-Reason header is attached and existing behavior is unchanged.

Summary of Changes

  • Store credentials and the normalized request reason in an immutable SupplierConfig record published via a single @Nullable private static volatile field in CloudSqlCredentialSupplier, eliminating visibility/torn-read/TOCTOU issues.
  • Validate credentials and fail-fast validate non-empty request reasons as printable US-ASCII (after trimming Unicode whitespace via CharMatcher.whitespace()) before publishing state or setting CredentialFactory.CREDENTIAL_FACTORY_PROPERTY.
  • Unify uninitialized-state handling in both create() and getCredentials() via checkState(...) (IllegalStateException), and override getCredentials() to return GoogleCredentials directly.
  • Make CloudSqlCredentialSupplier public final with an explicit documented public no-arg constructor for reflective instantiation, narrow internal constants to private static final, and use a named static nested RequestReasonCredentialsAdapter subclass of HttpCredentialsAdapter.
  • Add 33 unit test cases in CloudSqlCredentialSupplierTest (with resetForTesting() static state cleanup) covering header propagation, Unicode whitespace trimming, blank/null omission and state resets, 401-retry double-initialize() idempotency, reflective instantiation, uninitialized calls, invalid header characters, and non-mutation on invalid inputs.

Verification

  • ./gradlew :core:standardTest :core:registryToolIntegrationTest :core:checkstyleMain :core:checkstyleTest spotlessCheck javaIncrementalFormatCheck runPresubmits -x :console-webapp:buildConsoleWebapp passed (33/33 CloudSqlCredentialSupplierTest cases passing).
  • Tested end-to-end by running a read-only nomulus CLI SQL command (list_cursors) against sandbox: failed with 403 Forbidden without CLOUDSDK_CORE_REQUEST_REASON, and succeeded when run with a valid encoded CLOUDSDK_CORE_REQUEST_REASON set.

BUG= http://b/587970232


This change is Reviewable

@CydeWeys
CydeWeys requested a review from njshah301 November 19, 2026 22:51
@CydeWeys
CydeWeys force-pushed the cloud-sql-request-reason branch 2 times, most recently from a80b4ca to 3458ada Compare November 19, 2026 23:12

@CydeWeys CydeWeys left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

The core propagation mechanism is sound:

  • SDK wiring: jdbc-socket-factory-core's Connector invokes CredentialFactory.create() to build the SQLAdmin client, so subclassing HttpCredentialsAdapter attaches X-Goog-Request-Reason to outgoing Cloud SQL Admin API requests.
  • 401 retry idempotency: HttpCredentialsAdapter.handleResponse refreshes credentials and calls this.initialize(request) virtually; HttpHeaders.set(...) overwrites the header cleanly rather than appending duplicates.
  • OAuth token isolation: UserCredentials / ServiceAccountCredentials token refreshes use separate HttpRequestFactory instances, so X-Goog-Request-Reason stays scoped to Cloud SQL Admin API calls and is never sent to token endpoints.
  • Blast radius: CloudSqlCredentialSupplier.setupCredentialSupplier is only called from the CLI tool wiring (PersistenceModule.provideNomulusToolJpaTm), leaving server workloads unaffected; when CLOUDSDK_CORE_REQUEST_REASON is unset or blank, create() returns a plain HttpCredentialsAdapter.

Before merging, there are 4 Must-Fix items and 3 Should-Fix / polish items (detailed inline):

Must-Fix

  1. Validation/publication order (CloudSqlCredentialSupplier.java): System.setProperty(...) executes before checkNotNull(credential) and state updates, mutating the global JVM property on invalid input and exposing an uninitialized window.
  2. Thread safety & TOCTOU (CloudSqlCredentialSupplier.java): Non-volatile split static fields (credential, requestReason) and a double-read of requestReason in create() allow visibility issues, torn reads, and a TOCTOU NoSuchElementException; combine into a single @Nullable private static volatile immutable record read once per call.
  3. Asymmetric uninitialized errors (CloudSqlCredentialSupplier.java): Uninitialized create() throws a message-less NPE inside HttpCredentialsAdapter, while getCredentials() throws NPE with a message; unify both via a getConfig() helper using Preconditions.checkState (IllegalStateException).
  4. Test static state leak (CloudSqlCredentialSupplierTest.java): afterEach() restores the system property but leaves static credential/requestReason populated across tests sharing a Gradle worker JVM (forkEvery = 1 is disabled in core/build.gradle).

Should-Fix

  1. Fail-fast header character validation: Switch String::trim to String::strip and reject non-printable-ASCII header characters up front instead of relying on HttpURLConnection's partial checks.
  2. Class/constant hygiene: Make the class public final with an explicit documented public no-arg constructor for reflection, narrow the constants to private static final, use a static nested HttpCredentialsAdapter subclass, and clean up minor Javadoc/predicate nits.
  3. Unit test coverage: Add tests for uninitialized calls, set->unset/blank resets, null-credential non-mutation of the system property, double-initialize (401 retry idempotency + header preservation + handler registration), literal header assertion, reflective instantiation, and invalid header characters.

Comment on lines +35 to +38
public class CloudSqlCredentialSupplier implements CredentialFactory {

static final String REQUEST_REASON_ENV_VAR = "CLOUDSDK_CORE_REQUEST_REASON";
static final String REQUEST_REASON_HEADER = "X-Goog-Request-Reason";

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should-Fix (Class, constant visibility & reflective constructor):

  • Make CloudSqlCredentialSupplier public final class (compatible with CredentialFactoryProvider's reflective instantiation).
  • Add an explicit public CloudSqlCredentialSupplier() {} constructor with Javadoc noting it is required for CredentialFactory reflective loading, preventing accidental future refactoring to a private constructor.
  • Narrow REQUEST_REASON_ENV_VAR (unused outside this class) and REQUEST_REASON_HEADER (once tests assert the literal "X-Goog-Request-Reason") to private static final.
  • Minor Javadoc nit on line 43: Initialize -> Initializes (third-person fragment).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: made CloudSqlCredentialSupplier public final, added a documented public CloudSqlCredentialSupplier() {} constructor for reflective instantiation, narrowed both constants to private static final, and changed Initialize -> Initializes.

Comment on lines +40 to +41
private static GoogleCredentials credential;
private static Optional<String> requestReason = Optional.empty();

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must-Fix (Thread safety, @Nullable, and TOCTOU double-read):

jdbc-socket-factory-core runs background refresh tasks on a ListeningScheduledExecutorService alongside HikariCP threads. Keeping credential and requestReason in two plain non-volatile static fields has no happens-before guarantee, permits torn reads (new credential paired with old requestReason), omits @Nullable on credential, and creates a TOCTOU NoSuchElementException window in create() between requestReason.isEmpty() (line 60) and requestReason.get() (line 63).

Consider combining both fields into an immutable private record published via a single volatile reference and read once into a local variable per call:

private record SupplierConfig(GoogleCredentials credential, Optional<String> requestReason) {}

@Nullable private static volatile SupplierConfig config;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: combined state into private record SupplierConfig(GoogleCredentials credential, Optional<String> requestReason) {} held in @Nullable private static volatile SupplierConfig config; and read once per call via getConfig().

Comment on lines +51 to +53
System.setProperty(
CredentialFactory.CREDENTIAL_FACTORY_PROPERTY, CloudSqlCredentialSupplier.class.getName());
CloudSqlCredentialSupplier.credential = credential;
CloudSqlCredentialSupplier.credential = checkNotNull(credential, "credential");

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must-Fix (Side effect before validation and state publication):

System.setProperty(...) runs before checkNotNull(credential, "credential") and before state is assigned. Calling setupCredentialSupplier(null, ...) mutates the global cloudSql.socketFactory.credentialFactory system property before throwing NullPointerException, and even on valid inputs setting the property first exposes an uninitialized/stale state window to concurrent readers.

Validate all inputs first, publish the immutable config snapshot, and call System.setProperty(...) last:

GoogleCredentials validatedCredential = checkNotNull(credential, "credential");
Optional<String> normalizedReason = normalizeRequestReason(requestReason);
config = new SupplierConfig(validatedCredential, normalizedReason);
System.setProperty(
    CredentialFactory.CREDENTIAL_FACTORY_PROPERTY, CloudSqlCredentialSupplier.class.getName());

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: setupCredentialSupplier now validates credential and normalizes/validates requestReason first, publishes config next, and calls System.setProperty(...) last.

Comment on lines +54 to +55
CloudSqlCredentialSupplier.requestReason =
Optional.ofNullable(requestReason).map(String::trim).filter(reason -> !reason.isEmpty());

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should-Fix (Fail-fast HTTP header validation & Unicode whitespace):

OpenJDK's HttpURLConnection.checkMessageHeader only rejects \n when not followed by space/tab (producing a late, wrapped IllegalArgumentException inside HikariCP/JDBC connect). Bare \r, obs-fold (\r\n ), \0, C0 control characters (0x00-0x1F), DEL (0x7F), and non-ASCII characters (including NBSP \u00A0, which String::trim does not remove) bypass checkMessageHeader and are serialized raw onto the HTTP/1.1 wire, violating RFC 9110 §5.5 and causing HTTP frontends to reject the request.

Switch String::trim to String::strip, use Predicate.not(String::isEmpty), and validate printable US-ASCII fail-fast (without echoing raw control characters in the exception message):

private static final CharMatcher PRINTABLE_ASCII = CharMatcher.inRange(' ', '~');

private static Optional<String> normalizeRequestReason(@Nullable String requestReason) {
  Optional<String> normalized =
      Optional.ofNullable(requestReason).map(String::strip).filter(not(String::isEmpty));
  normalized.ifPresent(
      reason ->
          checkArgument(
              PRINTABLE_ASCII.matchesAllOf(reason),
              "%s must contain only printable ASCII characters",
              REQUEST_REASON_ENV_VAR));
  return normalized;
}

(If internal horizontal tabs should be allowed per RFC 9110 §5.5, .or(CharMatcher.is('\t')) can be added to the matcher.)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: added fail-fast CharMatcher.inRange(' ', '~') printable-ASCII validation (via checkArgument without echoing raw control characters) and not(String::isEmpty). Used Guava's CharMatcher.whitespace()::trimFrom rather than String::strip (with an inline comment) because Character.isWhitespace('\u00A0') is false in Java, so String::strip would reject "\u00a0" instead of omitting the header as requested.

Comment on lines +64 to +70
return new HttpCredentialsAdapter(credential) {
@Override
public void initialize(HttpRequest request) throws IOException {
super.initialize(request);
request.getHeaders().set(REQUEST_REASON_HEADER, reason);
}
};

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should-Fix (Static nested adapter class):

Prefer a named private static final class RequestReasonCredentialsAdapter extends HttpCredentialsAdapter over an anonymous inner class for cleaner stack traces and explicit static scoping (avoiding an implicit enclosing-instance reference).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: replaced the anonymous inner class with private static final class RequestReasonCredentialsAdapter extends HttpCredentialsAdapter.

Comment on lines +73 to 76
@Override
public GoogleCredentials getCredentials() {
return checkNotNull(credential, "Credential has not been initialized");
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must-Fix (Asymmetric uninitialized-state error handling between create() and getCredentials()):

Calling create() (lines 59-71) before setupCredentialSupplier(...) passes null into HttpCredentialsAdapter.<init>, which throws a message-less NullPointerException from inside google-auth-library, whereas getCredentials() throws NullPointerException("Credential has not been initialized").

Unify both methods through a getConfig() helper using Preconditions.checkState (IllegalStateException is more appropriate than NullPointerException for uninitialized state):

private static SupplierConfig getConfig() {
  SupplierConfig currentConfig = config;
  checkState(currentConfig != null, "CloudSqlCredentialSupplier has not been initialized");
  return currentConfig;
}

(Keeping the getCredentials() override itself is good — it avoids allocating a throwaway HttpCredentialsAdapter when callers only need GoogleCredentials.)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: unified both create() and getCredentials() through getConfig(), throwing IllegalStateException via Preconditions.checkState when uninitialized.

Comment on lines +52 to +60
@AfterEach
void afterEach() {
if (originalCredentialFactoryProperty == null) {
System.clearProperty(CredentialFactory.CREDENTIAL_FACTORY_PROPERTY);
} else {
System.setProperty(
CredentialFactory.CREDENTIAL_FACTORY_PROPERTY, originalCredentialFactoryProperty);
}
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must-Fix (Static state leak across tests):

core/build.gradle disables forkEvery = 1, so :core:standardTest reuses Gradle worker JVMs across multiple test classes. afterEach() restores CredentialFactory.CREDENTIAL_FACTORY_PROPERTY, but leaves CloudSqlCredentialSupplier's static credential and requestReason populated ("test-access-token" / "ticket/12345 investigate latency"), leaking state between test methods and into subsequent test classes in the same JVM.

Add a package-private @VisibleForTesting static void resetForTesting() (config = null;) on CloudSqlCredentialSupplier and call it here in afterEach() (and optionally beforeEach()).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: added package-private @VisibleForTesting static void resetForTesting() clearing config = null; and invoked it in both @BeforeEach and @AfterEach.

Comment on lines +111 to +118
@Test
void testSetup_nullCredential_throwsNullPointerException() {
assertThrows(
NullPointerException.class,
() ->
CloudSqlCredentialSupplier.setupCredentialSupplier(
null, "ticket/12345 investigate latency"));
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should-Fix (Test coverage gaps):

In testSetup_nullCredential_throwsNullPointerException, please also assert that the thrown message contains "credential" and that System.getProperty(CredentialFactory.CREDENTIAL_FACTORY_PROPERTY) remains unchanged (null) after the failure.

Additional test cases worth adding to cover the edge cases above:

  1. Uninitialized calls: Verify both new CloudSqlCredentialSupplier().create() and .getCredentials() throw IllegalStateException with a clear message before setup.
  2. State-reset transition: Call setupCredentialSupplier(TEST_CREDENTIALS, "ticket/12345") followed by setupCredentialSupplier(TEST_CREDENTIALS, null) (as well as "" and " "), verifying X-Goog-Request-Reason is cleared.
  3. 401 retry idempotency, header preservation & handler registration: Invoke initializer.initialize(request) twice on a request with a pre-existing header, asserting request.getHeaders().getHeaderStringValues("X-Goog-Request-Reason") has exactly one entry, request.getUnsuccessfulResponseHandler() is initializer, and pre-existing headers are preserved.
  4. Literal header oracle: Assert against the literal "X-Goog-Request-Reason" instead of importing CloudSqlCredentialSupplier.REQUEST_REASON_HEADER (line 18), so constant typos are caught and the constant can be private.
  5. Reflective instantiation: Verify Class.forName(System.getProperty(CredentialFactory.CREDENTIAL_FACTORY_PROPERTY)).asSubclass(CredentialFactory.class).getDeclaredConstructor().newInstance() succeeds.
  6. Invalid header characters: Assert setupCredentialSupplier throws IllegalArgumentException without mutating the system property/config on inputs containing \r, \n, \r\n, \0, or non-ASCII characters, and omits the header on Unicode whitespace ("\u00a0").

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f5a95ac: switched assertions to the literal "X-Goog-Request-Reason" and expanded CloudSqlCredentialSupplierTest to 33 passing cases covering uninitialized create()/getCredentials(), reflective instantiation, state-reset transitions (null, "", " ", "\u00a0"), double-initialize() 401-retry idempotency + header preservation + UnsuccessfulResponseHandler registration, null-credential NPE message + non-mutation, and invalid header character IllegalArgumentException + non-mutation.

Environments with GCP access-justification / request-reason enforcement
reject human calls to the Cloud SQL Admin API (GetConnectSettings /
GenerateEphemeralCert) with 403 Forbidden (NOT_AUTHORIZED) unless
requests include a valid X-Goog-Request-Reason HTTP header, even when
the caller holds the required IAM permissions.

The Cloud SQL JDBC Socket Factory does not read the standard
CLOUDSDK_CORE_REQUEST_REASON environment variable on its own, causing
nomulus CLI commands that connect to Cloud SQL to fail under these
restrictions. Propagate CLOUDSDK_CORE_REQUEST_REASON into the
X-Goog-Request-Reason header on outgoing Cloud SQL Admin API requests
so justified nomulus CLI commands pass validation and succeed.

Usage: run nomulus with CLOUDSDK_CORE_REQUEST_REASON set in the
environment to a request-reason value accepted by your organization's
policy (for example, exported by an access-justification session
wrapper or passed by a shell wrapper around nomulus):

  CLOUDSDK_CORE_REQUEST_REASON="<encoded-request-reason>" \
    nomulus -e production list_cursors --type=RECURRING_BILLING

Note that environments enforcing structured justification tokens expect
an encoded request-reason payload (typically minted by a wrapper CLI)
rather than raw free-form text. When CLOUDSDK_CORE_REQUEST_REASON is
unset or whitespace-only, no header is added and existing behavior is
unchanged.

Implementation details in CloudSqlCredentialSupplier:
- Store credentials and normalized request reason in an immutable
  record published via a single volatile reference so concurrent
  readers observe consistent snapshots without TOCTOU races.
- Validate credentials and fail-fast validate non-empty request reasons
  as printable US-ASCII (after trimming Unicode whitespace) before
  publishing state or setting the credential-factory system property.
- Override CredentialFactory.getCredentials() and unify uninitialized
  access checks in create() and getCredentials() via checkState().
- Make the class final with an explicit public no-arg constructor for
  reflective instantiation and use a static nested HttpCredentialsAdapter
  subclass to attach X-Goog-Request-Reason.
- Add comprehensive unit tests in CloudSqlCredentialSupplierTest
  covering header propagation, Unicode whitespace trimming, blank/null
  omission and resets, 401-retry initialize() idempotency, reflective
  instantiation, uninitialized calls, invalid header characters,
  non-mutation on invalid inputs, and static test state cleanup.

Tested end-to-end by running a read-only nomulus CLI SQL command
(list_cursors) against sandbox: failed with 403 Forbidden without
CLOUDSDK_CORE_REQUEST_REASON, and succeeded when run with a valid
encoded CLOUDSDK_CORE_REQUEST_REASON set.

BUG= http://b/587970232
@CydeWeys
CydeWeys force-pushed the cloud-sql-request-reason branch from 3458ada to f5a95ac Compare November 19, 2026 23:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant