Skip to content

fix(pool): fail requests queued behind a failed protocol probe together - #6241

Open
Taiwudao wants to merge 1 commit into
nodejs:mainfrom
Taiwudao:fix/pool-fail-queued-behind-failed-probe
Open

Taiwudao wants to merge 1 commit into
nodejs:mainfrom
Taiwudao:fix/pool-fail-queued-behind-failed-probe

Conversation

@Taiwudao

Copy link
Copy Markdown

This relates to...

Fixes #6238

Rationale

This is a regression introduced in 8.11.0 by #5799 ("avoid pool fan-out during ALPN
negotiation"). Requests to an HTTPS origin are held in the pool queue behind a single
"protocol probe" client until ALPN reveals whether the origin speaks h1 or h2.

If that probe's connection attempt fails, the pool retires it and hands the next queued
request to a new probe client. Each new client starts its connectTimeout when it is
created, so the time already spent waiting in the queue is not counted anywhere. The
result is that a backlog of N requests against an origin that accepts TCP but never
answers the TLS handshake rejects one connect timeout after another, the n-th only after
roughly n connect timeouts.

Measured with connectTimeout = 2000ms and four concurrent fetch() calls through one
Agent, against a local net server that accepts the connection and never answers the TLS
handshake:

before (undici 8.13.1):

2.5s, 5.1s, 7.6s, 10.1s

after this patch:

about 2.6s for each of the four requests

undici 8.5.0 rejected all four together, so this restores the pre-#5799 behavior for the
failed-probe case.

Changes

Bug Fixes

  • lib/dispatcher/pool-base.js: add [kFailQueue](err), which settles every request still
    waiting in the pool queue with one already-observed error. It reuses kShiftQueue, so
    aborted tombstones are skipped exactly as in kDrainQueue, and it mirrors how
    kDestroy/kAbortQueued settle queued work. It only settles the backlog that existed when
    it was called, so a handler that synchronously dispatches a new request does not
    inherit the error.

  • lib/dispatcher/pool.js: the connectionError handler now receives the connection error.
    When the failed target is the protocol probe, the backlog is settled with that error
    instead of being re-probed one request at a time.

The new path is deliberately narrow. It only runs when the failed client is the protocol
probe. While the probe gate is up, kGetDispatcher() returns no dispatcher for any other
request, so the queue can only be waiting on that probe. Pools that already established a
protocol, and connection errors on non-probe clients, keep the existing drain-and-retry
behavior.

Breaking Changes and Deprecations

N/A

Status

  • I have read and agreed to the Developer's Certificate of Origin
  • Tested
  • Benchmarked (optional)
  • Documented
  • Review ready

Notes for reviewers

test/issue-6238.js starts a net server that accepts the TCP connection and never answers
the TLS handshake, then asserts that four concurrent fetch() calls through one Agent all
reject with UND_ERR_CONNECT_TIMEOUT within 2 * connectTimeout.

The alternative shape is to count time spent queued behind the probe towards each
request's connect budget, for example by stamping the enqueue time on QueuedRequest and
checking a deadline in kDrainQueue. That keeps the retry, but it is a larger change and it
needs a decision on whether queue wait should consume a request's connect budget at all.
I went with settling the backlog because every queued request would otherwise retry the
same origin that the probe has just failed to reach, and because the observable result
matches the expectation in the issue. Happy to switch to the per-request deadline shape
if you prefer it.

One trade-off worth stating explicitly: with this patch a transient connect failure fails
the queued batch instead of retrying it on a fresh connection. Requests that arrive after
the probe failed still create a new probe and are retried normally. undici has no default
request-level retry (RetryHandler is a separate, opt-in interceptor), and the current
serial behavior turns an O(1) failure into O(n) connect timeouts, so failing together
seems like the better default here. If you would rather keep the retry, the per-request
deadline shape above is the way to go.

Signed-off-by: Taiwudao <359400496+Taiwudao@users.noreply.github.com>

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is the right default: it gives the pool backlog the same behavior a single Client already has when it fails to connect (onError fails every pending request with the same error).

A couple of things before landing:

  1. Please follow the client's exceptions. onError skips UND_ERR_SOCKET and UND_ERR_INFO, and on ERR_TLS_CERT_ALTNAME_INVALID it fails only the requests for the mismatched servername. kFailQueue currently fails the whole backlog on any error; I'd only fail it in the cases where the client itself would, and keep draining otherwise.
  2. After kFailQueue we return without calling kDrainQueue. If an error handler re-dispatches and that request gets queued (e.g. connection limit reached), nothing picks it up. A if (this[kQueued] > 0) this[kDrainQueue](...) before returning would cover it.
  3. Could you add tests for a non-probe connectionError still retrying, and for a request dispatched from inside the error handler not inheriting the error?

Otherwise LGTM.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent requests to an origin whose TLS handshake stalls time out one after another since 8.11.0

2 participants