SOLR-18480 prevent TLOG+PULL leader election stall by aborting fetcher

[kotman12]

https://issues.apache.org/jira/browse/SOLR-18480

AI Usage

I used Opus 5/5.5 to iterate on a solution for this. There were probably tens if not hundreds of refinements.

Description

During leader election a new leader will try to tear down its index fetcher. If the old leader process has stalled, this tear-down can take the minimum of graceful executor timeout and socket timeout (in our configuration it stalls for 60s). Since there is no point in gracefully shutting down an index fetch request that gets discarded anyway I propose we abort it without waiting. We have observed this issue in the wild and were lucky enough to reproduce it here.

Solution

Equip the HttpJettySolrClient with an eager abort that doesn't wait. I only implemented it for the jetty client as it appears to be the only client with robust request tracking support (and the only client currently relevant to IndexFetcher). I create a thin client for each individual fetch so as to isolate what gets aborted. I also added an opt-in for tracking abortable requests. I see no reason tracking of requests this shouldn't at least be an option. Although it would be nice to see if we can simply always track requests (i.e. if the performance penalty for this is small) but I didn't have time to properly benchmark this. Because the consequence is so far-reaching I have left it as an opt-in.

An alternative solution could be to tune how long we await for executor termination on abort vs graceful close. However, this is not as elegant and precise regarding timing and would only work for things we are waiting on which are scheduled on that executor.

Tests

Mainly in TlogLeaderElectionFrozenLeaderTest

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added a changelog entry for my change