Repository navigation
Size Postgres pools from a fleet connection budget (#223 C11-3) - #249
Conversation
Every process held a hand-set pool max, and the fleet's ceiling was their sum: about 320 of compose's 400 connections at one replica each. Scaling a deployment's replicas ran Postgres out of connections, and the saturation alert's raw threshold (250) had to be recounted by hand whenever anything changed. - POSTGRES_CONNECTION_BUDGET, POSTGRES_FLEET_SHARES and POSTGRES_POOL_SHARES set a process's pool max to budget x shares // fleet_shares (at least 1). The min is capped at that max. Unset, the explicit sizes apply as before. A budget without fleet shares, or a share larger than the fleet, is refused. - Each process checks the budget against the server's max_connections minus superuser_reserved_connections at startup, and logs an error when it doesn't fit. - The lookup submission bounds follow the derived pool size. - compose.yml puts all 28 DB-connecting services on one anchor (budget 360, 35 shares). That reproduces today's sizes: 10 a share, 30 for each lookup worker, and 10 per server process. test_connection_budget checks the share total against the services. - The monitor reads client backends and the usable max_connections in the same query. pg_connection_saturation now fires at 80% of what Postgres accepts, via the new pg_connection_pct metric. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
Codecov Report❌ Patch coverage is
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
The shared base image (#223 C11-5) is a compose service that is built, never run, and opens no pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
#248 moved MAX_LOOKUPS_IN_FLIGHT out of aragorn_lookup and aragorn_pathfinder into shepherd_utils/lookups.py. This takes devel's workers and derives the bound there from POOL_MAX_SIZE, as this branch did in the workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
|
Tox stopped before any test ran:
I couldn't re-run the job: the API returns 403 for this integration. Proposed fix, kept out of this PR because it's unrelated to the pool budget: - name: Set up Python
uses: actions/setup-python@v5
with:
- python-version: "3.10"
+ python-version: "3.12"Generated by Claude Code |
tox.ini's py312 and coverage envs need a Python 3.12 interpreter, but dev.yml installs only 3.10. tox had been using the 3.12 that came with the runner image, which disappeared between runs on 2026-11-10, so every PR's "Run tox" job failed before any test ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
Part 3 of 4 for #223 (C11-3, the "pool budget" option; PgBouncer is not included and is now #251).
Problem
Every process held a hand-set pool max, and the fleet's ceiling was the sum of those maxes: about 320 of compose's 400 connections at one replica each. Scaling a deployment's replicas multiplies that sum and runs Postgres out of connections. The saturation alert used a raw count (250) that had to be recounted by hand whenever a worker, pool size or replica count changed.
Change
Budget-derived pool sizes (
shepherd_utils/config.py,db.pool_sizes):POSTGRES_CONNECTION_BUDGETPOSTGRES_POOL_MAX_SIZE, so nothing changes unless it is set.POSTGRES_FLEET_SHARESPOSTGRES_POOL_SHARESbudget × shares // fleet_shares, at least 1, and its min is capped at that max.FLEET_SHARESand every pool shrinks to keep the fleet inside the budget.max_connections − superuser_reserved_connectionsand logs an error if it doesn't fit.MAX_LOOKUPS_IN_FLIGHT, inshepherd_utils/lookups.pysince One shared lookup submission; plain lookups fail loudly (#223 C6-3, C9-5) #248) follows the derived pool size.compose.yml:
x-postgres-budgetanchor (budget 360, 35 shares) is merged into all 28 DB-connecting services.docker compose configvalidates the file.Monitor:
pg_stat_activityquery now also returns client backends and the usablemax_connections, in one round trip.pg_connection_pctmetric drivespg_connection_saturation, which fires at 80% of what Postgres accepts. The idle floor is about 20%.pg_connection_countmetric still works.Also in this PR: CI installs Python 3.12
.github/workflows/dev.ymlinstalled Python 3.10, but tox'spy312andcoverageenvs need 3.12. Tox had been using the 3.12 that came with the runner image. That disappeared between runs on 2026-11-10, after which every PR'sRun toxfailed before any test ran (see the comment below). It's a one-line change:python-version: "3.12".Deployments (Helm)
Set
POSTGRES_CONNECTION_BUDGETfor the cluster's Postgres, and setPOSTGRES_FLEET_SHARESto Σ(replicas × processes × shares). Give the lookup workers and server theirPOSTGRES_POOL_SHARES. Until that's done, deployments keep their explicit sizes, but the new alert works either way.Tests
New
tests/unit/test_connection_budget.py:shepherd_baseservice from Build every image on one shared base image (#223 C11-5) #250 is skipped);FLEET_SHARESequals the computed sum; the budget fitsmax_connections.max_connections, stays quiet when the limit is unknown, and the shipped rule uses the share.The monitor poller fixture is updated for the new columns. I also ran both new SQL statements against Postgres 16.
Validation
MAX_LOOKUPS_IN_FLIGHTintoshepherd_utils/lookups.py, which now derives it fromPOOL_MAX_SIZE.test_query_parityfails locally only because of Python 3.13 caret formatting.black --check .is clean.Generated by Claude Code