Skip to content

Size Postgres pools from a fleet connection budget (#223 C11-3) - #249

Merged
maximusunc merged 4 commits into
develfrom
audit/223-pool-budget
Nov 11, 2026
Merged

maximusunc merged 4 commits into
develfrom
audit/223-pool-budget

Conversation

@maximusunc

@maximusunc maximusunc commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Part 3 of 4 for #223 (C11-3, the "pool budget" option; PgBouncer is not included and is now #251).

Problem

Every process held a hand-set pool max, and the fleet's ceiling was the sum of those maxes: about 320 of compose's 400 connections at one replica each. Scaling a deployment's replicas multiplies that sum and runs Postgres out of connections. The saturation alert used a raw count (250) that had to be recounted by hand whenever a worker, pool size or replica count changed.

Change

Budget-derived pool sizes (shepherd_utils/config.py, db.pool_sizes):

Setting Meaning
POSTGRES_CONNECTION_BUDGET Connections the whole fleet may hold. 0 (default) keeps the explicit POSTGRES_POOL_MAX_SIZE, so nothing changes unless it is set.
POSTGRES_FLEET_SHARES Sum of every process's shares, replicas and server processes included
POSTGRES_POOL_SHARES This process's weight (default 1)
  • A process's max is budget × shares // fleet_shares, at least 1, and its min is capped at that max.
  • When a deployment scales, it recomputes FLEET_SHARES and every pool shrinks to keep the fleet inside the budget.
  • A budget without fleet shares, or a share larger than the fleet, is refused at import.
  • Startup check: each process compares the budget with the server's max_connections − superuser_reserved_connections and logs an error if it doesn't fit.
  • The per-process lookup submission bound (MAX_LOOKUPS_IN_FLIGHT, in shepherd_utils/lookups.py since One shared lookup submission; plain lookups fail loudly (#223 C6-3, C9-5) #248) follows the derived pool size.

compose.yml:

  • A new x-postgres-budget anchor (budget 360, 35 shares) is merged into all 28 DB-connecting services.
  • The two lookup workers take 3 shares.
  • The server's 4 uvicorn processes take 1 share each.
  • That reproduces today's sizes exactly: 10 per share, 30 for each lookup worker and 10 per server process. It leaves 37 of the 397 usable connections free.
  • docker compose config validates the file.

Monitor:

  • The per-tick pg_stat_activity query now also returns client backends and the usable max_connections, in one round trip.
  • The new pg_connection_pct metric drives pg_connection_saturation, which fires at 80% of what Postgres accepts. The idle floor is about 20%.
  • The old pg_connection_count metric still works.

Also in this PR: CI installs Python 3.12

.github/workflows/dev.yml installed Python 3.10, but tox's py312 and coverage envs need 3.12. Tox had been using the 3.12 that came with the runner image. That disappeared between runs on 2026-11-10, after which every PR's Run tox failed before any test ran (see the comment below). It's a one-line change: python-version: "3.12".

Deployments (Helm)

Set POSTGRES_CONNECTION_BUDGET for the cluster's Postgres, and set POSTGRES_FLEET_SHARES to Σ(replicas × processes × shares). Give the lookup workers and server their POSTGRES_POOL_SHARES. Until that's done, deployments keep their explicit sizes, but the new alert works either way.

Tests

New tests/unit/test_connection_budget.py:

  • Sizing: explicit sizes without a budget; shares; scaling halves pools; min capped; minimum of 1; refusals.
  • Startup check: it logs an error when the budget is over the limit and stays quiet when it fits.
  • compose.yml: every DB service is on the budget with no stray max sizes (the build-only shepherd_base service from Build every image on one shared base image (#223 C11-5) #250 is skipped); FLEET_SHARES equals the computed sum; the budget fits max_connections.
  • Alert: it fires as a share of max_connections, stays quiet when the limit is unknown, and the shipped rule uses the share.

The monitor poller fixture is updated for the new columns. I also ran both new SQL statements against Postgres 16.

Validation


Generated by Claude Code

Every process held a hand-set pool max, and the fleet's ceiling was
their sum: about 320 of compose's 400 connections at one replica each.
Scaling a deployment's replicas ran Postgres out of connections, and
the saturation alert's raw threshold (250) had to be recounted by hand
whenever anything changed.

- POSTGRES_CONNECTION_BUDGET, POSTGRES_FLEET_SHARES and
  POSTGRES_POOL_SHARES set a process's pool max to
  budget x shares // fleet_shares (at least 1). The min is capped at
  that max. Unset, the explicit sizes apply as before. A budget without
  fleet shares, or a share larger than the fleet, is refused.
- Each process checks the budget against the server's max_connections
  minus superuser_reserved_connections at startup, and logs an error
  when it doesn't fit.
- The lookup submission bounds follow the derived pool size.
- compose.yml puts all 28 DB-connecting services on one anchor
  (budget 360, 35 shares). That reproduces today's sizes: 10 a share,
  30 for each lookup worker, and 10 per server process.
  test_connection_budget checks the share total against the services.
- The monitor reads client backends and the usable max_connections in
  the same query. pg_connection_saturation now fires at 80% of what
  Postgres accepts, via the new pg_connection_pct metric.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.30435% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.02%. Comparing base (1e08532) to head (7ed9acb).
⚠️ Report is 3 commits behind head on devel.

Files with missing lines Patch % Lines
shepherd_utils/db.py 90.00% 3 Missing ⚠️
workers/monitor/alerts.py 88.88% 0 Missing and 1 partial ⚠️
Files with missing lines Coverage Δ
shepherd_utils/config.py 100.00% <100.00%> (ø)
shepherd_utils/lookups.py 100.00% <100.00%> (ø)
workers/monitor/poller.py 53.92% <100.00%> (+0.25%) ⬆️
workers/monitor/alerts.py 69.36% <88.88%> (+0.75%) ⬆️
shepherd_utils/db.py 73.55% <90.00%> (+0.62%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f53952...7ed9acb. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The shared base image (#223 C11-5) is a compose service that is built,
never run, and opens no pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
#248 moved MAX_LOOKUPS_IN_FLIGHT out of aragorn_lookup and
aragorn_pathfinder into shepherd_utils/lookups.py. This takes devel's
workers and derives the bound there from POOL_MAX_SIZE, as this branch
did in the workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm

Copy link
Copy Markdown
Collaborator Author

Run tox failure on 8aa4cf6 is in CI setup, not this PR.

Tox stopped before any test ran:

py312: failed with could not find python interpreter matching any of the specs py312
coverage: failed with could not find python interpreter matching any of the specs python3.12

tox.ini runs py312 and coverage (basepython=python3.12), but .github/workflows/dev.yml installs only Python 3.10 via actions/setup-python. Tox had been relying on the runner image's own system 3.12. That interpreter was there for run #333 at 22:19 UTC and gone for this run at 00:11 UTC, so every PR will hit this until the workflow installs 3.12 itself. The same branch passed CI at fd5d013 before the devel merge.

I couldn't re-run the job: the API returns 403 for this integration.

Proposed fix, kept out of this PR because it's unrelated to the pool budget:

       - name: Set up Python
         uses: actions/setup-python@v5
         with:
-          python-version: "3.10"
+          python-version: "3.12"

Generated by Claude Code

tox.ini's py312 and coverage envs need a Python 3.12 interpreter, but
dev.yml installs only 3.10. tox had been using the 3.12 that came with
the runner image, which disappeared between runs on 2026-11-10, so
every PR's "Run tox" job failed before any test ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
@maximusunc
maximusunc merged commit 257a841 into devel Nov 11, 2026
2 checks passed
@maximusunc
maximusunc deleted the audit/223-pool-budget branch November 11, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants