Skip to content

feat: default healthcheck port and readiness probe - #163

Open
cherylaurelia wants to merge 1 commit into
pgdogdev:mainfrom
cherylaurelia:fix/default-readiness-probe
Open

cherylaurelia wants to merge 1 commit into
pgdogdev:mainfrom
cherylaurelia:fix/default-readiness-probe

Conversation

@cherylaurelia

@cherylaurelia cherylaurelia commented Nov 9, 2026 •

Copy link
Copy Markdown

Closes #121, using the approach suggested there. Without a readiness probe, a bad database setting plus kubectl rollout restart replaces every pod with one whose pools are all down. On kind this took 2 s, and every query then failed with connection pool ... is down. With this change and healthcheckPort set, the first new pod fails readiness, the rollout stops, and the old pods keep serving.

Who changes on upgrade: only installs that set healthcheckPort and not readinessProbe. They get a healthcheck container port and an HTTP readiness probe on it. healthcheckPort stays unset by default, so default renders are unchanged, and a user-set readinessProbe is kept. I left the chart version for you to bump.

It's readiness only, since a liveness probe would restart PgDog during a database outage. If Postgres is fully down, every pod goes NotReady, so clients connecting through the Service get connection refused instead of pool is down. The 10 s initial delay is a heuristic: pools report healthy until their first connect fails, which takes up to connect_timeout (5 s). A slower failure would still pass, and each pod takes about 10 s longer to become ready. The cleaner fix is in pgdog itself: pools start healthy (TargetHealth::new()), and starting them unknown until the first check would let this delay go.

Testing
  • ./test/test.sh passes. test/values-healthcheck-port.yaml renders the probe through the kubeconform loop; an invalid probe field fails it.
  • On kind, with healthcheckPort set: a host typo (3 runs), an unreachable IP, and one of two databases down. The rollout stopped every time while healthy pods kept serving. After a 45 s Postgres outage, pods were Ready again about 20 s after it came back.
  • Fork CI can't use the Blacksmith runners, so I ran the same commands locally. configcheck passes except for pgdog-config-extras, which fails the same way on main.

Written with Claude Code, which also ran the tests and the kind runs above.

🤖 Generated with Claude Code

Comment thread values.yaml Outdated
# healthcheckPort on which PgDog will expose its HTTP healthcheck endpoint.
# It returns 502 when every connection pool is down. Unless readinessProbe is
# set, the chart uses it as the readiness probe. Set to null to disable.
healthcheckPort: 8080

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't be enabling this by default.

Comment thread test/test.sh
exit 1
fi

echo ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove this and add a yaml test into the test folder instead? We automatically validate config generation there using kubeconform

The chart rendered no readiness probe, so Kubernetes treated a PgDog pod
whose pools are all down as ready, even with healthcheckPort set.

When healthcheckPort is set, declare it as a container port and, unless
the user has set readinessProbe, add an HTTP readiness probe on it. The
endpoint only fails when every pool is down, so one unreachable database
keeps the pod ready. Pools report healthy until their first connection
attempt fails (up to connect_timeout), hence the 10s initial delay.
healthcheckPort stays unset by default, so default renders are unchanged,
and a user-set readinessProbe is unchanged. No liveness probe: restarting
PgDog during a database outage would drop every client.

test/values-healthcheck-port.yaml renders the probe through the
kubeconform loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cherylaurelia
cherylaurelia force-pushed the fix/default-readiness-probe branch from 3eb07fc to 1a598ac Compare November 9, 2026 20:12
@cherylaurelia

Copy link
Copy Markdown
Author

thanks! made healthcheckPort opt-in (commented out by default, so no probe unless it's set) and moved the test to test/values-healthcheck-port.yaml so kubeconform picks it up

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Suggestion: Set the healthcheck and probes by default

2 participants