Repository navigation
[Backport release-1.6] feat(monitoring): alert when a namespace stays in Terminating - #5041
Open
europrinter (yankawai) wants to merge 2 commits into
Conversation
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
area/monitoring
A namespace that never finished deleting raised no alert of its own. A tenant namespace deleted directly, not through its Tenant, stayed in Terminating for about 28 hours while the finalizer of a Kamaji TenantControlPlane inside it waited for a DataStore secret after the tenant etcd was gone. The only signal was HelmReleaseNotReady of the tenant release, which kept flapping. kube-state-metrics exports kube_namespace_status_phase with a Terminating series per namespace that is 1 while it is being deleted. The rule takes the maximum per cluster and namespace, so the timer survives kube-state-metrics moving to another node, which changes its node and pod target labels. Healthy deletions finish in under a minute, and in about five for a tenant with a Kubernetes cluster, a VM and Postgres inside, so 30 minutes leaves wide room and still reports a stuck namespace within the hour. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> (cherry picked from commit 79627dc)
hack/test_helper.bash does not exist on release-1.6, and that branch runs hack/*.bats through hack/cozytest.sh, which has no load builtin. With the line kept from main the suite exits 127 before vmalert-tool runs. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
europrinter (yankawai)
force-pushed
the
backport/namespace-stuck-terminating-alert-1.6
branch
from
October 21, 2026 21:32
8c4e212 to
b78ea9d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Backport of #5040 to release-1.6.
On release-1.6 kube-state-metrics 2.15.0 with its
namespacescollector, the kube-state-metrics scrape, the alerts template, the VMAlert and vm-operator v0.68.4 are the same as on main, and no alert rule there readskube_namespace_status_phaseeither. The rule and its test data apply without changes. A second commit drops theload test_helperline from the Bats suite, because release-1.6 has nohack/test_helper.bashand runshack/*.batsthroughhack/cozytest.sh, where that line stops the suite with exit 127.On this branch
hack/cozytest.sh hack/namespace-deletion-alerts.batspasses, 5 cases with 11 checks. With the expression replaced byvector(1) == 0it fails the 4 checks that expect an alert, and each rule change listed in #5040 fails as many checks as on main. I ran every non-e2ehack/*.batsthroughhack/cozytest.sh, the waymake bats-unit-testsruns them here, with Helm 3.17.3 onPATH. 42 of 47 files pass,alert-severity-contractamong them with the new alert aswarning. The other 5 (cilium-ipam-migration,nightly-mirror_test,overlay-main-images_test,promote-retag_test,promote-rewrite-tags_test) fail the same way on release-1.6 itself in my local environment, macOS with bash 3.2 and BSD sed and awk. helm-unittest 1.1.2 passes the monitoring-agents chart with 2 of 2 tests, andhelm templaterenders the VMRule the same as the file. I have not run the rule in vmalert on a live cluster.Screenshots
Not a UI change.
Downstream repositories
The same three files as #5040, a new alert file, a Bats suite and its test data. No trigger map entry applies.
Release note