Repository navigation
fix: back off repeat stale alerts for endpoints held on a captain decision - #293
Merged
Merged
Conversation
A permanently unreachable endpoint whose decision was already with the captain kept re-alerting every watcher cycle, because the decision sat on a different task id and nothing about an unreachable endpoint changes. Link a captain call to the endpoint it covers and, while that call is open, back repeat wedge escalations off instead of firing once per cycle. The first escalation of an episode still fires; later reminders double up to a cap, each suppressed repeat is counted and logged, and the next reminder reports the folded count. A real change - reachable again, pane change, answer, release, or re-hold - drops the ledger and restores the unchanged escalation path. Unlinked endpoints keep today's ladder and deep-inspection wording. Contract: docs/configuration.md "Linked captain-decision backoff".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bin/fm-captain-hold.shadds a durable captain call -> endpoint link:link <task-id> --endpoint <endpoint-id>,hold ... --endpoint, and a read-onlyendpoint-open <endpoint-id> [--identity]predicate. Links live instate/captain-hold-endpoints/<endpoint-id>.linksand are removed when the call is answered, released, or closed.bin/fm-watch.shbacks repeat stale escalations off while that call is open. The first escalation of an episode still fires at once; the next reminder waits one base interval, and each later gap doubles up to the cap. Suppressed repeats are counted in the persistedstate/.held-alert-<window-key>ledger and logged, and the next reminder reports the folded count.FM_HELD_ALERT_BACKOFF_SECS=3600,FM_HELD_ALERT_BACKOFF_MAX_SECS=21600,FM_HELD_ALERT_RELEASE_CHECK_SECS=60. An endpoint with no linked open decision keeps the unchanged ladder anddemand-deep-inspectionwording.docs/configuration.md"Linked captain-decision backoff"; record mechanics:bin/fm-captain-hold.shheader.Evidence
Before: the same stale remote endpoint without a link keeps the old ladder, one wake per threshold:
The live report that motivated this change was
stale: remote:boatqa (idle ..., possible wedge, escalation 23, demand-deep-inspection: ...), fired about every four minutes for roughly twenty hours while the decision sat open.After: with a linked open decision the first wake names it, a watcher restart folds repeats inside the interval, and the next reminder reports the folded count:
test_held_alert_backoff_schedule_starts_at_base_intervalpins the base-then-doubling schedule,test_held_alert_backoff_release_restores_immediate_alertingcovers answering the decision,test_held_alert_backoff_clears_on_real_changecovers a recovered endpoint and a status change, andtest_held_alert_backoff_new_episode_is_not_suppressedcovers a released and re-held call. Final CI: 18 of 18 checks green.Merge Danger
Door: Two-way. The link records and ledgers are additive state; reverting restores the previous watcher behavior with no migration.
Blast Radius: Supervision noise only. An endpoint with no linked open decision keeps the existing escalation ladder, reason, and deep-inspection wording; nothing else reads the new state.
Testing
I ran the real watcher and captain-hold CLI against disposable fixture homes, with a fake remote beacon probe reporting a stale secondmate. I ran the 6 new held-alert watcher tests and the existing backlog captain-hold deferral test. I ran the full captain-hold lifecycle suite, including the new endpoint-link test. I also drove an adversarial corrupted link record and captured a live transcript of the backoff and folded-repeat reminders. Everything passed. I did not run the full repository suite (CI owns that), and I did not run bin/fm-lint.sh, because this step forbids linters.
Evidence: Targeted held-alert watcher test run
Evidence: Captain-hold lifecycle suite run
Evidence: Adversarial corrupt-link + live backoff transcript
watcher run 1: stale: remote:boatqa (idle 500s - captain decision boatqa-retire-decision is open, awaiting the captain, 0 repeats folded since the last reminder, ...) triage: absorbed stale (captain decision boatqa-retire-decision open, 1 repeats folded, next reminder in 6s) triage: absorbed stale (... 2 repeats folded, next reminder in 2s) watcher run 2: stale: remote:boatqa (... 2 repeats folded since the last reminder ...) corrupt link: endpoint-open rc=2 -> stale: remote:boatqa (idle 502s, possible wedge, escalation 1)Evidence: Subset runner script
~/.no-mistakes/evidence/01M6Z0088DBDD0G0VVKK3C36B2/held-adversarial-runner.sh)Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 4 issues found → auto-fixed ✅
bin/fm-watch.sh:524-FM_HELD_ALERT_BACKOFF_SECSnever becomes a real gap between reminders. On the first reminder of an episode the ledger is absent, so interval=base (3600). The reminder fires, then lines 524-526 double it before writing the ledger. The ledger stores 7200, and that is the window the fold check at line 509 uses. With defaults the schedule is: reminder at t0, then t0+2h, then t0+6h (4h gap), then 6h gaps after that. The 1h base gap never happens. The header comment (fm-watch.sh:383-385 'later repeats double this interval') and docs/configuration.md:177 both describe 3600 as the starting interval, and AC1 says 'at most one reminder per backoff interval'. Tests can't catch this because they set base==cap or cap=2*base. Fix: store the interval currently in effect and double it only when a reminder fires after a previous one (last != 0). Or keep the code and reword the docs to say the first gap is twice the base.docs/configuration.md:177- The docs say both backoff values 'must be positive integers of at most nine digits... invalid values fall back to their defaults'. fm-watch.sh:388-390 only rejects empty, leading-zero, and non-digit values. A 10+ digit value is accepted, and a large enough one overflowsinterval * 2at line 524. Either add the length check, e.g.?????????[0-9]*in the fallback case for both variables, or drop the nine-digit claim.bin/fm-watch.sh:1664- Afterheld_alert_release_tickthe code re-reads since_file and repeats the whole non-numeric timer-reset block (lines 1664-1672 copy 1655-1658). The release tick only ever writes a numeric value, so this second branch can't be reached from a valid since. Simpler: run the release tick before the firstsince=$(cat ...)read and keep one validation case.bin/fm-captain-hold.sh:2617- Simplification: the newunlink <task-id> [--endpoint]command (command_unlink, plus its usage/header/docs lines) is not required by the intent. The intent needs an explicit durable link that is dropped when the decision is answered or released.close_answered→release_endpoint_linksalready does that, andendpoint-openalready treats links to closed calls as not-held. Note also that unlinking a still-open call does not lift an active backoff until the next stale threshold, becauseheld_alert_release_tickchecks only whether the call is open, not whether the link still exists. Recommend removingunlink, or confirming it is wanted as a manual correction tool.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
bash held-subset-runner.sh(the fm-watch-triage.test.sh preamble plus 6 test_held_alert_backoff_* cases and test_wedge_backlog_captain_hold_defers_escalation): all 7 ok, exit 0bash tests/fm-captain-hold-lifecycle.test.sh: 32 ok, exit 0, including test_endpoint_links_track_open_callsbash held-adversarial-runner.sh: corrupted link-record schema leads to endpoint-open rc=2 and the watcher emits the ordinary 'possible wedge, escalation 1' with no ledgerLive transcript: two real watcher runs against a linked hold (base 10s, cap 40s) show the first reminder with 0 repeats folded, two absorbed repeats in the triage log, and a second reminder with '2 repeats folded'. The ledger interval goes 10 then 20✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.