Skip to content

fix: back off repeat stale alerts for endpoints held on a captain decision - #293

Merged
dnth merged 3 commits into
mainfrom
fm/fm-held-alert-backoff
Nov 9, 2026
Merged

dnth merged 3 commits into
mainfrom
fm/fm-held-alert-backoff

Conversation

@dnth

@dnth dnth commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

stale loop (unchanged hash, not busy)
  └─ wedge timer at FM_STALE_ESCALATE_SECS
       ├─ linked captain call open?
       │    ├─ no  -> unchanged escalation ladder (deep-inspection at threshold)
       │    └─ yes -> held_alert_tick
       │              ├─ first escalation  -> remind now
       │              ├─ inside interval   -> fold + count + triage log
       │              └─ interval elapsed  -> remind with "N repeats folded"
       │                                      then double the gap up to the cap
       └─ release tick each poll: call answered/released/re-held -> drop ledger, escalate now
  • bin/fm-captain-hold.sh adds a durable captain call -> endpoint link: link <task-id> --endpoint <endpoint-id>, hold ... --endpoint, and a read-only endpoint-open <endpoint-id> [--identity] predicate. Links live in state/captain-hold-endpoints/<endpoint-id>.links and are removed when the call is answered, released, or closed.
  • bin/fm-watch.sh backs repeat stale escalations off while that call is open. The first escalation of an episode still fires at once; the next reminder waits one base interval, and each later gap doubles up to the cap. Suppressed repeats are counted in the persisted state/.held-alert-<window-key> ledger and logged, and the next reminder reports the folded count.
  • Defaults: FM_HELD_ALERT_BACKOFF_SECS=3600, FM_HELD_ALERT_BACKOFF_MAX_SECS=21600, FM_HELD_ALERT_RELEASE_CHECK_SECS=60. An endpoint with no linked open decision keeps the unchanged ladder and demand-deep-inspection wording.
  • Contract: docs/configuration.md "Linked captain-decision backoff"; record mechanics: bin/fm-captain-hold.sh header.

Evidence

Before: the same stale remote endpoint without a link keeps the old ladder, one wake per threshold:

$ env -u FM_TASK_ID FM_WATCH_TRIAGE_TEST=test_held_alert_backoff_ignores_unlinked_endpoint bash tests/fm-watch-triage.test.sh
ok - an endpoint with no linked open decision keeps the unchanged escalation ladder and deep-inspection wording

The live report that motivated this change was stale: remote:boatqa (idle ..., possible wedge, escalation 23, demand-deep-inspection: ...), fired about every four minutes for roughly twenty hours while the decision sat open.

After: with a linked open decision the first wake names it, a watcher restart folds repeats inside the interval, and the next reminder reports the folded count:

$ env -u FM_TASK_ID FM_WATCH_TRIAGE_TEST=test_held_alert_backoff_folds_repeats_across_restart bash tests/fm-watch-triage.test.sh
ok - an open linked decision backs off repeat stale alerts across a watcher restart and reports folded repeats

test_held_alert_backoff_schedule_starts_at_base_interval pins the base-then-doubling schedule, test_held_alert_backoff_release_restores_immediate_alerting covers answering the decision, test_held_alert_backoff_clears_on_real_change covers a recovered endpoint and a status change, and test_held_alert_backoff_new_episode_is_not_suppressed covers a released and re-held call. Final CI: 18 of 18 checks green.

Merge Danger

Door: Two-way. The link records and ledgers are additive state; reverting restores the previous watcher behavior with no migration.

Blast Radius: Supervision noise only. An endpoint with no linked open decision keeps the existing escalation ladder, reason, and deep-inspection wording; nothing else reads the new state.

Testing

I ran the real watcher and captain-hold CLI against disposable fixture homes, with a fake remote beacon probe reporting a stale secondmate. I ran the 6 new held-alert watcher tests and the existing backlog captain-hold deferral test. I ran the full captain-hold lifecycle suite, including the new endpoint-link test. I also drove an adversarial corrupted link record and captured a live transcript of the backoff and folded-repeat reminders. Everything passed. I did not run the full repository suite (CI owns that), and I did not run bin/fm-lint.sh, because this step forbids linters.

  • Live validation: ✅ go - 9 of 9 scenarios driven live against the product
Scenario Result Live Evidence
AC1: stale unchanged remote endpoint with linked open captain hold emits one reminder per backoff interval, folding cycle repeats ✅ pass live held-subset-run.log (test_held_alert_backoff_folds_repeats_across_restart), held-adversarial-run.log transcript
Backoff schedule starts at the base interval, then doubles up to the cap ✅ pass live held-subset-run.log (test_held_alert_backoff_schedule_starts_at_base_interval); transcript ledger interval 10 then 20
AC4: backoff survives a watcher restart and the next reminder reports the folded count ✅ pass live held-subset-run.log; transcript shows '2 repeats folded since the last reminder' after restart
AC2: answering the linked decision restores immediate ordinary escalation on the next cycle ✅ pass live held-subset-run.log (test_held_alert_backoff_release_restores_immediate_alerting)
AC2: endpoint reachable again clears the ledger, and a status change surfaces its signal wake despite the open decision ✅ pass live held-subset-run.log (test_held_alert_backoff_clears_on_real_change)
New episode (released and re-held call) is not suppressed by the old backoff ✅ pass live held-subset-run.log (test_held_alert_backoff_new_episode_is_not_suppressed)
AC3: endpoint held but unlinked keeps the escalation ladder and demand-deep-inspection ✅ pass live held-subset-run.log (test_held_alert_backoff_ignores_unlinked_endpoint, test_wedge_backlog_captain_hold_defers_escalation)
Adversarial: corrupted link record never suppresses the alert ✅ pass live held-adversarial-run.log
Link CLI contract: link refuses non-calls, hold --endpoint records the link, endpoint-open reports identity, answer drops the link ✅ pass live captain-hold-lifecycle.log (test_endpoint_links_track_open_calls)
Evidence: Targeted held-alert watcher test run
ok - an open linked decision backs off repeat stale alerts across a watcher restart and reports folded repeats
ok - answering the linked decision restores the ordinary escalation on the next cycle
ok - a recovered endpoint clears the backoff and a status change is never suppressed
ok - an endpoint with no linked open decision keeps the unchanged escalation ladder and deep-inspection wording
ok - held-alert reminders start one base interval apart and double up to the cap
ok - a released and re-held linked call is a new episode that is never suppressed by the old backoff
ok - active backlog captain hold defers wedge escalation by default
bash $E/held-subset-runner.sh  54.01s user 27.06s system 106% cpu 1:15.98 total
Evidence: Captain-hold lifecycle suite run
ok - legacy history remains visible when reheld and identical answers use the new lifecycle mode
ok - reconciliation retries only match records from the current lifecycle
ok - current linked resolution stays suppressed and retries without duplicate records
ok - due rejects arguments and validates the test-only clock
ok - report-only unresolved decision is reproduced and completion refuses before loss
ok - non-forced scout teardown always requires durable inventory verification
ok - captain holds are idempotent, distinct, teardown-safe, Bearings-visible, and durably routed before close
ok - completion and verification validate origins before constructing paths
ok - ended visual review follows the same decision-hold completion owner
ok - resolved findings and decision-like prose do not create false holds
ok - terminal single-owner stale status decisions do not block empty inventory
actionable: task sample-mate-review-decision-release is held for the captain in this home but that did not reach the parent channel (rc=3)
ok - main-home and secondmate-home captain holds remain correctly routed
ok - resolve matches first/middle/last in quoted blocked_by and rejects a genuinely absent id
ok - a bound channel's captured answers close their captain holds at answer time
ok - a channel source with no decision binding closes nothing
ok - the answer path keeps every guard the unrouted close path already had
ok - the chat channel feeds the same keyed-answer intake a captured review does
ok - Deck-style keyed answers close plain task-id holds with provenance and steer the owning task
ok - reconcile requests stay durable, never close, and retire through evidence or a note
ok - the adapter splits reconcile selections from keyed answers
ok - a status close over an open hold surfaces through diverged
ok - a hold records its 14-day review-by date while staying live and visible
ok - endpoint links exist only for an open linked call and drop when it is answered
ok - a past-due call is marked due in OPEN DECISIONS while the hold and the spawn refusal survive
ok - whole-body due marking and identities agree with independent exact reads across lifecycle histories
ok - due advisory freshness rechecks deferral, blockers, and review date with lifecycle identity
ok - a --until deferral gates until its date and resurfaces after
ok - the board build writes fm-bearings-board.v1 plus durable fm-decision-card.v1 records
ok - an out-of-band close stays recordable through answer
ok - a close-mode answer refuses while the worker is live and --release, --close-live, and question closes behave
ok - an answered self-held call verifies after Done retention archives it, even under a fresh reused row
ok - the archive settles an answered call and never an unanswered or lost one
bash tests/fm-captain-hold-lifecycle.test.sh  278.90s user 75.34s system 107% cpu 5:31.00 total
Evidence: Adversarial corrupt-link + live backoff transcript

watcher run 1: stale: remote:boatqa (idle 500s - captain decision boatqa-retire-decision is open, awaiting the captain, 0 repeats folded since the last reminder, ...) triage: absorbed stale (captain decision boatqa-retire-decision open, 1 repeats folded, next reminder in 6s) triage: absorbed stale (... 2 repeats folded, next reminder in 2s) watcher run 2: stale: remote:boatqa (... 2 repeats folded since the last reminder ...) corrupt link: endpoint-open rc=2 -> stale: remote:boatqa (idle 502s, possible wedge, escalation 1)

--- endpoint-open before corruption:
boatqa-retire-decision	2026-11-09T02:20:53Z#0
rc=0
--- endpoint-open after corruption:
fm-captain-hold: cannot read the endpoint link record for boatqa
rc=2
--- watcher wake with corrupt link record:
stale: remote:boatqa (idle 502s, possible wedge, escalation 1)
ok - an unreadable link record never suppresses: ordinary escalation fires
--- link record:
schema=fm-captain-hold-endpoint.v1
task=boatqa-retire-decision
--- watcher run 1 wake at 10:21:01:
stale: remote:boatqa (idle 500s - captain decision boatqa-retire-decision is open, awaiting the captain, 0 repeats folded since the last reminder, rechecked on a backed-off cadence not a wedge; captain to answer or release the decision)
--- ledger:
boatqa-retire-decision	2026-11-09T02:20:57Z#0	0	1794190861	10
--- watcher run 2 wake at 10:21:14:
stale: remote:boatqa (idle 2s - captain decision boatqa-retire-decision is open, awaiting the captain, 2 repeats folded since the last reminder, rechecked on a backed-off cadence not a wedge; captain to answer or release the decision)
--- ledger:
boatqa-retire-decision	2026-11-09T02:20:57Z#0	0	1794190873	20
--- triage log (held lines):
[2026-11-09T10:21:06+0800] absorbed stale (captain decision boatqa-retire-decision open, 1 repeats folded, next reminder in 6s): remote:boatqa
[2026-11-09T10:21:10+0800] absorbed stale (captain decision boatqa-retire-decision open, 2 repeats folded, next reminder in 2s): remote:boatqa
ok - live transcript captured
Evidence: Subset runner script
#!/usr/bin/env bash
# tests/fm-watch-triage.test.sh - the always-on wake triage built into
# bin/fm-watch.sh and the shared classifier (bin/fm-classify-lib.sh). The watcher
# now absorbs the benign majority of wakes in bash and exits ONLY on an actionable
# wake, so firstmate's LLM re-arms once per actionable event instead of once per
# wake. These tests cover the classifier predicates as pure functions, then drive
# a real fm-watch.sh subprocess to assert the behavioral contract:
# provably-working no-verb wakes absorbed (no exit, no queue entry, suppressor
# advanced, beacon fresh), stopped-crew no-verb wakes surfaced (queue + exit),
# provably-working stale panes absorbed-then-escalated past the threshold,
# terminal-looking stale status lines overridden by an active run, the heartbeat
# backstop fail-safe, and afk coherence (no double-triage while the away-mode
# daemon owns supervision).
#
# Daemon-side classification/injection lives in fm-daemon.test.sh; watcher/lock
# liveness in fm-watcher-lock.test.sh; the durable-queue safety matrix in
# fm-wake-queue.test.sh.
set -u

# shellcheck source=tests/wake-helpers.sh
. {run-worktree}/tests/wake-helpers.sh
# shellcheck source=/dev/null
. "$ROOT/bin/fm-classify-lib.sh"

WATCH="${FM_WATCH_BIN:-$ROOT/bin/fm-watch.sh}"
DRAIN="$ROOT/bin/fm-wake-drain.sh"

TMP_ROOT=$(fm_test_tmproot fm-watch-triage-tests)

ack_stopped_cycle() {  # <state>
  local state=$1 err sequence generation
  err="$state/.test-cycle-drain.err"
  FM_STATE_OVERRIDE="$state" "$DRAIN" >/dev/null 2> "$err" || return 1
  sequence=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation [A-Za-z0-9._-][A-Za-z0-9._-]*$/\1/p' "$err")
  generation=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through [0-9][0-9]* --recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$err")
  rm -f "$err"
  [ -n "$sequence" ] && [ -n "$generation" ] || return 1
  FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through "$sequence" \
    --recovery-generation "$generation"
}

# Common watcher knobs: tight poll/grace, no check or heartbeat cadence unless a
# test overrides them, so a test only exercises the path it targets. FM_CREW_STATE_BIN
# points at the case's hermetic fake fm-crew-state.sh (installed by make_case) so the
# absorb-only-when-provably-working triage reads a canned verdict; a test fixes that
# verdict via FM_FAKE_CREW_STATE in its environment before calling watch_bg.
# Echo batching is default-on in production, so it is disabled here unless a case
# opts in through its config file: these cases assert which wake class a
# signal belongs to, and the batching suite owns the window's own behavior.
watch_bg() {  # <state> <fakebin> <out> [extra env assignments...]
  local state=$1 fakebin=$2 out=$3
  shift 3
  # env(1) parses the extra assignments after expansion; a bare "$@" word list
  # would make them the command name instead.
  env PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \
    FM_CONFIG_OVERRIDE="${state%/state}/config" \
    FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$@" "$WATCH" > "$out" &
}

# Wait up to <limit> 0.1s ticks while <pid> stays alive; 0 if still alive, 1 if it died.
wait_live() {
  local pid=$1 limit=${2:-30} i=0
  while [ "$i" -lt "$limit" ]; do
    kill -0 "$pid" 2>/dev/null || return 1
    sleep 0.1
    i=$((i + 1))
  done
  return 0
}

# Wait until the watcher has completed a whole poll cycle, or exited first.
# The beacon is touched at the top of each poll, so observing two distinct
# mtimes proves the cycle between them completed instead of passing vacuously
# while a loaded machine is still doing bounded watcher startup work.
wait_poll_cycle() {  # <state> <pid> [limit-ticks]
  local state=$1 pid=$2 limit=${3:-300} beat first now i=0
  beat="$state/.last-watcher-beat"
  rm -f "$beat"
  first=""
  while [ "$i" -lt "$limit" ]; do
    kill -0 "$pid" 2>/dev/null || return 1
    first=$(file_mtime "$beat")
    [ -n "$first" ] && break
    sleep 0.1
    i=$((i + 1))
  done
  while [ "$i" -lt "$limit" ]; do
    kill -0 "$pid" 2>/dev/null || return 1
    now=$(file_mtime "$beat")
    if [ -n "$now" ] && [ "$now" != "$first" ]; then
      return 0
    fi
    sleep 0.1
    i=$((i + 1))
  done
  return 1
}

wait_numeric_file() {
  local file=$1 limit=${2:-30} i=0 value
  while [ "$i" -lt "$limit" ]; do
    value=$(cat "$file" 2>/dev/null || true)
    case "$value" in
      ''|*[!0-9]*) ;;
      *) return 0 ;;
    esac
    sleep 0.1
    i=$((i + 1))
  done
  return 1
}

# Portable mtime in epoch seconds. Platform-detected, never the `stat -f || stat -c`
# fallback (which writes a partial filesystem dump on Linux; see fm-watch.sh).
file_mtime() {
  if [ "$(uname)" = Darwin ]; then stat -f %m "$1" 2>/dev/null; else stat -c %Y "$1" 2>/dev/null; fi
}

# Set <file>'s mtime to exactly <epoch> seconds, for aging a busy-turn marker by
# a precise amount (touch -t takes a local-time stamp, not an epoch, on both
# platforms, so convert via BSD `date -r` or GNU `date -d @`).
set_mtime() {  # <epoch> <file>
  local epoch=$1 f=$2 stamp
  if stamp=$(date -r "$epoch" +%Y%m%d%H%M.%S 2>/dev/null); then
    touch -t "$stamp" "$f"
  else
    stamp=$(date -d "@$epoch" +%Y%m%d%H%M.%S)
    touch -t "$stamp" "$f"
  fi
}

# Signature a primed .seen-* marker must hold so the per-poll signal scan does not
# fire on a pre-existing status (mirrors fm-watch.sh's stat_sig exactly).
seen_sig() {
  if [ "$(uname)" = Darwin ]; then stat -f '%i:%z:%Fm' "$1" 2>/dev/null; else stat -c '%i:%s:%.9Y' "$1" 2>/dev/null; fi
}

# Prime <file>'s .seen-* suppressor to its CURRENT signature, so the per-poll
# no-verb signal scan (which watches every *.turn-ended for an inode:size:mtime change)
# treats a just-created or just-backdated turn-ended marker as already seen.
# Busy-turn-age fixtures create/backdate turn-ended directly (there is no real
# harness touching it), so without this the marker's own first sighting would
# fire an unrelated "signal:" wake and mask the busy-turn-age assertion under
# test. Call again after any further touch/set_mtime on the same file.
prime_turnend_seen() {  # <file>
  local f=$1 base
  base=$(basename "$f" | tr '.' '_')
  printf '%s' "$(seen_sig "$f")" > "$(dirname "$f")/.seen-$base"
}

record_pi_busy() {  # <state-dir> <id>
  local state=$1 id=$2 gen
  gen=$("$ROOT/bin/fm-busy-event.sh" arm "$state" "$id")
  "$ROOT/bin/fm-busy-event.sh" apply "$state" "$id" busy --gen "$gen" \
    --source pi-ext --event agent-start
}

# --- echo-batching and churn fixtures ---------------------------------------
# Point the watcher at a case-local config dir and opt it into pane-churn absorb.
churn_config() {  # <case-dir>
  local cfg="$1/config"
  mkdir -p "$cfg"
  : > "$cfg/turnend-churn-absorb"
  printf '%s\n' "$cfg"
}

# Age the persisted echo batch's opening epoch so its window is due without a
# test sleeping for the production 60-300s bound. Only the first line's epoch
# moves; later entries keep their own times.
backdate_echo_batch() {  # <state-dir> <seconds-overdue>
  local state=$1 overdue=$2 file="$1/.echo-batch" stamp tmp
  stamp=$(( $(date +%s) - overdue ))
  tmp="$file.tmp.$$"
  awk -F '\t' -v s="$stamp" 'BEGIN { OFS = "\t" } { if (NR == 1) $1 = s; print }' \
    "$file" > "$tmp" && mv -f "$tmp" "$file"
}

wait_echo_batch_entries() {  # <state-dir> <count> [limit-ticks]
  local state=$1 want=$2 limit=${3:-100} i=0 got
  while [ "$i" -lt "$limit" ]; do
    got=0
    [ -f "$state/.echo-batch" ] && got=$(grep -c . "$state/.echo-batch" 2>/dev/null || true)
    [ "${got:-0}" -ge "$want" ] && return 0
    sleep 0.1
    i=$((i + 1))
  done
  return 1
}

# A task whose panes stay busy (so the staleness backbone does not pre-empt the
# batch), whose meta/marker pair is a live incarnation, and whose fake crew state
# is supplied by the caller.
make_echo_task() {  # <case-dir> <id> <gen>
  local dir=$1 id=$2 gen=$3
  local window="test:fm-$id"
  printf 'window=%s\nkind=ship\nharness=pi\nspawn_gen=%s\n' "$window" "$gen" > "$dir/state/$id.meta"
  : > "$dir/state/$id.turn-ended.$g

... [310426 bytes truncated] ...

s zero-alias tuned-window absent-default; do
    dir=$(make_case "echo-batch-config-$variant"); state="$dir/state"; fakebin="$dir/fakebin"
    out="$dir/watch.out"; mkdir -p "$dir/config"
    case "$variant" in
      invalid-default) window=30 ;;
      leading-zero) window=060 ;;
      octal-invalid) window=090 ;;
      oversized) window=99999999999999999999999999999999999999 ;;
      false-alias) window=false ;;
      uppercase-alias) window=OFF ;;
      no-alias) window=no ;;
      zero-alias) window=0 ;;
      *)               window=240 ;;
    esac
    printf '%s\n' "$window" > "$dir/config/wake-echo-batch"
    export FM_FAKE_CREW_STATE='state: parked · source: pane · waiting on the supervisor'
    case "$variant" in
      config-off)
        printf 'off\n' > "$dir/config/wake-echo-batch"
        ;;
    esac
    [ "$variant" != absent-default ] || rm -f "$dir/config/wake-echo-batch"
    watch_bg "$state" "$fakebin" "$out"
    pid=$!
    make_echo_task "$dir" echocfg g1
    case "$variant" in
      config-off)
        wait_for_exit "$pid" 40 || { reap "$pid"; fail "$variant: a disabled window still deferred: $(cat "$out")"; }
        [ ! -e "$state/.echo-batch" ] || fail "$variant: a disabled window wrote a batch"
        ;;
      absent-default|invalid-default|leading-zero|octal-invalid|oversized|false-alias|uppercase-alias|no-alias|zero-alias)
        wait_echo_batch_entries "$state" 1 || { reap "$pid"; fail "$variant: an invalid window did not fall back to the default"; }
        if [ "$variant" != absent-default ]; then
          wait_log_line "$state" 'echo-batch config invalid' \
            || { reap "$pid"; fail "$variant: the invalid value was not reported"; }
        fi
        wait_log_line "$state" 'window=120s' \
          || { reap "$pid"; fail "$variant: the fallback window was not the documented default: $(cat "$state/.watch-triage.log" 2>/dev/null)"; }
        reap "$pid"
        ;;
      *)
        wait_echo_batch_entries "$state" 1 || { reap "$pid"; fail "$variant: the configured window did not defer"; }
        wait_log_line "$state" 'window=240s' \
          || { reap "$pid"; fail "$variant: the configured window was not used"; }
        reap "$pid"
        ;;
    esac
    unset FM_FAKE_CREW_STATE
  done
  pass "config/wake-echo-batch selects 60-300 seconds or off, an absent file uses 120 seconds, and an invalid value falls back with one log line"
}

test_echo_batch_procevent_flush_does_not_deadlock() {
  local dir state out pid
  dir=$(make_case echo-batch-procevent-lock); state="$dir/state"; out="$dir/watch.out"
  printf '120\n' > "$dir/config/wake-echo-batch"
  export FM_FAKE_CREW_STATE='state: parked · source: pane · waiting on the supervisor'
  procevent_watch_bg "$dir" "$out"
  pid=$!
  make_echo_task "$dir" echope g1
  wait_echo_batch_entries "$state" 1 || { reap "$pid"; fail "the echo was not deferred before the process event"; }
  append_wake "$state" check procevent:batch-lock:1 'check: procevent fixture batch-lock 1'
  wait_for_exit "$pid" 100 || { reap "$pid"; fail "a pending batch deadlocked process-event delivery"; }
  grep -F 'check: process-event result captured: procevent:batch-lock:1' "$out" >/dev/null \
    || fail "the immediate output did not name the process event"
  grep "$(printf '\tsignal\t')" "$state/.wake-queue" | grep -F "$state/echope.turn-ended.g1" >/dev/null \
    || fail "the process-event wake lost the pending echo"
  [ ! -e "$state/.echo-batch" ] || fail "the process-event wake did not retire the pending batch"
  [ ! -d "$state/.wake-queue.lock" ] || fail "the process-event output retained the queue lock"
  unset FM_FAKE_CREW_STATE
  pass "a process-event wake flushes a pending echo batch without reacquiring its queue lock"
}

test_turnend_markers_preserve_full_task_identity() {
  local form rule dir state out capture marker pid key mapped id=alpha.turn-ended.beta
  local line='state: parked · source: pane · waiting on the supervisor'
  for form in bare generation; do
    for rule in churn baseline answered-gate no-change; do
      dir=$(make_case "marker-identity-$form-$rule"); state="$dir/state"; out="$dir/watch.out"; capture="$dir/pane.txt"
      printf 'window=test:fm-alpha\nkind=ship\nharness=pi\nspawn_gen=g1\n' > "$state/alpha.meta"
      printf 'window=test:fm-subject\nkind=ship\nharness=pi\n' > "$state/$id.meta"
      marker="$state/$id.turn-ended"
      if [ "$form" = generation ]; then
        printf 'spawn_gen=g1\n' >> "$state/$id.meta"
        marker="$marker.g1"
      fi
      : > "$marker"
      record_pi_busy "$state" "$id" >/dev/null
      blocking_capture "$capture"
      export FM_FAKE_CREW_STATE='state: unknown · source: none · stopped task'
      export FM_FAKE_CREW_STATE_alpha="$line"
      case "$rule" in
        churn)
          FM_FAKE_CREW_STATE_alpha='state: working · source: run-step · validating'
          : > "$dir/config/turnend-churn-absorb"
          key=test_fm-subject
          printf '%s' "$(hash_text "$(cat "$capture")")" > "$state/.hash-$key"
          ;;
        answered-gate)
          FM_FAKE_CREW_STATE_alpha='state: parked · source: run-step · parked at review: 1 finding(s) · run: 01RUN'
          printf '%s\n' 'needs-decision [key=nm-01RUN-review]: question' 'resolved [key=nm-01RUN-review]: answered' > "$state/alpha.status"
          prime_turnend_seen "$state/alpha.status"
          printf '120\n' > "$dir/config/wake-echo-batch"
          ;;
        no-change)
          printf '%s\tg1\t%s\n' "$(( $(date +%s) - 100 ))" "$line" > "$state/.last-delivered-alpha"
          printf '120\n' > "$dir/config/wake-echo-batch"
          ;;
      esac
      mapped=$(FM_STATE_OVERRIDE="$state" bash -c '. "$1"; fm_wake_status_key_map "$2" || exit; printf "%s" "$FM_WAKE_STATUS_KEY"' _ "$ROOT/bin/fm-wake-lib.sh" "${marker##*/}")
      [ "$mapped" = "$id.status" ] || fail "$form mapped the marker to another task's status"
      watch_bg "$state" "$dir/fakebin" "$out" FM_FAKE_TMUX_WINDOW=test:fm-subject FM_FAKE_TMUX_CAPTURE="$capture"
      pid=$!
      case "$rule" in
        answered-gate|no-change)
          wait_echo_batch_entries "$state" 1 || { reap "$pid"; fail "$form/$rule did not defer the marker"; }
          backdate_echo_batch "$state" 121
          ;;
      esac
      wait_for_exit "$pid" 100 || { reap "$pid"; fail "$form/$rule suppressed the stopped task's wake"; }
      grep -F "$marker" "$out" >/dev/null || fail "$form/$rule lost the marker in delivery"
      [ -f "$state/.last-delivered-$id" ] || fail "$form/$rule did not record the full task's baseline"
      if [ "$rule" != no-change ]; then
        [ ! -e "$state/.last-delivered-alpha" ] || fail "$form/$rule recorded the baseline under the truncated task id"
      fi
      unset FM_FAKE_CREW_STATE FM_FAKE_CREW_STATE_alpha
    done
  done
  pass "bare and generation-qualified markers preserve task IDs through classification, drops, baselines, and queue mapping"
}

test_echo_review_regressions() {
  test_turnend_markers_preserve_full_task_identity
  test_echo_batch_procevent_flush_does_not_deadlock
  test_procevent_surface_serializes_with_drain
  test_procevent_surface_crash_boundaries
  test_procevent_marker_failure_exits_and_replays
  test_turn_ended_churning_pane_absorbed
  test_status_and_turn_end_batch_never_uses_churn_evidence
  test_echo_batch_groups_turnend_echoes_into_one_wake
  test_echo_batch_survives_a_watcher_restart
  test_echo_batch_answered_gate_drop_and_near_misses
  test_echo_batch_no_change_drop_and_near_misses
  test_echo_batch_config_window_and_disable
}

case "${FM_WATCH_TRIAGE_TEST:-}" in
  '')
    ;;
  *)
    if declare -F "$FM_WATCH_TRIAGE_TEST" >/dev/null 2>&1; then
      "$FM_WATCH_TRIAGE_TEST"
      exit
    fi
    fail "unknown FM_WATCH_TRIAGE_TEST: $FM_WATCH_TRIAGE_TEST"
    ;;
esac

test_held_alert_backoff_folds_repeats_across_restart
test_held_alert_backoff_release_restores_immediate_alerting
test_held_alert_backoff_clears_on_real_change
test_held_alert_backoff_ignores_unlinked_endpoint
test_held_alert_backoff_schedule_starts_at_base_interval
test_held_alert_backoff_new_episode_is_not_suppressed
test_wedge_backlog_captain_hold_defers_escalation
  • Evidence: Adversarial runner script (local file: ~/.no-mistakes/evidence/01M6Z0088DBDD0G0VVKK3C36B2/held-adversarial-runner.sh)

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 4 issues found → auto-fixed ✅
  • ⚠️ bin/fm-watch.sh:524 - FM_HELD_ALERT_BACKOFF_SECS never becomes a real gap between reminders. On the first reminder of an episode the ledger is absent, so interval=base (3600). The reminder fires, then lines 524-526 double it before writing the ledger. The ledger stores 7200, and that is the window the fold check at line 509 uses. With defaults the schedule is: reminder at t0, then t0+2h, then t0+6h (4h gap), then 6h gaps after that. The 1h base gap never happens. The header comment (fm-watch.sh:383-385 'later repeats double this interval') and docs/configuration.md:177 both describe 3600 as the starting interval, and AC1 says 'at most one reminder per backoff interval'. Tests can't catch this because they set base==cap or cap=2*base. Fix: store the interval currently in effect and double it only when a reminder fires after a previous one (last != 0). Or keep the code and reword the docs to say the first gap is twice the base.
  • ℹ️ docs/configuration.md:177 - The docs say both backoff values 'must be positive integers of at most nine digits... invalid values fall back to their defaults'. fm-watch.sh:388-390 only rejects empty, leading-zero, and non-digit values. A 10+ digit value is accepted, and a large enough one overflows interval * 2 at line 524. Either add the length check, e.g. ?????????[0-9]* in the fallback case for both variables, or drop the nine-digit claim.
  • ℹ️ bin/fm-watch.sh:1664 - After held_alert_release_tick the code re-reads since_file and repeats the whole non-numeric timer-reset block (lines 1664-1672 copy 1655-1658). The release tick only ever writes a numeric value, so this second branch can't be reached from a valid since. Simpler: run the release tick before the first since=$(cat ...) read and keep one validation case.
  • ⚠️ bin/fm-captain-hold.sh:2617 - Simplification: the new unlink &lt;task-id&gt; [--endpoint] command (command_unlink, plus its usage/header/docs lines) is not required by the intent. The intent needs an explicit durable link that is dropped when the decision is answered or released. close_answered → release_endpoint_links already does that, and endpoint-open already treats links to closed calls as not-held. Note also that unlinking a still-open call does not lift an active backoff until the next stale threshold, because held_alert_release_tick checks only whether the call is open, not whether the link still exists. Recommend removing unlink, or confirming it is wanted as a manual correction tool.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 9 of 9 scenarios driven live against the product
Scenario Result Live Evidence
AC1: stale unchanged remote endpoint with linked open captain hold emits one reminder per backoff interval, folding cycle repeats ✅ pass live held-subset-run.log (test_held_alert_backoff_folds_repeats_across_restart), held-adversarial-run.log transcript
Backoff schedule starts at the base interval, then doubles up to the cap ✅ pass live held-subset-run.log (test_held_alert_backoff_schedule_starts_at_base_interval); transcript ledger interval 10 then 20
AC4: backoff survives a watcher restart and the next reminder reports the folded count ✅ pass live held-subset-run.log; transcript shows '2 repeats folded since the last reminder' after restart
AC2: answering the linked decision restores immediate ordinary escalation on the next cycle ✅ pass live held-subset-run.log (test_held_alert_backoff_release_restores_immediate_alerting)
AC2: endpoint reachable again clears the ledger, and a status change surfaces its signal wake despite the open decision ✅ pass live held-subset-run.log (test_held_alert_backoff_clears_on_real_change)
New episode (released and re-held call) is not suppressed by the old backoff ✅ pass live held-subset-run.log (test_held_alert_backoff_new_episode_is_not_suppressed)
AC3: endpoint held but unlinked keeps the escalation ladder and demand-deep-inspection ✅ pass live held-subset-run.log (test_held_alert_backoff_ignores_unlinked_endpoint, test_wedge_backlog_captain_hold_defers_escalation)
Adversarial: corrupted link record never suppresses the alert ✅ pass live held-adversarial-run.log
Link CLI contract: link refuses non-calls, hold --endpoint records the link, endpoint-open reports identity, answer drops the link ✅ pass live captain-hold-lifecycle.log (test_endpoint_links_track_open_calls)
  • bash held-subset-runner.sh (the fm-watch-triage.test.sh preamble plus 6 test_held_alert_backoff_* cases and test_wedge_backlog_captain_hold_defers_escalation): all 7 ok, exit 0
  • bash tests/fm-captain-hold-lifecycle.test.sh: 32 ok, exit 0, including test_endpoint_links_track_open_calls
  • bash held-adversarial-runner.sh: corrupted link-record schema leads to endpoint-open rc=2 and the watcher emits the ordinary 'possible wedge, escalation 1' with no ledger
  • Live transcript: two real watcher runs against a linked hold (base 10s, cap 40s) show the first reminder with 0 repeats folded, two absorbed repeats in the triage log, and a second reminder with '2 repeats folded'. The ledger interval goes 10 then 20
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

dnth added 3 commits November 9, 2026 09:16
A permanently unreachable endpoint whose decision was already with the
captain kept re-alerting every watcher cycle, because the decision sat on a
different task id and nothing about an unreachable endpoint changes.

Link a captain call to the endpoint it covers and, while that call is open,
back repeat wedge escalations off instead of firing once per cycle. The first
escalation of an episode still fires; later reminders double up to a cap,
each suppressed repeat is counted and logged, and the next reminder reports
the folded count. A real change - reachable again, pane change, answer,
release, or re-hold - drops the ledger and restores the unchanged escalation
path. Unlinked endpoints keep today's ladder and deep-inspection wording.

Contract: docs/configuration.md "Linked captain-decision backoff".
@dnth
dnth merged commit 0efe362 into main Nov 9, 2026
34 of 36 checks passed
@dnth
dnth deleted the fm/fm-held-alert-backoff branch November 9, 2026 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant