[wmousa]

Main port of #1503 (R26.3). See that PR for the full write-up.

Problem

On 2026-10-19 a customer cluster, freshly upgraded from R26.2-PRE, had two nodes shut down at 15:42:46. It stayed ACTIVE until 15:49:47: seven minutes with reduced redundancy and no status change.

1. The shutdown never refreshes the cluster status. shutdown_storage_node sets the node to OFFLINE itself, so the monitor's set_node_offline never runs for it. That is the only per-node path that calls update_cluster_status, so the status waits for the monitor's main-loop tick.
2. The main-loop tick was blocked by housekeeping. Retention housekeeping ran in that loop. Each sweep read the whole events table and cleared one key per expired event. R26.2-PRE never pruned events, so the first sweeps after the upgrade deleted months of them while the cluster status went unrefreshed.

Change

• shutdown_storage_node: once the node is OFFLINE, calls the new apply_node_loss_status(cluster_id) (monitor module). It applies only the downgrades a node loss can cause: ACTIVE → DEGRADED/SUSPENDED and DEGRADED → SUSPENDED. It uses the same set_cluster_status the monitor makes for those transitions. It deliberately does not run update_cluster_status, the monitor's full pass, because on a SUSPENDED cluster that pass calls cluster_activate, and a CLI shutdown must not start an activation or wait on one. Recovery transitions stay with the monitor. This is best effort: a failure is logged with a traceback and the shutdown still succeeds.
• Housekeeping thread: housekeeping runs on its own daemon thread, started by _start_housekeeping() in main(). The main loop only refreshes the status.
• _delete_old_logs(cluster_id): removes one cluster's expired events with a single clear_range(<cluster>/, <cluster>/<cutoff ms>). Event keys are <cluster_uuid>/<date ms>/<uuid>, so those events form one contiguous key range.

Differences from #1503

• Main-loop error log: main already logs this error with logger.exception, so that hunk keeps main's version.
• test_monitor_thread_supervision.py: those tests count every thread main() creates. They now patch _start_housekeeping instead of _run_periodic_housekeeping, so their per-node thread counts are unchanged.
• New test: the housekeeping thread is a daemon targeting _housekeeping_loop.

Integration CI failure (fixed)

test_cluster_status_is_refreshed_once_offline failed on R26.3 (py3.9) with 0 refreshes recorded.

• Cause: test_secondary_promotion.py::test_escalates_unreachable_primary assigned a MagicMock to sys.modules["simplyblock_core.services.storage_node_monitor"] and never restored it.
• Why 3.9 only: on Python 3.9, mock.patch resolves its target through package attributes, so the test patched the real module while the lazy import in shutdown_storage_node got the leaked mock. On 3.11+, patch resolves through sys.modules, which is why main passed.
• Fix: the leaking test now uses patch.dict(sys.modules, ...), which restores the real module.
• Reproduced locally: with a 3.9-style patch resolver in the same test order, the test fails without the fix and passes with it.

Tests

• ruff: clean.
• mypy: 2 errors, both in spdk_http_proxy_server.py (asyncio.open_unix_connection on Windows). This PR doesn't touch that file.
• Unit tier (Windows): 4062 passed against 4057 on the baseline. The same 35 failures appear on the untouched baseline (spdk_proxy_unit, lblk_persistent_names, …).
• Integration tier: not run locally, because it needs Docker + FDB. I ran test_shutdown_no_suspension.py locally with three extra FDB reads stubbed (get_active_lvol_migration, cancel_pending_node_restart_tasks, cancel_node_tasks): 17/17 pass. CI covers this tier.

🤖 Generated with Claude Code