Skip to content

fix(runtime): keep a core's wake descriptor open while a notifier can still write - #421

Closed
EnRaiha wants to merge 1 commit into
NodeDB-Lab:mainfrom
EnRaiha:fix/core-wake-eventfd-stale-fd
Closed

EnRaiha wants to merge 1 commit into
NodeDB-Lab:mainfrom
EnRaiha:fix/core-wake-eventfd-stale-fd

Conversation

@EnRaiha

@EnRaiha EnRaiha commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Share ownership of a data-plane core's wake descriptor between the core and the Control Plane's notifiers, so a notifier can never write to a descriptor number that was closed and handed to another file.

What changed

Area Change Paths
EventFd / EventFdNotifier The descriptor is held as Arc<OwnedFd>. The notifier is Clone, no longer Copy. The two Drop impls and the unsafe impl Send/Sync go away, because OwnedFd provides them. The self-pipe backend gets the same ownership. nodedb/src/data/eventfd.rs
Data plane bootstrap The notifier handed to the dispatcher is cloned, not copied. nodedb/src/bootstrap/data_plane.rs
Tests Two unit tests in eventfd.rs. nodedb/src/data/eventfd.rs

Why

EventFdNotifier held only the raw descriptor number. The core thread's EventFd closed that descriptor in Drop. When a core thread ended, for example on the panic at CoreLoop open, the Control Plane kept calling notify() from enqueue and wake_all_cores. The kernel had already given the freed number to the next open(). In a single-node boot the next opens were the catalog and Raft redb files, and each notify() wrote 1u64 into one of them. That destroyed the redb magic number, and the next boot stopped with Not a redb database: magic number mismatch. Full analysis and reproduction are in #419.

With the shared Arc<OwnedFd> the descriptor closes only when the core and the last notifier are gone. Its number cannot be reused while a notifier can still write, so a core that ends can no longer alter an unrelated file. A notify() for a core that is gone writes to its own eventfd, where nothing reads it.

Checks

  • Formatting: cargo fmt --all --check is clean.

  • Focused tests: cargo test -p nodedb --lib data::eventfd runs 7 tests, 7 passed (5 existing, 2 new). The new tests assert that a file opened after the core drops cannot get the notifier's descriptor number and stays empty after notify(), and that the descriptor outlives the core until the last clone drops.

  • Reproducer from runtime: EventFdNotifier keeps a raw fd past its core's death — notify() overwrites the header of whichever redb file reused it #419, compiled around the unmodified eventfd.rs from 4b17c1223: fails with the header overwritten (01 00 00 00 00 00 00 00 0a …). With this change it passes.

  • End to end, on a copy of a store that makes both core threads panic at open, booting repeatedly and checking the magic number of every redb file after each boot:

    Binary Runs Result
    4b17c1223 unmodified 2 runs files damaged in both: on boot 2 in the first run and on boot 1 in the second
    4b17c1223 with this change 1 run of 10 boots no file damaged. All 29 files valid after the 10th boot. Both cores panicked in boots 2 to 10, one in boot 1.
  • Clippy: cargo clippy -p nodedb --all-targets -- -D warnings stops on two clippy::nonminimal_bool errors that are on main and outside this change (nodedb/src/control/planner/sql_plan_convert/dml/vector_primary.rs:171, nodedb/src/data/executor/handlers/control/calvin_reply/stage.rs:94; rustc 1.96.1). With only that lint allowed locally (-A clippy::nonminimal-bool), clippy exits 0 across all targets for this branch. The full test suite was not run locally.

  • Pre-push preflight: the repo gate flagged clippy (lib) on this branch, from the same two nonminimal_bool errors, and nothing else once the commit messages were fixed. The local toolchain is rustc 1.96.1. CI runs clippy on stable, and the release pipeline for main at 4b17c1223 passed. The push therefore used PREFLIGHT_SKIP=1, for that one reason.

  • The self-pipe backend (non-Linux) changes the same way and was not compiled here. This was built and tested on Linux x86_64 only.

Closes #419

… still write

EventFdNotifier was Copy and held only the raw descriptor number of the
eventfd that wakes a data-plane core, while the core thread's EventFd closed
that descriptor in Drop. When a core thread ended, for example on the panic
at CoreLoop open, the Control Plane kept its notifier copies and kept
calling notify(). The kernel had already given the freed number to the next
open(), so each notify() wrote 1u64 into whichever file held it. In a
single-node boot that file was a catalog or Raft redb file, and the write
destroyed its magic number.

EventFd and EventFdNotifier now share the descriptor through Arc<OwnedFd>.
The notifier is Clone, no longer Copy, and the descriptor closes when the
core and the last notifier are gone, so its number cannot be reused while a
notifier can still write. The two Drop impls and the unsafe Send and Sync
impls go away: OwnedFd already provides them. The self-pipe backend gets the
same ownership.

One call site changes: the data plane bootstrap clones the notifier it hands
to the dispatcher.

Two tests: a file opened after the core drops cannot receive the descriptor
number and stays empty after notify(), and the descriptor outlives the core
until the last clone drops.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pv7TPc3Q7xPYXWzocBbZbq
Copilot AI balanced review requested due to automatic review settings October 16, 2026 06:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@EnRaiha

EnRaiha commented Oct 18, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #423. It shares the wake descriptor between the core and its notifiers the same way, keeps both ends of the non-Linux wake pipe together, and also covers the io_uring reader and the async bridge. Closing in favour of #423.

@EnRaiha EnRaiha closed this Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runtime: EventFdNotifier keeps a raw fd past its core's death — notify() overwrites the header of whichever redb file reused it

2 participants