runtime: EventFdNotifier keeps a raw fd past its core's death — notify() overwrites the header of whichever redb file reused it #419

Version / build tested against

origin/main @ 4b17c1223 (release build, nodedb 0.7.0-dev, wire format 1).

The Copy notifier over a raw fd is also in v0.6.0 and v0.6.1. nodedb/src/data/eventfd.rs is unchanged by every commit after 68217ebef.

Deployment mode

Origin — single node (local)

Engine(s) involved

Not engine-specific / unsure — Data Plane core wake (nodedb::data::eventfd) and Control Plane dispatch.

Summary

EventFdNotifier is Copy and stores only the raw descriptor number of the eventfd that wakes a data-plane core. The EventFd that owns the descriptor lives on the core thread and closes it in Drop. When a core thread ends, the descriptor closes. The Control Plane still holds copies of the notifier and still calls notify() on every enqueue and every wake_all_cores. The kernel gives the freed number to the next open(). In our run the next opens were the catalog and Raft *.redb files, so each later notify() executed write(fd, &1u64, 8) into one of them. redb files begin with a magic number. The write destroys it, and the next boot stops with Not a redb database: magic number mismatch. The store cannot start until the files are restored.

This is not a store-compatibility report (see #418). Any core death triggers it. The trigger in our run was the .expect("failed to open CoreLoop engines") at boot.

Steps to reproduce

Deterministic. No server and no data directory are needed. Add this test to the tests module of nodedb/src/data/eventfd.rs on origin/main and run it alone, because it depends on the kernel reusing the freed descriptor number:

#[test]
#[cfg(target_os = "linux")]
fn notify_after_the_core_dropped_writes_into_a_reused_descriptor() {
    let efd = EventFd::new().unwrap();
    let notifier = efd.notifier();
    drop(efd); // the core thread exits: close(fd)

    // A redb file opened afterwards gets the lowest free number: the one just closed.
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("system.redb");
    let mut header = b"redb\x1a\n\xa9\r\n".to_vec();
    header.resize(4096, 0);
    std::fs::write(&path, &header).unwrap();
    let file = std::fs::OpenOptions::new().read(true).write(true).open(&path).unwrap();

    notifier.notify(); // the Control Plane wakes a core that no longer exists
    drop(file);

    let after = std::fs::read(&path).unwrap();
    assert_eq!(&after[..9], &header[..9], "redb header overwritten: {:02x?}", &after[..16]);
}

cargo test -p nodedb --lib notify_after_the_core_dropped -- --test-threads=1
-- FAILED: redb header overwritten: [01, 00, 00, 00, 00, 00, 00, 00, 0a, 00, 00, 00, 00, 00, 00, 00]   <- WRONG

I compiled the unmodified eventfd.rs from 4b17c1223 in a stand-alone crate to run this. The file's five existing tests pass and this one fails.

End to end, boot any data directory in which a data-plane core thread panics while it opens its engines. On 4b17c1223 we reached that with a store written by the earlier main build c3d6c10f1. Its sparse/core-N.redb still held the old text.postings table, and e16f5be75 changed that table's key type. On a copy of that store, 4b17c1223 damaged two files on its second boot attempt. The logging build described below did the same on its second attempt.

Expected behavior

notify() never writes to a descriptor that the notifier does not own. The end of a core can never change an unrelated file. A wake for a core that is gone does nothing.

Actual behavior

notify() writes 1u64 to whatever file now owns the number. write() advances the shared file offset and redb does positional I/O, so repeated notifies land at offsets 0, 8, 16. First bytes after two boots of 4b17c1223 on the store copy:

A valid file starts 72 65 64 62 1a 0a a9 0d 0a (redb\x1a\n\xa9\r\n). The same signature appeared in the original incident, on a local build (4b17c1223 plus four startup-logging commits and an unrelated query-response change): system.redb held two consecutive 1u64 at offsets 0 and 8, and raft/group-3.redb held one at offset 0. Both files had the same modification time to the nanosecond. The core threads had panicked about 5 seconds earlier.

What actually happened? (check all that are true)

• Acknowledged/committed data was lost, corrupted, or silently wrong
• The server crashed, hung, or failed to start
• A security or isolation boundary was crossed
• Core functionality is broken with no acceptable workaround
• A workaround exists (rewrite the query, avoid one path, etc.)

Proposed severity

SEV-1 — Critical: stored data lost/corrupted, crash/won't start, or cross-tenant breach. The catalog and a Raft group were destroyed on disk. Each failed boot attempt can repeat the damage.

Reproducibility

Deterministic in the unit test above. End to end it is intermittent: the damage needs the Control Plane to notify a dead core after the freed number was reused.

Last known-good version / commit (if a regression)

Not a regression. The pattern is in v0.6.0.

Environment & logs

Linux x86_64, release build, data_plane_cores = 2. Log of the second boot on 4b17c1223:

nodedb: panic at nodedb/src/data/runtime/spawn.rs:80:14
nodedb: panic at nodedb/src/data/runtime/spawn.rs:80:14
WARN nodedb::control::distributed_applier::apply_loop::helpers: applying committed write failed reason=Internal { detail: "core-0 is gone; the request can never be executed" }
StartupError: server failed to start — aborting startup
schema registry rehydration failed: schema rehydration: failed to re-register collection (database DatabaseId(0)): data plane error: Internal { detail: "core-0 is gone; the request can never be executed" }

On the next start: catalog exists but could not be opened to load persisted WAL tombstones: failed to open catalog read-only: open catalog read-only: I/O error: Not a redb database: magic number mismatch.

Additional evidence (4b17c12)

Code path, pinned to 4b17c1223:

The Control Plane keeps booting after the cores die. It opens the Raft and catalog files and only then fails on the first request to a dead core. Those opens take the freed numbers.

Fix: #421. It shares ownership of the descriptor, so a notifier can never write to a number that was closed and handed to another file. EventFd holds Arc<OwnedFd> and EventFdNotifier clones the Arc and is no longer Copy. One call site changes (data_plane.rs#L379).