[vharseko]

Fixes #1227

Problem

A replication server can skip one change of a replica for a directory server which is catching up, and the state of that directory server then moves past the change for good. This is the one-change loss of #1225, which docker-convergence (combined) showed again in run 39407529872 on a build with #1219. #1227 has the analysis of that run.

A cursor on a changelog log reads the record it is positioned on in advance:

1. LogFileCursor.positionTo() keeps that record in initialRecord for the next next(), and leaves the file pointer after it.
2. When the head log file is rotated, Log.rotateHeadLogFile() moves every cursor open on it to the read-only file it was renamed to. It does so from the state InternalLogCursor.getState() saved: the file position and the current record, but not initialRecord.
3. A cursor positioned on the head log file and moved before it read anything therefore starts after the record it is positioned on. FileReplicaDBCursor takes the next one for the first change after its start CSN.

positionTo() and the first next() take the shared lock of the log one after the other. So a rotation which asks for the exclusive lock in between runs before the read. Two roads go through that window:

• MessageHandler.fillLateQueue() makes such cursors anew for every batch it reads.
• FileReplicaDBCursor.nextWhenCursorIsExhaustedOrNotCorrectlyPositionned() positions again each time it looks for the changes written since.

Fix

• Log.CursorState keeps the record read in advance, and the state moved to the rotated file carries it.
• LogFile.LogFileCursor.getInitialRecord() gives that record to the state.
• LogFile.getCursorInitialisedTo() gives it back to the cursor made on the rotated file.

Tests

In LogTest, on its log which rotates after two records:

• aCursorReadsTheRecordItIsPositionedOnWhenTheHeadLogFileIsRotatedBeforeItsFirstRead: a cursor positioned on the head log file reads that record first after a rotation. It covers every key matching and positioning strategy which reads a record in advance, 6 cases.
• aCursorPositionedAgainReadsTheRecordItIsPositionedOnWhenTheHeadLogFileIsRotatedBeforeItsNextRead: the same for a cursor which read the newest record and is positioned again once more were written. This is the road FileReplicaDBCursor takes.

Verification

• Without the fix: 7 of the 95 tests of LogTest fail, each one on the skipped record (expected key009, key010 or key012, read the record after it).

• With the fix: 95 of 95 pass.

• Mutants: each link of the fix was broken in turn, 5 mutants:

  • the state taken before the rotation drops the record;
  • the state moved to the rotated file drops it;
  • the cursor made from the state is not given it;
  • the cursor does not keep it;
  • the getter returns the current record instead.

  Each one fails all 7 new cases.

• Reactor (-Pprecommit verify -pl opendj-server-legacy -am, JDK 27, with javadoc), org/opends/server/replication/** and ChangelogBackendTestCase:

  • 3967 tests, 2 failures. Both failures are the embedded server failing to start in setUp of FileReplicaDBTest and ComputeBestServerTest, on 0.0.0.0:65534: Address already in use: another test server held the admin port.
  • Run again on their own, both classes pass (17 and 46 tests).
  • Every class of replication/server/changelog/file passes, and ChangelogBackendTestCase passes (31).

[maximthomas]

praise: the fix carries the record read in advance along the only road the cursor state travels, and the new tests are a real pin.

• CursorState is produced only by InternalLogCursor.getState (Log.java:1305, called from the rotation at :1084) and consumed only by updateOpenedCursorsOnHeadAfterRotation (:1051) → reinitializeTo (:1325); all three now carry initialRecord, and the invalid-state constructor clears it too (:1626).
• With Log.java and LogFile.java at the base and LogTest at the head, all 7 new rows fail on the record after the expected one (key009 → key010, key010 → key011, key012 → key013) and the other 88 pass; at the head LogTest is 95/95 on CI (ubuntu-latest, JDK 27).