Skip to content

[#339] Resume a tamper-evident CSV file with no event since the last rotation - #342

Merged
vharseko merged 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:issue-339-secure-csv-resume-after-rotation
Oct 19, 2026
Merged

vharseko merged 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:issue-339-secure-csv-resume-after-rotation

Conversation

@vharseko

@vharseko vharseko commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Fixes #339. Fixes #372. Fixes #373.

Resuming a file with no event since the last rotation

After a rotation, the live file holds the header and the signature chained from the archive. CsvSecureVerifier kept that signature for the chain but did not count it as a final signature, so the file failed with Missing final signature, and SecureCsvWriter could not resume it after a restart. The signature chained into row 2 now counts as the final one until an event follows. A file truncated to these two rows after later events is still rejected, as the current key in its keystore no longer matches.

Signatures around a rotation

SecureCsvWriter now tracks whether events were written since the last signature, and signs only then:

  • A rotation signs them whether or not the scheduled signature could be cancelled. When the event that rotated the file was the first one since a periodic signature, no signature was scheduled yet, and the archive ended with that event unsigned: Final HMAC key doesn't match expected value (Tamper-evident CSV audit: an archive rotated by the first event after a periodic signature ends unsigned #372).
  • No signature is written or scheduled with no event to cover. After a rotation on write, writeEvent() scheduled a signature that close() wrote into the new file over the last HMAC of the previous file, which the new file does not hold, so it failed at row 3. This is the failure that shouldRotateCsvAndKeyStoreFile asserted for its last archive (CAUD-225), and such a live file could not be resumed either.
  • A rotation before any signature, by the rotate action or by a time-based rotation of a topic with no event since startup, leaves the new file empty instead of writing an empty chained signature (,,), on which the verifier threw a NullPointerException when the writer resumed the file.
  • The header goes in the same write as the first event. Written on its own, it could rotate the file when a time-based rotation was due, and that rotation had no event to sign: the archive held only the header and failed with Missing final signature, and the event went into the new file without a header. This is likely after a restart over the empty live file that the rotation above leaves: its modification time counts as the last rotation, and the first event can come before the first rotation check.

A second close() does nothing. As close() now always runs the signature step, which flushes the writer, a second call with rotation threw IOException: Stream closed.

An event written while the periodic signature completes

writeEvent() scheduled a signature only when the scheduled one was done. The periodic task releases signatureLock and flushes the writer before its FutureTask is done, so an event written in this window scheduled none, and stayed unsigned until the next event, a rotation or close() (#373). A process killed meanwhile left a live file that failed verification at the next start.

The writer now keeps the scheduled task until it starts signing, in pendingSignature. writeEvent() schedules a signature for unsigned events when no task is pending, and a task clears pendingSignature once it holds the lock, before it signs: an event written after that comes after its signature, and schedules the next one, whether the task is done or not. A rotation or close() that cancels the scheduled task clears it too.

A task clears pendingSignature only if it is that task. cancel(false) returns true for a task that has started and waits for the lock, as a FutureTask stays NEW until its run() returns, and the task still runs. With a plain flag, this gives the interleaving that #373 notes: an event schedules a second task, the first one clears the flag when it gets the lock, and the next event schedules a third; close() cancels only the last one, and waits for the second up to the signature interval. The signature task is now one SignatureTask per schedule instead of one shared Runnable, so that a task can tell whether it is the pending one.

Verifier

CsvSecureVerifier also:

  • accepts an empty file whose keys match, as the rotation above archives one;

    This replaces the rule of [#331] Tamper-evident CSV audit: verify compressed archives, rename the keystore on the rotate action #332, which rejected every file without a header: a file truncated to nothing after events still fails, as the current key of its keystore moved on. CsvSecureVerifierTest.shouldNotVerifyEmptyFile now asserts that failure reason.

  • rejects a row 2 signature that is not base64;

  • rejects a signature row that holds other values: the data and HMAC columns of signature rows were not checked, so an event could be written into them;

  • reports a row with neither an HMAC nor a signature as an incorrect HMAC instead of throwing.

Tests

  • SecureCsvWriterTest: resuming after a forced rotation and after a rotation on write, an event that rotates the file after a periodic signature, a rotation before the first event, a file truncated to the chained signature, a malformed or empty chained signature, signature rows that hold other values, a first event whose write rotates the file, a second close(). shouldRotateCsvAndKeyStoreFile now expects every archive and the live file to pass, with the plain and the .gz suffix; it has 5 archives instead of 6, as the 6th came from the redundant signature of close().
  • SecureCsvWriterTest.shouldGenerateHMACColumn has a 60 s time-out. It fails when unsignedEvents is cleared after the signature row is written instead of before: that row then schedules another signature, and close() waits for it for the whole signature interval.
  • SecureCsvWriterTest, for Tamper-evident CSV audit: an event written while the periodic signature completes stays unsigned #373:
    • shouldSignAnEventWrittenWhileThePeriodicSignatureCompletes: the writer's flush(), which writeSignature() calls once the signature row is written, writes a second event the first time it runs. A signature must follow that event, and the live file must pass verification while the writer is still open. With the isDone() condition it fails: no second signature within 10 s.
    • shouldSignPeriodicallyAnEventWrittenAfterARotationCancelledTheScheduledSignature: a forced rotation cancels the scheduled signature, and the next event must get a periodic one. It fails when the cancel leaves pendingSignature set.
    • shouldScheduleNoOtherSignatureWhenARotationSignsWhileTheScheduledOneWaitsForTheLock: the test holds the lock until the scheduled signature waits for it, rotates and writes events, and close() must return in less than half the 3 s interval. When a task clears pendingSignature whatever it holds, close() takes 2.7 s.
  • CsvWriterTest.shouldCreateNewAuditFileAfterRotation verifies the live file, in place of a commented-out TODO.
  • The temporary directories of the SecureCsvWriterTest tests this PR adds or changes are under target.

Each fix was disabled in turn, and its test failed. audit/core and audit/handler-csv pass locally, and SecureCsvWriterTest passed 5 runs in a row.

Known limits

  • The signature chained into row 2 is not authenticated: a live file with no event since the rotation passes with any base64 value there. No tool checks the link between files (CsvSecureArchiveVerifier verifies each file on its own), the row can hold no event, and a truncation is caught by the keys.
  • Files written by earlier versions with a signature over the HMAC of the previous file at row 3, or with an empty chained signature, still fail verification: the first needs the HMAC of the previous archive, the second holds nothing to verify. To resume such a live file, move it and its .keystore aside before the restart.

Merging with #347 or #348 does not conflict, and #362 does not touch these files. #355 conflicts with master in the constructor of SecureCsvWriter, and with this PR in no other place.

@vharseko vharseko added bug security Security fixes and CVE remediation java Pull requests that update java code data-loss Data lost or irrecoverably overwritten labels Oct 16, 2026
@vharseko
vharseko requested a review from maximthomas October 16, 2026 17:58
@vharseko
vharseko force-pushed the issue-339-secure-csv-resume-after-rotation branch from d4688ce to 529de3a Compare October 16, 2026 18:39
@vharseko vharseko added the tests Test code changes label Oct 16, 2026

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The close is placed exactly where the leak was, and the tests measure it.

  • The SecureCsvWriter constructor's catch now closes csvWriter, which stops the RotatableWriter rotator. Removing that close turns all five rows of shouldStopRotatorAndFailWithIOExceptionWhenFileCannotBeResumed red ([pool-N-thread-1 alive]). Measured on JDK 26.
  • Restoring the RuntimeException wrapping turns testStartupOpensOtherTopicsWhenTamperEvidentFileCannotBeResumed red. Measured on JDK 26.
  • readInitialKey() == null is checked before verify(), so a missing keystore fails with a message that names the keystore, not with an IllegalStateException from the verifier.

issue (blocking): A resumed file with a row cut short, a wrong column count or an unclosed quote still fails the whole startup, not only its topic.

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:150, :201-205; commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/CsvSecureVerifier.java:97, :108

CsvSecureVerifier.verify() catches only SecureStorageException. super-csv's SuperCsvException from csvReader.read(header) therefore reaches the constructor's catch, which wraps it in RuntimeException. A probe against super-csv 2.4.0 got "the nameMapping array and the sourceList should be the same size" for a short row and "unexpected end of file while reading quoted column" for an unclosed quote. CsvAuditEventHandler.startup() catches only IOException, so the loop over the topics ends and AuditServiceImpl.startup() leaves the service in STARTING. A crash or a full disk during a write leaves the last row cut short, which is a common way for a file to become unresumable. The description says such a file "now always fails" with an IOException. A header line with one column fails on the same path: header[header.length - 2] throws ArrayIndexOutOfBoundsException.

// SecureCsvWriter.java:149-150 — import org.supercsv.exception.SuperCsvException;
CsvSecureVerifier verifier = new CsvSecureVerifier(csvFile, csvPreference, secureStorage);
final CsvSecureVerifier.VerificationResult verificationResult;
try {
    verificationResult = verifier.verify();
} catch (SuperCsvException e) {
    // A row cut short by a crash while it was written, or a row with a missing or extra column.
    throw new IOException("Resuming an existing CSV file but it cannot be parsed: " + csvFile, e);
}

// CsvSecureVerifier.java:97 — a header with fewer than two columns fails verification
if (header.length < 2 || !(HEADER_HMAC.equals(header[header.length - 2])
        && HEADER_SIGNATURE.equals(header[header.length - 1]))) {

Pin: add an unresumableFiles row that appends a line with fewer columns than the header to a written file. At this head it fails with the RuntimeException. With the fix it fails with the IOException.


question (non-blocking): Should an unreadable .keystore next to a new file fail the whole startup, as the description says, or only its topic, as the code now does?

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:118, :201-203

JcaKeyStoreHandler.init() opens an existing keystore path with KeyStoreBuilder.withKeyStoreFile(). For a mode-000 file or a directory, its new FileInputStream throws FileNotFoundException before anything is wrapped in IllegalStateException. The inner catch takes only IllegalStateException, so this IOException now passes through for a new file too, and startup() logs it for the topic. At base it was wrapped, and the startup failed. The description says a new file next to a keystore that cannot be loaded "still fails as before". Either answer is minor. If per-topic is intended, only the sentence changes. If "as before" is intended, the inner catch also takes IOException and wraps it when !fileAlreadyInitialized.

Pin: a 0-byte CSV next to a directory named <csv>.keystore, asserting the outcome you choose.


question (non-blocking): Is it intended that a topic that never got an event still fails the whole startup after a Password change?

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:123-125

A topic's keystore is stored when its writer is constructed, but its header is written only with the first event. An idle topic therefore leaves a 0-byte CSV next to a keystore. After the Password key changes, that keystore no longer loads. fileAlreadyInitialized is false, so the IllegalStateException is rethrown and wrapped, and the startup fails, the same as at base. The description says "a topic without a file still starts with a new keystore", which holds only when no keystore is left behind. If this is intended, the sentence needs "and no keystore". If not, dropping the rethrow at :123-125 makes such a topic fail on its own.


issue (non-blocking): The rotator can rotate a resumed file while the file is still being verified. This is pre-existing and not changed by this PR.

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:104, :239-240, :443-451

constructWriter() builds the RotatableWriter and registers the rotation hooks before the keys are read and verify() runs. The first rotation check runs one check interval later (default 5s). Take a resumed file that is already over maxFileSize or maxFileLifetime and takes longer than that interval to verify. It is renamed while it is being verified, and postRotationAction renames .keystore and swaps in random keys. verify() then reports "Final HMAC key doesn't match" for a valid file that has already been archived. This was traced in the code, not run. Suggested follow-up issue: start the rotation checks only once construction has succeeded.


nitpick (non-blocking): The IOException arm of the new close catch can never fire.

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:198

TextWriterAdapter.close() calls shutdown(). With rotation, RotatableWriter.shutdown() logs the IOException from its close(). Without rotation, TextWriter.Stream closes a PrintWriter, which swallows the exception. A failed close is therefore logged and not added as suppressed; only a RuntimeException can be. The catch has to stay because the compiler requires it. The description's "a failure of that close is added to the original one as suppressed" could say "a RuntimeException from that close".

vharseko added a commit that referenced this pull request Oct 18, 2026
… yet (#354)

Fixes #346.

`SecureCsvWriter` writes the CSV header with the first event, so the
audit file of a tamper-evident topic is empty after startup, and after
the `rotate` action of a topic without file rotation, until an event is
published. `CsvAuditEventHandler.getEntries()` passed the `null` header
of such a file to `convertDotNotationToSlashes()`:

- `readEvent()` threw the `NullPointerException` out of the call instead
of answering a promise;
- `queryEvents()` answered `BadRequestException`.

`getEntries()` now returns no entry for a file without a header:
`readEvent()` answers `NotFoundException`, and `queryEvents()` answers
an empty result. This also covers the empty file that a rotation before
the first signature leaves with #342.

## Tests


`CsvAuditEventHandlerTest.testReadingTamperEvidentAuditLogWithNoEventYet`
and `testQueryOnTamperEvidentAuditLogWithNoEventYet`, after startup and
after the `rotate` action. Each case first checks that the audit file is
empty, and fails without the fix as the issue describes. `audit/core`
and `audit/handler-csv` pass locally on JDK 11 and 27.

The change applies cleanly on #341, #342, #347 and #348.
@vharseko vharseko added the concurrency Race conditions, locking and thread-safety label Oct 18, 2026
@vharseko
vharseko force-pushed the issue-339-secure-csv-resume-after-rotation branch from 529de3a to 63e4495 Compare October 18, 2026 10:44
@vharseko

Copy link
Copy Markdown
Member Author

@maximthomas, this review seems to have been posted to #342 by mistake. Its text is identical to round 1 of #341 (#341 (review)), which you posted 55 minutes earlier, and it reviews code that only #341 contains: the close of csvWriter in the constructor's catch, the readInitialKey() == null check, the inner IllegalStateException catch around the keystore, and the tests shouldStopRotatorAndFailWithIOExceptionWhenFileCannotBeResumed and testStartupOpensOtherTopicsWhenTamperEvidentFileCannotBeResumed. In the commit it was posted on (529de3a), SecureCsvWriter.java:150 is a closing brace, :198 is a comment and :201-205 is getRandomBytes().

Every point is answered in #341 (#341 (comment)), and you approved its round 2:

  1. A resumed file that cannot be parsed: fixed in [#340] Tamper-evident CSV audit: close the file of a writer that fails to resume it #341, which turns any RuntimeException from verify() into an IOException.
  2. An unreadable .keystore next to a new file, and 3. an idle topic after a Password change: both are about the keystore catch that [#340] Tamper-evident CSV audit: close the file of a writer that fails to resume it #341 adds, and are answered there.
  3. The rotator during the verification: opened as Tamper-evident CSV audit: the rotator can rotate a resumed file while the writer still verifies it #343, fixed in [#343] Tamper-evident CSV audit: construct the rotatable writer once the resumed file is verified #355.
  4. The IOException arm of the close catch: about the catch that [#340] Tamper-evident CSV audit: close the file of a writer that fails to resume it #341 adds; its description was corrected.

The branch is now rebased onto master (63e4495), with the same patch; audit/core and audit/handler-csv pass. The description now names the open PRs it merges with: only #369 conflicts, in CsvSecureVerifier.

Could you dismiss this review and review the changes of #342 itself?

@vharseko
vharseko requested a review from maximthomas October 18, 2026 10:47

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the review above was posted by mistake, sorry.
Here is the right review:

praise: The signing model now follows what was written, not the state of the scheduled task.

  • unsignedEvents lets forceWriteSignature sign on every rotation and on close() whatever the scheduled Future did, which removes #372's unsigned last event in an archive.
  • holdsOnlyTheSignature closes the unchecked data and HMAC columns of signature rows.
  • shouldRotateCsvAndKeyStoreFile now verifies every archive and the live file with both the plain and the .gz suffix, in place of asserting the CAUD-225 failure. An empty .gz archive is a valid 20-byte gzip that the verifier reads to EOF, so it also takes the empty-file path.

issue (blocking): When the header write triggers a rotation, the archive gets only the header and the first event lands in the new live file without a header.

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:338-340, :269-273, :439-443

writeEvent writes the header on its own first, and RotatableWriter.write rotates inline after each write. Because the event has no HMAC yet, preRotationAction has nothing to sign and lastSignature stays null. postRotationAction then takes the new branch and sets headerWritten = false (:442), and writeHeader overwrites that with true when it returns (:273). This happens with a time-based policy when no event has been signed since startup. The most likely case is a restart over the empty live file that this branch leaves behind: the old mtime becomes lastRotationTime, and the first event comes before the rotator's first check. The archive then fails with Missing final signature, the live file fails with Found only 0 checked headers, and the next start reports the file as tampered. The pin below is red at 63e4495 (Missing final signature on the archive). It is green with the change below, and SecureCsvWriterTest passes 15/15 (JDK 26, macOS). The flag has to be set before the write. If it is set after the combined write, the signature that the rotation writes re-enters writeEvent and prepends a second header: The signature row 3 holds other values.

// SecureCsvWriter.writeEvent(Writer, Map): delete `if (!headerWritten) { writeHeader(headers); }` (:338-340),
// and in place of `writer.write(csvFormatter.formatEvent(extendedValues, extendedHeaders));` (:348):
String row = csvFormatter.formatEvent(extendedValues, extendedHeaders);
if (!headerWritten) {
    // The header and the first event in one write, so that a rotation this write triggers archives
    // both and signs the event; set first, as the signature that rotation writes comes from inside it.
    row = csvFormatter.formatHeader(extendedHeaders) + row;
    headerWritten = true;
}
writer.write(row);

postRotationAction's writeHeader and writeLastSignature (:446-448) can stay as they are, because they run with headerWritten already true. The headerWritten = false at :442 can stay as a guard.

Pin: an empty live file dated a day back, with a 1-hour rotation interval; its first event must leave one archive and a live file that both pass.

@Test
public void shouldSignTheFirstEventWhenItsWriteRotatesTheFile() throws Exception {
    final File actual = newSecureCsvFile();
    // an empty live file last rotated a day ago: with a 1 hour interval, the first write rotates it
    assertThat(actual.createNewFile()).isTrue();
    assertThat(actual.setLastModified(System.currentTimeMillis() - TimeUnit.DAYS.toMillis(1))).isTrue();
    final String header = "FOO";
    final CsvAuditEventHandlerConfiguration config = createRotatingSecureConfig(0);
    config.getFileRotation().setRotationInterval("1 hour");
    try (SecureCsvWriter secureCsvWriter = new SecureCsvWriter(
            actual, new String[]{header}, CsvPreference.EXCEL_PREFERENCE, config, keyStoreHandler, random)) {
        secureCsvWriter.writeEvent(singletonMap(header, "one"));
    }
    final List<File> archives = new TimeStampFileNamingPolicy(actual, "-yyyy.MM.dd-HH.mm.ss.SSS", "").listFiles();
    assertThat(archives).hasSize(1);
    assertPassesVerification(archives.get(0));
    assertPassesVerification(actual);
}

suggestion (non-blocking): No test checks that unsignedEvents is cleared before the signature row is written.

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:286-291

This order is what keeps the signature row's own writeEvent from scheduling a task under the new condition at :354. Move :288 after :291 and close() changes: it cancels the scheduled task, its signature row then schedules a new one, and awaitTermination waits for it. shouldGenerateHMACColumn still passes, but takes 300.8 s instead of under a second, and every other test stays green. No test sets a time bound that would catch this.

@Test(timeOut = 60_000)
public void shouldGenerateHMACColumn() throws Exception {

Pin: with this timeOut, the test is red under that mutant (close() takes 300.8 s) and green at the head, where the whole class runs in about 26 s. I did not run it with the timeOut added.


issue (non-blocking): If an event is written after the periodic signature task releases the lock but before it completes, no new signature is scheduled for that event.

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:354-362, :300-303

writeSignature unlocks (:301) and flushes (:302) before the task's run() returns, and FutureTask becomes done only after that. An event written in this window sees !scheduledSignature.isDone() and schedules nothing. On a quiet topic it stays unsigned until the next event, a rotation, or close(). If the process is killed in that state, the live file ends in an unsigned row. The race already exists on master with the same isDone() test. This PR's close() now always signs, which limits the problem to shutdowns that are not clean. A follow-up issue is enough. Sketch, not run:

private boolean signatureScheduled; // guarded by signatureLock

// writeSignature, first statement inside the lock, before the `if (!unsignedEvents)` guard:
signatureScheduled = false;

// writeEvent, in place of :354; set only once schedule() returned, so a RejectedExecutionException leaves it false:
if (unsignedEvents && !signatureScheduled) {
    logger.trace("Triggering a new signature task to be executed in {}", signatureInterval);
    try {
        scheduledSignature = scheduler.schedule(signatureTask, signatureInterval.getValue(),
                signatureInterval.getUnit());
        signatureScheduled = true;
    } catch (RejectedExecutionException e) {
        logger.error(e.getMessage(), e);
    }
}

suggestion (non-blocking): With rotation enabled, calling close() a second time now throws IOException("Stream closed").

commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:256-263, :300-303

forceWriteSignature now always calls writeSignature. When there is nothing to sign, it returns at :282, but its finally still calls flush(), which reaches the BufferedWriter that the first close() already closed. Master returned normally here. No caller closes twice: CsvAuditEventHandler.closeWriter removes the writer before it closes it. A guard at the top of close() makes it idempotent:

private boolean closed;

// close(), first statements:
if (closed) {
    return;
}
closed = true;

@vharseko

Copy link
Copy Markdown
Member Author

@maximthomas, the round is in 6b4ab04:

  1. The header write that rotates the file: fixed as you suggested. The header and the first event go in one write, and headerWritten is set before it. shouldSignTheFirstEventWhenItsWriteRotatesTheFile is your pin, and it also checks that the event is in the archive. At 63e4495 it fails with an archive that holds only the header. With the flag set after the write, it fails with The signature row 3 holds other values.
  2. The order of unsignedEvents = false: shouldGenerateHMACColumn now has timeOut = 60_000. With the assignment moved after the signature row, it fails with ThreadTimeoutException after 60 s, as close() waits in awaitTermination. At the head it passes.
  3. An event written while the periodic signature task completes: opened as Tamper-evident CSV audit: an event written while the periodic signature completes stays unsigned #373, and listed under the known limits. The sketch leaves one interleaving, which the issue describes. A rotation or close() signs while the task already runs and waits for the lock, so cancel(false) fails. An event then schedules a second task, the first one clears the flag, and the next event schedules a third. close() cancels only the last one, and waits for the second up to the signature interval.
  4. A second close(): it now does nothing. shouldDoNothingWhenClosedTwice fails at 63e4495 with IOException: Stream closed.

audit/core and audit/handler-csv pass locally, and SecureCsvWriterTest passed 5 runs in a row. The description is updated. Merging with #341, #347, #348, #353 or #355 still does not conflict, and #369 still conflicts in CsvSecureVerifier only.

@vharseko
vharseko requested a review from maximthomas October 19, 2026 07:44
…vent since the last rotation

CsvSecureVerifier now counts the signature chained into row 2 of a rotated
file as its final signature until an event follows, so a live file that holds
only the header and that signature passes verification, and SecureCsvWriter
can resume it after a restart. A file truncated to these two rows after later
events still fails, as the current key in its keystore no longer matches.

SecureCsvWriter now tracks whether events were written since the last
signature, and signs only then:
- a rotation signs them whether or not the scheduled signature could be
  cancelled: an event that rotated the file right after a periodic signature
  was archived unsigned;
- no signature is written or scheduled with no event to cover: after a
  rotation on write, close() wrote one into the new file over the last HMAC
  of the previous file, which failed verification at row 3 (CAUD-225 in
  shouldRotateCsvAndKeyStoreFile);
- a rotation before any signature leaves the new file empty instead of
  writing an empty chained signature, which the verifier could not read.

CsvSecureVerifier also accepts an empty file whose keys match, as such a
rotation archives one, rejects a row 2 signature that is not base64 and a
signature row that holds other values, and reports a row with neither an HMAC
nor a signature instead of throwing a NullPointerException.
…ile with its first event

Written on its own, the header could rotate the file when a time-based
rotation was due, as after a restart over an empty live file last modified
before the rotation interval. That rotation had no event to sign: the archive
held only the header and failed with "Missing final signature", and the event
went into the new file without a header. The header and the first event are
now written together, so that such a rotation archives both and signs the
event. headerWritten is set before the write, as the signature that the
rotation writes comes from inside it.

A second close() does nothing: with rotation, it flushed the closed writer and
threw "Stream closed".

shouldGenerateHMACColumn has a time-out, which fails a close() that waits for
a signature scheduled by its own signature row.
@vharseko
vharseko force-pushed the issue-339-secure-csv-resume-after-rotation branch from 6b4ab04 to 206d27e Compare October 19, 2026 08:24
@vharseko

Copy link
Copy Markdown
Member Author

@maximthomas, rebased onto the current master (28afe37, with #369): the head is 206d27e.

The only conflict was in CsvSecureVerifier.verifyHMAC(). This PR changed actualHMAC.equals(expectedHMAC) to expectedHMAC.equals(actualHMAC), so that a row with neither an HMAC nor a signature is reported as an incorrect HMAC instead of throwing a NullPointerException. #369 replaced that comparison with hmacCalculator.verify(actualHMAC, ...), which compares hmac(data).equals(expectedHmac) and takes a null as well, so the resolution keeps the code of master. The rest of the PR applied as is.

audit/handler-csv passes locally (87 tests).

…itten while the periodic signature completes

SecureCsvWriter scheduled a signature only when the scheduled one was
done. The periodic task releases the signature lock and flushes the
writer before its FutureTask is done, so an event written in this
window scheduled none, and stayed unsigned until the next event, a
rotation or close().

The writer now keeps the scheduled task until it starts signing:
writeEvent() schedules a signature for unsigned events when no task is
pending, and a task clears the pending one once it holds the lock, only
if it is that task. A successful cancel(false) clears it too, but does
not stop a task that has started and waits for the lock: that task then
leaves alone the one an event scheduled since. Otherwise two tasks stay
scheduled, and close() waits for the one it does not cancel, up to the
signature interval.
@vharseko

Copy link
Copy Markdown
Member Author

@maximthomas, #373, the non-blocking issue of your review (an event written while the periodic signature completes), is fixed in this PR now: 00f0a35, on top of 206d27e.

A correction to round 3 and to the note in #373: cancel(false) does not fail when the task has started and waits for the lock. It returns true, as a FutureTask stays NEW until its run() returns, and the task still runs. The outcome is the one described there: with a flag that every signature task clears, a rotation in this state leaves two tasks scheduled, and close() waits for the one it does not cancel, up to the signature interval. It happens on master as well.

So the writer keeps the pending task itself, in pendingSignature, with one SignatureTask per schedule instead of one shared Runnable:

  • writeEvent() schedules a signature for unsigned events when no task is pending.
  • A task clears pendingSignature once it holds the lock, before it signs, and only if it is that task. An event written after that comes after its signature, and schedules the next one, whether the task is done or not.
  • A successful cancel(false) in forceWriteSignature() clears it too. A task that it did not stop leaves alone the one that an event scheduled since.

Three tests, each red with its part of the fix disabled:

  • shouldSignAnEventWrittenWhileThePeriodicSignatureCompletes: the writer's flush(), which writeSignature() calls once the signature row is written, writes a second event. With the isDone() condition, no signature follows within 10 s, and the live file fails verification while the writer is open.
  • shouldSignPeriodicallyAnEventWrittenAfterARotationCancelledTheScheduledSignature: red when the cancel leaves pendingSignature set, as the event after the rotation then gets no periodic signature.
  • shouldScheduleNoOtherSignatureWhenARotationSignsWhileTheScheduledOneWaitsForTheLock: the test holds the lock until the scheduled signature waits for it, then rotates and writes events. When a task clears pendingSignature whatever it holds, close() takes 2.7 s with a 3 s interval; the test requires less than half of it.

audit/core and audit/handler-csv pass locally (90 tests in audit/handler-csv), and SecureCsvWriterTest passed 5 runs in a row. The description is updated: #373 moved from the known limits to its own section. Merging with #347 or #348 still does not conflict. #355 now conflicts with master itself, in the constructor of SecureCsvWriter, and this PR adds no other conflict to it.

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: Each fix in this round sits where its round-2 finding was, and each has a test that pins it.

  • SecureCsvWriter.writeEvent writes the header and the first event as one string (SecureCsvWriter.java:389-396). RotatableWriter.write rotates only after the write, so a rotation this write triggers archives both rows, and preRotationAction signs the event. shouldSignTheFirstEventWhenItsWriteRotatesTheFile covers this.
  • SignatureTask.run clears pendingSignature under signatureLock before it calls writeSignature (SecureCsvWriter.java:515-518). Both mutants were run locally on JDK 26. With the clear moved after writeSignature, shouldSignAnEventWrittenWhileThePeriodicSignatureCompletes fails. With the pendingSignature == this guard dropped, shouldScheduleNoOtherSignatureWhenARotationSignsWhileTheScheduledOneWaitsForTheLock fails: close() took 2694 ms against its 1.5 s bound.
  • close() returns early on a second call (SecureCsvWriter.java:272-276), pinned by shouldDoNothingWhenClosedTwice. The five new cases pass on all 9 CI matrix rows, where handler-csv runs 90 tests at this head against 78 at the base.

@vharseko
vharseko merged commit 0b37cce into OpenIdentityPlatform:master Oct 19, 2026
14 checks passed
@vharseko
vharseko deleted the issue-339-secure-csv-resume-after-rotation branch October 19, 2026 16:42
vharseko added a commit that referenced this pull request Oct 19, 2026
…the resumed file is verified (#355)

Fixes #343.

## Construction order

`SecureCsvWriter` constructed its `RotatableWriter` first, and the
`RotatableWriter` constructor schedules the rotation and retention
checks. Only then did the constructor read the keys and verify the file
it resumes. A check due on the resumed file, by `maxFileSize` or by
`rotationInterval` from its `lastModified()`, could run during the
construction: it renamed the file and its keystore, and
`postRotationAction` wrote new keys into a new keystore, through the
`secureStorage` that the verifier reads. A valid file, already archived,
then failed verification, and the topic did not start.

The writer is now constructed last, once the keys are read and the file
it resumes is verified. The first check runs one check interval after
the construction, on a writer that holds the keys and the last signature
of the file.

## A file that cannot be opened

The file is still opened, and closed at once, before the keys. A file
that cannot be opened, as one in a directory that cannot be created,
fails with an `IOException`, which `CsvAuditEventHandler.startup()` logs
for the topic. Without that, a new file would fail first when its
keystore is stored, with a `SecureStorageException` that the constructor
wraps in a `RuntimeException`, and the startup would fail for every
topic. Opening for append leaves the file and its `lastModified()` as
they are.

## A writer whose check interval is invalid

Found in the review. The `RotatableWriter` constructor opened its file
before it rejected an unlimited or zero rotation and retention check
interval, and the caller of a constructor that throws gets no writer to
close: the file stayed open until its stream was collected. An interval
under one millisecond passed that check, and failed after the open with
the `IllegalArgumentException` of `scheduleAtFixedRate`, which
`CsvAuditEventHandler.startup()` does not catch for the topic. The
constructor now builds the policies and checks the interval before it
opens the file, and rejects an interval under one millisecond with the
same `IOException`. That includes a negative one: `Duration` adds the
parts of a composite without an overflow check, and `9223372036854775807
ms, 1 ms` parses to a negative interval.

## Tests

- `SecureCsvWriterTest.shouldNotRotateResumedFileWhileVerifyingIt`: a
written file is resumed with rotation and a 50 ms check interval, with a
rotation due at the first check: its size at `maxFileSize`, or its
`lastModified()` two hours back with a 1 hour `rotationInterval`. A
`CommentMatcher` in the `CsvPreference` holds the verification for 500
ms on the first line. The construction succeeds, and the first check
after it rotates the file. The archive passes verification, and so does
the live file, with one event after the rotation. On Windows a file that
the verifier holds open cannot be renamed, and a rotation that fails to
rename it still resets the time of the last rotation: there, only the
`rotationInterval` row is sure to fail with the writer constructed
first.
- `SecureCsvWriterTest.shouldFailWithIOExceptionWhenFileCannotBeOpened`:
a file under a regular file fails with an `IOException`.
-
`SecureCsvWriterTest.shouldFailWithIOExceptionWhenCheckIntervalIsInvalidForNewFile`:
a new file with rotation and an `unlimited` check interval fails with
the `IOException` of the `RotatableWriter`.
-
`RotatableWriterTest.testRejectsInvalidRotationRetentionCheckIntervalBeforeOpeningTheFile`:
an `unlimited`, `zero`, `500 microseconds` or overflowed
`9223372036854775807 ms, 1 ms` check interval fails with an
`IOException`, and the file is not created.
`testRejectsInvalidRotationRetentionCheckIntervalWithRetentionOnlyBeforeOpeningTheFile`
does the same with a retention policy alone.
-
`RotatableWriterTest.testAcceptsAnyRotationRetentionCheckIntervalWithoutPolicy`:
with no policy, no check is scheduled, and the same intervals are
accepted. `testAcceptsOneMillisecondRotationRetentionCheckInterval`: one
millisecond is accepted.

Measured on JDK 27:

- With the `SecureCsvWriter` of `master`, both rows fail with "The CSV
file was tampered: The file contains no CSV header": with a 50 ms
interval, the first check runs while the keys are read, and renames the
file before the verifier opens it. With a 1 s interval and a 3 s hold,
both rows fail with the reason given in the issue, "Final HMAC key
doesn't match expected value".
- Without the open and close before the keys,
`shouldFailWithIOExceptionWhenFileCannotBeOpened` fails with the
`RuntimeException`, and so does
`CsvAuditEventHandlerTest.canConfigureCsvHandlerFromJsonAndRegisterWithAuditService`,
whose log directory `/path/to/audit/files/` cannot be created.
- With the writer constructed in the `try` block,
`shouldFailWithIOExceptionWhenCheckIntervalIsInvalidForNewFile` fails
with "RuntimeException: Error when initializing a secure CSV writer".
- With the `RotatableWriter` of `master`, both
`testRejectsInvalidRotationRetentionCheckInterval…` tests fail in their
four rows: the file exists for `unlimited` and `zero`, and `500
microseconds` and the overflowed interval throw an
`IllegalArgumentException`. With a guard that rejects only zero
milliseconds, the overflowed row fails in both with the
`IllegalArgumentException`.
- Three mutants of the guard each fail only their own test: without the
retention clause, the retention-only test in its four rows; without the
policy clause,
`testAcceptsAnyRotationRetentionCheckIntervalWithoutPolicy` in its four
rows; with `<= 1`,
`testAcceptsOneMillisecondRotationRetentionCheckInterval`.
- `audit/core` and `audit/handler-csv` pass locally, and the new tests
passed 3 runs in a row.

## Merging

- With #341: a conflict in the constructor. A failure in its `try` block
now comes before the writer is constructed, so no file or rotator thread
is left open (#340), and the close that #341 adds in the `catch` block
would find `csvWriter` still `null`: the close goes, and the rest of
#341 applies as is. Its tests that the rotator thread terminates still
hold, as the rotator does not start.
- With #342, #347 and #369: no conflict.

## Not in this PR

- `RotatableWriter` schedules its checks in its constructor, and
`registerRotationHooks()` sets the hooks after it, in a field that is
not volatile. The first check runs one check interval later. Not
observed; it follows from the memory model only.
- A new secure topic whose check interval is invalid stores new keys for
its empty file on each publish attempt, and a resumed one verifies its
file, before the writer fails. `master` leaked a `FileOutputStream` on
each attempt instead.
vharseko added a commit that referenced this pull request Oct 19, 2026
#347)

Fixes #334.
Fixes #350.
Fixes #349.
Fixes #365.
Fixes #363.
Fixes #364.
Fixes #351.

## The race

`CsvAuditEventHandler` looked up the writer of a topic without a lock;
only the reset after a failed write and the rotations took the handler
monitor. A publisher that had the writer when the `rotate` action closed
it wrote into the closed writer. With rotation and buffering disabled,
`TextWriter.Stream` wrote through a `PrintWriter`, which swallows the
`IOException`, so the retry never ran: the event was lost and
`publishEvent()` reported success. A stale `SecureCsvWriter` also stored
its keystore over the one of the new file.

With rotation enabled
([comment](#334 (comment))),
the `rotate` action looked up the writer without the monitor too, and
could call `forceRotation()` on a writer that a concurrent reset had
closed.

## The fix

The issue offers two ways out: make a write to a closed writer fail, so
that the retry opens the current writer, or make `rotate` wait for the
publishers that are already writing. This PR does the latter. A closed
writer that throws would not cover the path with rotation enabled, and
would leave the gap between the final signature and the close of the
stream in `SecureCsvWriter.close()` that #337 left to this issue. A
write to a closed writer now fails as well (#349, below), but only the
lock covers both paths.

A `ReentrantReadWriteLock` per topic guards the writer of the topic, and
replaces the handler monitor:

- a publisher looks up the writer and writes the event under the read
lock. If the topic has no writer, or the write fails, it retries under
the write lock, with a new writer unless another thread has replaced the
writer meanwhile;
- the `rotate` action with rotation enabled holds the read lock during
`forceRotation()`: publishers keep writing, and a reset cannot close the
writer meanwhile;
- the default rotation, `startup()` and `shutdown()` take the write
lock, so they wait for the events being written. The rotated file is
still compressed after the lock is released (#332).

With a lock per topic, the other topics keep publishing while the writer
of one topic is replaced. With a single lock they would not: new readers
of a `ReentrantReadWriteLock` also queue behind a waiting writer, so one
event being written plus a waiting rotation would stop every topic. The
retry now also opens a writer when a failed reset of another thread has
left the topic without one, where it threw a `NullPointerException`
before.

## Events after `shutdown()` (#350)

`shutdown()` closed the writers but left them in the map, and nothing
marked the handler as shut down. An event published afterwards went to
the closed writer of its topic. With rotation disabled, it was lost
while `publishEvent()` reported success, and a tamper-evident writer
still stored its next key, so the file failed verification and could not
be resumed. With rotation enabled, the closed `RotatableWriter` threw,
and the retry opened a new writer that nothing closed. The `rotate`
action after `shutdown()` also rotated the file and opened a new writer,
in both modes.

`shutdown()` now marks the handler as shut down before it takes the
first write lock, and removes each writer it closes from the map. A
topic without a writer then gets no new one:

- the retry under the write lock fails with a
`ServiceUnavailableException`, so the event is reported as lost, and is
written neither to a closed writer nor to a new one. A topic that has a
writer still resets it after a failed write, as `startup()` may have
reopened it already;
- the `rotate` action finds no writer, and fails with a
`ServiceUnavailableException`, as a publish does (#364, below).

An event that has the read lock of its topic when `shutdown()` gets
there is still written, and `shutdown()` waits for it. `startup()`
clears the mark once it has opened the writers, also when a writer
throws a `RuntimeException` (#340), so that a restarted handler opens a
writer again, at the next event, for a topic whose file it could not
open. Until `startup()` returns, an event or a `rotate` action on a
topic that it has not opened yet fails with a
`ServiceUnavailableException`, as the Javadoc of `startup()` now states.
Before, such an event went to the closed writer of its topic, as above,
and with rotation enabled `startup()` then replaced the writer that the
retry had opened, without closing it. `startup()` now closes a writer
that an event opened before it, or after a `startup()` that threw,
before it opens a new one.

The retry under the write lock also rethrows a `ResourceException` as
is. `ResourceException` is an `IOException`, so the
`ServiceUnavailableException` would have become an
`InternalServerErrorException`, and the `InternalServerErrorException`
of a failed reset would have been wrapped into another one, since the
reset moved into that `try`.

## Write failures (#349)

With rotation disabled, `StandardCsvWriter` and `SecureCsvWriter` write
through `TextWriter.Stream`, which wrote through a `PrintWriter`. A
`PrintWriter` never throws: it sets an error flag, which nothing read. A
write that failed, on a full disk, on an I/O error or on a closed
stream, was lost while `publishEvent()` reported success, the retry
never ran, and a tamper-evident writer still stored the next key after
the lost row, so the file failed verification and could not be resumed.

`TextWriter.Stream` now writes through a `BufferedWriter` over an
`OutputStreamWriter`, the two writers that the `PrintWriter` wrapped, so
the buffering and the charset are the same, and `write()` and `flush()`
throw the `IOException` of the stream with its message:

- the retry of the handler opens a new writer for the event, as it does
with rotation enabled, where `RotatableWriter` already threw;
- a tamper-evident writer stores the next key only after the row is
flushed, so it no longer stores the key of a row it could not write;
- with buffering enabled, `AsynchronousTextWriter` logs the failure,
which was silent before;
- a failure to close the stream is logged, as `RotatableWriter` does.

After a failure, the buffers of these writers are unusable: the encoder
of the `OutputStreamWriter` is left with its buffer flipped, so the next
flush writes nothing and returns, and a close writes a part of a failed
text longer than the buffers. Several publishers write to the same
`StandardCsvWriter` under the read lock, so the flush of the next one
would report success for an event that the failure dropped.
`TextWriter.Stream` therefore fails every write and flush after the
first failure, with that failure as the cause, and `shutdown()` then
closes the stream without writing what the buffers hold. A write or a
flush checks for a failure, and records its own, under the lock of the
writer: a flush that waits for a flush that fails would otherwise find
the buffers empty, and return. Every publisher whose event was in the
buffers gets the `IOException`, and the handler retries the event with a
new writer. A close that fails records its failure as well, and closes
the stream, which the close of the buffers leaves open on JDK 8 and 11,
and on older updates of 17 and 21. A caller that logs or throws only the
next failure, such as the flush in the `finally` block of
`SecureCsvWriter.writeSignature()`, still gives the first one, such as a
full disk.

`StandardCsvWriter` writes the header of a new file in its constructor,
which can now throw; it then closes the file, and the rotator thread
with rotation, as the caller gets no writer to close.

`write()` and `flush()` of `TextWriter.Stream` now declare
`IOException`. In this repository only the two CSV writers use the
class, through the `TextWriter` interface, which declared it already;
code that calls them on a `TextWriter.Stream` reference has to handle
it. `write(null)` throws a `NullPointerException`, where
`PrintWriter.print(null)` wrote `null`.

## `RotatableWriter` after a failure (#365)

`RotatableWriter`, which writes when rotation is enabled, had the same
buffers and the same flaw: with two publishers, the flush of the second
one returned after the flush of the first one failed; `close()`, which
the reset calls, wrote 8192 bytes of a failed event into the file; and a
rotation went on with a new writer, on which the flush of an event that
the failure dropped succeeded.

`TextWriter.Stream` and `RotatableWriter` now share `FailFastWriter`, a
package-private `Writer` that holds the logic above: it fails every
write and flush after the first write, flush or close that fails, with
that failure as the cause, and closes the stream without writing what
the buffers hold. `RotatableWriter` does not rotate a file after a
failure: the rotation checks log the failure, and `forceRotation()`
throws it, until the writer is replaced. It keeps the writer of the
failure until then, so its later flushes fail too.

`RotatableWriter` gets a package-private constructor that opens the file
through a `FileStreamOpener`, so that the tests can open a stream that
fails.

`RotationContext.getWriter()` now returns the `FailFastWriter`, which is
not a `BufferedWriter`. Its declared type is `Writer`, which the hooks
of this repository use; a `RotationHooks` that casts it to
`BufferedWriter` gets a `ClassCastException`.

## Buffered writes (#363)

With buffering enabled, `AsynchronousTextWriter` writes the records on
its own thread, and only logged a failure of the wrapped writer: the
handler never learned of it, never replaced the writer, and each record
logged a stack trace while the failure lasted.

`AsynchronousTextWriter` now keeps the first failure of the wrapped
writer, and fails the next `write()` and `flush()` with it as the cause,
so that the retry of the handler opens a new writer. It logs the first
failure with its stack trace, the next ones at debug level, and, at
shutdown, the number of records that could not be written. `flush()` now
throws the failure of the wrapped writer instead of logging it, and
declares `IOException`: code that calls it on an
`AsynchronousTextWriter` reference has to handle it. The first failure
is kept for the life of the writer, as the class Javadoc now states: the
caller replaces the writer after a failure, as the handler does. The
writers of this package that it wraps fail their later writes after a
failure too, so a writer that went on would not write again either.

`shutdown()` of a buffered handler now fails with an
`InternalServerErrorException` when the wrapped writer has failed
already, or fails to flush the records handed to it, as the handler does
without buffering when the flush fails. The records still queued are
written while the writer closes, after that flush: a failure there is
only logged, with their number. Every writer is still closed.

## Server errors (#364)

`CsvAuditEventHandler` reported an `IOException` of the audit file as a
`BadRequestException` (400): a failed write after the reset, a writer
that could not be opened, a file that could not be read or queried, and
a rotation that failed. These are failures of the server, which a client
may retry. They are now `InternalServerErrorException` (500), and
`queryEvents()` adapts its failure with
`ResourceExceptionsUtil.adapt()`, which keeps a `ResourceException` and
turns a `JsonValueException` into a `BadRequestException`. The `rotate`
action after `shutdown()` fails with a `ServiceUnavailableException`
(503), as a publish does. `BadRequestException` stays for an action
without a topic, an unknown action and an unknown topic. The `rotate`
action throws the exception of a rotation that does not happen after the
`try` of `forceRotation()`: a `ResourceException` is an `IOException`,
so its `catch` wrapped the exception into another one, whatever its
status.

## Failures after the event is written (#351)

The handler takes an `IOException` of a write for an event that is not
in the file: it closes the writer, opens a new one and writes the event
again. Two writers threw after the event was written:

- `SecureCsvWriter.writeEvent()` writes and flushes the row, then stores
the next HMAC key in the keystore of the file. When the key could not be
stored, the reset closed a writer whose file was intact, and the new
writer verified the file against the keystore, which still held the
previous key: it failed with "Final HMAC key doesn't match expected
value", and so did every later event of the topic, also once the
keystore could be written again. The writer now logs the failure and
goes on: the next row stores the key, and the event is reported as
published, as it is in the file.
- `RotatableWriter.write()` applies the rotation policies after it
writes the text, so a rotation that failed threw after the event was
written, and the handler wrote the event again to a new writer. With
tamper-evident logging, the rotation signs the file first, and a
keystore that cannot be stored failed it, so the reset broke the topic
as above. `write()` now logs the failure of the rotation, as the
scheduled checks do. A rotation that fails because the file cannot be
written, also when the file fails to close, leaves the writer failed
(#365): the flush that follows the write then fails, and the handler
writes the event again, which the failure dropped.

`CsvWriter.writeEvent()` now states that it does not throw once the row
is written, which the retry of the handler relies on.

## Tests

-
`CsvAuditEventHandlerTest.testActionToRotateFileKeepsEventBeingPublished`,
standard and tamper-evident: a publisher stops while it serializes a
field of its event, after it has the writer and before it writes to it,
and the `rotate` action runs meanwhile. Every event is then written
once, to the archive or to the new file, and with tamper-evident logging
both files pass verification with their own keystores. Without the fix,
the event being published is lost in both variants.
-
`CsvAuditEventHandlerTest.testActionToRotateFileLetsOtherTopicsPublish`:
an event of `activity` is written while the `rotate` action of `access`
waits for an event being written. With one lock for all topics, it times
out.
- `testActionToRotateFileCompressesOutsideHandlerMonitor` is renamed to
`testActionToRotateFileCompressesOutsideWriteLock`, as there is no
monitor any more.
- `CsvAuditEventHandlerTest.testPublishEventAfterShutdownFails`,
standard and tamper-evident, each with rotation disabled and enabled: an
event published after `shutdown()` fails with a
`ServiceUnavailableException` and is not in the file, and the
tamper-evident file passes verification. Without the fix, all four
variants report success.
- `CsvAuditEventHandlerTest.testActionToRotateFileAfterShutdownFails`,
with rotation enabled and disabled: the `rotate` action after
`shutdown()` fails and leaves the file as it is. Without the fix, it
rotates the file in both modes.
- `CsvAuditEventHandlerTest.testPublishEventAfterShutdownAndStartup`:
after `shutdown()` and `startup()`, a topic whose file could not be
opened at startup opens it at its next event.
- `TextWriterTest` (new, `audit/core`): a failure of the output stream
is thrown by `flush()` with its message; a write after `shutdown()`
throws; after a failed flush of two events, the next flush and write
throw too; the next failures have the first one as their cause; after a
failed write of a text longer than the buffers, `shutdown()` writes
nothing of it; a text written and flushed reaches the stream with its
byte count. Without the fix, all but the last fail: with the plain
`BufferedWriter`, the next flush returns, and `shutdown()` writes 8192
bytes of the failed text.
- `TextWriterTest.testStreamFailsTheFlushThatWaitsForAFailingFlush`: two
publishers flush at the same time, and the write of the first flush to
the stream fails while the second flush waits for it. Both flushes fail,
and nothing reaches the stream. Without the check under the lock, the
second flush returns.
- `TextWriterTest.testStreamFailsTheWriteThatWaitsForAFailingFlush`: a
publisher writes while the write of a flush to the stream fails. Both
fail, and nothing reaches the stream. Without the lock around the write,
the write returns, and puts its event into the buffers that the failure
left unusable.
- `TextWriterTest.testStreamClosesTheOutputStreamWhenTheShutdownFails`:
the close that writes the buffers fails, and the stream is closed.
Without the fix, the stream stays open on JDK 11. A probe of
`BufferedWriter.close()` over a stream that fails leaves the stream open
on 1.8.0_402 and 11.0.32 too, and closes it on 17.0.20, 21.0.12 and 27.
- `RotatableWriterTest.testFailsEveryWriteAndFlushAfterAFailure`,
`testDoesNotWriteTheTextOfAFailedWriteWhenClosed` and
`testDoesNotRotateAfterAFailure`, through a stream whose first write
fails. Without the fix, the second flush returns, `close()` writes 8192
bytes of the failed event, and `forceRotation()` rotates the file.
-
`RotatableWriterTest.testDoesNotRotateAfterTheRotationFailsToCloseTheFile`:
the close of a size rotation that `write()` triggers fails to write the
text, and the disk works again before the flush of the text.
`forceRotation()` and the flush fail, the file holds only the text
written before, and there is no archive. Without the fix,
`forceRotation()` rotates the file, and the flush succeeds on the new
file for the text that the failed close dropped.
- `AsynchronousTextWriterTest` (new, `audit/core`): after the wrapped
writer fails a write, a later `write()` fails with that failure as the
cause; `flush()` throws the failure of the wrapped writer, and the next
`write()` fails. Without the fix, no write fails, and `flush()` returns.
- `CsvAuditEventHandlerTest.testFailuresOfAuditFileAreServerErrors`:
with a directory in place of the audit file, a publish, a read, a query
and the `rotate` action fail with an `InternalServerErrorException`.
Without the fix, all four fail with a `BadRequestException`.
`testActionToRotateFileAfterShutdownFails` now expects a
`ServiceUnavailableException`, and
`testActionToRotateTamperEvidentFileKeepsFileWhenKeyStoreCannotBeRenamed`
an `InternalServerErrorException`.
-
`CsvAuditEventHandlerTest.testActionsThatTheRequestGetsWrongAreBadRequests`:
an action without a topic, an unknown action, and the `rotate` action of
an unknown topic fail with a `BadRequestException`.
-
`CsvAuditEventHandlerTest.testActionToRotateFileThatFailsIsServerError`:
with rotation enabled, the `rotate` action fails with an
`InternalServerErrorException` for a writer whose `forceRotation()`
throws an `IOException`, and for one whose `forceRotation()` returns
`false`, with the message of a rotation that does not happen. The
writers are put into the handler by reflection.
-
`CsvAuditEventHandlerTest.testPublishEventOpensWriterWhenTopicIsLeftWithoutOne`:
the writer of the topic removes itself from the handler, as a failed
reset of another publisher does, and fails the write. The event is
written once, to a new writer. Without the `newCsvWriter == null` check,
the publish fails with a `NullPointerException`.
-
`CsvAuditEventHandlerTest.testPublishEventDuringStartupResetsWriterOfReopenedTopic`:
after `shutdown()`, a writer is put into the handler by reflection, as
when `startup()` has reopened the topic and not returned yet, and fails
the write. The event is written once, to a new writer. Without the fix,
it fails with a `ServiceUnavailableException`.
- `CsvAuditEventHandlerTest.testStartupClosesTheWriterThatItReplaces`: a
writer put into the handler by reflection, as one that an event opened
after a `startup()` that threw, is closed by the next `startup()`.
Without the fix, `startup()` replaces it and leaves it open.
-
`CsvAuditEventHandlerTest.testRestartLetsOtherTopicsPublishWhenTamperEvidentKeyStoreCannotBeStored`:
after `shutdown()`, the file of the topic that `startup()` opens first
is deleted and its keystore made read-only, so the writer loads the
keystore but cannot store the first keys of the new file, and
`startup()` throws (#340). An event of the other topic is then
published. Without the fix, it fails with a
`ServiceUnavailableException`. The test fails if `startup()` does not
throw, so that it does not pass without the fix, and is skipped where
the keystore cannot be made read-only, as for root.
- `CsvWriterTest.shouldFailToWriteEventToClosedStandardCsvWriter` and
`shouldKeepSecureCsvFileVerifiableWhenWriteToClosedWriterFails`: an
event written to a closed writer fails with an `IOException`, and the
tamper-evident file still passes verification with its keystore. Without
the fix, both writes report success.
-
`CsvAuditEventHandlerTest.testPublishEventKeepsTamperEvidentWriterWhenKeyStoreCannotBeStored`,
with rotation enabled and disabled: a directory replaces the keystore
during one event, then the keystore is restored. Every event is
published and written once, and the file passes verification with its
keystore. Without the fix, that event fails with `RuntimeException:
Error when initializing a secure CSV writer` (#340), from the reset.
-
`CsvAuditEventHandlerTest.testPublishEventKeepsTamperEvidentWriterWhenRotationAfterItFails`:
with a size-based rotation, a directory replaces the keystore during the
event whose write rotates the file, so the signature of the rotation
fails. Every event is published and written once, to the archive that
the next event rotates, and the archive passes verification with its
keystore. Without the fix, the event fails the same way.
-
`RotatableWriterTest.testDoesNotFailTheWriteOfATextWhenTheRotationAfterItFails`:
a rotation hook fails at the write that follows a file of the maximum
size; the write and the flush return, and the text is in the file once.
Without the fix, the write throws the failure of the hook.

The default rotation under the read lock is caught by the first test,
and the compression under the write lock by the third. For #350, keeping
the closed writers in the map fails the last three tests, dropping the
check under the write lock fails the four variants of
`testPublishEventAfterShutdownFails`, and a `startup()` that does not
clear the mark fails the last test. For #349, storing the key before the
row is written fails
`shouldKeepSecureCsvFileVerifiableWhenWriteToClosedWriterFails` with
"Final HMAC key doesn't match expected value". For #351, throwing the
failure to store the key fails both variants of
`testPublishEventKeepsTamperEvidentWriterWhenKeyStoreCannotBeStored` and
`testPublishEventKeepsTamperEvidentWriterWhenRotationAfterItFails`, and
throwing the failure of the rotation from `write()` fails
`testPublishEventKeepsTamperEvidentWriterWhenRotationAfterItFails` and
`testDoesNotFailTheWriteOfATextWhenTheRotationAfterItFails`. A rotation
that logs the failed close of the file and goes on fails
`testDoesNotRotateAfterTheRotationFailsToCloseTheFile`. A
`BadRequestException` for the `IOException` of `forceRotation()`, or for
a rotation that does not happen, fails
`testActionToRotateFileThatFailsIsServerError`, and an
`InternalServerErrorException` for an unknown topic fails
`testActionsThatTheRequestGetsWrongAreBadRequests`. Throwing the
exception of a rotation that does not happen inside the `try` fails
`testActionToRotateFileThatFailsIsServerError` on its message. A retry
under the write lock that fails for any topic once the handler is shut
down fails `testPublishEventDuringStartupResetsWriterOfReopenedTopic`, a
`startup()` that does not close the writer it replaces fails
`testStartupClosesTheWriterThatItReplaces`, and a
`FailFastWriter.write()` outside the lock fails
`testStreamFailsTheWriteThatWaitsForAFailingFlush`. `audit/core`,
`audit/handler-csv`, `audit/handler-json` and `audit/handler-syslog`
pass locally on JDK 11 and 27, and the new tests of #334 passed 20 runs
in a row. One full run on JDK 11 failed
`CsvWriterTest.shouldCreateNewAuditFileAfterRotation`, on the archive of
an event that rotated the file after a periodic signature, which #342
fixes.

## Not covered by a test

- The `rotate` action with rotation enabled against a concurrent reset:
the reset needs a write that fails with an `IOException`, which a test
cannot cause with rotation enabled. The same lock covers that path, by
reasoning only.
- The retry of the handler after a write failure with rotation disabled
(#349): a test cannot make the file stream of a writer fail without a
full disk, so the writer tests close the stream instead, and
`TextWriterTest` uses a stream that fails.
- The close in the `StandardCsvWriter` constructor when the header
cannot be written, for the same reason.
- The retry of the handler after a failure with buffering enabled
(#363), for the same reason, and the log of `AsynchronousTextWriter`:
the tests of `audit/core` log through `slf4j-simple`, which a test
cannot read.
- The rotation checks after a failure (#365) run on a timer; they go
through the same `rotate()` as `forceRotation()`, which the test covers.

## Known limits

- A write that fails after part of its bytes reached the file leaves a
partial row, which the retry follows with the whole event.
- A tamper-evident writer computes the HMAC of a row, and moves to the
next key, before it writes the row. After a failed write, the reset
closes it: when a signature is due, the final signature covers the HMAC
of the row that is not in the file, and its write fails as well. The
file then ends with events that no signature covers, and the new writer
cannot resume it: the topic fails loudly, where it lost the row and
stored a key that broke the file silently before.
- With buffering enabled, the records queued before the failure shows
are reported as published, and are lost: their number is logged when the
writer is replaced. `shutdown()` does not fail for the records still
queued when it is called: their failure is only logged.
- Until the writer is replaced, the rotation checks log the failure at
each check interval.
- Until the keystore can be stored again, it holds the key of an earlier
row: the file fails verification meanwhile, and a handler stopped before
the next row cannot resume it. Each row logs the failure.
- A rotation that fails after it has renamed the file leaves the writer
half rotated. When the new file cannot be created or opened, the writer
keeps the closed one: the flush that follows the write fails, and the
handler writes the event again to a new file, while it is in the archive
already, as it did before. When a rotation hook fails after the rename,
the writer goes on with the new file, which may lack its header or, with
tamper-evident logging, the keys of its keystore; before, the write
threw, and the reset made the topic fail. The scheduled checks leave the
writer in the same states.

## Merging

- #348 makes `TextWriter.Stream` write in UTF-8. The two conflict in the
imports and the constructor of `TextWriter.Stream`; the merge is `new
FailFastWriter(stream, StandardCharsets.UTF_8)`.
- #341 and #353, merged meanwhile, add tests that expect a
`BadRequestException` for a topic whose file cannot be opened or resumed
(`testStartupOpensOtherTopicsWhenTamperEvidentFileCannotBeResumed`,
`testStartupWithAuditFileOfMoreColumnsThanTopicSchema`). With #364 the
handler answers an `InternalServerErrorException` there, and since the
rebase onto them they expect it. With #341, a file that cannot be
resumed fails with an `IOException`, and `startup()` opens the other
topics, so the restart test now fails the writer with a keystore that
cannot be stored, which is still a `RuntimeException`
(`testRestartLetsOtherTopicsPublishWhenTamperEvidentKeyStoreCannotBeStored`).
- #342 changes the condition that schedules a signature in
`SecureCsvWriter.writeEvent()`, two lines below the key store that this
PR stops throwing for. The two merge without a conflict, and so do this
PR and #355.

## Not in this PR

Found in the review of this change, older than it, and filed separately:

- #340, fixed by #341: a `RuntimeException` from `createCsvWriter()`
escapes `startup()`, `publishEvent()` and `handleAction()`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug concurrency Race conditions, locking and thread-safety data-loss Data lost or irrecoverably overwritten java Pull requests that update java code security Security fixes and CVE remediation tests Test code changes

Projects

None yet

2 participants