Repository navigation
[#339] Resume a tamper-evident CSV file with no event since the last rotation - #342
Conversation
d4688ce to
529de3a
Compare
maximthomas
left a comment
There was a problem hiding this comment.
praise: The close is placed exactly where the leak was, and the tests measure it.
- The
SecureCsvWriterconstructor'scatchnow closescsvWriter, which stops theRotatableWriterrotator. Removing that close turns all five rows ofshouldStopRotatorAndFailWithIOExceptionWhenFileCannotBeResumedred ([pool-N-thread-1 alive]). Measured on JDK 26. - Restoring the
RuntimeExceptionwrapping turnstestStartupOpensOtherTopicsWhenTamperEvidentFileCannotBeResumedred. Measured on JDK 26. readInitialKey() == nullis checked beforeverify(), so a missing keystore fails with a message that names the keystore, not with anIllegalStateExceptionfrom the verifier.
issue (blocking): A resumed file with a row cut short, a wrong column count or an unclosed quote still fails the whole startup, not only its topic.
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:150, :201-205; commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/CsvSecureVerifier.java:97, :108
CsvSecureVerifier.verify() catches only SecureStorageException. super-csv's SuperCsvException from csvReader.read(header) therefore reaches the constructor's catch, which wraps it in RuntimeException. A probe against super-csv 2.4.0 got "the nameMapping array and the sourceList should be the same size" for a short row and "unexpected end of file while reading quoted column" for an unclosed quote. CsvAuditEventHandler.startup() catches only IOException, so the loop over the topics ends and AuditServiceImpl.startup() leaves the service in STARTING. A crash or a full disk during a write leaves the last row cut short, which is a common way for a file to become unresumable. The description says such a file "now always fails" with an IOException. A header line with one column fails on the same path: header[header.length - 2] throws ArrayIndexOutOfBoundsException.
// SecureCsvWriter.java:149-150 — import org.supercsv.exception.SuperCsvException;
CsvSecureVerifier verifier = new CsvSecureVerifier(csvFile, csvPreference, secureStorage);
final CsvSecureVerifier.VerificationResult verificationResult;
try {
verificationResult = verifier.verify();
} catch (SuperCsvException e) {
// A row cut short by a crash while it was written, or a row with a missing or extra column.
throw new IOException("Resuming an existing CSV file but it cannot be parsed: " + csvFile, e);
}
// CsvSecureVerifier.java:97 — a header with fewer than two columns fails verification
if (header.length < 2 || !(HEADER_HMAC.equals(header[header.length - 2])
&& HEADER_SIGNATURE.equals(header[header.length - 1]))) {Pin: add an unresumableFiles row that appends a line with fewer columns than the header to a written file. At this head it fails with the RuntimeException. With the fix it fails with the IOException.
question (non-blocking): Should an unreadable .keystore next to a new file fail the whole startup, as the description says, or only its topic, as the code now does?
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:118, :201-203
JcaKeyStoreHandler.init() opens an existing keystore path with KeyStoreBuilder.withKeyStoreFile(). For a mode-000 file or a directory, its new FileInputStream throws FileNotFoundException before anything is wrapped in IllegalStateException. The inner catch takes only IllegalStateException, so this IOException now passes through for a new file too, and startup() logs it for the topic. At base it was wrapped, and the startup failed. The description says a new file next to a keystore that cannot be loaded "still fails as before". Either answer is minor. If per-topic is intended, only the sentence changes. If "as before" is intended, the inner catch also takes IOException and wraps it when !fileAlreadyInitialized.
Pin: a 0-byte CSV next to a directory named <csv>.keystore, asserting the outcome you choose.
question (non-blocking): Is it intended that a topic that never got an event still fails the whole startup after a Password change?
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:123-125
A topic's keystore is stored when its writer is constructed, but its header is written only with the first event. An idle topic therefore leaves a 0-byte CSV next to a keystore. After the Password key changes, that keystore no longer loads. fileAlreadyInitialized is false, so the IllegalStateException is rethrown and wrapped, and the startup fails, the same as at base. The description says "a topic without a file still starts with a new keystore", which holds only when no keystore is left behind. If this is intended, the sentence needs "and no keystore". If not, dropping the rethrow at :123-125 makes such a topic fail on its own.
issue (non-blocking): The rotator can rotate a resumed file while the file is still being verified. This is pre-existing and not changed by this PR.
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:104, :239-240, :443-451
constructWriter() builds the RotatableWriter and registers the rotation hooks before the keys are read and verify() runs. The first rotation check runs one check interval later (default 5s). Take a resumed file that is already over maxFileSize or maxFileLifetime and takes longer than that interval to verify. It is renamed while it is being verified, and postRotationAction renames .keystore and swaps in random keys. verify() then reports "Final HMAC key doesn't match" for a valid file that has already been archived. This was traced in the code, not run. Suggested follow-up issue: start the rotation checks only once construction has succeeded.
nitpick (non-blocking): The IOException arm of the new close catch can never fire.
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:198
TextWriterAdapter.close() calls shutdown(). With rotation, RotatableWriter.shutdown() logs the IOException from its close(). Without rotation, TextWriter.Stream closes a PrintWriter, which swallows the exception. A failed close is therefore logged and not added as suppressed; only a RuntimeException can be. The catch has to stay because the compiler requires it. The description's "a failure of that close is added to the original one as suppressed" could say "a RuntimeException from that close".
… yet (#354) Fixes #346. `SecureCsvWriter` writes the CSV header with the first event, so the audit file of a tamper-evident topic is empty after startup, and after the `rotate` action of a topic without file rotation, until an event is published. `CsvAuditEventHandler.getEntries()` passed the `null` header of such a file to `convertDotNotationToSlashes()`: - `readEvent()` threw the `NullPointerException` out of the call instead of answering a promise; - `queryEvents()` answered `BadRequestException`. `getEntries()` now returns no entry for a file without a header: `readEvent()` answers `NotFoundException`, and `queryEvents()` answers an empty result. This also covers the empty file that a rotation before the first signature leaves with #342. ## Tests `CsvAuditEventHandlerTest.testReadingTamperEvidentAuditLogWithNoEventYet` and `testQueryOnTamperEvidentAuditLogWithNoEventYet`, after startup and after the `rotate` action. Each case first checks that the audit file is empty, and fails without the fix as the issue describes. `audit/core` and `audit/handler-csv` pass locally on JDK 11 and 27. The change applies cleanly on #341, #342, #347 and #348.
529de3a to
63e4495
Compare
|
@maximthomas, this review seems to have been posted to #342 by mistake. Its text is identical to round 1 of #341 (#341 (review)), which you posted 55 minutes earlier, and it reviews code that only #341 contains: the close of Every point is answered in #341 (#341 (comment)), and you approved its round 2:
The branch is now rebased onto Could you dismiss this review and review the changes of #342 itself? |
maximthomas
left a comment
There was a problem hiding this comment.
Yes, the review above was posted by mistake, sorry.
Here is the right review:
praise: The signing model now follows what was written, not the state of the scheduled task.
unsignedEventsletsforceWriteSignaturesign on every rotation and onclose()whatever the scheduledFuturedid, which removes #372's unsigned last event in an archive.holdsOnlyTheSignaturecloses the unchecked data and HMAC columns of signature rows.shouldRotateCsvAndKeyStoreFilenow verifies every archive and the live file with both the plain and the.gzsuffix, in place of asserting the CAUD-225 failure. An empty.gzarchive is a valid 20-byte gzip that the verifier reads to EOF, so it also takes the empty-file path.
issue (blocking): When the header write triggers a rotation, the archive gets only the header and the first event lands in the new live file without a header.
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:338-340, :269-273, :439-443
writeEvent writes the header on its own first, and RotatableWriter.write rotates inline after each write. Because the event has no HMAC yet, preRotationAction has nothing to sign and lastSignature stays null. postRotationAction then takes the new branch and sets headerWritten = false (:442), and writeHeader overwrites that with true when it returns (:273). This happens with a time-based policy when no event has been signed since startup. The most likely case is a restart over the empty live file that this branch leaves behind: the old mtime becomes lastRotationTime, and the first event comes before the rotator's first check. The archive then fails with Missing final signature, the live file fails with Found only 0 checked headers, and the next start reports the file as tampered. The pin below is red at 63e4495 (Missing final signature on the archive). It is green with the change below, and SecureCsvWriterTest passes 15/15 (JDK 26, macOS). The flag has to be set before the write. If it is set after the combined write, the signature that the rotation writes re-enters writeEvent and prepends a second header: The signature row 3 holds other values.
// SecureCsvWriter.writeEvent(Writer, Map): delete `if (!headerWritten) { writeHeader(headers); }` (:338-340),
// and in place of `writer.write(csvFormatter.formatEvent(extendedValues, extendedHeaders));` (:348):
String row = csvFormatter.formatEvent(extendedValues, extendedHeaders);
if (!headerWritten) {
// The header and the first event in one write, so that a rotation this write triggers archives
// both and signs the event; set first, as the signature that rotation writes comes from inside it.
row = csvFormatter.formatHeader(extendedHeaders) + row;
headerWritten = true;
}
writer.write(row);postRotationAction's writeHeader and writeLastSignature (:446-448) can stay as they are, because they run with headerWritten already true. The headerWritten = false at :442 can stay as a guard.
Pin: an empty live file dated a day back, with a 1-hour rotation interval; its first event must leave one archive and a live file that both pass.
@Test
public void shouldSignTheFirstEventWhenItsWriteRotatesTheFile() throws Exception {
final File actual = newSecureCsvFile();
// an empty live file last rotated a day ago: with a 1 hour interval, the first write rotates it
assertThat(actual.createNewFile()).isTrue();
assertThat(actual.setLastModified(System.currentTimeMillis() - TimeUnit.DAYS.toMillis(1))).isTrue();
final String header = "FOO";
final CsvAuditEventHandlerConfiguration config = createRotatingSecureConfig(0);
config.getFileRotation().setRotationInterval("1 hour");
try (SecureCsvWriter secureCsvWriter = new SecureCsvWriter(
actual, new String[]{header}, CsvPreference.EXCEL_PREFERENCE, config, keyStoreHandler, random)) {
secureCsvWriter.writeEvent(singletonMap(header, "one"));
}
final List<File> archives = new TimeStampFileNamingPolicy(actual, "-yyyy.MM.dd-HH.mm.ss.SSS", "").listFiles();
assertThat(archives).hasSize(1);
assertPassesVerification(archives.get(0));
assertPassesVerification(actual);
}suggestion (non-blocking): No test checks that unsignedEvents is cleared before the signature row is written.
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:286-291
This order is what keeps the signature row's own writeEvent from scheduling a task under the new condition at :354. Move :288 after :291 and close() changes: it cancels the scheduled task, its signature row then schedules a new one, and awaitTermination waits for it. shouldGenerateHMACColumn still passes, but takes 300.8 s instead of under a second, and every other test stays green. No test sets a time bound that would catch this.
@Test(timeOut = 60_000)
public void shouldGenerateHMACColumn() throws Exception {Pin: with this timeOut, the test is red under that mutant (close() takes 300.8 s) and green at the head, where the whole class runs in about 26 s. I did not run it with the timeOut added.
issue (non-blocking): If an event is written after the periodic signature task releases the lock but before it completes, no new signature is scheduled for that event.
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:354-362, :300-303
writeSignature unlocks (:301) and flushes (:302) before the task's run() returns, and FutureTask becomes done only after that. An event written in this window sees !scheduledSignature.isDone() and schedules nothing. On a quiet topic it stays unsigned until the next event, a rotation, or close(). If the process is killed in that state, the live file ends in an unsigned row. The race already exists on master with the same isDone() test. This PR's close() now always signs, which limits the problem to shutdowns that are not clean. A follow-up issue is enough. Sketch, not run:
private boolean signatureScheduled; // guarded by signatureLock
// writeSignature, first statement inside the lock, before the `if (!unsignedEvents)` guard:
signatureScheduled = false;
// writeEvent, in place of :354; set only once schedule() returned, so a RejectedExecutionException leaves it false:
if (unsignedEvents && !signatureScheduled) {
logger.trace("Triggering a new signature task to be executed in {}", signatureInterval);
try {
scheduledSignature = scheduler.schedule(signatureTask, signatureInterval.getValue(),
signatureInterval.getUnit());
signatureScheduled = true;
} catch (RejectedExecutionException e) {
logger.error(e.getMessage(), e);
}
}suggestion (non-blocking): With rotation enabled, calling close() a second time now throws IOException("Stream closed").
commons/audit/handler-csv/src/main/java/org/forgerock/audit/handlers/csv/SecureCsvWriter.java:256-263, :300-303
forceWriteSignature now always calls writeSignature. When there is nothing to sign, it returns at :282, but its finally still calls flush(), which reaches the BufferedWriter that the first close() already closed. Master returned normally here. No caller closes twice: CsvAuditEventHandler.closeWriter removes the writer before it closes it. A guard at the top of close() makes it idempotent:
private boolean closed;
// close(), first statements:
if (closed) {
return;
}
closed = true;|
@maximthomas, the round is in 6b4ab04:
|
…vent since the last rotation CsvSecureVerifier now counts the signature chained into row 2 of a rotated file as its final signature until an event follows, so a live file that holds only the header and that signature passes verification, and SecureCsvWriter can resume it after a restart. A file truncated to these two rows after later events still fails, as the current key in its keystore no longer matches. SecureCsvWriter now tracks whether events were written since the last signature, and signs only then: - a rotation signs them whether or not the scheduled signature could be cancelled: an event that rotated the file right after a periodic signature was archived unsigned; - no signature is written or scheduled with no event to cover: after a rotation on write, close() wrote one into the new file over the last HMAC of the previous file, which failed verification at row 3 (CAUD-225 in shouldRotateCsvAndKeyStoreFile); - a rotation before any signature leaves the new file empty instead of writing an empty chained signature, which the verifier could not read. CsvSecureVerifier also accepts an empty file whose keys match, as such a rotation archives one, rejects a row 2 signature that is not base64 and a signature row that holds other values, and reports a row with neither an HMAC nor a signature instead of throwing a NullPointerException.
…ile with its first event Written on its own, the header could rotate the file when a time-based rotation was due, as after a restart over an empty live file last modified before the rotation interval. That rotation had no event to sign: the archive held only the header and failed with "Missing final signature", and the event went into the new file without a header. The header and the first event are now written together, so that such a rotation archives both and signs the event. headerWritten is set before the write, as the signature that the rotation writes comes from inside it. A second close() does nothing: with rotation, it flushed the closed writer and threw "Stream closed". shouldGenerateHMACColumn has a time-out, which fails a close() that waits for a signature scheduled by its own signature row.
6b4ab04 to
206d27e
Compare
|
@maximthomas, rebased onto the current The only conflict was in
|
…itten while the periodic signature completes SecureCsvWriter scheduled a signature only when the scheduled one was done. The periodic task releases the signature lock and flushes the writer before its FutureTask is done, so an event written in this window scheduled none, and stayed unsigned until the next event, a rotation or close(). The writer now keeps the scheduled task until it starts signing: writeEvent() schedules a signature for unsigned events when no task is pending, and a task clears the pending one once it holds the lock, only if it is that task. A successful cancel(false) clears it too, but does not stop a task that has started and waits for the lock: that task then leaves alone the one an event scheduled since. Otherwise two tasks stay scheduled, and close() waits for the one it does not cancel, up to the signature interval.
|
@maximthomas, #373, the non-blocking issue of your review (an event written while the periodic signature completes), is fixed in this PR now: 00f0a35, on top of 206d27e. A correction to round 3 and to the note in #373: So the writer keeps the pending task itself, in
Three tests, each red with its part of the fix disabled:
|
maximthomas
left a comment
There was a problem hiding this comment.
praise: Each fix in this round sits where its round-2 finding was, and each has a test that pins it.
SecureCsvWriter.writeEventwrites the header and the first event as one string (SecureCsvWriter.java:389-396).RotatableWriter.writerotates only after the write, so a rotation this write triggers archives both rows, andpreRotationActionsigns the event.shouldSignTheFirstEventWhenItsWriteRotatesTheFilecovers this.SignatureTask.runclearspendingSignatureundersignatureLockbefore it callswriteSignature(SecureCsvWriter.java:515-518). Both mutants were run locally on JDK 26. With the clear moved afterwriteSignature,shouldSignAnEventWrittenWhileThePeriodicSignatureCompletesfails. With thependingSignature == thisguard dropped,shouldScheduleNoOtherSignatureWhenARotationSignsWhileTheScheduledOneWaitsForTheLockfails:close()took 2694 ms against its 1.5 s bound.close()returns early on a second call (SecureCsvWriter.java:272-276), pinned byshouldDoNothingWhenClosedTwice. The five new cases pass on all 9 CI matrix rows, where handler-csv runs 90 tests at this head against 78 at the base.
…the resumed file is verified (#355) Fixes #343. ## Construction order `SecureCsvWriter` constructed its `RotatableWriter` first, and the `RotatableWriter` constructor schedules the rotation and retention checks. Only then did the constructor read the keys and verify the file it resumes. A check due on the resumed file, by `maxFileSize` or by `rotationInterval` from its `lastModified()`, could run during the construction: it renamed the file and its keystore, and `postRotationAction` wrote new keys into a new keystore, through the `secureStorage` that the verifier reads. A valid file, already archived, then failed verification, and the topic did not start. The writer is now constructed last, once the keys are read and the file it resumes is verified. The first check runs one check interval after the construction, on a writer that holds the keys and the last signature of the file. ## A file that cannot be opened The file is still opened, and closed at once, before the keys. A file that cannot be opened, as one in a directory that cannot be created, fails with an `IOException`, which `CsvAuditEventHandler.startup()` logs for the topic. Without that, a new file would fail first when its keystore is stored, with a `SecureStorageException` that the constructor wraps in a `RuntimeException`, and the startup would fail for every topic. Opening for append leaves the file and its `lastModified()` as they are. ## A writer whose check interval is invalid Found in the review. The `RotatableWriter` constructor opened its file before it rejected an unlimited or zero rotation and retention check interval, and the caller of a constructor that throws gets no writer to close: the file stayed open until its stream was collected. An interval under one millisecond passed that check, and failed after the open with the `IllegalArgumentException` of `scheduleAtFixedRate`, which `CsvAuditEventHandler.startup()` does not catch for the topic. The constructor now builds the policies and checks the interval before it opens the file, and rejects an interval under one millisecond with the same `IOException`. That includes a negative one: `Duration` adds the parts of a composite without an overflow check, and `9223372036854775807 ms, 1 ms` parses to a negative interval. ## Tests - `SecureCsvWriterTest.shouldNotRotateResumedFileWhileVerifyingIt`: a written file is resumed with rotation and a 50 ms check interval, with a rotation due at the first check: its size at `maxFileSize`, or its `lastModified()` two hours back with a 1 hour `rotationInterval`. A `CommentMatcher` in the `CsvPreference` holds the verification for 500 ms on the first line. The construction succeeds, and the first check after it rotates the file. The archive passes verification, and so does the live file, with one event after the rotation. On Windows a file that the verifier holds open cannot be renamed, and a rotation that fails to rename it still resets the time of the last rotation: there, only the `rotationInterval` row is sure to fail with the writer constructed first. - `SecureCsvWriterTest.shouldFailWithIOExceptionWhenFileCannotBeOpened`: a file under a regular file fails with an `IOException`. - `SecureCsvWriterTest.shouldFailWithIOExceptionWhenCheckIntervalIsInvalidForNewFile`: a new file with rotation and an `unlimited` check interval fails with the `IOException` of the `RotatableWriter`. - `RotatableWriterTest.testRejectsInvalidRotationRetentionCheckIntervalBeforeOpeningTheFile`: an `unlimited`, `zero`, `500 microseconds` or overflowed `9223372036854775807 ms, 1 ms` check interval fails with an `IOException`, and the file is not created. `testRejectsInvalidRotationRetentionCheckIntervalWithRetentionOnlyBeforeOpeningTheFile` does the same with a retention policy alone. - `RotatableWriterTest.testAcceptsAnyRotationRetentionCheckIntervalWithoutPolicy`: with no policy, no check is scheduled, and the same intervals are accepted. `testAcceptsOneMillisecondRotationRetentionCheckInterval`: one millisecond is accepted. Measured on JDK 27: - With the `SecureCsvWriter` of `master`, both rows fail with "The CSV file was tampered: The file contains no CSV header": with a 50 ms interval, the first check runs while the keys are read, and renames the file before the verifier opens it. With a 1 s interval and a 3 s hold, both rows fail with the reason given in the issue, "Final HMAC key doesn't match expected value". - Without the open and close before the keys, `shouldFailWithIOExceptionWhenFileCannotBeOpened` fails with the `RuntimeException`, and so does `CsvAuditEventHandlerTest.canConfigureCsvHandlerFromJsonAndRegisterWithAuditService`, whose log directory `/path/to/audit/files/` cannot be created. - With the writer constructed in the `try` block, `shouldFailWithIOExceptionWhenCheckIntervalIsInvalidForNewFile` fails with "RuntimeException: Error when initializing a secure CSV writer". - With the `RotatableWriter` of `master`, both `testRejectsInvalidRotationRetentionCheckInterval…` tests fail in their four rows: the file exists for `unlimited` and `zero`, and `500 microseconds` and the overflowed interval throw an `IllegalArgumentException`. With a guard that rejects only zero milliseconds, the overflowed row fails in both with the `IllegalArgumentException`. - Three mutants of the guard each fail only their own test: without the retention clause, the retention-only test in its four rows; without the policy clause, `testAcceptsAnyRotationRetentionCheckIntervalWithoutPolicy` in its four rows; with `<= 1`, `testAcceptsOneMillisecondRotationRetentionCheckInterval`. - `audit/core` and `audit/handler-csv` pass locally, and the new tests passed 3 runs in a row. ## Merging - With #341: a conflict in the constructor. A failure in its `try` block now comes before the writer is constructed, so no file or rotator thread is left open (#340), and the close that #341 adds in the `catch` block would find `csvWriter` still `null`: the close goes, and the rest of #341 applies as is. Its tests that the rotator thread terminates still hold, as the rotator does not start. - With #342, #347 and #369: no conflict. ## Not in this PR - `RotatableWriter` schedules its checks in its constructor, and `registerRotationHooks()` sets the hooks after it, in a field that is not volatile. The first check runs one check interval later. Not observed; it follows from the memory model only. - A new secure topic whose check interval is invalid stores new keys for its empty file on each publish attempt, and a resumed one verifies its file, before the writer fails. `master` leaked a `FileOutputStream` on each attempt instead.
#347) Fixes #334. Fixes #350. Fixes #349. Fixes #365. Fixes #363. Fixes #364. Fixes #351. ## The race `CsvAuditEventHandler` looked up the writer of a topic without a lock; only the reset after a failed write and the rotations took the handler monitor. A publisher that had the writer when the `rotate` action closed it wrote into the closed writer. With rotation and buffering disabled, `TextWriter.Stream` wrote through a `PrintWriter`, which swallows the `IOException`, so the retry never ran: the event was lost and `publishEvent()` reported success. A stale `SecureCsvWriter` also stored its keystore over the one of the new file. With rotation enabled ([comment](#334 (comment))), the `rotate` action looked up the writer without the monitor too, and could call `forceRotation()` on a writer that a concurrent reset had closed. ## The fix The issue offers two ways out: make a write to a closed writer fail, so that the retry opens the current writer, or make `rotate` wait for the publishers that are already writing. This PR does the latter. A closed writer that throws would not cover the path with rotation enabled, and would leave the gap between the final signature and the close of the stream in `SecureCsvWriter.close()` that #337 left to this issue. A write to a closed writer now fails as well (#349, below), but only the lock covers both paths. A `ReentrantReadWriteLock` per topic guards the writer of the topic, and replaces the handler monitor: - a publisher looks up the writer and writes the event under the read lock. If the topic has no writer, or the write fails, it retries under the write lock, with a new writer unless another thread has replaced the writer meanwhile; - the `rotate` action with rotation enabled holds the read lock during `forceRotation()`: publishers keep writing, and a reset cannot close the writer meanwhile; - the default rotation, `startup()` and `shutdown()` take the write lock, so they wait for the events being written. The rotated file is still compressed after the lock is released (#332). With a lock per topic, the other topics keep publishing while the writer of one topic is replaced. With a single lock they would not: new readers of a `ReentrantReadWriteLock` also queue behind a waiting writer, so one event being written plus a waiting rotation would stop every topic. The retry now also opens a writer when a failed reset of another thread has left the topic without one, where it threw a `NullPointerException` before. ## Events after `shutdown()` (#350) `shutdown()` closed the writers but left them in the map, and nothing marked the handler as shut down. An event published afterwards went to the closed writer of its topic. With rotation disabled, it was lost while `publishEvent()` reported success, and a tamper-evident writer still stored its next key, so the file failed verification and could not be resumed. With rotation enabled, the closed `RotatableWriter` threw, and the retry opened a new writer that nothing closed. The `rotate` action after `shutdown()` also rotated the file and opened a new writer, in both modes. `shutdown()` now marks the handler as shut down before it takes the first write lock, and removes each writer it closes from the map. A topic without a writer then gets no new one: - the retry under the write lock fails with a `ServiceUnavailableException`, so the event is reported as lost, and is written neither to a closed writer nor to a new one. A topic that has a writer still resets it after a failed write, as `startup()` may have reopened it already; - the `rotate` action finds no writer, and fails with a `ServiceUnavailableException`, as a publish does (#364, below). An event that has the read lock of its topic when `shutdown()` gets there is still written, and `shutdown()` waits for it. `startup()` clears the mark once it has opened the writers, also when a writer throws a `RuntimeException` (#340), so that a restarted handler opens a writer again, at the next event, for a topic whose file it could not open. Until `startup()` returns, an event or a `rotate` action on a topic that it has not opened yet fails with a `ServiceUnavailableException`, as the Javadoc of `startup()` now states. Before, such an event went to the closed writer of its topic, as above, and with rotation enabled `startup()` then replaced the writer that the retry had opened, without closing it. `startup()` now closes a writer that an event opened before it, or after a `startup()` that threw, before it opens a new one. The retry under the write lock also rethrows a `ResourceException` as is. `ResourceException` is an `IOException`, so the `ServiceUnavailableException` would have become an `InternalServerErrorException`, and the `InternalServerErrorException` of a failed reset would have been wrapped into another one, since the reset moved into that `try`. ## Write failures (#349) With rotation disabled, `StandardCsvWriter` and `SecureCsvWriter` write through `TextWriter.Stream`, which wrote through a `PrintWriter`. A `PrintWriter` never throws: it sets an error flag, which nothing read. A write that failed, on a full disk, on an I/O error or on a closed stream, was lost while `publishEvent()` reported success, the retry never ran, and a tamper-evident writer still stored the next key after the lost row, so the file failed verification and could not be resumed. `TextWriter.Stream` now writes through a `BufferedWriter` over an `OutputStreamWriter`, the two writers that the `PrintWriter` wrapped, so the buffering and the charset are the same, and `write()` and `flush()` throw the `IOException` of the stream with its message: - the retry of the handler opens a new writer for the event, as it does with rotation enabled, where `RotatableWriter` already threw; - a tamper-evident writer stores the next key only after the row is flushed, so it no longer stores the key of a row it could not write; - with buffering enabled, `AsynchronousTextWriter` logs the failure, which was silent before; - a failure to close the stream is logged, as `RotatableWriter` does. After a failure, the buffers of these writers are unusable: the encoder of the `OutputStreamWriter` is left with its buffer flipped, so the next flush writes nothing and returns, and a close writes a part of a failed text longer than the buffers. Several publishers write to the same `StandardCsvWriter` under the read lock, so the flush of the next one would report success for an event that the failure dropped. `TextWriter.Stream` therefore fails every write and flush after the first failure, with that failure as the cause, and `shutdown()` then closes the stream without writing what the buffers hold. A write or a flush checks for a failure, and records its own, under the lock of the writer: a flush that waits for a flush that fails would otherwise find the buffers empty, and return. Every publisher whose event was in the buffers gets the `IOException`, and the handler retries the event with a new writer. A close that fails records its failure as well, and closes the stream, which the close of the buffers leaves open on JDK 8 and 11, and on older updates of 17 and 21. A caller that logs or throws only the next failure, such as the flush in the `finally` block of `SecureCsvWriter.writeSignature()`, still gives the first one, such as a full disk. `StandardCsvWriter` writes the header of a new file in its constructor, which can now throw; it then closes the file, and the rotator thread with rotation, as the caller gets no writer to close. `write()` and `flush()` of `TextWriter.Stream` now declare `IOException`. In this repository only the two CSV writers use the class, through the `TextWriter` interface, which declared it already; code that calls them on a `TextWriter.Stream` reference has to handle it. `write(null)` throws a `NullPointerException`, where `PrintWriter.print(null)` wrote `null`. ## `RotatableWriter` after a failure (#365) `RotatableWriter`, which writes when rotation is enabled, had the same buffers and the same flaw: with two publishers, the flush of the second one returned after the flush of the first one failed; `close()`, which the reset calls, wrote 8192 bytes of a failed event into the file; and a rotation went on with a new writer, on which the flush of an event that the failure dropped succeeded. `TextWriter.Stream` and `RotatableWriter` now share `FailFastWriter`, a package-private `Writer` that holds the logic above: it fails every write and flush after the first write, flush or close that fails, with that failure as the cause, and closes the stream without writing what the buffers hold. `RotatableWriter` does not rotate a file after a failure: the rotation checks log the failure, and `forceRotation()` throws it, until the writer is replaced. It keeps the writer of the failure until then, so its later flushes fail too. `RotatableWriter` gets a package-private constructor that opens the file through a `FileStreamOpener`, so that the tests can open a stream that fails. `RotationContext.getWriter()` now returns the `FailFastWriter`, which is not a `BufferedWriter`. Its declared type is `Writer`, which the hooks of this repository use; a `RotationHooks` that casts it to `BufferedWriter` gets a `ClassCastException`. ## Buffered writes (#363) With buffering enabled, `AsynchronousTextWriter` writes the records on its own thread, and only logged a failure of the wrapped writer: the handler never learned of it, never replaced the writer, and each record logged a stack trace while the failure lasted. `AsynchronousTextWriter` now keeps the first failure of the wrapped writer, and fails the next `write()` and `flush()` with it as the cause, so that the retry of the handler opens a new writer. It logs the first failure with its stack trace, the next ones at debug level, and, at shutdown, the number of records that could not be written. `flush()` now throws the failure of the wrapped writer instead of logging it, and declares `IOException`: code that calls it on an `AsynchronousTextWriter` reference has to handle it. The first failure is kept for the life of the writer, as the class Javadoc now states: the caller replaces the writer after a failure, as the handler does. The writers of this package that it wraps fail their later writes after a failure too, so a writer that went on would not write again either. `shutdown()` of a buffered handler now fails with an `InternalServerErrorException` when the wrapped writer has failed already, or fails to flush the records handed to it, as the handler does without buffering when the flush fails. The records still queued are written while the writer closes, after that flush: a failure there is only logged, with their number. Every writer is still closed. ## Server errors (#364) `CsvAuditEventHandler` reported an `IOException` of the audit file as a `BadRequestException` (400): a failed write after the reset, a writer that could not be opened, a file that could not be read or queried, and a rotation that failed. These are failures of the server, which a client may retry. They are now `InternalServerErrorException` (500), and `queryEvents()` adapts its failure with `ResourceExceptionsUtil.adapt()`, which keeps a `ResourceException` and turns a `JsonValueException` into a `BadRequestException`. The `rotate` action after `shutdown()` fails with a `ServiceUnavailableException` (503), as a publish does. `BadRequestException` stays for an action without a topic, an unknown action and an unknown topic. The `rotate` action throws the exception of a rotation that does not happen after the `try` of `forceRotation()`: a `ResourceException` is an `IOException`, so its `catch` wrapped the exception into another one, whatever its status. ## Failures after the event is written (#351) The handler takes an `IOException` of a write for an event that is not in the file: it closes the writer, opens a new one and writes the event again. Two writers threw after the event was written: - `SecureCsvWriter.writeEvent()` writes and flushes the row, then stores the next HMAC key in the keystore of the file. When the key could not be stored, the reset closed a writer whose file was intact, and the new writer verified the file against the keystore, which still held the previous key: it failed with "Final HMAC key doesn't match expected value", and so did every later event of the topic, also once the keystore could be written again. The writer now logs the failure and goes on: the next row stores the key, and the event is reported as published, as it is in the file. - `RotatableWriter.write()` applies the rotation policies after it writes the text, so a rotation that failed threw after the event was written, and the handler wrote the event again to a new writer. With tamper-evident logging, the rotation signs the file first, and a keystore that cannot be stored failed it, so the reset broke the topic as above. `write()` now logs the failure of the rotation, as the scheduled checks do. A rotation that fails because the file cannot be written, also when the file fails to close, leaves the writer failed (#365): the flush that follows the write then fails, and the handler writes the event again, which the failure dropped. `CsvWriter.writeEvent()` now states that it does not throw once the row is written, which the retry of the handler relies on. ## Tests - `CsvAuditEventHandlerTest.testActionToRotateFileKeepsEventBeingPublished`, standard and tamper-evident: a publisher stops while it serializes a field of its event, after it has the writer and before it writes to it, and the `rotate` action runs meanwhile. Every event is then written once, to the archive or to the new file, and with tamper-evident logging both files pass verification with their own keystores. Without the fix, the event being published is lost in both variants. - `CsvAuditEventHandlerTest.testActionToRotateFileLetsOtherTopicsPublish`: an event of `activity` is written while the `rotate` action of `access` waits for an event being written. With one lock for all topics, it times out. - `testActionToRotateFileCompressesOutsideHandlerMonitor` is renamed to `testActionToRotateFileCompressesOutsideWriteLock`, as there is no monitor any more. - `CsvAuditEventHandlerTest.testPublishEventAfterShutdownFails`, standard and tamper-evident, each with rotation disabled and enabled: an event published after `shutdown()` fails with a `ServiceUnavailableException` and is not in the file, and the tamper-evident file passes verification. Without the fix, all four variants report success. - `CsvAuditEventHandlerTest.testActionToRotateFileAfterShutdownFails`, with rotation enabled and disabled: the `rotate` action after `shutdown()` fails and leaves the file as it is. Without the fix, it rotates the file in both modes. - `CsvAuditEventHandlerTest.testPublishEventAfterShutdownAndStartup`: after `shutdown()` and `startup()`, a topic whose file could not be opened at startup opens it at its next event. - `TextWriterTest` (new, `audit/core`): a failure of the output stream is thrown by `flush()` with its message; a write after `shutdown()` throws; after a failed flush of two events, the next flush and write throw too; the next failures have the first one as their cause; after a failed write of a text longer than the buffers, `shutdown()` writes nothing of it; a text written and flushed reaches the stream with its byte count. Without the fix, all but the last fail: with the plain `BufferedWriter`, the next flush returns, and `shutdown()` writes 8192 bytes of the failed text. - `TextWriterTest.testStreamFailsTheFlushThatWaitsForAFailingFlush`: two publishers flush at the same time, and the write of the first flush to the stream fails while the second flush waits for it. Both flushes fail, and nothing reaches the stream. Without the check under the lock, the second flush returns. - `TextWriterTest.testStreamFailsTheWriteThatWaitsForAFailingFlush`: a publisher writes while the write of a flush to the stream fails. Both fail, and nothing reaches the stream. Without the lock around the write, the write returns, and puts its event into the buffers that the failure left unusable. - `TextWriterTest.testStreamClosesTheOutputStreamWhenTheShutdownFails`: the close that writes the buffers fails, and the stream is closed. Without the fix, the stream stays open on JDK 11. A probe of `BufferedWriter.close()` over a stream that fails leaves the stream open on 1.8.0_402 and 11.0.32 too, and closes it on 17.0.20, 21.0.12 and 27. - `RotatableWriterTest.testFailsEveryWriteAndFlushAfterAFailure`, `testDoesNotWriteTheTextOfAFailedWriteWhenClosed` and `testDoesNotRotateAfterAFailure`, through a stream whose first write fails. Without the fix, the second flush returns, `close()` writes 8192 bytes of the failed event, and `forceRotation()` rotates the file. - `RotatableWriterTest.testDoesNotRotateAfterTheRotationFailsToCloseTheFile`: the close of a size rotation that `write()` triggers fails to write the text, and the disk works again before the flush of the text. `forceRotation()` and the flush fail, the file holds only the text written before, and there is no archive. Without the fix, `forceRotation()` rotates the file, and the flush succeeds on the new file for the text that the failed close dropped. - `AsynchronousTextWriterTest` (new, `audit/core`): after the wrapped writer fails a write, a later `write()` fails with that failure as the cause; `flush()` throws the failure of the wrapped writer, and the next `write()` fails. Without the fix, no write fails, and `flush()` returns. - `CsvAuditEventHandlerTest.testFailuresOfAuditFileAreServerErrors`: with a directory in place of the audit file, a publish, a read, a query and the `rotate` action fail with an `InternalServerErrorException`. Without the fix, all four fail with a `BadRequestException`. `testActionToRotateFileAfterShutdownFails` now expects a `ServiceUnavailableException`, and `testActionToRotateTamperEvidentFileKeepsFileWhenKeyStoreCannotBeRenamed` an `InternalServerErrorException`. - `CsvAuditEventHandlerTest.testActionsThatTheRequestGetsWrongAreBadRequests`: an action without a topic, an unknown action, and the `rotate` action of an unknown topic fail with a `BadRequestException`. - `CsvAuditEventHandlerTest.testActionToRotateFileThatFailsIsServerError`: with rotation enabled, the `rotate` action fails with an `InternalServerErrorException` for a writer whose `forceRotation()` throws an `IOException`, and for one whose `forceRotation()` returns `false`, with the message of a rotation that does not happen. The writers are put into the handler by reflection. - `CsvAuditEventHandlerTest.testPublishEventOpensWriterWhenTopicIsLeftWithoutOne`: the writer of the topic removes itself from the handler, as a failed reset of another publisher does, and fails the write. The event is written once, to a new writer. Without the `newCsvWriter == null` check, the publish fails with a `NullPointerException`. - `CsvAuditEventHandlerTest.testPublishEventDuringStartupResetsWriterOfReopenedTopic`: after `shutdown()`, a writer is put into the handler by reflection, as when `startup()` has reopened the topic and not returned yet, and fails the write. The event is written once, to a new writer. Without the fix, it fails with a `ServiceUnavailableException`. - `CsvAuditEventHandlerTest.testStartupClosesTheWriterThatItReplaces`: a writer put into the handler by reflection, as one that an event opened after a `startup()` that threw, is closed by the next `startup()`. Without the fix, `startup()` replaces it and leaves it open. - `CsvAuditEventHandlerTest.testRestartLetsOtherTopicsPublishWhenTamperEvidentKeyStoreCannotBeStored`: after `shutdown()`, the file of the topic that `startup()` opens first is deleted and its keystore made read-only, so the writer loads the keystore but cannot store the first keys of the new file, and `startup()` throws (#340). An event of the other topic is then published. Without the fix, it fails with a `ServiceUnavailableException`. The test fails if `startup()` does not throw, so that it does not pass without the fix, and is skipped where the keystore cannot be made read-only, as for root. - `CsvWriterTest.shouldFailToWriteEventToClosedStandardCsvWriter` and `shouldKeepSecureCsvFileVerifiableWhenWriteToClosedWriterFails`: an event written to a closed writer fails with an `IOException`, and the tamper-evident file still passes verification with its keystore. Without the fix, both writes report success. - `CsvAuditEventHandlerTest.testPublishEventKeepsTamperEvidentWriterWhenKeyStoreCannotBeStored`, with rotation enabled and disabled: a directory replaces the keystore during one event, then the keystore is restored. Every event is published and written once, and the file passes verification with its keystore. Without the fix, that event fails with `RuntimeException: Error when initializing a secure CSV writer` (#340), from the reset. - `CsvAuditEventHandlerTest.testPublishEventKeepsTamperEvidentWriterWhenRotationAfterItFails`: with a size-based rotation, a directory replaces the keystore during the event whose write rotates the file, so the signature of the rotation fails. Every event is published and written once, to the archive that the next event rotates, and the archive passes verification with its keystore. Without the fix, the event fails the same way. - `RotatableWriterTest.testDoesNotFailTheWriteOfATextWhenTheRotationAfterItFails`: a rotation hook fails at the write that follows a file of the maximum size; the write and the flush return, and the text is in the file once. Without the fix, the write throws the failure of the hook. The default rotation under the read lock is caught by the first test, and the compression under the write lock by the third. For #350, keeping the closed writers in the map fails the last three tests, dropping the check under the write lock fails the four variants of `testPublishEventAfterShutdownFails`, and a `startup()` that does not clear the mark fails the last test. For #349, storing the key before the row is written fails `shouldKeepSecureCsvFileVerifiableWhenWriteToClosedWriterFails` with "Final HMAC key doesn't match expected value". For #351, throwing the failure to store the key fails both variants of `testPublishEventKeepsTamperEvidentWriterWhenKeyStoreCannotBeStored` and `testPublishEventKeepsTamperEvidentWriterWhenRotationAfterItFails`, and throwing the failure of the rotation from `write()` fails `testPublishEventKeepsTamperEvidentWriterWhenRotationAfterItFails` and `testDoesNotFailTheWriteOfATextWhenTheRotationAfterItFails`. A rotation that logs the failed close of the file and goes on fails `testDoesNotRotateAfterTheRotationFailsToCloseTheFile`. A `BadRequestException` for the `IOException` of `forceRotation()`, or for a rotation that does not happen, fails `testActionToRotateFileThatFailsIsServerError`, and an `InternalServerErrorException` for an unknown topic fails `testActionsThatTheRequestGetsWrongAreBadRequests`. Throwing the exception of a rotation that does not happen inside the `try` fails `testActionToRotateFileThatFailsIsServerError` on its message. A retry under the write lock that fails for any topic once the handler is shut down fails `testPublishEventDuringStartupResetsWriterOfReopenedTopic`, a `startup()` that does not close the writer it replaces fails `testStartupClosesTheWriterThatItReplaces`, and a `FailFastWriter.write()` outside the lock fails `testStreamFailsTheWriteThatWaitsForAFailingFlush`. `audit/core`, `audit/handler-csv`, `audit/handler-json` and `audit/handler-syslog` pass locally on JDK 11 and 27, and the new tests of #334 passed 20 runs in a row. One full run on JDK 11 failed `CsvWriterTest.shouldCreateNewAuditFileAfterRotation`, on the archive of an event that rotated the file after a periodic signature, which #342 fixes. ## Not covered by a test - The `rotate` action with rotation enabled against a concurrent reset: the reset needs a write that fails with an `IOException`, which a test cannot cause with rotation enabled. The same lock covers that path, by reasoning only. - The retry of the handler after a write failure with rotation disabled (#349): a test cannot make the file stream of a writer fail without a full disk, so the writer tests close the stream instead, and `TextWriterTest` uses a stream that fails. - The close in the `StandardCsvWriter` constructor when the header cannot be written, for the same reason. - The retry of the handler after a failure with buffering enabled (#363), for the same reason, and the log of `AsynchronousTextWriter`: the tests of `audit/core` log through `slf4j-simple`, which a test cannot read. - The rotation checks after a failure (#365) run on a timer; they go through the same `rotate()` as `forceRotation()`, which the test covers. ## Known limits - A write that fails after part of its bytes reached the file leaves a partial row, which the retry follows with the whole event. - A tamper-evident writer computes the HMAC of a row, and moves to the next key, before it writes the row. After a failed write, the reset closes it: when a signature is due, the final signature covers the HMAC of the row that is not in the file, and its write fails as well. The file then ends with events that no signature covers, and the new writer cannot resume it: the topic fails loudly, where it lost the row and stored a key that broke the file silently before. - With buffering enabled, the records queued before the failure shows are reported as published, and are lost: their number is logged when the writer is replaced. `shutdown()` does not fail for the records still queued when it is called: their failure is only logged. - Until the writer is replaced, the rotation checks log the failure at each check interval. - Until the keystore can be stored again, it holds the key of an earlier row: the file fails verification meanwhile, and a handler stopped before the next row cannot resume it. Each row logs the failure. - A rotation that fails after it has renamed the file leaves the writer half rotated. When the new file cannot be created or opened, the writer keeps the closed one: the flush that follows the write fails, and the handler writes the event again to a new file, while it is in the archive already, as it did before. When a rotation hook fails after the rename, the writer goes on with the new file, which may lack its header or, with tamper-evident logging, the keys of its keystore; before, the write threw, and the reset made the topic fail. The scheduled checks leave the writer in the same states. ## Merging - #348 makes `TextWriter.Stream` write in UTF-8. The two conflict in the imports and the constructor of `TextWriter.Stream`; the merge is `new FailFastWriter(stream, StandardCharsets.UTF_8)`. - #341 and #353, merged meanwhile, add tests that expect a `BadRequestException` for a topic whose file cannot be opened or resumed (`testStartupOpensOtherTopicsWhenTamperEvidentFileCannotBeResumed`, `testStartupWithAuditFileOfMoreColumnsThanTopicSchema`). With #364 the handler answers an `InternalServerErrorException` there, and since the rebase onto them they expect it. With #341, a file that cannot be resumed fails with an `IOException`, and `startup()` opens the other topics, so the restart test now fails the writer with a keystore that cannot be stored, which is still a `RuntimeException` (`testRestartLetsOtherTopicsPublishWhenTamperEvidentKeyStoreCannotBeStored`). - #342 changes the condition that schedules a signature in `SecureCsvWriter.writeEvent()`, two lines below the key store that this PR stops throwing for. The two merge without a conflict, and so do this PR and #355. ## Not in this PR Found in the review of this change, older than it, and filed separately: - #340, fixed by #341: a `RuntimeException` from `createCsvWriter()` escapes `startup()`, `publishEvent()` and `handleAction()`.
Fixes #339. Fixes #372. Fixes #373.
Resuming a file with no event since the last rotation
After a rotation, the live file holds the header and the signature chained from the archive.
CsvSecureVerifierkept that signature for the chain but did not count it as a final signature, so the file failed withMissing final signature, andSecureCsvWritercould not resume it after a restart. The signature chained into row 2 now counts as the final one until an event follows. A file truncated to these two rows after later events is still rejected, as the current key in its keystore no longer matches.Signatures around a rotation
SecureCsvWriternow tracks whether events were written since the last signature, and signs only then:Final HMAC key doesn't match expected value(Tamper-evident CSV audit: an archive rotated by the first event after a periodic signature ends unsigned #372).writeEvent()scheduled a signature thatclose()wrote into the new file over the last HMAC of the previous file, which the new file does not hold, so it failed at row 3. This is the failure thatshouldRotateCsvAndKeyStoreFileasserted for its last archive (CAUD-225), and such a live file could not be resumed either.rotateaction or by a time-based rotation of a topic with no event since startup, leaves the new file empty instead of writing an empty chained signature (,,), on which the verifier threw aNullPointerExceptionwhen the writer resumed the file.Missing final signature, and the event went into the new file without a header. This is likely after a restart over the empty live file that the rotation above leaves: its modification time counts as the last rotation, and the first event can come before the first rotation check.A second
close()does nothing. Asclose()now always runs the signature step, which flushes the writer, a second call with rotation threwIOException: Stream closed.An event written while the periodic signature completes
writeEvent()scheduled a signature only when the scheduled one was done. The periodic task releasessignatureLockand flushes the writer before itsFutureTaskis done, so an event written in this window scheduled none, and stayed unsigned until the next event, a rotation orclose()(#373). A process killed meanwhile left a live file that failed verification at the next start.The writer now keeps the scheduled task until it starts signing, in
pendingSignature.writeEvent()schedules a signature for unsigned events when no task is pending, and a task clearspendingSignatureonce it holds the lock, before it signs: an event written after that comes after its signature, and schedules the next one, whether the task is done or not. A rotation orclose()that cancels the scheduled task clears it too.A task clears
pendingSignatureonly if it is that task.cancel(false)returnstruefor a task that has started and waits for the lock, as aFutureTaskstaysNEWuntil itsrun()returns, and the task still runs. With a plain flag, this gives the interleaving that #373 notes: an event schedules a second task, the first one clears the flag when it gets the lock, and the next event schedules a third;close()cancels only the last one, and waits for the second up to the signature interval. The signature task is now oneSignatureTaskper schedule instead of one sharedRunnable, so that a task can tell whether it is the pending one.Verifier
CsvSecureVerifieralso:accepts an empty file whose keys match, as the rotation above archives one;
This replaces the rule of [#331] Tamper-evident CSV audit: verify compressed archives, rename the keystore on the rotate action #332, which rejected every file without a header: a file truncated to nothing after events still fails, as the current key of its keystore moved on.
CsvSecureVerifierTest.shouldNotVerifyEmptyFilenow asserts that failure reason.rejects a row 2 signature that is not base64;
rejects a signature row that holds other values: the data and HMAC columns of signature rows were not checked, so an event could be written into them;
reports a row with neither an HMAC nor a signature as an incorrect HMAC instead of throwing.
Tests
SecureCsvWriterTest: resuming after a forced rotation and after a rotation on write, an event that rotates the file after a periodic signature, a rotation before the first event, a file truncated to the chained signature, a malformed or empty chained signature, signature rows that hold other values, a first event whose write rotates the file, a secondclose().shouldRotateCsvAndKeyStoreFilenow expects every archive and the live file to pass, with the plain and the.gzsuffix; it has 5 archives instead of 6, as the 6th came from the redundant signature ofclose().SecureCsvWriterTest.shouldGenerateHMACColumnhas a 60 s time-out. It fails whenunsignedEventsis cleared after the signature row is written instead of before: that row then schedules another signature, andclose()waits for it for the whole signature interval.SecureCsvWriterTest, for Tamper-evident CSV audit: an event written while the periodic signature completes stays unsigned #373:shouldSignAnEventWrittenWhileThePeriodicSignatureCompletes: the writer'sflush(), whichwriteSignature()calls once the signature row is written, writes a second event the first time it runs. A signature must follow that event, and the live file must pass verification while the writer is still open. With theisDone()condition it fails: no second signature within 10 s.shouldSignPeriodicallyAnEventWrittenAfterARotationCancelledTheScheduledSignature: a forced rotation cancels the scheduled signature, and the next event must get a periodic one. It fails when the cancel leavespendingSignatureset.shouldScheduleNoOtherSignatureWhenARotationSignsWhileTheScheduledOneWaitsForTheLock: the test holds the lock until the scheduled signature waits for it, rotates and writes events, andclose()must return in less than half the 3 s interval. When a task clearspendingSignaturewhatever it holds,close()takes 2.7 s.CsvWriterTest.shouldCreateNewAuditFileAfterRotationverifies the live file, in place of a commented-out TODO.SecureCsvWriterTesttests this PR adds or changes are undertarget.Each fix was disabled in turn, and its test failed.
audit/coreandaudit/handler-csvpass locally, andSecureCsvWriterTestpassed 5 runs in a row.Known limits
CsvSecureArchiveVerifierverifies each file on its own), the row can hold no event, and a truncation is caught by the keys..keystoreaside before the restart.Merging with #347 or #348 does not conflict, and #362 does not touch these files. #355 conflicts with
masterin the constructor ofSecureCsvWriter, and with this PR in no other place.