Add external body storage for audit by rbev · Pull Request #6029 · Particular/ServiceControl

rbev · 2026-10-20T06:20:26Z

Problem

Audit's SQL Server and PostgreSQL persisters discard oversized and binary message bodies. Oversized text retains a searchable prefix, but the body API cannot return the complete body. Error's SQL persisters already support external storage for these bodies.

Change

Reuse Error's FileSystem, Azure Blob, and S3 providers for Audit by extracting them into ServiceControl.Persistence.BodyStorage.

Keep small valid text bodies inline. Store oversized, binary, content-encoded, non-UTF-8, and NUL-containing bodies externally.
Retain a UTF-8-safe inline search prefix for oversized text and serve the complete body through the body API.
Await external writes before inserting message rows, including when disposing an abandoned ingestion batch.
Delete external bodies before removing expired rows or partitions. A storage failure leaves the database records available for the next sweep.
Provision storage during setup and register the existing disk-space custom check for FileSystem storage.
Update persistence manifests, test fixtures, development configuration, and the testing tool.

RavenDB behavior is unchanged. Existing TooLarge and NotText rows retain their previous API responses; discarded bodies cannot be recovered. The new external-body state uses the existing integer column, so no schema migration is needed.

Required configuration

External storage is required for Audit SQL Server and PostgreSQL instances. Existing instances must add ServiceControl.Audit/MessageBody/StorageType and the selected provider's settings before restarting after an upgrade.

<add key="ServiceControl.Audit/MessageBody/StorageType" value="FileSystem" />
<add key="ServiceControl.Audit/MessageBody/FileSystem/StoragePath" value="D:\ServiceControl.Audit\bodies" />

MaxBodySizeToStore becomes the inline threshold, not a limit on the complete stored body. Azure defaults to the audit-bodies container; S3 defaults to the audit-bodies/ key prefix.

Each instance must use a separate directory, container, or key prefix. Sharing a location can let one instance's retention process delete another instance's bodies.

Design constraints

Sharing the providers avoids duplicating their storage formats, compression, authentication, and stream handling. Error retains its existing configuration and behavior.

Audit retention deletes by hourly prefixes on SQL Server and daily prefixes on PostgreSQL. This preserves PostgreSQL's partition-drop path without scanning message rows. SQL Server only visits hours containing rows, so bodies orphaned by failed inserts into otherwise empty hours can require manual cleanup.

The current design and configuration are documented in Message body storage.

…into audit-external-bodies

warwickschroeder · 2026-10-23T04:14:03Z

 | `--tag <tag>` | Test a specific ServiceControl image tag, for example a PR prerelease like `pr-1234` |
 | `--persistence <type>` | Persistence for the error instance: `RavenDb` (default), `SqlServer`, or `PostgreSql` |
 | `--error-ingestion-scale-unit <n>` | Start `n` additional error-ingestion-only ServiceControl instances |
+| `--azureblob-body-storage "<connection string>"` | Store message bodies in Azure Blob Storage instead of the container file system. Applies only to `SqlServer` and `PostgreSql`. The error instance and its scale-out units share one container, and each audit instance gets its own |


is this missing a "-" between azure and blob?

warwickschroeder · 2026-10-23T04:46:51Z

+
+External storage is required. The instance fails to start when `MessageBody/StorageType` is missing, which includes existing SQL Server and PostgreSQL Audit instances that upgrade without adding it. Setup creates the directory, container, or bucket.
+
+Settings use the `ServiceControl.Audit/` prefix in the application settings file. The validation matches the Error instance, whose settings use the `ServiceControl/` prefix.


Need to add to docs PR, if not already

warwickschroeder · 2026-10-23T04:50:48Z

+| `MessageBody/MinCompressionSize` | all, minimum body size in bytes to compress | 4096 |
+| `MessageBody/FileSystem/StoragePath` | `FileSystem`, required | none |
+| `MessageBody/FileSystem/DataSpaceRemainingThreshold` | `FileSystem`, free space percentage below which the custom check fails | 15 |
+| `MessageBody/Azure/ConnectionString` | `AzureBlob`, shared key or SAS | none |


So we are only supporting authentication via SAS token? What about Managed Identity or other Entra identities?

warwickschroeder · 2026-10-23T04:51:40Z

@@ -0,0 +1,79 @@
+# Message body storage


Any ADRs worth extracting for the message body design?

warwickschroeder · 2026-10-23T05:00:32Z

+            var storage = new RecordingBodyStorage();
+            await configuration.CreateRetention(storage).SweepNow();
+
+            Assert.That(storage.DeletedPrefixes, Has.Some.Matches<string>(bodyId.StartsWith));


Has.Some.Matches(bodyId.StartsWith) also passes for a day prefix, or "2", which would delete every body still in retention.

warwickschroeder · 2026-10-23T05:02:14Z

+                return external is null
+                    ? MessageBodyView.NoContent()
+                    : MessageBodyView.FromStream(external.Stream, external.ContentType, external.BodySize, version);


When the row says External but storage has nothing, this answers 204 with no log. A LogWarning with the body id and row here would make it traceable.

warwickschroeder · 2026-10-23T05:06:12Z

+        await foreach (var blob in container.GetBlobsAsync(BlobTraits.None, BlobStates.None, prefix, cancellationToken: cancellationToken))
+        {
+            await container.GetBlobClient(blob.Name).DeleteIfExistsAsync(cancellationToken: cancellationToken);
+        }


This deletes one blob per round trip, in sequence. An hour of audit traffic with binary bodies can be hundreds of thousands of blobs, so the sweep can take as long as the hour it is clearing, and the rows (and on PostgreSQL the partition drop) wait behind it. S3 already deletes 1,000 per call.

BlobBatchClient.DeleteBlobsAsync (256 per batch) would keep Azure in the same range.

warwickschroeder · 2026-10-23T05:08:29Z

+        foreach (var filePath in Directory.EnumerateFiles(StoragePath, $"{prefix}*.body"))
+        {
+            cancellationToken.ThrowIfCancellationRequested();
+            if (Path.GetFileName(filePath).StartsWith(prefix, StringComparison.Ordinal))
+            {
+                TryDelete(filePath);
+            }
+        }


Bodies are one flat directory, and this enumeration reads every file in it to find one hour's worth. With a month of audit bodies that's tens of millions of entries per hourly sweep.

Would per-hour subdirectories ({StoragePath}/{yyyyMMddHH}/{guid}.body) work for audit, so the delete becomes Directory.Delete?

warwickschroeder · 2026-10-23T05:11:16Z

        catch (RequestFailedException ex) when (ex.Status is (int)HttpStatusCode.Conflict or (int)HttpStatusCode.PreconditionFailed)
        {
            // A blob for this immutable body already exists.
        }


This will swallow every 409 and 412 as "already exists". 409 is also ContainerBeingDeleted and ContainerDisabled, so writes during a container recreate succeed silently and audit rows point at bodies that were never stored.

Filtering on ex.ErrorCode is BlobErrorCode.BlobAlreadyExists or BlobErrorCode.ConditionNotMet would keep the intended case.

johnsimons · 2026-10-23T06:19:07Z

So this would be a new feature?
Because if I correctly recall, Raven does the same, correct?

johnsimons · 2026-10-23T06:20:48Z

So MaxBodySizeToStore is the setting that controls whether it is stored or not?

Add external body storage for audit

0b9b1b9

rbev requested review from abparticular, johnsimons and warwickschroeder October 20, 2026 06:20

rbev self-assigned this Oct 20, 2026

rbev added 2 commits October 20, 2026 14:53

Add load to trigger external body storage

9ddc908

Allow running with azure body storage

1e0a965

rbev marked this pull request as ready for review October 21, 2026 05:29

rbev added 5 commits October 21, 2026 13:30

Merge branch 'master' of https://github.com/Particular/ServiceControl …

5c4baca

…into audit-external-bodies

Add s3 large body support to test tool

0a86806

Fix persistence test compilation after body storage changes

bb1b459

Add body storage to audit tests

157be91

Don't fail the test on failed cleanup

fbe726b

warwickschroeder requested changes Oct 23, 2026

View reviewed changes

Navigation Menu

Repository navigation

Add external body storage for audit - #6029

Add external body storage for audit#6029
rbev wants to merge 8 commits into
masterfrom
audit-external-bodies

rbev commented Oct 20, 2026 •

edited

Loading

Uh oh!

warwickschroeder Oct 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

warwickschroeder Oct 23, 2026

Uh oh!

johnsimons commented Oct 23, 2026

Uh oh!

johnsimons commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		External storage is required. The instance fails to start when `MessageBody/StorageType` is missing, which includes existing SQL Server and PostgreSQL Audit instances that upgrade without adding it. Setup creates the directory, container, or bucket.

		Settings use the `ServiceControl.Audit/` prefix in the application settings file. The validation matches the Error instance, whose settings use the `ServiceControl/` prefix.

Conversation

rbev commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Required configuration

Design constraints

Uh oh!

warwickschroeder Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnsimons commented Oct 23, 2026

Uh oh!

johnsimons commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rbev commented Oct 20, 2026 •

edited

Loading

warwickschroeder Oct 23, 2026 •

edited

Loading