Repository navigation
Keep Redis to hot payloads: cold store, sealed responses, unified ARS children, Redis roles and Sentinel (#252) - #259
Merged
Conversation
… stage 1) Redis kept every query's payloads for the full redis_ttl because it was the only place an ARA response could be read back. This adds the cold store the plan on #252 calls for, without yet changing what Redis keeps. - payload_store gains payload classes (ARS messages, ARA responses, ARA logs) and a second backend: an S3 bucket (PAYLOAD_S3_BUCKET), using the directory layout as object keys. The directory backend is unchanged. - finish_query archives each finished response as the pipeline left it, before the ARS handoff or callback, and the query's logs after its last flush. Best-effort: retried, bounded to 30s, logged, never fails the query. - GET /response/{id} and /arax/response/{id} fall back to the archive once the Redis copy is gone. - The ARS watchdog purges archived responses after ARA_RESPONSE_RETENTION_DAYS (30) and, under disk pressure, evicts them before any ARS tree. Eviction is skipped for a bucket. - compose mounts the payload directory into finish_query. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
The archive was two objects per query (the response, and its logs) and
dropped the parameters echo once the query expired from Redis.
Each query is now one self-contained object of three zstd frames: the
stored response and the query, both as the compressed bytes Redis held,
and a tail finish_query writes with the final status, description and
logs. Nothing is decompressed or parsed in finish_query, so archiving
costs it only the compressed size of a response.
- finish_query reads the response's compressed bytes before the ARS
handoff or callback and writes the object at the end, after its last
log flush, so the archived logs are complete.
- GET /response/{id} takes the object apart and serves an ordinary TRAPI
Response, with the parameters from the archived query.
- The separate ara-logs/ payload class is gone.
- The /arax/response fallback is removed: that route sends every
UUID-shaped id to the ARS lookup, so it never reached the archive.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
…e 2) Stage 1 added the cold copies; nothing in Redis got smaller. With a cold store configured, Redis now lets go of a payload once its cold copy exists. Without one, behaviour is unchanged. - finish_query: once a query is archived, its query, response and logs are left REDIS_HOT_TTL (30 min) instead of the rest of REDIS_TTL. A failed archive write leaves the full TTL. - ARS children keep no durable copy of their own. A child records the response id of its ARA query (ars_message.ara_response_id, set when the child is created) and, once its Redis copy has expired, is served from that response's archive: the ARA's own answer with its parameters and logs. A child whose ARA never answered keeps a durable copy as before. - Only the newest merged version keeps a durable copy: a version that finishes Done drops the copy of the one it supersedes. - When a parent completes, every Redis copy in its tree is given REDIS_HOT_TTL, and a read from a durable copy re-warms Redis for only that long. - merge_message deletes a callback's payload once it is folded (with or without a store). A batch given up on as unmergeable keeps its payloads. The new column and its upgrade path were run against a real Postgres 16. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
…#258) The de-federated ARS only premerges answers from ARAs this deployment hosts, so several steps inherited from upstream either did nothing useful or hid ARA bugs. - Intake no longer splices the query's logs into the child's payload. The merge folds only each child's message and the logs stay in the query's log list; an expired child is still served from its ARA's archived response, logs included. - Intake and premerge are one pool call: the ARA's response is loaded once, counted and scored, premerged, and saved once under the child. The intake state machine, its guards, notification and terminal shapes are unchanged; a delivery that ends at intake still stores the response once. - Null stripping stays but is reported as a WARNING on the query (step one of removing it); count_stripped_nulls returns the count. - decorate_edges_with_infores is gone: edges whose sources lack the ARA's infores are reported as a WARNING and left as sent. - Phantom support graphs are still removed, now reported as an ERROR, and a response with no auxiliary_graphs no longer skips the pass. The reports are logged in the pool child through pool_task_logger and folded into the task's query logs. Goldens changed by the infores change are declared divergences; parity register deviation 21 records all of it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
Once finish_query claims a query, its response is about to be delivered and archived, but any task still running for the query (a merge pass in flight, a late callback, a redelivered or reclaimed task, a slow operation) could still save over it. finish_query now seals the response (response_sealed:<id>, TTL redis_ttl, so it outlives the response) after its own too-large rewrite and before it reads the response for delivery, hand-off or archiving. Every message write checks the seal atomically with the write (WATCH/MULTI/EXEC, retried on WatchError), in both the async and the sync paths, and a refused write raises ResponseSealedError. The task lifecycle and merge_message treat that as "the query is done": one WARNING, ACK, stop, without handing the query on or failing it. The lifecycle also checks the seal before running an operation. Tests that mocked the Redis client's get/set now use fakeredis, since a save is a transaction. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
…es (#252) A child had its own pk and its own payload: the ARA's answer after the ARS had premerged it, with a pointer column to the ARA's archived response for reads after the hot window. So a child read differently hot and cold, the same answer sat in Redis twice, and premerge needed to write a payload the response seal (#257) now protects. A child's pk is now the response id its ARA's query is dispatched with, so the child and the response are one entry: written by the ARA's pipeline, sealed by finish_query, never written by the ARS. - A child's payload is the ARA's answer as the ARA would deliver it (the stored response with the query's parameters and logs), read from Redis while hot and from the archive afterwards: the same bytes either way. A child that is not terminal has no payload yet. - What premerge makes of a child is the merge's working copy, stored at ars:premerged:<pk>, saved only when the response validated, read by the fold and deleted once the merge has attempted the child. - The "Internal ARS Server Error" entry of a child ending E/500 goes to the ARS query's logs instead of into the child's payload. - A failed dispatch no longer saves the query under the child. - ars_message.ara_response_id and its index are gone (never released). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
Archived ARA responses were purged purely by age and evicted first under disk pressure, whoever they belonged to. So a retained ARS tree, or one backing a live cache entry, lost its children's answers after 30 days, and evicting a tree did not count or free them. An ARS child's pk is its ARA's response id, so an archive either is an ARS child's or belongs to a directly submitted query: - finish_query records the archive's size on the child's row (data_size, leaving updated_at alone). The tree's retention then counts it, keeps it for as long as the tree keeps its payloads, and drops it with them. - The age purge and the disk-pressure archive eviction ask Postgres who owns each candidate, in batches, and delete only archives no tree owns. A failed lookup deletes nothing. The new queries were run against a real Postgres 16. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
#252 stage 4, #201) The broker, data, lock and logs clients were each built by hand against redis_host, in six places, so none of them could be moved to a Redis of its own or follow a Sentinel failover. shepherd_utils/redis_clients.py now builds all of them: - Each role can be given its own Redis (REDIS_<ROLE>_HOST / _PORT / _PASSWORD), keeping its database number. Unset, every role is on redis_host as before. - With REDIS_SENTINELS and REDIS_SENTINEL_MASTER set, clients ask Sentinel for the current primary and follow it across a failover; a role with a Redis of its own names its master with REDIS_<ROLE>_SENTINEL_MASTER. Commands that find the primary gone or demoted are retried. - The Sentinel pool blocks for a free connection like the direct pools, where redis-py's own Sentinel pool raises. Discovery, the blocking pool and both failover cases (primary demoted, primary killed) are tested against real Redis and Sentinel processes, skipped where redis-server is not installed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
… and Sentinel (#201) redis.conf still had what production dropped: hourly snapshots (a second fork, and another full copy of the dataset on the AOF's volume) and logging to a file on the data volume, where neither `docker logs` nor the log pipeline sees it. - Snapshots off; the AOF is the persistence. - Log to stdout. - no-appendfsync-on-rewrite yes, so a rewrite cannot stall writes past the clients' socket timeouts. - Comments say how to size maxmemory against the container and the volume. The README gains the Redis roles (which settings move a role to its own Redis, and the eviction policy each wants) and the Sentinel setup (client settings, what a failover costs, the per-node Redis settings from #201). The modified redis.conf was loaded by a real redis-server. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
#201) Redis roles can now live on servers of their own, and behind Sentinel. The monitor only ever looked at the broker's Redis, so a data Redis filling up, evicting, failing to persist or restarting would have gone unseen, and its restart would not have finished the queries whose payloads it lost. The poller now takes one INFO per distinct instance. snapshot["redis"] stays the broker role's instance; snapshot["redis_instances"] has them all. Memory, eviction and persistence alerts are evaluated per instance with separate state. The broker role's instance keeps its alert names, cooldown keys, run_id key and history series; any other instance gets the same names with "@<instance>" appended (history: "redis@<instance>:<metric>"). With one Redis the alerts and their text are unchanged. A restart is mapped to what the instance holds: the data role's runs the lost-payload sweep, the broker role's the lost-task sweep, and one shared Redis both, as before. Restarts of several instances fold into one sweep. Behind Sentinel a failover changes run_id too and is swept the same way; it is reported as redis_failover when the new primary carries the old one's replication id. New for #201: replica-loss and replica-lag alerts; each instance's write rate and eviction window (maxmemory / write rate) in the snapshot and history, with an alert when an instance near its cap would evict keys younger than the longest query budget; and a persistence alert that says whether an AOF write, an AOF rewrite or an RDB snapshot failed, since only some of those lose data. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
`docker compose up --build` failed before building anything on Compose v5.5.1: "service X declares unknown service shepherd_base as additional contexts", naming a different service on each run. That version drops a profile-disabled service before it resolves `additional_contexts`, and shepherd_base was behind the `base` profile so that `up` would not run it. `scale: 0` keeps the service in the project on every Compose version and still gives it no container. Validated with `docker compose config` and a dry-run build on Compose v5.6.1; `up` itself was not run (no Docker daemon available). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
The defaults were a production deployment's: payloads kept in Redis for 3 days, a 6gb maxmemory in a 25g container. A laptop running compose paid for that, while a deployment overrides all of it anyway. - redis_ttl defaults to 4 hours. It only has to outlast the longest query; finished work is read from the cold store. - redis.conf's maxmemory is 2gb and compose gives the broker 4g. - The broker entrypoint takes REDIS_MAXMEMORY, so a deployment raises maxmemory without shipping its own redis.conf. A deployment now needs to set REDIS_MAXMEMORY (unless it already supplies its own Redis config), and REDIS_TTL if it has no cold store. The maxmemory default and override were checked against a real redis-server started through the entrypoint. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements stages 1 to 4 of the plan on #252, plus the two issues that work came to depend on. Redis now holds a payload only while it is hot; finished work is read from a cold store.
Part of #252 (the stage 0 load test is still to do). Closes #257. Closes #258. Part of #201 (the Sentinel deployment itself is outside this repo). Partly addresses #222.
What changed
Cold store (stage 1)
payload_storehas two backends: a directory (ARS_PAYLOAD_DIR, unchanged) and an S3 bucket (PAYLOAD_S3_BUCKET), with the same layout in both.finish_queryarchives every finished ARA query, direct or ARS-dispatched, as one object of three zstd frames: the stored response, the query, and a tail with the final status, description and logs. The first two are copied as compressed bytes; nothing is decompressed infinish_query.GET /response/{id}falls back to the archive once the Redis copy is gone.Smaller Redis (stage 2)
REDIS_HOT_TTL(30 minutes). If the archive write failed they keep the fullREDIS_TTL.Sealed responses (#257)
finish_queryseals a response when it claims the query. Every later write checks the seal atomically and is refused; the late task logs one warning, acks and stops.Premerge cleanup (#258)
An ARS child is its ARA's response
ars:premerged:<id>), deleted once the merge has attempted the child.Retention (stage 3)
Redis roles and availability (stage 4, #201)
shepherd_utils/redis_clients.py.REDIS_<ROLE>_HOST. Unset, everything is one Redis as before.REDIS_SENTINELSandREDIS_SENTINEL_MASTER, clients follow the current primary across a failover.redis.conf: snapshots off, logs to stdout,no-appendfsync-on-rewrite yes.Defaults and compose
REDIS_TTL4 hours, Redismaxmemory2 GB, broker container 4 GB.REDIS_MAXMEMORY.scale: 0in place of a profile, which fixesdocker compose up --buildon Compose v5.5.1.Behaviour changes to review
sourcesnow fails validation, dropping that ARA's results from the merge. The old infores stamping used to repair it.redis_persistence_failingis reworded; a failed AOF rewrite alone is now a warning.REDIS_TTLapplies.Deployment notes
finish_queryneeds the store: the shared volume mounted, or the bucket settings plus an IAM role.REDIS_MAXMEMORYon the broker unless the chart already supplies its own Redis config; the image default is now 2 GB.REDIS_MEMORY_LIMITon the monitor for the footprint alert.ara/; it would delete a retained tree's answers.boto3is a new base dependency, imported only when a bucket is configured.Testing
redis-serveris not installed, which includes CI); the brokerredis.confandREDIS_MAXMEMORYoverride.docker compose upwithscale: 0was validated byconfigand a dry-run build, not by running it.🤖 Generated with Claude Code
https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ