Skip to content

Keep Redis to hot payloads: cold store, sealed responses, unified ARS children, Redis roles and Sentinel (#252) - #259

Merged
maximusunc merged 14 commits into
develfrom
claude/adoring-knuth-9d7e9j
Nov 12, 2026
Merged

maximusunc merged 14 commits into
develfrom
claude/adoring-knuth-9d7e9j

Conversation

@maximusunc

@maximusunc maximusunc commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Implements stages 1 to 4 of the plan on #252, plus the two issues that work came to depend on. Redis now holds a payload only while it is hot; finished work is read from a cold store.

Part of #252 (the stage 0 load test is still to do). Closes #257. Closes #258. Part of #201 (the Sentinel deployment itself is outside this repo). Partly addresses #222.

What changed

Cold store (stage 1)

  • payload_store has two backends: a directory (ARS_PAYLOAD_DIR, unchanged) and an S3 bucket (PAYLOAD_S3_BUCKET), with the same layout in both.
  • finish_query archives every finished ARA query, direct or ARS-dispatched, as one object of three zstd frames: the stored response, the query, and a tail with the final status, description and logs. The first two are copied as compressed bytes; nothing is decompressed in finish_query.
  • Archiving is best-effort: retried, bounded to 30 seconds, logged, and never fails the query.
  • GET /response/{id} falls back to the archive once the Redis copy is gone.

Smaller Redis (stage 2)

  • Once a query is archived, its query, response and logs get REDIS_HOT_TTL (30 minutes). If the archive write failed they keep the full REDIS_TTL.
  • When an ARS parent completes, every Redis copy in its tree gets the hot TTL. Cold reads re-warm Redis for the same window.
  • Only the newest merged version keeps a cold copy.
  • Lookup callback payloads are deleted once folded. A batch given up on as unmergeable keeps its payloads.

Sealed responses (#257)

  • finish_query seals a response when it claims the query. Every later write checks the seal atomically and is refused; the late task logs one warning, acks and stops.

Premerge cleanup (#258)

  • Logs are no longer added to the premerged payload.
  • Intake and premerge are one load and one save.
  • Nulls are still stripped, now with a warning naming the ARA (step one of removing the step).
  • The ARA's infores is no longer stamped on edges; missing ones are reported as a warning.
  • Phantom support graphs are still removed, now reported as an error.

An ARS child is its ARA's response

  • A child's ID is the response ID its ARA's query is dispatched with, so the two are one Redis entry and one archive. The ARS never writes it.
  • A child fetch returns the ARA's own answer with the query's parameters and logs, identical hot and cold, and nothing until the child is terminal.
  • Premerge output is a private working copy (ars:premerged:<id>), deleted once the merge has attempted the child.

Retention (stage 3)

  • An ARS tree owns its children's archives: counted, kept (including for retained trees and live cache entries) and dropped with the tree.
  • The age sweep and disk-pressure eviction delete only archives no tree owns, decided by asking Postgres. A failed lookup deletes nothing.

Redis roles and availability (stage 4, #201)

  • All Redis clients are built in shepherd_utils/redis_clients.py.
  • Each role (broker, data, lock, logs) can be given its own Redis with REDIS_<ROLE>_HOST. Unset, everything is one Redis as before.
  • With REDIS_SENTINELS and REDIS_SENTINEL_MASTER, clients follow the current primary across a failover.
  • The monitor watches every Redis instance, with new failover, replica-loss, replica-lag and eviction-window alerts. With one Redis, existing alert names and history are unchanged.
  • redis.conf: snapshots off, logs to stdout, no-appendfsync-on-rewrite yes.

Defaults and compose

  • Defaults are sized for a local stack: REDIS_TTL 4 hours, Redis maxmemory 2 GB, broker container 4 GB.
  • The broker entrypoint takes REDIS_MAXMEMORY.
  • The base image service uses scale: 0 in place of a profile, which fixes docker compose up --build on Compose v5.5.1.

Behaviour changes to review

  • ARS child fetches return the raw ARA response, not the premerged one. Deliberate deviation from upstream; see parity register deviations 20 to 22.
  • Intermediate merged versions are readable only while the tree is hot.
  • An edge with no sources now fails validation, dropping that ARA's results from the merge. The old infores stamping used to repair it.
  • redis_persistence_failing is reworded; a failed AOF rewrite alone is now a warning.
  • Without a cold store configured, storage behaviour is unchanged, but the shorter default REDIS_TTL applies.

Deployment notes

  • Deploy when the ARS is idle. There is no compatibility with the old ARS layout; queries in flight will not complete correctly.
  • finish_query needs the store: the shared volume mounted, or the bucket settings plus an IAM role.
  • Set REDIS_MAXMEMORY on the broker unless the chart already supplies its own Redis config; the image default is now 2 GB.
  • Set REDIS_MEMORY_LIMIT on the monitor for the footprint alert.
  • Do not put an S3 lifecycle rule on ara/; it would delete a retained tree's answers.
  • boto3 is a new base dependency, imported only when a bucket is configured.

Testing

  • Unit suite: 2,594 passed, 246 skipped.
  • Run against real services locally: the new Postgres queries (Postgres 16); Sentinel discovery, the blocking pool and both failover cases (real Redis and Sentinel processes; these tests skip where redis-server is not installed, which includes CI); the broker redis.conf and REDIS_MAXMEMORY override.
  • Not yet run on a live stack. The S3 backend is tested against an in-process fake, not a real bucket. The monitor's failover path and dashboard changes are unit-tested only. docker compose up with scale: 0 was validated by config and a dry-run build, not by running it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ

… stage 1)

Redis kept every query's payloads for the full redis_ttl because it was the
only place an ARA response could be read back. This adds the cold store the
plan on #252 calls for, without yet changing what Redis keeps.

- payload_store gains payload classes (ARS messages, ARA responses, ARA
  logs) and a second backend: an S3 bucket (PAYLOAD_S3_BUCKET), using the
  directory layout as object keys. The directory backend is unchanged.
- finish_query archives each finished response as the pipeline left it,
  before the ARS handoff or callback, and the query's logs after its last
  flush. Best-effort: retried, bounded to 30s, logged, never fails the query.
- GET /response/{id} and /arax/response/{id} fall back to the archive once
  the Redis copy is gone.
- The ARS watchdog purges archived responses after
  ARA_RESPONSE_RETENTION_DAYS (30) and, under disk pressure, evicts them
  before any ARS tree. Eviction is skipped for a bucket.
- compose mounts the payload directory into finish_query.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
The archive was two objects per query (the response, and its logs) and
dropped the parameters echo once the query expired from Redis.

Each query is now one self-contained object of three zstd frames: the
stored response and the query, both as the compressed bytes Redis held,
and a tail finish_query writes with the final status, description and
logs. Nothing is decompressed or parsed in finish_query, so archiving
costs it only the compressed size of a response.

- finish_query reads the response's compressed bytes before the ARS
  handoff or callback and writes the object at the end, after its last
  log flush, so the archived logs are complete.
- GET /response/{id} takes the object apart and serves an ordinary TRAPI
  Response, with the parameters from the archived query.
- The separate ara-logs/ payload class is gone.
- The /arax/response fallback is removed: that route sends every
  UUID-shaped id to the ARS lookup, so it never reached the archive.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
…e 2)

Stage 1 added the cold copies; nothing in Redis got smaller. With a cold
store configured, Redis now lets go of a payload once its cold copy exists.
Without one, behaviour is unchanged.

- finish_query: once a query is archived, its query, response and logs are
  left REDIS_HOT_TTL (30 min) instead of the rest of REDIS_TTL. A failed
  archive write leaves the full TTL.
- ARS children keep no durable copy of their own. A child records the
  response id of its ARA query (ars_message.ara_response_id, set when the
  child is created) and, once its Redis copy has expired, is served from
  that response's archive: the ARA's own answer with its parameters and
  logs. A child whose ARA never answered keeps a durable copy as before.
- Only the newest merged version keeps a durable copy: a version that
  finishes Done drops the copy of the one it supersedes.
- When a parent completes, every Redis copy in its tree is given
  REDIS_HOT_TTL, and a read from a durable copy re-warms Redis for only
  that long.
- merge_message deletes a callback's payload once it is folded (with or
  without a store). A batch given up on as unmergeable keeps its payloads.

The new column and its upgrade path were run against a real Postgres 16.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
…#258)

The de-federated ARS only premerges answers from ARAs this deployment
hosts, so several steps inherited from upstream either did nothing useful
or hid ARA bugs.

- Intake no longer splices the query's logs into the child's payload. The
  merge folds only each child's message and the logs stay in the query's
  log list; an expired child is still served from its ARA's archived
  response, logs included.
- Intake and premerge are one pool call: the ARA's response is loaded once,
  counted and scored, premerged, and saved once under the child. The
  intake state machine, its guards, notification and terminal shapes are
  unchanged; a delivery that ends at intake still stores the response once.
- Null stripping stays but is reported as a WARNING on the query (step one
  of removing it); count_stripped_nulls returns the count.
- decorate_edges_with_infores is gone: edges whose sources lack the ARA's
  infores are reported as a WARNING and left as sent.
- Phantom support graphs are still removed, now reported as an ERROR, and
  a response with no auxiliary_graphs no longer skips the pass.

The reports are logged in the pool child through pool_task_logger and
folded into the task's query logs. Goldens changed by the infores change
are declared divergences; parity register deviation 21 records all of it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
Once finish_query claims a query, its response is about to be delivered
and archived, but any task still running for the query (a merge pass in
flight, a late callback, a redelivered or reclaimed task, a slow
operation) could still save over it.

finish_query now seals the response (response_sealed:<id>, TTL
redis_ttl, so it outlives the response) after its own too-large rewrite
and before it reads the response for delivery, hand-off or archiving.
Every message write checks the seal atomically with the write
(WATCH/MULTI/EXEC, retried on WatchError), in both the async and the
sync paths, and a refused write raises ResponseSealedError. The task
lifecycle and merge_message treat that as "the query is done": one
WARNING, ACK, stop, without handing the query on or failing it. The
lifecycle also checks the seal before running an operation.

Tests that mocked the Redis client's get/set now use fakeredis, since a
save is a transaction.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
…es (#252)

A child had its own pk and its own payload: the ARA's answer after the ARS
had premerged it, with a pointer column to the ARA's archived response for
reads after the hot window. So a child read differently hot and cold, the
same answer sat in Redis twice, and premerge needed to write a payload the
response seal (#257) now protects.

A child's pk is now the response id its ARA's query is dispatched with, so
the child and the response are one entry: written by the ARA's pipeline,
sealed by finish_query, never written by the ARS.

- A child's payload is the ARA's answer as the ARA would deliver it (the
  stored response with the query's parameters and logs), read from Redis
  while hot and from the archive afterwards: the same bytes either way.
  A child that is not terminal has no payload yet.
- What premerge makes of a child is the merge's working copy, stored at
  ars:premerged:<pk>, saved only when the response validated, read by the
  fold and deleted once the merge has attempted the child.
- The "Internal ARS Server Error" entry of a child ending E/500 goes to
  the ARS query's logs instead of into the child's payload.
- A failed dispatch no longer saves the query under the child.
- ars_message.ara_response_id and its index are gone (never released).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
Archived ARA responses were purged purely by age and evicted first under
disk pressure, whoever they belonged to. So a retained ARS tree, or one
backing a live cache entry, lost its children's answers after 30 days, and
evicting a tree did not count or free them.

An ARS child's pk is its ARA's response id, so an archive either is an ARS
child's or belongs to a directly submitted query:

- finish_query records the archive's size on the child's row (data_size,
  leaving updated_at alone). The tree's retention then counts it, keeps it
  for as long as the tree keeps its payloads, and drops it with them.
- The age purge and the disk-pressure archive eviction ask Postgres who
  owns each candidate, in batches, and delete only archives no tree owns.
  A failed lookup deletes nothing.

The new queries were run against a real Postgres 16.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
#252 stage 4, #201)

The broker, data, lock and logs clients were each built by hand against
redis_host, in six places, so none of them could be moved to a Redis of
its own or follow a Sentinel failover.

shepherd_utils/redis_clients.py now builds all of them:

- Each role can be given its own Redis (REDIS_<ROLE>_HOST / _PORT /
  _PASSWORD), keeping its database number. Unset, every role is on
  redis_host as before.
- With REDIS_SENTINELS and REDIS_SENTINEL_MASTER set, clients ask Sentinel
  for the current primary and follow it across a failover; a role with a
  Redis of its own names its master with REDIS_<ROLE>_SENTINEL_MASTER.
  Commands that find the primary gone or demoted are retried.
- The Sentinel pool blocks for a free connection like the direct pools,
  where redis-py's own Sentinel pool raises.

Discovery, the blocking pool and both failover cases (primary demoted,
primary killed) are tested against real Redis and Sentinel processes,
skipped where redis-server is not installed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
… and Sentinel (#201)

redis.conf still had what production dropped: hourly snapshots (a second
fork, and another full copy of the dataset on the AOF's volume) and logging
to a file on the data volume, where neither `docker logs` nor the log
pipeline sees it.

- Snapshots off; the AOF is the persistence.
- Log to stdout.
- no-appendfsync-on-rewrite yes, so a rewrite cannot stall writes past the
  clients' socket timeouts.
- Comments say how to size maxmemory against the container and the volume.

The README gains the Redis roles (which settings move a role to its own
Redis, and the eviction policy each wants) and the Sentinel setup (client
settings, what a failover costs, the per-node Redis settings from #201).

The modified redis.conf was loaded by a real redis-server.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
#201)

Redis roles can now live on servers of their own, and behind Sentinel. The
monitor only ever looked at the broker's Redis, so a data Redis filling up,
evicting, failing to persist or restarting would have gone unseen, and its
restart would not have finished the queries whose payloads it lost.

The poller now takes one INFO per distinct instance. snapshot["redis"] stays
the broker role's instance; snapshot["redis_instances"] has them all. Memory,
eviction and persistence alerts are evaluated per instance with separate
state. The broker role's instance keeps its alert names, cooldown keys,
run_id key and history series; any other instance gets the same names with
"@<instance>" appended (history: "redis@<instance>:<metric>"). With one Redis
the alerts and their text are unchanged.

A restart is mapped to what the instance holds: the data role's runs the
lost-payload sweep, the broker role's the lost-task sweep, and one shared
Redis both, as before. Restarts of several instances fold into one sweep.
Behind Sentinel a failover changes run_id too and is swept the same way; it
is reported as redis_failover when the new primary carries the old one's
replication id.

New for #201: replica-loss and replica-lag alerts; each instance's write rate
and eviction window (maxmemory / write rate) in the snapshot and history,
with an alert when an instance near its cap would evict keys younger than the
longest query budget; and a persistence alert that says whether an AOF write,
an AOF rewrite or an RDB snapshot failed, since only some of those lose data.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
`docker compose up --build` failed before building anything on Compose
v5.5.1: "service X declares unknown service shepherd_base as additional
contexts", naming a different service on each run. That version drops a
profile-disabled service before it resolves `additional_contexts`, and
shepherd_base was behind the `base` profile so that `up` would not run it.

`scale: 0` keeps the service in the project on every Compose version and
still gives it no container.

Validated with `docker compose config` and a dry-run build on Compose
v5.6.1; `up` itself was not run (no Docker daemon available).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
The defaults were a production deployment's: payloads kept in Redis for 3
days, a 6gb maxmemory in a 25g container. A laptop running compose paid for
that, while a deployment overrides all of it anyway.

- redis_ttl defaults to 4 hours. It only has to outlast the longest query;
  finished work is read from the cold store.
- redis.conf's maxmemory is 2gb and compose gives the broker 4g.
- The broker entrypoint takes REDIS_MAXMEMORY, so a deployment raises
  maxmemory without shipping its own redis.conf.

A deployment now needs to set REDIS_MAXMEMORY (unless it already supplies
its own Redis config), and REDIS_TTL if it has no cold store.

The maxmemory default and override were checked against a real
redis-server started through the entrypoint.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAnJGKGh1MT3uKvawZ
@maximusunc maximusunc changed the title Archive finished ARA responses in a cold store with an S3 backend (#252 stage 1) Keep Redis to hot payloads: cold store, sealed responses, unified ARS children, Redis roles and Sentinel (#252) Nov 12, 2026
@codecov

codecov Bot commented Nov 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.03704% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.52%. Comparing base (d40627e) to head (38c9c0a).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Patch % Lines
shepherd_utils/db.py 79.41% 9 Missing and 5 partials ⚠️
workers/monitor/poller.py 88.88% 3 Missing and 7 partials ⚠️
shepherd_utils/ars/db.py 92.30% 6 Missing and 2 partials ⚠️
shepherd_utils/ars/payload_store.py 92.23% 4 Missing and 4 partials ⚠️
shepherd_utils/response_archive.py 94.83% 8 Missing ⚠️
shepherd_utils/ars/premerge.py 83.33% 2 Missing and 4 partials ⚠️
shepherd_utils/redis_clients.py 94.54% 2 Missing and 4 partials ⚠️
workers/merge_message/worker.py 80.76% 4 Missing and 1 partial ⚠️
workers/monitor/redis_instances.py 92.18% 5 Missing ⚠️
shepherd_utils/ars/retention.py 93.22% 3 Missing and 1 partial ⚠️
... and 8 more
Files with missing lines Coverage Δ
shepherd_server/base_routes.py 91.92% <100.00%> (+0.10%) ⬆️
shepherd_utils/ars/aras.py 100.00% <ø> (ø)
shepherd_utils/ars/lifecycle.py 92.40% <100.00%> (+0.40%) ⬆️
shepherd_utils/ars/trapi.py 92.45% <100.00%> (+1.14%) ⬆️
shepherd_utils/broker.py 83.05% <100.00%> (+1.30%) ⬆️
shepherd_utils/config.py 100.00% <100.00%> (ø)
shepherd_utils/response_limit.py 88.67% <100.00%> (+0.67%) ⬆️
workers/ars_fanout/worker.py 91.26% <100.00%> (+0.78%) ⬆️
workers/ars_merge/worker.py 84.10% <100.00%> (+0.15%) ⬆️
workers/finish_query/worker.py 92.06% <100.00%> (+0.18%) ⬆️
... and 18 more

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25c8fb6...38c9c0a. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit 893a3d8 into devel Nov 12, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/adoring-knuth-9d7e9j branch November 12, 2026 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants