Skip to content

Batch the price daemons' rate lookups - #79

Open
peachbits wants to merge 7 commits into
masterfrom
matthew/rates-fetch-throttle
Open

peachbits wants to merge 7 commits into
masterfrom
matthew/rates-fetch-throttle

Conversation

@peachbits

@peachbits peachbits commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

  • Yes
  • No

Dependencies

none

Description

priceChangeDaemon on push2 has been fetching rates from https://rates2-us1.edge.app, a host that no longer exists. Because ratesCache never cached a failed fetch, every price-change event in every loop retried it and logged a full stack trace: about 17 GB of priceChangeDaemon.log a day, which filled the server's disk to 97% on 2026-10-21. Price-change alerts were not being sent.

Part 1: stop the flood (first two commits). The cache treats a failed fetch like a missing rate for that 5-minute bucket, so the other events in the bucket skip instead of retrying, and logs one warning per bucket that names the configured ratesServer, so a wrong host shows up in a single line. The next bucket tries the server again. A non-2xx reply (the rates load balancer answers 429 with an empty body when a client exceeds 300 requests per 100 s) is reported as HTTP 429 from <ratesServer> instead of invalid json response body. makeRatesCache takes the fetch function as a parameter so the cache can be unit-tested without the network.

The wrong host itself is a config problem, fixed separately by pointing ratesServer in the push-settings/services document back at https://rates2.edge.app (the code's own default). After that change, production showed the second half of the problem: a loop over the ~50,000 price-change events wants about 7,000 distinct (currency pair, 5-minute bucket) lookups, one GET /v2/exchangeRate each, and the load balancer rate-limited most of them, so most alerts were still skipped.

Part 2: batch the lookups (remaining commits). Both price daemons now read the event stream in pages of 500, work out every rate the page will ask for, and fetch the ones not yet cached through POST /v2/exchangeRates in chunks of at most 100, no faster than two requests a second. The per-event logic is unchanged and reads from the cache. A price-change loop makes about 70 requests instead of 7,000; the price-level loop about 24.

  • RatesCache.prefetch(requests) skips cached keys, de-duplicates, chunks at 100 grouped by bucket, and matches replies by (pair, date) with a positional fallback, since the live server answers in request order but does not document it. A failed chunk (429, network error, bad JSON) caches null for all of its keys and logs once per bucket. getRate still does a single GET for a key that was never prefetched, as a safety net.
  • priceChangeLookups(event, now) and checkPriceChange share priceChangeFromTime, so the prefetch cannot drift from what the check asks for; priceLevelLookups(event, now) walks nested all/any triggers the way checkTrigger does. checkPriceChange moved to src/util/priceChange.ts so it can be tested (the daemon file runs runDaemon() at import).
  • now is taken once per page rather than once per event, so a 5-minute boundary mid-page cannot push the rest of the page onto the safety net. event.triggered can be stamped a few seconds early for a send late in a page.
  • Each loop ends with one Rates lookups info line: requests, batches, failedBatches, nullRates, singleFetches. After deploy, expect roughly 70 batches, 0 failed and about 0 single fetches per price-change loop.

Tests: 95 passing with a fake fetch and an injected sleep, no network. test/ratesCache.test.ts covers dedupe, cached-key skipping, the 100-item cap and request bodies, in-order, shuffled, re-dated, short and unrelated replies, null rates, rejected batches (429, network error, HTML body) with one warning per bucket and no retries, pacing (three chunks, two sleeps, also across calls) and the counters. test/priceChange.test.ts pins the from-time for every triggered case against what checkPriceChange requests, plus a threshold send. test/triggers.test.ts does the same for nested price-level triggers. test/pages.test.ts covers full, short and empty pages and lazy pulling. eslint, tsc and the sucrase build pass on Node 24; the server runs on Node 16 today and the change uses nothing newer.

Rollout: deploy to push2, start priceLevelDaemon first and watch two loops' Rates lookups lines, then priceChangeDaemon.

A dead rates server made priceChangeDaemon log a full stack trace for
every event in the stream, about 17 GB a day on push2, because a failed
fetch was never cached and each event retried it. Treat a failed fetch
like a missing rate for that 5-minute bucket, and log a single warning
that names the configured ratesServer so a wrong host is obvious.

makeRatesCache takes the fetch function as a parameter so the cache can
be tested without the network.
@peachbits
peachbits marked this pull request as ready for review October 22, 2026 19:28
A rate-limited load balancer answers 429 with an empty body, which
node-fetch reports as "invalid json response body". Check response.ok
first so the per-bucket warning says "HTTP 429 from <ratesServer>".
@cursor

cursor Bot commented Oct 22, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

The price daemons ask the rates server for one rate per GET, and a
loop over the price-change events wants about 7,000 distinct (pair,
5-minute bucket) lookups. The rates load balancer allows 300 requests
per 100 seconds, so most of them came back 429 and most alerts were
skipped.

Add `prefetch(requests)`: it drops lookups already in the cache,
de-duplicates the rest, and POSTs them to /v2/exchangeRates in chunks
of at most 100, no faster than two requests a second. Replies are
matched by (pair, date) with a positional fallback, since the server
answers in request order but does not document it. A failed chunk
caches `null` for all of its keys and logs once per bucket, through
the same set the single fetch uses. `getRate` is unchanged and still
fetches a key that was never prefetched.

The cache also counts its work, so a daemon can log one line per loop.
The daemons will prefetch rates for one page of events at a time, so
they need to group an async stream into arrays while still pulling
rows lazily, one page in memory at a time.
The price-change daemon file calls runDaemon() at import time, so
nothing in it can be unit-tested. Move checkPriceChange to
src/util/priceChange.ts and split out priceChangeFromTime(), the one
place that decides which historical time a check compares against.
priceChangeLookups() uses the same function to list the three rates
an event needs (now, a day back, an hour back), so a batched prefetch
cannot drift from what the check asks for.

Behaviour is unchanged; the tests pin the from-time for every
`triggered` case and the message substitution.
List the rates checkEventTrigger will ask for when it checks an
event, walking nested all/any triggers with their per-child state the
same way checkTrigger does and skipping price levels that have already
fired. checkEventTrigger now takes the check time as an optional
argument, so a caller can prefetch for exactly that time.
Read the event stream in pages of 500, prefetch every rate the page
needs in a few POSTs, then run the unchanged per-event logic from the
cache. A loop over the price-change events now makes about 70
requests instead of 7,000, inside the load balancer's limit of 300
per 100 seconds, and the price-level daemon about 24.

`now` is taken once per page instead of once per event, so the
lookups a check makes are exactly the prefetched ones; the difference
is the few seconds a page takes to process.

Each loop ends with a `Rates lookups` info line carrying the request,
batch, failed-batch and missing-rate counts, which is the number to
watch after deploy.
@peachbits peachbits changed the title Log one warning per bucket when rates fetch fails Batch the price daemons' rate lookups Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant