Repository navigation
Conversation
A dead rates server made priceChangeDaemon log a full stack trace for every event in the stream, about 17 GB a day on push2, because a failed fetch was never cached and each event retried it. Treat a failed fetch like a missing rate for that 5-minute bucket, and log a single warning that names the configured ratesServer so a wrong host is obvious. makeRatesCache takes the fetch function as a parameter so the cache can be tested without the network.
peachbits
marked this pull request as ready for review
October 22, 2026 19:28
A rate-limited load balancer answers 429 with an empty body, which node-fetch reports as "invalid json response body". Check response.ok first so the per-bucket warning says "HTTP 429 from <ratesServer>".
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
The price daemons ask the rates server for one rate per GET, and a loop over the price-change events wants about 7,000 distinct (pair, 5-minute bucket) lookups. The rates load balancer allows 300 requests per 100 seconds, so most of them came back 429 and most alerts were skipped. Add `prefetch(requests)`: it drops lookups already in the cache, de-duplicates the rest, and POSTs them to /v2/exchangeRates in chunks of at most 100, no faster than two requests a second. Replies are matched by (pair, date) with a positional fallback, since the server answers in request order but does not document it. A failed chunk caches `null` for all of its keys and logs once per bucket, through the same set the single fetch uses. `getRate` is unchanged and still fetches a key that was never prefetched. The cache also counts its work, so a daemon can log one line per loop.
The daemons will prefetch rates for one page of events at a time, so they need to group an async stream into arrays while still pulling rows lazily, one page in memory at a time.
The price-change daemon file calls runDaemon() at import time, so nothing in it can be unit-tested. Move checkPriceChange to src/util/priceChange.ts and split out priceChangeFromTime(), the one place that decides which historical time a check compares against. priceChangeLookups() uses the same function to list the three rates an event needs (now, a day back, an hour back), so a batched prefetch cannot drift from what the check asks for. Behaviour is unchanged; the tests pin the from-time for every `triggered` case and the message substitution.
List the rates checkEventTrigger will ask for when it checks an event, walking nested all/any triggers with their per-child state the same way checkTrigger does and skipping price levels that have already fired. checkEventTrigger now takes the check time as an optional argument, so a caller can prefetch for exactly that time.
Read the event stream in pages of 500, prefetch every rate the page needs in a few POSTs, then run the unchanged per-event logic from the cache. A loop over the price-change events now makes about 70 requests instead of 7,000, inside the load balancer's limit of 300 per 100 seconds, and the price-level daemon about 24. `now` is taken once per page instead of once per event, so the lookups a check makes are exactly the prefetched ones; the difference is the few seconds a page takes to process. Each loop ends with a `Rates lookups` info line carrying the request, batch, failed-batch and missing-rate counts, which is the number to watch after deploy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CHANGELOG
Does this branch warrant an entry to the CHANGELOG?
Dependencies
none
Description
priceChangeDaemonon push2 has been fetching rates fromhttps://rates2-us1.edge.app, a host that no longer exists. BecauseratesCachenever cached a failed fetch, every price-change event in every loop retried it and logged a full stack trace: about 17 GB ofpriceChangeDaemon.loga day, which filled the server's disk to 97% on 2026-10-21. Price-change alerts were not being sent.Part 1: stop the flood (first two commits). The cache treats a failed fetch like a missing rate for that 5-minute bucket, so the other events in the bucket skip instead of retrying, and logs one warning per bucket that names the configured
ratesServer, so a wrong host shows up in a single line. The next bucket tries the server again. A non-2xx reply (the rates load balancer answers 429 with an empty body when a client exceeds 300 requests per 100 s) is reported asHTTP 429 from <ratesServer>instead ofinvalid json response body.makeRatesCachetakes the fetch function as a parameter so the cache can be unit-tested without the network.The wrong host itself is a config problem, fixed separately by pointing
ratesServerin thepush-settings/servicesdocument back athttps://rates2.edge.app(the code's own default). After that change, production showed the second half of the problem: a loop over the ~50,000 price-change events wants about 7,000 distinct (currency pair, 5-minute bucket) lookups, oneGET /v2/exchangeRateeach, and the load balancer rate-limited most of them, so most alerts were still skipped.Part 2: batch the lookups (remaining commits). Both price daemons now read the event stream in pages of 500, work out every rate the page will ask for, and fetch the ones not yet cached through
POST /v2/exchangeRatesin chunks of at most 100, no faster than two requests a second. The per-event logic is unchanged and reads from the cache. A price-change loop makes about 70 requests instead of 7,000; the price-level loop about 24.RatesCache.prefetch(requests)skips cached keys, de-duplicates, chunks at 100 grouped by bucket, and matches replies by (pair, date) with a positional fallback, since the live server answers in request order but does not document it. A failed chunk (429, network error, bad JSON) cachesnullfor all of its keys and logs once per bucket.getRatestill does a single GET for a key that was never prefetched, as a safety net.priceChangeLookups(event, now)andcheckPriceChangesharepriceChangeFromTime, so the prefetch cannot drift from what the check asks for;priceLevelLookups(event, now)walks nestedall/anytriggers the waycheckTriggerdoes.checkPriceChangemoved tosrc/util/priceChange.tsso it can be tested (the daemon file runsrunDaemon()at import).nowis taken once per page rather than once per event, so a 5-minute boundary mid-page cannot push the rest of the page onto the safety net.event.triggeredcan be stamped a few seconds early for a send late in a page.Rates lookupsinfo line:requests,batches,failedBatches,nullRates,singleFetches. After deploy, expect roughly 70 batches, 0 failed and about 0 single fetches per price-change loop.Tests: 95 passing with a fake fetch and an injected sleep, no network.
test/ratesCache.test.tscovers dedupe, cached-key skipping, the 100-item cap and request bodies, in-order, shuffled, re-dated, short and unrelated replies,nullrates, rejected batches (429, network error, HTML body) with one warning per bucket and no retries, pacing (three chunks, two sleeps, also across calls) and the counters.test/priceChange.test.tspins the from-time for everytriggeredcase against whatcheckPriceChangerequests, plus a threshold send.test/triggers.test.tsdoes the same for nested price-level triggers.test/pages.test.tscovers full, short and empty pages and lazy pulling.eslint,tscand the sucrase build pass on Node 24; the server runs on Node 16 today and the change uses nothing newer.Rollout: deploy to push2, start
priceLevelDaemonfirst and watch two loops'Rates lookupslines, thenpriceChangeDaemon.