Repository navigation
Run dashboard source lookups in the background - #101
Merged
Merged
Conversation
Follow-ups from the second review of #93. - Each shared lookup now runs in its own task with a 10 s timeout. Reads still wait at most 2 s, but a read that gives up no longer cancels the lookup, so a slow-but-working config database still answers and the answer is cached for later reads. - A read that stops waiting while its lookup is still queued for a check slot is "busy", not a failure: it doesn't log the failure warning and counts as result="busy" in rush_dashboard_source_checks_total. It is still unverified, so long ranges get the retryable 503. - A failure starts a new outage log line when none is recorded or the last failure is older than 60 s, so a later outage is logged even if no lookup succeeded in between. - When the cache is full it drops expired entries and then the oldest tenth, instead of clearing everything. - Tests: a slow lookup answers a later read with one call, busy reads don't count as failures, oldest-first eviction, outage logging, and the busy case end to end.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups from the second review of #93.
Changes
busy:result="busy"inrush_dashboard_source_checks_total, separate fromunknown.seen_classis nowseen_window, since it holds the time window.Tests
cargo test: 786 lib, 54 bin, 0 failed.a_slow_lookup_still_answers_later_reads. The lookup takes 250 ms and the first read gives up at 100 ms. A read arriving at 180 ms joins the same lookup and getsReadable, with one lookup call in total.reads_waiting_for_a_check_slot_are_busy_not_failed. With the only slot held, the read isBusyand no failure is recorded. Once the slot frees up, the queued lookup finishes and is cached.dashboard_upgrade_is_shared_cached_and_boundednow asserts oldest-first eviction.source_check_outages_log_once_and_again_after_a_quiet_periodcovers the outage logging.busycase.cargo clippy --all-targets: no warnings on the changed lines.Docs
/docs/query-limitsinrepos/marketingneedsbusyadded to therush_dashboard_source_checks_totalresults.