Skip to content

Run dashboard source lookups in the background - #101

Merged
mzupan merged 1 commit into
mainfrom
fix/dashboard-check-background-lookup
Nov 15, 2026
Merged

mzupan merged 1 commit into
mainfrom
fix/dashboard-check-background-lookup

Conversation

@mzupan

@mzupan mzupan commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Follow-ups from the second review of #93.

Changes

  • Background lookups. Before, the shared dashboard/notebook lookup ran inside the request of the read that started it. When that read gave up after 2 s, the lookup was cancelled and the next waiter restarted it from scratch, so a lookup slower than 2 s never finished. Each lookup now runs in its own task with a 10 s timeout. Reads still wait at most 2 s, but the lookup keeps going after they give up, and its answer is cached for the next reads.
  • Busy is not a failure. A read that stops waiting while its lookup is still queued for one of the 8 check slots now gets busy:
    • It doesn't log the "checks are failing" warning.
    • It counts as result="busy" in rush_dashboard_source_checks_total, separate from unknown.
    • It's still unverified, so a long range gets the retryable 503 as before.
  • Outage logging. A failure logs a new warning when none is recorded or the last failure is older than 60 s. A later outage is now logged even if no lookup succeeded since the previous one.
  • Eviction. When the cache is full it drops expired entries, then the oldest tenth, instead of clearing everything.
  • Test naming. The end-to-end test's seen_class is now seen_window, since it holds the time window.

Tests

  • cargo test: 786 lib, 54 bin, 0 failed.
  • New a_slow_lookup_still_answers_later_reads. The lookup takes 250 ms and the first read gives up at 100 ms. A read arriving at 180 ms joins the same lookup and gets Readable, with one lookup call in total.
  • New reads_waiting_for_a_check_slot_are_busy_not_failed. With the only slot held, the read is Busy and no failure is recorded. Once the slot frees up, the queued lookup finishes and is cached.
  • dashboard_upgrade_is_shared_cached_and_bounded now asserts oldest-first eviction.
  • source_check_outages_log_once_and_again_after_a_quiet_period covers the outage logging.
  • The end-to-end test gains a busy case.
  • The four timing-sensitive tests passed 10 out of 10 runs.
  • cargo clippy --all-targets: no warnings on the changed lines.

Docs

/docs/query-limits in repos/marketing needs busy added to the rush_dashboard_source_checks_total results.

Follow-ups from the second review of #93.

- Each shared lookup now runs in its own task with a 10 s timeout. Reads
  still wait at most 2 s, but a read that gives up no longer cancels the
  lookup, so a slow-but-working config database still answers and the
  answer is cached for later reads.
- A read that stops waiting while its lookup is still queued for a check
  slot is "busy", not a failure: it doesn't log the failure warning and
  counts as result="busy" in rush_dashboard_source_checks_total. It is
  still unverified, so long ranges get the retryable 503.
- A failure starts a new outage log line when none is recorded or the
  last failure is older than 60 s, so a later outage is logged even if no
  lookup succeeded in between.
- When the cache is full it drops expired entries and then the oldest
  tenth, instead of clearing everything.
- Tests: a slow lookup answers a later read with one call, busy reads
  don't count as failures, oldest-first eviction, outage logging, and the
  busy case end to end.
@mzupan
mzupan merged commit 8da0c16 into main Nov 15, 2026
6 checks passed
@mzupan
mzupan deleted the fix/dashboard-check-background-lookup branch November 15, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant