Make the calendar finite for crawlers

[brianglass]

Why

Traffic has grown 5–9× in two weeks, and most of it is crawlers working through an effectively infinite calendar. In the peak hour (Oct 11, 04:00 UTC), 78% of requests, 6,974 of 8,980, were for dates outside is_indexable's ±5-year window.

• Who: Meta's meta-externalagent was reading the 2280s and ClaudeBot the 1790s.
• Why the existing hints don't help: almost none of these requests carry a referer. The crawlers work from stored URL lists, so the pages' noindex/nofollow can't stop them.

What

• orthocal.middleware.finite_calendar_for_crawlers: returns 404 to self-declared crawlers (user agent contains bot, crawl, spider or externalagent) for any URL whose year/month/day falls outside the window.
  • It covers the readings, calendar and API routes, and runs before any calendar work.
  • People and API clients, such as the mobile app's Expo … CFNetwork agent, are untouched.
  • The 404 is Cache-Control: no-store with X-Robots-Tag: noindex, nofollow. The middleware sits above cache_control, so Firebase's CDN never caches a crawler's 404 and serves it to people.
• robots.txt: also disallows /api/greek/ and /api/slavic/, which the older /api/gregorian/-style entries didn't cover.

Meta isn't blocked: it can still crawl the ±5-year window. The sitemap already stays within it (±1 year).

Test plan

• New test_crawler_window:
  • Meta and Claude crawlers get an uncacheable 404 outside the window and the page inside it
  • a browser, the mobile app, and an empty user agent get any date
  • calendar and API routes are covered
  • pages without a date are untouched
• Full suite passes (240 tests, 1 skipped)
• After deploy: out-of-window crawler requests should show as 404s with ~ms latency, and Cloud Run request volume and billable time should drop

🤖 Generated with Claude Code