Skip to content

TUL: one REST request and a static 404 for missing handles (dspace-customers#1087) - #1651

Open
vidiecan wants to merge 1 commit into
customer/TULfrom
fix/issue-1087-fast-404-tul
Open

vidiecan wants to merge 1 commit into
customer/TULfrom
fix/issue-1087-fast-404-tul

Conversation

@vidiecan

@vidiecan vidiecan commented Oct 21, 2026 •

Copy link
Copy Markdown

References

  • dataquest-dev/dspace-customers#1087: TUL outage on 2026-10-19 18:45. A crawler requested junk suffixes of real handles (/handle/15240/<n>/pt-br.html/en.html/..., 120 junk URLs per item, ~1,700 req/s). Every request was a full SSR with about 19 REST requests, each holding a database connection, until Postgres ran out of connections.

Description

One check in server.ts, before the SSR: for /handle/<prefix>/<anything> and /id/<type>/<id> it makes the one REST request LookupGuard would make first (pid/find, or dso/find for /id/uuid/<uuid>).

  • 404 from the REST API: the answer is a 404 with the static app shell (index.html, 9.8 KB, the same file as the existing CSR fallback), Cache-Control: no-store, no ETag/ranges. The browser renders the usual "not found" page from it.
  • Anything else (302 for an existing object, 401/403, an error, no answer within 3 s): the page renders as before.

Handle suffixes are never judged from the path, only the REST API decides. Paths that Angular's router could read differently than a plain path (;, =, (), %, empty segments) are not checked and render as before. The check is skipped when SSR (universal.preboot) is off.

The specific junk patterns of this crawler stay in the nginx config (#1087), together with the rate limits for /handle/ and /server/api/pid/find, which the bot also calls directly.

Measurements

Production build, SSR against the TUL REST API (read-only proxy); CPU = Node process CPU time per request:

Request Before After
/handle/15240/5308/pt-br.html/x (handle not found) 229 ms CPU, 19 REST requests 7–9 ms CPU, 1 REST request, 404 + 9.8 KB
/id/handle/15240/5308/junk, /id/uuid/<missing uuid> same same as above
/handle/15240/5308, /handle/15240/5308/, /handle/15240/5308;a=b 302 to the item 302 to the item
/handle/15240/nope= (not checked, see above) 19 REST requests unchanged
/items/<existing uuid> 200 200
/foo/bar, /items/<missing uuid> full SSR unchanged (no single REST request can decide these)

Instructions for Reviewers

  • A cache-miss SSR of an existing handle makes one extra REST request (the probe answers 302, then the app looks the handle up again).
  • For an object that really is missing, the 404 now comes before the login/end-user-agreement redirects.
  • Earlier versions of this PR (a route allowlist; a browser-only menu resolver; a 503 on an overloaded REST API; a shared lookup-params module with specs) are dropped to keep the change to one file. They are reviewed in TUL: one REST request and a static 404 for missing handles (dspace-customers#1087) #1651's history; the last one is cedd656be7.
  • Reviewed in several rounds with Codex (gpt-6-astra). Found there and kept: = and ; paths are not checked (Angular reads /handle/1/2=x as handle 1/2); the 404 must not be cacheable or rangeable; a deadline over the whole probe.

Tests

  • Manually against the TUL REST API: missing handles give 404 + app shell after one REST request; existing handles redirect; existing items render; a 500 from pid/find renders the page as before; with a REST API that never answers, the probe gives up after 3 s and the render starts, as before.
  • Full test:headless green, lint on server.ts clean except the pre-existing unused existsSync, build:prod OK.

@vidiecan vidiecan changed the title TUL: answer missing pages with a static 404 instead of an SSR render (crawler flood, dspace-customers#1087) TUL: cheap SSR 404 - render "page not found" and identifier lookups without server-side menus (dspace-customers#1087) Oct 21, 2026
@vidiecan vidiecan changed the title TUL: cheap SSR 404 - render "page not found" and identifier lookups without server-side menus (dspace-customers#1087) TUL: cheap 404s - one REST lookup for missing identifiers, "page not found" without server-side menus (dspace-customers#1087) Oct 21, 2026
@vidiecan vidiecan changed the title TUL: cheap 404s - one REST lookup for missing identifiers, "page not found" without server-side menus (dspace-customers#1087) TUL: cheap 404s - one REST lookup for missing objects, "page not found" without server-side menus (dspace-customers#1087) Oct 21, 2026
@vidiecan
vidiecan force-pushed the fix/issue-1087-fast-404-tul branch from cedd656 to 78b0e0d Compare October 22, 2026 10:35
@vidiecan vidiecan changed the title TUL: cheap 404s - one REST lookup for missing objects, "page not found" without server-side menus (dspace-customers#1087) TUL: one REST request and a static 404 for missing handles (dspace-customers#1087) Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant