[vidiecan]

References

dspace-customers issue 1083, finding R03 (shortener input validation).

No upstream fix: ufal/clarin-dspace clarin-v7 has no change to ExternalHandleRestRepository beyond dtq-dev (git log --remotes=ufal --not origin/dtq-dev on the file; gh search prs --repo ufal/clarin-dspace for shortener, magicLindat, ExternalHandle). DSpace/DSpace has no such controller.
Related: the delimiter check was added and reverted in #1508 ("Bad shortener input is a separate issue").

Description

The anonymous shortener (POST/PUT /api/services/handles) stored input its own host policy did not cover:

• @magicLindat@ in title (or another metadata field) shifts the parts of the stored record. The resolver takes the last part as the destination, so a title ending in https://evil.example/ makes the short link redirect there, whatever the allowed hosts are.
• java.net.URL reads https://evil.example\@lindat.cz/ as host lindat.cz (allowed); browsers end the host at \ and go to evil.example. file:// and other schemes passed when the host was allowed.
• Host rules compared the raw spelling: BLOCKED.example, blocked.example. and fullwidth ｂｌｏｃｋｅｄ.example passed a blocked\.example deny rule. The same applies to the deny rules shipped on customer/TUL, sav, uk, zcu-data, zcu-pub (.*\\.com;.*\\.xyz;...): https://EXAMPLE.COM/some/path was not denied.
• reportemail was not checked; the record size was unbounded (handle.url is varchar without a limit on PostgreSQL); a subprefix too long for the handle column reached the database (500).
• A failed PUT returned the SQLException message to the client.

Fix:

• URL parsed with java.net.URI; only http/https with an ASCII host.
• Host allow/deny rules match case-insensitively, without the trailing root dot.
• @magicLindat@ rejected in title, reportemail, datasetName, datasetVersion, query (reportemail too: EmailValidator accepts a quoted local part such as "x@magicLindat@y"@example.org).
• reportemail checked with commons-validator EmailValidator.
• Stored record limited to 8192 characters (the H2 handle.url column) on POST and PUT; <prefix>/<subprefix>-<random> limited to the 256 characters of the handle column.
• PUT 500 returns a generic message; the exception is logged.

Behaviour changes:

• A URL with unencoded space, ", <>, {}, |, \, ^, backtick or a bare % is rejected (400). Clients must percent-encode.
• A host must be ASCII: send IDN hosts as punycode (xn--...). Not mapped with java.net.IDN on purpose: it is IDNA2003, browsers use UTS testIssue 316 #46, and the two disagree (e.g. faß.de).
• Host rules are now case-insensitive (https://LINDAT.CZ/ is allowed by the default rule). URL rules still match the raw URL.
• An invalid reportemail or a too long subprefix returns 400.
• PUT does not check the handle/PID format or re-validate the stored title, e-mail etc., so legacy handles keep their update path. One exception: PUT returns 400 when the stored metadata plus the new URL exceed 8192 characters.

Instructions for Reviewers

Changes:

• ExternalHandleRestRepository: URL/host/e-mail/delimiter/size checks, generic PUT error.
• ExternalHandleRestRepositoryIT: 9 tests; POST fixtures use a valid e-mail so the existing host/URL rule tests fail for the URL, not the e-mail.

Tests:

• createRejectsMagicDelimiterInMetadata: @magicLindat@ in metadata moved the redirect destination outside the host rules.
• createRejectsUrlWithBackslashBeforeAt: java.net.URL checked a different host than browsers visit.
• createRejectsNonHttpUrl: a non-http(s) scheme passed with an allowed host.
• hostRulesIgnoreCaseRootDotAndUnicodeSpelling: case, root-dot and fullwidth spellings passed a deny rule; an uppercase allowed host was refused.
• createRejectsInvalidReportEmail: a non-address reportemail was stored.
• createRejectsRecordLongerThanUrlColumn: POST stored a record larger than the URL column.
• createRejectsSubprefixLongerThanHandleColumn: a too long subprefix reached the database.
• updateRejectsRecordLongerThanUrlColumn: PUT stored a record larger than the URL column.
• updateErrorDoesNotRevealDatabaseDetails: PUT echoed the SQLException message.

How to test:
mvn -pl dspace-server-webapp -DskipIntegrationTests=false -Dit.test=ExternalHandleRestRepositoryIT verify

Local run (JDK 11, Windows), ExternalHandleRestRepositoryIT: before the fix the 9 new tests fail (200 or 500 instead of 400, or the leaked message) and the 15 existing ones pass; after the fix 24/24 pass. Checkstyle: 0 violations.

Not in scope:

• Login/CAPTCHA/rate limits.
• Numeric IPv4 spellings (http://2130706433/) are not normalised. Host rules run without DNS, so an IP deny rule can always be bypassed by a host name that resolves to that IP. No shipped config has an IP deny rule.
• The default shortener.post.host.whitelist.regexps = lindat(\\.mff\\.cuni)?.cz has an unescaped . before cz. Besides the two intended hosts it only matches names outside public DNS (lindatXcz, lindat.mff.cuniXcz). It is customer configuration and is not changed here.
• Records already stored.