Repository navigation
Conversation
added 2 commits
October 27, 2026 15:50
The OAI-ORE ingest crosswalk fetched every href in a remote resource map through URL.openStream(), guarded only by a scheme allow-list and an optional, off-by-default host-prefix check. A collection admin could point a harvest at a hostile OAI source and make the server fetch internal hosts such as 127.0.0.1 or the cloud metadata endpoint 169.254.169.254. Add a self-contained egress guard in org.dspace.harvest.ore: - BlockedAddressPredicate blocks loopback, private, link-local, CGNAT, reserved, documentation and cloud-metadata ranges on the raw address bytes, unwrapping IPv4-in-IPv6 and defeating decimal/hex/IPv6-embedding obfuscation. - OreUrlValidator fails closed: http/https only, no userinfo, resolvable host, host confined to the collection's oai_source (or a configured prefix), and every resolved address checked against the predicate. - SafeResourceFetcher follows redirects by hand, re-validating and re-resolving each hop, pins the socket to the validated address, blocks https->http downgrades, applies connect/read timeouts and a response size cap. Wire the crosswalk to the fetcher with a policy anchored on the oai_source (set from OAIHarvester.processRecord via HarvestPolicyAware); callers with no harvest context get the fail-closed policy. A rejected resource skips and rolls back that record so a bitstream-less item is not committed. Reconcile oai.cfg: drop the obsolete validateUrlPrefix key and document the new blockInternalAddresses / allowedInternalHosts / maxRedirects / timeout / maxBytes keys; allowedUrlPrefix remains the cross-host escape hatch.
Prove the ORE fetch connects to the address the validator vetted and does not resolve the host a second time: a resolver wired into the client's connection manager records every lookup, and the fetch must reach the server with no lookup recorded. This guards the socket pinning (an explicit address and port on the HttpHost) that defeats DNS rebinding between check and connect. Also note in oai.cfg that redirects are re-confined per hop, so a file URL that redirects to a different host must have that host in allowedUrlPrefix.
There was a problem hiding this comment.
🟡 Changes recommended
Active moderate and critical issues remain in OAIHarvester: failed downloads can orphan assetstore files, and rejected attacker-supplied URLs may expose reusable credentials in warning logs.
2 open issues
What changed in this PR
This PR hardens OAI-ORE bitstream harvesting against SSRF by validating destinations, blocking internal addresses, pinning connections to validated DNS results, controlling redirects, enforcing transfer limits, and handling rejected resources safely.
Changes:
- Added ORE URL validation, address filtering, egress policies, redirect controls, DNS pinning, timeouts, and response-size limits.
- Integrated policy-aware fetching and rollback handling into OAI harvesting.
- Documented the new security configuration and added comprehensive unit tests.
| File | Reviewed change |
|---|---|
dspace/config/modules/oai.cfg |
Documents host confinement, internal-address controls, redirect limits, timeouts, and byte caps. |
dspace-api/src/test/java/org/dspace/harvest/ore/SafeResourceFetcherTest.java |
Tests redirects, downgrade prevention, response limits, and DNS pinning. |
dspace-api/src/test/java/org/dspace/harvest/ore/OreUrlValidatorTest.java |
Tests URL parsing, host confinement, DNS policy, and address restrictions. |
dspace-api/src/test/java/org/dspace/harvest/ore/BlockedAddressPredicateTest.java |
Tests blocked internal, reserved, embedded, and obfuscated addresses. |
dspace-api/src/main/java/org/dspace/harvest/ore/SafeResourceFetcher.java |
Performs pinned, bounded, timeout-controlled fetching with per-hop redirect validation. |
dspace-api/src/main/java/org/dspace/harvest/ore/RejectionReason.java |
Defines ORE resource rejection categories. |
dspace-api/src/main/java/org/dspace/harvest/ore/OreUrlValidator.java |
Validates schemes, authority, host confinement, DNS results, and destination addresses. |
dspace-api/src/main/java/org/dspace/harvest/ore/OreResourceRejectedException.java |
Represents rejected ORE resource fetches and their reasons. |
dspace-api/src/main/java/org/dspace/harvest/ore/OreEgressPolicy.java |
Builds immutable collection-specific ORE egress policies. |
dspace-api/src/main/java/org/dspace/harvest/ore/HarvestPolicyAware.java |
Provides policy injection for harvest-aware crosswalks. |
dspace-api/src/main/java/org/dspace/harvest/ore/BlockedAddressPredicate.java |
Identifies private, reserved, metadata, multicast, and obfuscated internal addresses. |
dspace-api/src/main/java/org/dspace/harvest/OAIHarvester.java |
Applies ORE policies and rolls back rejected or failed records. Critical: rejection logging exposes attacker-supplied query strings; log redacted URLs or only the rejection reason. Moderate: database rollback does not clean assetstore files already written before failure, leaving orphaned files. |
dspace-api/src/main/java/org/dspace/content/crosswalk/OREIngestionCrosswalk.java |
Routes resource downloads through the safe fetcher. Moderate: the response/client can leak when downstream bitstream creation fails before consuming the stream; use try-with-resources to close it on every exit path. |
Review effort: Lite
…ate ORE resources - Redact the query string and userinfo from a rejected ORE URL before logging it in rejectRecord: a rejected presigned S3/CDN URL can carry reusable credentials there. - Close the fetched stream (and the HTTP connection/client it holds) on every path via try-with-resources, including when bitstream creation fails part way through. - Validate every aggregated resource URL up front, before any bitstream is created, so a URL rejected by the egress policy cannot leave the bitstreams of earlier resources in the same record orphaned in the assetstore after the record is rolled back.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Security — block SSRF in OAI-ORE harvest ingestion (GHSA-qff6)
Will not be merged now
Part of dataquest-dev/dspace-customers#1104, see #1083.
Problem
A collection administrator (COLLECTION/WRITE) who configures a harvest of type METADATA_AND_BITSTREAMS can make the server fetch every resource href listed in the remote OAI-ORE document. The only existing guard (
validResourceUri) did a scheme allow-list plus an off-by-default, bypassable host-prefix check — no IP-range blocking and no DNS-rebind protection — so a malicious ORE document could drive the server tohttp://127.0.0.1:…,http://169.254.169.254/…(cloud metadata),http://10.x/…, etc. This is SSRF (the upstream GHSA-qff6 scheme fix is already present but insufficient).Fix
All ORE resource fetches now go through a new self-contained
org.dspace.harvest.orepackage:oai_sourcehost, resolve DNS and reject if any resolved address is internal/reserved.getAllByName.Scope is the SSRF core only — no DB migration, no REST/CLI, no frontend.
Host confinement is now unconditional (previously the host check was off by default). Existing
harvest_type=3collections that legitimately pull bitstreams from a host different from theiroai_source— including CDN hosts and bitstream URLs that redirect to another host (e.g. presigned S3) — will now be blocked until an admin adds those hosts tooai.harvester.ore.file.allowedUrlPrefix. Failure is fail-safe: the record is skipped (status OAI_ERROR,lastHarvestednot advanced), no crash. Redirects are re-confined per hop, so cross-host redirect targets must also be allow-listed. The obsoletevalidateUrlPrefixkey is removed; new keys (blockInternalAddresses,allowedInternalHosts,maxRedirects, timeouts,maxBytes) are documented inoai.cfg.Note: when a forward proxy (
http.proxy.host) is configured, the proxy performs the connect, so address checks/pinning become advisory — SSRF protection then depends on the proxy's egress rules (documented).Out of scope
The OAI ReM / metadata fetch to the admin-configured
oai_sourceendpoint itself is not attacker-controlled by the ORE document and is left unchanged (separate, lower-severity concern).Tests
33 unit tests (run, passing): address-block coverage (incl. obfuscated/IPv6-embedded), URL validation, redirect-to-metadata rejection with per-hop revalidation, redirect cap, https→http block, byte caps, and an anti-rebind pinning test that fails if the connection ever re-resolves. The harvester rollback path is code-verified (an executable test would require the full harvest stack).
🤖 Generated with Claude Code