Skip to content

fix(security): close bitstream stream leak on HEAD/304/412/416/S3-redirect - #1559

Merged
milanmajchrak merged 7 commits into
dtq-devfrom
sec/bitstream-stream-close
Oct 28, 2026
Merged

milanmajchrak merged 7 commits into
dtq-devfrom
sec/bitstream-stream-close

Conversation

@Kasinhou

@Kasinhou Kasinhou commented Oct 27, 2026 •

Copy link
Copy Markdown

References

  • dspace-customers issue 1104, R43 (from the security audit in dspace-customers issue 1083)
  • Port of upstream DSpace PR 12249 by Bram Maegerman, taken from its dspace-8_x port, DSpace PR 12534. Upstream issue: DSpace issue 12244. Upstream has no 7.x port.

Description

Port of the upstream fix for bitstream streams that were never closed. /api/core/bitstreams/{uuid}/content now opens the stored file only when it writes the body (a citation cover page is still built up front), so HEAD, 304, 412, 416 and the S3 redirect (302) no longer leave it open, and request-a-copy closes the files it attaches.

Instructions for Reviewers

List of changes in this PR:

  • BitstreamResource.java is the vanilla dspace-8_x file with one difference. In the cover-page fallback in getCoverpageByteArray(), vanilla leaves the stream of the original file open (also on main). Here it is closed with try-with-resources.
  • RequestItemEmailNotifier.java gets the same hunks as DSpace PR 12534: the attachment streams are closed in finally. On dtq-dev the single-file branch sits inside the check from dtq-dev PR 1520, so that hunk is one level deeper.
  • BitstreamRestControllerIT: the four tests from DSpace PR 12534, unchanged. Our own tests cover what upstream does not check: HEAD, 304, 412, 416 and the S3 redirect never open the file, a failed cover page closes its streams, and a GET for a file missing in the assetstore ends in 500. The upstream HEAD test only checks that HEAD closes what it opens.

How to test:

mvn -B -pl dspace-server-webapp,dspace -am -P-assembly verify -DskipUnitTests=false -Dtest=RequestItemEmailNotifierTest -DfailIfNoTests=false -DskipIntegrationTests=false -Dit.test=BitstreamRestControllerIT,RequestItemRepositoryIT

On a running instance with a local assetstore, send curl -I .../server/api/core/bitstreams/<uuid>/content a few times and run lsof -p <tomcat pid> | grep assetstore. Before, every HEAD left one more file open until garbage collection. After, none.

On S3, each response without a body also downloaded the first chunk of the file (5 MB by default) to a temp file. Now it downloads no file content, unless a citation cover page has to be built.

Changes a client can see when a file is missing in the assetstore and no citation cover page is built:

  • HEAD returns 200 instead of 500, because HEAD no longer opens the file. GET still returns 500.
  • With S3 direct download, HEAD and GET get the usual 302 instead of 500, and S3 returns the error.

Checklist

  • My PR is created against the main branch of code (unless it is a backport or is fixing an issue specific to an older branch).
  • My PR is small in size (e.g. less than 1,000 lines of code, not including comments & integration tests). Exceptions may be made if previously agreed upon.
  • My PR follows all coding best practices based on the Code Conventions Guide.
  • My PR passes Checkstyle validation based on the Code Style Guide.
  • My PR includes Javadoc for all new (or modified) public methods and classes. It also includes Javadoc for large or complex private methods.
  • My PR passes all tests and includes new/updated Unit or Integration Tests based on the Code Testing Guide.
  • My PR includes details on how to test it. I've provided clear instructions to reviewers on how to successfully test this fix or feature.
  • If my PR includes new libraries/dependencies (in any pom.xml), I've made sure their licenses align with the DSpace BSD License based on the Licensing of Contributions documentation.
  • If my PR modifies REST API endpoints, I've opened a separate REST Contract PR related to this change.
  • If my PR includes new configurations, I've provided basic technical documentation in the PR itself.
  • If my PR fixes an issue ticket, I've linked them together.

🤖 Generated with Claude Code

Matus Kasak added 2 commits October 27, 2026 13:21
…responses

The /content download opened the storage InputStream eagerly while building
response headers (getChecksum()/contentLength() called fetchDocument() which
cached an open stream). The stream was only closed when Spring wrote the
response body, so HEAD, 304, 412 and the S3 redirect (302) returned without a
body and leaked the stream (an S3 connection for S3 assetstores).

fetchDocument() now caches only the etag and length (and materializes the
cover page into a byte[]); getInputStream() opens the stream lazily when the
body is actually written, via a fresh short-lived Context.
The existing testS3DirectDownloadRedirect is gated behind @IfProfileValue and
is skipped in a standard CI run, leaving the highest-severity leak path (the
S3 302 redirect) without a CI-enforced guard. This adds an ungated test that
drives the redirect branch with a mocked presigned-URL service and runtime
config, asserts a 302 and verifies the storage stream is never opened.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Citation-page materialization and fallback stream handling can still leak on bodyless paths, and the test must restore the original S3 configuration value.

1 open issue
What changed in this PR

This pull request updates bitstream downloads to defer storage stream creation until the response body is written, with regression coverage for bodyless responses, redirects, and stream closure.

Changes:

  • Caches bitstream metadata and lazily retrieves normal content streams.
  • Adds coverage for HEAD, conditional responses, S3 redirects, and successful downloads.
  • Preserves fresh-stream behavior for repeated or range-based reads.
File Summary
dspace-server-webapp/​src/​test/​java/​org/​dspace/​app/​rest/​BitstreamRestControllerIT.java Adds regression coverage for bodyless responses, redirects, and stream usage.
dspace-server-webapp/​src/​main/​java/​org/​dspace/​app/​rest/​utils/​BitstreamResource.java Implements lazy bitstream retrieval and cached document metadata.

Review effort: Lite

@milanmajchrak

Copy link
Copy Markdown
Collaborator

Why the file stayed open

The server opened the file while it built the headers. Spring closed it only after it wrote the body. With no body, nothing closed it. This PR opens the file only when Spring writes the body.

flowchart LR
  R["GET or HEAD /content"] --> H["build headers"]
  H -->|"before: opens the file"| B1{"body?"}
  B1 -->|"200"| C1["closed after the body"]
  B1 -->|"HEAD, 304, 412, 416, 302"| L["stays open until GC"]
  H -->|"after: does not open it"| B2{"body?"}
  B2 -->|"200"| C2["opened, sent, closed"]
  B2 -->|"HEAD, 304, 412, 416, 302"| N["nothing to close"]
  style L stroke:red,stroke-width:2px
Loading
Case Before After
304: item page visited again, thumbnail cached in the browser file left open not opened
HEAD: monitoring, crawler, curl -I file left open not opened
302: S3 with direct download downloads up to 5 MB from S3 to disk, then redirects redirects at once
200: normal download opened, sent, closed same

412 and 416 work like HEAD, but are rare. On S3, opening a file first downloads up to 5 MB to a temp file. So on S3 each such request costs a download and a temp file. It does not keep an S3 connection open.

A non-matching If-Match ends in 416 without a body, like HEAD, 304 and
412, so the storage stream must not be opened there either.

The file is now opened only when Spring writes the body. A test checks
that a file missing in the assetstore still ends in 500, not in an
empty 200.

The tests added in this PR used var. The rest of the repo uses explicit
types, so they now declare BitstreamStorageService.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milanmajchrak milanmajchrak changed the title fix(security): close bitstream stream leak on HEAD/304/412/S3-redirect fix(security): close bitstream stream leak on HEAD/304/412/416/S3-redirect Oct 28, 2026
milanmajchrak and others added 3 commits October 28, 2026 14:33
Port of upstream DSpace PR 12249 by Bram Maegerman, taken from its
dspace-8_x port, DSpace PR 12534 (merge commit 6c8b05f).

BitstreamResource.java is now the dspace-8_x file with one change: the
cover-page fallback still closes the stream of the original file.
RequestItemEmailNotifier.java gets the hunks from 12534 and closes the
attachment streams after the mail is sent.

The four tests from 12534 are copied as they are. Our own tests stay
only for cases upstream does not test: 304, 412, 416, the S3 redirect,
the cover-page fallback and a missing file. Our HEAD test repeated the
upstream one and is removed. The gated S3 test is back to the dtq-dev
version, the ungated S3 test covers the same case.

Co-authored-by: Bram Maegerman <bram.maegerman@telenet.be>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The upstream HEAD test only checks that a stream is closed if it was
opened. This test checks that HEAD on /content does not call retrieve
on the storage at all, like the 304, 412 and 416 tests do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milanmajchrak
milanmajchrak merged commit e838ca3 into dtq-dev Oct 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants