Repository navigation
Upgrade vanilla base 7.6.5 → 7.6.7 - #1542
Open
milanmajchrak wants to merge 1461 commits into
Open
milanmajchrak wants to merge 1461 commits into
milanmajchrak wants to merge 1461 commits into
Conversation
…_x/log4j.version-2.25.3
(cherry picked from commit 570fb9c)
(cherry picked from commit 1cb8827)
[Port dspace-7_x] DSpace 8.x bug fix link checker
Bumps the build-tools group with 4 updates: [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin), [org.codehaus.mojo:xml-maven-plugin](https://github.com/mojohaus/xml-maven-plugin), [org.codehaus.mojo:license-maven-plugin](https://github.com/mojohaus/license-maven-plugin) and [org.codehaus.mojo:buildnumber-maven-plugin](https://github.com/mojohaus/buildnumber-maven-plugin). Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.9.0 to 0.10.0 - [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits) Updates `org.codehaus.mojo:xml-maven-plugin` from 1.2.0 to 1.2.1 - [Release notes](https://github.com/mojohaus/xml-maven-plugin/releases) - [Commits](mojohaus/xml-maven-plugin@1.2.0...xml-maven-plugin-1.2.1) Updates `org.codehaus.mojo:license-maven-plugin` from 2.7.0 to 2.7.1 - [Release notes](https://github.com/mojohaus/license-maven-plugin/releases) - [Commits](mojohaus/license-maven-plugin@2.7.0...2.7.1) Updates `org.codehaus.mojo:buildnumber-maven-plugin` from 3.2.1 to 3.3.0 - [Release notes](https://github.com/mojohaus/buildnumber-maven-plugin/releases) - [Commits](mojohaus/buildnumber-maven-plugin@3.2.1...buildnumber-maven-plugin-3.3.0) --- updated-dependencies: - dependency-name: org.sonatype.central:central-publishing-maven-plugin dependency-version: 0.10.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: build-tools - dependency-name: org.codehaus.mojo:xml-maven-plugin dependency-version: 1.2.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: build-tools - dependency-name: org.codehaus.mojo:license-maven-plugin dependency-version: 2.7.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: build-tools - dependency-name: org.codehaus.mojo:buildnumber-maven-plugin dependency-version: 3.3.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: build-tools ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the fasterxml group with 4 updates: [com.fasterxml:classmate](https://github.com/FasterXML/java-classmate), [com.fasterxml.jackson.core:jackson-annotations](https://github.com/FasterXML/jackson), [com.fasterxml.jackson.core:jackson-core](https://github.com/FasterXML/jackson-core) and [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson). Updates `com.fasterxml:classmate` from 1.7.1 to 1.7.3 - [Commits](FasterXML/java-classmate@classmate-1.7.1...classmate-1.7.3) Updates `com.fasterxml.jackson.core:jackson-annotations` from 2.20 to 2.21 - [Commits](https://github.com/FasterXML/jackson/commits) Updates `com.fasterxml.jackson.core:jackson-core` from 2.20.1 to 2.21.0 - [Commits](FasterXML/jackson-core@jackson-core-2.20.1...jackson-core-2.21.0) Updates `com.fasterxml.jackson.core:jackson-databind` from 2.20.1 to 2.21.0 - [Commits](https://github.com/FasterXML/jackson/commits) Updates `com.fasterxml.jackson.core:jackson-databind` from 2.20.1 to 2.21.0 - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml:classmate dependency-version: 1.7.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: fasterxml - dependency-name: com.fasterxml.jackson.core:jackson-annotations dependency-version: '2.21' dependency-type: direct:production update-type: version-update:semver-minor dependency-group: fasterxml - dependency-name: com.fasterxml.jackson.core:jackson-core dependency-version: 2.21.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: fasterxml - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-version: 2.21.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: fasterxml - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-version: 2.21.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: fasterxml ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [org.apache.bcel:bcel](https://github.com/apache/commons-bcel) from 6.11.0 to 6.12.0. - [Changelog](https://github.com/apache/commons-bcel/blob/master/RELEASE-NOTES.txt) - [Commits](apache/commons-bcel@rel/commons-bcel-6.11.0...rel/commons-bcel-6.12.0) --- updated-dependencies: - dependency-name: org.apache.bcel:bcel dependency-version: 6.12.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) from 3.6.3 to 3.6.4. - [Release notes](https://github.com/dnsjava/dnsjava/releases) - [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog) - [Commits](dnsjava/dnsjava@v3.6.3...v3.6.4) --- updated-dependencies: - dependency-name: dnsjava:dnsjava dependency-version: 3.6.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [org.postgresql:postgresql](https://github.com/pgjdbc/pgjdbc) from 42.7.8 to 42.7.9. - [Release notes](https://github.com/pgjdbc/pgjdbc/releases) - [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md) - [Commits](pgjdbc/pgjdbc@REL42.7.8...REL42.7.9) --- updated-dependencies: - dependency-name: org.postgresql:postgresql dependency-version: 42.7.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the amazon-s3 group with 2 updates: software.amazon.awssdk:s3 and [software.amazon.awssdk.crt:aws-crt](https://github.com/awslabs/aws-crt-java). Updates `software.amazon.awssdk:s3` from 2.38.8 to 2.39.6 Updates `software.amazon.awssdk.crt:aws-crt` from 0.39.4 to 0.40.1 - [Release notes](https://github.com/awslabs/aws-crt-java/releases) - [Commits](awslabs/aws-crt-java@v0.39.4...v0.40.1) --- updated-dependencies: - dependency-name: software.amazon.awssdk:s3 dependency-version: 2.39.6 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: amazon-s3 - dependency-name: software.amazon.awssdk.crt:aws-crt dependency-version: 0.40.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: amazon-s3 ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the apache-commons group with 4 updates in the / directory: [commons-codec:commons-codec](https://github.com/apache/commons-codec), org.apache.commons:commons-dbcp2, org.apache.commons:commons-pool2 and [org.apache.commons:commons-text](https://github.com/apache/commons-text). Updates `commons-codec:commons-codec` from 1.20.0 to 1.21.0 - [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt) - [Commits](apache/commons-codec@rel/commons-codec-1.20.0...rel/commons-codec-1.21.0) Updates `org.apache.commons:commons-dbcp2` from 2.13.0 to 2.14.0 Updates `org.apache.commons:commons-pool2` from 2.12.1 to 2.13.1 Updates `org.apache.commons:commons-text` from 1.14.0 to 1.15.0 - [Changelog](https://github.com/apache/commons-text/blob/master/RELEASE-NOTES.txt) - [Commits](apache/commons-text@rel/commons-text-1.14.0...rel/commons-text-1.15.0) --- updated-dependencies: - dependency-name: commons-codec:commons-codec dependency-version: 1.21.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: apache-commons - dependency-name: org.apache.commons:commons-dbcp2 dependency-version: 2.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: apache-commons - dependency-name: org.apache.commons:commons-pool2 dependency-version: 2.13.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: apache-commons - dependency-name: org.apache.commons:commons-text dependency-version: 1.15.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: apache-commons ... Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit cfff804)
[Port dspace-7_x] Move same-collection check to inheritDefaultPolicies move method (manual port)
…_x/org.apache.bcel-bcel-6.12.0 Bump org.apache.bcel:bcel from 6.11.0 to 6.12.0
…_x/dnsjava-dnsjava-3.6.4 Bump dnsjava:dnsjava from 3.6.3 to 3.6.4
…_x/fasterxml-8fce214327 Bump the fasterxml group with 4 updates
…_x/io.grpc-grpc-context-1.78.0 build(deps): bump io.grpc:grpc-context from 1.77.0 to 1.78.0
…_x/org.postgresql-postgresql-42.7.9 Bump org.postgresql:postgresql from 42.7.8 to 42.7.9
…_x/build-tools-57bb4f0b4d Bump the build-tools group with 4 updates
…_x/apache-commons-92ef38cb7b
Bumps the test-tools group with 7 updates: | Package | From | To | | --- | --- | --- | | [org.testcontainers:testcontainers-localstack](https://github.com/testcontainers/testcontainers-java) | `2.0.2` | `2.0.3` | | [io.netty:netty-buffer](https://github.com/netty/netty) | `4.2.7.Final` | `4.2.9.Final` | | [io.netty:netty-transport](https://github.com/netty/netty) | `4.2.7.Final` | `4.2.9.Final` | | [io.netty:netty-transport-native-unix-common](https://github.com/netty/netty) | `4.2.7.Final` | `4.2.9.Final` | | [io.netty:netty-common](https://github.com/netty/netty) | `4.2.7.Final` | `4.2.9.Final` | | [io.netty:netty-handler](https://github.com/netty/netty) | `4.2.7.Final` | `4.2.9.Final` | | [io.netty:netty-codec](https://github.com/netty/netty) | `4.2.7.Final` | `4.2.9.Final` | Updates `org.testcontainers:testcontainers-localstack` from 2.0.2 to 2.0.3 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/main/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@2.0.2...2.0.3) Updates `io.netty:netty-buffer` from 4.2.7.Final to 4.2.9.Final - [Commits](netty/netty@netty-4.2.7.Final...netty-4.2.9.Final) Updates `io.netty:netty-transport` from 4.2.7.Final to 4.2.9.Final - [Commits](netty/netty@netty-4.2.7.Final...netty-4.2.9.Final) Updates `io.netty:netty-transport-native-unix-common` from 4.2.7.Final to 4.2.9.Final - [Commits](netty/netty@netty-4.2.7.Final...netty-4.2.9.Final) Updates `io.netty:netty-common` from 4.2.7.Final to 4.2.9.Final - [Commits](netty/netty@netty-4.2.7.Final...netty-4.2.9.Final) Updates `io.netty:netty-handler` from 4.2.7.Final to 4.2.9.Final - [Commits](netty/netty@netty-4.2.7.Final...netty-4.2.9.Final) Updates `io.netty:netty-codec` from 4.2.7.Final to 4.2.9.Final - [Commits](netty/netty@netty-4.2.7.Final...netty-4.2.9.Final) --- updated-dependencies: - dependency-name: org.testcontainers:testcontainers-localstack dependency-version: 2.0.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: test-tools - dependency-name: io.netty:netty-buffer dependency-version: 4.2.9.Final dependency-type: direct:production update-type: version-update:semver-patch dependency-group: test-tools - dependency-name: io.netty:netty-transport dependency-version: 4.2.9.Final dependency-type: direct:production update-type: version-update:semver-patch dependency-group: test-tools - dependency-name: io.netty:netty-transport-native-unix-common dependency-version: 4.2.9.Final dependency-type: direct:production update-type: version-update:semver-patch dependency-group: test-tools - dependency-name: io.netty:netty-common dependency-version: 4.2.9.Final dependency-type: direct:production update-type: version-update:semver-patch dependency-group: test-tools - dependency-name: io.netty:netty-handler dependency-version: 4.2.9.Final dependency-type: direct:production update-type: version-update:semver-patch dependency-group: test-tools - dependency-name: io.netty:netty-codec dependency-version: 4.2.9.Final dependency-type: direct:production update-type: version-update:semver-patch dependency-group: test-tools ... Signed-off-by: dependabot[bot] <support@github.com>
…_x/test-tools-f978810084
…_x/amazon-s3-87bef71f21 build(deps): bump the amazon-s3 group with 2 updates
(cherry picked from commit 8a18ca6)
(cherry picked from commit 14d4f4f)
(cherry picked from commit f22766a)
(cherry picked from commit 396e843)
[Port dspace-7_x] Fix NullPointerException in VersionHistoryService.getVersion() method
… not date filter (DSpace#10056) * 121661: [Issue#10055] startsWith on date browse should return items starting from that date, not with that exact year DSpace#10055 * 121661: [Issue#10055, PR#10056] Use upperBound year when sorting by startsWith date descending, so a year implies => up to end of year, month implied => up to end of month
[Port dspace-7_x] Type date browse startsWith functionality - input date as start date, not date filter
(cherry picked from commit 00e4979)
[Port dspace-7_x] Application hardening: filter requests for JSPs and other unwanted patterns
…ent-7_x [Port dspace-7_x] Curation I/O improvements
Applies the DSpace 7.6.5->7.6.7 delta onto the CLARIN fork dtq-dev as a two-parent merge with merge-base dspace-7.6.5. Resolves the 26 conflicting files per the section-4 recipes plus out-of-marker fixups: keep AWS SDK v1 S3 stack (aws-java-sdk-s3 @ 1.12.792 + s3mock), dedupe Swordv2IT depositItemWithEmbargo, bump json-smart pin to 2.6.0. Build-caught fixups (vanilla API changes / new tests vs CLARIN code): - AbstractCurationTask: re-added dereference(Context,String) helper that vanilla removed (inlined into perform()); CLARIN ItemMetadataQAChecker still calls it. - ItemServiceTest: updated findItemsWithEdit/countItemsWithEdit to the new String-query signature from PR DSpace#11953 (empty query preserves old behavior). - RequestItemRepositoryIT.testDeleteBitstreamRemovesRequestItem: reattach the bitstream/item to the session after the committed REST POST, so CLARIN updateItemFilesMetadata (run inside bitstream delete) sees an attached item. Validation: mvn install -P-assembly BUILD SUCCESS (all modules); VersioningTest 6/6, RequestItemRepositoryIT#testDeleteBitstreamRemovesRequestItem green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hib ITs - clarin-dspace.cfg: webui.content_disposition_inline allowlist (vanilla flipped blocklist->allowlist; text/plain, audio, tiff, csv would force-download). video/ogg is listed alongside audio/ogg because bitstream-formats.xml maps the .ogg extension to video/ogg, so audio/ogg alone would never match. - AuthenticationRestControllerIT: 2 new vanilla tests now use setAuthenticationMethodSequence() instead of unreliable setProperty() - test local.cfg: pin the vanilla allowlist for the test environment, otherwise the new CLARIN override reaches the ITs and breaks the upstream BitstreamRestControllerIT#checkContentDispositionOfFormats assertion on text/plain
Live rehearsal showed the previous wording ("must stay previewable inline") is
untrue for real file sizes: BitstreamRestController also enforces
webui.content_disposition_threshold, code default 8 MB, so most real audio/video
and large TIFF scans are sent as attachment despite being allowlisted.
Comment only - no value change. Raising the threshold widens the inline-render
surface and is left as a deliberate deployment decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OAI (regression introduced by taking 7.6.7, breaks CLARIN's VLO/OLAC harvesting): vanilla 7.6.7 added ItemUtils.sanitize() and applies it to every metadata value before the XOAI serializer escapes it again, so any value containing & < > " ' reaches harvesters double-escaped. Measured live on cmdi and olac: REST: u767xssprobe <script>alert(1)</script> end cmdi: u767xssprobe &lt;script&gt;alert(1)&lt;/script&gt; end sanitize() now removes characters XML 1.0 forbids instead of escaping, which is what its own javadoc says it does. Upstream defect - should be reported upstream. content-disposition: the previous allowlist restored inline for only 14 of the 96 registered formats; 7.6.5 allowed 90 (every image/* was inline unconditionally). Added the omitted formats LINDAT actually serves - WebVTT subtitles, AIFF/AU/MPEG audio, JPEG2000/WebP/AVIF/BMP images, Quicktime. Also added the parameterised "text/plain; charset=utf-8": the allowlist is matched by exact string equality, so the registry's extension-less "License" format could never match a bare "text/plain" token and license bitstreams always downloaded. Still deliberately absent: HTML/XML/TEI/SVG/JS. Threshold left at the 8 MB default. Verified: dspace-oai compiles, checkstyle clean, sanitize() behaviour unit-checked (metachars pass through, illegal C0/noncharacters removed, tab/LF/CR and CJK kept). Test config pin keeps the vanilla value, so upstream BitstreamRestControllerIT is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit replaced escapeXml10 with a regex to stop the double escaping, but escapeXml10 was an AggregateTranslator that ALSO ran UnicodeUnpairedSurrogateRemover, and the regex reproduced only the control-char part. Consequence: a lone surrogate in metadata (truncated 4-byte character, mojibake corpus) makes the StAX writer throw "Broken surrogate pair" inside XOAI.index(), and because that is caught per item the record is silently dropped from the OAI index. Measured before this fix: 2048 code units broke indexing, against 0 for vanilla 7.6.7. Verified over the whole BMP: 0 XML-1.0-illegal code units leak, 0 legal ones are removed, valid surrogate pairs survive, unpaired ones do not, and XML metacharacters still pass through unescaped. Known and intentional difference from escapeXml10: C0/DEL controls U+007F-U+009F are left as-is instead of being turned into &#NNN; entities. They are legal XML 1.0 characters (only XML 1.1 requires escaping them), so well-formedness is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7.6.7 rewrote S3BitStoreService from com.amazonaws (v1) to software.amazon.awssdk (v2 CRT client). The original plan rejected that and kept v1; the user reversed it: "if 7.6.7 is on AWS2, go to AWS2". v1 is past AWS end-of-support anyway. The fork owns three S3 classes vanilla does not know, so this was not a file takeover: - S3BitStoreService = vanilla 7.6.7 + exactly three CLARIN items (47 added lines): getFile(Bitstream), `protected` client so the subclasses can reuse it, and a configurable pathStyleAccessEnabled (vanilla hardcodes forcePathStyle(true) on an endpoint override; keeping the property preserves the fork's v1 behaviour and keeps bitstore.xml valid). getFile is byte-identical to the fork's own v9 implementation (5e930d1) so the branches do not diverge. - SyncS3BitStoreService - hand-ported, including the explicit multipart path. - S3DirectDownloadServiceImpl - presigning moved to S3Presigner, built from the same credentials/region/endpoint as the bitstore client. - S3BitStoreServiceIT - taken from 7.6.7 wholesale; the fork's copy was byte-identical to 7.6.5, so no CLARIN test was lost. LocalStack replaces s3mock, which only ever spoke v1. - ClarinS3BitStoreServiceIT - new. The CLARIN-only paths had no test at all. - bitstore.xml / assetstore.cfg - the merge had dropped vanilla's four new tuning keys (targetThroughputGbps, minPartSizeBytes, maxConcurrency, s3ChecksumAlgorithm). Two defects the new tests caught, neither visible by reading the code: 1. v1 returned part ETags unquoted, v2 returns them quoted. The multipart checksum comparison would have failed on every part, and since that path logs and swallows, completeMultipartUpload would never run - a silent upload failure. 2. SyncS3BitStoreService.put() never closed the FileOutputStream on the local assetstore copy. Pre-existing, not a port regression, but it leaks a handle per stored bitstream and made remove() silently fail to delete the local file. Evidence (local, this head): - mvn test-compile -DskipTests: 13/13 modules SUCCESS - S3BitStoreServiceIT Tests run: 15, Failures: 0, Errors: 0 - ClarinS3BitStoreServiceIT Tests run: 6, Failures: 0, Errors: 0 - S3DirectDownloadServiceTest Tests run: 8, Failures: 0, Errors: 0 - checkstyle 0 violations; `git grep com.amazonaws` finds nothing s3Store is lazy-init and nothing else instantiates it, so springWiringBindsStoreProperties loads the bean from bitstore.xml on purpose - it is the only check that the blank maxConcurrency binds to a null Integer and that s3ChecksumAlgorithm binds to the enum. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**Silent data loss on multipart upload.** `uploadPart` was handed a `BoundedInputStream`
over a `FileInputStream`, which cannot be reset, so the SDK refused to retry a part after
any transient error ("Request cannot be retried, because the request stream could not be
reset"). The v1 SDK passed the file plus an offset and retried freely. The failure then
landed in a catch that logged and swallowed, `completeMultipartUpload` never ran, and
`put()` carried on setting size and checksum - DSpace recorded a bitstream that S3 does
not hold, with no error anywhere.
Now uses the public `AsyncRequestBody.fromFile(FileRequestBodyConfiguration)` with
`position`/`numBytesToRead`, which is file-backed and therefore retryable; this also
removes the hand-rolled `openPart` helper and its executor. A failed upload now aborts the
multipart upload (parts were previously leaked and billed) and throws instead of returning
normally.
**The multipart loop was never executed more than once.** 50 MB part size against a 23-byte
fixture meant one iteration with offset 0, so the offset arithmetic and last-part handling
this port rewrote had no coverage at all. `uploadPartSizeBytes` is now settable and
`syncStoreUploadsInMultipleParts` uploads 5 MB + 4 KB across a real part boundary, verifying
the object byte for byte.
**A tripwire in `amazonClientBuilderBy` left all 21 S3 tests green.** Every test injects a
ready-made client, so `FunctionalUtils.getDefaultOrBuild` never called the supplier and
`endpointOverride`, `forcePathStyle`, `maxConcurrency` and the part sizes never ran - the
fork's headline delta, configurable `pathStyleAccessEnabled`, was verified only at its
getter. `clientBuilderAppliesEndpointAndPathStyle` builds a client through that method and
uses it against LocalStack. Verified it detects breakage: removing `endpointOverride` turns
it red, restoring it turns it green.
**Content-Disposition had two implementations and the newer one was wrong.** It escaped `"`
but not `\`, so a bitstream named `evil\` terminated the quoted string and swallowed
`filename*`; and it used `URLEncoder` directly, so a space arrived as `+`, which RFC 8187
reads literally - re-introducing the bug vanilla had just fixed. Extracted the correct
implementation to `org.dspace.util.ContentDispositionUtils`, now used by both call sites.
**Enabling direct downloads silently disabled inline preview.** The redirect hardcoded
`attachment`, discarding the disposition the allowlist and the 8 MB threshold had just
computed. `generatePresignedUrl` takes the caller's disposition; `BitstreamRestController`
passes the exact header the non-redirect path would have sent.
**Presigner region could diverge from the client's.** It hardcoded `us-east-1` while the
client leaves the region to the default provider chain when no credentials are configured,
so presigned URLs would be signed for the wrong region on an IAM role outside us-east-1. It
now mirrors `init()` branch for branch.
**A 403 was reported as "bucket absent".** `doesBucketExist` treated everything except
`NoSuchBucketException` as absent, so `init()` would try to create a bucket that already
exists; a least-privilege policy denies that and the assetstore comes up dead. Only 404 and
`NoSuchBucketException` now answer false. `doesObjectExist` likewise stops reporting a 403
or a timeout as "object does not exist" with the cause hidden at DEBUG.
**`s3ChecksumAlgorithm` was dead configuration.** `bitstore.xml` wires
`SyncS3BitStoreService`, which overrides `put()`, so the parent's `putObject` - the only
consumer - never ran. `uploadFluently` now applies it. Also `${assetstore.s3.endpoint:}` is
self-defaulting, matching vanilla.
Tests: S3BitStoreServiceIT 15, ClarinS3BitStoreServiceIT 8, S3DirectDownloadServiceTest 11,
ContentDispositionUtilsTest 9 - all 0 failures, checkstyle 0 violations. Maven does not
recompile edited sources reliably here, so every run used
`-Dmaven.compiler.useIncrementalCompilation=false`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ItemUtils.sanitize()` decides what VLO and OLAC receive, was changed twice in this PR, and had no test at all - a reviewer called it the highest-risk file in the upgrade for exactly that reason. `OAIPMHBundleExposureIT` already drove `ItemUtils.retrieveMetadata()`, so the harness existed; this uses the same serialisation path as `DSpaceXOAIItemCacheService.put()` and re-parses the result, because writing alone proves nothing - `U+FFFE` serialises happily and only fails on the harvester's parser. Four cases: metacharacters round-trip exactly once (the double-escaping 7.6.6 introduced); control characters and unpaired surrogates are removed rather than throwing inside `XOAI.index()`, which catches per item and would silently drop the record; supplementary characters and `U+1FFFE` survive while `U+FFFE` does not; and the output always re-parses. Verified non-vacuous: making `sanitize()` return its input unchanged turns three of the four red, with `WritingXmlException` and `SAXParseException` - the exact failure modes the fix exists to prevent. Two things the test itself established, both recorded in comments so nobody re-derives them: `U+0000` never reaches the sanitiser because PostgreSQL cannot store a NUL in a text column, and DSpace replaces C0 controls with a space before persisting, so the spacing around a removed control character is the persistence layer's doing and not something this test can assert on. Also fixes the import order `ContentDispositionUtils` was inserted at in `BitstreamByHandleRestController` (checkstyle). Tests: OAIPMHSanitizeIT 4, OAIPMHBundleExposureIT 3, 0 failures, checkstyle 0 violations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither dtq-dev nor dspace-7.6.7 ends this file with a newline. The upgrade merge added one while resolving the last conflict, so the file had an extra hunk against both sides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the 21 commits dtq-dev gained since the upgrade was cut, including 6114281 from dtq-dev-767-base. Conflicts: - LDAPAuthentication: both sides added the same e-mail fallback. Kept the upgrade side, which matches vanilla 7.6.7. The groupmap guard and debug logging from dtq-dev merged cleanly. - HttpHeadersInitializer: took dtq-dev, which escapes \ and " in the ASCII filename fallback. Without it a name with a quote ends the quoted-string early. - BitstreamByHandleRestController: took dtq-dev, which keeps its own copy of vanilla's fallback (transliterates diacritics) instead of the shared ContentDispositionUtils. Its ITs expect that output. ContentDispositionUtils stays for the S3 presigned-URL path. - BitstreamRestControllerIT: took dtq-dev (indent fix and the new quoted-name test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the dtq-dev merge the handle endpoint builds its own header (diacritics are transliterated), but the S3 redirect still used the shared builder (non-ASCII becomes _). Pass the endpoint's header to the presigned URL, as BitstreamRestController does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S3 sends this header itself, so Tomcat's clean-up of CR/LF does not apply. Replace control characters with spaces, as Tomcat would. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BitstreamRestController passes its Content-Disposition to S3, so a stub on the 4-argument overload never matches and returns null. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…6.7-sync-dtq-dev ItemExportServiceImpl: take the dtq-dev side, it already contains the 7.6.7 owning-collection null check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…6.7-sync-dtq-dev Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…repeatable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment said TEI and XML use the sanitized CLARIN preview. They don't: the preview only handles text and archives, so these files just download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
References
dtq-dev-767-base), with currentdtq-devmerged in and a few small fixes on top.Description
Moves the CLARIN-DSpace backend from vanilla 7.6.5 to 7.6.7, with the security fixes from both releases.
Instructions for Reviewers
List of changes in this PR:
81dac88c07merges thedspace-7.6.7tag with two parents. Only the 7.6.5 to 7.6.7 changes were applied, so the next vanilla release is a normal merge.ItemUtils.sanitize()from 7.6.6 escaped OAI values twice, socmdiandolacshowed&lt;. It now only removes characters that XML 1.0 forbids.clarin-dspace.cfggets a CLARIN allowlist (text, audio, TIFF, CSV). HTML, XML and SVG still download.dtq-devis merged in three times (b96b1ed04c,510a8e5a2b,8af46176b0).669c5c4cc7,e77c4622c9).submission-forms_cs.xmldoes the same now, so the Czech and English forms match (bae4f1493d).Before deploying:
DSPACE_REST_IMAGEin the dev-5 env file uses a tag this build updates.dspace database migrateskips them. Rundspace database migrate ignoredon each instance.V7.6_2025.10.29deletes request-a-copy rows of deleted bitstreams, so back uprequestitemfirst.dspace index-discovery -b.dspace checker -ais now-i. Update crons that use it.How to test:
.txtfile. Both show inline. An.htmlor.xmlfile downloads.cmdifor an item with<in the title shows<, not&lt;.The presigned S3 redirect was tested with mocks and LocalStack only, not with real S3.
Checklist
pom.xml), I've made sure their licenses align with the DSpace BSD License based on the Licensing of Contributions documentation.🤖 Generated with Claude Code