Repository navigation
Conversation
|
The "Ensure PR format" check is only waiting for a label. Could a maintainer add |
| <!-- Pre-built JNI libraries that release CI puts in jni/ before | ||
| building the release artifacts. They are binaries, not source, | ||
| and are several hundred MB. See GH-743. --> | ||
| <exclude>jni/**</exclude> |
There was a problem hiding this comment.
This covers the extracted jni/ directory, but the binaries job in rc.yml also leaves the compressed tarballs at the repo root and never deletes them:
mv artifacts/*/*.tar.gz .
tar -xf apache-arrow-java-*.tar.gz --strip-components=1
tar -xvzf jni-linux-x86_64.tar.gz
...
So they are still packaged. From the published 19.0.0 zip:
61.6 MB jni-linux-aarch_64.tar.gz
66.0 MB jni-linux-x86_64.tar.gz
36.5 MB jni-macos-aarch_64.tar.gz
44.8 MB jni-macos-x86_64.tar.gz
14.6 MB jni-windows-x86_64.tar.gz
With the current matrix, the next RC zip would still carry about 100MB of the same native libraries, just tarred. The local test with a fake jni/ directory doesn't reproduce the CI layout, which is why this didn't show up.
NB: I don't think we should consider that for 20.0.0, we can do the cleanup after this release.
| <directory>.</directory> | ||
| <outputDirectory></outputDirectory> | ||
| <useDefaultExcludes>true</useDefaultExcludes> | ||
| <excludes> |
There was a problem hiding this comment.
The same job leaves a few other non source files in the working directory. They are also in the 19.0.0 zip and are not covered here:
apache-arrow-java-19.0.0.tar.gz: the git archive source release, nested inside this oneartifacts/release-source/*.sha256and*.sha512, plus emptyartifacts/jni-*/directories- an empty
binaries/directory: the dist dirjni_full_build.shcreates before running Maven
We could add an exclude for each of these, but then the list has to track whatever the CI job happens to leave behind.
| <!-- Use our own descriptor instead of the Apache parent's | ||
| "source-release" one so that pre-built JNI libraries | ||
| in jni/ aren't included in the source release. --> | ||
| <descriptorRefs combine.self="override"/> |
There was a problem hiding this comment.
Rather than forking the descriptor, I would fix this where the files come from. The source-release descriptor zips the whole working directory, and the binaries job unpacks its downloads into that same directory. Two options:
- Build from a clean subdirectory in
rc.yml, so downloads and outputs sit next to the source tree instead of inside it:
mv artifacts/*/*.tar.gz .
mkdir arrow-java
tar -xf apache-arrow-java-*.tar.gz -C arrow-java --strip-components=1
tar -xvzf jni-linux-x86_64.tar.gz
...
then run ci/scripts/jni_full_build.sh arrow-java jni binaries, checkout apache/arrow-testing to arrow-java/testing, and point "Prepare docs" at arrow-java/target/site/apidocs. That handles the tarballs, artifacts/, binaries/ and the nested source tarball in one go.
- Set
<skipAssembly>true</skipAssembly>on this execution, asbom/pom.xmlalready does. The source release we vote on and publish to dist.apache.org is thegit archivetarball.
Either way we drop dev/release/source-release.xml. Otherwise we would have to keep it in sync with the upstream descriptor on every org.apache:apache parent bump. 19.0.0 already leaked .mvn/.develocity/ because of that kind of drift.
What's Changed
The
apache-releaseprofile buildsarrow-java-root-<version>-source-release.zipwith the Apache parent'ssource-releaseassembly descriptor, which packs the whole working directory. In the release CI (ci/scripts/jni_full_build.sh . jni binaries) the pre-built JNI libraries are unpacked intojni/in the source tree before Maven runs, so they end up in that zip (~870 MB uncompressed for 18.3.0; the 19.0.0 zip is still ~455 MB).This adds
dev/release/source-release.xml, a copy of thesource-releasedescriptor fromapache-source-release-assembly-descriptor1.8 (the version used by the Apache parent POM) with one extra exclude for the top-leveljni/directory, and points the root POM'ssource-release-assemblyexecution at it. Nothing else about the zip changes;jni/directories undersrc/(Java packages) are not affected. I lefttesting/in, since it is needed to run the tests from the source release.Testing: I created a fake
jni/arrow_dataset_jni/x86_64/libarrow_dataset_jni.so(5 MB) in the checkout and ranmvn -pl .,bom -Papache-release package -Dgpg.skip=true -DskipTests ...:main: the zip is 8.5 MB, 2077 entries, and containsjni/arrow_dataset_jni/x86_64/libarrow_dataset_jni.sojni/entriesmvn -pl .,bom -Pspotless spotless:checkpasses.AI assistance disclosure: this change and description were prepared with the help of an AI coding assistant; the checks above were run locally.
Closes #743.