Conversation
Same shape as the Arsenal PoC: Yarn pinned via packageManager (with its integrity hash) rather than a committed release binary. - .yarnrc.yml: nodeLinker node-modules, Scality-scoped approvedGitRepositories, and networkConcurrency: 1 (Git dependencies with Yarn 1 lockfiles are bootstrapped by Yarn Classic, which races on its shared cache when run in parallel -- see YN0058 in Docker builds). npmMinimalAgeGate is left at Yarn's 1-day default. - Build scripts stay disabled globally; the 12 packages that need compiling are allowlisted via dependenciesMeta. Yarn 1 by contrast lets every package run install scripts. - fcntl carries no checksum: it is a Git dependency with a node-gyp build, so Yarn repacks it and the archive is not reproducible. Of the 17 Git-sourced entries it is the only unstable one -- publishing scality/node-fcntl to npm would remove the need for this.
- corepack enable must precede actions/setup-node: its cache: yarn probe shells out to Yarn to find the cache folder, and Yarn 1 refuses once packageManager is set (actions/setup-node#1027). - --frozen-lockfile -> --immutable, and drop --network-concurrency (rejected on the CLI; now a .yarnrc.yml setting). - Berry removed the global --silent flag, so 'yarn run --silent X' becomes 'yarn X'; Berry is quiet by default.
- yarn install --production exits 1 under Berry; the replacement is yarn workspaces focus --production, which needs no plugin. - workspaces focus cannot enforce lockfile immutability -- verified: with a deliberately stale lockfile and enableImmutableInstalls it still exits 0 and installs the drift. A preceding 'yarn install --immutable --mode=skip-build' restores that guarantee cheaply, since it resolves without building. - yarn global add was removed in Berry; nyc now comes from npm, matching the node-gyp/typescript installs already in this stage. - .yarnrc.yml has to be COPYed in for any of the above to apply. DockerfileMem is left alone: it is FROM node:6-slim, unreferenced anywhere, and last touched in 2022 -- Berry needs Node 18+.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
@@ Coverage Diff @@
## development/9.5 #6293 +/- ##
===================================================
- Coverage 86.54% 86.50% -0.05%
===================================================
Files 213 213
Lines 14615 14615
===================================================
- Hits 12649 12642 -7
- Misses 1966 1973 +7
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
- yarn global add was removed in Berry, and CI still used it for typescript in two jobs (tests.yaml:101 and :175). This is why every test job failed within ~20s. Now npm install -g, matching the Dockerfile. - Berry does not run pre/post lifecycle scripts for custom scripts. Verified: with pre/main/post defined, Yarn 1 runs all three and Berry runs only the main one. postcover writes the lcov report, so coverage was silently not being produced; CI now calls it explicitly. - The production image would not have started. It does COPY . (so it has package.json with packageManager: yarn@4.18.0) and runs CMD ["yarn","start"], but Corepack was only enabled in the builder stage, so the image's bundled Yarn 1 refused to run at all. Verified in a node:22.14.0-bookworm-slim container both ways: it fails without corepack enable and prints the script output with it, including against production-pruned node_modules.
.github/actions/setup-ci is used by 11 of the 14 test jobs and was never
migrated -- I had only grepped .github/workflows. It had no corepack
enable, used 'yarn global add', and passed --ignore-engines,
--frozen-lockfile and --network-concurrency, all of which Berry rejects.
This is why every functional-test job failed.
tests/functional/{report,raw-node,healthchecks} have a package.json but
are not workspaces, so Berry hard-errors on 'cd X && yarn test' ('the
nearest package directory doesn't seem to be part of the project').
That broke ft_management, ft_node, ft_node_routes, ft_gcp and
ft_healthchecks. They now invoke mocha directly, like the 20 sibling
ft_* scripts already do.
The production image enabled Corepack but did not install Yarn, so every
container start fetched the CLI over the network -- fatal for air-gapped
deployments. Yarn is now baked in, with COREPACK_HOME shared and
world-readable so images/federation (which runs as 'scality') can use
it. Verified offline as a non-root user.
Also: drop aws-sdk, es5-ext, level and protobufjs from the build
allowlist (their install scripts only print notices), disable telemetry,
and add the missing !.yarn/releases gitignore exception.
…ists Telemetry was disabled on review advice, but that was not asked for and Berry's default is fine here. approvedGitRepositories is not optional hardening as the comment implied: Yarn's default is an empty allowlist, which blocks every Git fetch. Verified by removing it -- the install fails with YN0080 on fcntl. The real choice is only between Yarn's migration-written "**" and a scoped list. All 11 Git resolutions here use the .git clone form, which is the path that enforces the allowlist, so scoping it to Scality is meaningful rather than decorative.
lint: Berry passes `--` through to the script rather than stripping it,
so `yarn lint -- --max-warnings 0` made eslint treat --max-warnings as a
filename ("No files matching the pattern"). Yarn 1 stripped it.
utapi-v2-tests: utapi requires levelup, memdown and encoding-down but
declares none of them. Yarn 1's hoisting happened to put levelup@4 within
its reach; Berry hoists levelup@0.19, and utapi then calls the v4 API
against it, so the server never starts and the job times out waiting for
port 8000. packageExtensions declares them on utapi's behalf, with the
same ranges as scality/utapi#1336 so the lockfile is untouched.
utapi 8.2.7 declares levelup, memdown and encoding-down itself (scality/utapi#1336, UTAPI-126), so Cloudserver no longer has to declare them on its behalf. Verified: levelup still resolves to 4.4.0 with the block removed, and the server starts with ENABLE_UTAPI_V2=t, which is what utapi-v2-tests was timing out on. The bump spans 8.2.4..8.2.7, so it also brings UTAPI-112, UTAPI-122 (which rolls @senx/warp10 back to 1.0.14) and UTAPI-123 -- a git tag cannot be cherry-picked.
Contributor
Author
|
Superseded by #6306, which carries the same work from a branch with a ticket (CLDSRV-1004), restructured into 6 atomic commits and with all review feedback from this PR applied. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / WIP — not for merge. Second PoC for ZENKO-4051 (open since 2022), after Arsenal#2706. Cloudserver was chosen because it exercises what a library could not: a Docker image build, a production install, and 7 Git dependencies.
Measured: Berry is ~2x faster for developers
Same machine, same cold/warm state, both serialised (Yarn 1 already used
--network-concurrency 1everywhere):The warm number is the day-to-day one — every branch switch or dependency change. There is further upside: Berry at default concurrency was 107 s cold, so once the dependency repos are themselves on Berry, cold approaches 3.4x.
Security posture improves
Berry disables build scripts by default. The 12 packages that genuinely need compiling are allowlisted in
dependenciesMeta; under Yarn 1 every package may run install scripts.yarn installsilently wrotenpmMinimalAgeGate: 0andapprovedGitRepositories: ["**"]into.yarnrc.ymlduring migration — both reverted. Re-read that file after migrating, before committing.What Docker forced us to learn
yarn install --productionexits 1 under Berry — not deprecated-but-working, broken. Replacement isyarn workspaces focus --production(no plugin needed).workspaces focuscannot enforce lockfile immutability. Verified: with a stale lockfile andenableImmutableInstalls: trueit still exits 0 and installs the drift. A precedingyarn install --immutable --mode=skip-buildrestores what--frozen-lockfilegave us, without building anything.yarn global addwas removed;nycnow comes from npm.networkConcurrency: 1is load-bearing. Dropping Yarn 1's flag as obsolete was wrong. Git dependencies that still carry Yarn 1 lockfiles get bootstrapped by Yarn Classic, and concurrent Classic installs corrupt each other on its shared cache (YN0058, ENOENT while unpacking). This only reproduces in Docker. Berry rejects the CLI flag but honours the setting. It can be dropped once those repos move to Berry.Checksums
Of 17 Git-sourced lockfile entries, exactly one is unstable across a cold reinstall:
fcntl. It is node-gyp-built, so Yarn repacks it and the archive is not byte-reproducible. Deps that merely runtscinprepare(arsenal,scubaclient,utapi) are perfectly stable.Its checksum is therefore omitted. This is not durable — Yarn re-adds one on every install, so any developer who commits after installing reintroduces a machine-specific hash. The real fix is to publish
scality/node-fcntlto npm asfcntl@0.3.0: the repo is public and Apache-2.0, the bare name is unclaimed, andpackage.jsonalready declaresname: "fcntl", so consumers change one line with no code change and no registry auth. Only Arsenal and Backbeat consume it.State
Verified locally: install,
--immutable, Docker image build (--target builder), eslint (0 errors), prettier. The test suite has not been run locally — CI is the check for that.DockerfileMemuntouched:FROM node:6-slim, unreferenced anywhere, last modified 2022, and Berry needs Node 18+.