Repository navigation
chore(ci): manual workflow to deploy build images and AMIs - #25606
Closed
spalladino wants to merge 3 commits into
Closed
spalladino wants to merge 3 commits into
spalladino wants to merge 3 commits into
Conversation
charlielye
requested changes
Oct 6, 2026
charlielye
left a comment
charlielye
left a comment
Contributor
There was a problem hiding this comment.
i think we can just run ./build-images/bootstrap.sh deploy
the result will update two files with the new ami ids.
we'd then want to push those changes back to branch we ran on.
spalladino
changed the title
Contributor
Author
|
Thanks for the review @charlielye, updated |
spalladino
added a commit
that referenced
this pull request
Oct 8, 2026
## Summary Moves the pinned Foundry toolchain from 1.4.1 to 1.8.5 (latest release, published 2026-10-05) everywhere the repo pins it, adapts `l1-contracts` to the behaviour changes shipped in Foundry 1.5 through 1.8, targets the Osaka EVM that mainnet runs, and introduces build image tag `3.1` so this toolchain can be published without disturbing `next`. Pins updated: - `bootstrap.sh` (`expected_abs_foundry_version`, which the toolchain check greps from `forge --version` / `anvil --version`) - `build-images/src/Dockerfile` (`FOUNDRY_VERSION`, the build/devbox image) - `scripts/setup-container.sh` ## Adaptations required by the new toolchain - **`forge fmt` output changed (Foundry 1.5 formatter).** 48 Solidity files under `l1-contracts` reformatted with `forge fmt` 1.8.5 so `forge fmt --check` passes. Every change is whitespace-only: stripping all whitespace from each file before and after yields identical token streams (checked for all 48 files). - **`foundry.toml`: `isolate = false`.** Foundry 1.8 runs forge tests with `--isolate` by default, which executes each top-level call as its own transaction. That charges cold-access and intrinsic gas per call, which pushed the large validator-set tests (`RollupGetters`, `tmnt333`, `flushEntryQueue` fuzz) past the gas limit, and failed the `getCurrentEpochCommittee` gas-growth assertion. Pinning it off keeps the semantics the suite was written against. It does not affect the committed gas reports: a gas report (`FORGE_GAS_REPORT=true`) isolates every call whatever this is set to, and the same report is byte-identical with it on or off. Regenerating `gas_report.json` under 1.8.5 reproduces every committed `min`/`max`; only fuzz-driven call counts and averages drift (the 1.5 fuzzer samples inputs differently even with `--fuzz-seed 42`). `gas_benchmark.md` regenerated under 1.8.5 is identical to what `next` produces under 1.4.1, and `partial_epoch_proof_gas_report.json` reproduces exactly. - **`foundry.toml`: `evm_version = 'osaka'`.** Mainnet runs the Osaka EVM (Fusaka activated; the current fork reported by `eth_config` is the second BPO fork of 2026-01-07, with no next fork scheduled), and Foundry 1.7+ defaults forge and anvil to Osaka. Building with the Osaka target produces byte-identical creation and runtime code for all 125 `src/` contracts (solc 0.8.30 emits no Osaka-only opcodes here), the full suite passes, the Rollup gas report matches on every `min`/`max`, and nothing in `src/` uses the modexp precompile that EIP-7883 reprices. solc 0.8.30 still labels `osaka` experimental in its help text; it is the chain we deploy to, so the tests should simulate it. - **`foundry.toml`: dropped `variable_override_spacing`.** Not a recognised `[fmt]` key in 1.8.5 (every forge invocation warned about it); `override_spacing = false` on the next line is the key that is honoured. - **`test/governance/governance/tmnt331.t.sol`.** `Governance__CallerCannotBeSelf()` declares no parameters, but the test encoded an `address` argument into its `expectRevert` data. Older forge matched that leniently; 1.8.5 compares revert data exactly, so the test now encodes the selector alone. - **`scripts/forge_broadcast.js`.** `forge script` 1.8.5 rejects `--batch-size` (`error: unexpected argument '--batch-size' found`), which broke `scripts/test_rollup_upgrade.sh`. The wrapper now passes `--slow` (send the next tx only after the previous one is confirmed) in the automining-anvil case where it previously used `--batch-size 1`, and leaves forge's default batching elsewhere. Real-chain deploys therefore no longer send in batches of 8; there is no flag left that reproduces that. If strictly sequential live deploys are preferred, pass `--slow` unconditionally. - **`l1-contracts/bootstrap.sh build_src` and `barretenberg/sol/bootstrap.sh build_sol` no longer run `forge install`.** Under 1.8.5 `forge install` syncs every submodule of the repository to `foundry.lock` and exits 1 when the lock names a revision a shallow clone does not have. Both lock files list `l1-contracts/lib/circuits`, `labs` and `noir/noir-repo` behind the revisions the repository records, so the cache-miss build failed before compiling anything (1.4.1 exits 0 and touches nothing). The `git submodule update --init --recursive ./lib` that follows already checks the libraries out at the recorded revisions. - **`l1-contracts/bootstrap.sh build_verifier` ends with a full `forge build`.** Under 1.8.5, running `forge test` after an explicit-path `forge build` so that forge incrementally compiles only the handful of leftover files (`shouting.t.sol`, `test/script/*`, scripts) produces inconsistent artifacts: 3 of 3 such runs failed 177 tests with `EvmError: Revert` at the same ~2.93M gas point in every Rollup-deploying suite. A full build before the artifacts are uploaded means `forge test` has nothing left to compile; that sequence passed every time. ## Build image `3.1` - `build-images/bootstrap.sh` `version`, `build-images/run.sh`, `.devcontainer/dev/devcontainer.json`, `ci3/bootstrap_ec2`, `ci3/docker_isolate`, `ci3/aws/ami_update.sh` and `ci3/tests/signal_test` now reference `3.1`. CI runners boot from an AMI with `3.0` already cached and `bootstrap_ec2` runs `docker run aztecprotocol/devbox:<tag>` without pulling first, so a re-pushed `3.0` would never reach them and would break every other PR on `next` (whose checkout pins 1.4.1). A fresh tag is pulled on first use and leaves `next` untouched. - `build-images/bootstrap.sh build_ec2` and `ci3/aws/ami_update.sh` are ported to the current `aws_request_instance` contract (state directory, `KEY_NAME`, `aws_terminate_instance <state_dir>`); they still called the pre-March-2026 three-argument form and failed with `state_dir: unbound variable` before doing anything. The port was exercised end to end against stubbed `aws`, `ssh` and `docker` commands: it exits 0, requests and terminates four instances, pushes the manifests and writes both AMI id files, where the scripts as they are on `next` fail with `$4: unbound variable` before requesting anything. It has not been run against real AWS. **To go green, the `3.1` images (`build`, `devbox`, `sysbox`, amd64 + arm64 + manifest) must be published to Docker Hub.** #25606 adds a workflow that runs this branch's own deploy script; a laptop runbook is in the PR thread as a fallback. Refreshing the AMIs (`ci3/aws/ami_update.sh`, commit the new `ci3/aws/ami_id_*`) only speeds up cold starts and can follow later from the CI AWS account. A mainframe admin also needs to point `/usr/local/bin/launch_sysbox` at `sysbox:3.1`. ## Verified locally with Foundry 1.8.5 binaries | Check | Result | | --- | --- | | `forge fmt --check` | clean | | `forge build` (bootstrap's file set) | compiles | | `forge test` (full suite, Osaka target) | 1383 passed, 0 failed, 3 skipped (1386) | | `scripts/test_rollup_upgrade.sh` (anvil 1.8.5 + forge script broadcast) | completed successfully | | `scripts/check_contract_sizes.sh` (both profiles, Osaka target) | within EIP-170 | | `solhint` | unchanged warnings only | | Bytecode, Osaka vs Prague target | identical for all 125 `src/` contracts (metadata aside) | ## Things to know - `forge build` now emits ~245 non-fatal `forge-lint` warnings because 1.8 added many detectors. A stacked PR on this branch brings that to zero without changing bytecode. - The existing CI command `forge test --match-contract MerkleCheck --ffi` matches no contract on `next` (no `MerkleCheck` contract exists under `test/`); this predates the bump and still exits 0. - The anvil-related skip of `uniswap_trade_on_l1_from_l2.test.ts` in `.test_patterns.yml` is untouched. --- *Created by [claudebox](https://claudebox.work/v2/sessions/f38155cc1541e786/jobs/1) · group: `slackbot` · requested by Mike (@iAmMichaelConnor) · [Slack thread](https://aztecfoundation.slack.com/archives/D0B2N7W1WJD/p1791282577517869?thread_ts=1791282577.517869&cid=D0B2N7W1WJD)* --------- Co-authored-by: Charlie <5764343+charlielye@users.noreply.github.com> Co-authored-by: Santiago Palladino <santiago@aztec-labs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a manual
Publish build imagesworkflow that runs./build-images/bootstrap.sh deployfor a given branch from CI: it buildsaztecprotocol/build,devboxandsysboxon EC2 for amd64 and arm64, pushes them and the multi-arch manifests to Docker Hub, bakes new CI AMIs, and commits the newci3/aws/ami_id_{amd64,arm64}back to that branch.First use: publish tag
3.1for #25602 (Foundry 1.8.5).How it works
resolve(no secrets): resolves the branch to its head SHA, reads the tag frombuild-images/bootstrap.sh, and fails ifci3/aws/ami_update.shcaches a different tag.deploy(environment: master): first checks Docker Hub (refuses to overwrite an existing tag unlessallow_overwrite; inmode=amis-onlyrequires the tag to exist). The check lives here so "Re-run failed jobs" cannot republish a tag; after a failed AMI step, dispatch again withmode=amis-only. Then AWS via the existing OIDC role, the build-instance SSH key decoded as in.github/ci3.sh, andbootstrap.sh deploy(orupdate_amis). AMI ids are uploaded as an artifact on success.commit-amis: validates the AMI ids and pushes a commit to the branch with the bot token. It runs no script from the branch, and the push is non-force from the resolved SHA, so it is rejected if the branch moved.next, which is also what themasterenvironment's branch policy allows.Security trade-off
The target branch's own scripts run with the Docker Hub credentials, the CI AWS role and the build-instance SSH key. Today anyone who can dispatch workflows on
nextcould use that. Follow-up: move it to a dedicated environment with required reviewers.Needs checking before the first run
sg-0ccd4e5df0dcca0c9(used in SSH mode) allows port 22 from GitHub-hosted runners.ec2:CreateImageandec2:DescribeImages;ami_update.shloops on the image waiter until the 120-minute job timeout if it does not.masterenvironment'sDOCKERHUB_PASSWORDbelongs toaztecprotocolci, whichbuild-images/bootstrap.shhardcodes.build_ec2/ami_update.shas ported in chore: bump Foundry to 1.8.5 #25602 have not run yet; watch the first dispatch.Follow-ups