ostree-ext: override root dir metadata on merge commit by Rolv-Apneseth · Pull Request #2571 · bootc-dev/bootc

Rolv-Apneseth · 2026-10-15T13:27:44Z

When building the merge commit in write_merge_commit_impl(), write_dfd_to_mtree() inherits the root dir metadata from the temp checkout dir, which causes its mode to be 0o755 rather than 0o555.

Manually override it to uid=0, gid=0, and importantly mode=0o555. This matches the expectations of the filesystem RPM and systemd's /usr/lib/tmpfiles.d/root.conf.

Without this change, systemd-tmpfiles fails with fchmod() of / failed: Read-only file system on systems with a read-only root since systemd v261.

See coreos/fedora-coreos-tracker#2238

Assisted-by: llm

I've manually confirmed that this resolves the root permissions issues (0o755 -> 0o555) locally by comparing output from systems with bootc 1.17.1 to systems with this patch included, using both:

RPM overrides in cosa builds of Fedora CoreOS
Bootc builds using packages built using just build and run using just test-tmt-nobuild --preserve-vm readonly

Example output without patch:

[root@ibm-p8-kvm-03-guest-02 ~]# stat -c '%a' /
755
[root@ibm-p8-kvm-03-guest-02 ~]# ostree ls -d $(rpm-ostree status -b --json | jq -r '.deployments[0].checksum') /
d00755 0 0      0 /

Vs with the patch:

[root@ibm-p8-kvm-03-guest-02 ~]# stat -c '%a' /
555
[root@ibm-p8-kvm-03-guest-02 ~]# ostree ls -d $(rpm-ostree status -b --json | jq -r '.deployments[0].checksum') /
d00555 0 0      0 /

Of course, let me know if I'm missing anything or if any changes are required.

cgwalters · 2026-10-15T14:31:28Z

+/// mode=0o555, preserving any existing xattrs (e.g. SELinux labels).
+///
+/// This ensures the committed root directory permissions match the
+/// `filesystem` RPM and systemd's `/usr/lib/tmpfiles.d/root.conf`.


In bootc upstream we shouldn't have anything that is hardcoded about RPM or a specific distro/OS - while this is just a comment, I would make that clear.

That said! For this specific case again there's a HUGE detail in that in OCI images it's not required to include the root, and many don't!

More on that in coreos/fedora-coreos-tracker#2238 (comment) - please include direct links for that here.

So we should handle this - if the input OCI image has permissions for the root we should use them. If it doesn't, then let's match what composefs-rs is doing and take the permissions from the /usr directory.

(or of course...in theory we could look at the tmpfiles.d config now...but I am not sure I want to have parsing those be in the truly load bearing path)

I am not a fan of propagating hardcoding 0o555 in more places.

(Note contrast w/your rpm-ostree PR - there I think we can assume that the RPM content set includes metadata for the root)

I think here though...it is complicated and subtle in the true general case, but I'd just be happier if we hardcoded propagaing the perms of /usr (after unpacking all layers) instead to match composefs-rs.

Ah OK. Thanks for the breakdown. I will change the approach to propagate /usr perms instead then and we'll see how that looks.

The thing that's way easier on the composefs side is it always operates on an in-memory metadata tree so we can easily express "Option" to know whether or not we found metadata for the root (or any arbitrary directory) whereas for an actual kernel filesystem there's no concept of "unset permissions/mtime".

Right, yeah. I'm not sure how that should be handled.

I updated the approach to copy all the metadata like I believe https://github.com/composefs/composefs-rs/blob/main/crates/composefs/src/generic_tree.rs#L837 is doing. But I'm just now noticing you said perms of /usr specifically. Is that all we want propagated?

Let's do exactly the same as composefs-rs, all metadata: ownership, mtime, xattrs etc.

When building the merge commit in `write_merge_commit_impl()`, `write_dfd_to_mtree()` inherits the root dir metadata from the temp checkout dir, which causes its mode to be 0o755 rather than 0o555. Since systemd v261, this triggers a systemd-tmpfiles error on systems with read-only root: "fchmod() of / failed: Read-only file system". OCI images don't reliably include root directory metadata, so to match the approach used by composefs-rs [1], let's copy the metadata from /usr to the root. See: - coreos/fedora-coreos-tracker#2238 - coreos/fedora-coreos-tracker#2238 (comment) [1]: https://docs.rs/composefs/0.9.4/composefs/generic_tree/struct.FileSystem.html#method.copy_root_metadata_from_usr Assisted-by: llm Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>

cgwalters

LGTM that said I think a readonly tmt test that asserts / matches the rpm database would make sense in addition to a unit test.

Can be a followup

Rolv-Apneseth · 2026-10-15T20:10:53Z

LGTM that said I think a readonly tmt test that asserts / matches the rpm database would make sense in addition to a unit test.

Can be a followup

Yep that makes sense. I can do that in this one, just probably tomorrow

Rolv-Apneseth · 2026-10-15T20:15:39Z

I think the CI failures are #2572

Rolv-Apneseth · 2026-10-16T09:35:42Z

Actually, @cgwalters there's a problem with the new approach:

[core@cosa-devsh ~]$ rpm -ql --dump filesystem | grep -E '^/(usr |\s)'
/ 0 1768521600 0000000000000000000000000000000000000000000000000000000000000000 040555 root root 0 0 0 X
/usr 0 1768521600 0000000000000000000000000000000000000000000000000000000000000000 040755 root root 0 0 0 X

How would you suggest proceeding? Would you be OK with hard-coding the permissions after copying the other metadata from /usr?

cgwalters · 2026-10-16T12:33:17Z

Holy cow. This is...such a giant mess.

Look at https://src.fedoraproject.org/rpms/filesystem/blob/rawhide/f/filesystem.spec#_403 - why are the perms for /usr 0o755 but the perms for /usr/bin are 0o555? It seems just nonsensical.

OK @cgwalters-bot p0 task sol+luna worker - dig through the OCI container images of fedora, opensuse, debian, ubuntu, arch: unpack their OCI archives via skopeo copy to oci: dir to access teh raw tarballs see if these container images have a . directory at all, report its perms if present or not. But then, try to find the source code (filesystem rpm equivalent) for each OS and report what should be the perms of / from the perspective of the package system. Finally also "ground truth" crosscheck find a stock "cloud image" e.g. a thing with cloud-init, use virt-install's cloud init support to boot it and ssh in, and report the perms of that. Use bootc-dev/cgwalters-devspace-sandbox for CI runners.

The poll can now act on the operator's comments outside the repositories it follows. Today those comments are invisible to everything hosted (tracker#649; bootc-dev/bootc#2571 is one). With `operatorComments.repositories` set in the trusted config (off by default, 1 to 10 `OWNER/*` or `OWNER/REPO` entries), each pass reads up to three pages of one search, `commenter:OPERATOR updated:>=CURSOR`, and saves the cursor and page to resume at the first thread it is not done with. On a pull request the bot opened, the operator's new comments start a fix round, counted against its 2 rounds. On anything else they start a triage run. Either way the task quotes the comments word for word. Each run is claimed on its thread by the comment ids first, as fix rounds are, so a comment is never acted on twice, even if the state file's cursor is lost. A run GitHub refuses (a 4xx) is recorded on its claim or marker, without using the round, and tried again up to three times; a fix claim no marker followed within the run timeout is superseded. Trust and scope: only comments whose author is `operator` count. Runs go through the existing `--dispatch-workflow` and its per-pass limit of 3. With the setting on, though, `--apply` also posts claims and markers on threads in repositories it does not follow, and starts runs on them. Which repositories a fix run may push to is then bounded only by the tracker's dispatch workflow. Verified: `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked`, the three CI `cargo run` commands and `agentic-job handback scan` all exited 0. The new tests use a fake GitHub: fix round with the quote, triage, a non-operator comment ignored, no replay with or without the cursor, the wait/left bounds, 71 threads within the lag (40 at one time), an incomplete search, refused and uncertain dispatches, and a stale fix claim. Nothing was run against the live API. It is unverified that `org:` and `repo:` qualifiers OR together in issue search. Left: the tracker's `agent-board.json` entry (`bootc-dev/*`), and the short-status and review-site listing. Neither is in this repository; decision lines are this side of it. Self-review: 1 pass, no blocker. After it I made two narrowing fixes: a fix round only on pull requests the bot authored (a third party's PR in the bot's fork now gets triage), and a no-change exit in the fix brief. Follow-ups it left open: - A triage claim from a pass that died, or a dispatch whose outcome is unknown (a 5xx or lost response), leaves the comment handled with no run. The bot cannot label upstream. - While a thread is held, a pass reads at most three pages past it. - An upstream maintainer can edit the operator's comment before a pass reads it. - `</comment>` in a body closes the quote early. - Scope and `follow` matching is case-sensitive. - An older binary rejects a state file that has `operator_cursor`. Generated-by: AI

…king issue On a pull request the bot did not open, the poll now gives the operator's request a tracking issue in the dispatch repository, because the dispatch workflow refuses triage and research runs whose item is a pull request (bootc-dev/bootc#2571 hit this). The run takes that issue as its item and answers there; the claim on the pull request records it once the run is dispatched, so later comments are quoted on the same issue. A `/bot` or `@BOT` line asking for research or an investigation gets a `research` run in place of `triage`. Only a public repository's comments are copied to the tracker, so a private source is never published by it. Upstream admins can edit the bot's claims, so a mapped issue is reused only when it is open, in the tracker, opened by the bot and titled for this pull request; one that is gone is replaced. Verified with the CI fmt, clippy, test and CLI commands; nothing was run against live GitHub. Generated-by: AI

cgwalters-bot · 2026-10-16T14:34:55Z

Picked up: a research run on a RHEL 10 runner with KVM covers the container images, the packaging sources and booted cloud images. It reports in bootc-dev/cgwalters-devspace-sandbox#20.

Generated-by: https://github.com/cgwalters/#llms

cgwalters-bot · 2026-10-16T15:04:41Z

The research is done: bootc-dev/cgwalters-devspace-sandbox#20 (layer digests, source lines and image checksums are there). Short version:

Container layers. A ./ entry is present in openSUSE Tumbleweed, openSUSE Leap 16 and Debian 13 (0755, 0:0). It is absent in Fedora 44, Ubuntu 24.04 and Arch.
Packaging sources. Only Fedora's filesystem.spec gives / a mode, and that mode is 0555, with /usr 0755 and /usr/bin 0555. openSUSE and Arch set no mode for /. Debian and Ubuntu base-files run a recursive chmod 755 that includes ..
Why Fedora does it. A 2009 commit cites rhbz#517575: remove root's write permission so that capability-dropped uid-0 processes need CAP_DAC_OVERRIDE to write. Nothing found explains why /usr was left at 0755. Comment 3 on that bug (yours, 2013) already questioned the scheme, since files like /usr/bin/bash stay owner-writable.
Booted stock cloud images (virt-install and cloud-init, under KVM):
- Fedora Cloud 44: / 755, /usr 755, /usr/bin 555. So the booted / doesn't match the spec's 0555, and the image build step responsible wasn't identified.
- Debian 13 genericcloud: all three 755.

Generated-by: https://github.com/cgwalters/#llms

Rolv-Apneseth · 2026-10-16T15:51:11Z

Fedora Cloud 44: / 755, /usr 755, /usr/bin 555. So the booted / doesn't match the spec's 0555, and the image build step responsible wasn't identified.

Quite a mess indeed, but that one is particularly surprising

cgwalters · 2026-10-16T18:01:08Z

I think ultimately, it's still technically most correct for us to mirror the composefs-rs, because we have to handle OCI images without . regardless, and as the research shows many of them omit it.

Quite a mess indeed, but that one is particularly surprising

Yes...why the Fedora cloud image is 0o755 I don't know. But there's like, a whole giant chain of stuff involved after RPM is invoked, I mean, ultimately it could be a stray package post or it could be in Kiwi...

What boggles my mind more is - as far as I can tell, the Debian base-files package claims to own / but if there's a thing in there defining the perms of / I cannot find it obviously.

And what further is amazing, in dpkg (Debian 13) the expected metadata is not stored out of band, only md5 (!!!!) for file contents, so dpkg -V base-files has nothing to say if you chmod a dir.

Here's another data point, the official CentOS Stream 10 image in GCP is using 0o555.

OK it goes deeper - the filesystem RPM in OpenSUSE does not include / (wow!) - so what defines the permissions of it? Who even knows...

# grep ID /etc/os-release
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="16.0"
# rpm -qf /
file / is not owned by any package
#

Rolv-Apneseth · 2026-10-16T18:50:53Z

I think ultimately, it's still technically most correct for us to mirror the composefs-rs, because we have to handle OCI images without . regardless, and as the research shows many of them omit it.

As in, keep this as is, without forcing root permissions to 0o555? Or how do you suggest proceeding here?

Yes...why the Fedora cloud image is 0o755 I don't know. But there's like, a whole giant chain of stuff involved after RPM is invoked, I mean, ultimately it could be a stray package post or it could be in Kiwi...

What boggles my mind more is - as far as I can tell, the Debian base-files package claims to own / but if there's a thing in there defining the perms of / I cannot find it obviously.

And what further is amazing, in dpkg (Debian 13) the expected metadata is not stored out of band, only md5 (!!!!) for file contents, so dpkg -V base-files has nothing to say if you chmod a dir.

Here's another data point, the official CentOS Stream 10 image in GCP is using 0o555.

OK it goes deeper - the filesystem RPM in OpenSUSE does not include / (wow!) - so what defines the permissions of it? Who even knows...

Oh man...

cgwalters · 2026-10-16T19:45:16Z

I took this back to systemd/systemd#41431 (comment)

cgwalters · 2026-10-16T20:44:18Z

OK I think this PR will become correct in combination with https://src.fedoraproject.org/rpms/filesystem/pull-request/44 so let's merge it

github-actions Bot added the area/ostree Issues related to ostree label Oct 15, 2026

bootc-bot Bot requested a review from jeckersb October 15, 2026 13:28

Rolv-Apneseth mentioned this pull request Oct 15, 2026

compose: hard-code root directory permissions to 0o555 coreos/rpm-ostree#5644

Open

Rolv-Apneseth force-pushed the root-permissions branch from 5286ccc to a447789 Compare October 15, 2026 14:14

cgwalters requested changes Oct 15, 2026

View reviewed changes

Rolv-Apneseth force-pushed the root-permissions branch from a447789 to 7b970f0 Compare October 15, 2026 19:28

cgwalters approved these changes Oct 15, 2026

View reviewed changes

cgwalters-bot mentioned this pull request Oct 16, 2026

research: Permissions of / in distro container images, packaging sources and booted cloud images (bootc#2571) bootc-dev/cgwalters-devspace-sandbox#20

Open

Rolv-Apneseth mentioned this pull request Oct 16, 2026

systemd-tmpfiles fails to change permissions of / coreos/fedora-coreos-tracker#2238

Open

cgwalters mentioned this pull request Oct 16, 2026

Help users with incorrect / permission bits systemd/systemd#41431

Merged

cgwalters approved these changes Oct 16, 2026

View reviewed changes

Conversation

Rolv-Apneseth commented Oct 15, 2026

Uh oh!

cgwalters Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Rolv-Apneseth Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Rolv-Apneseth Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Rolv-Apneseth Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Rolv-Apneseth commented Oct 15, 2026

Uh oh!

Rolv-Apneseth commented Oct 15, 2026

Uh oh!

Rolv-Apneseth commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters-bot commented Oct 16, 2026

Uh oh!

cgwalters-bot commented Oct 16, 2026

Uh oh!

Rolv-Apneseth commented Oct 16, 2026

Uh oh!

cgwalters commented Oct 16, 2026

Uh oh!

Rolv-Apneseth commented Oct 16, 2026

Uh oh!

cgwalters commented Oct 16, 2026

Uh oh!

cgwalters commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Rolv-Apneseth commented Oct 16, 2026 •

edited

Loading

cgwalters commented Oct 16, 2026 •

edited

Loading