Enforce guest RLIMIT_AS and RLIMIT_DATA by xalestar · Pull Request #485 · sysprog21/elfuse

xalestar · 2026-11-06T18:23:46Z

Intent: brk, mmap, mremap and mprotect hold a guest to its RLIMIT_AS and RLIMIT_DATA, and fail with ENOMEM past them, as Linux does. Follow-up 3 of #448.

Failure fixed: on main, both limits are stored and never checked. With RLIMIT_AS 24 MiB above VmSize, two 16 MiB mmaps both succeed. With RLIMIT_DATA at 24 MiB, sbrk(8 MiB) then sbrk(32 MiB) both succeed. On Linux 6.18 the second call fails with ENOMEM.

This PR:

Counting: mem_vm_totals sums two totals from the region table under mmap_lock: all mapped pages (RLIMIT_AS) and private writable pages (RLIMIT_DATA). Each page counts once, and the shim's infra reserve is left out. [stack] counts at the 8188 KiB elfuse maps up front, where Linux counts 132 KiB right after exec, so a limit sized from a Linux run leaves about 8 MiB less room (Charge [stack] at its grown extent, as Linux does #492).
mmap and mremap: growth past either limit fails with ENOMEM, as may_expand_vm does. MAP_FIXED is charged only its net growth, as mmap_region does. A MREMAP_FIXED move is checked with its destination still counted, and a refused move leaves the destination mapped, as Linux does since 6.17.
Error order: when a request exceeds a limit, EBADF, EEXIST and EACCES still come first, in do_mmap's order.
brk: the page-aligned growth is checked, and the break extent is held to the soft RLIMIT_DATA whatever the heap's protection, as check_data_rlimit does. The data segment that Linux adds to that extent is not tracked (brk does not count the data segment toward RLIMIT_DATA #487).
mprotect: adding PROT_WRITE to a private mapping fails with ENOMEM at the first mapping that RLIMIT_DATA refuses, as mprotect_fixup does. The part below it is changed in one call, which does not stop at a hole or a refused mapping as Linux does (mprotect changes a whole range instead of stopping at the first failing mapping #489).
RLIMIT_DATA of 0: a soft limit of 0 falls back to the hard limit, except for brk.
SysV shm: an attach is mapped shared, as on Linux, so it counts toward RLIMIT_AS only. Its region is named /SYSV<key> (deleted) as on Linux and carries a serial that no other region has, so it never merges with another mapping. A fork child shares it with the parent when the attach starts on a host page (16 KiB on Apple Silicon) and no other mapping shares its last host page; otherwise the child gets a private copy, as before. shmdt finds the attach from the mappings, as Linux does, so an attach moved with mremap is detached at its new address; it writes back and unmaps only the parts still mapped, and fails with EINVAL when none is there. A shmdt that finds only part of an attach leaves the rest attached until it too is gone, as Linux does. An attach the guest unmapped is detached at the next shmat or shmdt, and fork children hand out serials past the inherited ones. A fork child's shmdt still fails with EINVAL, as it did before (SysV shmdt fails with EINVAL in a fork child #490).
Left for later: running counters instead of the table scan (RLIMIT_AS and RLIMIT_DATA totals lose records once the region table is stale #488).

Verified on a430b30 (same tree as 66367a0), based on 170b5df, against the Linux 6.18 oracle:

make check-format passes. make check: 81 passed, 1 failed, 2 skipped; the failure is nslookup timing out on live DNS, which times out the same way on 3115982. make test-matrix-elfuse-aarch64: 318 passed, 0 failed, 10 skipped. make test-matrix-qemu-aarch64 : 292 passed, 0 failed, 35 skipped.
New test-rlimit-mem: 3 of 3 pass under elfuse and on Linux 6.18.60. Steps 8 and 12 expect the MREMAP_FIXED order of 6.17 and later, so a kernel before 6.17 fails them (seen on 6.12.115); the CI qemu lane still boots such a kernel, see CI qemu lane boots stale fixtures, not Linux 6.18 #491. On 170b5df, all 3 fail. Each step added in review fails with its fix reverted or on the commit before it, except step 14 of data_child (a fork child shares a SysV attach), which 170b5df never reaches; a probe of it on 170b5df shows the child getting a private copy.
Not tested: the EEXIST and EACCES order on the high-VA MAP_FIXED path, which runs only under Rosetta.

cubic-dev-ai

2 issues found across 6 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/syscall/mem.c">

<violation number="1" location="src/syscall/mem.c:2699">
P1: Both new limit helpers trust incomplete region totals after `regions_tracker_stale` is set, allowing later mmap, mremap, brk, or mprotect requests to bypass finite RLIMIT_AS/RLIMIT_DATA limits. Refuse finite-limit checks while the tracker is stale instead of treating the incomplete totals as authoritative.

(Based on your team's feedback about refusing stale tracker limit checks.)</violation>
</file>

<file name="tests/test-matrix.sh">

<violation number="1" location="tests/test-matrix.sh:759">
P3: This adds one unconditional test pass to both aarch64 lanes, but the recorded `EXPECTED_BASELINES` floors are unchanged. Raise them to 256 and 231 respectively so the matrix inventory and baseline accounting stay synchronized.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic}

cubic-dev-ai · 2026-11-06T18:29:14Z

+                              uint64_t replace_start,
+                              uint64_t replace_end)
+{
+    uint64_t as_limit = sys_rlimit_cur(LINUX_RLIMIT_AS);


P1: Both new limit helpers trust incomplete region totals after regions_tracker_stale is set, allowing later mmap, mremap, brk, or mprotect requests to bypass finite RLIMIT_AS/RLIMIT_DATA limits. Refuse finite-limit checks while the tracker is stale instead of treating the incomplete totals as authoritative.

(Based on your team's feedback about refusing stale tracker limit checks.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/syscall/mem.c, line 2699: <comment>Both new limit helpers trust incomplete region totals after `regions_tracker_stale` is set, allowing later mmap, mremap, brk, or mprotect requests to bypass finite RLIMIT_AS/RLIMIT_DATA limits. Refuse finite-limit checks while the tracker is stale instead of treating the incomplete totals as authoritative. (Based on your team's feedback about refusing stale tracker limit checks.) </comment> <file context> @@ -2654,6 +2655,129 @@ static bool heap_piece_is_plain(const guest_region_t *r) + uint64_t replace_start, + uint64_t replace_end) +{ + uint64_t as_limit = sys_rlimit_cur(LINUX_RLIMIT_AS); + uint64_t data_limit = data_rlimit(); + if (as_limit == LINUX_RLIM_INFINITY && data_limit == LINUX_RLIM_INFINITY) </file context>

Not changed, by design. Refusing every request with a finite limit once regions_tracker_stale is set would fail every later mmap, mremap and brk of the guest until execve, even ones far under the limit, and the table goes stale from a split that finds 4096 entries in use, which a JIT that splits with mprotect can reach. #435 tried the refusal (3c1bbd4) and took it out for that reason. With a stale table the totals are short by the lost records, so a limit can be passed by at most those records; mem_rlimit_allows still refuses everything beyond them.

cubic-dev-ai · 2026-11-06T18:29:14Z

    test_check "$runner" "test-cloexec" "PASS" "$bindir/test-cloexec"
    test_rc "$runner" "test-exec-limits" 0 "$bindir/test-exec-limits"
    test_rc "$runner" "test-rlimit-fsize" 0 "$bindir/test-rlimit-fsize"
+    test_rc "$runner" "test-rlimit-mem" 0 "$bindir/test-rlimit-mem"


P3: This adds one unconditional test pass to both aarch64 lanes, but the recorded EXPECTED_BASELINES floors are unchanged. Raise them to 256 and 231 respectively so the matrix inventory and baseline accounting stay synchronized.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test-matrix.sh, line 759: <comment>This adds one unconditional test pass to both aarch64 lanes, but the recorded `EXPECTED_BASELINES` floors are unchanged. Raise them to 256 and 231 respectively so the matrix inventory and baseline accounting stay synchronized.</comment> <file context> @@ -756,6 +756,7 @@ run_unit_tests() test_check "$runner" "test-cloexec" "PASS" "$bindir/test-cloexec" test_rc "$runner" "test-exec-limits" 0 "$bindir/test-exec-limits" test_rc "$runner" "test-rlimit-fsize" 0 "$bindir/test-rlimit-fsize" + test_rc "$runner" "test-rlimit-mem" 0 "$bindir/test-rlimit-mem" test_rc "$runner" "test-session" 0 "$bindir/test-session" test_check "$runner" "test-pidfd" "0 failed" "$bindir/test-pidfd" </file context>

Not changed. EXPECTED_BASELINES is a floor well under the live count, not a count: it reads 255 and 230, while the two lanes pass 318 and 292 on this branch. Raising it by one would leave it as far from the real numbers as it is now.

jserv · 2026-11-07T04:15:18Z

+ *
+ * Returns true when the request fits both limits. Caller holds mmap_lock.
+ */
+static bool mem_rlimit_allows(const guest_t *g,


Follow-up: once regions_tracker_stale is set, the dropped records fall out of both totals for good, so a guest that pushes the table to GUEST_MAX_REGIONS and then splits mappings can grow past either limit. Running vm_total and vm_data counters, updated wherever a mapping is added, removed or reprotected and kept even when the table has no free slot, would close that gap and also replace the full-table scan on every mmap under a finite limit.

Agreed, left for a follow-up: once regions_tracker_stale is set, the dropped records stay out of both totals, and running vm_total/vm_data counters would close that and drop the scan. Tracked in #488.

jserv · 2026-11-07T04:15:21Z

+     * limit of 0 is not for brk.
+     */
+    uint64_t data_cur = sys_rlimit_cur(LINUX_RLIMIT_DATA);
+    if (data_cur != LINUX_RLIM_INFINITY && new_off - g->brk_base > data_cur)


Follow-up: check_data_rlimit adds end_data - start_data, and the comment above says that extent is not tracked. The ELF loader could record the writable PT_LOAD bounds at load and exec, and fork could carry them over. Then brk is refused at the same break Linux refuses, not one data segment later.

Agreed, left for a follow-up: #487 tracks recording the writable PT_LOAD bounds at load and exec, and carrying them through fork.

jserv · 2026-11-07T04:50:10Z

+     * limit of 0 is not for brk.
+     */
+    uint64_t data_cur = sys_rlimit_cur(LINUX_RLIMIT_DATA);
+    if (data_cur != LINUX_RLIM_INFINITY && new_off - g->brk_base > data_cur)


As a follow-up: check_data_rlimit adds end_data - start_data, and nothing in guest_t records it, so a binary with a large .data/.bss gets more break than Linux allows. Recording the writable segment extent at ELF load time would close this and remove the caveat in the comment here.

Agreed, left for a follow-up in #487.

jserv · 2026-11-07T06:12:52Z

+        if (off <= UINT64_MAX - length &&
+            mprotect_exceeds_data(g, off, off + length, &stop)) {
+            if (stop > off) {
+                int64_t err = sys_mprotect(g, addr, stop - off, prot);


The prefix goes through the whole-range sys_mprotect, which does not walk mapping by mapping the way mprotect_fixup does. guest_region_range_has_ro_shared_backing refuses the whole prefix with EACCES before the mappings below the shared one change, and a hole inside the prefix gets page tables and the new permissions, as with any range today, where Linux would stop there with ENOMEM. Step 8 of data_child only checks the errno, so it passes either way. A follow-up could walk the range in address order and stop at the first hole, refused mapping or RLIMIT_DATA refusal. Until that lands, the description should not claim Linux's partial state for this path.

Agreed. Left the walk for a follow-up in #489, which also covers the hole case on main: mprotect over a range with a hole returns 0 and changes the pages past it, where Linux 6.12 stops with ENOMEM. In 899bc57, the mprotect_exceeds_data comment now says the part below the refused mapping is changed with one whole-range sys_mprotect, and the PR body no longer claims Linux's partial state for this path.

jserv · 2026-11-07T11:52:40Z

                             */
    uint64_t overlay_end;   /* Host-page-aligned overlay end (exclusive). */
    char name[64];          /* Label: "[heap]", "[stack]", ELF path, etc. */
+    uint64_t shm_attach;    /* Nonzero for a SysV shm attach: a serial unique


mremap rebuilds its regions through add_mremap_region, which goes through guest_region_add_ex_owned and does not carry shm_attach. An attach grown or shrunk in place by mremap loses its serial, so it can merge with a neighbor again, and shmdt on it fails with EINVAL with the attach still mapped. Carry the serial through mremap_track_t into the re-adds, or refuse to move an attach and say so.

Fixed in 5c25e3c: mremap_track_t now carries shm_attach, and add_mremap_region passes it to each re-add, so an attach resized or moved by mremap keeps its serial. Step 21 shrinks an attach and grows it back in place, then detaches; it fails without the carried serial and passes under elfuse and on Linux 6.18. A moved attach still cannot be detached at its new address, which is in #490.

xalestar · 2026-11-07T12:43:30Z

Runtime (Release) on 7d91b6c failed test-rlimit-mem at steps 8 and 12 in the qemu lane. That is what a pre-6.17 kernel gives: since a85dc37186a5 (v6.17) check_prep_vma charges MREMAP_FIXED growth before mremap_to unmaps the destination, and the steps expected that order.

The runner restores its fixtures from $HOME/.cache/elfuse-ci/test-fixtures (build.yml), and fetch-fixtures.sh runs only when fixture files are missing, so a cache filled before 1442b3c would still boot 6.12. My local clone had the same stale fixtures until today, and its VM was 6.12.103. I cannot see the runner's uname -r in the log, so this is an inference: could you check the cached kernel, or clear the cache so the next run fetches 6.18?

Meanwhile b644e96 makes steps 8 and 12 expect the order of the running kernel, from its uname release. elfuse reports 6.18, so it is still held to the 6.17+ order; the test passes under elfuse and on Linux 6.12.115 and 6.18.60.

The same run also failed test-futex-requeue-race on the qemu side ("the waiter never returned"). This PR does not touch futex code, and I could not rerun the job.

xalestar · 2026-11-07T14:03:31Z

1874a72 holds steps 8 and 12 of test-rlimit-mem to the Linux 6.18 MREMAP_FIXED order again, instead of following the running kernel. The CI qemu lane is expected to fail those two steps until its fixtures are refreshed: the runner cache still holds a kernel older than 6.17, and nothing in CI rebuilds it for the 6.18 pin (#491). On a local 6.18.60 VM all 3 cases pass; a 6.12.115 VM fails exactly steps 8 and 12, the same as CI. @jserv, clearing $HOME/.cache/elfuse-ci/test-fixtures on the runner would let the lane rebuild the fixtures.

cubic-dev-ai

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test-rlimit-mem.c">

<violation number="1" location="tests/test-rlimit-mem.c:411">
P2: This unconditional destination-preservation assertion makes `test-rlimit-mem` fail on supported Linux kernels before 6.17, where the refused move unmaps `dst` before returning `ENOMEM`. Keep the kernel-version-aware assertion or pin the QEMU fixture to Linux 6.17+ before requiring the newer behavior.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic}

cubic-dev-ai · 2026-11-07T14:06:38Z

+            MAP_FAILED ||
+        errno != ENOMEM)
+        return 12;
+    if (msync(dst, 4096, MS_ASYNC) < 0)


P2: This unconditional destination-preservation assertion makes test-rlimit-mem fail on supported Linux kernels before 6.17, where the refused move unmaps dst before returning ENOMEM. Keep the kernel-version-aware assertion or pin the QEMU fixture to Linux 6.17+ before requiring the newer behavior.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test-rlimit-mem.c, line 411: <comment>This unconditional destination-preservation assertion makes `test-rlimit-mem` fail on supported Linux kernels before 6.17, where the refused move unmaps `dst` before returning `ENOMEM`. Keep the kernel-version-aware assertion or pin the QEMU fixture to Linux 6.17+ before requiring the newer behavior.</comment> <file context> @@ -423,9 +408,7 @@ static int data_child(void) - errno = 0; - int dst_mapped = msync(dst, 4096, MS_ASYNC) == 0; - if (dst_mapped != mremap_checks_first() || (!dst_mapped && errno != ENOMEM)) + if (msync(dst, 4096, MS_ASYNC) < 0) return 12; </file context>

Intended. elfuse reports Linux 6.18 to the guest and is held to it, so steps 8 and 12 expect the 6.17+ order. The CI qemu lane fails them only because its cached fixtures still boot a kernel older than 6.17; that is #491.

jserv · 2026-11-07T14:14:08Z

    if (src_reg->backing_fd >= 0 && track->backing_fd < 0)
        return false;
    str_copy_trunc(track->name, src_reg->name, sizeof(track->name));
+    track->shm_attach = src_reg->shm_attach;


The serial now follows an attach through a moving mremap, but the shm_attach_t record keeps the old guest_gva. After mremap(seg, n, n, MREMAP_MAYMOVE | MREMAP_FIXED, dst), shmdt(dst) fails with EINVAL where Linux detaches it, and shmdt(seg) retires the record as stale and drops the host attach without writing back, so every write made after the move is lost. Worth a follow-up: update the record's address when mremap moves a region that carries a serial, or have sys_shmdt find the record by the serial of the region at the address it is given.

Fixed in 87263f9 rather than left for a follow-up, because the stale-record retirement from 77af41f was what dropped the writes. sys_shmdt now finds the record the way Linux finds the attach: a region carrying the record's serial at or above the address, whose offset into the attach matches its distance from the address. It writes back and unmaps from there. A record is stale only when no region anywhere carries its serial. Step 23 of data_child moves an attach with MREMAP_FIXED, writes through the new address, and checks that shmdt at the old address fails with EINVAL, shmdt at the new one succeeds, and a new attach sees the write. It fails on 1874a72 and passes under elfuse and on Linux 6.18.

jserv · 2026-11-07T14:14:09Z

+ * the infra reserve, each record as region_counted_span counts it. Caller holds
+ * mmap_lock.
+ */
+static void mem_vm_totals(const guest_t *g, uint64_t *total, uint64_t *data)


[stack] is counted at its full fixed extent (STACK_SIZE, 8 MiB), while Linux charges the stack VMA only as far as it has grown, about 132 KiB right after exec. A guest whose ulimit -v was sized from a Linux run gets roughly 8 MiB less address space here, and test-rlimit-mem cannot see it because it takes its base from elfuse's own VmSize. Either state this in the description and in the comment on mem_vm_totals, or open a follow-up to charge [stack] at the extent Linux would have.

Measured right after exec: [stack] is 8188 KiB under elfuse and 132 KiB on Linux 6.18.60 (VmSize 8880 kB against 840 kB). 5fba3aa states the gap in the comment on mem_vm_totals, the PR description now says so too, and #492 tracks charging [stack] at the extent it has grown to.

jserv · 2026-11-07T14:14:10Z

-                 int fd,
-                 int64_t offset)
+
+/* sys_mmap, but the region it records gets the name @name and the shm attach


The blank line separates the existing "Why this stays one function" comment from mmap_tagged, the function it now describes, and leaves two block comments stacked one after the other. Fold them into a single comment directly above mmap_tagged.

Folded in c52ab11: one comment directly above mmap_tagged, with what it does first and why it stays one function after.

jserv · 2026-11-07T14:14:13Z

    }
    if (!entry) {
        pthread_mutex_unlock(&shm_lock);
+        sys_munmap(g, (uint64_t) gva, PAGE_ALIGN_UP(seg_size));


A stale record is now retired only by a shmdt at that same address. A guest that munmaps whole attaches without calling shmdt (Linux detaches on unmap) still leaks one record and one host attach per attach, and after 64 of them every shmat fails here with ENOMEM. Before giving up, run the check sys_shmdt already uses (mem_attach_next finds no part of the record's range) across the table and retire the records with nothing mapped. The caller already holds mmap_lock.

Fixed in 87263f9: sys_shmat (and sys_shmdt) first retires every record whose serial no region carries any more, so a munmap of a whole attach detaches it at the next call. On macOS the host limit comes before the table's: kern.sysv.shmseg is 8, so on 1874a72 attaching and unmapping in a loop fails the 9th shmat with EMFILE, long before the 64 records run out. Step 22 of data_child now attaches and unmaps 80 times without shmdt; it fails on 1874a72 and passes under elfuse and on Linux 6.18.

mmap, mremap, brk and mprotect stored both limits and never checked them. They now refuse a request that would take the mapped total past RLIMIT_AS, or the private writable total past RLIMIT_DATA, with ENOMEM, as may_expand_vm does. Both totals are summed from the region table under mmap_lock, so they cannot drift from it; each page counts once, and the shim's infra reserve, which a Linux process has no mapping for, is left out. With neither limit set the check is two loads. The charge follows Linux. MAP_FIXED is charged only its net growth, as mmap_region does. A MREMAP_FIXED move is checked with its destination still counted and leaves it mapped when refused, as Linux does since 6.17. brk is charged its page-aligned growth, and the break extent is held to the soft RLIMIT_DATA whatever the heap's protection, as check_data_rlimit does. mprotect adding PROT_WRITE to a private mapping fails at the first mapping RLIMIT_DATA refuses, as mprotect_fixup does. A soft RLIMIT_DATA of 0 falls back to the hard limit, except for brk. The errors do_mmap reports before the limit still come first: EBADF, EEXIST, and EACCES for a descriptor whose access mode forbids the mapping. [stack] counts at the full extent elfuse maps up front, about 8 MiB more than Linux counts right after exec.

xalestar · 2026-11-07T16:48:07Z

Folded the 25 review commits into two, with the same tree as e9b0abe:

9cb3b2e Enforce guest RLIMIT_AS and RLIMIT_DATA: 3c5651e, 32f2e50, dcc95d5, 899bc57, d494ebb, 4373870, db0e62e, 7d91b6c, b644e96, 1874a72, 5fba3aa, 32b2986, and the shared-page counting and mmap_refused_errno parts of ee59d66 and 1399c9c.
3115982 Map and detach SysV shm attaches as Linux does: b7cb41f, a1d4794, 612bc96, 693c305, 5c25e3c, 77af41f, 87263f9, c76f376, c52ab11, 684fd8a, e9b0abe, and the shm parts of ee59d66 and 1399c9c.

A hash cited in an earlier reply is listed above under the commit that now holds its change. Each commit builds and passes test-rlimit-mem.

jserv · 2026-11-07T22:44:42Z

+ * attaches (kern.sysv.shmseg, 8 by default on macOS) and a table slot. The
+ * caller holds mmap_lock and not shm_lock.
+ */
+static void shm_retire_unmapped(guest_t *g)


Retirement only runs at the next shmat or shmdt, so two cases still differ from Linux. After execve, the new image may never call either, so every attach of the old image keeps its host attach (and keeps an IPC_RMIDed segment alive) until the process exits. And an attach the guest munmaps without shmdt loses its writes, because the guest pages are gone before this runs, while on Linux those pages are the segment. As a follow-up, write back and detach from the sys_munmap path, and write back and drop every record at exec. That would also make this lazy sweep unnecessary.

Agreed, and kept out of this PR as you suggest: opened #497 to write back and detach from the sys_munmap path, and to write back and drop every record at exec. That removes the lazy sweep.

jserv · 2026-11-07T22:44:44Z

+    int n = 0;
+    pthread_mutex_lock(&shm_lock);
+    SHM_FOR_EACH (e) {
+        if (e->active && !mem_attach_mapped(g, e->attach)) {


mem_attach_mapped walks the whole region table once for each active record, so every shmat and shmdt pays up to MAX_SHM_ATTACH times nregions iterations under mmap_lock (64 x 4096 with a full table). Walk g->regions once instead, and for each region with a nonzero shm_attach mark the matching record, so regions without a serial cost nothing.

Fixed in 66367a0. shm_retire_unmapped collects the serials of the active records and walks g->regions once, marking each record whose serial a region carries. A region without a serial costs one test. The check at the end of shmdt uses the same helper with one serial.

jserv · 2026-11-07T22:44:45Z

+    for (uint64_t cur = gva;
+         mem_attach_next(g, gva, cur, end, entry.attach, &lo, &hi, NULL);
+         cur = hi) {
+        int64_t rc = sys_munmap(g, lo, hi - lo);


An attach grown in place past its segment is only partly detached. After shmat of a one-page segment and mremap(seg, 4096, 8192, 0), the region carries the serial over [seg, seg + 8192), but end stops at PAGE_ALIGN_UP(entry.size). So shmdt unmaps the first page and leaves the second one mapped with the serial: the record stays active, the host attach is held, and the page keeps counting toward RLIMIT_AS with no address left that can detach it. Linux unmaps the first matching VMA whole and bounds only the later ones by the segment size, and then only whole VMAs. Take the first piece to its region's end, and apply the end bound only to the pieces after it.

Fixed in d2ea761. shmdt now takes the first piece to its region's end and applies the segment bound only to the later pieces, detaching one only when it ends within the bound, as ksys_shmdt does. The same sequence also hung shmdt: with the region reaching past end, mem_attach_next kept returning an empty piece at end. It now returns false once gva reaches end.

Step 26 of test-rlimit-mem attaches a one-page segment, grows it in place to two pages, writes to it and detaches. It then checks that msync fails on both pages and that a reattach sees the write. Without the fix it times out. With only the bound check it fails at step 26. With the fix it passes in both lanes (QEMU lane on 6.18).

A SysV shm attach was private anonymous memory, so it counted toward RLIMIT_DATA, a fork child got a copy, and shmdt left the mapping in place. A Linux attach is VM_SHARED: it counts toward RLIMIT_AS only, a fork child shares it, and shmdt unmaps it. The attach is now mapped shared anonymous (a fork child shares it when it starts on a host page), named /SYSV<key> (deleted) as Linux names it, and its regions carry a serial unique to the attach, kept across fork, so the attach never merges with another mapping. The region struct grew, so the fork IPC magic moves to ELFT. shmdt finds the attach from the mappings, as Linux does: a region with the serial whose offset into the attach matches its distance from the address. It writes back and unmaps only the parts found there: the first one whole, even when mremap grew it past the segment, and the later ones only when they end within the segment size. So an attach moved with mremap detaches at its new address, and the record stays live while another part is mapped. A record whose serial no region carries is retired at the next shmat or shmdt, so unmapping attaches does not exhaust the 8 host attaches macOS allows a process.

xalestar · 2026-11-08T13:17:41Z

Folded the two review fixes into the shm commit; the tree is unchanged from 66367a0. d2ea761 (mremap-grown attach, r4504981011) and 66367a0 (one region walk, r4504980924) are now part of a430b30 "Map and detach SysV shm attaches as Linux does". 9cb3b2e is unchanged.

jserv · 2026-11-08T15:35:28Z

+    int64_t err = -LINUX_ENOMEM;
+    if (overlaps)
+        err = -LINUX_EEXIST;
+    else if (!is_anon &&


mmap_refused_errno only knows EBADF, EEXIST and EACCES, so once a limit is exceeded a descriptor that cannot be mapped at all (a pipe, a socket, a directory) gets ENOMEM here. do_mmap returns ENODEV for those before mmap_region reaches may_expand_vm, and nothing ahead of this check (mmap_check_args covers only FUSE) catches them. Test the host fd type with fstat and return -LINUX_ENODEV ahead of the EACCES test, or list ENODEV as a known gap next to the error order in the description.

jserv · 2026-11-08T15:35:29Z

+    }
+    pthread_mutex_unlock(&shm_lock);
+    for (int i = 0; i < nhost; i++)
+        shmdt(host[i]);


Retiring here throws away whatever the guest wrote through the attach since shmat, because only sys_shmdt copies back to host_addr. The sequence shmat, write, munmap (or a MAP_FIXED over the attach), then shmat of the same segment again reads the old contents, while Linux keeps the writes in the segment. It is no worse than before (the old shmdt copied whatever sat at the stale address), but the description should say so. The fix belongs in a follow-up: map the host segment into the guest through the shared overlay path that MAP_SHARED files already use, instead of copying it. That also lets two live attaches of one segment see each other's writes.

cubic-dev-ai Bot reviewed Nov 6, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

jserv reviewed Nov 6, 2026

View reviewed changes