Skip to content

fix: let the forced old-page gate fixture move its block, and bound moved by selected live from below (#9659) - #14927

Merged
TheHypnoo merged 3 commits into
PerryTS:mainfrom
raliclo:fix/9659-defrag-accounting
Nov 18, 2026
Merged

TheHypnoo merged 3 commits into
PerryTS:mainfrom
raliclo:fix/9659-defrag-accounting

Conversation

@raliclo

@raliclo raliclo commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The old_page_forced_defrag workload of the memory-stability gate (compile-smoke, full tier) was red for two independent reasons. Both are fixed here, and both were measured rather than inferred.

  1. The fixture never moved anything on macOS. test_old_page_defrag_target_gate_emits_trace panicked at oldgen.rs:913 ("forced old-page defrag should rewrite the shadow root to the moved parent"). The source block was not movable, so nothing moved.
  2. The report asserted the wrong bound. It checked old_page_moved_bytes <= old_page_selected_live_bytes. That is the "old_page_moved_bytes(80) > old_page_selected_live_bytes(32)" failure in the issue. The pass moves more than the selected live bytes by design.

What was measured

All numbers come from upstream 690e83901d on macOS arm64, with temporary eprintln! instrumentation in evacuate_selected_old_pages_collecting, arena_alloc_gc_old and the fixture. The instrumentation is not part of this PR.

Why nothing moved. The parent's source block on main held four occupants:

occupant type bytes flags marked movable?
shape keys array Array 64 TENURED|SHAPE_SHARED|ARENA no no: GC_FLAG_SHAPE_SHARED
shape keys array Array 72 TENURED|SHAPE_SHARED|ARENA no no
parent Object 40 TENURED|ARENA|MARKED yes yes
dead string String 48 TENURED|ARENA no yes

Measured cycle values were selected_pages=1, selected_live=40, reclaimable=184, moved=0, released=0. The whole-block pass is all-or-nothing (#7913/#9772), so the two keys arrays make it decline.

Where the two arrays come from:

  • Measured: step markers place both allocations inside the fixture's own shape_descriptor_ensure(null, 0, 1) call in alloc_old_test_object.
  • Measured: the 2 MiB "finish the current block" filler is placed in a different block (for example filler at 0x108100008, arrays and parent at 0x103000008…). It never isolates the parent.
  • From the code: on macOS BLOCK_SIZE is 1 MiB, so a 2 MiB filler gets an oversized block of its own.
  • Measured: minting the descriptor before the filler doesn't help, because the parent still lands next to the keys arrays.

What the pass moves once it can move. I re-ran the fixture with the shape minted first and allocations continued until a new old block starts. The parent's block then held three occupants: a 4096-byte unmarked filler on another page of the same block, the parent, and the 48-byte dead string.

field value
old_page_selected_pages 1
old_page_selected_live_bytes 40 (the parent)
old_page_reclaimable_bytes 48
occupants moved 3
bytes moved, marked 40
bytes moved, unmarked 4144 (48 on the selected page, 4096 on another page of the block)
old_page_moved_bytes 4184
released_original_bytes 4184

So moved > selected_live is the designed behavior: evacuate_selected_old_pages_collecting copies every indexed occupant of the source block, marked or not, on selected pages or not. The report's ceiling contradicts that. The selection's live bytes are a floor instead, because every live byte on a selected page is in the block that moves. The existing ceiling moved <= released still holds and is kept.

Changes

  • crates/perry-runtime/src/gc/tests/oldgen.rs (test fixture only): mint the parent's shape first, then allocate until a new old block starts (old_arena_block_for_test). This works whatever the platform's BLOCK_SIZE is.
  • scripts/copied_minor_fallback_report.py: replace moved > selected_live → error with 0 < moved < selected_live → error, and keep moved > released → error.
  • tests/tooling/infrastructure/test_copied_minor_fallback_report.py: old_page_cycle takes the three byte counts. Three tests cover them:
    • the measured block (40 / 4184 / 4184) passes
    • a pass that moves less than the selected live bytes fails
    • moved > released still fails

Related issue

Refs #9659. The gate is still red after this PR for reasons outside the old-page accounting; see the gate run below.

Test plan

macOS-only local workaround: perry-runtime's lib test binary does not link on macOS at this commit, because snap_census/inventory.rs references ELF-only linker symbols. Every cargo test -p perry-runtime below used a local linker wrapper (CARGO_TARGET_AARCH64_APPLE_DARWIN_LINKER) that aliases those 9 symbols to _main for the test binary only. It is not part of this PR.

Fixture test, reverting only the fixture change to prove it:

PERRY_GC_TRACE=1 PERRY_GC_FORCE_EVACUATE=1 PERRY_GC_OLD_DEFRAG=1 \
  cargo test --release -p perry-runtime --lib -- --test-threads=1 test_old_page_defrag_target_gate_emits_trace
main fixture: panicked at crates/perry-runtime/src/gc/tests/oldgen.rs:913:5 (parent not moved)
this PR:      ok

The gate's old-page canaries also pass:

  • cargo test --release -p perry-runtime --lib -- test_old_page_: 18 passed
  • PERRY_GC_FORCE_EVACUATE=1 PERRY_GC_OLD_DEFRAG=1 cargo test … gc::tests::oldgen::: 27 passed

Report on the real trace from the fixed fixture (the gc_cycle JSON line it emits):

python3 scripts/copied_minor_fallback_report.py --workload old_page_forced_defrag=<trace> --target-collector-gates
main script: old_page_forced_defrag:1: evacuation.old_page_moved_bytes(4184) > evacuation_policy.old_page_selected_live_bytes(40)
this PR:     (no old-page error)

Both runs also print target collector gates require at least one copied-minor workload, because only the one workload was passed.

Report unit tests (python3 tests/tooling/infrastructure/test_copied_minor_fallback_report.py): 27 run, OK. With main's script, the two new bound tests fail, as expected.

Full perry-runtime lib suite with this PR (RUST_TEST_THREADS=1 cargo test --release -p perry-runtime --lib): 6854 passed, 11 failed. The failures are:

  • 7 snap_image::tests::*. These read the ELF section symbols the macOS wrapper aliases; Linux-only per aa769b3.
  • 3 gc::tests::lazy_intrinsic_towers::*
  • gc::roots::runtime_handles::tests::handle_storage_teardown_clears_cache_before_late_scope_drop

None of these is in the module this PR touches. The one snap_image test that is newer than this macOS workaround, the_capture_names_each_reason_replay_cannot_rebuild_and_counts_routes, fails the same way with the fixture change reverted: the same panic at snap_image/tests.rs:835.

Full gate with this PR (./scripts/run_memory_stability_tests.sh, macOS, after cargo build-perry --release, sidecars, npm ci and scripts/build_instr_runtime.sh):

The gate still reports FAIL: 4, but the old-page workload now passes.

  • PASS [target-gc] old_page_forced_defrag. On main this workload panics before the report, which leaves the six target workloads' report blocked.

  • The target-collector report now runs. Its old-page accounting from the gate's own trace: selected_pages=1, selected_live_bytes=40, reclaimable_bytes=48, old_page_moved_bytes=4184, released_original_bytes=4184, pooled_bytes=1048576. The report finds no old-page error.

  • What still fails is outside this PR. It was measured the same way on main by running the report by hand on main's traces:

    • [target-gc] architecture stress gates, from the report's other assertions:
      • string_heavy and raw_numeric_layouts: malloc_registry_rebuilds=1, want 0
      • string_heavy: no mutable or shadow root source evidence
      • large_object_barriers: missing large-object exclusion telemetry
      • raw_numeric_layouts: no raw numeric object field slots were skipped
      • pointer_free_numeric: no pointer-free slots were skipped
    • [gc-trace] default copied minor churn: all 10 cycles are full / not_attempted. This is the unscoped-assertion pattern the issue's first comment describes, in the GC acceptance telemetry step.
    • [gc-trace] copied-minor fallback evidence: string_churn has no mutable or shadow root source evidence.
    • [canary] issue #1425 Fastify/ws manual gc: listen EAGAIN on 127.0.0.1 in this sandbox. This is environmental.
  • rustfmt is clean on the touched lines.

  • Full lint (BASE_SHA=<merge-base> bash scripts/run_lint_gates.sh): 18 of 151 gates FAIL. The FAIL set is identical, by gate name, to full runs on two pristine main checkouts from the same day (18 each).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CUQanVsnitcrmYLAWWPc3p

Summary by CodeRabbit

  • Bug Fixes
    • Updated old-page evacuation accounting to allow moved bytes to exceed selected live bytes when whole source blocks are moved. Checks still flag cases where fewer bytes move than selected live bytes or moved bytes exceed released original storage.
  • Tests
    • Expanded old-page workload coverage across platforms and added checks for valid and invalid byte-accounting results.

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The old-page report gate now requires positive moved bytes to be at least the selected live bytes and no more than the released original bytes. The report tests cover these bounds, and the runtime test changes how it arranges the parent’s source block.

Changes: Old-page defrag accounting

Layer / File(s) Summary
Prepare the old-page source-block layout
crates/perry-runtime/src/gc/tests/oldgen.rs
The test initializes the shape, resets the old allocator, and allocates objects across a block boundary before allocating the parent.
Update and test moved-byte bounds
scripts/copied_minor_fallback_report.py, tests/tooling/infrastructure/test_copied_minor_fallback_report.py, changelog.d/14927-old-page-gate-fixture-and-bound.md
The report gate rejects positive moved bytes below selected live bytes and retains the upper bound against released original bytes. Tests cover both bounds and a passing case where moved bytes exceed selected live bytes. The changelog records the workload and bounds.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related issues

Merge Risk: Minimal · up to c636e

This change fixes a memory-stability gate's fixture and relaxes its moved-bytes check to match how whole-block evacuation actually counts. It does not alter production runtime behavior, and no concrete merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main fix: the forced old-page gate fixture now moves its block, and the report bounds moved bytes by selected live bytes from below. It matches the changeset (oldgen.rs fixture, co…
Description check Passed The description follows the template sections: Summary, Related issue (Refs #9659), Type of change, Platforms (macOS tested, arm64), Changes, and a detailed Test plan with commands and results. It exp…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main fix: the forced old-page gate fixture now moves its block, and the report bounds moved bytes by selected live bytes from below. It matches the changeset (oldgen.rs fixture, copied_minor_fallback_report.py bound change, and its tests). The issue number is included, which is acceptable.

Full details: Description check

Explanation

The description follows the template sections: Summary, Related issue (Refs #9659), Type of change, Platforms (macOS tested, arm64), Changes, and a detailed Test plan with commands and results. It explains the root cause, measurements, and reasoning for the bound change. Some template items are not ticked (Type of change, Platforms checkboxes), and the Changes section is a bulleted list naming paths. The Checklist section is not addressed, and the changelog entry (changelog.d/14927...) is referenced in the summary but the PR-number mismatch with #9659 is not explained. These are minor gaps and do not make the description largely incomplete.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5f1c5578-402f-4eb7-b819-3071a516d1da
Commits

Reviewing files that changed from the base of the PR and between c266c7e and c636e99.

Files selected for processing: 3

  • crates/perry-runtime/src/gc/tests/oldgen.rs
  • scripts/copied_minor_fallback_report.py
  • tests/tooling/infrastructure/test_copied_minor_fallback_report.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

@TheHypnoo TheHypnoo added area:tooling-ci CLI driver, scripts, CI, release effort:S Hours: one local fix impact:med A feature, module, package or platform subset platform:macos Specific to macos type:chore Refactor, CI, docs, cleanup labels Nov 18, 2026
ralic and others added 3 commits November 18, 2026 23:46
…oved by selected live from below (PerryTS#9659)

test_old_page_defrag_target_gate_emits_trace minted the parent's shape after
its 2 MiB filler, so the shape's keys arrays (GC_FLAG_SHAPE_SHARED, never
moved) landed in the parent's source block and the all-or-nothing block
evacuation declined. Where BLOCK_SIZE is 1 MiB (macOS) the filler takes an
oversized block of its own and never isolates anything. Mint first, then
allocate until a new old block starts.

copied_minor_fallback_report asserted old_page_moved_bytes <=
old_page_selected_live_bytes. The pass moves every indexed occupant of the
source block, unmarked ones and other pages' included; measured on the fixed
fixture: selected_live 40, moved 4184 (40 marked + 4144 unmarked), released
4184. Require moved >= selected_live whenever anything moved, and keep
moved <= released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CUQanVsnitcrmYLAWWPc3p
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CUQanVsnitcrmYLAWWPc3p
@TheHypnoo
TheHypnoo force-pushed the fix/9659-defrag-accounting branch from c266c7e to c636e99 Compare November 18, 2026 23:00
@TheHypnoo
TheHypnoo merged commit 9041965 into PerryTS:main Nov 18, 2026
100 of 107 checks passed
@raliclo
raliclo deleted the fix/9659-defrag-accounting branch November 18, 2026 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tooling-ci CLI driver, scripts, CI, release effort:S Hours: one local fix impact:med A feature, module, package or platform subset platform:macos Specific to macos type:chore Refactor, CI, docs, cleanup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants