Skip to content

perf(cargo): align dev/test hybrid debuginfo to prevent full-test OOM - #1066

Open
fleeto wants to merge 1 commit into
libra-tools:mainfrom
fleeto:fix/full-test-resource-limits
Open

fleeto wants to merge 1 commit into
libra-tools:mainfrom
fleeto:fix/full-test-resource-limits

Conversation

@fleeto

@fleeto fleeto commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Aligns [profile.dev] and [profile.test] in Cargo.toml to a hybrid debuginfo configuration (debug = "line-tables-only" for workspace/test crates, debug = 0 for external dependencies) while preserving the existing [profile.dev.package."*"] opt-level = 2 optimization.

Root Cause

Previously, Cargo.toml enabled opt-level = 2 on all external dependencies in dev/test profiles while leaving Cargo's default debug = true (full DWARF). Because libra links a large dependency graph across ~180 integration/unit test binaries:

  • A single heavy test target (agent_bridge_checkpoint_test) produced a 2.001 GiB ELF (1.906 GiB / 95.3% .debug_* sections) and drove rust-lld to 8.936 GiB RSS (14.003 GiB cgroup peak, 701 memory.high throttle events even at --jobs 1).
  • Running the full test suite (cargo test --all / cargo nextest run --all) concurrently linked multiple multi-GiB binaries and overwhelmed developer machines with memory and page-cache pressure.

Changes

  • Set [profile.dev] and [profile.test] to debug = "line-tables-only" (preserving file:line resolution for libra and test code, verified via addr2line).
  • Set [profile.dev.package."*"] and [profile.test.package."*"] to debug = 0 (eliminating external dependency DWARF bloat and allowing cargo build and cargo test to reuse dependency artifacts).
  • Documented override knobs (CARGO_PROFILE_DEV_DEBUG=2 / CARGO_PROFILE_TEST_DEBUG=2 for full workspace DWARF, --config 'profile.test.package."*".debug=2' for dependency DWARF, and CARGO_PROFILE_TEST_DEBUG=0 for minimal footprint).
  • Zero changes to production code, test assertions, debug_assertions, opt-level, or [profile.release].

Measured Impact (Cgroup: CPUQuota=400%, MemoryHigh=14G, MemoryMax=18G, MemorySwapMax=1G)

Single Heavy Target (agent_bridge_checkpoint_test, --jobs 1)

Metric Before (debug = true) After (Hybrid) Delta
Cgroup memory.peak 14.003 GiB (701 high events) 8.533 GiB (0 high events) -39.1%
rust-lld peak VmRSS 9,150.74 MiB (8.936 GiB) 1,819.40 MiB (567.44 MiB anon) -80.1%
Output ELF size 2,048.58 MiB (2.001 GiB) 195.14 MiB (101.05 MiB .debug_*) -90.5%
addr2line on libra::* & test crate Exact file.rs:line Exact file.rs:line Preserved

Controlled Full-Suite Acceptance (cargo nextest run --all --locked --offline --build-jobs 2 --test-threads 4 --no-fail-fast --retries 2)

  • Build phase (180 test binaries): completed in 5m 29s; net target/debug disk growth 17.65 GiB (13.27 GiB across 352 changed artifacts); 0 OOM / 0 memory.max / 0 swap.
  • Runtime phase (12,015 tests): completed in 42m 46.5s (11,930 passed, 51 skipped, 85 pre-existing environment/umask 0002 & registry drift failures unrelated to debuginfo); runtime cgroup anon peak was only 1.316 GiB.
  • dev/test cache reuse: cargo build --jobs 2 produced a 347 MiB target/debug/libra (8.74 GiB cgroup peak, 0 high events), followed by cargo test --no-run --test agent_bridge_checkpoint_test --jobs 2 in 1.461s with 0 rebuilt artifacts.

Verification

  • cargo fmt --all --check (formatting clean; note local host lacks nightly rustfmt for unstable import grouping)
  • cargo clippy --locked --offline --lib --test agent_bridge_checkpoint_test -- -D warnings
  • cargo nextest run --locked --offline --build-jobs 2 --test-threads 4 --retries 0 --no-fail-fast --test agent_bridge_checkpoint_test --test compat_release_homebrew_verify --test compat_all_production_unwrap_guard --test agent_bridge_protocol_test (16/16 passed)
  • Full-suite cargo nextest run --all --locked --offline --build-jobs 2 --test-threads 4 --no-fail-fast --retries 2 under cgroup limits (0 OOM / 0 swap)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant