Guest processes are limited to 64 threads #494

Activity

1. xalestar commented on Nov 7, 2026

   

   xalestar

   on Nov 7, 2026

   CollaboratorAuthor

   More actions

   The 64 is a Hypervisor.framework limit, not something elfuse picked. hv_vm_get_max_vcpu_count reports 64, and the 65th hv_vcpu_create in one VM fails with HV_NO_RESOURCES (0xfae94005). elfuse gives every guest thread its own vCPU, so MAX_THREADS and the uint64_t SP_EL1 bitmask only mirror that cap. Raising MAX_THREADS alone would move the failure from clone to hv_vcpu_create.

   Probe: one VM, then 300 pthreads that each call hv_vcpu_create and stay parked.

   max_vcpu_count=64
   created=64
   fail at 64: 0xfae94005   (236 times)
   

   hv_vm_config has no setting that raises this. It offers only IPA size and IPA granule.

   The demand is real Go behavior, not something elfuse adds. Peak threads= from GODEBUG=schedtrace=10 for the gofmt command above, one value per run:

   Where	GOMAXPROCS	Peak threads
   elfuse	15 (default, 15 host CPUs)	65, 65, 65 (the cap)
   Linux 6.18 (QEMU, 4 vCPUs)	15	48, 126, 112
   elfuse	4	13, 25
   Linux 6.18	4	7, 24, 7

   At GOMAXPROCS=4 the two are in the same range, so elfuse does not inflate the count. The Linux VM runs 15 Ps on 4 vCPUs, so a real 15-core Linux machine may peak lower.

   Memory is not the limit. The EL1 stacks take 256 KiB at the top of the 2 MiB shim data block, and the shim globals take about 8.4 KiB at the bottom. For a guest to run more than 64 threads, a guest thread would have to stop owning a vCPU for its whole life, for example by taking a parked thread's registers off its vCPU while it blocks in a host syscall.
2. xalestar

   added

   bugSomething isn't workingSomething isn't working

   enhancementNew feature or requestNew feature or request

   on Nov 8, 2026
3. xalestar commented on Nov 8, 2026

   

   xalestar

   on Nov 8, 2026

   CollaboratorAuthor

   More actions

   A possible way past the cap, as a design sketch. Nothing here is implemented or measured yet.

   Go needs more than 64 threads because threads block in syscalls, not because more than 64 run guest code at once. So the target is narrower than a full N:M scheduler: at most 64 guest threads on a vCPU at a time, any number parked in a host syscall.

   Hypervisor.framework binds a vCPU to the host thread that created it (per Apple's documentation, not tested here), so a guest thread cannot move to another vCPU itself. Its register state has to be saved and loaded by the thread that owns the target vCPU. That gives three roles:

   Role	Count	Does
   vCPU runner thread	at most 64, each owns one vCPU	loads a runnable guest context, runs hv_vcpu_run, handles fast syscalls inline as today
   per-guest-thread host pthread	one per guest thread, no cap	runs that thread's blocking syscalls
   saved guest context	one per guest thread	GPRs, FP/SIMD state, per-thread system registers

   A blocking syscall (futex wait, blocking read/accept/poll, nanosleep) then goes: SVC exit on the runner, runner saves the context and hands the call to the thread's host pthread, runner loads the next runnable context; when the host call returns, the pthread stores the result and marks the context runnable for any free runner. Fast syscalls stay on the runner, so lat_syscall null should not move; the blocking path pays one host-thread handoff plus a register reload. With more than 64 runnable contexts the rest wait for a runner; time slicing through hv_vcpus_exit can come later.

   What makes it a large change (counts from grep on current main):

   • current_thread is _Thread_local (src/runtime/thread.h) and has 152 uses; 25 _Thread_local objects in all, including the TLBI request in src/core/guest.c. Each one means "this guest thread" today and would have to be split into per-runner and per-context state.
   • src/syscall makes 182 hv_vcpu_* calls. A handler that runs on the blocking pthread cannot touch a vCPU and has to edit the saved context instead, so the blocking set has to be audited and the register-touching handlers (signal delivery, rt_sigreturn, clone, exec, ptrace) kept on the runner.
   • A signal to a parked thread edits its saved context; one to a running thread still needs a forced exit.
   • The SP_EL1 slot bitmask in src/runtime/thread.c and the shim_data slots become per-vCPU instead of per-thread.
   • A futex wake makes a context runnable instead of waking a pthread that owns a vCPU.
   • gdbstub and fork-state enumerate threads through vCPUs and would enumerate contexts instead.

   Alternatives that do not work: raising MAX_THREADS moves the failure to hv_vcpu_create; one process cannot hold more than one VM; capping GOMAXPROCS still failed inside make bench-ci.

   Suggested order: release the vCPU on futex wait only, with lat_syscall null and a futex ping-pong measured before and after; then extend it to blocking file and socket syscalls; then rerun the gofmt reproducer above at the default GOMAXPROCS, 30 runs. The futex step alone does not cover gofmt, which blocks in file I/O.