You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This morning it has been observed that there is a backlog in node-test-commit jobs which have hit their limit of 15 concurrent jobs currently in flight.
The reason for this appears to have been an issue with the node-test-binaries-windows-native-suites jobs which do not appear to be correctly receiving the results from each initiated axis.
I have aborted all of the stuck jobs (35426 through 35431) although I note that a couple of @jasnell's jobs that were in the queue (35426 and 35431) had already been stopped. the follow PR tests are the ones that will have been affected (linking them so the originators are aware of what's happened
changed the title [-]Windows test multiconfig jobs hanging and not acknowledging completion of subjobs[/-][+]Windows/arm machines unavailable[/+]on Oct 9, 2026
Looks like our Windows/arm machines are all offline which is preventing that axis from running. I have removed the two axes of the test jobs native-suites and js-suites until those machines can be recovered:
win11-vs2022-arm64-COMPILED_BY-vs2022-arm64
win11-vs2022-arm64-COMPILED_BY-vs2022_clang-arm64
Noting that I've left the building phase in place as that is cross compiled from an x64 system.
Stefan has brought them back online and I have reactivated the jobs. We should endeavor to check that the machines are correctly auto-starting jenkins agents for the future.
Systems came back online overnight and the queues have cleared. Closing again.
I saw this last night and brought them back online. I couldn't find out why it happened, but this seems to happen more often than it should... we should put some mechanism in place to catch this early, notify us, and ideally help us identify why it keeps happening. This seems ARM64-specific. Windows 10 machines also go offline from time to time, but that is because of the Upgrade screen after reboot, which must be manually skipped (plus we will remove those machines soon)
Activity
[-]Windows test multiconfig jobs hanging and not acknowledging completion of subjobs[/-][+]Windows/arm machines unavailable[/+]sxa commentedon Oct 9, 2026
Looks like our Windows/arm machines are all offline which is preventing that axis from running. I have removed the two axes of the test jobs native-suites and js-suites until those machines can be recovered:
Noting that I've left the building phase in place as that is cross compiled from an x64 system.
Assigning to @StefanStojanovic
sxa commentedon Oct 12, 2026
Stefan has brought them back online and I have reactivated the jobs. We should endeavor to check that the machines are correctly auto-starting jenkins agents for the future.
sxa commentedon Oct 20, 2026
@StefanStojanovic Were you able to confirm if these machines will auto-start the jenkins agent going forward? If so we can probably close this.
StefanStojanovic commentedon Nov 3, 2026
Forgot to close this earlier. Yes, it can be closed.
sxa commentedon Nov 18, 2026
Re-opening as this has recurred - all four windows/arm agents are currently offline.
sxa commentedon Nov 19, 2026
Systems came back online overnight and the queues have cleared. Closing again.
StefanStojanovic commentedon Nov 19, 2026
I saw this last night and brought them back online. I couldn't find out why it happened, but this seems to happen more often than it should... we should put some mechanism in place to catch this early, notify us, and ideally help us identify why it keeps happening. This seems ARM64-specific. Windows 10 machines also go offline from time to time, but that is because of the Upgrade screen after reboot, which must be manually skipped (plus we will remove those machines soon)