Skip to content

feat: improve CI recovery and add availability and workload commands - #1204

Open
panva wants to merge 6 commits into
nodejs:mainfrom
panva:fix-ci-resume-recovery
Open

panva wants to merge 6 commits into
nodejs:mainfrom
panva:fix-ci-resume-recovery

Conversation

@panva

@panva panva commented Sep 25, 2026 •

Copy link
Copy Markdown
Member
  • Check Jenkins' Resume build action before scanning failures or attempting a resume. If unavailable, follow recorded resume ancestry to the nearest eligible build for the same PR and approved commit, checking failures from every visited run.
  • Keep --check-for-duplicates blocking duplicate runs for the same approved commit. Suggest resume-ci or ncu-ci resume when a resume action is available, and provide manual recovery guidance otherwise.
  • Add ncu-ci available to check whether Jenkins is quieting down and whether node-test-pull-request is enabled and buildable.
  • Add ncu-ci workload to print the number of running and queued PR jobs, including builds waiting for downstream tests, without double-counting requests moving between the queue and executors.
  • Print the matching failure excerpt and console URL when changed files prevent a resume, with bounded output for large diagnostics.
  • Exclude propagated make exit summaries from file attribution while retaining direct Makefile diagnostics and failures in changed tests.

The two additions are aimed at allowing node's auto-start-ci to skip piling on more jobs and exiting gracefully before touching labels if CI is unavailable.

@panva panva changed the title handle unavailable CI resume actions ncu-ci: handle unavailable CI resume actions Sep 25, 2026
@panva
panva requested review from aduh95 and mcollina September 25, 2026 09:44
@codecov

codecov Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.12%. Comparing base (b431f15) to head (2c640e7).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1204      +/-   ##
==========================================
+ Coverage   61.80%   63.12%   +1.31%     
==========================================
  Files          61       65       +4     
  Lines       11880    12257     +377     
==========================================
+ Hits         7343     7737     +394     
+ Misses       4537     4520      -17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aduh95

aduh95 commented Sep 25, 2026

Copy link
Copy Markdown
Contributor
  • Allow --check-for-duplicates to start fresh CI when a completed failed or aborted run has no usable resume path. Resume lookup errors still block duplicates.

I'm not sure we want to do that, it's probably a better idea to defer to manual action in this case as it indicates that something went wrong (I'd expect that in most cases, manual action in question would be to rebase the PR, as that likely indicates that the change is too old)

@panva
panva force-pushed the fix-ci-resume-recovery branch from 2a6b933 to f649f61 Compare September 25, 2026 11:57
@panva panva changed the title ncu-ci: handle unavailable CI resume actions feat: improve ncu-ci recovery and add available and workload Sep 25, 2026
@panva

panva commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

@aduh95 reverted.

Check Jenkins' context menu before scanning failures or posting a resume.
When the action is missing, follow recorded resume ancestors for the same
PR and approved commit, checking failures from every run along the way.

Report unavailable actions and request failures with recovery guidance.

Signed-off-by: Filip Skokan <panva.ip@gmail.com>
Assisted-by: Codex
Check resume availability before suggesting a recovery action. Keep
same-commit duplicates blocked when neither the latest run nor an eligible
ancestor can resume, and guide users to inspect the run and recover
manually. Suggest resume-ci only when a resume action is available.

Signed-off-by: Filip Skokan <panva.ip@gmail.com>
Assisted-by: Codex
Check Jenkins shutdown state and PR job readiness before processing CI
requests. Report unavailable or unknown states with a nonzero exit code.

Signed-off-by: Filip Skokan <panva.ip@gmail.com>
Assisted-by: Codex
@panva
panva force-pushed the fix-ci-resume-recovery branch from 62688b3 to 7512639 Compare September 25, 2026 13:55
Count running and queued PR CI jobs for workload threshold checks.
Include PR builds waiting for downstream tests and count requests only
once when they move from the queue to an executor between reads.

Signed-off-by: Filip Skokan <panva.ip@gmail.com>
Assisted-by: Codex
Print the matching diagnostic and console log URL when a changed file
prevents resuming CI. Bound large excerpts and preserve early download
cancellation.

Assisted-by: Codex
Signed-off-by: Filip Skokan <panva.ip@gmail.com>
A failed make target names its Makefile even when an unrelated test
failed. Exclude these summaries from file attribution while retaining
direct Makefile diagnostics and failures in changed tests.

Assisted-by: Codex
Signed-off-by: Filip Skokan <panva.ip@gmail.com>
@panva
panva force-pushed the fix-ci-resume-recovery branch from 7512639 to 2c640e7 Compare September 25, 2026 14:34
@panva panva changed the title feat: improve ncu-ci recovery and add available and workload feat: improve CI recovery and add availability and workload commands Sep 25, 2026
@panva

panva commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Print the matching failure excerpt and console URL when changed files prevent a resume, with bounded output for large diagnostics.

This is what that comment in https://redirect.github.com/nodejs/node/pull/65879 would like now.

Failed to resume CI

   ✖  test/parallel/test-inspector-dom-storage-malformed.js
   ℹ  https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win11-arm64-COMPILED_BY-vs2022_clang-arm64/43507/consoleText
✖  Refusing to resume CI: failures reference files changed by this PR
Full Auto Start CI output
[Earlier progress output omitted]

   ✖  test/parallel/test-inspector-dom-storage-malformed.js
   ℹ  https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=3,nodes=win11-arm64-COMPILED_BY-vs2022_clang-arm64/43507/consoleText
not ok 731 parallel/test-inspector-dom-storage-malformed
  ---
  duration_ms: 464.00300
  severity: fail
  exitcode: 1
  stack: |-
    [Routine debugger output omitted]

    [err] child process crashed, signal SIGTERM
    AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
    + actual - expected

      Comparison {
    +   message: 'Could not read DOM storage items: storage is unavailable'
    -   message: 'Could not read DOM storage items: the backing file is malformed'
      }

        at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
        at async d:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-inspector-dom-storage-malformed.js:74:3 {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: [Object],
      expected: [Object],
      operator: 'rejects',
      diff: 'simple'
    }
    1
  ...
✖  Refusing to resume CI: failures reference files changed by this PR

View workflow run

@panva

panva commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Exclude propagated make exit summaries from file attribution while retaining direct Makefile diagnostics and failures in changed tests.

Refs: nodejs/node#66228 (comment)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants