Skip to content

Disappearing Jenkins jobs #4542

Open
Open

Description

@pimterry

Sometimes Jenkins jobs seem to simply disappear from the history. Each of the ranges below no longer exists, even though the preceding and subsequent builds do:

Each of those ranges has disappeared. When this happens, we end up doing unnecessary reruns which wastes time and CI usage.

Doing a broader survey through the history, searching for #78X on that page, it looks like this currently affects about 30% of builds older after 24 hours:

  • All #786 builds are present
  • #785: 72/100
  • #784: 71/100
  • #783: 62/100
  • #782: 53/100
  • #781: 72/100
  • #780: 70/100

Activity

  1. sxa commented on Nov 21, 2026

    @sxa
    Member

    It looks like it's not a coincidence that you can't see the last resumed link ... it's looking like the missing ones are specifically all of the resumed jobs. The missing builds have a <com.tikal.jenkins.plugins.multijob.MultiJobResumeControl> element on disk with resolves-to="hudson.model.Run$Replacer" in the build.xml on the file system. Removing that section fixes it although (I presume it will no longer know what the original job was. I've done this for 79359 (and told jenkins to reload it's configuration) and you can see that it's now visible although the node-test-commit job underneath it - also a resumed multijob - has the same problem and can't be clicked on

    image

    To look a little further I have also removed the problematic section from node-test-commit#94402 and it now shows correctly in the UI with its subjobs when you click on node-test-pull-request#79359

    image

    I will also note that while it's not necessarily related there is a mismatch between the version of the multijob plugin (which is the one that provides resume functionality) between the node-test-pull-request job definition (669.v9d96a_d9c71b_0) and the one showing in each build of the jobs (682.v4ed91b_52d621 which is the latest one). node-test-commit shows a different version in the configuration - 677.v7ffc23d6a_4c2 but that may just show the version from the last time the job was edited.

    It seems likely that this is related to lazy loading of jobs (perhaps jenkins cannot dereference the linked original job when it loads the resume) but it is unclear at this point if it's related to jenkins itself or the multijob plugin. The plugin did have an update three months ago although this wasn't reported until a bit later (although that could have just been when we did a jenkins upgrade). It would be fairly easy to try and download the plugin to the previous version (677) although I'm not sure I want to be doing any maintenance right now over a weekend (even if that does mean it's a quiet day in the CI!)

    I'll also note that we should also be under no illusions about the multijob plugin - it is considered deprecated and we have an issue relating to this. I've been looking at some initial prototyping of a pipeline-based replacement for these jobs. Having said that I'm tempted to downgrade the plugin to the last version and raise an issue in their repository to at least see if others have noticed it.

  2. sxa commented on Nov 21, 2026

    @sxa
    Member

    Noting that it seems quite likely that some of the security fixes that went into jenkins core version 2.580 were the cause of this. Possibly some of the code relating to https://github.com/jenkinsci/jenkins/blame/ea3f958268fe87ed86a5d1c2d90c5a99426b97ce/core/src/main/java/hudson/util/RobustReflectionConverter.java#L540 - more debug logging would probably be able to confirm this if we chose to do it, but it seems that the deprecated multijob plugin may have fallen foul of security fixes and would need modified to continue to work.

  3. sxa commented on Nov 21, 2026

    @sxa
    Member

    While I'm loathed to patch a deprecated plugin it may be possible ...

    From some analysis by throwing AI at it, here is the current code in ResumeCause.java that may be the cause of the breakage:

    public class ResumeCause extends Cause.UpstreamCause {
        public ResumeCause(Run<?, ?> up) {
            super(up);  // ← This stores Run object, blocked by SECURITY-3972
        }
    }
    

    There are a couple of potential fixes that could work for this but they would be patches to the multijob plugin. Here is one suggested by IBM Bob which came up with something more quickly than I could (I have not tested/verified this): jenkins-multijob-plugin-2580-fix.patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions