Skip to content

blog: add post about Camel Core CI improvements - #1831

Merged
apupier merged 1 commit into
apache:mainfrom
apupier:blogpostAboutCICare
Nov 16, 2026
Merged

apupier merged 1 commit into
apache:mainfrom
apupier:blogpostAboutCICare

Conversation

@apupier

@apupier apupier commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview is available at https://pr-1831--camel.netlify.app

@davsclaus davsclaus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TL;DR

Comment — a well-documented post on how the Camel Core CI went from a few green builds a year to several a day. The front matter, images, tables and reference-style Develocity links all render correctly in the Netlify preview, and the PR counts I checked match GitHub (3,326 merged on main in 2025, 415 backports to 4.18.x in 2026). Ready to merge once the numbers question below is settled.

Blockers

  • None

Other findings: 1 question (build-time figures), 2 markup suggestions, 2 optional content ideas. See the inline comments.

Thanks for writing this up, Aurélien. The before/after tables and the Map.<String, String> entry witness trick are really useful to share.


This review covers project conventions and content checks only. It is not a substitute for specialized review tools or static analysis.

This review was generated by an AI agent and may contain inaccuracies. Please verify all suggestions before applying.

| Metric | A year ago | Today |
|--------|-----------|-------|
| Green Jenkins builds on main | 1 to 4 per year | several per days |
| Duration of the Jenkins build for main | 5h45| 1h24 |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question (non-blocking): the build times in the post don't seem to match each other:

  • here: main build 5h45 → 1h24
  • the -T1C table further down: ppc64le 5h45 → 1h41, x86_64 with Sonar 5h20 → 1h47, x86_64 without Sonar 4h09 → 56min
  • "What's next": "our 1h25 build with Sonar"

The 5h45 matches ppc64le, but 1h24 is none of the "after" values. If 1h24 is the wall-clock time of the whole matrix and the table shows single cells, or the numbers were measured at different times, one sentence saying so would help. Otherwise readers may think one of the figures is a mistake.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this top-level includes several improvements, not only the -T1C, which explains the difference with the other specific -T1C table

* Lack of synchronization between expected state of the system and next steps in the test

Some generic solutions:
* Use of @Isolated, this is often the easy solution, at the cost of performance.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (markup): the rest of the post puts annotations, flags and paths in code spans (`@Isolated` on line 241, `-T1C` in the headings), but a few are plain text: @Isolated here, -T1C on line 143, .m2 on line 84 and etc/scripts/regen.sh on line 132.

Suggested change
* Use of @Isolated, this is often the easy solution, at the cost of performance.
* Use of `@Isolated`, this is often the easy solution, at the cost of performance.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: @Isolated, -T1C, .m2 and etc/scripts/regen.sh are now in code spans.


On top of that, we started hitting the **GitHub Actions limits**, which are **shared among the whole Apache GitHub organization**. The queue time can go up to dozens of hours on worst days, regularly dozens of minutes. On a 30 minutes job that's quite consequent and slowing down the development process.

It was time to give a bit more love to our CIs. The focus was first on Jenkins, but several actions benefitted to both of them. This is an on-going work but let's share some of the improvements results today as we reached an important milestone last week on Jenkins CI.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional idea: most readers of the blog use Camel and don't contribute to it, and for them the payoff of this work is indirect. A stable, faster CI is what makes the jump to 820 backports this year possible, and it lets LTS patch releases ship sooner with more fixes. A sentence here saying what this means for people who use Camel would give them a reason to keep reading. How to say it is your call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I added an introduction stating that a stable and fast CI is what allows a higher throughput of new features and CVE fixes, and that faster, more reliable feedback means more changes backported to the maintained branches and security fixes reaching users sooner.

Some generic solutions:
* Use of @Isolated, this is often the easy solution, at the cost of performance.
* Increase timeout. Beware, it is often the first solution that AI agents provide, it is not that often working. Need to check first if the test is close to the timeout when passing, otherwise it is usually another problem and it will only increase the time before the test is failing.
* Use more conditional checks with `Awaitility` to ensure that systems are ready, or what we want to test is in the expected state

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional idea: the Camel Integration Testing Pitfalls post from last week covers the route-testing side (timer-driven routes, eager initialization, weak assertions), so the two posts go well together. For example:

Suggested change
* Use more conditional checks with `Awaitility` to ensure that systems are ready, or what we want to test is in the expected state
* Use more conditional checks with `Awaitility` to ensure that systems are ready, or what we want to test is in the expected state
For pitfalls in how the routes themselves are tested, see also [Camel Integration Testing Pitfalls](/blog/2026/10/camel-testing-pitfalls/).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the pointer. I did not add the cross-link in this post, to keep it focused on the CI side.

- Investigate/improve Sonar which is not leveraging parallelization or multi-core CPUs at all. Currently adding 30 minutes on our 1h25 build with Sonar.
- Investigate build and test time of modules on critical paths
- Investigate other Camel projects
- Apply new recommendation policies for GitHub Workflows which are emerging in Apache community, for instance from discussions in Slack channel [#project-workflow-optimizations](https://the-asf.slack.com/archives/C0B835NQG6P)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note (non-blocking): only people with an ASF Slack account can open the-asf.slack.com, so most readers will hit a login page. You could say that in the text, or link to a public thread on a mailing list if there is one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the text now says it is the ASF Slack and requires an Apache account or an invitation.

@apupier
apupier force-pushed the blogpostAboutCICare branch from 53115a4 to e208820 Compare November 13, 2026 14:00
@apupier
apupier marked this pull request as ready for review November 13, 2026 14:18
@apupier
apupier requested review from davsclaus and oscerd November 13, 2026 14:18
Comment on lines +289 to +295
- (Re)activate test parallelization for some modules where it was disabled. Can be both from Maven JUnit config or with `@Isolated` or even `@Order` JUnit annotations.
- Tweak test parallelization with the best number of threads and forks.
- Investigate/improve Sonar which is not leveraging parallelization or multi-core CPUs at all. Currently adding 30 minutes to our 1h25 build with Sonar.
- Investigate build and test time of modules on critical paths. A [critical path analysis](https://issues.apache.org/jira/browse/CAMEL-24290?focusedId=18143592&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18143592) based on Develocity build scans lists the most interesting candidates.
- Investigate other Camel projects
- Apply new recommendation policies for GitHub Workflows which are emerging in Apache community, for instance from discussions in Slack channel [#project-workflow-optimizations](https://the-asf.slack.com/archives/C0B835NQG6P) (ASF Slack, requires an Apache account or an invitation)
- Leverage the new [CI usage dashboard](https://ci-test.apache.org/#/overview?) to monitor the usage of our jobs and to spot the ones consuming the most shared resources

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have more ideas but I prefer to stay simple here

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview is available at https://pr-1831--camel.netlify.app

@apupier
apupier force-pushed the blogpostAboutCICare branch from e208820 to 2711f21 Compare November 16, 2026 07:47
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview is available at https://pr-1831--camel.netlify.app

Co-authored-by: IBM Bob 2.2.1
Signed-off-by: Aurélien Pupier <apupier@ibm.com>
@apupier
apupier force-pushed the blogpostAboutCICare branch from 2711f21 to a020f66 Compare November 16, 2026 12:46
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview is available at https://pr-1831--camel.netlify.app

@apupier
apupier merged commit dc2c4a3 into apache:main Nov 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants