Skip to content

test(testkit): one Elasticsearch container per test JVM, reset before each suite - #463

Merged
fupelaqu merged 1 commit into
mainfrom
feature/CI-1
Nov 13, 2026
Merged

fupelaqu merged 1 commit into
mainfrom
feature/CI-1

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Integration suites now share one Elasticsearch container per forked test JVM, instead of starting one each. Before each suite, the container is reset to the state it had right after it started. In PR #462's CI run, the Elasticsearch phase started 222 containers, and booting them took 54.5 min. With this change the same phase should start 16.

Sharing is switched on by a system property, softclient4es.testkit.shared-container=true. Only the Test / javaOptions of elasticsql's five Elasticsearch modules set it. Without the property the testkit behaves as before, so a downstream repository that does not set the property sees no change.

What changes

  • SharedElasticContainers: a registry that holds one container per JVM for each configuration key. The key is the image tag plus the elasticsearch.yml text, rendered by the existing renderer.
    • When a container starts, the registry waits once for the licence to be active and takes a snapshot of the cluster.
    • A shutdown hook stops the containers. Ryuk stays as the backstop.
    • A container that fails to start is never registered.
  • ClusterReset runs in start(), before each suite.
    • It first inspects everything, then deletes, in this order:
      • every index whose name does not start with .;
      • legacy, index and component templates;
      • pipelines;
      • stored scripts;
      • enrich policies;
      • on the secure container only: users, roles and API keys created after the snapshot.
    • Then it closes the cached clients (ElasticClientFactory.shutdown()).
    • It never deletes a system object: one whose name starts with . or contains @, one marked _meta.managed, one already in the snapshot, or a reserved user.
    • A kind of object no suite should leave behind makes the reset fail and name the suite that left it, without deleting anything: watchers, transforms, cluster settings, or a change of licence.
    • Each list endpoint must answer 200 (read) or 404 (nothing to delete). Any other status fails the suite and names the method, path and version.
  • ElasticDockerTestKit: start() and stop() check whether elasticContainer is a shared container, by identity. The property only chooses the default elasticContainer. A suite that overrides elasticContainer (for example a basic-licence container) keeps its own container and today's lifecycle, with or without the property.
  • ElasticContainerSharing: an opt-out member, sharedContainer. GatewayApiIntegrationSpec and ReplGatewayIntegrationSpec set it to false, because they assert on the whole cluster. They keep a container each.
  • Control suites: SharedContainerResetSpec (one runner per client module) and JavaClientSharedSecureResetSpec (es8, secure container). They plant every kind of object, then check that the reset deletes them and keeps the system ones. They also check that a refused kind fails the next reset with the right suite name.
  • Logs:
    • one [es-container] line per container started;
    • one [es-reset] line per reset, with its duration and what it deleted;
    • one [es-leaks] line after each suite: open scroll and search contexts, and script compilations.
    • -oD adds each test's duration in the five Elasticsearch modules.
  • Workflows (build.yml and release.yml):
    • a "Runner info" step (lscpu, nproc, free -m, docker info);
    • the JUnit XML reports uploaded as the test-reports artifact, even when tests fail.
  • CONTRIBUTING.md documents:
    • the property and the opt-out;
    • how to run a module with the property removed, and with the suite order reversed;
    • a warning that suites must stay serial within a module.

Verification

Elasticsearch 8.18.3, es8java/test, main against this branch:

run suites succeeded / failed / canceled containers time
main 53 992 / 0 / 3 48 688 s
branch 55 (53 + 2 control suites) 997 / 0 / 3 4 217 s
branch, suite order reversed 55 997 / 0 / 3 4 204 s
branch, property removed, control suites filtered out 53 992 / 0 / 3 48 653 s
  • Same results as main: in all three branch runs, every suite of main gives the same counts (succeeded, failed, canceled, ignored, pending, aborted) as on main. The counts come from the test log, because the JUnit XML records a canceled test as a pass.
  • Reversed run: it runs the suites in exactly the reverse order of the forward run.
  • Run with the property removed: it starts every container as unshared and prints no reset or leak line.
  • Other checks, all with no failure:
    • control suites on Elasticsearch 8.18.3: 5/0;
    • ClusterResetPlanSpec: 19/0;
    • ElasticDockerTestKitLifecycleSpec: 5/0 (no Docker);
    • sbt "+ compile" (2.12.20 and 2.13.16);
    • headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck.
  • Elasticsearch 6.8, 7.17 and 9.0 run on CI only. This PR's CI run is the check for them; see the next section.

CI run of this PR

Compared with #462's run, using its complete job log:

#462 (main + #462) this PR
runner westus eastus, 4 vCPU, 16 GB
sbt + compile 4.7 min 4.7 min
sbt compile test 110.4 min 48.1 min
Elasticsearch containers started 222 16
time spent booting them 54.5 min 5.7 min
  • Containers by module: es6rest 3, es6jest 3, es7rest 3, es8java 4, es9java 3. In each module, the two gateway suites keep their own container; the other suites share one (es8java also has the secure container).
  • About 49 of the 62 minutes saved are container boots. The rest is runner speed, which varies between runs.
  • Resets: 229 resets took 21.4 s in total (median 0.06 s, maximum 0.94 s) and deleted 1,082 objects. No reset refused. After every suite there were 0 open search contexts and 0 open scrolls.
  • Counts: every module equals feat(core): describe a JOIN view body per table without running it #462's counts plus the suites this PR adds (succeeded / failed / canceled; aborted 0 everywhere):
module #462 this PR added
es6rest 53 suites, 905 / 0 / 18 54 suites, 908 / 0 / 18 1 control suite, 3 tests
es6jest 50 suites, 914 / 0 / 18 51 suites, 917 / 0 / 18 1 control suite, 3 tests
es7rest 49 suites, 906 / 0 / 3 50 suites, 909 / 0 / 3 1 control suite, 3 tests
es8java 53 suites, 992 / 0 / 3 55 suites, 997 / 0 / 3 2 control suites, 5 tests
es9java 48 suites, 907 / 0 / 3 49 suites, 910 / 0 / 3 1 control suite, 3 tests

Release note

  • Downstream rebuild: the published softclient4es{6,7,8,9}-core-testkit changes. ElasticDockerTestKit gains members, and GatewayApiIntegrationSpec and ReplGatewayIntegrationSpec now mix in ElasticContainerSharing. Test classes that extend them must be rebuilt.
  • No case-class arity change.
  • Without the property, downstream test behaviour and container count do not change. Each repository can opt in later.

Recorded, not fixed

  • None.

🤖 Generated with Claude Code

… each suite (CI-1)

Story CI-1. Reviewed: r0:must-fix, r1:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fupelaqu
fupelaqu marked this pull request as ready for review November 13, 2026 17:29
@fupelaqu
fupelaqu merged commit e4605cb into main Nov 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant