[#1090] Check replication convergence under load next to the Docker tests, and run the Docker tests of both images after their builds

[vharseko]

Fixes #1090 with the base variant (points 1, 2, 4, 5 and 6 of the proposal). The disturbances of point 3 follow one at a time in later PRs, under #1208.

What it does

.github/scripts/replication/run.sh <image> writes to every directory server of a replication topology at once, then checks that the replicas hold the same data. It runs in CI and on a laptop with Docker alone; every setting comes from the environment (lib.sh).

1. Topology (topology-up.sh): TOPOLOGY=rs534 - four directory servers without a replication server and two replication servers without data, the topology of Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 - or TOPOLOGY=combined - four combined servers. Every server joins through the background join of the image (OPENDJ_REPLICATION_TYPE=simple) in its REPLICATION_ROLE (Docker image: the background join has no server role, so a standalone replication server or a directory server without one needs the deprecated one-shot types #1178), and the run waits for every join to end, not only for every container to be healthy: a join goes on repairing the replication server lists after that. The run checks who runs a replication server and that every directory server holds the seeded users.

2. Load (load.sh), on every directory server at once for DURATION seconds, from containers of the same image, over plain LDAP with eight connections per client kept open and bound once (--noRebind), up to LOAD_MOD_RATE modifications and LOAD_ADD_RATE adds or deletes a second per directory server (325 and 65), with no container memory limit and the heap bounded (SERVER_JAVA_ARGS -server -Xmx1g, CLIENT_JAVA_ARGS -Xmx256m): modrate on the shared users from every server (concurrent modifications of one entry, Replication of modify operations on same DN may be replayed out of order #139), addrate with fifo deletes under a uid prefix of each server (no naming conflicts). The values modrate writes start with the name of their server: the rate tools seed rand and randstr alike in every client, so all of them would write the same values in the same order, a change lost on its way could hide behind the same value written later by another server, and a value in a diff would not tell which server made it. Writes only: a read sends nothing to the replication servers. A rate tool exits with 0 whatever it got back, and one whose connection breaks stops early, still with 0, leaving out the interval that would count the error: so each client must have done work, reported no errors in its CSV output, exited with 0, printed nothing besides its CSV output and covered the whole load.

3. Convergence (monitor.sh): every replication server knows every directory server of the domain and none misses a change, on three samples in a row that span a monitoring period (60 s: a replication server computes missing-changes once a period, and a server its last computation does not know reads as 0) - a missing monitor entry is not a zero. A sampler records missing-changes and approximate-delay for the peaks, asking every replication server at once (each ask is an ldapsearch, a JVM that takes tens of seconds to start on a busy runner); stopping the sampler removes what the sample it cuts short leaves behind.

4. Checks on the data (assert-converged.sh), since missing-changes reaching 0 is not enough:

   • online: the same equality searches on indexed attributes return the same DNs on every replica (Replicated objectClass value additions are not written to the objectClass equality index #1021-class index drift, which an export does not show); no ds-sync-conflict entry; every replication server connected to every other one; no SYNC error, TLS handshake failure or replay give-up in any errors log after the topology came up. A search or a read of a log that fails is an error of its own: searches that failed alike on every replica would give the same empty answer;
   • offline, servers stopped: export-ldif --offline of every replica without ds-sync-hist/ds-sync-state, a guard against empty exports, ldifdiff against the first replica (the differences go to diff/), verify-index;
   • the changelogs, servers stopped: changelog-gaps.py reads the changelog of every replication server, streamed out by docker cp, and reports the sequence numbers missing among the changes each replica made during the load. A directory server numbers its changes one after the other (CSNGenerator.newCSN()) and no operation of the load fails, so a gap is a change that reached no replication server - which the data shows only until a later change overwrites it, and missing-changes never (Replication: two deletes a directory server made while switching replication servers under load reached no replication server, and missing-changes still reached 0 #1195). A change lost after the changelog, on its way to a directory server, leaves no gap there; the data shows it (Replication: a replication server skipped minutes of one replica's changes for a directory server catching up, and missing-changes still reached 0 #1206). Every changelog must hold changes of every directory server, so that records the script no longer reads do not pass for a changelog without gaps. What it cannot see: a change lost at either end of a replica's changes has no gap after or before it, and a directory server restarted during the load would read as a lost change - its ReplicaOfflineMsg takes a sequence number that only offline.state holds, and only the newest one; the run restarts none.

   The offline checks and the changelogs are checked after failed online checks as well, so that a run says which changes are missing where.

5. What is left behind (collect.sh): summary.md and the job summary - convergence time, how often the monitor was sampled, peak missing-changes and approximate-delay (seconds) and the reconnections to a replication server per directory server (msgID 62, whatever made them: a switch the broker decided on, 188 or 213, a connection the replication server closed or that broke; changes made around one were lost in Replication: two deletes a directory server made while switching replication servers under load reached no replication server, and missing-changes still reached 0 #1195), counted also when a run fails before its checks, the changes missing from the changelogs, throughput per client, time per phase; on failure state.tgz with every server's container log, logs/, config/ and changelogDb. OUT must be new or empty: the files of an earlier run would mix into the summary.

CI

The registry of a build job is a service of that job and is gone with it, so the image goes over as an artifact:

build-maven ─┬─ build-docker         (every platform, as before; + linux/amd64 image → artifact)
             └─ build-docker-alpine  (same)
                    ├─ docker-test        matrix: default, alpine   - the test steps of both jobs
                    └─ docker-convergence matrix: rs534, combined   - this run, DURATION=900

docker-test holds the test steps of build-docker and build-docker-alpine once. Expanded per flavor, its steps were byte for byte the old ones (checked with a script); since then two settings changed on purpose: the containers of both flavors run with 512m (alpine had 1g), and both benchmark artifacts are kept 5 days (3 and 90 before). The PDB vs JE benchmark stays with the default image. The tests, Trivy and the benchmarks now run next to the convergence check instead of after the build in the same job.

Three couplings of the jobs, each stated in a comment where it happens:

• A red docker-convergence row turns the Build run red, and on a push to master deploy.yml then skips the snapshot deploy and the publishing of the docs. rs534 fails until Replication: a replication server skipped minutes of one replica's changes for a directory server catching up, and missing-changes still reached 0 #1206 is fixed and combined until Replication: two deletes a directory server made while switching replication servers under load reached no replication server, and missing-changes still reached 0 #1195 is.
• docker-test runs when either image was built (as the tests of each image ran in its own build job before): a matrix row cannot need its own build only, so the row of an image that was not built fails at its download.
• The linux/amd64 images, about 250 MB each, are kept 3 days: "Re-run failed jobs" does not build them again, so it works within those days, and "Re-run all jobs" after them.

Testing

This round (the review of maximthomas): no CI run yet. Each check it changes was run on stubs and on the artifact of run 38893291531, against the scripts before the round:

case	before	now
a client whose output stops at 390 of 900 s (39 lines)	passes	red: stopped after 39 of 89 intervals
a client that printed a connection error	passes	red
a client that exited with 1	passes	red
the indexed searches failing alike on every replica	pass	red
the ds-sync-conflict search failing on one replica	passes	red
the errors log of one server that cannot be read	passes	red
UPDATE_TYPES = {} in changelog-gaps.py, on the changelog of cv-r0	passes	red: no change of the load by cv-d0 ... cv-d3
converged on every sample	after 3 samples, 21 s	after 9 samples, 63 s; CONVERGED_IN is the first of them
converged on 4 samples, behind on the fifth at 30 s	done after the third	the fifth starts the series again
rm fails in the stop of the sampler	the shell exits	the run goes on
a directory server without a server id	passes	red
a run that fails before the online checks	reconnections -	counted

What the round does not change came out the same before and after: the client output of a whole run, the same answers everywhere, a conflict entry, answers that differ, the whole changelog of cv-r0 (green, green, red, red, green). The reconnections in the errors logs of that run, which go on to the shutdown: cv-d2 8 - each of its 8 switches of #1198 connected it again to the replication server it left -, cv-d1 1, its reconnection while the servers stopped, the others 0. run.sh with an OUT that holds a file exits with 2 before it touches Docker. The wait for the end of the joins is the one of docker-test-replication.sh, which runs on both images; it has not run here (no Docker on the host). build.yml: actionlint 1.7.7, no findings.

CI, run 38893291531 (the sixth round: values of its own for every modrate client, the stop of the sampler): docker-test (default) and docker-test (alpine) green, combined green, converged 212 s after the load. rs534 failed rightly, again on a loss on the way to a directory server (#1206): cv-d2 kept 12 entries the others had deleted - 7 of cv-d0, 5 of cv-d3 - while both changelogs held every change; it converged 40 s after the load. cv-d2 switched replication servers 8 times (msgID 188), and each switch connected it again to the replication server it left - #1198, fixed by #1205 since, which this branch is now on. Before CI, with start_client stubbed, each modrate client got a template of its own (description:cv-d0-%2$s, description:cv-d1-%2$s, ...), which String.format - as ModRate calls it - turns into description:cv-d0-AbCdEfGhIjKlMnOp; stopping the sampler in the middle of a sample (a stubbed admin_search that takes 3 s) left its sample.* directory behind before that round and left nothing after it.

CI, run 38865878001 (the fifth round: every replication server asked at once, the switch count, the changelog check): every job that finished was green but docker-convergence (rs534) (the docker-test jobs were still running when this round was pushed). rs534 failed rightly, on a loss of another kind than #1195: a replication server skipped minutes of the changes of one directory server for another that was catching up - about 70 thousand changes of cv-d2 never reached cv-d3, about 24 thousand of cv-d3 never reached cv-d2 - with nothing logged and missing-changes at 0 (converged 348 s after the load), now #1206. The changelog check found no gap in the changelog of either replication server - every change was there, the loss was on the way out - and none in combined, so it raised no false alarm on a complete changelog. The sampler took 22 samples in 900 s on rs534 (13 in the fourth round) and counted 10 switches of cv-d1 (the pattern of #1198), none elsewhere. Two things this round changes came out of it: which server made a stale value took the changelog to tell, since every modrate client wrote the same values, and a sample.* directory the stop of the sampler had cut short was uploaded with the run. So docker-convergence (rs534) can fail until #1206 is fixed, as combined until #1195.

The fifth round, before CI: changelog-gaps.py, the switch count and the summary were run on the state of the failed combined run of #1195 (run 38605759421 attempt 1, artifact convergence-combined) with docker stubbed: the replica of cv-c3 misses 81 changes in all four changelogs - the number the analysis of #1195 found independently - the other three replicas none, also over the whole history including the setup of the topology; cv-c3 switched replication servers once. Asking the replication servers at once took one ask's time for four of them. A whole local run was not possible (no Docker on the host at the time).

CI, run 38755406531 (the fourth round, 325 modifications and 65 adds or deletes a second for 900 s, heaps of 1 GB and 256 MB): every job green. The clients reached 92-98% of their ceilings, about 1500 writes a second over the topology. rs534 converged 269 s after the load: cv-d1 and cv-d2 fell behind steadily, about 300 thousand changes and 320 s at the end of the load, and caught up. combined converged 83 s after the load. The sampler took 13 samples in 900 s there, one ask after the other, which this round changes.

CI, run 38694855431 (the third round, 325 modifications and 65 adds or deletes a second for 900 s, no bound on any heap): both docker-convergence runners lost communication with GitHub in the middle of the run, with no log and no artifact left. Fourteen JVMs that may each take a quarter of a 16 GB runner are the likely cause, so the heaps are bounded again. docker-test (default) failed in Docker test replication (dj-r1 did not join through dj-r0, the joins on empty volumes of #1178), which this PR does not touch.

CI, run 38605759421 (the second round, at 250 modifications and 50 adds or deletes a second for 300 s): the clients reached their ceilings, about 1180 writes a second over the topology. rs534 passed. combined failed in attempt 1, and rightly: two deletes cv-c3 made while its directory server switched replication servers reached no replication server, the other replicas kept both entries, and missing-changes reached 0 all the same - #1195. Attempt 2 of the same job passed, with peaks of missing-changes up to 58078 and of approximate-delay up to 105 s: the loss depends on a delete landing in the window of a switch. So docker-convergence (combined) can fail until #1195 is fixed.

CI, run 38528537312 (the first round of this PR, before the load moved to plain LDAP on open connections): every job green. docker-convergence took 14 minutes for each topology, next to docker-test (103 and 117 minutes). rs534 converged 22 s after the load, combined 75 s; the clients reached 10-30 operations a second each, since each operation opened an LDAPS connection and bound first, which the second round removed.

Locally, Docker 29.8.2, 4 CPU / 4 GB, image built from this branch:

• combined and rs534 with the load for 120 s: green; combined for 300 s: green, converged 83 s after the load, ldifdiff found no attribute that differs beyond the two excluded.
• combined with the load of this round (plain LDAP, open connections, writes only) for 60 s: green, converged 158 s after the load.
• INJECT_DIVERGENCE=1 (one change and one entry on one replica only, through the replication repair control): red on the indexed searches and on ldifdiff, whose output is exactly the injected change.
• Each guard red on its own case: an export guard demanding more entries than there are, a planted SYNC error in an errors log, a stopped replication server (the others see 2 of 3 peers), a directory server whose monitor entry is gone (the wait times out and prints the monitor), a load client that reaches no server or binds with a wrong password.
• The load checks once summed the client rates in the host's locale, where awk reads 0.5 as 0 with a decimal comma; the run sets LC_ALL=C (red on a saved client output before, green after).
• build.yml: actionlint 1.7.7, no findings.

Not done: rs534 with the load for 300 s locally. The first run failed on a load client that printed nothing (the run now records each client's exit code and OOM kill); the second was stopped for lack of memory on the host. On 4 CPU / 4 GB with the host itself busy, the servers get a few operations a second from each client, so the local runs check the logic, not the throughput. Once, locally, a combined server did not turn healthy within an hour while the topology came up; its logs were lost to the cleanup, which KEEP=1 and state.tgz now prevent.

[maximthomas]

praise: The harness checks the data itself, and its checks hold up against real runs.

• changelog-gaps.py reads the real changelog: on the convergence-rs534 artifact of run 38865878001 it parsed 327697-344522 changes per replica with contiguous sequence numbers, and cv-d0's 344522 matches its clients' 318.5 + 64.3 ops/s over 900 s.
• docker-test, expanded per flavor with the matrix values substituted, is the old test steps of build-docker / build-docker-alpine except the three changes the description states (512m for alpine, 5-day retention, PDB vs JE on default only).
• assert_offline refuses exports that failed alike (assert-converged.sh:69-73), and verify-index's error count cannot wrap to 0 (Utils.filterExitCode caps it at 255).

---

question (blocking): Should a red docker-convergence row on a push to master skip the whole of deploy.yml?

.github/workflows/build.yml:1105-1144, .github/workflows/deploy.yml:18-21, :44

docker-convergence has no continue-on-error and no job-level if, so it runs on every push of build.yml:18-19, master included. deploy.yml runs package-deploy-maven only when the Build run concluded success (:44). The description expects rs534 to fail until #1206 and combined until #1195, and both issues are open. So every master push with a red row loses the snapshot deploy, the docs site and wiki publish and the launcher refresh, and the description does not mention that. If that is intended, say so in the description and in a comment on the job. If not, keep the row visible but out of the run's conclusion until both issues are fixed:

  docker-convergence:
    # #1090: writes to every directory server of a replication topology at once, then checks
    # that the replicas hold the same data; next to docker-test, on the image of build-docker
    needs: build-docker
    # rs534 fails until #1206 and combined until #1195 are fixed: a red row must not turn the
    # Build run red, which deploy.yml reads as "skip the snapshot deploy and the site publish"
    continue-on-error: true

This relies on GitHub's documented job-level continue-on-error behaviour; this review did not run it.

---

issue (non-blocking): The online checks read a failed search or log read as a pass.

.github/scripts/replication/assert-converged.sh:166, :175, :142; .github/scripts/replication/run.sh:83

assert_online || checks=... turns errexit and the ERR trap off inside the function, so the status of ldaps_search | awk | sort at :166 is lost. A search that fails the same way on every DS gives equal empty dns/*.txt files, and cmp passes. [ -z "$(ldaps_search … | grep '^dn:')" ] at :175 drops the status in any context: a failed conflict search on the node that holds conflicts reads as "no conflicts", and nothing offline checks conflicts. since_mark | grep … || true at :142 reads an errors log that could not be read as "no replication errors". A probe under run.sh's set -eE -o pipefail and trap ran the or-call to its end, nothing flagged, exit 0. assert_rs_mesh (:152) stays as is, since got=0 is flagged, and so does record_switches (:137), which counts rather than checks:

    for n in $DS_NODES; do
      ldaps_search $n "$BASE_DN" sub "$f" 1.1 >"$OUT/dns/$i.$n.ldif" \
        || { echo "::error::$f failed on $n"; failed=1; }
      awk '/^dn: /{ print tolower($0) }' "$OUT/dns/$i.$n.ldif" | sort >"$OUT/dns/$i.$n.txt"
    done
  # …
  for n in $DS_NODES; do
    # the load makes no naming conflicts: each server adds under a uid prefix of its own
    if ! out=$(ldaps_search $n "$BASE_DN" sub "(ds-sync-conflict=*)" 1.1); then
      echo "::error::the ds-sync-conflict search failed on $n"; failed=1
    elif grep -q '^dn:' <<<"$out"; then
      echo "::error::$n holds ds-sync-conflict entries"; failed=1
    fi
  done
  for n in $ALL_NODES; do
    if ! lines=$(since_mark $n); then
      echo "::error::the errors log of $n could not be read"; failed=1; continue
    fi
    lines=$(grep -E "category=SYNC severity=ERROR|SSLHandshakeException|bad_certificate" <<<"$lines" || true)
    [ -z "$lines" ] || { printf '%s\n' "$lines" >"$OUT/logs/$n.errors"; echo "::error::$n logged replication errors, see logs/$n.errors"; failed=1; }
  done

Add out to the local line of assert_online. errors_since_mark then has no caller left.

---

issue (non-blocking): check_client passes a client that stopped before the end of the load.

.github/scripts/replication/load.sh:99-107

check_client looks at the exit code and the OOM kill only on its "did no work" branch, and it never compares the row count with DURATION. Take a client whose kept-open connection drops at t≈400 s. PerformanceRunner calls stopTool(true), StatsThread.stopRecording(true) skips the last row, and the tool exits 0. The check then sees 39 rows, ops > 0 and no errors, and passes, even though the load stopped at 400 of 900 s and a request failed. A request that fails after the server assigned its CSN then shows up as a changelog gap reported as a lost change. Run 38865878001 printed 90-91 rows for 900 s at --statInterval 10:

check_client() { # <client>
  awk -F, -v c=$1 -v exit_state="$(cat "$OUT/load/$1.exit")" -v want=$((DURATION / 10 - 1)) '
    /^Time \(seconds\)/ { for (i = 1; i <= NF; i++) { if ($i == "Recent throughput (ops/second)") t = i; if ($i == "Errors/second") e = i }; next }
    t && $1 ~ /^[0-9.]+$/ { rows++; ops += $t; errors += $e; last = $0 }
    END {
      if (!rows || ops == 0) { printf "::error::%s did no work (exit code and OOM kill: %s)\n", c, exit_state; exit 1 }
      if (errors > 0) { printf "::error::%s got errors back\n", c; exit 1 }
      if (exit_state != "0 false") { printf "::error::%s ended with exit code and OOM kill %s\n", c, exit_state; exit 1 }
      if (rows < want) { printf "::error::%s stopped after %d of %d intervals\n", c, rows, want; exit 1 }
      printf "%s: %d intervals, last: %s\n", c, rows, last
    }' "$OUT/load/$1.csv"
}

---

issue (non-blocking): The switch count leaves out the load-balancing switches (msgID 213).

.github/scripts/replication/assert-converged.sh:131-139

ReplicationBroker (:2600-2619) logs NOTE_LOAD_BALANCE_REPLICATION_SERVER (213) on the other branch of the same decision as 188, then makes the same reStart(true). In rs534 a DS that leaves an overloaded RS logs 213, so the summary's switches column reads 0 for a run that did switch.

    echo "$n $(since_mark $n | grep -E "msgID=(188|213) " | grep -cF "for domain \"$BASE_DN\"" || true)"

---

issue (non-blocking): sampler_stop's rm can fail a green run.

.github/scripts/replication/monitor.sh:91-101, :33-40

kill/wait $SAMPLER reach only the sampler subshell. The admin_search | awk >"$dir/$rs" & children it has forked live on. If a child opens its file after rm -rf has read the directory, rmdir fails with ENOTEMPTY and rm exits 1. Under set -eE that marks a green run failed, and inside the ERR trap errexit stops collect and the cleanup. The window is rare (between a fork and its open), but nothing guards it.

    rm -rf "$OUT"/sample.* || true

---

suggestion (non-blocking): Count the switches of a run that fails before the online checks too.

.github/scripts/replication/run.sh:40, .github/scripts/replication/assert-converged.sh:162

record_switches has one caller, assert_online. A load check or a wait_converged timeout, the failures a switch under load would explain, therefore leave "-" in the column while the containers are still up.

trap 'code=$?; [ "$BASH_SUBSHELL" -eq 0 ] || exit $code; sampler_stop; [ -f "$OUT/switches.txt" ] || [ ! -f "$OUT/logs/${DS_NODES%% *}.mark" ] || record_switches || true; timings; collect; summarize failed; [ "$KEEP" = 1 ] || cleanup; exit $code' ERR

---

question (non-blocking): Is it intended that a failed build of one flavor skips the docker tests of both?

.github/workflows/build.yml:634-637

docker-test needs both builds, so a failed or cancelled build-docker-alpine, for example on one of its QEMU platforms, skips docker-test (default) as well. At BASE the default image's tests, Trivy and the benchmarks ran in build-docker. The run is red either way, so this loses signal but gives no wrong verdict: Minor. If it is not intended, gate on either build. The row whose image was never uploaded then fails at its download. A job-level if cannot read matrix; this review did not run either point:

    needs: [build-docker, build-docker-alpine]
    if: ${{ !cancelled() && (needs.build-docker.result == 'success' || needs.build-docker-alpine.result == 'success') }}

---

question (non-blocking): Is the 1-day retention of the image artifacts a deliberate storage trade-off?

.github/workflows/build.yml:544-549, :627-632

On a red convergence row, which is expected while #1195 and #1206 are open, "Re-run failed jobs" does not re-run the passed build jobs. More than 24 h later it fails at "Download the image". At BASE a re-run rebuilt from the 5-day build-maven artifact. The tars are about 250 MB each (264012084 and 250900074 bytes on run 38865878001). If the retention is not deliberate, keep them for retention-days: 5. If it is, a comment saying that a late re-run needs "Re-run all jobs" would do.

---

suggestion (non-blocking): Make the three converged samples span one monitoring period.

.github/scripts/replication/monitor.sh:64-80

A replication server rebuilds missing-changes once per monitoring-period, 60 s by default and set nowhere under .github. A server the snapshot does not hold reads as 0 (ReplicationDomainMonitorData.getMissingChanges). Three samples about 5 s apart can therefore all read one all-zero snapshot taken while changes are still in flight. The data checks would then fail on changes still on their way: a false red, and a CONVERGED_IN that is too short. Not run: it needs a run that logs the monitor build times or injects a DS switch after the load. No such early convergence was seen on the cited runs.

wait_converged() {
  local start=$SECONDS hits=0 first=
  until [ $hits -ge 3 ] && [ $((SECONDS - first)) -ge 60 ]; do
    if converged_once; then hits=$((hits + 1)); first=${first:-$SECONDS}; else hits=0; first=; fi

---

suggestion (non-blocking): Wait for the joins to end, not only for "healthy", before marking the logs.

.github/scripts/replication/topology-up.sh:46-51

join.sh touches BOOTSTRAP_COMPLETE (:965) before repair_replication_servers (:967), which can still wait for peers and rewrite replication-server lists. Such a write after mark_logs would put its errors into the run's errors check and a broker restart into the switch count. On run 38865878001 no repair wrote anything: after membership the container logs show only "waiting for … to register" lines. So nobody has seen this road; it was not traced through join.sh:509-565. docker-test-replication.sh:182 (bootstrap_ended, used at :781) already waits for exactly this.

---

suggestion (non-blocking): The changelog check passes when no change of a directory server matches the parser.

.github/scripts/replication/changelog-gaps.py:81-82, :123-124; .github/scripts/replication/assert-converged.sh:92-109

A record with an unknown type byte, or a changelog with no *.log file, is skipped, and the script exits 0 with nothing to report. A format drift that only filters records out therefore leaves the #1195 detector vacuous while CI stays green. The parser matches today's layout, as shown above.

    for id in $(awk '{ print $1 }' "$OUT/server-ids.txt"); do
      grep -q "^replica $id (" "$OUT/changelog/$n.txt" \
        || { echo "::error::the changelog of $n holds no change of the load by $(grep "^$id " "$OUT/server-ids.txt" | cut -d' ' -f2)"; failed=1; }
    done

Pin: with this loop in assert_changelogs, the mutant UPDATE_TYPES = set() turns the run red.

---

suggestion (non-blocking): No CI row shows that the checks can fail.

.github/workflows/build.yml:1129-1136, .github/scripts/replication/lib.sh:40, .github/scripts/replication/run.sh:74-77

INJECT_DIVERGENCE is never set in CI. These mutants stay green on every row: cmp -s replaced by true, the ldifdiff || {...} dropped, case 1) mapped to 0) in assert_changelogs.

      - name: Convergence self-test (the checks must report the injected divergence)
        if: matrix.topology == 'combined'
        shell: bash
        env:
          TOPOLOGY: combined
          DURATION: "60"
          SAMPLE_DATA: "1000"
          INJECT_DIVERGENCE: "1"
          OUT: ${{ runner.temp }}/replication-selftest
        run: |
          ! .github/scripts/replication/run.sh "localhost:5000/${GITHUB_REPOSITORY,,}:${{ needs.build-docker.outputs.release_version }}" > "$RUNNER_TEMP/selftest.log" 2>&1
          grep -q "(sn=Diverged) returns other entries" "$RUNNER_TEMP/selftest.log"

---

suggestion (non-blocking): The changelog check cannot see a lost first or last change of a replica.

.github/scripts/replication/changelog-gaps.py:140-141

Gaps are counted only between consecutive changes inside the window. If a replica's last change of the load, or a tail of them, reaches no replication server, the check prints "0 missing". The data shows such a loss only until another server overwrites it (assert-converged.sh:87-91). Pin: count the gap between the last change before the window and the first one inside it, and have run.sh make one marker change on every DS after run_load and before LOAD_END_MS. A changelog with a replica's last load change removed then goes red.

---

suggestion (non-blocking): State the workflow couplings this PR keeps where they happen.

.github/workflows/build.yml:1105, :637, :549, :632

No test runs the workflow graph. Whatever the answers to the three questions above keep (the deploy skip, the cross-flavor skip, the 1-day re-run limit), a comment on the job naming the consequence is the only pin.

---

note (non-blocking): Not checked

• docker-test (default|alpine) and docker-convergence (rs534|combined) have not run at bbe9484. The head commit changed the modrate values in load.sh and the sampler stop in monitor.sh after run 38865878001, and this review has no Docker daemon. It closes when those four checks finish on this head.

[vharseko]

Round 1: e4a8431, on master with #1205. The description is updated.

question (blocking): should a red docker-convergence row skip the whole of deploy.yml? Yes, that is intended. A comment on the job and the CI section of the description say so now, with #1206 and #1195 named as what keeps rs534 and combined red. Run 38893291531 at bbe9484 is such a run: rs534 failed again, on #1206.

issue: the online checks read a failed search or log read as a pass. Fixed as you wrote it. The output of each indexed search, its error output included, goes to dns/<i>.<node>.ldif, so a failed search says why there; errors_since_mark is gone. With ldaps_search and since_mark stubbed, under set -eE -o pipefail and the trap of run.sh, called as assert_online || …: searches that fail alike on every replica, a conflict search that fails on one, and an errors log that cannot be read were green before and are red now; a conflict entry and different answers are red in both.

issue: check_client passes a client that stopped before the end of the load. Fixed with your row count and exit check, and one more: any line besides the CSV output is an error. A connection that breaks in the last interval leaves 89 rows, as many as want, but handleConnectionError prints its message to the error output, which docker logs … 2>&1 writes to the same file. The 16 client outputs of the last two runs hold nothing else, so this raises no false alarm. Green before and red now: the output of run 38893291531 cut at 39 lines, the whole output with a connection error appended, and the whole output with exit code 1. --statInterval comes from STAT_INTERVAL in one place.

issue: the switch count leaves out the load-balancing switches (msgID 213). Right, and 213 is not the only other road: a replication server that closes the connection (63) and a connection that breaks reconnect without 188 as well. The column counts every new connection now - msgID 62 (96 where the generation IDs differ), which the broker logs at the end of every connect (ReplicationBroker:838) - and is called reconnections. On the errors logs of run 38893291531 it gives cv-d2 8: each of its eight 188 switches connected it again to RS(15571), the server it said it was leaving - #1198, fixed by #1205.

issue: sampler_stop's rm can fail a green run. || true, with the reason in the comment. With rm stubbed to fail, the shell exited before and goes on now.

suggestion: count the switches of a run that fails before the online checks too. Done in the trap, as you wrote it. OUT must be new or empty now (run.sh exits with 2 otherwise): the guard on reconnections.txt would take the file of an earlier run in the same directory, and samples.txt is appended to. On the state of run 38893291531 a failure before the online checks writes the column (8 for cv-d2) where it wrote -.

question (non-blocking): is it intended that a failed build of one flavor skips the docker tests of both? No; your if: is in, with a comment.

question (non-blocking): is the 1-day retention of the image artifacts a deliberate storage trade-off? It is 3 days now, with a comment that a later re-run needs "Re-run all jobs".

suggestion: make the three converged samples span one monitoring period. Done: MONITORING_PERIOD=60, and CONVERGED_IN counts to the first sample of the series, so that it does not read 60 s or more on every run. With converged_once and sleep stubbed, converged on every sample ends after 9 samples at 63 s (before: 3 samples, 21 s), and a miss on the fifth sample at 30 s starts the series again (before, the wait ended at the third).

suggestion: wait for the joins to end, not only for "healthy", before marking the logs. Done: bootstrap_ended of docker-test-replication.sh is in lib.sh, and topology_up waits for it after the health checks, within the same deadline. Not run here: no Docker on the host.

suggestion: the changelog check passes when no change of a directory server matches the parser. Your loop is in, and record_server_ids fails now when a directory server has no id, so the loop cannot be empty. Your mutant (UPDATE_TYPES = {}) on the changelog of cv-r0 of run 38893291531: green before, red now for all four replicas.

suggestion: no CI row shows that the checks can fail. Not in this PR. For now the real rows show it: on run 38893291531 rs534 went red through the indexed searches and through ldifdiff - the cmp and the ldifdiff of your mutants - and combined went red through the data on #1195 before. A self-test row pays once the fixes of #1206 and #1195 turn the rows green, so it belongs with those fixes; until then it is one more runner on every Build. The mutant case 1) → 0) of assert_changelogs stays unpinned meanwhile; the loop above pins that the check reads anything at all.

suggestion: the changelog check cannot see a lost first or last change of a replica. Right; documented - in the docstring of changelog-gaps.py and in the description - not changed:

• The first end has nothing to anchor to. Over its whole history the changelog of cv-r0 holds the same 345069 changes of cv-d2 as the window, from sequence number 2. Number 1 is not a lost change: a directory server numbers its ReplicaOfflineMsg like a change (PendingChanges.putReplicaOfflineMsg), and the changelog keeps only the newest of them, in offline.state rather than a log file (ReplicationEnvironment.notifyReplicaOffline) - cv-d2's holds 345071, its stop after the checks. It also means that a run which restarts a directory server in the window would read each restart as a lost change, which Replication convergence run: disturb the topology during the load, one disturbance per variant #1208 has to take into account.
• The last end: a lost change of the tail escapes the data only when another server overwrites it within seconds; a lost add or delete, and a modify nobody overwrites, are in the ldifdiff. A marker change after the load would close it; it is left out as a narrow window that nothing seen points to.

suggestion: state the workflow couplings this PR keeps where they happen. Done: comments on docker-convergence, at the if: of docker-test and at both retention-days; the CI section of the description lists the three.

note: not checked. Run 38893291531 at bbe9484 has finished: docker-test (default) and docker-test (alpine) green, combined green (converged 212 s after the load), rs534 red on #1206 - cv-d2 kept 12 entries the others had deleted, 7 of cv-d0 and 5 of cv-d3, with both changelogs complete. This head has not run in CI yet.

build.yml: actionlint 1.7.7, no findings.