Skip to content

Run race tests and bounded protocol fuzzing in CI - #14

Open
nirmalhk7 wants to merge 5 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/protocol-reliability-ci
Open

nirmalhk7 wants to merge 5 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/protocol-reliability-ci

Conversation

@nirmalhk7

@nirmalhk7 nirmalhk7 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Existing CI runs protocol tests but does not exercise the race detector or active fuzz campaigns. This adds one race job and five bounded fuzz jobs, with documented reproduction commands and failure evidence.

Each fuzz target runs for 10 seconds on pull requests, pushes to main and manual runs, or 60 seconds on the daily schedule. Fuzz steps have a 10-minute timeout inside a 15-minute job, and failures attempt to upload logs and corpus inputs for 14 days. Only pull-request runs cancel in-progress jobs; both reliability checkouts disable credential persistence. The workflow uses pull_request, read-only contents permissions and no secrets.

RELIABILITY.md documents transport coverage, timestamp regressions, the limited assertions made by operator/client fuzzing and the limits of artifact retention. CONFORMANCE.md links those checks before the final product-specific section. Historical bug-report notes are absent from the maintained guide.

#12 and #13 have merged. This branch incorporates current main at cfe78d1399c1f395c77eda0be991b3206dfdb0cf through an append-only merge, preserving published history. The diff against main is now exactly three files: the new reliability workflow, RELIABILITY.md and the conformance link. All protocol source, existing tests/helpers, SPEC, dependency declaration and golden vectors match main.

Verification on 78124faace9b38eeb3c6345c8d9f3d8e27fc6301:

  • Full go test -race ./... -count=1, build and vet pass.
  • All five 10-second fuzz campaigns pass with two workers each.
  • All 55 independent Python conformance checks pass.
  • actionlint passes all workflows; Go formatting, dependency-leaf and diff-whitespace checks pass.

Verification used an isolated copy of the exact committed tree with Go 1.27.1 and loopback access for real HTTP tests. Hosted CI remains a separate approval gate; prior-head results do not establish a pass on this revision. The daily schedule starts after the workflow reaches the default branch. No protocol release, consumer bump, wire change or production-readiness claim is included. The integration commit carries a DCO sign-off.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Nirmal. This is careful CI work, and the docs are honest about what fuzzing does and doesn't prove.

I reviewed only the top commit, 1b084aa, against its parent f3e09c5 (the #13 head). It changes .github/workflows/reliability.yml, RELIABILITY.md and the new CONFORMANCE.md section.

What was not run: I did not build or run any code from this stack locally. The three fork runs for 1b084aa (CI, dependency-leaf and Protocol reliability) are still waiting for a maintainer to approve them, so this head has no build, test, race or fuzz results yet. The only reported check is DCO, which passes. Everything below comes from reading the code.

What looks right

  • Safe for fork PRs: it uses pull_request, not pull_request_target, sets permissions: contents: read, uses no secrets, and passes matrix values through env rather than inline in run:.
  • shell: bash on the fuzz step makes GitHub add -o pipefail, so go test ... | tee still fails the step when the fuzzer finds something.
  • Fuzzing is time-bounded: 10s on PRs, pushes and manual runs, 60s on the schedule, with a 15-minute job limit. Each matrix entry matches exactly one fuzz function, and go test -race ./... covers every package.
  • Putting this in its own workflow leaves ci.yml unchanged, so the lean-code invariant test still finds go test ./... and the Python reproducer.

Before approval

  1. The fork runs need to be approved and to pass: CI, dependency-leaf and all 6 reliability jobs. The race job has never run in this repo. A new push, for example with the doc fixes below, would also give you runs against current main, which now includes #15.
  2. Merge order is #12, then #13, then #14. This PR carries #13's Verify changes; #13 is now approved at f3e09c5. If you rebase, please keep #12's commit: #13's decode_fuzz_test.go uses testCoordinator, roundTripFunc, trackedBody and clientCalls from #12's transport_test.go.

Should fix (fine in the same push)

  • RELIABILITY.md:66-84, "Bug-Build-Test report draft": this section is notes about this PR, such as "On the parent implementation…" and "No Slack receipt was submitted…", and it will go stale once merged. Please remove it from the file and file it through the Bug-Build-Test workflow, so the find gets credited, or move it into the PR description. Lines 1-64 are good.

Nits

  • CONFORMANCE.md:60: "Reliability checks" comes after "Product-specific notes (last, per the ordering rule)" at :56. Please move it up, for example after "Dependency declaration" (:52-54).
  • reliability.yml:14-16: push, schedule and manual runs on main share one concurrency group with cancel-in-progress: true. A merge can cancel the daily 60s run, and the daily run can cancel a push run. Suggest cancel-in-progress: ${{ github.event_name == 'pull_request' }}.
  • reliability.yml:33 and :57-63: if a fuzz job ever hits the 15-minute job limit, GitHub cancels it and the if: failure() upload is skipped, so RELIABILITY.md:59 wouldn't hold in that case. Add timeout-minutes: 10 to the "Bounded fuzz campaign" step (a step timeout counts as a failure), or use if: failure() || cancelled().
  • RELIABILITY.md:51-52 says the existing checks "remain mandatory", but only DCO is a required check on main. Suggest "continue to run in ci.yml and dependency-leaf.yml".
  • RELIABILITY.md:48-49: beyond the seeds, FuzzOperatorVerification can in practice only check panic safety, because random inputs won't produce a valid ed25519 signature. FuzzClientJSON's success path only checks that the body was closed. One sentence saying so would keep the doc's claims exact.
  • Optional: add persist-credentials: false to both checkouts. SHA-pinning the actions would be better as a separate PR that covers all three workflows.

There are no wire, record or identity changes here. Once the runs are green and #12 and #13 are in, this looks ready apart from the items above.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>

Copy link
Copy Markdown
Contributor Author

Resolved the integration conflict in 78124faace9b38eeb3c6345c8d9f3d8e27fc6301 by merging current main, preserving append-only history. #12 and #13 are merged. Upstream's expanded transport tests and context-aware helpers are retained unchanged, including the helpers used by the fuzz targets.

The PR diff is now only .github/workflows/reliability.yml, RELIABILITY.md and the reliability section in CONFORMANCE.md. All review fixes are retained: historical report notes removed from the guide, product-specific notes last, cancellation restricted to PRs, a 10-minute fuzz step timeout, accurate existing-CI and fuzz-coverage wording, and credential persistence disabled on both reliability checkouts. SHA-pinning all workflows remains the separate follow-up suggested in the review.

Verification on the new head:

  • go test -race ./... -count=1: all packages pass.
  • go vet ./... and go build ./...: pass.
  • Each of the following passes with -run '^$' -fuzztime 10s -parallel 2: FuzzMessageVerification (root), FuzzVerifySig (canon), FuzzOperatorVerification (operator), FuzzHandlerJSON and FuzzClientJSON (transport/httpjson).
  • python3 testdata/reproduce_spec.py: 55 checks, all match.
  • actionlint .github/workflows/*.yml: passes.
  • test -z "$(gofmt -l .)", dependency-leaf check and git diff --check: pass.

Tests ran from an isolated copy of committed tree 953c29dbe78fd9979d1aa0be26335ec005ca9065, using Go 1.27.1 and GOCACHE=/private/tmp/protocol-pr14-go-cache. Real HTTP tests ran with loopback access. SPEC, golden vectors, protocol source and existing tests match main. Local success does not replace hosted checks: CI, dependency-leaf and all six reliability jobs still need successful results on this head before approval. Scheduled execution begins only on the default branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants