Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ cargo fmt --all # max_width = 120
cargo machete --with-metadata # unused dependencies, dev ones included
```

An x86-only `cfg` arm never compiles on an Apple machine, so type-check the other target before pushing anything `cfg`-gated (needs only `rustup target add x86_64-unknown-linux-gnu`; `bench` and the CLI are left out because jemalloc's build script wants a cross `gcc`):
An x86-only `cfg` arm never compiles on an Apple machine, so type-check the other target before pushing anything `cfg`-gated (needs only `rustup target add x86_64-unknown-linux-gnu`; `bench` and the binaries are left out because jemalloc's build script wants a cross `gcc`):

```bash
RUSTFLAGS="-C target-cpu=znver4" \
Expand Down
19 changes: 19 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 18 additions & 0 deletions bins/leanvm-sidecar/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[package]
name = "leanvm-sidecar"
version.workspace = true
edition.workspace = true
publish = false

[lints]
workspace = true

[dependencies]
lean_vm = { path = "../../crates/lean_vm" }
bench.workspace = true
libc.workspace = true
tracing.workspace = true
clap = { version = "4", features = ["derive"] }

[dev-dependencies]
leanvm_sidecar_client = { path = "../../crates/leanvm_sidecar_client" }
131 changes: 131 additions & 0 deletions bins/leanvm-sidecar/PROTOCOL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# leanvm-sidecar protocol, version 1

`leanvm-sidecar` serves leanXMSS aggregation, recursion and verification to one client process over one shared file. The server creates the file; the client opens it read-write and maps all of it `MAP_SHARED`. Every integer is little-endian. Every offset is in bytes from the start of the file.

Clients that implement this document: `crates/leanvm_sidecar_client` (Rust), `clients/go`, `clients/java`, `clients/typescript` and `clients/nim`. Each exposes `send`, which places a payload in the arena and queues its request, and `try_recv`, which returns the next response if there is one.

The file holds a header, a request ring, a response ring and an arena. Payloads live in the arena; the rings carry 64-byte messages that point into it.

## Header

Twelve `u64` words at offset 0. The client reads them once after mapping; only `epoch` changes afterwards.

| offset | field | meaning |
| --- | --- | --- |
| 0 | `magic` | the ASCII bytes `LEANVMSH` |
| 8 | `version` | 1 |
| 16 | `epoch` | incremented each time the server starts |
| 24 | `ring_cap` | slots in each ring, a power of two |
| 32 | `total` | file size |
| 40 | `req_ring` | offset of the request ring |
| 48 | `resp_ring` | offset of the response ring |
| 56 | `arena_off` | offset of the arena |
| 64 | `arena_len` | arena size, `total - arena_off` |
| 72 | `pub_key_size` | bytes in a leanXMSS public key |
| 80 | `message_len` | bytes in a message |
| 88 | `sig_size` | bytes in a leanXMSS signature |

A client checks `magic` and `version` and takes every other number from the header.

A request in flight when the server stops is lost. A client that reads a different `epoch` than the one it saw when it sent a request stops waiting for that request and sends it again. Requests still queued in the request ring survive a restart and are served.

## Rings

Each ring is single-producer, single-consumer: the client produces requests and consumes responses, the server the reverse. A client with several threads serializes its own access.

| offset in ring | field |
| --- | --- |
| 0 | `head`, `u64`: messages consumed, written only by the consumer |
| 64 | `tail`, `u64`: messages produced, written only by the producer |
| 128 | `ring_cap` slots of 64 bytes |

Both counters only grow. Message `i` is in slot `i mod ring_cap`.

- Produce: read `tail` and `head`. If `tail - head == ring_cap` the ring is full: wait and retry. Otherwise write the message to slot `tail mod ring_cap`, then store `tail + 1`.
- Consume: read `head` and `tail`. If they are equal the ring is empty. Otherwise read slot `head mod ring_cap`, then store `head + 1`.

`head` and `tail` are read and written with atomic 64-bit operations, and the store that follows a slot access must be a releasing (or sequentially consistent) store: `sync/atomic` in Go, a `VarHandle` `getVolatile`/`setVolatile` on the mapped buffer in Java. There is no wake-up primitive. Both sides poll; the server polls an empty request ring every 100 microseconds.

## Arena

The server never allocates in the arena. The client places each request's payload wherever it likes and names, in the request, a second region for the response. The client alone decides when a region is free again: a request's two regions are the server's from the moment the request is produced until its response is consumed.

The server copies the payload out before decoding it, so a client may not rely on the server seeing a later write.

## Messages

Request, 64 bytes:

| offset | field | meaning |
| --- | --- | --- |
| 0 | `id` `u64` | echoed in the response |
| 8 | `op` `u32` | below |
| 12 | `flags` `u32` | below |
| 16 | `req_off` `u64` | payload region |
| 24 | `req_len` `u64` | |
| 32 | `resp_off` `u64` | response region |
| 40 | `resp_cap` `u64` | its capacity |
| 48 | `log_inv_rate` `u32` | PCS rate of the proof to make, 1 through 4; higher is a smaller proof and slower proving. Ignored by `VERIFY` |
| 52 | zero | |

Response, 64 bytes:

| offset | field | meaning |
| --- | --- | --- |
| 0 | `id` `u64` | the request's |
| 8 | `status` `u32` | below |
| 12 | `detail` `u32` | depends on `status` |
| 16 | `resp_len` `u64` | bytes written at `resp_off`, except under `RESP_TOO_SMALL` |
| 24 | zero | |

Both regions must lie inside the arena. Responses come in request order.

## Payloads

Building blocks:

- `claims` := `n u32 | n x (pubkey | message)`. An aggregate's claims are strictly sorted on the bytes of `pubkey | message`, with no duplicate; a proof is rejected against claims that are not.
- `proof` := `len u64 | len bytes`. The bytes are opaque to the client. A proof carries no claims: the two always travel together.
- `signatures` := `n u32 | n x (pubkey | message | signature)`, in any order. A repeated `(pubkey, message)` counts once.

| op | name | request payload | response body |
| --- | --- | --- | --- |
| 1 | `AGGREGATE` | `signatures` | `proof \| claims` |
| 2 | `RECURSE` | `n u32 \| n x (claims \| proof)` | `proof \| claims` |
| 3 | `VERIFY` | `claims \| proof` | empty |
| 4 | `VERIFY_NARROW` | `claims \| proof \| claims` | `proof \| claims` |
| 5 | `EXTEND` | `n u32 \| n x (claims \| proof) \| signatures` | `proof \| claims` |

- `AGGREGATE` proves that every signature is valid. The signatures are assumed valid: an invalid one fails the whole request with `AGGREGATION_FAILED`, without saying which.
- `RECURSE` proves the union of its children's claims. With flag `VERIFY_CHILDREN` (bit 0) the server verifies each child first and reports the first bad one as `INVALID_PROOF` with its index. Without it the client asserts the children are verified; a bad child then fails the request late as `AGGREGATION_FAILED`, usually without its index.
- `VERIFY` checks a proof against the claims given. `OK` means exactly these public keys signed exactly these messages.
- `VERIFY_NARROW` verifies the proof against the first claims, then makes a proof of the second claims, which must be a subset of the first.
- `EXTEND` is `RECURSE` and `AGGREGATE` in one proof: it proves the union of its children's claims and the signatures' claims. `VERIFY_CHILDREN` and the rules for signatures apply as in those two.

`RECURSE` and `EXTEND` take 1 to 16 children, the kept proof included.

| flag | bit | ops | meaning |
| --- | --- | --- | --- |
| `VERIFY_CHILDREN` | 0 | `RECURSE`, `EXTEND` | verify each child in the payload before proving |
| `KEEP_PROOF` | 1 | every op but `VERIFY` | the server keeps the proof it returns, in place of any it held |
| `USE_KEPT` | 2 | `RECURSE`, `EXTEND` | the kept proof is one more child, after those in the payload, which may then be none |

The server holds at most one kept proof, in memory. It lets a client grow a proof over several `EXTEND` requests without sending it back each time: `AGGREGATE` with `KEEP_PROOF`, then `EXTEND` with `USE_KEPT | KEEP_PROOF` and a payload of `0 u32 | signatures`, and the last `EXTEND` with `USE_KEPT` alone. A request with `USE_KEPT` that succeeds ends the keeping of that proof, so without `KEEP_PROOF` nothing is kept afterwards. A request that fails changes nothing, and `USE_KEPT` with nothing kept is `BAD_REQUEST`. A proof that did not fit its response region (`RESP_TOO_SMALL`) is kept all the same. The kept proof is never verified again, and a server restart loses it.

The response body repeats the claims because the server sorts and deduplicates them.

## Status

| status | name | `detail` | body |
| --- | --- | --- | --- |
| 0 | `OK` | 0 | the op's response |
| 1 | `BAD_REQUEST` | 0 | text |
| 2 | `RESP_TOO_SMALL` | 0 | nothing; `resp_len` is the capacity needed |
| 3 | `INVALID_PROOF` | index of the proof in the request | text |
| 4 | `AGGREGATION_FAILED` | reason, below | text |

`BAD_REQUEST` is a region outside the arena, an unknown op, or a payload that does not decode or has trailing bytes. The text of a failure is UTF-8 for a human, cut to `resp_cap`, possibly inside a character.

`RESP_TOO_SMALL` is found only after the proof is made, and the proof is discarded: size the response region generously.

`AGGREGATION_FAILED` reasons: 1 invalid child, 4 invalid signature, 11 proving failed (an invalid signature or child that was not caught earlier, or too many signatures for one proof), 12 invalid input (nothing to prove, a retained claim that is not among the proof's, too many children, `log_inv_rate` out of range); the text says which. 2, 3 and 5 through 10 are no longer sent: 12 replaces them.
97 changes: 97 additions & 0 deletions bins/leanvm-sidecar/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# leanvm-sidecar

A long-lived leanVM prover and verifier for one client process on the same machine. The client and the sidecar share one file on tmpfs: payloads sit in an arena inside it, and two rings of 64-byte messages carry requests and responses. The client can be written in any language that can map a file and do an atomic 64-bit load and store.

It serves five operations, on leanXMSS signatures only:

| op | what it does |
| --- | --- |
| `AGGREGATE` | prove a batch of `(pubkey, message, signature)` |
| `RECURSE` | prove the union of up to 16 proofs' claims |
| `VERIFY` | check a proof against its claims |
| `VERIFY_NARROW` | check a proof, then prove a subset of its claims |
| `EXTEND` | prove the union of up to 16 proofs' claims and a batch of signatures |

[`PROTOCOL.md`](PROTOCOL.md) is the byte-level specification: the file layout, the messages, the payload encodings and the status codes.

## Build and run

Release only: the guest compiler overflows the debug stack.

```bash
cargo build --release -p leanvm-sidecar
target/release/leanvm-sidecar --path /dev/shm/leanvm --threads 16
```

The sidecar builds the guest, runs its warm-up, creates the channel file and prints `serving <path>`. The file appears only once it is ready, so a client may open it as soon as it exists. Stop the sidecar with a signal; it holds no state to flush.

| flag | default | meaning |
| --- | --- | --- |
| `--path` | required | the channel file; put it on tmpfs (`/dev/shm`) |
| `--threads` | one per physical core | prover threads; verification always runs on one |
| `--arena-mib` | 64 | arena size; the payload and the response region of every request in flight have to fit |
| `--ring-cap` | 64 | slots in each ring, a power of two |
| `--warmup-signatures` | 1500 | signatures in the warm-up proofs; 0 skips the warm-up |
| `--warmup-rates` | `1,2` | the `log_inv_rate` values to warm up |
| `--tracing` | off | print a timing tree per request (`RUST_LOG` sets the level) |

The warm-up is the discarded pass the benchmarks run before they measure: for each rate, one aggregation, one verification and one 16-wide recursion over signatures the sidecar makes for itself. Without it the first request of each shape is slower than the rest. Set `--warmup-signatures` to the largest batch expected and `--warmup-rates` to the rates requests will ask for.

The whole file is faulted in at start, so the arena's memory is committed from then on.

Requests are served one at a time, in order. A request in flight when the sidecar stops is lost; requests still queued are served after a restart on the same file. The header's `epoch` changes on every start, which is how a client notices.

## Clients

Each client exposes the same two calls, over raw payload bytes in the encodings of `PROTOCOL.md`:

- `send(op, flags, log_inv_rate, payload, resp_cap)` copies the payload into the arena, reserves `resp_cap` bytes for the response and queues the request. It returns the request's id, or reports that the ring or the arena is full; it never blocks.
- `try_recv()` (`tryRecv` in Java, TypeScript and Nim) returns the next response with its body copied out, or nothing if the sidecar has not produced one.

| language | where | notes |
| --- | --- | --- |
| Rust | [`crates/leanvm_sidecar_client`](../../crates/leanvm_sidecar_client) | depends on `libc` alone; links nothing of the prover |
| Go | [`clients/go`](clients/go) | package `leanvmsidecar`, standard library only |
| Java | [`clients/java`](clients/java) | `leanvmsidecar.Client`, Java 17 or later, no dependencies; channel files under 2 GiB |
| TypeScript | [`clients/typescript`](clients/typescript) | Node 18 or later; `client.ts` plus a small C addon (`mmap.c`) that maps the file, since Node cannot; `./build.sh` builds both into `dist/` |
| Nim | [`clients/nim`](clients/nim) | module `leanvmsidecar`, standard library only; needs the GCC or Clang backend for the atomics; the entry point is `openClient` |

A client is for one thread at a time. Open one only while no request of an earlier client is outstanding: the sidecar would write that request's response into arena space the new client hands out again.

The clients build no payloads and parse no bodies. The end-to-end runners below are the worked example in each language: `crates/leanvm_sidecar_client/examples/e2e.rs`, `clients/go/e2e/main.go`, `clients/java/leanvmsidecar/E2e.java`, `clients/typescript/e2e.ts` and `clients/nim/e2e.nim`.

## End-to-end test

```bash
bins/leanvm-sidecar/e2e.sh <rust|go|java|ts|nim> <threads> [--signatures N] [--batch-size N] [--aggregate-rate R] [--recurse-rate R]
```

| option | default | meaning |
| --- | --- | --- |
| `--signatures` | 1440 | signatures per batch, 1 to 1500 |
| `--batch-size` | `--signatures` | signatures per sub-batch |
| `--aggregate-rate` | 2 | `log_inv_rate` of each batch's proof |
| `--recurse-rate` | 1 | `log_inv_rate` of the recursion |

The script builds the sidecar and the chosen language's runner, starts the sidecar with `<threads>` prover threads, and has the runner make sixteen `AGGREGATE` requests of `--signatures` signatures each, one after the other, then one `RECURSE` over the sixteen proofs and one `VERIFY` of the result. With `--batch-size`, each batch is instead built up in sub-batches of that size, the last one short: the first is an `AGGREGATE`, and each later one an `EXTEND` of the proof so far with the next signatures. The sidecar keeps that proof between the requests (flags `KEEP_PROOF` and `USE_KEPT`), so the runner sends only signatures. These run at `log_inv_rate` 1, except the last, which makes the batch's proof at `--aggregate-rate`. Each step prints its request, its response and the time from `send` to the response, as the client sees it. The script stops the sidecar and removes the channel file when the runner exits.

```bash
bins/leanvm-sidecar/e2e.sh go 16 # Go client, 16 threads, defaults
bins/leanvm-sidecar/e2e.sh java 32 --signatures 1500 --aggregate-rate 1 # Java client, full batches, aggregate at rate 1
bins/leanvm-sidecar/e2e.sh nim 32 --signatures 1400 --batch-size 150 # Nim client, each batch as 9 x 150 + 1 x 50
```

The signatures are `testdata/signatures-24000.bin`: 24000 records of `pubkey | message | signature`, every record a different signer, signer `i` on message `i mod 4`. The signers are mock keys, real only at the leaf index they sign at; their signatures verify like any other. If the file is missing the script regenerates it with `examples/gen_signatures.rs`, which takes a minute or two.

The Go runner needs `go` on `PATH`. The Java runner uses `javac` and `java` if they are on `PATH`, and otherwise the `eclipse-temurin:21-jdk` image through `docker`. The TypeScript runner needs `node`, `npm` and a C compiler: the script runs `clients/typescript/build.sh`, which installs the TypeScript compiler from npm on first use. The Nim runner uses `nim` if it is on `PATH`, and otherwise builds and runs in the `nimlang/nim:2.2.0` image through `docker`.

## Tests

```bash
cargo test --release -p leanvm-sidecar -p leanvm_sidecar_client
(cd bins/leanvm-sidecar/clients/go && go test ./...)
(cd bins/leanvm-sidecar/clients/typescript && ./build.sh && npm test)
nim c -r --outdir:target/leanvm-sidecar bins/leanvm-sidecar/clients/nim/leanvmsidecar.nim
```

The sidecar's test runs every op, and its failure cases, through the Rust client against the serving loop in one process.
Loading
Loading