Skip to content
This repository was archived by the owner on Oct 23, 2026. It is now read-only.
openIE-devPublic archive

About

Implementation-independent conformance suite for RCE

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Moved (23 October 2026). This repository now lives in openIE-dev/rce as conformance/, with its full history, so that it is checked from the same commit as the specification and the reference checker and can no longer drift from them. Changes go there; this repository is kept only as a pointer.

rce-conformance

An implementation-independent conformance suite for RCE. Every test is a plain .rce file with a metadata block. The harness talks to any implementation through a line-oriented JSON adapter protocol and reports, per normative clause of the RCE book, which tests exercise it and whether they pass.

The guiding rule: tests encode what the laws of computing require (IEEE 754, two's complement, injectivity of the log digest, replay equals reopen), not whatever the spec text or the reference happens to say. Where the two differ, the test follows the law and the difference is filed in FINDINGS.md with a derived fix. No test encodes a known bug.

Layout

  • clauses.json: every clause extracted from spec/book/*.md (rule ids such as W-PAR-1 and HC-15, theorem rows, MUST/SHALL sentences as CHnn-M-<hash>), each with a stable id and a profile (language, handler-spec, tooling, proposed, editorial).
  • tests/<area>/: the tests, one area per directory.
  • harness/: rcetest.py (file format) and run.py (runner).
  • adapters/: the reference interpreter, the Rust, Python, JS and Scala emitters, and the wasm playground.
  • tools/: clause extraction, harvesting, generators, tagging, coverage, emission parity, tree-sitter cross-check, mutation testing.
  • findings/ and FINDINGS.md: every place an implementation or the spec disagrees with the law.
  • reports/: coverage, per-adapter results, parity, mutation score.

See INTERPRETING.md for the file format and ADAPTERS.md for the protocol.

Running

Current limit. The adapter binary rce-conform is built from public rce (cd spec/checker && cargo build --release --bin rce-conform, source src/bin/conform.rs, present since openIE-dev/rce cdae5cf). Tests that use $RCE_ENGINE (the handler-spec profile, appendix D) need the engine's rce, which is not public, so those cannot run against public rce. The lexing-parsing area was run against public rce main on 17 October 2026: 133 of 133 pass. The reference and the three emitter reports were rerun against public rce 4a3dc1f (next section); the other reports were produced against the engine repository, whose history is not public. The clause paths below use the public layout (spec/book).

export RCE_CONFORM=/path/to/rce-conform   # cargo build -p rce-spec-check --bin rce-conform
python3 harness/run.py --adapter adapters/reference.json --jobs 8
python3 tools/coverage.py
python3 tools/parity.py

Status

Generated from reports/ by tools/readme_status.py (--check verifies it). Every figure below is counted from the report files.

  • Tests: 1979 test files under tests/ (1983 .rce files, of which 4 named _FIXTURE are support files, not tests); the reference run has 1979 results; the coverage report counts 1979. The three counts agree.
  • Coverage: language profile 205 of 205 normative clauses covered (100.0%). Other profiles: tooling 5 of 7, retired 1 of 2, handler-spec 8 of 14, editorial 0 of 1, proposed 3 of 6.
  • Reference: 1979 pass, 0 fail of 1979.
  • Emitters: python 1459 pass / 0 fail / 520 not applicable or unsupported; js 1459 pass / 0 fail / 520 not applicable or unsupported; rust 1459 pass / 0 fail / 520 not applicable or unsupported. Parity with the reference: python 1460 agree, 0 disagree; js 1460 agree, 0 disagree; rust 1460 agree, 0 disagree.
  • Scala emitter: 52 pass, 1189 fail, 591 unsupported of 1832 results (146 tests have no Scala result). Scala does not yet agree with the reference.
  • Wasm playground: 419 pass, 16 fail, 1504 unsupported of 1939 results.
  • Lean core model: 210 pass, 1 fail, 1766 unsupported of 1977 results.
  • Tree-sitter grammar 00b639a: 1773 of 1773 programs agree with the reference parser (0 disagreements).
  • Mutation: 200 of 712 schema sites sampled (seed 1); 185 killed (92.5%) on the first run. After survivor-targeted tests: 94.5%; after round 3: 99.0%; round 4 reports 99.5% counting mutant 133 as equivalent. Survivors are in reports/mutation.json. The mutation baseline (the unmutated mutation build) fails 20 tests; that is not the reference run, which has 0 failures.

Run against public rce 4a3dc1f (one language, 18 October 2026)

The reference, emit-python, emit-js and emit-rust results in reports/ were rerun against public rce main 4a3dc1f, with rce-conform and rce built from that commit ($RCE_ENGINE set to the same rce). Before the tests were converted to the one-language rules the reference run had 1,575 pass and 404 fail; the converted suite has 1,965 pass and 14 fail.

  • The 14 not re-run: the 14 handler-spec tests (appendix D) need the engine's rce, which is not public; their committed results (pass, engine run) are kept, and they fail here because public rce has no spec declarations.
  • Not re-run, results kept as committed (minus the removed emit06-reject): emit-scala (needs a Scala toolchain run that was not made), wasm-playground (needs a playground build of the new checker) and lean-core (needs the Lean build). Their results predate the conversion of 371 programs and the rename of emit06-reject to emit06-accept, so for those tests they describe the earlier program text. reports/tree-sitter.json was not regenerated either; it still lists the earlier program text.
  • Conversions: programs were changed only as the one-language rules require (_name for unused names, let _ = for dropped results, loop bound N, len_bytes, renamed reserved words ct and label, externs taking &World). Expectations that encode the retired &mut write-back (T33) were re-derived under F.4.4 and checked by hand (write-back-methods, with its two generated fuel tests). s4-std-modules now expects W-UNUSED-1, because an unused use is refused. emit06-reject became emit06-accept because par is part of the language. The 12 version mutants were removed, and clauses W-VER-1 and the 2.7 version sentence are in the retired profile.

Partial runs against public rce (17 October 2026)

The committed reports/ were produced against the engine repository, not public rce; the reference adapter (rce-conform) is not in public rce, so no reference report can be regenerated from public sources. Two partial runs were made against public rce main 4304048 with a check-only adapter and the emit-python adapter (the counts are from those runs, not from reports/):

  • Check tests: 436 pass, 12 fail of 448 (1,543 further tests need an operation the check-only adapter does not offer). Causes of the 12: six W-SES-4 tests (session use after move, send, close or in a loop) are reported under W-SES-2 or W-OWN-1, or accepted, so public rce does not enforce W-SES-4 as written; three W-TY-3 tests (int and float mixing, narrowing, equality) are reported as W-TY-1; w-mod-2-cycle (module cycle) is accepted; w-state-3-borrow-field is reported as W-TY-2 and W-TY-1; w-trait-1-impl-mismatch is accepted.
  • emit-python: 1,304 pass, 156 fail, 531 not applicable. Causes: (1) 103 tests (102 in forms-log and f6-signed-zero-state) expect the emitted program to take --trace and print trace HEAD; emitted Python from public rce has no --trace line, so the adapter sees the genesis head. (2) 27 concurrency/generated tests are float reductions in for ... par whose oracle is the fixed balanced tree of spec C.8; public rce evaluates them as the left fold (tree-prod-006 gives 1.857142857142857 where the tree gives 1.8571428571428568). Float multiplication and addition are not associative, so the two orders differ in the last digits; the specification sentence that the tree equals the left fold holds only for associative operations. (3) 13 tests expect -0 where public rce prints 0 (ceil of a tiny negative, and -0 state). (4) 11 overflow tests agree on the trap but print a different out-of-range value in the message. The other two are n9-size-limit (public rce finishes where the test expects a size-limit trap) and r5-generic-call (the emitter refuses a generic call with unknown type T).

None of these is a suite error; each is a place where public rce and the engine behind the committed reports differ. They are not fixed in public rce here because the -0 text and several of these paths are being changed in a pending export.

Convergence notes

The suite converges on ideas already established elsewhere. These notes cite only material read while building it.

  • Test262 (INTERPRETING.md in tc39/test262): metadata lives in the test file, negative tests state the phase and the error type they expect, flags change how a file is run, and files named with _FIXTURE are support files rather than tests. Here expect: reject plus a diagnostic code plays the role of the negative phase and type, and fixtures: names support modules.
  • The WebAssembly spec testsuite (README of WebAssembly/testsuite and test/core/README): assertions about traps, invalid modules and exhaustion are first-class alongside assertions about results. Here a run states outcome: trap:<kind>, refused:<error>, done or deadlock, and fuel exhaustion is a trap kind.
  • ACATS (the ACATS overview page and the User's Guide, section 3.1): tests are organised by test objective tied to rules of the reference manual. Here every test lists the clause ids it exercises and the coverage report lists every clause with no test.

License

Licensed under either of Apache License, Version 2.0 (LICENSE-APACHE) or the MIT license (LICENSE-MIT), at your option. Copyright (c) 2026 Open Interface Engineering, Inc. See COPYRIGHT.

About

Implementation-independent conformance suite for RCE

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages