A shared platform for evaluating AI-generated code, the datasets that feed and judge it, and the evals themselves, with isolation, audit trails, access control and cost governance.
Status: early development. Built: local engine and sandbox, scoring and compare, hard gate and CI workflow, multi-tenant API with OIDC, PostgreSQL storage with tenant isolation, audit log, object storage, durable queue, suite registry. Not built: the worker process, live model providers, judges, datasets, SSO login flow, Kubernetes runners.
Docs: What Evalry is | How to use it (local, CI gate, server, API)
| Module | Purpose |
|---|---|
evalry.schemas |
Version 1 public schemas: task YAML (Task, load_task) and run record JSON (RunRecord, SampleResult, GraderScore) |
evalry.runner.sandbox |
Docker sandbox for untrusted code: pinned image, no network, non-root, read-only root, capped CPU/memory/processes. Default rpc isolation runs the solution and the hidden tests in separate containers joined by a host relay |
evalry.graders.python_tests |
python_tests: hidden tests run in the sandbox |
evalry.generation.base |
Generator interface, offline FakeGenerator, extract_code, logged generation calls |
evalry.scheduler.engine |
grade and run_sample: the steps shared by the CLI and suite runs |
evalry.graders.cache |
Grading cache keyed by code, tests, grader version, image and seed (--cache-dir); timeouts are never cached |
evalry.scheduler.jobs |
Idempotent jobs keyed by their inputs, in-process priority queue, per-project concurrency and budget limits, retries, crash recovery |
evalry.scheduler.worker |
submit_run, work, collect: the loop that runs queued jobs through the engine |
evalry.audit.chain |
Append-only, hash-chained audit log (AuditLog.verify detects edits, removals and, with a stored head, truncation) |
evalry.gates.hard |
Hard gate: pass/block on deterministic graders for an approved suite version, exceptions with a named approver and expiry, signed decisions recorded in the audit log |
evalry.access.store |
Roles (viewer/author/approver/admin), tenant and project scoping in one data-access layer; cross-tenant access is structurally impossible and every denial, grant and write is audited |
evalry.access.service |
Request-level operations behind roles: submit_suite (author) and gate_run (approver; every exception's named approver must hold the approver role). The local CLI does not go through it |
evalry.api |
FastAPI app under /v1: OIDC bearer-token verification (RS256/ES256 only), project and member management, run import and read, gate and decision endpoints. State is injected; environment and the gate key are server config |
evalry.storage (backend, pgstore, db, objects) |
Storage behind the access layer: in-memory or PostgreSQL (tenant in every key, row-level security, immutable runs and decisions via triggers), plus plain-SQL migrations with checksums |
evalry.storage (objects, pgjobs) |
Content-addressed object storage (file or S3-compatible) for run code, and the durable Postgres job queue with leases. Migrations 0003 (jobs) and 0004 (immutable suite registry) |
evalry.scheduler.suite |
Run every task x N samples; per-task mean/min/max/std and pass@k |
evalry.scoring.stats |
Composite score with hard floors, unbiased pass_at_k, per-task stats |
evalry.scoring.compare |
Seeded paired bootstrap over tasks and the regression gate |
evalry.billing.cost |
Pre-run cost estimate and budget stop; rates are inputs |
evalry.log |
get_logger, the only way to log: LogQuill with redaction attached |
Python 3.12+, uv, and a running Docker daemon (untrusted code only ever runs in a container).
id: fizzbuzz
language: python
prompt: Write fizzbuzz(n).
tests: assert fizzbuzz(3) == 'Fizz' # hidden from generatorsUnknown fields are rejected. Optional fields: tier (1 to 4), reference, tags, sensitivity, owner. Only Python tasks
can be graded so far.
# Grade one solution file. Exit code: 0 perfect score, 1 lower score, 2 usage or file error.
uv run evalry run task.yaml solution.py [--out run.json] [--suite-version V] [--timeout S] [--isolation rpc|inprocess]
# Run a directory of task YAML files, N samples each, from recorded model replies.
# replies.json: {"task_id": ["reply for sample 0", "reply for sample 1", ...]}
uv run evalry suite tasks/ --replies replies.json --samples 5 -k 1 --out run.json
# --cache-dir DIR reuse grades across runs
# --workers N run N jobs at once (a project's concurrency limit still applies); ~2x faster at 4 workers on a laptop
# --rates R.json --budget USD print the estimate first; refuse to start (and stop mid-run) over budget
# --project P --actor A --audit-log audit.jsonl record run and job events in the audit log
# Compare two run records of the same suite version (paired bootstrap, 95% interval).
# Exit code: 0 no regression or too small to gate, 1 regression gate blocks, 2 bad input.
uv run evalry compare baseline.json candidate.json [--threshold 0.05]
# Apply database migrations (connection string from the environment, never a flag).
EVALRY_ADMIN_DSN='host=... user=... password=... dbname=evalry' uv run evalry migrate
# Hard gate: block a release when correctness falls below the floor. Exit code: 0 pass, 1 block, 2 bad input.
# Rule YAML: name, suite_version (the approved one, never "latest"), grader_version, [floor, min_pass_rate, required_tasks].
# Production decisions must be signed with the key in EVALRY_GATE_KEY; dev and staging record rehearsals.
# Only `rpc` isolation results can gate. Exceptions: YAML list of task_id, approver, reason, expires_at.
uv run evalry gate run.json --rule gate.yaml --audit-log audit.jsonl [--exceptions exc.yaml] [--environment production] [--out decision.json]
Regression gates need at least 100 tasks and 5 samples per task. Smaller runs print the delta and interval as advisory only. Run records store the task hash, grader versions, runner image digest and seed so a score can be reproduced.
cp deploy/.env.example deploy/.env # set both passwords; never commit deploy/.env
docker compose --env-file deploy/.env -f deploy/docker-compose.yml up -d --wait
The compose file also starts MinIO (object storage). It creates two least-privilege roles, neither a superuser nor BYPASSRLS: evalry_app (the API, bound to one tenant) and evalry_worker (claims queued jobs across tenants; it can read and update the jobs table and read suites, nothing else). Set EVALRY_PG_APP_PASSWORD, EVALRY_PG_WORKER_PASSWORD, EVALRY_S3_ACCESS_KEY and EVALRY_S3_SECRET_KEY in deploy/.env. An existing database needs the worker role created by hand. Run evalry migrate as the
admin, then connect the application as evalry_app; PostgresBackend refuses a superuser connection because PostgreSQL
superusers bypass row-level security.
# one-time: schema, then the first tenant (operator actions; they use the admin connection)
export EVALRY_ENV=staging EVALRY_DB_APP_DSN='host=... user=evalry_app password=... dbname=evalry'
export EVALRY_DB_ADMIN_DSN='host=... user=postgres password=... dbname=evalry'
EVALRY_ADMIN_DSN="$EVALRY_DB_ADMIN_DSN" uv run evalry migrate
export EVALRY_OIDC_ISSUER=https://login.example.com EVALRY_OIDC_AUDIENCE=evalry-api \
EVALRY_OIDC_JWKS_URL=https://login.example.com/.well-known/jwks.json
uv run evalry bootstrap-tenant acme alice # alice = the identity-provider subject of the first admin
uv run evalry-api # binds 127.0.0.1:8000 unless EVALRY_HOST / EVALRY_PORT say otherwise
| Variable | Meaning |
|---|---|
EVALRY_ENV |
dev, staging or production (required). Only production records official gate decisions |
EVALRY_OIDC_ISSUER, EVALRY_OIDC_AUDIENCE, EVALRY_OIDC_JWKS_URL |
Token verification (RS256/ES256 only); the JWKS URL must be https. EVALRY_TENANT_CLAIM names the tenant claim (default tenant_id) |
EVALRY_DB_APP_DSN |
PostgreSQL connection as the least-privilege evalry_app role. Required in staging and production |
EVALRY_DB_ADMIN_DSN |
Admin connection, used only by bootstrap-tenant |
EVALRY_GATE_KEY |
Signing key for gate decisions. Required in production |
EVALRY_S3_BUCKET, EVALRY_S3_ACCESS_KEY, EVALRY_S3_SECRET_KEY, EVALRY_S3_ENDPOINT |
Object storage for run code. Required in staging and production; dev may use EVALRY_OBJECT_DIR instead |
| dev only | EVALRY_OIDC_PUBLIC_KEY_FILE (a PEM instead of a JWKS URL), EVALRY_DEV_TENANTS=acme:alice,..., EVALRY_AUDIT_FILE. Without a database dev uses the in-memory store and a local audit file |
Every configuration problem is reported at once, and secret values are never printed. Audit events go to one hash chain per tenant in the database; a database outage returns 503 without echoing the connection string.
uv sync
uv run ruff format . && uv run ruff check .
uv run mypy src tests
uv run pytest # sandbox and Postgres tests need Docker and skip without it (Postgres tests start a throwaway container)
- Log only through
evalry.log.get_logger(component); neverprintor bareloggingin library code. - A score change fails
tests/golden. Updatetests/golden/expected.jsononly when the change is intended. - Tests never call a real model: use
FakeGenerator.
- The sandbox shares the host kernel; a stronger isolation option (gVisor or microVMs) is planned.
rpcisolation (default) keeps the hidden tests out of the solution's reach, but the solution sees the test inputs, and tests that need classes, mutated arguments, callbacks or generators fail closed (score 0, exit 4). Use--isolation inprocessfor those; its results are advisory and must not gate.rpcruns two containers per sample: about 0.2 s warm (1.3 s for the first, cold call), measured on a laptop.- Suite runs can use
--workers Nthreads but are not resumable across process restarts; there is no generation cache yet. - Budget stops are checked when a job is claimed, so parallel jobs already running can overshoot by up to the number of workers.
- Running suites through the API is not possible yet: the API stores suites and run records and records gates, and the durable queue exists, but the worker process and an enqueue endpoint are not built. SSO is token verification only (no login flow); roles are assigned in the project, not read from the identity provider, and reads are not audited (denials, grants and writes are).