AI Engineer · RL environments and LLM evaluation · Python, Docker, AWS
I build the environments that LLM coding agents train and get graded in. My day job is turning real merged GitHub pull requests into reproducible Docker tasks with hidden tests, then checking that agents actually solve them instead of gaming the grader. I've shipped 2,000+ of these environments and 2,000+ agent trajectories for SFT and RL post-training.
I'm open to AI engineer, LLM eval and Python backend roles, remote or relocation.
I read the code of LLM eval and interpretability libraries I use, find real bugs, and send fixes with regression tests. Every one of these started as a bug I hit or spotted myself.
| Project | ⭐ | PR | Status | What I fixed |
|---|---|---|---|---|
| SAELens decoderesearch |
1.6k | #776 |  |
random_firing_probabilities built a fresh torch.Generator(), which starts from a fixed seed, so unseeded calls returned identical probabilities every time |
| delphi EleutherAI |
280 | #192 |  |
split_quantiles floored the per-quantile count, so 25 samples over 10 quantiles gave 20, and fewer samples than quantiles gave 0 with no warning |
| nnsight ndif-team |
1.1k | #800 |  |
Reassembling a multi-invoke .skip() lost the container type: namedtuple outputs raised TypeError and ModelOutput/OrderedDict came back as plain dicts |
| TransformerLens TransformerLensOrg |
4k | #1965 |  |
Eval helpers hardcoded "cuda", which crashed on CPU and Apple MPS machines |
| lighteval huggingface |
2.6k | #1465 |  |
The LiveCodeBench sandbox's fake stdout had no binary buffer, so correct solutions were graded as wrong |
| sparsify EleutherAI |
740 | #146 |  |
handle_arg_string used str.isnumeric(), so -5 silently became -5.0 and Unicode numerals like ½ crashed |
Bug reports
- langchain-ai/deepagents#7477:
executetimeout leaves the command's child processes running (fix ready on my fork) - langchain-ai/deepagents#7478:
FilesystemBackendwrites truncate in place, so a crash mid-write destroys the file - langchain-ai/deepagents#7479:
langchain-google-genaiis a required dependency but never imported - huggingface/lighteval#1464: LCB grader fails correct solutions that write via
sys.stdout.buffer
My main open-source project. It mines PRs merged after a model's cutoff, rebuilds each repo at the base commit in Docker, and grades agents the way SWE-bench style benchmarks do. Then it tries hard to cheat its own grader.
| Repo | What it does |
|---|---|
| gauntlet-harness | The core. PR mining, 3-run F2P/N2P/S2P/P2P test classification, an 8-check leak scan, rubric judging, a Gymnasium env (gauntlet/SWE-v0) and exports for SWE-bench, SFT, GRPO and DPO |
| gauntlet-cheatcatcher | 11 reward-hacking rules that flag test tampering, conftest tricks and hardcoded outputs |
| gauntlet-repo2env | Turns a repo into env images and SWE-style tasks, with grading hardened against test edits |
| gauntlet-sandbox | Docker sandboxes that record every agent step as a trajectory |
| gauntlet-timemachine | Replay, branch and diff trajectories, plus an HTML viewer |
| gauntlet-arena | FastAPI app where a human and an AI agent race on the same task, with a leaderboard |
| gauntlet | Umbrella repo. One docker compose up runs the whole thing |
Gauntlet Live (Nov 2026): 53 verified tasks out of 186 PRs across 10 repos. The oracle solves 53/53 and the cheater policies get caught 53/53. Red-teaming my own grader found a "stealth" cheat that scored 7/10, so I rebuilt the canary tests until it dropped to 0/60. CI runs a nightly calibration on random tasks.
Daily: Python · Docker · pytest · Git · AWS (ECR, S3, Lambda, Glue) · Linux · GitHub Actions
LLM and RL: Claude, GPT and GLM APIs · LLM-as-judge · Hugging Face · PyTorch · Terminal-Bench / Harbor · Multi-SWE-bench · LangChain
Backend and data: FastAPI · PostgreSQL · Redis · SQL · Snowflake · ETL pipelines
- Ethara AI (via Quess Corp) · Software Engineer, RL Environments and Evaluation · 2026 to now
- NavAdr Technologies · Software Engineer (Data) · 2025 to 2026. Python ETL into Snowflake, 200K+ records a day
- GlobalLogic · Associate Analyst · 2024 to 2025. CI/CD with Docker and GitHub Actions on AWS
- quizbuzz-night: live multiplayer trivia buzzer. 2 to 8 players join from their phones with a room code, no login
- neetcode-submissions: my DSA practice in Python