[Hackathon] fix(test): retrying create-batch --run or a several-id test run with the same idempotency key starts (and bills) every run again #492

Verified against (commit SHA or released version)

988c035

Discord Username / User ID

shubhdeep_chhabra

What does this improvement do?

--idempotency-key is documented as the way to retry safely: test create-batch --help says
"pin one yourself for safe retries", and test run --help says "opaque key for safe retries … Auto-generated
keys print to stderr". Retrying with the same key is supposed to make the server return the runs the first
attempt already started. On two batch paths the CLI does not send the same keys on the retry, so the server
cannot recognise the repeat and starts every run again. Each frontend run costs credits.

1. test create-batch --run (src/commands/test.ts:4387). Each trigger gets
cli-batch-run-<random UUID>, even when --idempotency-key K is passed. On a retry with K the batch create
is recognised as a repeat (same test ids come back), but every run trigger carries a new random key, so all of
them are started a second time. The sibling test create --run already gets this right: it derives
<K>:run (test.ts:1626).

2. Several-id test run through a tunnel (--local <port>, a saved loopback environment, or
--all --local; runTestRunLocalBatch, test.ts:11253-11261). With --idempotency-key K the keys are
K:<testId>, which is correct. Without it, the CLI prints one auto-minted key, uses it as-is for the
first test only, and gives every other test a random key that is never shown. Retrying with the printed
key produces <printed>:<testId> for every test, which matches none of the keys sent the first time.

When this bites: a CI step is killed or times out after the server accepted the runs, and the CI retry (or a
coding agent following the help text) runs the same command again with the same key.

Steps to reproduce

A fake API that implements idempotency the way the real one does (a repeated key returns the stored response;
a new key on a trigger starts a new run), then the built CLI run twice with the same key:

$ testsprite --endpoint-url http://127.0.0.1:48911 --output json test create-batch \
    --plans plans3.jsonl --run --no-wait --skip-preflight --idempotency-key ci-build-42   # twice

runs started on the server: 6
  POST /tests/test_f1/runs key=cli-batch-run-8a37b462-…
  POST /tests/test_f2/runs key=cli-batch-run-a6cb2c48-…
  POST /tests/test_f3/runs key=cli-batch-run-e224ebd4-…
  POST /tests/test_f1/runs key=cli-batch-run-fb3b9132-…   ← retry: new keys, 3 more runs
  POST /tests/test_f2/runs key=cli-batch-run-9b1ba549-…
  POST /tests/test_f3/runs key=cli-batch-run-e5186529-…

For path 2, the keys sent by test run a b c --local <port> and by the retry with the printed key:

import http from 'node:http';
const store = new Map(); let runsStarted = 0; const log = [];
const send = (res, status, body) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(body)); };
http.createServer((req, res) => {
  let raw = ''; req.on('data', c => (raw += c)); req.on('end', () => {
    const p = new URL(req.url, 'http://x').pathname.replace('/api/cli/v1', '');
    const key = req.headers['idempotency-key'];
    if (p === '/__stats') return send(res, 200, { runsStarted, log });
    log.push(`${req.method} ${p}${key ? ' key=' + key : ''}`);
    if (key && store.has(key)) return send(res, 200, store.get(key));
    let body;
    if (req.method === 'POST' && p === '/tests/batch')
      body = { results: ['test_f1', 'test_f2', 'test_f3'].map((testId, i) => ({ specIndex: i, testId, status: 'created' })), summary: { total: 3, created: 3, failed: 0 } };
    else if (req.method === 'POST' && /^\/tests\/[^/]+\/runs$/.test(p))
      body = { runId: `run_${++runsStarted}`, status: 'queued', enqueuedAt: new Date().toISOString(), codeVersion: 'v1', targetUrl: 'https://example.com' };
    else return send(res, 404, { error: { code: 'NOT_FOUND', message: 'no route', nextAction: '', requestId: 'r', details: {} } });
    if (key) store.set(key, body);
    send(res, 200, body);
  });
}).listen(48911);

Details / implementation notes

• Derive every per-test trigger key from the one key the user has (passed or printed), the same way the code
  already derives <key>:run, <key>:chunkN and the test-list group keys:
  • create-batch --run: <createKey>:run:<testId> (never the create key itself, which would replay the
    create response);
  • several-id test run: <key>:<testId> for every test, auto-minted keys included.
• Keep the derived key within 256 characters by trimming the base, as the existing derivations do.
• Tests: a retry with the same key (passed or printed) re-sends identical trigger keys; different tests still
  get distinct keys. No new dependency.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.