Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,5 +44,8 @@ jobs:
- name: Typecheck
run: pnpm -r typecheck

- name: Install bubblewrap
run: sudo apt-get update && sudo apt-get install -y bubblewrap

- name: Test
run: pnpm -r test
23 changes: 13 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,19 +90,21 @@ ASIL can run against any HTTP server speaking the OpenAI-compatible `/v1/chat/co

### Running against untrusted repos — sandbox

ASIL executes target-repo code (`pnpm install`, build, tests) inside disposable worktrees. To keep that safe:
ASIL executes target-repo commands (`pnpm install`, build, tests, typecheck, and the scanner's search commands) inside a bubblewrap jail. The command can write only its working directory. It gets a private `/tmp` and `HOME`, not the operator's home directory. Network is off, except for `pnpm install` and `npm install`, which need a registry. Install still passes `--ignore-scripts` unless `ASIL_ALLOW_INSTALL_SCRIPTS=1`.

- **Secrets are stripped from the execution environment.** The runner that executes target-repo commands gets a scrubbed env — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and git/GitHub tokens are never in scope, so a malicious `postinstall`/build/test script can't exfiltrate them. Git and `gh` use a separate allowlisted environment that keeps only path, locale, git identity, and the GitHub token those commands need.
- **Install lifecycle scripts are disabled by default** (`pnpm install --ignore-scripts`). Opt back in with `ASIL_ALLOW_INSTALL_SCRIPTS=1` only for repos you trust.
`bwrap` must be on `PATH`. If it is missing or cannot start, the command fails. Set `ASIL_SANDBOX=off` only when you accept unsandboxed execution; grind prints a warning and continues.

This is process-level hardening. It removes the credential-exfiltration vector for install, build, and test, and it adds the following limits:
Git and `gh` stay outside the jail. They use an allowlisted environment with path, locale, git identity, and the GitHub token. Commits set `core.hooksPath` to `/dev/null` and pass `--no-verify`.

- Git and `gh` do not inherit model API keys. Commits set `core.hooksPath` to `/dev/null` and pass `--no-verify`, so hooks in the target repository do not run.
- Worktrees live under the system temp directory and include a hash of the repository path, so two repositories cannot delete each other's worktrees. Cleanup refuses to delete anything else.
- Patch file reads stay inside the target repository. Relative paths configured for usage data, the queue, skills, and domain answers cannot climb out of the repository. An absolute path is used as given, unless it contains `..` and resolves outside the repository.
- `auto grind` does not commit, push, or open a pull request unless you pass `--apply`. Without that flag a task that passes the gates stays queued for a later run.
Other limits:

It does **not** sandbox the filesystem or network. A hostile repository's build or test can still read local files or make network calls. For an untrusted repository, run ASIL inside a container that does not share your home directory or network.
- `REPO_ROOT` is required for `auto grind`, including `--dry-run`. It does not default to the current directory.
- Worktrees live under the system temp directory and include a hash of the repository path. Cleanup refuses to delete anything else.
- Patch reads stay inside the target repository. Relative paths for usage data, the queue, skills, and domain answers cannot leave the repository. An absolute path is used as given unless it contains `..` and resolves outside the repository.
- `auto grind` does not commit, push, or open a pull request unless you pass `--apply`. `--apply` updates the loop's `auto/*` branches and opens pull requests. Without that flag a task that passes the gates stays queued.
- Before scanning, grind prints whether each scanned category loaded a skill file. A missing file uses the built-in instructions. `--require-skills` exits before scanning when any scanned category is missing its skill.

`--dry-run` does not call a model or change git. It still runs the scanner's test, typecheck, and search commands, inside the jail.

### Languages — Python profile

Expand Down Expand Up @@ -218,7 +220,8 @@ Every LLM call in B and A flows through a `CostCheckpoint`. Each checkpoint is b
|---|---|---|---|
| `ANTHROPIC_API_KEY` | env | required | Primary LLM |
| `OPENAI_API_KEY` | env | required | Adversarial gate (different provider) |
| `REPO_ROOT` | env | current working directory | Absolute path to the target repo. Set this explicitly. |
| `REPO_ROOT` | env | required for `auto grind` | Absolute path of the target repository. `think` and `report` still default to the current directory. |
| `ASIL_SANDBOX` | env | on | Set to `off` to run target-repository commands without bubblewrap. |
| `ASIL_USAGE_DATA_DIR` | env | `<REPO_ROOT>/.asil/usage-data` | Cost tracker state. Relative values must stay inside the repo. An absolute value is used as given unless `..` leaves the repo. |
| `ASIL_QUEUE_PATH` | env | `<REPO_ROOT>/.asil/usage-data/queue.json` | Task queue. Same path rules as `ASIL_USAGE_DATA_DIR`. |
| `ASIL_SKILLS_PATH` | env | `<REPO_ROOT>/.asil/skills` | Markdown skills. Files are `{name}.md` or `{name}/SKILL.md` directly under this directory. Same path rules as `ASIL_USAGE_DATA_DIR`. |
Expand Down
11 changes: 6 additions & 5 deletions examples/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ packages:

## 2. Set up environment

Create `.env` next to `pnpm-workspace.yaml` in the ASIL checkout. The CLI loads that file and no other:
Create `.env` next to `pnpm-workspace.yaml` in the ASIL checkout. The CLI loads that file and no other. `auto grind` refuses to start unless `REPO_ROOT` is set. Target-repository commands run inside bubblewrap (`bwrap` on `PATH`). Set `ASIL_SANDBOX=off` only when you accept unsandboxed execution.

```
ANTHROPIC_API_KEY=sk-ant-…
Expand Down Expand Up @@ -80,10 +80,11 @@ pnpm --filter asil-runners auto grind --max-tasks 3 --skip dependency-update,doc

The runner will:

1. Scan and queue tasks
2. Pause on any unresolved `// DOMAIN_QUESTION:` markers and offer triage
3. For each task: isolate, execute, typecheck, test, self-review × 3, adversarial gate, domain guard. `--apply` then commits, pushes, and opens a pull request. Without `--apply` the task stays queued.
4. Print a summary with cost spent and PRs opened
1. Print whether each scanned category loaded a skill file. Pass `--require-skills` to stop before scanning when one is missing.
2. Scan and queue tasks. Scanner commands run inside the jail.
3. Pause on any unresolved `// DOMAIN_QUESTION:` markers and offer triage
4. For each task: isolate, execute, typecheck, test, self-review × 3, adversarial gate, domain guard. `--apply` then commits, pushes, and opens a pull request, including updates to the loop's `auto/*` branches. Without `--apply` the task stays queued.
5. Print a summary with cost spent and PRs opened

## 7. Inspect the spend

Expand Down
2 changes: 2 additions & 0 deletions packages/asil-improvement-loop/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ export {
type CycleEvent,
} from './cycle-detector.js';
export {
SCANNED_CATEGORIES,
scanCodebase,
scanTestFailures,
scanTypeErrors,
Expand Down Expand Up @@ -52,6 +53,7 @@ export {
type FileFetcher,
} from './executor.js';
export {
findSkillFile,
loadSkillMarkdown,
skillCandidatePaths,
SKILL_NAME_ALIASES,
Expand Down
9 changes: 9 additions & 0 deletions packages/asil-improvement-loop/src/scanner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,15 @@ export interface ScannerDeps {
* `profile` defaults to TypeScript for backwards compatibility with
* callers built before the multi-language refactor.
*/
/** Categories `scanCodebase` actually runs. */
export const SCANNED_CATEGORIES = [
'test-failure',
'type-error',
'todo-resolution',
'coverage-gap',
'dead-code',
] as const;

export async function scanCodebase(
repoRoot: string,
deps: ScannerDeps,
Expand Down
11 changes: 10 additions & 1 deletion packages/asil-improvement-loop/src/skill-loader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
* 3. `skills/{name}.md` (older installs that nested a skills/ folder)
* 4. `skills/{name}/SKILL.md`
*/
import { readFileSync } from 'node:fs';
import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';

/**
Expand Down Expand Up @@ -48,6 +48,15 @@ export function skillCandidatePaths(skillName: string): string[] {
return paths;
}

/** Absolute path of the first matching skill file, or null when none exist. */
export function findSkillFile(skillsPath: string, skillName: string): string | null {
for (const rel of skillCandidatePaths(skillName)) {
const abs = join(skillsPath, rel);
if (existsSync(abs)) return abs;
}
return null;
}

/**
* Load skill Markdown from disk. Returns an empty string when nothing
* matches so callers can fall back to inline defaults.
Expand Down
32 changes: 30 additions & 2 deletions packages/asil-runners/src/__tests__/run-a-flags.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
import { describe, expect, it } from 'vitest';
import { parseFlags } from '../run-a.js';
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { afterEach, describe, expect, it } from 'vitest';
import { formatSkillStatus, parseFlags, skillStatuses } from '../run-a.js';

describe('parseFlags', () => {
it('does not publish unless --apply is set', () => {
Expand All @@ -19,4 +22,29 @@ describe('parseFlags', () => {
expect(flags?.dryRun).toBe(true);
expect(flags?.apply).toBe(true);
});

it('requires skills only when --require-skills is set', () => {
expect(parseFlags([])?.requireSkills).toBe(false);
expect(parseFlags(['--require-skills'])?.requireSkills).toBe(true);
});
});

describe('skillStatuses', () => {
let dir: string;
afterEach(() => rmSync(dir, { recursive: true, force: true }));

it('reports a loaded file and a missing skill', () => {
dir = mkdtempSync(join(tmpdir(), 'asil-skills-'));
mkdirSync(dir, { recursive: true });
writeFileSync(join(dir, 'code-simplification.md'), '# skill\n');
const rows = skillStatuses(dir, ['test-failure', 'todo-resolution', 'coverage-gap', 'dead-code']);
expect(rows).toHaveLength(1);
expect(rows[0]?.category).toBe('type-error');
expect(rows[0]?.path).toBe(join(dir, 'code-simplification.md'));
expect(formatSkillStatus(rows[0]!)).toContain('code-simplification.md');

const missing = skillStatuses(dir, ['type-error', 'test-failure', 'coverage-gap', 'dead-code']);
expect(missing[0]?.path).toBeNull();
expect(formatSkillStatus(missing[0]!)).toContain('missing, using built-in instructions');
});
});
137 changes: 137 additions & 0 deletions packages/asil-runners/src/__tests__/sandbox.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { afterEach, describe, expect, it } from 'vitest';
import {
bubblewrapAvailable,
buildBwrapArgs,
commandAllowsNetwork,
createSandboxedCommandRunner,
toolchainRoot,
} from '../sandbox.js';

const noFs = {
exists: () => false,
resolveCommand: () => null,
pathEnv: '',
};

describe('commandAllowsNetwork', () => {
it('allows network only for pnpm and npm install', () => {
expect(commandAllowsNetwork('pnpm', ['install', '--frozen-lockfile'])).toBe(true);
expect(commandAllowsNetwork('npm', ['install'])).toBe(true);
expect(commandAllowsNetwork('pnpm', ['test'])).toBe(false);
expect(commandAllowsNetwork('pnpm', ['run', '-r', 'build'])).toBe(false);
expect(commandAllowsNetwork('grep', ['-n', 'TODO'])).toBe(false);
});
});

describe('buildBwrapArgs', () => {
it('drops the network and binds only the working directory for a test command', () => {
const args = buildBwrapArgs('pnpm', ['test'], '/work/repo', noFs);
expect(args).toContain('--unshare-net');
expect(args).toContain('--clearenv');
const bindAt = args.indexOf('--bind');
expect(args.slice(bindAt, bindAt + 3)).toEqual(['--bind', '/work/repo', '/work/repo']);
expect(args).toContain('--setenv');
expect(args).toContain('HOME');
expect(args).toContain('/tmp/home');
if (process.env.HOME && process.env.HOME !== '/tmp/home') {
expect(args).not.toContain(process.env.HOME);
}
});

it('keeps the network for pnpm install', () => {
const args = buildBwrapArgs('pnpm', ['install', '--ignore-scripts'], '/work/repo', noFs);
expect(args).not.toContain('--unshare-net');
});

it('read-only binds a toolchain that lives outside the system directories', () => {
const args = buildBwrapArgs('pnpm', ['test'], '/work/repo', {
exists: (path) => path === '/usr',
resolveCommand: (command) =>
command === 'pnpm' ? '/opt/node/bin/pnpm' : null,
pathEnv: '/opt/node/bin',
});
expect(args).toContain('/usr');
expect(args).toContain('/opt/node');
expect(args.filter((part) => part === '--ro-bind').length).toBe(2);
});

it('refuses to bind the filesystem root', () => {
expect(() => buildBwrapArgs('true', [], '/', noFs)).toThrow(/working directory is \//);
});
});

describe('toolchainRoot', () => {
it('uses the parent of a bin directory', () => {
expect(toolchainRoot('/opt/node/bin/node')).toBe('/opt/node');
expect(toolchainRoot('/usr/bin/grep')).toBe('/usr');
});
});

describe('createSandboxedCommandRunner', () => {
const prev = process.env.ASIL_SANDBOX;
afterEach(() => {
if (prev === undefined) delete process.env.ASIL_SANDBOX;
else process.env.ASIL_SANDBOX = prev;
});

it('reports a missing bubblewrap binary instead of running the command unsandboxed', async () => {
const runner = createSandboxedCommandRunner({
...process.env,
ASIL_SANDBOX: 'on',
PATH: '',
});
const result = await runner.run('true', [], { cwd: tmpdir() });
expect(result.exitCode).not.toBe(0);
expect(result.stderr).toMatch(/bwrap|bubblewrap|ENOENT/i);
});
});

describe('bubblewrap jail', () => {
const available = bubblewrapAvailable();
let dir: string;
afterEach(() => {
if (dir) rmSync(dir, { recursive: true, force: true });
});

it.skipIf(!available)(
'cannot read a sibling directory or open an outbound socket',
async () => {
dir = mkdtempSync(join(tmpdir(), 'asil-jail-'));
const work = join(dir, 'work');
const outside = join(dir, 'secret.txt');
mkdirSync(work);
writeFileSync(outside, 'secret');
writeFileSync(
join(work, 'probe.js'),
`
const fs = require('fs');
const net = require('net');
let readOk = false;
try { fs.readFileSync(process.argv[2]); readOk = true; } catch {}
const sock = net.connect({ host: '1.1.1.1', port: 443 });
const finish = (netOpen) => {
console.log(readOk ? 'READ_OK' : 'READ_BLOCKED');
console.log(netOpen ? 'NET_OPEN' : 'NET_BLOCKED');
process.exit(readOk || netOpen ? 2 : 0);
};
sock.setTimeout(1500);
sock.on('connect', () => finish(true));
sock.on('timeout', () => finish(true));
sock.on('error', () => finish(false));
`,
);
const runner = createSandboxedCommandRunner();
const result = await runner.run(
'node',
[join(work, 'probe.js'), outside],
{ cwd: work },
);
expect(result.stdout).toContain('READ_BLOCKED');
expect(result.stdout).toContain('NET_BLOCKED');
expect(result.exitCode).toBe(0);
},
);
});
5 changes: 4 additions & 1 deletion packages/asil-runners/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,11 @@ const HELP = `
Set ANTHROPIC_API_KEY and OPENAI_API_KEY in the environment or in
the repository root .env (next to pnpm-workspace.yaml).
OPENAI_API_KEY is only needed for grind (adversarial gate).
REPO_ROOT is required for grind. It does not default to the
current directory.
grind does not commit, push, or open a pull request unless you
pass --apply.
pass --apply. Target-repository commands run inside bubblewrap
unless ASIL_SANDBOX=off.
`.trimStart();

export async function runCli(
Expand Down
Loading
Loading