Skip to content

Own --tests_url/--tests_dir in the sweep instead of failing on position - #52

Merged
maximusunc merged 1 commit into
mainfrom
claude/helmsdeep-performance-runner-y0q51y
Oct 2, 2026
Merged

maximusunc merged 1 commit into
mainfrom
claude/helmsdeep-performance-runner-y0q51y

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

--tests_url belongs to the harness's download subcommand, and the sweep puts passthrough arguments in front of the subcommand, so passing it after -- dies on "unrecognized arguments" -- an error that says the flag is wrong when only its position is. That is a bad failure to hand someone editing a CronJob manifest by hand, where the feedback loop is a deploy.

The sweep already had --tests_url and just needed to say so; --tests_dir, its load counterpart, was missing entirely, so a sweep over a suite outside the bundled directory had no way to say where it lived. Add it, place both after their subcommand where argparse accepts them, and reject either one appearing in the passthrough with an error naming the flag and where it goes.

The CronJob example now also spells out that --performance_profile is the only thing selecting which queries HelmsDeep sends. The suite's test assets don't, so a job named for pathfinder tests but running the default profile measures inferred queries instead -- and nothing about its output would say so.

Verified against a locally served suite zip: downloaded from a custom URL, parsed, ran, exit 0; and the misplaced form now stops with the guidance instead of the harness's own error.

Claude-Session: https://claude.ai/code/session_01Uu7mLrTmyXzLSY8kyR3tjD

`--tests_url` belongs to the harness's `download` subcommand, and the sweep
puts passthrough arguments in front of the subcommand, so passing it after `--`
dies on "unrecognized arguments" -- an error that says the flag is wrong when
only its position is. That is a bad failure to hand someone editing a CronJob
manifest by hand, where the feedback loop is a deploy.

The sweep already had --tests_url and just needed to say so; --tests_dir, its
`load` counterpart, was missing entirely, so a sweep over a suite outside the
bundled directory had no way to say where it lived. Add it, place both after
their subcommand where argparse accepts them, and reject either one appearing
in the passthrough with an error naming the flag and where it goes.

The CronJob example now also spells out that --performance_profile is the only
thing selecting which queries HelmsDeep sends. The suite's test assets don't,
so a job named for pathfinder tests but running the `default` profile measures
inferred queries instead -- and nothing about its output would say so.

Verified against a locally served suite zip: downloaded from a custom URL,
parsed, ran, exit 0; and the misplaced form now stops with the guidance instead
of the harness's own error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uu7mLrTmyXzLSY8kyR3tjD
@maximusunc
maximusunc merged commit 45e3ea1 into main Oct 2, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/helmsdeep-performance-runner-y0q51y branch October 2, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants