Skip to content

TEST: technique acceptance - #4016

Open
Roman Lutz (romanlutz) wants to merge 4 commits into
microsoft:mainfrom
romanlutz:romanlutz-scenario-technique-acceptance
Open

Roman Lutz (romanlutz) wants to merge 4 commits into
microsoft:mainfrom
romanlutz:romanlutz-scenario-technique-acceptance

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Closes the remaining Scenario GUI technique-selection acceptance gap with eight desktop/narrow Playwright journeys against the real backend, registry, preset storage, estimate service and execution path. The fixture uses deterministic local targets, converters and scoring, private SQLite/configuration/preset storage, and no external providers or developer environment files. Successful API responses are not intercepted. Only the loading/error cases deliberately hold one estimate or return one HTTP 503.

The journeys cover an actual preset edit, scenario-relative defaults, untouched preset values, aggregate expansion, independent baseline settings, keyboard/focus recovery, retained selections and displayed totals through failed/pending estimates, recovered totals/status, Run preview and persisted execution. Each primary journey completes exactly the two selected techniques plus baseline and verifies the actual request, saved configuration, plan and successful results.

The tightly scoped UI fixes associate unavailable technique/baseline controls with their existing explanations, add role="status" to the two existing updating/stale estimate elements, and apply the existing touch-target class to both preview buttons. Coarse-pointer checks wait for the dialog to settle and measure at least 44px without fixed sleeps. Backend sizing/compatibility ownership and desktop behavior are unchanged. CI/package/Playwright wiring is feature-specific and additive.

This is an independent leaf based on main including the ScenarioDetail split in #3526 (0f5f6589e92824268b5151409ad19c8593e1dc5a), not stacked on the dataset or History follow-ups. ScenarioDetail.tsx exactly matches that upstream revision; the approved attributes live in the extracted technique, estimate and preview components. Dataset-label/count/metadata semantics are not changed.

Tests and Documentation

Validated head: aaaad689981ec0dd48c67f0309179ba277dc9501, clean worktree. Independent Reviewer critically reviewed all 14 leaf files after the split against upstream 0f5f6589e92824268b5151409ad19c8593e1dc5a: APPROVE, no actionable findings. Its initial finding about missing rendered estimate-recovery proof was fixed and explicitly confirmed resolved in the post-split review.

From frontend:

Exact command Result
npm run test:e2e:scenario-techniques -- --reporter=line,json Passed build and all 8 real-backend journeys in 111.415s; 0 skipped, failed or flaky tests.
npm test -- --runInBand src\components\Scenarios\ScenarioDetail.test.tsx src\components\ScenarioPresets\ScenarioPresetEditor.test.tsx --silent --json --outputFile="$evidence\jest-report.json" Passed both complete suites, 212 tests. $evidence is the retained exact-head evidence directory.
npm run build Passed TypeScript and Vite production build, with the compatibility stamp bound to the validated head.
npx eslint e2e\scenario-techniques.spec.ts playwright.scenario-techniques.config.ts src\components\Scenarios\ScenarioDetail.tsx src\components\Scenarios\ScenarioTechniqueSection.tsx src\components\Scenarios\ScenarioEstimateSection.tsx src\components\Scenarios\ScenarioRunPreviewDialog.tsx src\components\Scenarios\ScenarioTechniqueSelector.tsx src\components\Scenarios\ScenarioDetail.test.tsx src\components\ScenarioPresets\ScenarioPresetEditor.test.tsx Passed.
npx playwright test --list --reporter=line Passed discovery: 334 ordinary tests in 26 files, excluding the dedicated technique spec.

Explicit acceptance/config TypeScript checking also passed with this command from frontend:

node --input-type=module -e "import ts from 'typescript'; import path from 'node:path'; const config = ts.readConfigFile('tsconfig.json', ts.sys.readFile); const parsed = ts.parseJsonConfigFileContent(config.config, ts.sys, process.cwd()); const program = ts.createProgram([path.join('e2e', 'scenario-techniques.spec.ts'), 'playwright.scenario-techniques.config.ts'], {...parsed.options, types: ['node']}); const diagnostics = ts.getPreEmitDiagnostics(program); console.log(ts.formatDiagnosticsWithColorAndContext(diagnostics, {getCanonicalFileName: x => x, getCurrentDirectory: ts.sys.getCurrentDirectory, getNewLine: () => ts.sys.newLine})); process.exit(diagnostics.length ? 1 : 0);"

From the repository root:

Exact command Result
uv run --no-sync pytest tests\unit\backend\test_scenario_techniques_fixture.py --basetemp .pyrit_cache\technique-fixture-post-split-unit -q Passed 3 pre-mutation containment regressions.
uv run --no-sync ruff check frontend\e2e\fixtures\scenario_techniques_backend.py tests\unit\backend\test_scenario_techniques_fixture.py Passed.
uv run --no-sync ruff format --check frontend\e2e\fixtures\scenario_techniques_backend.py tests\unit\backend\test_scenario_techniques_fixture.py Passed, both files formatted.
uv run --no-sync ty check frontend\e2e\fixtures\scenario_techniques_backend.py tests\unit\backend\test_scenario_techniques_fixture.py --output-format concise Passed.
uv run --no-sync python build_scripts\check_async_suffix.py frontend\e2e\fixtures\scenario_techniques_backend.py Passed.

The README documents fixture isolation, deliberate fault injection, the acceptance command and screenshot provenance. Eighteen screenshot/provenance pairs, two successful execution records, both test reports and a checksummed verification manifest are retained in the session Library under scenario-techniques\aaaad689981ec0dd48c67f0309179ba277dc9501. Each viewport captures preset edit, loading, failure, keyboard focus, preview, completed run and all three baseline states. Verification confirmed the clean source/compatibility head, all expected request/result shapes and private fixture paths. All six named external synthetic preset JSON paths and their six hidden lock sidecars were absent; no external artifacts were changed. Earlier captures retain their original head binding.

No unresolved review or acceptance blocker. Existing Vite chunk-size/Pydantic warnings and Windows connection-reset logging did not fail the final suite. External-provider/live suites and JupyText were not run because no provider integration or notebooks changed.

Exercise real preset editing, scenario defaults, estimate recovery and launch with isolated provider-free data. Retain desktop/narrow provenance and execution evidence, explain disabled choices, and use existing touch targets for preview actions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the technique descriptions, existing estimate status roles, and preview touch targets in their extracted components.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 20:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the technique-selection acceptance coverage and accessibility changes.

No open issues 🎉

What changed in this PR

Adds real-backend acceptance coverage for Scenario GUI technique selection, building on the ScenarioDetail component split in #3526. UI changes focus on accessible explanations, estimate status announcements, and touch-friendly preview actions.

Changes:

  • Adds eight desktop/narrow browser journeys covering presets, defaults, baseline selection, estimate recovery, and persisted execution.
  • Introduces a provider-free backend fixture with isolated storage and containment regression tests.
  • Adds dedicated Playwright/CI wiring and documents execution and evidence capture.
File Description
tests/​unit/​backend/​test_scenario_techniques_fixture.py Tests preset-write and lock-file containment.
frontend/​src/​components/​Scenarios/​ScenarioTechniqueSelector.tsx Associates unavailable baseline controls with explanations.
frontend/​src/​components/​Scenarios/​ScenarioTechniqueSection.tsx Adds accessible technique-limit and baseline explanations.
frontend/​src/​components/​Scenarios/​ScenarioRunPreviewDialog.tsx Applies touch-target sizing to preview actions.
frontend/​src/​components/​Scenarios/​ScenarioEstimateSection.tsx Announces updating and stale estimate states.
frontend/​src/​components/​Scenarios/​ScenarioDetail.test.tsx Covers keyboard navigation, status announcements, and descriptions.
frontend/​src/​components/​ScenarioPresets/​ScenarioPresetEditor.test.tsx Tests the unavailable baseline’s accessible description.
frontend/​README.md Documents isolated checks and evidence provenance.
frontend/​playwright.scenario-techniques.config.ts Configures the dedicated real-backend browser suite.
frontend/​playwright.config.ts Excludes the dedicated suite from ordinary discovery.
frontend/​package.json Adds the technique acceptance test command.
frontend/​e2e/​scenario-techniques.spec.ts Implements eight browser journeys and evidence capture.
frontend/​e2e/​fixtures/​scenario_techniques_backend.py Provides isolated local scenarios and estimate fault controls.
.github/​workflows/​frontend_tests.yml Adds a dedicated technique acceptance CI shard.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants