Skip to content

Fix flaky html2 tests with deterministic waits - #5894

Merged
Eugene (OEvgeny) merged 3 commits into
mainfrom
OEvgeny/fix-flaky-html2-tests
Oct 18, 2026
Merged

Eugene (OEvgeny) merged 3 commits into
mainfrom
OEvgeny/fix-flaky-html2-tests

Conversation

@OEvgeny

Copy link
Copy Markdown
Collaborator

Changelog Entry

Not required, development only.

Description

Six html2 tests failed intermittently on CI or under load because they asserted or took a snapshot before asynchronous work had finished. Each test now waits for the state it checks, with waitFor() from @testduet/wait-for or existing page conditions. There are no retries and no sleeps. Snapshots are unchanged.

Test Cause Fix
fluentTheme/carousel.html The first snapshot waited for the outgoing message only. The bot response could arrive during the snapshot, so the card images were captured blank. Wait for both activities and their images before the first snapshot
markdown/codeBlockCopyButton/behavior.denied.html The copy button calls navigator.permissions.query() before it sets aria-disabled, but the test asserted right after pressing ENTER waitFor() on aria-disabled
attachment/adaptiveCard/custom.*.html postActivity() resolves before Web Chat renders the bot response. The snapshots only waited until the screen stopped changing, so the response could be missing if rendering was late. Wait for the expected number of activities and for the transcript to scroll to the bottom before each snapshot. Wait for the response to each button click before the next click.
activity/viewCodeButton.scroll.html 2 ARROW_DOWN presses were sent back to back. Chrome sometimes drops a keyboard scroll that arrives while the previous smooth scroll is finishing. Wait for scrollend after each key press
activity/legacy/unknownActivity.html The telemetry exception event is sent from an effect after the error box renders, but the test asserted right after numActivitiesShown(2) waitFor() on the console error and on the exception event, as in unknownAttachment.html

Design

  • behavior.denied.html was converted from a Babel script to a module script so that it can import waitFor(), as the feedbackForm tests do. The JSX is now createElement() and renders the same DOM.
  • custom.*.html uses the existing pageConditions.scrollToBottomCompleted(). This helper includes a 500 ms settle time. Locally, each variant took 17–20 s both before and after the change.
  • unknownAttachment.html asserted its telemetry exception event without waiting, which has the same race. It now waits too.
  • The existing wait for the 3 buttons in custom.skip.html returned a boolean, so waitFor() never retried. It now uses expect().
  • The changed tests are formatted with Prettier in a separate commit (whitespace only).

Follow-ups (not seen failing on CI, not changed here):

  • markdown/codeBlockCopyButton/behavior.html, behavior.highlighted.html and adaptiveCards/behavior.html take the "Copied" snapshot right after ENTER, but the "Copied" state is also set asynchronously. With the permission query delayed by 200 ms, the snapshot differs.

Specific Changes

  • fluentTheme/carousel.html: wait for numActivitiesShown(2) and allImagesLoaded() before the first snapshot

  • markdown/codeBlockCopyButton/behavior.denied.html: module script, waitFor() on aria-disabled

  • attachment/adaptiveCard/custom.skip.html (shared by custom.{copilot,fluent}[.dark].html): numActivitiesShown() and scrollToBottomCompleted() before each snapshot and after each click, expect() in the existing waitFor()

  • activity/viewCodeButton.scroll.html: wait for scrollend after each ARROW_DOWN

  • activity/legacy/unknownActivity.html, unknownAttachment.html: waitFor() on the console error and the telemetry exception event

  • I have added tests and executed them locally

  • I have updated CHANGELOG.md

  • I have updated documentation

Review Checklist

This section is for contributors to review your work.

  • Accessibility reviewed (tab order, content readability, alt text, color contrast)
  • Browser and platform compatibilities reviewed
  • CSS styles reviewed (minimal rules, no z-index)
  • Documents reviewed (docs, samples, live demo)
  • Internationalization reviewed (strings, unit formatting)
  • package.json and package-lock.json reviewed
  • Security reviewed (no data URIs, check for nonce leak)
  • Tests reviewed (coverage, legitimacy)

Eugene Olonov added 2 commits October 17, 2026 09:00
- fluentTheme/carousel.html: wait for the bot response and its images before the first snapshot, the response could arrive during the snapshot and the card images were captured blank.
- markdown/codeBlockCopyButton/behavior.denied.html: the copy button checks the clipboard permission asynchronously, wait for aria-disabled with waitFor().
- attachment/adaptiveCard/custom.*.html: wait for the bot responses and the scroll before each snapshot, and for the response to each click before the next click. The wait for the 3 buttons now uses expect().
- activity/viewCodeButton.scroll.html: wait for each keyboard scroll to end before the next ARROW_DOWN, Chrome may drop a keyboard scroll that arrives while the previous smooth scroll is ending.
- activity/legacy/unknownActivity.html, unknownAttachment.html: wait for the console error and the telemetry exception event, the exception is tracked in an effect after the error box is rendered.
@OEvgeny
Eugene (OEvgeny) marked this pull request as ready for review October 17, 2026 22:01
Copilot AI balanced review requested due to automatic review settings October 17, 2026 22:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@OEvgeny
Eugene (OEvgeny) enabled auto-merge (squash) October 18, 2026 05:41
@OEvgeny
Eugene (OEvgeny) merged commit 2ec2e9f into main Oct 18, 2026
32 checks passed
@OEvgeny
Eugene (OEvgeny) deleted the OEvgeny/fix-flaky-html2-tests branch October 18, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants