Skip to content

[#175] Run the openig-ui QUnit suite in headless Chrome during the Maven test phase - #179

Merged
vharseko merged 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:ui-qunit-headless-chrome
Oct 22, 2026
Merged

vharseko merged 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:ui-qunit-headless-chrome

Conversation

@vharseko

@vharseko vharseko commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Fixes #175.

The qunit task had been commented out of the build task since 2019 (f270c42a), and grunt qunit itself timed out: grunt-contrib-qunit 1.3 / PhantomJS cannot load the ES2015 app any more. The four suites in src/test/js/tests (39 tests) gave no signal.

Runner

  • grunt-contrib-qunit 1.3.0 → 10.2.0 (headless Chrome via Puppeteer; Chrome is downloaded during npm install), QUnit 2.26 from npm instead of the qunit:1.15.0 Maven artifact (copy-dependencies-qunit execution, the two dependencies and the root pom's two qunit download items removed). Puppeteer is pinned to 24.43.1 so that every fresh install tests against the same Chrome (openig-ui has no lockfile; the plugin itself asks for ^24.0.0).
  • Chrome runs with --allow-file-access-from-files: the test page is opened from file:// and the app fetches translations and templates through XHR, which Chrome blocks for local files by default (PhantomJS did not). puppeteer.args replaces the arguments the plugin would build itself, so they are rebuilt next to it: CHROMIUM_FLAGS if set, else --no-sandbox when CI is set. Building as root (e.g. in a container) therefore needs CI=true or CHROMIUM_FLAGS=--no-sandbox, as with the plugin alone.
  • New grunt test task (copy:qunit + qunit), bound to the Maven test phase as a separate execution rather than put back into build: mvn verify fails on a failing JS test, and -DskipTests skips the suite through frontend-maven-plugin's built-in skipTests parameter (same layout as OpenAM's npm-test execution, Run frontend Karma tests during the Maven test phase OpenAM#902). The plugin never reads maven.test.skip, so a maven-test-skip profile, active on -Dmaven.test.skip=true, skips that execution as well; <skip> on the execution itself would have replaced ${skip.grunt} for it.

Harness (config.js / run.js)

Two problems only showed up once Chrome actually ran the page:

  1. config.js required main and ../test/run together. The paths the test suites' dependencies resolve through (i18next, backbone, …) are set by the require.config inside main.js; after grunt build that is the uglified bundle, which loads slower than the tiny run.js, so the dependencies were resolved before the config existed (Script error for "i18next"). Historically qunit ran in build-dev before requirejs, so the race was won by luck. Now run.js registers the EVENT_APP_INITIALIZED listener before main is required and loads the suites inside that listener.
  2. The optimised bundle carries its own copy of jQuery, lodash, bootstrap, …; the test page has already loaded jQuery as a module (needed by MockServer/EventManager), so executing the bundle created a second jQuery instance — bootstrap (shim) attached $.fn.modal to the new global while the AMD bootstrap-dialog got the old one (Cannot read properties of undefined (reading 'Constructor')). The test config therefore starts the app from the unbundled target/compose/main.js (paths.main); everything else is still served from target/www.

Tests

  • Migrated to the QUnit 2 API: module() → QUnit.module(), QUnit.asyncTest + QUnit.start() → QUnit.test (all callbacks under test are synchronous). PhantomJS Function.prototype.bind polyfill removed.
  • assert.throws(…, transformService.TransformServiceException("invalidModel"), …) called the constructor without new, so expected was undefined and the assertions only checked that something was thrown. They now use a validator that checks name and errorType; this exposed that the "unknown filter type" test expected invalidModel while transformFilter throws unknownFilterType.
  • The "no 'name'" and "no 'baseURI'" fixtures each left both attributes empty, and RouteModel.validate() stops at the first missing one, so neither test could fail if the check it names was removed. Each fixture now leaves out only the attribute under test.
  • New StylesheetTests: every @import of the built css/structure.css must load. Less keeps a CSS @import as it is and a stylesheet that fails to load does not fail the page, which is how the CodeMirror path below stayed broken unnoticed.

Also

  • structure.less still imported ../libs/CodeMirror-4.10.0/lib/codemirror.css; Fix build frontend codemirror lib error #36 moved the CodeMirror location to libs/codemirror5-4.10.0 in main.js only, so the stylesheet has been a 404 in the built UI since then (CodeMirror is used for the route JSON view). Path fixed; StylesheetTests now covers it.

Verification

  • mvn -pl openig-ui -am verify: grunt-test → 40 tests completed, 0 failed, BUILD SUCCESS.
  • -DskipTests and -Dmaven.test.skip=true: grunt-test → Skipping execution.; -Dskip.grunt=true still skips both grunt and grunt-test.
  • Negative checks against the built copies (each → 1 failed, grunt exit 6):
    • a flipped expectation in DataFilterTests;
    • transformError("invalidModel") in the unknown filter type test (Actual: TransformServiceException: UnknownTypeOfFilter);
    • the name branch, then the baseURI branch, removed from RouteModel.validate() → the matching "no 'name'" / "no 'baseURI'" test fails;
    • the old CodeMirror-4.10.0 path in structure.css → css/../libs/CodeMirror-4.10.0/lib/codemirror.css is loaded fails.
  • Chrome arguments: none → --allow-file-access-from-files; CI=true → + --no-sandbox; CHROMIUM_FLAGS="--no-sandbox --disable-gpu" → + --no-sandbox --disable-gpu (suite green); CHROMIUM_FLAGS wins over CI, as in the plugin.
  • grunt eslint: 0 errors; the 5 warnings are pre-existing in src/main/js.
  • Remaining console noise in the test run is expected: file:///api/info (Footer → ServerInfoService.getInfo() with no backend, handled by its .fail) and the resulting "Unknown error" message from the commons REST error handler.

@vharseko vharseko added javascript Pull requests that update Javascript code ci Build, CI/CD, and GitHub Actions changes tests Adds or updates tests dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation labels Sep 18, 2026

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The suite is back as a real gate, and the fixes it surfaced are the right ones.

  • The grunt-test execution in the test phase makes mvn verify fail on a red QUnit test: frontend-maven-plugin 1.6 turns grunt's non-zero exit into a MojoFailureException. The whole build-maven matrix (ubuntu/macos/windows) runs the 39 tests green.
  • The transformError(...) validator replaces assert.throws(..., transformService.TransformServiceException("invalidModel"), ...), whose expected was undefined. The tighter check exposed the wrong invalidModel expectation in the unknown-filter-type test.
  • config.js calls run(MockServer.instance) before require(["main"]), so the EVENT_APP_INITIALIZED listener cannot lose the race against the bundle.

issue (non-blocking): Setting puppeteer.args drops grunt-contrib-qunit's CHROMIUM_FLAGS override, not only its CI-only --no-sandbox.

openig-ui/Gruntfile.js:53-56

grunt-contrib-qunit 10.2.0 builds its arguments from (process.env.CHROMIUM_FLAGS || (process.env.CI ? '--no-sandbox' : '')).split(' ') (tasks/qunit.js:257-259), and Object.assign replaces them with options.puppeteer.args (qunit.js:276-281). With this PR, mvn verify run as root with CI unset (a Maven Docker image, a local container) launches Chrome without --no-sandbox, and Chrome refuses to run as root. The plugin's documented workaround, CHROMIUM_FLAGS=--no-sandbox, now has no effect, so the only ways out are CI=true or -DskipTests. Not run here (no Docker daemon). No workflow in this repo builds as root without CI.

        // Setting puppeteer.args replaces the arguments grunt-contrib-qunit would build itself
        // (CHROMIUM_FLAGS, else --no-sandbox under CI), so they are re-created here.
        chromiumArgs = ["--allow-file-access-from-files"].concat(
            (process.env.CHROMIUM_FLAGS || (process.env.CI ? "--no-sandbox" : "")).split(" ").filter(Boolean)
        );

question (non-blocking): Is a build that passes only -Dmaven.test.skip=true meant to keep running the JS suite?

openig-ui/pom.xml:96-104

frontend-maven-plugin 1.6 binds the grunt goal's skipTests to ${skipTests} and skip to ${skip.grunt}, and never reads maven.test.skip. So mvn verify -Dmaven.test.skip=true now downloads and launches headless Chrome and fails on a red JS test, where before nothing in openig-ui ran a browser. No workflow in this repo passes that flag on its own. It is a Minor either way. If it is not intended, honour the flag on this execution:

          <execution>
            <id>grunt-test</id>
            <goals>
              <goal>grunt</goal>
            </goals>
            <phase>test</phase>
            <configuration>
              <arguments>test</arguments>
              <skip>${maven.test.skip}</skip>
            </configuration>
          </execution>

plus <maven.test.skip>false</maven.test.skip> in the module's <properties>. Note that <skip> also replaces ${skip.grunt} for this execution. Or, if it is intended: add a README line saying only -DskipTests skips the JS suite.


suggestion (non-blocking): The 'no name' and 'no baseURI' tests cannot fail if the check they name is removed.

openig-ui/src/test/js/tests/TransformServiceTests.js:57, :71

RouteModel.validate() checks id, then name, then baseURI, and returns on the first failure (RouteModel.js:65-77). The 'no baseURI' fixture {id: "modelID", url: ""} leaves name at its default "", and url is not an attribute validate() reads. The model is therefore invalid because of the name, and deleting the baseURI branch keeps the test green. The 'no name' fixture has the mirror gap: its default baseURI: "" keeps it red-proof if the name check is removed. errorType is invalidModel for every branch, so the new validator cannot tell them apart.

            const applicationWithEmptyName = new RouteModel({
                id: "modelID",
                name: "",
                baseURI: "http://www.example.com"
            });
            // ...
            const applicationWithEmptyBaseUrl = new RouteModel({
                id: "modelID",
                name: "routeName",
                baseURI: ""
            });

Pin: with one missing attribute per fixture, deleting the baseURI (or name) branch of validate() makes transformRoute pass its isValid() check (TransformService.js:247) and return a config, so assert.throws turns red.


suggestion (non-blocking): Nothing pins the CodeMirror stylesheet path fix, because failed page loads never fail the QUnit run.

openig-ui/src/main/resources/css/structure.less:33

grunt-contrib-qunit 10.2.0 only logs page console messages (tasks/qunit.js:390-412), has no requestfailed handler, and passes when combinedRunEnd.status === 'passed' (qunit.js:453). Every CI job already logs Failed to load resource: net::ERR_FILE_NOT_FOUND (the expected api/info call) and stays green. Less keeps a .css @import verbatim, so reverting this line to ../libs/CodeMirror-4.10.0/... would add one more 404 to the log, and all 39 tests would still pass.

        QUnit.test("codemirror.css is applied", (assert) => {
            const editor = $("<div class='CodeMirror'></div>").appendTo("#qunit-fixture");
            assert.equal(editor.css("height"), "300px", "structure.less imports codemirror.css");
        });

Pin: .CodeMirror { height: 300px } comes only from codemirror.css, so a broken import turns this case red. It must run after css/structure.css has loaded.


suggestion (non-blocking): The Chrome the suite runs on is not pinned.

openig-ui/package.json:16

grunt-contrib-qunit 10.2.0 and qunit 2.26.0 are pinned exactly, but puppeteer arrives through the plugin's "puppeteer": "^24.0.0", and openig-ui has no lockfile. Puppeteer's postinstall chooses the Chrome build. Two fresh installs, such as the PR run and a later release run, can test against different browsers, so the suite can change colour without a commit. At BASE, phantomjs-prebuilt always fetched PhantomJS 2.1.1.

    "puppeteer": "24.43.1",

Or: pin it as an exact devDependency at whatever 24.x you verify. npm dedupes the plugin's ^24 range onto it.


suggestion (non-blocking): The root pom still downloads qunit 1.15.0 js and css, which nothing uses any more.

pom.xml:799-812

This PR removed the only consumers: openig-ui's two qunit dependencies and copy-dependencies-qunit. git grep -i qunit over *.xml/*.yml finds no other reference. The two artifactItems are now dead configuration that still costs a cdnjs fetch on every build. Delete both.

…t phase

grunt-contrib-qunit 1.3 / PhantomJS could no longer load the application, so the
"qunit" task had been commented out of the build since 2019 and the JavaScript
tests gave no signal.

- grunt-contrib-qunit 10 (Puppeteer) with QUnit 2.26 from npm instead of the
  qunit 1.15 Maven artifact; Chrome runs with --allow-file-access-from-files as
  the app fetches translations and templates through XHR from file://
- new "grunt test" task bound to the Maven test phase: "mvn verify" fails on a
  failing JS test, -DskipTests skips the suite
- harness: register the EVENT_APP_INITIALIZED listener before starting the app
  and load the suites only afterwards (their dependencies resolve through the
  require.config of main.js); start the app from the unbundled
  target/compose/main.js, as the optimised bundle re-executes jQuery and the
  other libraries the test page has already loaded
- tests migrated to the QUnit 2 API; PhantomJS Function.prototype.bind polyfill
  removed
- tests: assert.throws validates the TransformServiceException type instead of
  an undefined "expected" (the constructor was called without new); the unknown
  filter type test now expects "unknownFilterType"
- structure.less: CodeMirror stylesheet path follows the libs/codemirror5-4.10.0
  location main.js was switched to in OpenIdentityPlatform#36

Fixes OpenIdentityPlatform#175
The application loads ViewManager asynchronously while navigating to its
first view, so the synchronous require() in QUnit.testStart only worked
when the 500 ms delay before QUnit.start() happened to be long enough;
on the windows-latest/Java 11 runner it was not.
…xtures

- Rebuild grunt-contrib-qunit's own Chrome arguments (CHROMIUM_FLAGS, else
  --no-sandbox under CI) next to --allow-file-access-from-files, so
  CHROMIUM_FLAGS=--no-sandbox works again when building as root.
- Skip the grunt-test execution on -Dmaven.test.skip=true through a profile;
  <skip> on the execution would replace ${skip.grunt}.
- Pin puppeteer 24.43.1, so a fresh install always tests the same Chrome.
- Leave out only the attribute under test in the 'no name' / 'no baseURI'
  fixtures, so each fails if its check in RouteModel.validate() is removed.
- Check that every @import of the built structure.css loads; a broken path
  (as the CodeMirror one fixed here) never failed the page.
- Drop the qunit 1.15.0 artifacts from the root pom; nothing uses them.
@vharseko
vharseko force-pushed the ui-qunit-headless-chrome branch from bdd58ba to 7ee6158 Compare October 22, 2026 12:08
@vharseko

Copy link
Copy Markdown
Member Author

The branch is rebased onto the current master. The only conflict was the bad,id assertion from #174 in OpenIGValidatorsTests.js: it is kept, and QUnit.start() after it is dropped.

CHROMIUM_FLAGS (issue): fixed as suggested. chromiumArgs now adds CHROMIUM_FLAGS if set, else --no-sandbox under CI, after --allow-file-access-from-files. With CHROMIUM_FLAGS="--no-sandbox --disable-gpu", Chrome gets both flags and the suite is green. The README now tells a build running as root to set CI=true or CHROMIUM_FLAGS=--no-sandbox.

-Dmaven.test.skip=true (question): not intended, so the flag now skips the suite. I used a different mechanism, though. <skip>${maven.test.skip}</skip> on the execution would replace ${skip.grunt}. Then -Dskip.grunt=true would skip grunt build but still run grunt test against an unbuilt target/www. Instead, a maven-test-skip profile in openig-ui/pom.xml turns on with -Dmaven.test.skip=true and sets <skip>true</skip> on grunt-test only. Checked: with -DskipTests or -Dmaven.test.skip=true, grunt-test logs Skipping execution., and -Dskip.grunt=true still skips both executions.

Name / baseURI fixtures: fixed as suggested. Each model now leaves out only the attribute under test. Checked against the built copy: with the name branch, and then the baseURI branch, removed from RouteModel.validate(), the matching test fails.

CodeMirror stylesheet: covered, but by a different test than the one suggested. ThemeManager appends the theme <link>s without waiting for them to load, and the suites start after a fixed _.delay(500). A css("height") check would therefore depend on timing, and also on CodeMirror's default height. The new StylesheetTests instead fetches the built css/structure.css and checks that every @import in it loads (all 9 do). It does not depend on when the stylesheets finish loading, and it covers every import, not only CodeMirror. With the old CodeMirror-4.10.0 path in the built structure.css, css/../libs/CodeMirror-4.10.0/lib/codemirror.css is loaded fails (grunt exit 6).

Puppeteer: pinned to 24.43.1, the 24.x the suite was verified on (and the newest 24.x). npm dedupes the plugin's ^24.0.0 onto it.

Root pom qunit 1.15.0: both artifactItems removed.

mvn -pl openig-ui -am verify: 40 tests completed, 0 failed (the 39 plus StylesheetTests). The PR description is updated to match.

@vharseko
vharseko requested a review from maximthomas October 22, 2026 12:09

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The suite now runs for real, and the new checks were measured to bite.

  • StylesheetTests turns red when structure.less gets the old ../libs/CodeMirror-4.10.0 import back (1 failed at StylesheetTests.js:47; 40/40 green otherwise).
  • The maven-test-skip profile does what README.md:61 promises: under -Dmaven.test.skip=true the grunt-test execution logs "Skipping execution."
  • The transformError(...) validators (TransformServiceTests.js:35) replace the undefined expectations, which is what exposed the unknownFilterType mismatch.

@vharseko
vharseko merged commit a01df62 into OpenIdentityPlatform:master Oct 22, 2026
14 checks passed
@vharseko
vharseko deleted the ui-qunit-headless-chrome branch October 22, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Build, CI/CD, and GitHub Actions changes dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation javascript Pull requests that update Javascript code tests Adds or updates tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openig-ui: the QUnit runner is effectively dead (grunt qunit / PhantomJS times out, task commented out of build)

2 participants