Repository navigation
[#175] Run the openig-ui QUnit suite in headless Chrome during the Maven test phase - #179
Conversation
maximthomas
left a comment
There was a problem hiding this comment.
praise: The suite is back as a real gate, and the fixes it surfaced are the right ones.
- The
grunt-testexecution in thetestphase makesmvn verifyfail on a red QUnit test: frontend-maven-plugin 1.6 turns grunt's non-zero exit into aMojoFailureException. The whole build-maven matrix (ubuntu/macos/windows) runs the 39 tests green. - The
transformError(...)validator replacesassert.throws(..., transformService.TransformServiceException("invalidModel"), ...), whoseexpectedwasundefined. The tighter check exposed the wronginvalidModelexpectation in the unknown-filter-type test. config.jscallsrun(MockServer.instance)beforerequire(["main"]), so theEVENT_APP_INITIALIZEDlistener cannot lose the race against the bundle.
issue (non-blocking): Setting puppeteer.args drops grunt-contrib-qunit's CHROMIUM_FLAGS override, not only its CI-only --no-sandbox.
openig-ui/Gruntfile.js:53-56
grunt-contrib-qunit 10.2.0 builds its arguments from (process.env.CHROMIUM_FLAGS || (process.env.CI ? '--no-sandbox' : '')).split(' ') (tasks/qunit.js:257-259), and Object.assign replaces them with options.puppeteer.args (qunit.js:276-281). With this PR, mvn verify run as root with CI unset (a Maven Docker image, a local container) launches Chrome without --no-sandbox, and Chrome refuses to run as root. The plugin's documented workaround, CHROMIUM_FLAGS=--no-sandbox, now has no effect, so the only ways out are CI=true or -DskipTests. Not run here (no Docker daemon). No workflow in this repo builds as root without CI.
// Setting puppeteer.args replaces the arguments grunt-contrib-qunit would build itself
// (CHROMIUM_FLAGS, else --no-sandbox under CI), so they are re-created here.
chromiumArgs = ["--allow-file-access-from-files"].concat(
(process.env.CHROMIUM_FLAGS || (process.env.CI ? "--no-sandbox" : "")).split(" ").filter(Boolean)
);question (non-blocking): Is a build that passes only -Dmaven.test.skip=true meant to keep running the JS suite?
openig-ui/pom.xml:96-104
frontend-maven-plugin 1.6 binds the grunt goal's skipTests to ${skipTests} and skip to ${skip.grunt}, and never reads maven.test.skip. So mvn verify -Dmaven.test.skip=true now downloads and launches headless Chrome and fails on a red JS test, where before nothing in openig-ui ran a browser. No workflow in this repo passes that flag on its own. It is a Minor either way. If it is not intended, honour the flag on this execution:
<execution>
<id>grunt-test</id>
<goals>
<goal>grunt</goal>
</goals>
<phase>test</phase>
<configuration>
<arguments>test</arguments>
<skip>${maven.test.skip}</skip>
</configuration>
</execution>plus <maven.test.skip>false</maven.test.skip> in the module's <properties>. Note that <skip> also replaces ${skip.grunt} for this execution. Or, if it is intended: add a README line saying only -DskipTests skips the JS suite.
suggestion (non-blocking): The 'no name' and 'no baseURI' tests cannot fail if the check they name is removed.
openig-ui/src/test/js/tests/TransformServiceTests.js:57, :71
RouteModel.validate() checks id, then name, then baseURI, and returns on the first failure (RouteModel.js:65-77). The 'no baseURI' fixture {id: "modelID", url: ""} leaves name at its default "", and url is not an attribute validate() reads. The model is therefore invalid because of the name, and deleting the baseURI branch keeps the test green. The 'no name' fixture has the mirror gap: its default baseURI: "" keeps it red-proof if the name check is removed. errorType is invalidModel for every branch, so the new validator cannot tell them apart.
const applicationWithEmptyName = new RouteModel({
id: "modelID",
name: "",
baseURI: "http://www.example.com"
});
// ...
const applicationWithEmptyBaseUrl = new RouteModel({
id: "modelID",
name: "routeName",
baseURI: ""
});Pin: with one missing attribute per fixture, deleting the baseURI (or name) branch of validate() makes transformRoute pass its isValid() check (TransformService.js:247) and return a config, so assert.throws turns red.
suggestion (non-blocking): Nothing pins the CodeMirror stylesheet path fix, because failed page loads never fail the QUnit run.
openig-ui/src/main/resources/css/structure.less:33
grunt-contrib-qunit 10.2.0 only logs page console messages (tasks/qunit.js:390-412), has no requestfailed handler, and passes when combinedRunEnd.status === 'passed' (qunit.js:453). Every CI job already logs Failed to load resource: net::ERR_FILE_NOT_FOUND (the expected api/info call) and stays green. Less keeps a .css @import verbatim, so reverting this line to ../libs/CodeMirror-4.10.0/... would add one more 404 to the log, and all 39 tests would still pass.
QUnit.test("codemirror.css is applied", (assert) => {
const editor = $("<div class='CodeMirror'></div>").appendTo("#qunit-fixture");
assert.equal(editor.css("height"), "300px", "structure.less imports codemirror.css");
});Pin: .CodeMirror { height: 300px } comes only from codemirror.css, so a broken import turns this case red. It must run after css/structure.css has loaded.
suggestion (non-blocking): The Chrome the suite runs on is not pinned.
openig-ui/package.json:16
grunt-contrib-qunit 10.2.0 and qunit 2.26.0 are pinned exactly, but puppeteer arrives through the plugin's "puppeteer": "^24.0.0", and openig-ui has no lockfile. Puppeteer's postinstall chooses the Chrome build. Two fresh installs, such as the PR run and a later release run, can test against different browsers, so the suite can change colour without a commit. At BASE, phantomjs-prebuilt always fetched PhantomJS 2.1.1.
"puppeteer": "24.43.1",Or: pin it as an exact devDependency at whatever 24.x you verify. npm dedupes the plugin's ^24 range onto it.
suggestion (non-blocking): The root pom still downloads qunit 1.15.0 js and css, which nothing uses any more.
pom.xml:799-812
This PR removed the only consumers: openig-ui's two qunit dependencies and copy-dependencies-qunit. git grep -i qunit over *.xml/*.yml finds no other reference. The two artifactItems are now dead configuration that still costs a cdnjs fetch on every build. Delete both.
…t phase grunt-contrib-qunit 1.3 / PhantomJS could no longer load the application, so the "qunit" task had been commented out of the build since 2019 and the JavaScript tests gave no signal. - grunt-contrib-qunit 10 (Puppeteer) with QUnit 2.26 from npm instead of the qunit 1.15 Maven artifact; Chrome runs with --allow-file-access-from-files as the app fetches translations and templates through XHR from file:// - new "grunt test" task bound to the Maven test phase: "mvn verify" fails on a failing JS test, -DskipTests skips the suite - harness: register the EVENT_APP_INITIALIZED listener before starting the app and load the suites only afterwards (their dependencies resolve through the require.config of main.js); start the app from the unbundled target/compose/main.js, as the optimised bundle re-executes jQuery and the other libraries the test page has already loaded - tests migrated to the QUnit 2 API; PhantomJS Function.prototype.bind polyfill removed - tests: assert.throws validates the TransformServiceException type instead of an undefined "expected" (the constructor was called without new); the unknown filter type test now expects "unknownFilterType" - structure.less: CodeMirror stylesheet path follows the libs/codemirror5-4.10.0 location main.js was switched to in OpenIdentityPlatform#36 Fixes OpenIdentityPlatform#175
The application loads ViewManager asynchronously while navigating to its first view, so the synchronous require() in QUnit.testStart only worked when the 500 ms delay before QUnit.start() happened to be long enough; on the windows-latest/Java 11 runner it was not.
…xtures
- Rebuild grunt-contrib-qunit's own Chrome arguments (CHROMIUM_FLAGS, else
--no-sandbox under CI) next to --allow-file-access-from-files, so
CHROMIUM_FLAGS=--no-sandbox works again when building as root.
- Skip the grunt-test execution on -Dmaven.test.skip=true through a profile;
<skip> on the execution would replace ${skip.grunt}.
- Pin puppeteer 24.43.1, so a fresh install always tests the same Chrome.
- Leave out only the attribute under test in the 'no name' / 'no baseURI'
fixtures, so each fails if its check in RouteModel.validate() is removed.
- Check that every @import of the built structure.css loads; a broken path
(as the CodeMirror one fixed here) never failed the page.
- Drop the qunit 1.15.0 artifacts from the root pom; nothing uses them.
bdd58ba to
7ee6158
Compare
|
The branch is rebased onto the current
Name / baseURI fixtures: fixed as suggested. Each model now leaves out only the attribute under test. Checked against the built copy: with the CodeMirror stylesheet: covered, but by a different test than the one suggested. Puppeteer: pinned to Root pom qunit 1.15.0: both
|
maximthomas
left a comment
There was a problem hiding this comment.
praise: The suite now runs for real, and the new checks were measured to bite.
StylesheetTeststurns red whenstructure.lessgets the old../libs/CodeMirror-4.10.0import back (1 failed atStylesheetTests.js:47; 40/40 green otherwise).- The
maven-test-skipprofile does whatREADME.md:61promises: under-Dmaven.test.skip=truethegrunt-testexecution logs "Skipping execution." - The
transformError(...)validators (TransformServiceTests.js:35) replace theundefinedexpectations, which is what exposed theunknownFilterTypemismatch.
Fixes #175.
The
qunittask had been commented out of thebuildtask since 2019 (f270c42a), andgrunt qunititself timed out: grunt-contrib-qunit 1.3 / PhantomJS cannot load the ES2015 app any more. The four suites insrc/test/js/tests(39 tests) gave no signal.Runner
grunt-contrib-qunit1.3.0 → 10.2.0 (headless Chrome via Puppeteer; Chrome is downloaded duringnpm install), QUnit 2.26 from npm instead of thequnit:1.15.0Maven artifact (copy-dependencies-qunitexecution, the two dependencies and the root pom's twoqunitdownload items removed). Puppeteer is pinned to 24.43.1 so that every fresh install tests against the same Chrome (openig-ui has no lockfile; the plugin itself asks for^24.0.0).--allow-file-access-from-files: the test page is opened fromfile://and the app fetches translations and templates through XHR, which Chrome blocks for local files by default (PhantomJS did not).puppeteer.argsreplaces the arguments the plugin would build itself, so they are rebuilt next to it:CHROMIUM_FLAGSif set, else--no-sandboxwhenCIis set. Building as root (e.g. in a container) therefore needsCI=trueorCHROMIUM_FLAGS=--no-sandbox, as with the plugin alone.grunt testtask (copy:qunit+qunit), bound to the Maventestphase as a separate execution rather than put back intobuild:mvn verifyfails on a failing JS test, and-DskipTestsskips the suite through frontend-maven-plugin's built-inskipTestsparameter (same layout as OpenAM'snpm-testexecution, Run frontend Karma tests during the Maven test phase OpenAM#902). The plugin never readsmaven.test.skip, so amaven-test-skipprofile, active on-Dmaven.test.skip=true, skips that execution as well;<skip>on the execution itself would have replaced${skip.grunt}for it.Harness (
config.js/run.js)Two problems only showed up once Chrome actually ran the page:
config.jsrequiredmainand../test/runtogether. The paths the test suites' dependencies resolve through (i18next,backbone, …) are set by therequire.configinsidemain.js; aftergrunt buildthat is the uglified bundle, which loads slower than the tinyrun.js, so the dependencies were resolved before the config existed (Script error for "i18next"). Historicallyqunitran inbuild-devbeforerequirejs, so the race was won by luck. Nowrun.jsregisters theEVENT_APP_INITIALIZEDlistener beforemainis required and loads the suites inside that listener.MockServer/EventManager), so executing the bundle created a second jQuery instance —bootstrap(shim) attached$.fn.modalto the new global while the AMDbootstrap-dialoggot the old one (Cannot read properties of undefined (reading 'Constructor')). The test config therefore starts the app from the unbundledtarget/compose/main.js(paths.main); everything else is still served fromtarget/www.Tests
module()→QUnit.module(),QUnit.asyncTest+QUnit.start()→QUnit.test(all callbacks under test are synchronous). PhantomJSFunction.prototype.bindpolyfill removed.assert.throws(…, transformService.TransformServiceException("invalidModel"), …)called the constructor withoutnew, soexpectedwasundefinedand the assertions only checked that something was thrown. They now use a validator that checksnameanderrorType; this exposed that the "unknown filter type" test expectedinvalidModelwhiletransformFilterthrowsunknownFilterType.RouteModel.validate()stops at the first missing one, so neither test could fail if the check it names was removed. Each fixture now leaves out only the attribute under test.StylesheetTests: every@importof the builtcss/structure.cssmust load. Less keeps a CSS@importas it is and a stylesheet that fails to load does not fail the page, which is how the CodeMirror path below stayed broken unnoticed.Also
structure.lessstill imported../libs/CodeMirror-4.10.0/lib/codemirror.css; Fix build frontend codemirror lib error #36 moved the CodeMirror location tolibs/codemirror5-4.10.0inmain.jsonly, so the stylesheet has been a 404 in the built UI since then (CodeMirror is used for the route JSON view). Path fixed;StylesheetTestsnow covers it.Verification
mvn -pl openig-ui -am verify:grunt-test→40 tests completed, 0 failed, BUILD SUCCESS.-DskipTestsand-Dmaven.test.skip=true:grunt-test→Skipping execution.;-Dskip.grunt=truestill skips bothgruntandgrunt-test.1 failed, grunt exit 6):DataFilterTests;transformError("invalidModel")in the unknown filter type test (Actual: TransformServiceException: UnknownTypeOfFilter);namebranch, then thebaseURIbranch, removed fromRouteModel.validate()→ the matching "no 'name'" / "no 'baseURI'" test fails;CodeMirror-4.10.0path instructure.css→css/../libs/CodeMirror-4.10.0/lib/codemirror.css is loadedfails.--allow-file-access-from-files;CI=true→+ --no-sandbox;CHROMIUM_FLAGS="--no-sandbox --disable-gpu"→+ --no-sandbox --disable-gpu(suite green);CHROMIUM_FLAGSwins overCI, as in the plugin.grunt eslint: 0 errors; the 5 warnings are pre-existing insrc/main/js.file:///api/info(Footer→ServerInfoService.getInfo()with no backend, handled by its.fail) and the resulting "Unknown error" message from the commons REST error handler.