Repository navigation
[#158] Check every attribute before an XML update changes the entry - #188
Conversation
… lastname The tests that pin what a failed update leaves behind relied on update applying attributes before it rejects one (OpenIdentityPlatform#158). They now fail midway through a DOM error instead: a lastname nested in another element of the entry, which removeChild rejects after the entry's own lastname is gone (OpenIdentityPlatform#182). Behaviour is unchanged; the tests keep their count.
…s the entry update() checked each attribute just before replacing it, so a rejected attribute left the attributes before it, and its own old value, removed or replaced in the document, and dispose() saved that half-applied entry. It now checks all attributes first, in the same order and with the same messages, and only then changes the entry. A DOM failure while the values are applied (OpenIdentityPlatform#182) can still leave the entry half changed.
…the required type check Two pins the final review found missing: - failedUpdateLeavesTheFileInAgreementWithMemory gains a row where the attribute that fails midway is the first one applied, so its removal alone must mark the document changed. With firstname applied first, markDirty() after removeChildrenFromElement went unnoticed. - rejectedUpdateLeavesTheEntryAndTheFileAlone gains a row with a wrongly typed required attribute, whose values are read before the type check: the message stays "contains invalid type", and nothing is changed.
…t failing attribute is reported A rejected update with is-deleted given as the String "true" (not updatable and of illegal type) followed by an unsupported attribute must report "is-deleted is not updatable.". Red at 659e8b0 by mtime; red by message when the not-updatable and illegal-type checks are swapped, when the last failure is thrown, or when the attributes are checked in reverse.
vharseko
left a comment
There was a problem hiding this comment.
The split into a check pass and an apply pass does what #158 asks: every check now runs before markDirty() and the first removeChildrenFromElement, and I found no case where a rejected update still changes the entry or the file. Three requests, all in code this PR adds or moves; line references are to ef2aa72.
1. A required attribute is converted twice (L326, L345). For a required attribute, findAttributeValue runs in the required block and again four lines below, on the same attribute, with the same result. Before this PR the second call sat in the apply loop; now the two are next to each other in the check loop. For __PASSWORD__, which the test schema declares required (<xsd:element ref="icf:__PASSWORD__"/> with no minOccurs) and as a GuardedString, every password update decrypts the value twice and leaves two plaintext Strings behind. Keeping the list from the required block and calling findAttributeValue at L345 only when it is still null keeps the check order that row 8 of rejectedUpdateLeavesTheEntryAndTheFileAlone pins: a wrongly typed required value is still rejected by the first call, with the same message.
2. checkedAttributes as a list of Map.Entry (L311, L351, L355-L357). The checked attributes are kept as AbstractMap.SimpleImmutableEntry<String, List<String>>, so the apply loop reads getKey() / getValue() as name and values, and the PR adds the java.util.AbstractMap import for it. Map.Entry also suggests one entry per key, while the loop deliberately keeps a repeated name in order (repeatedAttributeKeepsTheLastValue), which invites a later "simplification" into a Map. A small private holder, such as private static final class CheckedAttribute { final String name; final List<String> values; }, would say what the pair is.
3. firstAndLastNames repeats lastNames (XMLHandlerReloadTests.java#L893-L912). The new helper copies the null-safe read of an attribute from lastNames, which the tests at L366-L378 still use. Extracting the read, for example private static String stringValue(ConnectorObject object, String name) returning null for a missing attribute, and using it in both helpers keeps the two from rendering a missing attribute differently later.
Not raised here, as they are tracked separately or left out of this PR on purpose:
…d name the checked pair - update() reuses the values the required check read instead of converting the attribute again, so a password is decrypted once per update. - The checked attributes are kept as CheckedAttribute (name, values), a list rather than a map, so a repeated name is still applied in order. - XMLHandlerReloadTests reads a possibly missing attribute through one stringValue helper in lastNames and firstAndLastNames.
|
All three are in 6654b15.
Every mutant in the body's test-strength table was re-run on the new code and fails the same tests as before. |
vharseko
left a comment
There was a problem hiding this comment.
All three requests from the previous round are addressed in 6654b15.
updatekeeps the values the required block reads and converts the attribute again only when they are stillnull;findAttributeValuealways returns a list, so thenullcheck cannot mistake an empty result for an unread one. A wrongly typed required value is still rejected by the first call, with the same message, as row 8 pins. That the conversion now happens once is not pinned by a test, which the PR body states; I see no way to pin it short of spying on theGuardedString, so I am fine leaving it.CheckedAttributenames the pair, and its Javadoc records why it is a list rather than a map. TheAbstractMapimport is gone.lastNamesandfirstAndLastNamesboth read throughstringValue.
Nothing new in the fix commit. The branch is based on 659e8b0; since then master changed only OpenICF-xml-connector/pom.xml (#185), with no overlap. build-maven is still running at the time of this review; the approval assumes it comes back green.
…et evicted scripts unload (#196) Fixes #165. `runScriptOnConnector` no longer compiles a new class into the script engine's loader on every call. Each distinct script text is compiled once per engine and kept in a bounded LRU cache. Classes that leave the cache, and every cached class on `release()`, are unregistered so that they can be unloaded. With the default size an engine keeps at most 64 runScriptOnConnector scripts, each with its closures and declared classes, however many calls it serves. - **One class per text, in its own loader.** `ScriptedConfiguration.getScriptOnConnectorClass` compiles a text with a `ScriptOnConnectorLoader`, a child of the engine's loader, which keeps the new classes out of the engine's class cache. Before, every call went through `getGroovyScriptEngine().getGroovyClassLoader().parseClass(...)`. Each class then stayed in the engine loader, and in `GroovyScriptEngine`'s script cache, until `release()`. - **Script-root classes are the engine's.** A class whose source is in the script roots, such as `Helper`, resolves through `GroovyScriptEngine.loadScriptByName`, as it does in the engine's own compilations. A text therefore uses the `Helper` that the operation scripts use. The engine compiles it when it is first needed or after its source changes, and a script-root class still shadows a class of the connector's class loader. Other classes resolve through the engine's loader. - **A cached text runs against the classes it was compiled against.** Its code links a class by name the first time each reference runs. `ScriptOnConnectorLoader` keeps every class the compiler resolved for the text, and the loaders of the script-root classes among them. Its `loadClass` returns a resolved class, or else a class that one of those loaders defined: a nested class, a closure or another class compiled from the same source. Without that, code that first ran after an operation script had recompiled an edited `Helper` linked the new classes. `@CompileStatic` code then failed with `NoSuchMethodError` where it was compiled against an old signature, and with a `LinkageError` (loader constraint violation) where it reached a class such as `Helper.Greeting` only through the return type of a `Helper` method. - **The engine's configuration and compiler lock.** A text compiles with the engine's compiler configuration, the one `getRoots(CompilerConfiguration, GroovyClassLoader)` receives, so with the default imports and with whatever a subclass adds there, a `SecureASTCustomizer` for example. It compiles under the lock that the engine's loader holds while it compiles an operation script or a script-root class, as texts compiled on master: the monitor of the loader's `GroovyClassLoader.sourceCache`. The engine therefore cannot recompile a script-root class while a text resolves the classes it uses, and the customizers of the configuration are not used by two compilations at once. The compiler resolves the class nodes of an `ImportCustomizer` in place. `sourceCache` is protected and the loader is a private class of `GroovyScriptEngine`, so the field is read by reflection; Groovy 2.2.1, 2.4.21, 4.0.22 and 5.0.3 have it and lock it in `GroovyScriptEngine`'s loader. `getImportCustomizer(ImportCustomizer)` and `getImports()` are called once per engine, as on master. - **One compilation at a time, no lock for a cached text.** The configuration monitor is held only to read the engine and the cache, and the cache's monitor only around map operations. Texts compile one at a time, and not while the engine compiles, as on master. A cached text takes no lock and does not wait for a compilation. If two threads compile the same new text, the first one cached wins and the other's classes are unregistered. - **What a text adds or compiles through its loader goes to the engine.** A text's class loader delegates `addURL`, `addClasspath` and `parseClass` to the loader that compiled the text, now its `ScriptOnConnectorLoader`. That loader passes them on to the engine's loader, as on master. The operation scripts and the other texts load classes from a URL that a text adds, and see a class that a text compiles at run time. That compilation takes the engine's compiler lock, and the class resolves `Helper` as the operation scripts do. The text itself compiles through a separate method, into its own loader. - **Bounded, least recently used first.** An engine keeps 64 scripts by default. The system property `org.identityconnectors.framework.groovy.scriptOnConnectorCacheSize` changes the size. It is read when an engine is created. A value that is not a positive integer logs a WARN and falls back to 64. - **Evicted and released classes can unload.** Eviction and `release()` call `InvokerHelper.removeClass` on every class of the script's child loader, closures and declared classes included, outside the cache monitor. The cache is created with the engine once its customizer has succeeded. `release()` clears it last in its `finally`. - **Run from Java.** `ScriptedConnectorBase.runScriptOnConnector` now calls `evaluateScriptOnConnector`, which runs `InvokerHelper.createScript(...).run()` in Java. A Groovy call site caches the metaclass of the last script it ran, and so keeps that class alive. The arguments, the `log` binding and the `catch` blocks are unchanged. - **Without a cache, as on master.** While the customizer runs, the engine is not published and there is no cache. A subclass that overrides `getGroovyScriptEngine()` without calling super has no cache either. A text then compiles through the engine's loader, with the engine's configuration, as on master, and is not cached. The configuration of an engine that a subclass builds, a `SecureASTCustomizer` for example, therefore still applies to its texts. If `GroovyClassLoader.sourceCache` cannot be read, there is no cache either, and a WARN is logged when `ScriptedConfiguration` is loaded. - **A broken script-root class still fails as a compilation error.** The engine compiles it, and its `CompilationFailedException` reaches the caller, as on master. A `ScriptException` from `loadScriptByName`, which a `GroovyScriptEngine` subclass may throw, also fails the text's compilation with a `CompilationFailedException`, not a `GroovyBugError`. Behaviour changes: - a script's class name is stable while its text stays cached; - static fields, metaclass changes and classes declared in a script survive between calls with the same text; - classes declared in one text are no longer visible to other texts or to the operation scripts, and a text can no longer replace a script-root class for everyone else; - a text's class loader answers `getURLs`, `findResource`, `findResources`, `getResourceLoader`, `setResourceLoader`, `getLoadedClasses` and `clearCache` for its `ScriptOnConnectorLoader`, not for the engine's loader. `getResource` and `getResourceAsStream` give the same answers as on master; - a cached script runs against the classes it was compiled against, script-root classes such as `Helper` and the classes compiled with them included, until it is evicted or the configuration is released. This holds even after their source changes on disk and the operation scripts move to the new classes. Master recompiled the text on every call and so picked up the change; a newly compiled text still does; - a class evicted while another thread runs it loses any metaclass changes it made. Its metaclass is rebuilt on its next dynamic call, after which the class is only softly held; - a compilation that finishes after `release()` goes into the orphaned cache, so its class is only softly held. Not changed: - operation scripts (`loadScript`, `evaluate`) are still compiled and cached by the engine; - the customizer lifecycle and the exceptions `runScriptOnConnector` maps. ## Test strength All in `ScriptOnConnectorCacheTest`. The ed25f73, ce3a86f, 37b50d1 and 21f99b4 cells are runs of the new tests against the production code of those earlier heads of this PR. | arm | case | evidence | |---|---|---| | one class per text | `#testSameTextReusesTheCompiledClass` | compile-red at c328553 | | classes stay out of the engine loader | `#testScriptClassStaysOutOfTheEngineLoader` | compile-red at c328553; mutants "child loader's parent is `getParentLoader()`, not the engine's loader" and "the text compiles through the `parseClass` it passes on to the engine" killed | | the text's configuration has the default imports | `#testScriptSeesTheDefaultImports` | mutant "text configuration without the import customizer" killed | | script-root classes resolve | `#testScriptSeesTheClassesInTheScriptRoots` | kept behaviour | | a text shares script-root classes with the operation scripts | `#testScriptSharesScriptRootClassesWithOperationScripts` | red at ed25f73; mutants "no engine resolver" and "default resolver first" killed | | … also after a source edit, the text compiled first | `#testNewScriptSharesTheEditedScriptRootClassWithOperationScripts` | red at ed25f73; both resolver mutants killed | | script roots shadow the connector's classes | `#testScriptRootClassShadowsAClassOfTheConnectorLoader` | red at ed25f73; both resolver mutants killed | | a cached text keeps the class it was compiled against | `#testCachedScriptKeepsTheScriptRootClassItWasCompiledAgainst` | kept behaviour, green at ed25f73 | | … also where its code first runs after the engine recompiled the class, nested classes included | `#testCachedScriptLinksTheScriptRootClassesItWasCompiledAgainst` | red at ce3a86f (`hi hi`); mutant "no `loadClass` override" killed | | … a class of a script-root source that the text names without that source's own class | `#testCachedScriptLinksTheEngineClassesItWasCompiledAgainst` | kept behaviour, green at 37b50d1; mutants "no lookup of the resolved classes" and "keep only the script-root classes" killed | | … statically compiled code, with classes reached only through the methods of a script-root class | `#testCachedStaticScriptLinksTheClassesCompiledWithAScriptRootClass` | red at 37b50d1 (`LinkageError`, loader constraint violation); mutants "no lookup in the script-root loaders" and "nested classes only" killed | | texts compiled together share a script-root class | `#testScriptsCompiledTogetherShareTheScriptRootClasses` | red at ed25f73; mutant "no engine compiler lock" killed (five rounds of four threads: a race, so a likely kill, not a certain one) | | a text and an operation script that load a script-root class together share it | `#testTextAndOperationScriptThatLoadAScriptRootClassTogetherShareIt` | red at 37b50d1 (two copies of `Helper`); mutant "no engine compiler lock" killed | | texts compile together on a new engine | `#testScriptsCompileTogetherOnANewEngine` | red at ed25f73 (`unable to resolve class org$forgerock$openicf$misc$scriptedcommon$ICFObjectBuilder`); mutant "no engine compiler lock" killed, with the engine's configuration shared | | what a subclass adds to the engine's configuration in `getRoots` applies to texts | `#testScriptCompilesWithWhatASubclassAddsToTheEngineConfiguration` | red at 37b50d1 (`unable to resolve class RootsList`); mutant "a new configuration built as the engine's" killed | | binding reaches the script | `#testScriptSeesItsBinding` | compile-red at c328553 | | a URL that a text adds reaches the operation scripts | `#testUrlThatATextAddsToItsLoaderReachesTheOperationScripts` | red at 37b50d1 (`ClassNotFoundException: drv.Driver`); mutant "no `addURL` override" killed | | a class that a text compiles at run time reaches the operation scripts | `#testClassThatATextParsesReachesTheOperationScripts` | red at 21f99b4 (`MissingPropertyException: No such property: Parsed`); mutant "the `parseClass` override takes the engine's lock but compiles into the text's loader" killed | | … and waits for the engine's compilation | `#testClassThatATextParsesWaitsForTheEngineCompilation` | red at 21f99b4 (`parseClass` returns while the engine compiles an operation script) | | `release()` unregisters and recompiles | `#testReleaseUnregistersTheCachedClasses` | compile-red at c328553 | | … also when the release closure throws | `#testReleaseUnregistersTheCachedClassesWhenTheReleaseClosureThrows` | mutant "clear the cache after the `try`/`finally`" killed | | customizer before publish | `#testCustomizerCanRunAScriptOnConnector` | mutant without the null-cache guard killed (`NullPointerException`) | | … and its script is not cached | `#testScriptRunByTheCustomizerIsNotCached` | mutant "cache created before the customizer runs" killed | | an engine built by a subclass compiles with its own configuration | `#testEngineBuiltByASubclassCompilesTheScriptWithItsOwnConfiguration` | red at ce3a86f (`unable to resolve class EngineList`); mutants "no cache: compile with the child loader anyway" and "no null-cache guard" killed | | a hit does not wait for a compile | `#testCachedScriptDoesNotWaitForAnotherCompilation` | mutants `synchronized getScriptOnConnectorClass` and "cache lookup under the engine's compiler lock" killed (`TimeoutException`) | | size property: name, default, `5`, ` 7 `, `0`, `-3`, `many` | `#testCacheSizeComesFromTheSystemProperty` | compile-red at f4b1286 | | LRU, not insertion order | `#testLeastRecentlyUsedScriptIsEvicted` | mutant `accessOrder` `false` killed | | eviction unregisters closures too | `#testEvictedScriptClassesAreUnregistered` | mutants "unregister only the main class" and "no unregister in `put`" killed | | an evicted class unloads | `#testEvictedScriptClassCanBeUnloaded` | same two mutants killed | | the connector reuses the class | `#testConnectorReusesTheCompiledScript` | red at a40030c | | arguments and `log` binding | `#testConnectorScriptSeesItsArgumentsAndTheLogger` | kept behaviour, green at a40030c; mutants "`log` is `Log.getLog(Object.class)`" and "`evaluateScriptOnConnector` rebinds `log` to the script's class" killed | | a connector-run class unloads once evicted | `#testScriptRunByTheConnectorCanBeUnloadedOnceEvicted` | red at a40030c; mutant Groovy-side `createScript(...).run()` killed (the call site keeps the class) | | broken script-root class → compile error | `#testBrokenScriptRootClassFailsAsACompilationError` | red at a872c5b (`GroovyBugError`); both resolver mutants killed | | … also from the connector | `#testConnectorReportsABrokenScriptRootClassAsACompilationError` | red at a872c5b (`GroovyBugError`); both resolver mutants killed | | a `ScriptException` from the engine → compile error | `#testScriptExceptionFromTheEngineFailsAsACompilationError` | red at 37b50d1 (`GroovyBugError`); mutants "`ConnectorException.wrap`" and "no cause" killed | The two unload tests run `System.gc()` for up to about 5 s and return as soon as the class is gone. They rely on the JVM's default soft-reference policy, which surefire leaves unchanged. Not pinned by a test: - the first-wins rule when two threads compile the same new text; - the WARN for an invalid size, and reading the size once per engine; - unregistering closure classes on `release()` (eviction pins the shared `unregister()`); - the fallback without a cache, and its WARN, when `GroovyClassLoader.sourceCache` cannot be read; - that `loadClass` takes from the loader of a script-root class only the classes that loader defined. A mutant that takes any class the loader returns passes: the two differ only for a class that the script-root class linked before the engine replaced it. `mvn -pl OpenICF-groovy-connector,OpenICF-ssh-connector verify` on JDK 26: - groovy: `Tests run: 179, Failures: 0, Errors: 0, Skipped: 42`. The 42 skipped tests are the five `*SampleTest` classes, which need a private configuration; - ssh: `Tests run: 135, Failures: 0, Errors: 0, Skipped: 120`. The skipped sample tests need a live host. Master is at 4648e38 now. #188, #191, #194 and #197 touch none of these files. The branch merges into it cleanly, and the groovy module then runs `Tests run: 210, Failures: 0, Errors: 0, Skipped: 42`.
XMLHandlerImpl.updatenow checks every attribute before it changes the entry, so a rejected update leaves both the document and the file as they were.Before,
updatechecked and applied one attribute at a time. A check that failed on a later attribute left the earlier ones already replaced, and the single-valued check ran after the attribute's old values were removed. The framework callsdispose()in afinallyblock, anddispose()saved the half-applied entry.update(uid, {firstname=CHANGED, is-deleted=true})is-deleted is not updatable.; the file is saved withfirstname=CHANGED(depending on the set's iteration order)ageage contains values of illegal type;firstname=CHANGEDsavedaddAttributeValues(uid, firstname += F1b)onfirstname=F1Data field: firstname is not multivalued can not have more than one value; the entry is saved withoutfirstnamefirstnamestaysF1update(uid, {lastname=[A, B]})lastnamedispose()does not rewrite the file.Attribute.equalscompares the values too) is still applied in order, the last value winning.Three tests pinned what a rejected update left behind, by relying on the old behaviour:
XMLHandlerReloadTests#failedUpdateLeavesTheFileInAgreementWithMemory,XMLHandlerIndexTests#failedRenameLeavesNoStaleName(nowrenameFailingMidwayLeavesNoStaleName), and the failed-rename step ofXMLHandlerIndexTests#indexAgreesWithXQueryAfterChanges. They now fail midway on a DOM error instead: alastnamenested in another element of the entry, whichremoveChildrejects after the entry's ownlastnameis gone (#182). They still check that the file agrees with memory and the index with the document after such a failure.Not fixed here: a DOM failure while the values are applied (#182: a nested entry or value, a lookup by qualified name) still leaves the entry half changed, and
dispose()saves it.Test strength
XMLHandlerImpl.java:309-355all checks before the first changeXMLHandlerReloadTests#rejectedUpdateLeavesTheEntryAndTheFileAlone, 9 rows (the message, the file's mtime and bytes, then the entry in memory);XMLHandlerIndexTests#rejectedRenameKeepsTheOldName;XMLConnectorTests#failedAddAttributeValuesKeepsTheOldValue:314unsupportedrejectedRenameKeepsTheOldNameremoveChildrenFromElement:321not updatable:328required without values:331required blank:341illegal typefindAttributeValuethen throws "contains invalid type"):351single-valuedfailedAddAttributeValuesKeepsTheOldValue:326-342a required attribute's values are read before the type checklastname=Integer5: "lastname contains invalid type. Value(s) should be of type java.lang.String"):311-355checks in the given order, the first failure thrownis-deleted= the String "true", both not updatable and of illegal type, then an unsupported attribute: "is-deleted is not updatable."):363markDirty()per applied attributeupdateIsInTheFileWhenTheLastUserLeaves,repeatedAttributeKeepsTheLastValue, both rows offailedUpdateLeavesTheFileInAgreementWithMemoryupdateWithNoAttributesLeavesTheFileAloneupdateWithNoAttributesLeavesTheFileAlonefailedUpdateLeavesTheFileInAgreementWithMemory,updateWithNoAttributesLeavesTheFileAlonefailedUpdateLeavesTheFileInAgreementWithMemoryrow 2 (the failing attribute is the first one applied)removeChildrenFromElement:354checked attributes kept in order, repeats includedrepeatedAttributeKeepsTheLastValue:380rekeyedinfinallyrenameFailingMidwayLeavesNoStaleName,indexAgreesWithXQueryAfterChanges("failed rename midway")Not pinned:
:346, the reuse of a required attribute's values. Converting the attribute a second time returns the same list, so no test tells the two apart.mvn -pl OpenICF-xml-connector verify: 242 unit tests (227 + 15) and 1 IT, green on JDK 26 and JDK 11.Fixes #158