[kfir4444]

What

Fourteen fixes for failures that appeared when running ARC at scale on a machine-learned potential (UMA through the ASE adapter) next to a DFT reference route. The benchmark was 116 C/H/O species and 25 H-abstraction reactions, thermo and rates, on a PBS cluster with several ARC runs at once. Each fix is its own commit with a test where one is possible. Most are in the ASE adapter; the rest are general (the job cap, restarts, the IRC check, rotor-scan checks, Arkane species files).

Changes

ASE adapter (ase_adapter.py, scripts/ase_script.py)

1. Local queue jobs had no script. On a local server that is a queue (PBS here), upload_files() copies only check files, but the submit script runs $JOB_DIR/ase_script.py. Every ASE queue job failed. set_files() now copies the script into the job directory.
2. Rotor scans of ethers and peroxides failed with "part of a ring". rotor_top() built its neighbor list with ASE's default skin=0.3, which ASE adds to every cutoff, so heavy atoms counted as bonded up to 2.42 Å apart: the carbons of C–O–C (2.33 Å), or C and the far O of C–O–O. Now skin=0.0.
3. Scans of a torsion through a near-linear chain failed (FixInternals.adjust_positions did not converge). ARC defines, e.g., the OH rotor of HOC≡COH as H–O–C≡C, with O–C≡C at 175°, and a dihedral through three collinear atoms is undefined. well_defined_torsion() replaces such an end atom by the first atom beyond it that is off the pivot axis. That measures the same rotation about the same bond. If one side is linear to its end, the scan raises, since no torsion exists. Gaussian scanned the same torsions.
4. Every MLIP TS failed ARC's IRC check. There were two problems in the IRC job:
   • It reported the TS as its final geometry. ARC builds each IRC species from that geometry and optimizes it, and a minimizer started at a saddle stays there, so both "endpoints" were the TS. The final geometry is now the last point of the path.
   • Sella's inner loop converges each step to min(fmax, 0.01) eV/Å. At the opt default of 0.001, it failed on the first step off UMA's CH₄ + OH saddle (IRCInnerLoopConvergenceFailure), because that is the level of an MLIP's force noise. The IRC now has its own irc_fmax (default 0.01) and keep_going=True, and a path cut short is kept with a warning. The endpoints only seed ARC's IRC species, which ARC optimizes anyway.

Scheduler and checks

5. A scan-troubleshooting crash ended the whole run. For an inconsistent ASE scan, ARC's "freeze" remedy called ics_to_scan_constraints(), which raises NotImplementedError for YAML outputs. trsh_scan_job() now raises a TrshError, which the Scheduler already handles by giving up that rotor.

6. Correct IRC endpoints of O-atom reactions were rejected. For O(³P) + H₂, the endpoints were H₂···O and OH···H, but ARC perceives a lone O as a closed-shell singlet, which is not isomorphic with the reactant. The bond-list fallback then failed too, because it compares bonds under one atom map, while the two H atoms of H₂ are interchangeable. Isolated atoms now match by element. The existing bond-list test now disables the isomorphism tier, so that it still tests the bond-list tier.

7. max_simultaneous_jobs counted the user's other jobs. The cap compared every job ID in the queue with the limit. A second ARC run of the same user (or any other job) could starve a run forever, since the waiting loop doesn't process the run's own finished jobs meanwhile. It now counts only the run's own job IDs (own_job_ids()).

8. A restarted run crashed on TS conformer jobs still in flight. run_ts_conformer_jobs() respawned them, reset job_dict[label]['conf_opt'], and save_restart_dict() then raised a KeyError for the restored job names. It now returns if conformer jobs of that TS are already running.

9. The end of a run crashed when a TS failed its IRC check. The Scheduler deletes that TS's IRC species from its own list only, so save_project_info_file() raised a KeyError, which skipped Arkane and output.yml. ARC now drops those species from its list too.

10. Arkane ignored the multiplicity ARC wrote. The species template wrote spinMultiplicity, but Arkane reads spin_multiplicity from species files. Logs that state their own multiplicity were unaffected, but an output without it (e.g., a YAML output) had none.

11. ARC accepted rotor scans that stopped partway. A Gaussian scan that ends on an "Internal coordinate error" is invalidated in check_scan_job(). But for more than five points, scan_quality_check() then ran on the points the scan reached and overwrote that verdict, so a smooth partial scan became a successful rotor (e.g., HOC≡COH, stopped at point 12 of 46). Arkane rejects such a log, and since ARC runs one Arkane job for all species, one of them left the whole project without thermo; on a TS, without rates. The quality check now runs only on a scan that is not already invalid.

12. A restarted run crashed generating conformers for an IRC species. On restart, an IRC endpoint species whose opt had not finished went through run_conformer_jobs() like any species, and ARC often has no 2D graph for these complexes (AttributeError: 'NoneType' object has no attribute 'copy' in cheat_sheet()). IRC species take their geometry from the IRC path, so they now get no conformers.

13. A flat rotor scan crashed the run. A scan whose 46 energies were identical (nothing turned: the rotating group lay on the axis) has no peaks or valleys, and determine_rotor_symmetry() raised on min() of the empty peak list. Such a scan is now invalidated before its symmetry is sought.

14. A restarted run re-optimized species it had already optimized and scanned. Two paths spawned an optimization from a species' conformers again:

    • At start, schedule_jobs() optimizes any species that has conformer energies but no geometry. A restart file saved while conformer jobs ran still holds the force-field energies of those that had not reported, so the restart optimized the force field's best conformer while the restored conformer jobs were still running.
    • When those jobs ended, ARC determined the conformer again and spawned another optimization. Conformer job names left in running_jobs repeated this on later restarts (butanal's two, across three restarts), after the rotor scans of the first optimization had run.

    In the benchmark, butanal and ethanol ended on a different conformer from the one their rotors were scanned on (0.56 and 0.16 kJ/mol apart), so Arkane combined the scans of one conformer with the frequencies of another. Both paths now go through optimize_most_stable_conformer(). It spawns nothing while conformer jobs of the species are running, or once the species has an optimization (or composite job) running or converged. Troubleshooting that re-runs conformers deletes the species' jobs first, which clears both conditions.

Checks

• Production: these fixes ran on the zeus benchmark (three routes: UMA-S 1.2, UMA-M 1.1, ωB97X-D/def2-TZVP; DLPNO-CCSD(T)-F12 single points).
  • With 4, both UMA TSs of a CH₄ + OH / CH₄ + H test pass the IRC check on their first guess. Before it, both had used up every guess.
  • With 6, all four of UMA's O-atom TSs pass: O + H₂, CH₄, C₂H₆ and CH₂O.
  • 5 and 13 each replaced a crash of a 116-species run, and 12 a crash on restarting one.
  • With 11, the DFT route got its thermo and rates back: six such scans had blocked Arkane for every species.
  • 14 was found after the runs: the DFT route's butanal and ethanol rotor scans were of a different conformer from their frequencies, and the archived restart files and logs show the extra optimizations. It has not run in production yet.
• Tests, locally: ase_test.py, trsh_test.py, checks/ts_test.py, scheduler_test.py, main_test.py and statmech/arkane_test.py pass (scheduler_test.py run with -n0; under xdist a few of its tests share files and fail at random, also without this PR's last commit). The exceptions are three tests that already fail on main in this environment, because they run Arkane: test_check_rxn_e0, test_compute_rxn_e0, test_run_statmech_using_molecular_properties.

Tests

• test_set_files_puts_the_script_in_a_local_queue_job_directory (1)
• test_rotor_top_does_not_bond_geminal_heavy_atoms: dimethyl ether (2)
• test_well_defined_torsion: HOC≡COH, ethane, and a linear acyl with no torsion (3)
• test_an_irc_job_ends_at_the_end_of_its_path: a stand-in Sella; checks the endpoint, irc_fmax and keep_going (4)
• test_trsh_scan_job_cannot_freeze_in_a_software_without_constraints (5)
• test_check_irc_matches_an_isolated_atom_by_element: the UMA O + H₂ endpoints (6)
• test_check_scan_job_keeps_an_internal_coordinate_error_invalid (11)
• test_run_conformer_jobs_skips_irc_species: an IRC species gets no conformers, an ordinary one still does (12)
• test_check_scan_job_invalidates_a_flat_scan (13)
• test_optimize_most_stable_conformer_once: conformer jobs still running, an optimization running or converged, a composite job running, and the ordinary case (14)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.12%. Comparing base (6da9d99) to head (b62a524).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1072      +/-   ##
==========================================
- Coverage   67.15%   67.12%   -0.04%     
==========================================
  Files         123      123              
  Lines       43675    43706      +31     
  Branches    11153    11161       +8     
==========================================
+ Hits        29332    29338       +6     
- Misses      11203    11224      +21     
- Partials     3140     3144       +4     

Flag	Coverage Δ
functionaltests	67.12% <ø> (-0.04%)	⬇️
unittests	67.12% <ø> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 11 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.