Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,22 @@ jobs:
- name: Build
run: cmake --build build -j "$(nproc)"

- name: Python environment
# --system-site-packages: use the apt NumPy/matplotlib/pytest that the
# compiled module was built against; add gemmi (test references) and
# the docs tools.
run: |
python3 -m venv --system-site-packages .venv
.venv/bin/pip install --quiet -r tests/requirements.txt -r docs/requirements.txt

- name: Python tests (pytest)
env:
DIFFUSE_PYTHON_PATH: ${{ github.workspace }}/build/python
DIFFUSE_REQUIRE_PYTHON: "1"
MPLBACKEND: Agg
# DISCUS_SUITE is not set here, so the legacy macro tests inside
# pytest are skipped; the next step runs them in parallel.
run: python3 -m pytest tests/python -v --junitxml=pytest-results.xml
run: .venv/bin/python -m pytest tests/python -v -rxs --junitxml=pytest-results.xml

- name: Legacy macro tests
# Tests tagged mc run with "seed 1234" on one thread (repeatable). A
Expand All @@ -73,12 +81,7 @@ jobs:
DIFFUSE_PYTHON_PATH: ${{ github.workspace }}/build/python
MPLBACKEND: Agg
# Warnings are errors, so broken docstrings or links fail the build.
# --system-site-packages: use the apt NumPy/matplotlib that the
# compiled module was built against.
run: |
python3 -m venv --system-site-packages .venv-docs
.venv-docs/bin/pip install --quiet -r docs/requirements.txt
.venv-docs/bin/python -m sphinx -W --keep-going -b html docs docs/_build/html
run: .venv/bin/python -m sphinx -W --keep-going -b html docs docs/_build/html

- name: Upload documentation
if: success() || failure()
Expand Down
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ targets `claude-python-app`.
FC=gfortran CC=gcc CXX=g++ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
-DDIFFUSE_PYTHON=ON -DPGPLOT_DIR=/usr/lib/pgplot5
cmake --build build -j
pip install -r tests/requirements.txt # gemmi
DIFFUSE_PYTHON_PATH=build/python python3 -m pytest tests/python
python3 tools/run_legacy_tests.py --exe build/suite/prog/discus_suite -j 4 --tags fast,mc
```
Expand Down
58 changes: 58 additions & 0 deletions migration/WP0_Test_Baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -436,3 +436,61 @@ whole Python process, so after any `diffuse` call, relative paths in the
user's script pointed somewhere else. `Session.run()` now restores the
caller's working directory (test `test_working_directory_kept`).

### 0.6 + 0.7 for WP3 and WP4 – structures and single-crystal Fourier

Golden data and coverage for the first two work packages that will be
rewired, checked against independent references wherever possible (new
`tests/requirements.txt`: gemmi).

`tests/python/test_wp3_structures.py` (26 tests):

* Space-group expansion of 10 structures (monoclinic C2/c to cubic, origin
choice 1, hexagonal, a CIF, one with molecules) agrees site by site with
gemmi's symmetry operations; the CIF cell parameters too.
* Supercells contain every unit-cell site exactly once per cell.
* `save` → `read stru`, CIF and RMCProfile export → import round trips give
the same atoms; DISCUS's CIF read by gemmi gives the same cell; VASP export
checked by parsing (metric, counts, coordinates).
* Known failure (strict xfail): SHELX export/import round trip.

`tests/python/test_wp4_fourier.py` (16 tests):

* fcc extinctions and equal intensities; Debye-Waller factor
exp(−B|d*|²/2) to 1e-3; X-ray form factors within 1 % of IT92 (gemmi).
* Anomalous dispersion: Friedel pairs in ZnSe differ by the ratio the
structure factor predicts (0.5 %); centrosymmetric nickel stays symmetric;
without dispersion pairs are equal; tabulated `moa1` values (golden).
* Golden maps for X-rays, neutrons and electrons; `energy` = `wvle`.
* `lots` with `seed`: repeats exactly with 1 and 4 threads; golden.
* NUFFT agrees with the standard technique for the 50×50 layer on the
`04.sro3` grid; known failure (strict xfail) for small crystals.

**Findings** (details in `WP0_Test_Coverage_Gaps.md`):

1. **NUFFT gives wrong maps for small crystals** on many grids (e.g. Ni
4×4×4, hk0 −3…3: R = 1.7), but is exact for others (10×10 cells, the
50×50 layer, a 9×9 grid over 0…4). The default technique is unaffected.
Probably related to the rare Infinity/NaN map in `04.sro3`, which uses
NUFFT. Needs fixing before WP4 wraps the technique option.
2. **SHELX export** writes every atom of the cell together with the
space-group symmetry (and occupancy 0.5): re-reading CuO gives 32 atoms
instead of 8.
3. **`disp on` does not switch on anomalous dispersion**; only `disp anom`
does. Any macro using `disp on` silently gets no dispersion.
4. **Fractions in the legacy atom format are silently misread.** `.cell`
files have two atom formats. The comma format with a column header
(`atoms x, y, z, Biso` / `co 1./3., 2./3., 0.0, 1.0`) runs the values
through the interpreter, so `1./3.` works. The legacy format (bare
`atoms`, values separated by blanks) uses a list-directed read, which
stops at `/`: `1./3.` becomes 1., without a warning. Mixing the two (bare
`atoms` with commas, or a header with blanks) gives an error.
`01.Simulation/calculate.cell` uses the legacy format and gets Co at
(0,0,0) instead of (1/3,2/3,0).
5. **`export.mac`**: its failures are mostly outdated reference files and
typos; two checks pass falsely because `compare_files.mac` treats a
missing file as "no differences".
6. `Crystal.atoms()` (Python) failed on structures with molecules; fixed.

Checked: `pytest tests/python` 80 passed, 2 known failures, the same with
`OMP_NUM_THREADS` 1 and 4.

17 changes: 9 additions & 8 deletions migration/WP0_Test_Coverage_Gaps.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ How the numbers are made, and their limits:

| Gap | Today | Why it matters |
|---|---|---|
| Structure export/import: `export` (SHELX, CIF, RMCProfile, VASP …), `import` | Not reached. `01.Simulation/export.mac` exists with 9 checks but is **not** in `test_all.mac`, and it fails 6 of 9 (see 4.1) | The Python `Crystal` class must read and write the same files |
| `save` options (`format`, `incl`, `omit`, `prop`, `asym` …) | 4 / 13 options | Round trip Python → Fortran → file |
| `spacegroup`, `wyckoff`, `asym`, `symmetry` details | Not reached / 14 of 24 sub-commands | Space-group expansion is wrapped in WP3 |
| Structure export/import: `export` (SHELX, CIF, RMCProfile, VASP …), `import` | **Covered** (`tests/python/test_wp3_structures.py`): CIF and RMCProfile export → import round trips give the same atoms; CIF also read by gemmi; VASP checked by parsing. **SHELX export is broken** (writes the whole cell *and* the symmetry; known-failure test). `01.Simulation/export.mac` itself still fails 6 of 9 (see 4.1) | The Python `Crystal` class must read and write the same files |
| `save` options (`format`, `incl`, `omit`, `prop`, `asym` …) | `save` + `read stru` round trip covered (with molecules); other options still 4 / 13 | Round trip Python → Fortran → file |
| `spacegroup`, `wyckoff`, `asym`, `symmetry` details | Space-group expansion **covered**: 10 structures (C2/c, Cmcm, Cmc2₁, Cmm2, F-43m, Pn-3m, Fm-3m, P4mm, P6₃/mmc, I4₁/amd CIF) agree site by site with gemmi; supercells checked. `wyckoff`, `asym`, `symmetry` still open. Found: in the legacy (blank-separated) atom format, fractions such as `1./3.` are silently misread; the comma format evaluates them (see 4.1) | Space-group expansion is wrapped in WP3 |
| `transform` (new cell, origin shift) | No sub-commands reached | Cell transformations are part of the data model |
| `d2r`, `r2d`, `index2cell`, `vprod` | Mostly not reached | Small metric helpers; cheap exact tests |
| KUPLOT `load` formats (`csv`, `h5`, `nexus`, `mrc`, …), `ksave`, `dsave` | 3 / 16 formats | Data-file parity when I/O moves to h5py |
Expand All @@ -66,10 +66,10 @@ How the numbers are made, and their limits:

| Gap | Today | Why it matters |
|---|---|---|
| Radiation and energy: `electron`, `energy` (instead of `wvle`) | Not reached | Every radiation type needs a golden result |
| Anomalous dispersion `delf`, single `hkl`, `layer` | Not reached | Options the API must expose |
| `set technique` (standard vs. NUFFT) | Used once | The two methods should agree: a cheap consistency test |
| Only `set aver` and `set technique` used | – | Check the other `set` options (filters, lots variants, ADPs) |
| Radiation and energy: `electron`, `energy` (instead of `wvle`) | **Covered** (`tests/python/test_wp4_fourier.py`): golden maps for X-rays, neutrons, electrons; `energy` = `wvle`; X-ray form factors within 1 % of IT92; Debye-Waller factor exact; fcc extinctions | Every radiation type needs a golden result |
| Anomalous dispersion `delf`, single `hkl`, `layer` | `delf`/`disp anom` **covered**: Friedel pairs in ZnSe match the structure-factor prediction (0.5 %), nickel stays Friedel-symmetric, tabulated `moa1` values golden. Note: `disp on` does *not* switch dispersion on (only `anom` does). `hkl`, `layer` still open | Options the API must expose |
| `set technique` (standard vs. NUFFT) | **NUFFT is wrong for small crystals on many grids** (e.g. Ni 4×4×4, hk0 −3…3: R = 1.7, spurious maxima every 0.08–0.16 in h); exact for the 50×50 layer and some size/grid combinations. Known-failure test; the default technique is unaffected. Likely related to the rare broken Fourier result in `04.sro3`, which uses NUFFT | The two methods should agree: a cheap consistency test |
| Only `set aver` and `set technique` used | `lots` with `seed` covered: repeats exactly (1 and 4 threads), golden; ADPs covered. Filters, `symm` modes still open | Check the other `set` options (filters, lots variants, ADPs) |
| `diff-four` | Not reached | |
| Strong checks exist (R < 0.001 against `RESULTS/`) | 02.1d, 02.1d_periodic, 02.2d | Good model for the rest |

Expand Down Expand Up @@ -159,7 +159,8 @@ through the new runner, none of them passes today:

| Macro | Result | Likely cause |
|---|---|---|
| `01.Simulation/export.mac` (9 checks) | 6 FAIL, 3 PASS | SHELX and CIF exports differ from `Original/` in 2 atoms. Either the export changed (**possible regression, needs a human look**) or the reference files are outdated. The macro also writes `cmm2_export.rmc6f` but compares `cmm2_export.rmcf6` (typo) |
| `01.Simulation/export.mac` (9 checks) | 6 FAIL, 3 PASS (two of the passes are false) | Re-checked: the CIF, VASP and POSCAR differences are format changes (version string, newer CIF symmetry block), not wrong atoms; the CIF round trip is exact (Python tests). The SHELX export *is* wrong (whole cell written with the space-group symmetry). The RMCProfile and SCATTY comparisons name files that do not exist (`.rmcf6` typo; `version:7` writes `.rmc6f.rmc7`), and `compare_files.mac` counts only `diff`'s output, so a missing file **passes**. The RMCProfile import check fails. Suggest: compare atoms instead of text, and make `compare_files.mac` fail on a missing file |
| `01.Simulation/calculate.cell` | – | Uses the legacy atom format (bare `atoms`, values separated by blanks) with `1./3.`. Only the comma format with a column header (`atoms x, y, z, Biso`) evaluates expressions; in the legacy format Fortran's list-directed read stops at `/`, so DISCUS reads 1. and puts Co at (0,0,0) instead of (1/3,2/3,0), without a warning. Fix the file (decimals or comma format); consider making the legacy reader reject `/` |
| `04.SRO/SRO_2/sro.2.mac 0.7,-0.7,1.5` | FAIL | Commented out in `test_all.mac`; the result is outside its tolerance |
| `04.SRO/SRO_MOLECULES/sro.2.mac 0.7,-0.7,1.5` | ABORTED: error opening a file | Input file missing or renamed |
| `08.Nanoparticle_example/Q/nano.mac 30, zno` | ABORTED: KUPLOT "Incompatible data sets for KCAL" | Output format changed since the reference was made |
Expand Down
26 changes: 22 additions & 4 deletions python/diffuse/crystal.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,7 @@ def atoms(self) -> tuple[np.ndarray, np.ndarray]:
s.run(f"discus\n save\n outfile {out.name}\n write all\n run\n exit\nexit",
cwd=s.workdir)
lines = out.read_text().splitlines()
start = next(i for i, line in enumerate(lines) if line.split()[:1] == ["atoms"]) + 1
rows = [line.replace(",", " ").split() for line in lines[start:] if line.strip()]
names = np.array([r[0] for r in rows])
xyz = np.array([[float(v) for v in r[1:4]] for r in rows])
names, xyz = _parse_stru_atoms(lines)
if len(names) != self.natoms:
raise DiffuseError(f"saved structure has {len(names)} atoms, expected {self.natoms}")
self._atoms = (names, xyz)
Expand All @@ -97,6 +94,27 @@ def __repr__(self) -> str:
return f"Crystal('{self.path.name}'{sc}, natoms={self.natoms}, lattice=({cell}))"


def _parse_stru_atoms(lines: list[str]) -> tuple[np.ndarray, np.ndarray]:
"""Atom names and coordinates from the atom section of a saved .stru file.

The section can contain ``molecule ...`` blocks between the atom lines;
an atom line is a name followed by three numbers.
"""
start = next(i for i, line in enumerate(lines) if line.split()[:1] == ["atoms"]) + 1
names, xyz = [], []
for line in lines[start:]:
words = line.replace(",", " ").split()
if len(words) < 4 or words[0].lower() == "molecule":
continue
try:
coords = [float(v) for v in words[1:4]]
except ValueError:
continue
names.append(words[0])
xyz.append(coords)
return np.array(names), np.array(xyz, dtype=float).reshape(-1, 3)


def read_structure(path: str | os.PathLike,
supercell: tuple[int, int, int] | int | None = None) -> Crystal:
"""Read a structure file into DISCUS.
Expand Down
9 changes: 9 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,18 @@ golden data in `tests/golden/<id>/`. `test_diffuse_api.py` tests the
script `python/examples/powder_demo.py` (needs matplotlib). Build with `-DDIFFUSE_PYTHON=ON`, then:

```bash
pip install -r tests/requirements.txt # gemmi, for independent references
DIFFUSE_PYTHON_PATH=/path/to/build/python pytest tests/python
```

Besides the API tests, `test_wp3_structures.py` and `test_wp4_fourier.py`
are the safety net for the WP3 (structures, I/O) and WP4 (single-crystal
Fourier) work: they compare DISCUS with gemmi's space groups and form
factors and with physical rules, and hold golden data. Tests marked
`xfail(strict=True)` document known DISCUS bugs; they turn red when the bug
is fixed, so the mark can be removed. Without gemmi the cross-checks are
skipped.

| Variable | Meaning |
|---|---|
| `DIFFUSE_PYTHON_PATH` | folder that holds `suite_python*.so` (`<build>/python`) |
Expand Down
Binary file added tests/golden/wp4.fourier/layer_lots_seed4321.npz
Binary file not shown.
42 changes: 42 additions & 0 deletions tests/golden/wp4.fourier/meta.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
{
"layer_lots_seed4321": {
"date": "2026-11-16T18:34:06+00:00",
"git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9",
"machine": "x86_64",
"numpy": "2.5.3",
"platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39",
"r_max": 1e-05
},
"znse_111_moa1_anomalous": {
"date": "2026-11-16T18:34:05+00:00",
"git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9",
"machine": "x86_64",
"numpy": "2.5.3",
"platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39",
"r_max": 1e-05
},
"znse_hk0_electron": {
"date": "2026-11-16T18:34:05+00:00",
"git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9",
"machine": "x86_64",
"numpy": "2.5.3",
"platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39",
"r_max": 1e-05
},
"znse_hk0_neutron": {
"date": "2026-11-16T18:34:05+00:00",
"git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9",
"machine": "x86_64",
"numpy": "2.5.3",
"platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39",
"r_max": 1e-05
},
"znse_hk0_xray": {
"date": "2026-11-16T18:34:05+00:00",
"git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9",
"machine": "x86_64",
"numpy": "2.5.3",
"platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39",
"r_max": 1e-05
}
}
Binary file added tests/golden/wp4.fourier/znse_111_moa1_anomalous.npz
Binary file not shown.
Binary file added tests/golden/wp4.fourier/znse_hk0_electron.npz
Binary file not shown.
Binary file added tests/golden/wp4.fourier/znse_hk0_neutron.npz
Binary file not shown.
Binary file added tests/golden/wp4.fourier/znse_hk0_xray.npz
Binary file not shown.
Loading
Loading