diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 21a321f3..648aea58 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -48,6 +48,14 @@ jobs: - name: Build run: cmake --build build -j "$(nproc)" + - name: Python environment + # --system-site-packages: use the apt NumPy/matplotlib/pytest that the + # compiled module was built against; add gemmi (test references) and + # the docs tools. + run: | + python3 -m venv --system-site-packages .venv + .venv/bin/pip install --quiet -r tests/requirements.txt -r docs/requirements.txt + - name: Python tests (pytest) env: DIFFUSE_PYTHON_PATH: ${{ github.workspace }}/build/python @@ -55,7 +63,7 @@ jobs: MPLBACKEND: Agg # DISCUS_SUITE is not set here, so the legacy macro tests inside # pytest are skipped; the next step runs them in parallel. - run: python3 -m pytest tests/python -v --junitxml=pytest-results.xml + run: .venv/bin/python -m pytest tests/python -v -rxs --junitxml=pytest-results.xml - name: Legacy macro tests # Tests tagged mc run with "seed 1234" on one thread (repeatable). A @@ -73,12 +81,7 @@ jobs: DIFFUSE_PYTHON_PATH: ${{ github.workspace }}/build/python MPLBACKEND: Agg # Warnings are errors, so broken docstrings or links fail the build. - # --system-site-packages: use the apt NumPy/matplotlib that the - # compiled module was built against. - run: | - python3 -m venv --system-site-packages .venv-docs - .venv-docs/bin/pip install --quiet -r docs/requirements.txt - .venv-docs/bin/python -m sphinx -W --keep-going -b html docs docs/_build/html + run: .venv/bin/python -m sphinx -W --keep-going -b html docs docs/_build/html - name: Upload documentation if: success() || failure() diff --git a/CLAUDE.md b/CLAUDE.md index fc732ad9..6339f06c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,6 +24,7 @@ targets `claude-python-app`. FC=gfortran CC=gcc CXX=g++ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \ -DDIFFUSE_PYTHON=ON -DPGPLOT_DIR=/usr/lib/pgplot5 cmake --build build -j +pip install -r tests/requirements.txt # gemmi DIFFUSE_PYTHON_PATH=build/python python3 -m pytest tests/python python3 tools/run_legacy_tests.py --exe build/suite/prog/discus_suite -j 4 --tags fast,mc ``` diff --git a/migration/WP0_Test_Baseline.md b/migration/WP0_Test_Baseline.md index 910a6c19..a9cfb7d2 100644 --- a/migration/WP0_Test_Baseline.md +++ b/migration/WP0_Test_Baseline.md @@ -436,3 +436,61 @@ whole Python process, so after any `diffuse` call, relative paths in the user's script pointed somewhere else. `Session.run()` now restores the caller's working directory (test `test_working_directory_kept`). +### 0.6 + 0.7 for WP3 and WP4 – structures and single-crystal Fourier + +Golden data and coverage for the first two work packages that will be +rewired, checked against independent references wherever possible (new +`tests/requirements.txt`: gemmi). + +`tests/python/test_wp3_structures.py` (26 tests): + +* Space-group expansion of 10 structures (monoclinic C2/c to cubic, origin + choice 1, hexagonal, a CIF, one with molecules) agrees site by site with + gemmi's symmetry operations; the CIF cell parameters too. +* Supercells contain every unit-cell site exactly once per cell. +* `save` → `read stru`, CIF and RMCProfile export → import round trips give + the same atoms; DISCUS's CIF read by gemmi gives the same cell; VASP export + checked by parsing (metric, counts, coordinates). +* Known failure (strict xfail): SHELX export/import round trip. + +`tests/python/test_wp4_fourier.py` (16 tests): + +* fcc extinctions and equal intensities; Debye-Waller factor + exp(−B|d*|²/2) to 1e-3; X-ray form factors within 1 % of IT92 (gemmi). +* Anomalous dispersion: Friedel pairs in ZnSe differ by the ratio the + structure factor predicts (0.5 %); centrosymmetric nickel stays symmetric; + without dispersion pairs are equal; tabulated `moa1` values (golden). +* Golden maps for X-rays, neutrons and electrons; `energy` = `wvle`. +* `lots` with `seed`: repeats exactly with 1 and 4 threads; golden. +* NUFFT agrees with the standard technique for the 50×50 layer on the + `04.sro3` grid; known failure (strict xfail) for small crystals. + +**Findings** (details in `WP0_Test_Coverage_Gaps.md`): + +1. **NUFFT gives wrong maps for small crystals** on many grids (e.g. Ni + 4×4×4, hk0 −3…3: R = 1.7), but is exact for others (10×10 cells, the + 50×50 layer, a 9×9 grid over 0…4). The default technique is unaffected. + Probably related to the rare Infinity/NaN map in `04.sro3`, which uses + NUFFT. Needs fixing before WP4 wraps the technique option. +2. **SHELX export** writes every atom of the cell together with the + space-group symmetry (and occupancy 0.5): re-reading CuO gives 32 atoms + instead of 8. +3. **`disp on` does not switch on anomalous dispersion**; only `disp anom` + does. Any macro using `disp on` silently gets no dispersion. +4. **Fractions in the legacy atom format are silently misread.** `.cell` + files have two atom formats. The comma format with a column header + (`atoms x, y, z, Biso` / `co 1./3., 2./3., 0.0, 1.0`) runs the values + through the interpreter, so `1./3.` works. The legacy format (bare + `atoms`, values separated by blanks) uses a list-directed read, which + stops at `/`: `1./3.` becomes 1., without a warning. Mixing the two (bare + `atoms` with commas, or a header with blanks) gives an error. + `01.Simulation/calculate.cell` uses the legacy format and gets Co at + (0,0,0) instead of (1/3,2/3,0). +5. **`export.mac`**: its failures are mostly outdated reference files and + typos; two checks pass falsely because `compare_files.mac` treats a + missing file as "no differences". +6. `Crystal.atoms()` (Python) failed on structures with molecules; fixed. + +Checked: `pytest tests/python` 80 passed, 2 known failures, the same with +`OMP_NUM_THREADS` 1 and 4. + diff --git a/migration/WP0_Test_Coverage_Gaps.md b/migration/WP0_Test_Coverage_Gaps.md index 708d6e89..d665cd57 100644 --- a/migration/WP0_Test_Coverage_Gaps.md +++ b/migration/WP0_Test_Coverage_Gaps.md @@ -55,9 +55,9 @@ How the numbers are made, and their limits: | Gap | Today | Why it matters | |---|---|---| -| Structure export/import: `export` (SHELX, CIF, RMCProfile, VASP …), `import` | Not reached. `01.Simulation/export.mac` exists with 9 checks but is **not** in `test_all.mac`, and it fails 6 of 9 (see 4.1) | The Python `Crystal` class must read and write the same files | -| `save` options (`format`, `incl`, `omit`, `prop`, `asym` …) | 4 / 13 options | Round trip Python → Fortran → file | -| `spacegroup`, `wyckoff`, `asym`, `symmetry` details | Not reached / 14 of 24 sub-commands | Space-group expansion is wrapped in WP3 | +| Structure export/import: `export` (SHELX, CIF, RMCProfile, VASP …), `import` | **Covered** (`tests/python/test_wp3_structures.py`): CIF and RMCProfile export → import round trips give the same atoms; CIF also read by gemmi; VASP checked by parsing. **SHELX export is broken** (writes the whole cell *and* the symmetry; known-failure test). `01.Simulation/export.mac` itself still fails 6 of 9 (see 4.1) | The Python `Crystal` class must read and write the same files | +| `save` options (`format`, `incl`, `omit`, `prop`, `asym` …) | `save` + `read stru` round trip covered (with molecules); other options still 4 / 13 | Round trip Python → Fortran → file | +| `spacegroup`, `wyckoff`, `asym`, `symmetry` details | Space-group expansion **covered**: 10 structures (C2/c, Cmcm, Cmc2₁, Cmm2, F-43m, Pn-3m, Fm-3m, P4mm, P6₃/mmc, I4₁/amd CIF) agree site by site with gemmi; supercells checked. `wyckoff`, `asym`, `symmetry` still open. Found: in the legacy (blank-separated) atom format, fractions such as `1./3.` are silently misread; the comma format evaluates them (see 4.1) | Space-group expansion is wrapped in WP3 | | `transform` (new cell, origin shift) | No sub-commands reached | Cell transformations are part of the data model | | `d2r`, `r2d`, `index2cell`, `vprod` | Mostly not reached | Small metric helpers; cheap exact tests | | KUPLOT `load` formats (`csv`, `h5`, `nexus`, `mrc`, …), `ksave`, `dsave` | 3 / 16 formats | Data-file parity when I/O moves to h5py | @@ -66,10 +66,10 @@ How the numbers are made, and their limits: | Gap | Today | Why it matters | |---|---|---| -| Radiation and energy: `electron`, `energy` (instead of `wvle`) | Not reached | Every radiation type needs a golden result | -| Anomalous dispersion `delf`, single `hkl`, `layer` | Not reached | Options the API must expose | -| `set technique` (standard vs. NUFFT) | Used once | The two methods should agree: a cheap consistency test | -| Only `set aver` and `set technique` used | – | Check the other `set` options (filters, lots variants, ADPs) | +| Radiation and energy: `electron`, `energy` (instead of `wvle`) | **Covered** (`tests/python/test_wp4_fourier.py`): golden maps for X-rays, neutrons, electrons; `energy` = `wvle`; X-ray form factors within 1 % of IT92; Debye-Waller factor exact; fcc extinctions | Every radiation type needs a golden result | +| Anomalous dispersion `delf`, single `hkl`, `layer` | `delf`/`disp anom` **covered**: Friedel pairs in ZnSe match the structure-factor prediction (0.5 %), nickel stays Friedel-symmetric, tabulated `moa1` values golden. Note: `disp on` does *not* switch dispersion on (only `anom` does). `hkl`, `layer` still open | Options the API must expose | +| `set technique` (standard vs. NUFFT) | **NUFFT is wrong for small crystals on many grids** (e.g. Ni 4×4×4, hk0 −3…3: R = 1.7, spurious maxima every 0.08–0.16 in h); exact for the 50×50 layer and some size/grid combinations. Known-failure test; the default technique is unaffected. Likely related to the rare broken Fourier result in `04.sro3`, which uses NUFFT | The two methods should agree: a cheap consistency test | +| Only `set aver` and `set technique` used | `lots` with `seed` covered: repeats exactly (1 and 4 threads), golden; ADPs covered. Filters, `symm` modes still open | Check the other `set` options (filters, lots variants, ADPs) | | `diff-four` | Not reached | | | Strong checks exist (R < 0.001 against `RESULTS/`) | 02.1d, 02.1d_periodic, 02.2d | Good model for the rest | @@ -159,7 +159,8 @@ through the new runner, none of them passes today: | Macro | Result | Likely cause | |---|---|---| -| `01.Simulation/export.mac` (9 checks) | 6 FAIL, 3 PASS | SHELX and CIF exports differ from `Original/` in 2 atoms. Either the export changed (**possible regression, needs a human look**) or the reference files are outdated. The macro also writes `cmm2_export.rmc6f` but compares `cmm2_export.rmcf6` (typo) | +| `01.Simulation/export.mac` (9 checks) | 6 FAIL, 3 PASS (two of the passes are false) | Re-checked: the CIF, VASP and POSCAR differences are format changes (version string, newer CIF symmetry block), not wrong atoms; the CIF round trip is exact (Python tests). The SHELX export *is* wrong (whole cell written with the space-group symmetry). The RMCProfile and SCATTY comparisons name files that do not exist (`.rmcf6` typo; `version:7` writes `.rmc6f.rmc7`), and `compare_files.mac` counts only `diff`'s output, so a missing file **passes**. The RMCProfile import check fails. Suggest: compare atoms instead of text, and make `compare_files.mac` fail on a missing file | +| `01.Simulation/calculate.cell` | – | Uses the legacy atom format (bare `atoms`, values separated by blanks) with `1./3.`. Only the comma format with a column header (`atoms x, y, z, Biso`) evaluates expressions; in the legacy format Fortran's list-directed read stops at `/`, so DISCUS reads 1. and puts Co at (0,0,0) instead of (1/3,2/3,0), without a warning. Fix the file (decimals or comma format); consider making the legacy reader reject `/` | | `04.SRO/SRO_2/sro.2.mac 0.7,-0.7,1.5` | FAIL | Commented out in `test_all.mac`; the result is outside its tolerance | | `04.SRO/SRO_MOLECULES/sro.2.mac 0.7,-0.7,1.5` | ABORTED: error opening a file | Input file missing or renamed | | `08.Nanoparticle_example/Q/nano.mac 30, zno` | ABORTED: KUPLOT "Incompatible data sets for KCAL" | Output format changed since the reference was made | diff --git a/python/diffuse/crystal.py b/python/diffuse/crystal.py index b81edaa2..2b47e477 100644 --- a/python/diffuse/crystal.py +++ b/python/diffuse/crystal.py @@ -78,10 +78,7 @@ def atoms(self) -> tuple[np.ndarray, np.ndarray]: s.run(f"discus\n save\n outfile {out.name}\n write all\n run\n exit\nexit", cwd=s.workdir) lines = out.read_text().splitlines() - start = next(i for i, line in enumerate(lines) if line.split()[:1] == ["atoms"]) + 1 - rows = [line.replace(",", " ").split() for line in lines[start:] if line.strip()] - names = np.array([r[0] for r in rows]) - xyz = np.array([[float(v) for v in r[1:4]] for r in rows]) + names, xyz = _parse_stru_atoms(lines) if len(names) != self.natoms: raise DiffuseError(f"saved structure has {len(names)} atoms, expected {self.natoms}") self._atoms = (names, xyz) @@ -97,6 +94,27 @@ def __repr__(self) -> str: return f"Crystal('{self.path.name}'{sc}, natoms={self.natoms}, lattice=({cell}))" +def _parse_stru_atoms(lines: list[str]) -> tuple[np.ndarray, np.ndarray]: + """Atom names and coordinates from the atom section of a saved .stru file. + + The section can contain ``molecule ...`` blocks between the atom lines; + an atom line is a name followed by three numbers. + """ + start = next(i for i, line in enumerate(lines) if line.split()[:1] == ["atoms"]) + 1 + names, xyz = [], [] + for line in lines[start:]: + words = line.replace(",", " ").split() + if len(words) < 4 or words[0].lower() == "molecule": + continue + try: + coords = [float(v) for v in words[1:4]] + except ValueError: + continue + names.append(words[0]) + xyz.append(coords) + return np.array(names), np.array(xyz, dtype=float).reshape(-1, 3) + + def read_structure(path: str | os.PathLike, supercell: tuple[int, int, int] | int | None = None) -> Crystal: """Read a structure file into DISCUS. diff --git a/tests/README.md b/tests/README.md index 49ded9f3..3717810c 100644 --- a/tests/README.md +++ b/tests/README.md @@ -86,9 +86,18 @@ golden data in `tests/golden//`. `test_diffuse_api.py` tests the script `python/examples/powder_demo.py` (needs matplotlib). Build with `-DDIFFUSE_PYTHON=ON`, then: ```bash +pip install -r tests/requirements.txt # gemmi, for independent references DIFFUSE_PYTHON_PATH=/path/to/build/python pytest tests/python ``` +Besides the API tests, `test_wp3_structures.py` and `test_wp4_fourier.py` +are the safety net for the WP3 (structures, I/O) and WP4 (single-crystal +Fourier) work: they compare DISCUS with gemmi's space groups and form +factors and with physical rules, and hold golden data. Tests marked +`xfail(strict=True)` document known DISCUS bugs; they turn red when the bug +is fixed, so the mark can be removed. Without gemmi the cross-checks are +skipped. + | Variable | Meaning | |---|---| | `DIFFUSE_PYTHON_PATH` | folder that holds `suite_python*.so` (`/python`) | diff --git a/tests/golden/wp4.fourier/layer_lots_seed4321.npz b/tests/golden/wp4.fourier/layer_lots_seed4321.npz new file mode 100644 index 00000000..12db1553 Binary files /dev/null and b/tests/golden/wp4.fourier/layer_lots_seed4321.npz differ diff --git a/tests/golden/wp4.fourier/meta.json b/tests/golden/wp4.fourier/meta.json new file mode 100644 index 00000000..024ca416 --- /dev/null +++ b/tests/golden/wp4.fourier/meta.json @@ -0,0 +1,42 @@ +{ + "layer_lots_seed4321": { + "date": "2026-11-16T18:34:06+00:00", + "git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9", + "machine": "x86_64", + "numpy": "2.5.3", + "platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39", + "r_max": 1e-05 + }, + "znse_111_moa1_anomalous": { + "date": "2026-11-16T18:34:05+00:00", + "git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9", + "machine": "x86_64", + "numpy": "2.5.3", + "platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39", + "r_max": 1e-05 + }, + "znse_hk0_electron": { + "date": "2026-11-16T18:34:05+00:00", + "git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9", + "machine": "x86_64", + "numpy": "2.5.3", + "platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39", + "r_max": 1e-05 + }, + "znse_hk0_neutron": { + "date": "2026-11-16T18:34:05+00:00", + "git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9", + "machine": "x86_64", + "numpy": "2.5.3", + "platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39", + "r_max": 1e-05 + }, + "znse_hk0_xray": { + "date": "2026-11-16T18:34:05+00:00", + "git_commit": "81ad356099c37d8b45ff632c2dc914011e0523d9", + "machine": "x86_64", + "numpy": "2.5.3", + "platform": "Linux-6.18.44-fc-v470-x86_64-with-glibc2.39", + "r_max": 1e-05 + } +} diff --git a/tests/golden/wp4.fourier/znse_111_moa1_anomalous.npz b/tests/golden/wp4.fourier/znse_111_moa1_anomalous.npz new file mode 100644 index 00000000..5366072f Binary files /dev/null and b/tests/golden/wp4.fourier/znse_111_moa1_anomalous.npz differ diff --git a/tests/golden/wp4.fourier/znse_hk0_electron.npz b/tests/golden/wp4.fourier/znse_hk0_electron.npz new file mode 100644 index 00000000..bfa69cee Binary files /dev/null and b/tests/golden/wp4.fourier/znse_hk0_electron.npz differ diff --git a/tests/golden/wp4.fourier/znse_hk0_neutron.npz b/tests/golden/wp4.fourier/znse_hk0_neutron.npz new file mode 100644 index 00000000..fc071024 Binary files /dev/null and b/tests/golden/wp4.fourier/znse_hk0_neutron.npz differ diff --git a/tests/golden/wp4.fourier/znse_hk0_xray.npz b/tests/golden/wp4.fourier/znse_hk0_xray.npz new file mode 100644 index 00000000..3f81a73e Binary files /dev/null and b/tests/golden/wp4.fourier/znse_hk0_xray.npz differ diff --git a/tests/python/test_wp3_structures.py b/tests/python/test_wp3_structures.py new file mode 100644 index 00000000..9b86f456 --- /dev/null +++ b/tests/python/test_wp3_structures.py @@ -0,0 +1,265 @@ +"""WP3 safety net: structure reading, space-group expansion, supercells, export/import. + +Golden data for the WP3 work package (Python Crystal model and I/O). The +references are independent of DISCUS where possible: + +* space-group expansion is compared with gemmi's symmetry operations + (tests are skipped if gemmi is not installed); +* export/import round trips must give back the same atoms; +* supercells must contain every unit-cell site once per cell. + + DIFFUSE_PYTHON_PATH=/python pytest tests/python/test_wp3_structures.py +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import numpy as np +import pytest + +REPO = Path(__file__).resolve().parents[2] +T = REPO / "tests" + +# (file, space group as written in the file, atoms in the unit cell) +STRUCTURES = [ + (T / "02.Calculation_Scattering/cuo.cell", "C2/c", 8), # monoclinic + (T / "14.Decoration/Deco_01/CELL/bgamma.cell", "Cmcm", 12), + (T / "14.Decoration/Deco_01/CELL/boehmite.cell", "Cmc21", 12), + (T / "08.Nanoparticle_example/COMPLEX/CELL/znse_zincblende.cell", "F-43m", 8), + (T / "02.Calculation_Scattering/cu2o.cell", "Pn-3m", 6), # origin choice 1 + (T / "01.Simulation/cmm2_scat.cell", "Cmm2", 12), # with molecules + (T / "01.Simulation/p4mm.cell", "P4mm", 8), + (T / "03.Pair_Distribution_Function/ni.cell", "Fm-3m", 4), + (T / "03.Pair_Distribution_Function/water_hex.cell", "P63/mmc", 4), # hexagonal +] +ANATASE = REPO / "python" / "examples" / "anatase.cif" +CUO = STRUCTURES[0][0] + + +@pytest.fixture +def dc(suite): + import diffuse + return diffuse + + +# ------------------------------------------------------------------ helpers +def parse_cell(path: Path) -> tuple[str, list[tuple[str, list[float]]]]: + """Space group and asymmetric unit of a DISCUS .cell file.""" + spcgr, atoms, in_atoms = None, [], False + for raw in path.read_text().splitlines(): + line = raw.split("#")[0].split("!")[0].strip() + if not line: + continue + words = line.replace(",", " ").split() + key = words[0].lower() + if key == "spcgr": + spcgr = words[1] + elif key == "atoms": + in_atoms = True + elif in_atoms and len(words) >= 4 and key != "molecule": + try: + atoms.append((words[0].upper(), [float(v) for v in words[1:4]])) + except ValueError: + continue + return spcgr, atoms + + +def wrap(d): + """Map fractional differences into [-0.5, 0.5).""" + return np.mod(np.asarray(d) + 0.5, 1.0) - 0.5 + + +def element(name: str) -> str: + """Element symbol from a DISCUS atom name: 'CU2+' -> 'CU'.""" + return re.match(r"[A-Za-z]+", name).group(0).upper() + + +def assert_same_sites(a, b, *, names=lambda n: n.upper(), tol=1e-4): + """Two lists of (name, xyz) describe the same set of sites (mod 1).""" + assert len(a) == len(b), f"{len(a)} sites vs {len(b)}" + free = list(range(len(b))) + for name, x in a: + for k in free: + m, y = b[k] + if names(m) == names(name) and np.allclose(wrap(np.subtract(x, y)), 0, atol=tol): + free.remove(k) + break + else: + raise AssertionError(f"site {name} {np.round(x, 5)} has no partner") + + +def sites(crystal): + names, xyz = crystal.atoms() + return list(zip(names, np.mod(xyz, 1.0))) + + +def gemmi_expand(spcgr: str, asym): + gemmi = pytest.importorskip("gemmi") + group = gemmi.find_spacegroup_by_name(spcgr) + assert group is not None, f"gemmi does not know {spcgr}" + out = [] + for name, xyz in asym: + found = [] + for op in group.operations(): + v = np.mod(np.array(op.apply_to_xyz(xyz)), 1.0) + if not any(np.allclose(wrap(v - q), 0, atol=1e-4) for q in found): + found.append(v) + out += [(name, v) for v in found] + return out + + +# ------------------------------------------------------ space-group expansion +@pytest.mark.parametrize("path, spcgr, natoms", STRUCTURES, ids=lambda v: getattr(v, "stem", None)) +def test_space_group_expansion_matches_gemmi(dc, path, spcgr, natoms): + """DISCUS's unit cell = gemmi's symmetry applied to the asymmetric unit.""" + file_spcgr, asym = parse_cell(path) + assert file_spcgr == spcgr + cr = dc.read_structure(path) + assert cr.natoms == natoms + assert_same_sites(sites(cr), gemmi_expand(spcgr, asym)) + + +def test_cif_expansion_matches_gemmi(dc): + gemmi = pytest.importorskip("gemmi") + st = gemmi.read_small_structure(str(ANATASE)) + ref = [(s.type_symbol, np.mod(np.array(s.fract.tolist()), 1.0)) + for s in st.get_all_unit_cell_sites()] + cr = dc.read_structure(ANATASE) + assert cr.natoms == 12 + assert_same_sites(sites(cr), ref, names=element) + cell = st.cell + assert cr.lattice == pytest.approx((cell.a, cell.b, cell.c, cell.alpha, cell.beta, cell.gamma)) + + +HCP_HEAD = "title hcp\nspcgr P63mc\ncell 3.0 3.0 4.8989797 90.0 90.0 120.0\n" + + +def test_cell_comma_format_evaluates_expressions(dc, tmp_path): + """Comma format with a column header: atom values go through the + interpreter, so '1./3.' is one third.""" + f = tmp_path / "hcp.cell" + f.write_text(HCP_HEAD + "atoms x, y, z, Biso\nco 1./3., 2./3., 0.0, 1.0\n") + _, xyz = dc.read_structure(f).atoms() + assert_same_sites([("CO", [1 / 3, 2 / 3, 0.0]), ("CO", [2 / 3, 1 / 3, 0.5])], + [("CO", x) for x in np.mod(xyz, 1.0)]) + + +def test_cell_legacy_format_misreads_fractions(dc): + """Legacy format (bare 'atoms', values separated by blanks): no + expressions; Fortran list-directed input stops at '/', so '1./3.' is + read as 1. without a warning. 01.Simulation/calculate.cell uses this + format and puts Co at (0,0,0) instead of hcp's (1/3,2/3,0). Documents + the current behaviour; change this test if the reader learns + expressions or rejects such lines.""" + cr = dc.read_structure(T / "01.Simulation/calculate.cell") + _, xyz = cr.atoms() + assert np.allclose(np.mod(xyz[:, :2], 1.0), 0.0) + + +# ------------------------------------------------------------- supercells +@pytest.mark.parametrize("supercell", [(2, 2, 2), (3, 1, 2), (4, 4, 1)]) +def test_supercell_repeats_every_site_once_per_cell(dc, supercell): + unit = dc.read_structure(CUO) + unit_sites = sites(unit) + big = dc.read_structure(CUO, supercell=supercell) + ncells = int(np.prod(supercell)) + assert big.natoms == ncells * unit.natoms + names, xyz = big.atoms() + cells = np.floor(xyz + 1e-6).astype(int) # which cell each atom is in + assert len({tuple(c) for c in cells}) == ncells + for axis, n in enumerate(supercell): + assert np.ptp(cells[:, axis]) == n - 1 # n consecutive cells per axis + for name, x in unit_sites: # every site once in every cell + hits = [c for m, y, c in zip(names, xyz, cells) + if m == name and np.allclose(wrap(y - x), 0, atol=1e-5)] + assert len({tuple(c) for c in hits}) == ncells and len(hits) == ncells + + +# --------------------------------------------------------- save / export +def test_save_and_read_back(dc, tmp_path): + cr = dc.read_structure(T / "01.Simulation/cmm2_scat.cell", supercell=(2, 2, 1)) + dc.run_macro("discus\n save\n outfile saved.stru\n write all\n run\n exit\nexit", + cwd=tmp_path) + back = dc.read_structure(tmp_path / "saved.stru") + assert back.natoms == cr.natoms + assert back.lattice == pytest.approx(cr.lattice) + n1, x1 = cr.atoms() + n2, x2 = back.atoms() + assert list(n1) == list(n2) + np.testing.assert_allclose(x1, x2, atol=1e-6) + + +def export_import(dc, path, fmt, ext, tmp_path): + src = dc.read_structure(path) + ref = sites(src) + dc.run_macro(f"discus\n export {fmt}, rt.{ext}\nexit", cwd=tmp_path) + dc.run_macro(f"discus\n import {fmt}, rt.{ext}\nexit", cwd=tmp_path) + made = [p for p in tmp_path.iterdir() if p.suffix in (".stru", ".cell")] + assert len(made) == 1, f"import made {made}" + return src, ref, dc.read_structure(made[0]) + + +@pytest.mark.parametrize("path", [CUO, ANATASE, T / "01.Simulation/cmm2_scat.cell", + T / "02.Calculation_Scattering/cu2o.cell"], + ids=lambda p: p.stem) +def test_cif_export_import_round_trip(dc, path, tmp_path): + src, ref, back = export_import(dc, path, "cif", "cif", tmp_path) + assert back.lattice == pytest.approx(src.lattice, abs=1e-4) + assert_same_sites(ref, sites(back)) + + +def test_exported_cif_read_by_gemmi(dc, tmp_path): + """An independent reader sees the same unit cell in DISCUS's CIF.""" + gemmi = pytest.importorskip("gemmi") + cr = dc.read_structure(CUO) + dc.run_macro("discus\n export cif, out.cif\nexit", cwd=tmp_path) + st = gemmi.read_small_structure(str(tmp_path / "out.cif")) + ref = [(s.type_symbol, np.mod(np.array(s.fract.tolist()), 1.0)) + for s in st.get_all_unit_cell_sites()] + assert_same_sites(sites(cr), ref, names=element) + + +@pytest.mark.parametrize("path", [CUO, ANATASE, T / "01.Simulation/cmm2_scat.cell"], + ids=lambda p: p.stem) +def test_rmcprofile_export_import_round_trip(dc, path, tmp_path): + """RMCProfile stores elements without charges ('Cu' for 'CU2+').""" + _, ref, back = export_import(dc, path, "rmcprofile", "rmc6f", tmp_path) + assert_same_sites(ref, sites(back), names=element) + + +def test_vasp_export(dc, tmp_path): + """VASP 4 layout: lattice vectors, counts per element, direct coordinates.""" + cr = dc.read_structure(CUO) + dc.run_macro("discus\n export vasp, out.vasp\nexit", cwd=tmp_path) + lines = (tmp_path / "out.vasp").read_text().splitlines() + scale = float(lines[1]) + vectors = scale * np.array([[float(v) for v in lines[i].split()] for i in (2, 3, 4)]) + counts = [int(v) for v in lines[5].split()] + assert lines[6].strip().lower().startswith("direct") + xyz = np.array([[float(v) for v in line.split()[:3]] for line in lines[7:7 + sum(counts)]]) + a, b, c = np.linalg.norm(vectors, axis=1) + alpha = np.degrees(np.arccos(vectors[1] @ vectors[2] / (b * c))) + beta = np.degrees(np.arccos(vectors[0] @ vectors[2] / (a * c))) + gamma = np.degrees(np.arccos(vectors[0] @ vectors[1] / (a * b))) + assert (a, b, c, alpha, beta, gamma) == pytest.approx(cr.lattice, abs=1e-4) + names, ref = cr.atoms() + assert counts == [int((names == n).sum()) for n in dict.fromkeys(names)] + np.testing.assert_allclose(xyz, ref, atol=1e-6) + + +@pytest.mark.xfail(strict=True, reason=( + "export shelx writes every atom of the unit cell but keeps the space-group " + "symmetry (LATT/SYMM) and sets occupancies to 0.5; re-importing CuO gives " + "32 atoms instead of 8. Remove this mark once the export is fixed.")) +def test_shelx_export_import_round_trip(dc, tmp_path): + _, ref, back = export_import(dc, CUO, "shelx", "ins", tmp_path) + assert_same_sites(ref, sites(back), names=element) + + +def test_shelx_export_refuses_origin_choice_2(dc, tmp_path): + """SHELX needs the centre of symmetry at the origin; DISCUS says so.""" + dc.read_structure(ANATASE) + with pytest.raises(dc.MacroError, match="1bar not at origin"): + dc.run_macro("discus\n export shelx, out.ins\nexit", cwd=tmp_path) diff --git a/tests/python/test_wp4_fourier.py b/tests/python/test_wp4_fourier.py new file mode 100644 index 00000000..a4be9ead --- /dev/null +++ b/tests/python/test_wp4_fourier.py @@ -0,0 +1,242 @@ +"""WP4 safety net: single-crystal Fourier options, checked against physics. + +Golden data for the WP4 work package (the Fourier shim). Where possible the +reference is physics rather than an earlier DISCUS run: + +* fcc extinction rules and equal intensities (nickel, neutrons, no ADPs); +* the Debye-Waller factor exp(-B |d*|^2 / 2); +* X-ray form factors against the International Tables (IT92, via gemmi); +* anomalous dispersion: Friedel pairs differ in non-centrosymmetric ZnSe by + the amount the structure factor predicts, and stay equal in nickel; +* energy and wavelength settings agree; seeded 'lots' runs repeat exactly; +* the NUFFT technique against the standard one. + +The DISCUS commands are run directly (diffuse.run_macro), so options that +the Python API does not offer yet are covered too. + + DIFFUSE_PYTHON_PATH=/python pytest tests/python/test_wp4_fourier.py +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pytest + +from compare import check_golden, rvalue + +REPO = Path(__file__).resolve().parents[2] +NI = REPO / "tests/03.Pair_Distribution_Function/ni.cell" # Fm-3m, B = 0.2 +ZNSE = REPO / "tests/08.Nanoparticle_example/COMPLEX/CELL/znse_zincblende.cell" # F-43m +LAYER = REPO / "tests/02.Calculation_Scattering/example3.stru" # 50x50 Cu/Au layer +A_NI, B_NI = 3.5234, 0.2 +HK0 = dict(ll="-3,-3,0", lr="3,-3,0", ul="-3,3,0", na=61, no=61) + + +@pytest.fixture +def dc(suite): + import diffuse + return diffuse + + +def fourier(dc, *, ll, lr, ul=None, na, no=1, radiation="neutron", temp="ign", + extra=""): + """Run the fourier menu with the given settings; return the KUPLOT data. + + A line gives (x, intensity); a plane gives (x, y, intensity[na, no]). + ``extra`` holds further fourier-menu commands (wvle, disp, delf, set ...). + """ + s = dc.session() + out = s.workdir / "wp4.inte" + out.unlink(missing_ok=True) + s.run(f""" +discus + fourier + reset + {radiation} + ll {ll} + lr {lr} + ul {ul or ll} + na {na} + no {no} + abs h + ord k + temp {temp} + disp off + set aver, 0 + lots off +{extra} + run + exit + output + outf {out.name} + value inte + form stan + run + exit +exit +kuplot + rese + load {"xy" if no == 1 else "ni"}, {out.name} +exit +""", cwd=s.workdir) + if no == 1: + x, i = s.dataset(1) + return x.astype(float), i.astype(float) + x, y, z = s.dataset_2d(1) + return x.astype(float), y.astype(float), z.astype(float) + + +# ------------------------------------------------------------ basic physics +def test_fcc_extinction_and_equal_intensities(dc): + """Ni, 4x4x4 cells, neutrons, no ADPs: hk0 reflections only for h, k even, + all of the same intensity; zero at mixed parity and between Bragg peaks.""" + dc.read_structure(NI, supercell=4) + h, k, z = fourier(dc, ll="0,0,0", lr="4,0,0", ul="0,4,0", na=9, no=9) + hh, kk = np.meshgrid(h, k, indexing="ij") + allowed = (np.isclose(hh % 2, 0) & np.isclose(kk % 2, 0)) + np.testing.assert_allclose(z[allowed], z.max(), rtol=1e-5) + assert np.all(z[~allowed] < 1e-6 * z.max()) + + +def test_debye_waller_factor(dc): + """With ADPs, each reflection is scaled by exp(-B |d*|^2 / 2).""" + dc.read_structure(NI, supercell=4) + plane = dict(ll="0,0,0", lr="4,0,0", ul="0,4,0", na=9, no=9) + h, k, cold = fourier(dc, temp="ign", **plane) + _, _, warm = fourier(dc, temp="use", **plane) + hh, kk = np.meshgrid(h, k, indexing="ij") + peak = cold > 1e-6 * cold.max() + d2 = (hh**2 + kk**2) / A_NI**2 + np.testing.assert_allclose(warm[peak] / cold[peak], np.exp(-B_NI * d2[peak] / 2), rtol=1e-3) + + +def test_xray_form_factor_matches_it92(dc): + """X-ray / neutron intensity follows f(s)^2 of the International Tables.""" + gemmi = pytest.importorskip("gemmi") + dc.read_structure(NI, supercell=4) + plane = dict(ll="0,0,0", lr="4,0,0", ul="0,4,0", na=9, no=9) + h, k, neutron = fourier(dc, radiation="neutron", **plane) + _, _, xray = fourier(dc, radiation="xray", **plane) + hh, kk = np.meshgrid(h, k, indexing="ij") + peak = neutron > 1e-6 * neutron.max() + s = np.sqrt((hh**2 + kk**2)[peak]) / A_NI / 2 # sin(theta)/lambda + f = np.array([gemmi.Element("Ni").it92.calculate_sf(float(v) ** 2) for v in s]) + ratio = xray[peak] / neutron[peak] / f**2 + assert np.ptp(ratio) / ratio.mean() < 0.01 + + +def test_energy_equals_wavelength(dc): + """'energy E' (keV) gives the same map as 'wvle 12.3984/E'.""" + dc.read_structure(ZNSE, supercell=3) + *_, a = fourier(dc, radiation="xray", extra=" wvle 1.0", **HK0) + *_, b = fourier(dc, radiation="xray", extra=" energy 12.398419843", **HK0) + assert rvalue(a, b) < 1e-6 + + +# -------------------------------------------------------- anomalous dispersion +ANOMALOUS = " wvle 1.0\n delf ZN, -1.5, 0.7\n delf SE, -3.0, 0.5\n disp anom" +LINE_111 = dict(ll="-3,-3,-3", lr="3,3,3", na=61) # h = k = l from -3 to 3 + + +def test_without_dispersion_friedel_pairs_are_equal(dc): + dc.read_structure(ZNSE, supercell=3) + _, i = fourier(dc, radiation="xray", extra=" wvle 1.0", **LINE_111) + np.testing.assert_allclose(i, i[::-1], rtol=1e-6) + + +def test_dispersion_breaks_friedel_law_in_znse(dc): + """Zincblende: F(hhh) = 4 (f_Zn + f_Se exp(i pi 3h/2)). With f = f0 + f' + i f'' + the 111 and -1-1-1 intensities differ by a predictable ratio.""" + gemmi = pytest.importorskip("gemmi") + dc.read_structure(ZNSE, supercell=3) + x, i = fourier(dc, radiation="xray", extra=ANOMALOUS, **LINE_111) + i111, i_111 = i[np.argmin(abs(x - 1))], i[np.argmin(abs(x + 1))] + s = np.sqrt(3) / 5.496 / 2 + f_zn = gemmi.Element("Zn").it92.calculate_sf(s**2) - 1.5 + 0.7j + f_se = gemmi.Element("Se").it92.calculate_sf(s**2) - 3.0 + 0.5j + expected = abs(f_zn - 1j * f_se) ** 2 / abs(f_zn + 1j * f_se) ** 2 + assert i111 / i_111 == pytest.approx(expected, rel=5e-3) + assert abs(i111 / i_111 - 1) > 0.01 + + +def test_dispersion_keeps_friedel_law_in_centrosymmetric_nickel(dc): + dc.read_structure(NI, supercell=3) + _, i = fourier(dc, radiation="xray", + extra=" wvle 1.0\n delf NI, -2.0, 1.0\n disp anom", **LINE_111) + assert rvalue(i, i[::-1]) < 1e-4 + + +def test_disp_on_is_not_anomalous(dc): + """Only 'disp anom' switches dispersion on; 'disp on' leaves it off.""" + dc.read_structure(ZNSE, supercell=3) + _, i = fourier(dc, radiation="xray", + extra=ANOMALOUS.replace("disp anom", "disp on"), **LINE_111) + np.testing.assert_allclose(i, i[::-1], rtol=1e-6) + + +def test_internal_dispersion_tables_golden(dc): + """Tabulated f', f'' for Mo K-alpha1 ('wvle moa1', 'delf all, internal').""" + dc.read_structure(ZNSE, supercell=3) + x, i = fourier(dc, radiation="xray", + extra=" wvle moa1\n delf all, internal\n disp anom", **LINE_111) + assert rvalue(i, i[::-1]) > 0.005 # Friedel pairs differ + check_golden("wp4.fourier", "znse_111_moa1_anomalous", r_max=1e-5, h=x, intensity=i) + + +# ------------------------------------------------------- radiation goldens +@pytest.mark.parametrize("radiation", ["xray", "neutron", "electron"]) +def test_radiation_golden(dc, radiation): + dc.read_structure(ZNSE, supercell=3) + h, k, z = fourier(dc, radiation=radiation, temp="use", extra=" wvle 0.71", **HK0) + assert np.all(z >= 0) + check_golden("wp4.fourier", f"znse_hk0_{radiation}", r_max=1e-5, h=h, k=k, intensity=z) + + +# --------------------------------------------------------------- technique +def test_nufft_matches_standard_for_layer(dc): + """NUFFT and the standard technique agree for the 50x50 layer on the grid + that 04.SRO/SRO_3 uses.""" + dc.read_structure(LAYER) + plane = dict(ll="-10,-10,0", lr="10,-10,0", ul="-10,10,0", na=201, no=201) + *_, standard = fourier(dc, radiation="xray", extra=" set technique:turbo", **plane) + *_, nufft = fourier(dc, radiation="xray", extra=" set technique:nufft", **plane) + assert rvalue(standard, nufft) < 1e-4 + + +@pytest.mark.xfail(strict=True, reason=( + "NUFFT gives wrong maps for small crystals on many grids: Ni 4x4x4, hk0 " + "-3..3 with 61x61 points has R = 1.7 against the standard technique, with " + "spurious maxima every 0.08-0.16 in h. Some size/grid combinations are " + "exact (10x10 cells, or a 9x9 grid over 0..4). Remove the mark once fixed.")) +def test_nufft_matches_standard_for_small_crystal(dc): + dc.read_structure(NI, supercell=4) + *_, standard = fourier(dc, extra=" set technique:turbo", **HK0) + *_, nufft = fourier(dc, extra=" set technique:nufft", **HK0) + assert rvalue(standard, nufft) < 1e-4 + + +# -------------------------------------------------------------------- lots +LOTS = " lots box, 10, 10, 1, 20, yes" +LOTS_PLANE = dict(ll="0,0,0", lr="2,0,0", ul="0,2,0", na=81, no=81) + + +def lots_map(dc, seed): + dc.read_structure(LAYER) + dc.run_macro(f"seed {seed}") + return fourier(dc, radiation="xray", extra=LOTS, **LOTS_PLANE) + + +def test_lots_with_seed_repeat_exactly(dc): + *_, first = lots_map(dc, 4321) + *_, again = lots_map(dc, 4321) + *_, other = lots_map(dc, 999) + assert np.all(np.isfinite(first)) + np.testing.assert_array_equal(first, again) + assert rvalue(first, other) > 0.01 # another seed, other lots + + +def test_lots_golden(dc): + h, k, z = lots_map(dc, 4321) + check_golden("wp4.fourier", "layer_lots_seed4321", r_max=1e-5, h=h, k=k, intensity=z) diff --git a/tests/requirements.txt b/tests/requirements.txt new file mode 100644 index 00000000..66eb284f --- /dev/null +++ b/tests/requirements.txt @@ -0,0 +1,4 @@ +# Extra packages for the Python tests (pytest, NumPy and matplotlib come from +# the system or your environment). gemmi gives independent crystallographic +# references: space-group operations and International Tables form factors. +gemmi>=0.6