Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,53 @@ jobs:
if [ -n "$unexpected" ]; then
echo "unexpected skips:"; echo "$unexpected"; exit 1
fi

one-extra:
# The leg above installs the core alone and then everything. These install
# one extra each, so the extras' boundaries are tested in both directions:
# the extra's own modules import without the others, and every test that
# needs another extra skips rather than erroring at collection.
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- {extra: spec, modules: "ymf.schema, ymf.compose, ymf.generated"}
- {extra: units, modules: "ymf.units"}
- {extra: symbolic, modules: "ymf.symbolic, ymf.schema"}
name: linux-64 / py3.12 / ymf[${{ matrix.extra }}] alone
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Build sdist
run: |
python -m pip install --upgrade pip build
python -m build --sdist

- name: Install ymf[${{ matrix.extra }}] alone
shell: bash
run: |
python -m venv /tmp/one
/tmp/one/bin/pip install "$(ls dist/ymf-*.tar.gz)[${{ matrix.extra }}]" pytest

- name: Its modules import without the other extras
shell: bash
run: |
cd /tmp
/tmp/one/bin/python -c "import ymf, ${{ matrix.modules }}; print('ymf', ymf.__version__, 'imports ${{ matrix.modules }}')"

- name: Run test suite
shell: bash
run: |
# The tests that need another extra skip themselves; a collection
# error or a failure here is a test, or a module, that assumes an
# extra it does not declare.
export PATH=/tmp/one/bin:$PATH
python -c "import ymf; print(ymf.__file__)" | grep -v "/src/ymf/"
python -m pytest tests -q -rs -p no:cacheprovider
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ ymf version that wrote it (`ymf:`), and a reader refuses one from a newer ymf.

| Piece | State |
|---|---|
| Archive data model, YAML I/O, structural checks (`ymf.archive`) | **Implemented.** Tested against real Proteus output; used by Proteus's `ymf_io` branch, aimed at Proteus 2.0 |
| Archive data model, YAML I/O, structural checks (`ymf.archive`) | **Implemented.** Tested against real Proteus output; used by Proteus 2.0 (its `main` line) |
| `.ymf` → `.xmf` conversion (`ymf2xmf`, `ymf.xdmf`) | **Implemented.** Lossless round trip; output opens in ParaView. Reads XDMF from other tools too, refusing by name what the model can't hold ([docs/xdmf-model.yaml](docs/xdmf-model.yaml)) |
| Self-contained archives: arrays inline, no HDF5 | **Implemented.** `data_item_for(array, inline=True)`; `write_archive.py --inline` |
| Problem-spec schema v0.2 (`ymf.schema`) | **Implemented.** See the [known gaps](docs/problem-specification.md#known-gaps) |
Expand Down Expand Up @@ -87,7 +87,7 @@ A specification, abridged from
[examples/kovasznay_flow.ymf](examples/kovasznay_flow.ymf):

```yaml
Problem:
model:
name: "Kovasznay flow, Re = 40"
physical_model:
provenance: llm_derived
Expand Down Expand Up @@ -132,16 +132,16 @@ Writing an archive, from
[examples/write_archive.py](examples/write_archive.py):

```python
from ymf.archive import (new_domain, add_uniform_step, topology, geometry,
from ymf.archive import (new_approximation, add_uniform_step, topology, geometry,
attribute, data_item_for, write_ymf)

domain = new_domain("Mesh Spatial_Domain")
domain = new_approximation("Mesh Spatial_Domain")
topo = topology("Triangle", len(elements), data_item_for(elements, "heat.h5:/elements"))
geom = geometry(data_item_for(nodes, "heat.h5:/nodes"))
for k, t in enumerate(times):
add_uniform_step(domain, t, topo, geom,
[attribute("T", data_item_for(T[k], "heat.h5:/T_t%d" % k))])
write_ymf(domain, "heat.ymf", extra={"Problem": problem})
write_ymf(domain, "heat.ymf", extra=problem) # the spec: model, solution_paths, ...
```

`data_item_for` reads shape, type and precision off the array, so the
Expand Down
50 changes: 28 additions & 22 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,46 @@
# YMF documentation

Start with the two guides. They describe what exists today. The rest are
design drafts and background research, kept for their reasoning.
Start with the two guides. They describe what exists today. Design notes say
what was decided and whether it is built. History and research are kept for
their reasoning, not as a description of the code.

## Guides (current)

| Document | What it covers |
|---|---|
| [problem-specification.md](problem-specification.md) | Writing and validating a problem spec: structure, branching, provenance, units, known gaps |
| [problem-specification.md](problem-specification.md) | Writing and validating a problem spec: structure, branching, composition, provenance, units, known gaps |
| [archive-format.md](archive-format.md) | Archives: the archive of a spec (outputs keyed by their input, studies, reruns, `extends`), and the archive a solver writes: files on disk, the domain model, parallel output, XDMF conversion, use in Proteus |
| [xdmf-model.yaml](xdmf-model.yaml) | The archive model laid out in the order of XDMF's own [Model and Format](https://xdmf.org/index.php/XDMF_Model_and_Format.html) page: each XDMF element and attribute, the `.ymf` key that holds it, and whether it is supported. For readers who know XDMF. |
| [xdmf-model.yaml](xdmf-model.yaml) | The archive model laid out in the order of XDMF's own [Model and Format](https://xdmf.org/index.php/XDMF_Model_and_Format.html) page: each XDMF element and attribute, the `.ymf` key that holds it, and whether it is supported. For readers who know XDMF. Checked by `tests/test_xdmf_model.py`. |
| [changes.md](changes.md) | Gaps that have been closed, dated, with the commit that closed each. The guides describe the result; this says why it is that way. |

## Design drafts
## Design notes

| Document | Status |
A design note is written before the code, records the decisions taken on it,
and lands in `design/` with the code that implements it.

| Note | Status |
|---|---|
| [ymf-schema.md](ymf-schema.md) | The original v0.1 schema draft. Still the best statement of the design principles and of the intended `vvuq` shape. Its §4 "Archive Configuration" and §5 "XDMF Preservation" were superseded by the implemented [archive format](archive-format.md), and the title expands the old working name. |
| [ymf-schema-v0.2-delta.md](ymf-schema-v0.2-delta.md) | The v0.2 changes on top of v0.1, which `ymf.schema` implements. §7 lists what was deferred to v0.3. |
| [design/time-outside-space.md](design/time-outside-space.md) | **Implemented** 2026-10-26: archive format 2, one list of instants with the spaces inside each. |

## History

Superseded drafts, kept for the reasoning behind decisions still in force.
Each starts with a note saying what replaced it.

Both drafts use `strong_form.unknown_provenance`. That key has since been renamed to `strong_form.provenance`, matching every other block, and a structured unknown may now carry its own `provenance`. See [Provenance](problem-specification.md#provenance).
| Document | What it was |
|---|---|
| [history/schema-v0.1.md](history/schema-v0.1.md) | The original v0.1 schema draft. Its § 3 is still the only description of the intended `vvuq` shape. |
| [history/schema-v0.2-delta.md](history/schema-v0.2-delta.md) | The v0.2 changes on top of v0.1, which `ymf.schema` implements. |
| [history/session-2026-07-29.md](history/session-2026-07-29.md) | Notes from the session where YMF was designed, under its earlier name YDMF. |

## Background research

These were drafted with an LLM research agent, and have not all been
checked against their sources. Treat specific numbers and API names in
them as leads to verify, not facts.
Drafted with an LLM research agent, and not all checked against their
sources. Treat specific numbers and API names in them as leads to verify,
not facts. Each says so in its own header.

| Document | Covers | Caution |
|---|---|---|
| [ymf-units-report.md](ymf-units-report.md) | Unit libraries, dimensional analysis, non-dimensionalization | The basis for `ymf.units`. |
| [data-sources-report.md](data-sources-report.md) | Topography, bathymetry, land cover, roughness and forcing datasets | The basis for `ymf.data_sources`; dataset details not re-verified. |
| [csdms-bmi-report.md](csdms-bmi-report.md) | CSDMS Standard Names and the Basic Model Interface | **Contains known errors.** CSDMS is the *Community Surface Dynamics Modeling System*, not what §1 says. The registry fetched at the time listed 2,653 names, not "~4,000+". The real naming grammar is `object__quantity` (double underscore), not the pattern the report gives. Its "Registry API" section and example variable names were never confirmed against a source. |

## History

[session-summary-2026-07-29.md](session-summary-2026-07-29.md) is the notes
from the session where YMF was designed, under its earlier name YDMF. It
records the original intent, including ideas that have not been built (the
`ibvp`/sympy symbolic layer, multi-backend dispatch).
| [research/units.md](research/units.md) | Unit libraries, dimensional analysis, non-dimensionalization | The basis for `ymf.units`. |
| [research/data-sources.md](research/data-sources.md) | Topography, bathymetry, land cover, roughness and forcing datasets | The basis for `ymf.data_sources`; dataset details not re-verified. |
| [research/csdms-bmi.md](research/csdms-bmi.md) | CSDMS Standard Names and the Basic Model Interface | **Contains known errors**, listed in its header. |
45 changes: 28 additions & 17 deletions docs/archive-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A YMF archive is how a simulation records its output: the heavy arrays go
in HDF5, and a small YAML document (`.ymf`) says what each array is — which
mesh, which time, which field, what shape and type. This half of YMF is
implemented, tested against real solver output, and is what Proteus writes
on its `ymf_io` branch.
on its 2.0 line (`main`).

The code is `ymf.archive` (data model and YAML I/O), `ymf.xdmf` (conversion
to XDMF), and the `ymf2xmf` command. All three depend on `pyyaml` alone.
Expand All @@ -16,8 +16,10 @@ which parts a `.ymf` archive represents.
An archive takes two forms. **The archive of a spec** is the problem
specification as realized, followed by every output computed from it;
`ymf.closure` reads and writes it, and Proteus's `ymf_run` produces it.
Its outputs each hold an approximation in **the solver's form**, the
`domain` tree a solver writes and the rest of this page describes.
Its outputs each hold an approximation in **the solver's form**: the
time/space/field tree a solver writes, which the rest of this page describes.
The same tree is called `domain` at the top of a solver's own document and
`approximation` inside an output of the archive of a spec.

- [The archive of a spec](#the-archive-of-a-spec)
- [Files on disk](#files-on-disk)
Expand All @@ -39,7 +41,7 @@ was realized, then what was computed from it.

```yaml
ymf: 0.2.1 # the ymf that wrote it; a newer one is refused on read
Problem: {...} # the spec, composed and as realized: no extends,
model: {...} # the spec, composed and as realized: no extends,
solution_paths: {...} # every override listed under composition
vvuq: {...}
composition: {sources: [...], overrides: [...]}
Expand All @@ -60,7 +62,7 @@ outputs:
solve: {by: {ymf: ..., proteus_git: ..., numpy: ..., platform: ...}, created: ...}
storage: poisson_2365d177bb1f.h5 # or: inline
verification: {l2_errors: {u: 0.0105...}}
approximation: {times: [...]} # the solver's domain tree
approximation: {times: [...]} # the solver's time/space/field tree
reproduced: [{created: ..., by: {...}}] # each rerun that matched, bitwise
```

Expand Down Expand Up @@ -164,7 +166,7 @@ ymf_archive_version: 2 # refused on read if it isn't a version this ymf k
domain: # the time/space/field tree -- see below
times: [...]
extra: # optional: anything with no XDMF equivalent
Problem: {...} # typically the problem spec that produced the run
model: {...} # typically the problem spec that produced the run
solution_paths: {...}
```

Expand Down Expand Up @@ -208,7 +210,7 @@ domain:
- time: 10.0
...
extra:
Problem:
model:
name: Transient heat conduction, unit square
...
```
Expand All @@ -218,6 +220,15 @@ only to keep the listing short.)

## The domain model

The functions that build and check this tree are named for what it is in
the archive of a spec, an *approximation*: `new_approximation`,
`validate_approximation`, `canonicalize_approximation`,
`select_approximation`, `inline_approximation`, `rebase_approximation`,
`approximation_arrays`. The names they had before 2026-11 (`new_domain`,
...) are aliases, kept for one release, so that `domain` can mean the region
a problem is posed on. The solver file's `domain:` key is unchanged: it is
part of `ymf_archive_version: 2`.

A domain is plain dicts and lists — no classes — so it costs nothing to
build, gathers across MPI natively, and serializes directly.

Expand Down Expand Up @@ -273,7 +284,7 @@ of a domain, `space_names(domain)` and `series(domain, name)`;
(`transpose_collections`): steps of different spaces are joined on exact
time, a repeated time pairs up by occurrence, and a space missing at a time
is omitted from that instant. Nothing is rewritten on disk. Code that built
a domain with `new_domain(name)` / `add_collection` / `add_uniform_step(...,
a domain with `new_approximation(name)` / `add_collection` / `add_uniform_step(...,
collection=)` keeps working and is upgraded when the domain is validated or
written; `{"time_collection": {...}}` (singular) is accepted as before.

Expand All @@ -289,7 +300,7 @@ is rejected when the archive is written:
| `data_type` | `Float Int UInt Char UChar` |
| `format` | `HDF XML Binary` |

**Defaults.** `canonicalize_domain()` fills every defaultable field
**Defaults.** `canonicalize_approximation()` fills every defaultable field
explicitly, and the file on disk is always canonical. The one default worth
knowing is that XDMF's default `Precision` is **4** bytes, which is wrong
for `float64`. Use `data_item_for()`, which reads the precision off the
Expand All @@ -304,10 +315,10 @@ disagree with what's in the file.

```python
import h5py, numpy as np
from ymf.archive import (new_domain, add_instant, uniform_grid, topology, geometry,
from ymf.archive import (new_approximation, add_instant, uniform_grid, topology, geometry,
attribute, data_item_for, write_ymf)

domain = new_domain()
domain = new_approximation()
with h5py.File("heat.h5", "w") as h5:
h5["nodes"], h5["elements"] = nodes, elements # mesh written once
topo = topology("Triangle", len(elements), data_item_for(elements, "heat.h5:/elements"))
Expand All @@ -318,7 +329,7 @@ with h5py.File("heat.h5", "w") as h5:
attribute("T", data_item_for(T, "heat.h5:/T_t%d" % k)),
])})

write_ymf(domain, "heat.ymf", extra={"Problem": problem})
write_ymf(domain, "heat.ymf", extra=problem) # the spec: model, solution_paths, ...
```

[`examples/write_archive.py`](../examples/write_archive.py) is the
Expand Down Expand Up @@ -398,7 +409,7 @@ YMF splits checking across a trust boundary:
- A **solver** trusts the validated input and writes the archive. On that
path the checks are only structural and cost O(1) per field.

`write_ymf()` runs `validate_domain()` by default. It checks required keys,
`write_ymf()` runs `validate_approximation()` by default. It checks required keys,
the closed sets above, and that no grid has two fields with the same name
and centering (viewers key fields by name within point data and within
cell data, so a duplicate silently hides the first; the same name cell-
Expand Down Expand Up @@ -439,7 +450,7 @@ wrote view.xmf
`-c/--collection` (a name or part of one), `-f/--field` and `-s/--steps`
(Python indices and slices: `-1`, `0:10`, `::5`) cherry-pick what the `.xmf`
describes; the `.ymf` is never changed. In code the same selection is
`ymf.archive.select_domain(domain, collections=, fields=, steps=)`.
`ymf.archive.select_approximation(domain, collections=, fields=, steps=)`.

ParaView and VisIt open the `.xmf` directly. `extra` is carried into it as
a base64-encoded JSON `<Information Name="YMF" Value="..."/>` element, a
Expand Down Expand Up @@ -472,8 +483,8 @@ as they are written and the full document once, at close.

## Use in Proteus

Proteus's `ymf_io` branch (aimed at the 2.0 line) replaces its hand-built
XDMF XML with this model:
Proteus's 2.0 line (`main`, since the `ymf_io` branch merged) replaces its
hand-built XDMF XML with this model:

- Every field goes through one `write_field` call, which builds its
`Attribute` + `DataItem`. Fields beyond the solution itself are declared
Expand All @@ -497,7 +508,7 @@ domain used to hold one temporal collection per function space, each with
its own list of instants, because XDMF does; it now holds one list of
instants with the spaces inside each. The reasons, the migration and the
Proteus changes are in
[design-time-outside-space.md](design-time-outside-space.md). In short:
[design/time-outside-space.md](design/time-outside-space.md). In short:
the Proteus writer already refused a space missing at an instant; an
instant is now the unit of an appended record, so a killed run loses a whole
instant and never one space of it; and `append_step` no longer names
Expand Down
Loading
Loading