From ba823b040cc05b6b2406418b7f5d920b8436d00f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 16:49:30 -0300 Subject: [PATCH] docs: groups, vector point source, block-wise reading and CLI reference - operators/pipeline_files/mapbiomas: `groups` of majority and minority, the distance-to-points example, the contents of run.json - new guides: command line and block-wise reading - nav and CHANGELOG [Unreleased] Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01S4SjCckAQ8koVsqjYipUHr --- CHANGELOG.md | 4 ++ docs/architecture/operators.md | 5 +++ docs/guides/blocks.md | 62 ++++++++++++++++++++++++++ docs/guides/cli.md | 80 ++++++++++++++++++++++++++++++++++ docs/guides/mapbiomas.md | 24 ++++++++++ docs/guides/pipeline_files.md | 33 ++++++++++++++ mkdocs.yml | 2 + 7 files changed, 210 insertions(+) create mode 100644 docs/guides/blocks.md create mode 100644 docs/guides/cli.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 07a672b..b100c6b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Documentation +- New guides: command line (`validate`, `fetch`, `run`, `export`, `plot`) and block-wise reading (the block budget, which operators and sources are read by blocks, measured runs). +- `groups` of `majority` and `minority` (MapBiomas level-1 classes) in the operators page, the pipeline-file parameters and the MapBiomas guide; the point-source example of `distance`; the contents of `run.json`. + --- ## [0.6.0] - 2026-10-14 diff --git a/docs/architecture/operators.md b/docs/architecture/operators.md index 7977d6b..1021bae 100644 --- a/docs/architecture/operators.md +++ b/docs/architecture/operators.md @@ -60,6 +60,11 @@ These operators receive a high-resolution array snapped to the target grid origi Each of the operators above accepts `params = {"subcells": n}`, a cap on the fine pixels per cell along each axis (see `GridAligner._align_fine`). +`majority` and `minority` accept `params = {groups = {label = [codes]}}`: the map is read at a coarser level, and the +result is the label of the group that covers most (least) of the cell. The labels are integers (TOML keys are written as +strings, `"1"`), a code cannot be in two groups, and codes in no group, and nodata, are ignored. With it one derivation +turns a map of leaf classes into the MapBiomas level-1 classes; see [MapBiomas](../guides/mapbiomas.md#level-1-classes). + The last three also produce `coverage_purity` and `dominance_purity` as coordinates of the output `DataArray` (persisted in the Zarr alongside the variable). ### Proximity — vector (and passthrough for raster) diff --git a/docs/guides/blocks.md b/docs/guides/blocks.md new file mode 100644 index 0000000..40cce5e --- /dev/null +++ b/docs/guides/blocks.md @@ -0,0 +1,62 @@ +# Block-wise reading + +A national raster does not fit in memory: one year of MapBiomas for Brazil has 154,470 × 146,501 pixels, 22.6 GB at one +byte per pixel. DisSCube does not load it. It processes the grid in **blocks of rows and columns**, and the peak +memory depends on the size of a block, not on the size of the source. + +## What happens in a block + +For each block of grid cells DisSCube + +1. finds the window of the source that covers the block (in the source's CRS, clipped to the source); +2. reads only that window — MapBiomas and PRODES are tiled GeoTIFFs, so the rest is not fetched; +3. reprojects it with GDAL to the grid projection (nearest neighbour onto fine pixels for the class operators, which + count them; averaging for `mean`); +4. aggregates the pixels into the values of the block's cells, and writes them to the product. + +The blocks of one source are written to the same product, so the result is the grid, not a set of tiles. + +## How big a block is + +The budget is the number of **source pixels** aligned at once: 50 million by default. It is measured, not assumed: +a strip of grid rows that is wide in a projected grid can cover a far larger area of a longitude/latitude source than +its height suggests. So DisSCube starts from the whole grid and halves a block along its larger side until the block +that covers the most source pixels is within the budget. If the whole grid is within it, there are no blocks. + +Change the budget with the environment variable `DISSCUBE_BLOCK_PIXELS` (or `block_pixels=` in +`CubeClient.derive`): a smaller budget lowers the peak and makes more, smaller reads. + +## When a source is not read by blocks + +A source is derived whole when + +- it is **not a raster** (a vector layer is reprojected, clipped to the grid and used whole; it is small); +- a variable uses an operator that is not block-safe. The block-safe ones are `majority`, `minority`, `percentage`, + `mean`, `median`, `std`, `sum`, `min` and `max`. `distance`, `min_distance`, `count`, `area`, `attribute`, `presence` and + `network_cost` look at neighbours or at the whole layer; +- a variable uses `fill = "nearest"`, because a cell then depends on its neighbours; +- the file cannot be opened as a plain raster (a NetCDF subdataset, some URLs). + +The Brazil Data Cube source (STAC) is not block-read either: the window of each item is read whole, and the items of the +period are reduced to a per-pixel `median` (or the `reducer` of the source) before the aggregation. It is the case with +the highest peak of the runs below, from a small source. + +## Time + +Each derived variable re-reads and re-aligns the source, so a pipeline with several variables from the same raster +makes several passes over it. Reading is what takes the time. + +## Measured runs + +On one core of a desktop computer (Intel Core i7-7700T, 15 GB of RAM), reported in +[disscube-grids](https://github.com/LambdaGeo/disscube-grids), one folder per case: + +| Case | Source | Grid | Time | Peak memory | +|---|---|---|---|---| +| Brazil | MapBiomas C11, 2000 (22.6 × 10⁹ pixels) | 5.28 km; 900 × 1,040 | 1 h 43 min | 3.9 GB | +| Legal Amazon | PRODES, 2008 and 2024 (≈ 9.7 × 10⁹ pixels each) | 5.28 km; 680 × 520 | 27 min | 2.9 GB | +| Pará, classes | MapBiomas 2024 and PRODES 2024 | 5 km; 280 × 291 | 2 min 12 s | 1.2 GB | +| Pará, NDVI | BDC MODIS NDVI, 16-day composites | 5 km; 280 × 291 | 7 min 15 s | 4.2 GB (source not block-read) | + +Each case was run once; the times show that the run is feasible, not how fast it is compared with other tools. Peak +memory is the maximum resident set size reported by `/usr/bin/time -v`. diff --git a/docs/guides/cli.md b/docs/guides/cli.md new file mode 100644 index 0000000..c4178c9 --- /dev/null +++ b/docs/guides/cli.md @@ -0,0 +1,80 @@ +# Command line + +The `disscube` command works on a [pipeline file](pipeline_files.md). Five subcommands: + +| Command | What it does | Network | +|---|---|---| +| `disscube validate FILE` | checks the pipeline file and prints the plan; nothing is fetched | no | +| `disscube fetch FILE` | downloads the remote `file` sources of the pipeline and checks their SHA-256 | yes | +| `disscube run FILE` | fetches the sources, derives the variables and, with `--output`, exports them | yes | +| `disscube export FILE -o OUT` | exports variables from a workspace that already has them | no | +| `disscube plot TIF` | draws the bands of an exported GeoTIFF in a PNG | no | + +## `validate` + +```bash +disscube validate sources.toml # the plan, or the first error +disscube validate sources.toml --json # the same as JSON +``` + +An unknown operator, a `percentage` without `class_code`, a source that does not exist or a parameter the operator does +not know fail here, before any download. + +## `fetch` + +Downloads the `file` sources that declare a `url` (and a `sha256`) into the workspace cache, and verifies them. It is +the step to run once before working offline. + +## `run` + +```bash +disscube run sources.toml --workspace data/sources --output data/sources.tif +``` + +| Option | Meaning | +|---|---| +| `--workspace DIR` | where the run keeps its files; default: the `workspace` of the file, otherwise a folder named after the file | +| `--output`, `-o FILE` | export the derived variables: GeoTIFF, or netCDF if the name ends in `.nc` | +| `--dry-run` | simulate the plan without downloading or computing | +| `--json` | print the execution report as JSON | +| `-v`, `--verbose` | log each step | + +The workspace receives `catalog.db`, `store/`, `raw/` and `run.json` (see +[what a run leaves behind](pipeline_files.md#what-a-run-leaves-behind)). The exported GeoTIFF has one band per variable +and per year, named after the variable; `-o` with an existing workspace redoes only what changed. + +## `export` + +```bash +disscube export sources.toml -o data/cellspace.tif --workspace data/sources --variables l1_class +``` + +Exports from an existing workspace without deriving anything. `--variables` selects some of the derived variables +(default: all); `--workspace` defaults to `data/cube` or the workspace of the file. The same output is made by +`run --output`. + +## `plot` + +```bash +pip install 'disscube[plot]' # needs matplotlib +disscube plot data/sources.tif -o look.png --cols 2 --crop +``` + +A quick look, not a publication figure: one panel per band; the scale follows the values (shares in [0, 1] as 0 to +100 %, a few integer codes as classes, otherwise continuous between the 2nd and 98th percentiles); cells without data +are white. + +| Option | Meaning | +|---|---| +| `--output`, `-o FILE` | PNG; default: the GeoTIFF's name with `.png` | +| `--bands NAME …` | only these bands or variables | +| `--year N` | only the bands of this year | +| `--cols N` | panels per row | +| `--shared-colorbar` | one colour bar for all panels, when all are shares | +| `--crop` | trim the empty margin (the bounding box of the cells with data) | +| `--dpi N` | resolution of the PNG (default 150) | + +## Exit status + +`0` when the command finished; `2` for an error of the pipeline file, a missing optional extra or an unreadable +file, with the message on the standard error (as JSON when `--json` is given). diff --git a/docs/guides/mapbiomas.md b/docs/guides/mapbiomas.md index 385a234..1b202af 100644 --- a/docs/guides/mapbiomas.md +++ b/docs/guides/mapbiomas.md @@ -49,6 +49,30 @@ example open sea outside the mapped area). The reader treats 0 as nodata and writes the local file with `nodata=0`, so it never counts as a class in `majority` or `percentage`. +## Level-1 classes + +The leaf classes of the legend are grouped in five level-1 classes: Forest, Herbaceous and shrubby vegetation, Farming, +Non vegetated area and Water. `majority` reads a map at that level with `groups`, in one derivation and without a +reclassified copy of the map. For Collection 11 (legend of 13 August 2026): + +```toml +[[derive]] +target = "l1_class" +source = "lulc_{year}" +operator = "majority" + +[derive.params.groups] +"1" = [1, 3, 4, 5, 6, 7, 49] # forest, savanna, mangrove, floodable forest, flooded savanna, wooded sandbank +"2" = [10, 11, 12, 29, 32, 50, 77, 84] # wetland, grassland, herbaceous and shrub formation, rocky outcrop, tidal flat, salt marsh, herbaceous sandbank +"3" = [9, 14, 15, 18, 19, 20, 21, 35, 36, 39, 40, 41, 46, 47, 48, 62] # pasture, crops, forest plantation, mosaic of uses +"4" = [22, 23, 24, 25, 30, 75, 91] # beach and dune, urban, mining, photovoltaic, wind farm, other +"5" = [26, 31, 33] # river, lake and ocean; aquaculture +``` + +A code that is in no group is ignored, as is nodata (0): a class missing from the table is a class that disappears from +the map, so check the table against the legend of the collection in use. The share of a class, instead of the dominant +one, is `operator = "percentage"` with `class_code = [3, 4, 5, 6, 49]` (a list is the share in any of them). + ## Classes `disscube.sources.mapbiomas.CLASSES` names the classes most common in diff --git a/docs/guides/pipeline_files.md b/docs/guides/pipeline_files.md index dbc7f2f..bd48f71 100644 --- a/docs/guides/pipeline_files.md +++ b/docs/guides/pipeline_files.md @@ -104,6 +104,32 @@ relative to the pipeline file, so a folder with its TOML and data is portable. Every source block also accepts `name` and `years`. +### A vector source: distance to points + +A point, line or polygon layer is declared as a `file` source, like a raster, and the operator is the one that +summarises it in a cell. The distance from each cell to the nearest municipal seat, measured in a projected CRS +(`EPSG:5880`, in metres) although the grid is in BDC Albers: + +```toml +[[source]] +id = "seats" +type = "file" +path = "zip://ana/GEOFT_CIDADE_2016.zip" # a shapefile inside a zip +url = "https://…/GEOFT_CIDADE_2016.zip" # where it is downloaded from, if not there +sha256 = "0311f4267c4e…" # checked after the download + +[[derive]] +target = "dist_seats" +source = "seats" +operator = "distance" +params = { crs = "EPSG:5880" } +``` + +The distance is computed for every cell of the grid, including those outside the area of interest (the maps may mask +them), and the layer is not clipped to the grid: a point outside it can be the nearest one. A vector source is +not read by blocks (see [Block-wise reading](blocks.md)); it is small. The complete file, for the state of Pará, is +`cases/para/seats.toml` in [disscube-grids](https://github.com/LambdaGeo/disscube-grids). + ### Taking files as they come A `file` source can say how to read the file, so that the data need no @@ -147,6 +173,7 @@ of = ["state_paved", "federal_paved"] |---|---|---| | `distance` | `crs` | measure in this CRS — e.g. a projected one, for metres on a geographic grid | | `percentage`, `majority`, `minority`, `std`, `median` | `subcells` | at most this many fine pixels per cell along each axis: bounds the memory of a fine source over a large grid (a 100 m raster on a 1/12° grid would give ~90 × 90) | +| `majority`, `minority` | `groups` | `{label = [codes]}`: read a map of leaf classes at a coarser level; the result is the label of the group (codes in no group are ignored). See [MapBiomas](mapbiomas.md#level-1-classes) | | `sum` | `area` | vector polygons: share the value in proportion to the intersected area (default `false`) | | `sum` | `column` | vector: numeric column to sum (default: the target name) | | `std` | `ddof` | `0` (default, population) or `1` (sample standard deviation) | @@ -175,6 +202,12 @@ other loops or conditions: what does not fit belongs in Python. └── run.json this run: DisSCube version, pipeline file and checksum, sources, products ``` +`run.json` (a copy is kept in `runs/.json`) holds, in this order: the DisSCube version that ran +(`disscube`); the `pipeline` file, with its SHA-256 and name; the `started` and `finished` times (UTC); the `grid`; +the `sources`, each with its `id`, `type`, the local `file`, the SHA-256 `checksum` of that file and its `time`; the +`derived` products, each with its `target`, `source`, `spec_hash`, `times` and the Zarr `file`; and, if the run exported, +the `exported` path. With the pipeline file, the checksums and the version, the same grid can be rebuilt and checked. + Each provenance file gains a `pipeline` entry with the pipeline file's path, name and SHA-256, so every derived variable can be traced back to the exact file that declared it. diff --git a/mkdocs.yml b/mkdocs.yml index eaee26c..0c812df 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -40,6 +40,8 @@ nav: - TerraME Fill Cells Correspondence: terrame_fill_correspondence.md - Guides: - Pipeline files (TOML): guides/pipeline_files.md + - Command line: guides/cli.md + - Block-wise reading: guides/blocks.md - BDC Integration: guides/bdc.md - MapBiomas: guides/mapbiomas.md - "Land-cover maps: PRODES and SITS": guides/landcover.md