Fix merge gaps, merge with DuckDB in vec merge

[m-mohr]

Stacked on #55.

Fixes gaps in vec merge and merge_parquet around collection metadata vs. column data, and lets vec merge use DuckDB. Updates to follow vecorel/specification#8

• vec merge merges local GeoParquet files that are all in the target CRS with DuckDB, without loading them into memory; it falls back to the in-memory merge otherwise. New --engine option, --crs first keeps the first dataset's CRS (default remains EPSG:4326).
• Breaking: vec merge keeps all properties by default; -i restricts to core + given properties, -e removes any property. Warns if required properties are dropped.
• Schemas of a collection that occurs in multiple datasets are united instead of overwritten.
• Array/object constants are hydrated correctly: they were spread over rows, dropped, or failed.
• A missing collection is filled from a single schemas entry.
• Collection-only properties that differ between datasets are removed with a warning.
• merge_parquet:
  • hydrates constants with proper types (dates, binary, arrays, objects) and NaN as null
  • checks and suffixes ids per collection, and checks required properties per collection
  • recomputes the bbox, which was empty for GeoParquet 1.0 parts
• All-null columns are no longer dehydrated, which wrote NaN into the collection metadata or failed for nullable integers.
• get_pyarrow_type no longer mutates schemas with patternProperties.
• Validation checks that every feature has a collection listed in schemas, checks required properties per collection, and checks the schemas of all collections, not just the first.
• CHANGELOG restructured into Keep a Changelog categories.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Property filtering is inconsistent across engines, and two validation and warning paths can miss or mishandle invalid data.

Review effort: Balanced
Findings: None

Resolved since last review (3)

• Null IDs incorrectly counted as duplicates
• Missing collection column causes KeyError
• Excluded collection property is unintentionally restored

Previously missed (3)

In code that hasn't changed since last review

Geometry exclusion behaves inconsistently across engines

vecorel_cli/​conversion/​duckdb.py:658

properties can omit geometry after --exclude geometry, but this restores it only for the DuckDB path. The GeoPandas path receives the exclusion unchanged and will either fail to write a spatial file or omit the geometry, while DuckDB silently keeps it. Reject geometry exclusion or normalize the property set before engine selection so both engines behave consistently.

Unhashable collection values abort validation

vecorel_cli/​validation/​geoparquet.py:145

A malformed non-scalar collection value makes unique()/set(...) raise TypeError: unhashable type before the validator can report the column's invalid type. Validation should handle arbitrary input values and record them as unknown rather than aborting.

Removed collection-only properties are incorrectly treated as present

vecorel_cli/​vecorel/​ops.py:84

This removes every collection-only property from missing, even when filtering removed that property from the merged collection. Excluding a required collection-only property declared by a retained external schema therefore produces an invalid file without the promised warning. Only treat collection-only properties that are still present in collection as satisfied.

[ivorbosloper]

Review of what this adds on top of #55. 1 and 2 are reproduced with small scripts; the rest is from reading the code.

1. encoding/base.py:130: --engine geopandas fails on an array constant without a schema (tags=["x","y"] in both parts: ArrowTypeError); with a schema it stays a column on every row. DuckDB handles both. Suggest hydrating only the keys the merged collection drops, as merge_parquet does.
2. conversion/duckdb.py:405: results now depend on the engine. A null in a required property fails on DuckDB (with converter advice, and "collection" IS NULL repeated per group) but merges on geopandas; empty geometries are dropped on DuckDB only.
3. conversion/duckdb.py:83: _constants_table catches every exception and falls back to an inferred type, so an overflow or unparsable date is written with the wrong type and no warning.
4. vecorel/schemas.py:339: two versions of one extension (e.g. fiboa v0.2 and v0.3) are kept side by side in one collection; differing Vecorel versions do raise.
5. conversion/duckdb.py:420: the id check and suffixing use (collection, id) even with one collection, which slows the large-conversion path; the composite key is only needed with several collections.
6. merge.py:140: with -e, a non-GeoParquet input is read in full once for its column names and again for the merge.
7. conversion/duckdb.py:399: the null condition still quotes "{target}" by hand instead of _sql_name.
8. merge.py:153: get_duckdb_blocker duplicates the CRS check of _common_crs; two copies can disagree.
9. vecorel/ops.py:81: with -i, a required collection-only property is dropped without a warning.
10. vecorel/ops.py:29: rows with a null collection and no collection in the metadata now make create-geoparquet and the in-memory merge raise; merge_parquet swallows the same error and fails later.

[ivorbosloper]

Code for the review findings: #58, one commit per finding, so you can take them one by one.

[ivorbosloper]

Tested on the full ES 2025 edition (50 provinces, 17.8M rows) on a 125 GB machine, with this branch (without #58):

• fiboa convert es per province with merge_parquet (ES: convert province by province, then merge fiboa/cli#338), and 50 separate conversions merged with fiboa merge --crs first (DuckDB engine): identical results and both validate.
• fiboa convert + fiboa merge is 5x faster because portolan runs some converts in parallel