Enable global applicability by ekatef · Pull Request #8 · PyPSA/grid-builder

ekatef · 2026-09-27T08:38:38Z

The PR brings in functionality to enable accuracy of GridBuilder when being applied to any country of the world.

Disclaimer

The implementation is build an a combination of PyPSA-Earth approach with learnings from the regional grid studies conducted during previous year by PyPSA-meets-Earth and MapYourGrid initiatives. I have updated the authors list accordingly, though it is not exhaustive yet.

Major changes

Restore country field since it is crucial for correct assignment of conductor types. Bring-in regional line types #7 provides more details on that.
Improved reliability and accuracy of the parameters clean-up according to results of the global grid-building runs.
Enhanced treatment of the frequency field to enable generalisation.
Increase reliability for non-perfect data, specifically for the countries where some of the data features can be missing.
Added connections with the data-producing initiatives on which the project builds up.
Enable usage of custom data.
Added treatment of DC lines using a combination of PyPSA-Earth experience with the relation concept proposed in PyPSA-Eur.
Bug-fixing, removal of hard-coding and human proof-read of the documentation.

Reviewer checklist

There are no pip dependencies in the module's environment files (workflow/envs/).
All rules use pathvars (e.g., <results>) in their inputs and outputs.
The integration test-suite is successful, including:
- pre-commit.ci tests pass.
- tests pass for all relevant OS configurations (linux, osx, windows).
Module documentation is up-to-date, including:
- INTERFACE.yaml mentions all relevant pathvars and wildcards.
- README.md describes how to use the module and has the necessary citations.

ekatef · 2026-09-27T08:45:17Z

@davide-f it would be good to have your looks on that. The major goal of this PR is to enable global applicability of GridBuilder creating a base to port results of this base if the regional grid modelling studies from global grid modelling.

To facilitate generalisation, it can be handy to use composite_clean_build branch which has been created to incorporate into PyPSA-Eath grid modelling the approaches amended to it in PyPSA-Eur.

davide-f · 2026-09-28T23:06:22Z

Thanks Katia! This is to confirm that I'm going through it. The work is quite extensive and I'll need a few days to complete it

ekatef · 2026-10-01T07:53:43Z

A technical note: GridBuilder needs some technical fixes right now to enable compatibility with modelblocks. To facilitate things, it would be good to fix #4 first before merging this PR.

davide-f

Hi, nice extension to the DC! Comments follow.
It would be great to see two tests for example one image for Italy and one for Congo to see how the DC systems are rendered in the two systems.

Cross-linking this comment related to this PR #6 (comment)

davide-f · 2026-09-28T13:42:37Z

+
+    @field_validator("files")
+    @classmethod
+    def validate_file_names(cls, v: list[str]) -> list[str]:


Is the filename convenction necessary?

I'd recommend to include a validation of the minimum required columns of the files to ensure consistency with OSM-retrieved data

davide-f · 2026-09-28T13:43:15Z

    frame: gpd.GeoDataFrame,
    name: str,
    color: list[int] | str,
-    *,


is this wanted? It makes sense for robustness of the code

davide-f · 2026-09-28T22:42:47Z


 _VALID_REGIONS: frozenset[str] = frozenset(get_all_valid_codes())

+#: The fixed set of raw OSM features to enable compartibility with custom data.


Suggested change

#: The fixed set of raw OSM features to enable compartibility with custom data.

# The fixed set of raw OSM features to enable compatibility with custom data.

davide-f · 2026-09-28T22:44:02Z

+
+
+def _merge_country_codes(values: Any) -> str:
+    """Clean-up country codes for multy-country entries.


Suggested change

"""Clean-up country codes for multy-country entries.

"""Clean-up country codes for multi-country entries.

Please improve description to describe how codes are cleaned

For lines, it would be interesting to have country0 and country1 denoting the country of the endings. Can be an issue for future PRs

davide-f · 2026-09-28T22:48:24Z

+  # earth: repl_circuits. "2/3" and "2-1" would strip to 23 and 21.
+  - exact: ["2/3", "2"]


I am unsure of this. It would be worth to comment it and add a TODO to check it

davide-f · 2026-10-03T09:30:09Z

    return all_transformers[["transformer_id", *columns]]


+def _add_converters(


I'd recommend improving the docstrings and describe the approach.
That helps also understanding what priorities to give when adopting a common solution.

davide-f · 2026-10-03T09:30:56Z

+                    (ac_buses["voltage"] - dc_bus["voltage"]).abs().idxmin()
+                ]
+                pairing = "same_station"
+            elif station_id in converter_stations:


This if/elif condition defines a priority order that needs better documentation

davide-f · 2026-10-03T09:37:28Z

+    # An explicit zero means the way carries no live conductor: a ground wire
+    # ("ground") or a retired one ("1 disused"). Every branch below floors the
+    # count at one circuit, so without this the way would enter the network as
+    # a live single-circuit line. PyPSA-Earth reaches the same end by letting
+    # the zero through and dropping it in filter_circuits.
+    bool_no_conductors = (~df_lines["cleaned"]) & (
+        ((df_lines["cables"] == "0") & (df_lines["circuits"] == ""))
+        | (df_lines["circuits"] == "0")
+    )
+    df_lines.loc[bool_no_conductors, "circuits"] = "0"
+    df_lines.loc[bool_no_conductors, "cleaned"] = True


Have you identified such cases in OSM data?

davide-f · 2026-10-03T09:39:59Z

+    """Minimum AC voltage [V] for ``country``: its regional override, else the network default.
+
+    For a cross-border element the most permissive threshold wins, so an
+    interconnector survives whenever either side would keep it.
+    """


We may not need the overcomplication of a minimum voltage by country, rather one value for each execution. This is not a hard comment though

davide-f · 2026-10-03T17:35:38Z

+    # point at the same voltage must not share a bus.
    endpoints = (
-        endpoints.groupby(["voltage", "geometry"])
+        endpoints.groupby(["voltage", "dc", "geometry"])


As endpoints also depend on ac/dc, their bus_id may change accordingly. The risk is that AC and DC systems may have the same ID otherwise

ekatef added 30 commits September 26, 2026 13:11

Restore country field

2579c25

Polish LLMish wording

cdfe34e

Removing excessive details which stem from agentic testing

07e58b7

Bringing in global mappings

918107d

Add human TODOs to OSM corrections

e4816fa

Add OSM features to config validator

fd923da

Add treatment of country codes for cross-border elements

37bf53f

Clarify docstring

d683649

Restore under_construction values

b997f1b

Maintain output structure for empty entries

69f9d48

Correct citation

9cf45ee

Sketch custom data functionality

932fad7

Fix LMM artifacts in README

e2a2db9

Fix LLM artifacts in config/README

f42224b

Add TODO for CRS hardcoding in schema

a7edddd

Increase quality of README

544cfda

Fix refuse

5e2c23f

Quick fix of interactive frontend

7f55d29

Port frequency correction mappings

35a4d35

Generalize treatment of frequency

b1c1dab

Enhance cleanup

0d4f431

Improve treatment of cables

a26a6c7

Refactor output for clean script

d17ec65

Add frequency-related parameters to config and schema

b6fffa6

Bug-fix to enable customization of voltage filtering threshold

d1d1628

Integrate PyPSA-Earth approach for processing DC lines

a403ac2

Integrate DC field into filtering

53e2c99

Implement relationships into DC treatment

1a9c19f

Amend power field for HVDC

b3b9db5

Define function for force_ac treatment

2ba595d

ekatef added 20 commits September 26, 2026 17:53

Improve voltage filtering

23b2788

Improve treatment of empty geometries

4055f76

Add cables-to-underground transformation

922f240

Minor polishes

035520e

Integrate DC treatment into network build

5dec475

Implement adding converters

57aae73

Add DC part into script validation schema

8b46d6c

Add voltage threshold into script validation schema

92dea8c

Add converters to plotting

f04e049

Document DC and frequency treatment

9d4bf91

Fetch power field for DC lines

a29aaa5

Add processing of empty converters

ae2c1f0

Improve clarity of docstring

1098f50

Add converters to plotting

765b537

Minor clean-up

00b7bc9

Add details to description of buffer radius

d659cc1

Fix hardcoding

6cdbb66

Minor style fixes

43950b9

Apply formatting fixes

2fe33e8

Revise README

14caf00

ekatef requested a review from davide-f September 27, 2026 08:38

ekatef added 4 commits September 27, 2026 10:58

Add minimal testing config for Colombia

78fac48

Enhance test config

fdcc6cd

Sketch integration of newly introduced features

07d7d24

Externalize custom files input functionality

9745dfa

davide-f reviewed Oct 3, 2026

View reviewed changes


		_VALID_REGIONS: frozenset[str] = frozenset(get_all_valid_codes())

		#: The fixed set of raw OSM features to enable compartibility with custom data.

	#: The fixed set of raw OSM features to enable compartibility with custom data.
	# The fixed set of raw OSM features to enable compatibility with custom data.



		def _merge_country_codes(values: Any) -> str:
		"""Clean-up country codes for multy-country entries.

	"""Clean-up country codes for multy-country entries.
	"""Clean-up country codes for multi-country entries.

		# earth: repl_circuits. "2/3" and "2-1" would strip to 23 and 21.
		- exact: ["2/3", "2"]

		return all_transformers[["transformer_id", *columns]]


		def _add_converters(

Conversation

ekatef commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Disclaimer

Major changes

Reviewer checklist

Uh oh!

ekatef commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davide-f commented Sep 28, 2026

Uh oh!

ekatef commented Oct 1, 2026

Uh oh!

davide-f left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ekatef commented Sep 27, 2026 •

edited

Loading

ekatef commented Sep 27, 2026 •

edited

Loading