Skip to content

Replace Zenodo DEM tiles with EarthDataHub access #30

Open

Description

@LinhHo

Is your feature request related to a problem? Please describe.
Slope data are currently derived from 30 m-resolution elevation data (GEDTM30), which are stored on Zenodo as 30° × 30° tiles. This tile-based approach can be cumbersome when the selected region spans multiple tiles or covers only a small fraction of a tile, potentially causing unnecessary data downloads, storage overhead, and preprocessing time.

As illustrated by Issue #3, this currently requires additional clipping logic, implemented through a rio-clip wrapper.

Describe the solution you'd like
I propose exploring the use of the Copernicus DEM GLO-30 dataset on EarthDataHub, which provides 30 m-resolution elevation data through an API with Zarr-based access.

Instead of downloading and managing static tiles from Zenodo, the workflow could retrieve elevation data for the selected spatial extent directly from EarthDataHub. This could reduce unnecessary data transfers and local storage requirements while simplifying regional data access.

Describe alternatives you've considered
The alternative is to replace the current Zenodo data retrieval step with an EarthDataHub API request for the same spatial extent, leaving the downstream slope-calculation workflow unchanged as far as possible.

This approach would require an EarthDataHub access token and appropriate handling of authentication and data-access errors.

Additional context
This is a suggestion for enhancement. I have not thoroughly assessed the elevation data from EarthDataHub.

Activity

  1. irm-codebase commented on Oct 26, 2026

    @irm-codebase
    Member

    @LinhHo I agree on the move to EarthDataHub. Writing a snakemake wrapper that provides this should be feasible.
    We could couple this with #27 to enable the module to:

    • require an EarthDataHub key (as user input), which will be used to download a series of supported EarthDataHub sources (using the extent of the provided shapes to limit large downloads). The EarthDataHub wrapper is used internally to do this efficiently.
    • Alternatively, allow the user to provide their own rasters in case they want to use a particular datasource. See Enable user-provided files and standardise processing methods #27 for details.

    Out of curiosity, though, which version of the module are you running? There were some improvements to the GDAL clipping method I implemented some time ago that mitigate the rate limit issue (but do not fully avoid it).

  2. irm-codebase commented on Oct 27, 2026

    @irm-codebase
    Member

    @LinhHo additional question: are you using the tiny_files: True configuration option?

  3. LinhHo commented on Oct 29, 2026

    @LinhHo
    ContributorAuthor

    @irm-codebase Not yet! I just remember from what I worked on previously.
    But now I'm checking to run this area potentials globally. I might need to run with 20° × 20° tiles rather than using geoparquet of all countries, so your issue #27 might be relevant. Is there any progress on it? I might help a bit there if necessary.

    Another question: Do we have an option to run at a coarser resolution than GlobCover? I only need a resolution of ERA5 data of 0.25° × 0.25°.

  4. irm-codebase commented on Oct 29, 2026

    @irm-codebase
    Member

    @LinhHo not yet unfortunately. This is a feature I really want, but I won't have time to implement this for a while, unfortunately.

    Regarding running this globally: you could create a GeoParquet dataset that has the tile squares instead of countries. As long as you comply with the schema used by the module, it does not really matter what shapes you feed it.

  5. LinhHo commented on Nov 18, 2026

    @LinhHo
    ContributorAuthor

    @irm-codebase Sorry I only have time to get back to this now. A tile square doesn't work because this module requires shape_class, especially maritime for offshore wind. I will just loop through a country list. Stefan mentioned that the module can run in parallel to reduce the computing time. Is that correct?

    Also, what about the option to run at a coarser resolution than GlobCover, e.g., 0.25° × 0.25°? I can make a pull request if necessary.

  6. irm-codebase commented on Nov 19, 2026

    @irm-codebase
    Member

    No worries @LinhHo !
    Some feedback:

    Replacing the DEM tiles

    Unfortunately, I do not think that method would work well. The module processes things based on the provided shapefile's boundary (lat / lon, or equivalent if it is in a projected CRS). Looping over a list of countries would be very brittle: sometimes countries include remote colonies or contested territories. This will inevitably lead to issues as the spatial extent of whatever dataset you are trying to access (e.g., DEM GLO-30) might not match the dataset used to create the shapes.
    Examples:

    • France: it's NUTS datasets sometimes include colonies in Latin America.
    • China: Taiwan is contentious and its assignation differs between countries.

    The module does split processing per country_id or shape_id. So if you meant processing using the bounds of each of those cases, that would work. But it likely would be slow: it's often better to make one request to EarthDataHub for the full extent rather than many smaller ones because the rectangular bounds of each shape can overlap, leading to duplicated data download.

    My suggestion would be to implement #27.
    If this is adopted, the user could download whichever dataset they want and provide it as a filtering layer (DEM Glo 30 would be a continuous raster input). This would allow users to handle spatial extents to fit their needs, and to choose whichever sources they prefer. The module could then keep GEDTM30 as its default slope dataset, which users may choose to use if desired.

    Coarser resolution

    Running at a coarser resolution might be attractive, but that is a separate issue to this, so I'd treat it as such.
    It could just be a value users can configure, I suppose?

  7. LinhHo commented on Nov 19, 2026

    @LinhHo
    ContributorAuthor

    @irm-codebase Thanks for the suggestion! I agree that looping through all countries is troublesome. I will generate the tiles including polygons of land and maritime derived from a global geoboundaries file. I think this is different from #27 though, as I still use the same datasets provided by the module. The tiles will be the same as GEDTM30 to avoid overlapping downloaded files.

    Regarding the coarser resolution, land cover information from multiple pixels would be resampled into one large pixel and thus risk losing the technology's area potential when suitable land cover is a minority. Even when converting into fractions of land cover types, the module does not yet support land cover data with multiple layers. I still need to think about how to preserve this information better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions