Skip to content

Data Sources#

Design Concept#

earthlens is designed following the Template/Factory design pattern to create an abstract class as a template for different data sources.

The main objective is to provide a unified API for all remote sensing data sources, where you only have to worry about the domain of your data (date range and spatial extent) and the package does everything in the backend.

earthlens provides a unified API across 61 providers — see All supported providers below for the full index, or jump straight to a provider's own reference page (docs/reference/<id>/introduction.md) for its full walkthrough.

Note

Many data sources (Google Earth Engine, ECMWF, EUMETSAT, Sentinel Hub, …) require authentication keys. See the Authentication page for setup instructions, or each provider's own reference page for provider-specific credential details.

The API takes a few parameters to determine the domain of your data:

  • Date range: start, end, and temporal_resolution
  • Spatial extent: lat_lim (latitude limits) and lon_lim (longitude limits)
  • If lat_lim and lon_lim are not provided, the EarthLens class defaults to longitude [-180, 180] and latitude [-90, 90].
from earthlens.core import EarthLens

start = "2009-01-01"
end = "2009-01-10"
temporal_resolution = "daily"
latlim = [4.19, 4.64]
lonlim = [-75.65, -74.73]

Each data source has different climate variables/datasets. To discover available variables, use the Catalog class for each data source (see Data Catalog).

Info

The downloaded data format differs based on the data source. CHIRPS and ECMWF have a post_download function that converts the NetCDF format into GeoTIFF using the pyramids GIS package.

Note

In future versions, lat_lim and lon_lim will be deprecated and replaced by a GeoDataFrame containing a polygon geometry.

All supported providers#

Each provider's id links to its own reference page for the full walkthrough, authentication details, and catalog. Pass the data_source value to EarthLens(...).

Air quality#

Provider data_source
AirNow (US EPA) airnow
European Environment Agency eea-aq
OpenAQ openaq
Sensor.Community sensor-community

Biodiversity & protected areas#

Provider data_source
GBIF gbif
IUCN Red List iucn
OBIS obis
Protected Planet (UNEP-WCMC) wdpa

Climate reanalysis & projections#

Provider data_source
Copernicus Climate Data Store (ECMWF) ecmwf
NOAA Physical Sciences Laboratory climate-indices
WCRP CMIP6 cmip6

Disasters & risk#

Provider data_source
FDSN fdsn
GDACS gdacs
EM-DAT disaster impacts emdat, gdis
NASA FIRMS firms
ThinkHazard! (GFDRR/World Bank) thinkhazard

Elevation & bathymetry#

Provider data_source
Copernicus DEM (ESA) dem
GEBCO bathymetry

Glaciers & cryosphere#

Provider data_source
NSIDC Randolph Glacier Inventory glaciers

Humanitarian data#

Provider data_source
Humanitarian Data Exchange (UN OCHA) hdx

Hydrology#

Provider data_source
GloH2O MSWEP / MSWX mswep, mswx
NOAA National Water Model nwm
USGS National Water Information System usgs-water

Multi-mission imagery & data platforms#

Provider data_source
AWS Open Data amazon-s3
EUMETSAT eumetsat
Google Earth Engine gee
JAXA jaxa
NASA Earthdata earthdata
NOAA GOES-R goes
STAC (SpatioTemporal Asset Catalog) stac
Sentinel Hub sentinel-hub
openEO openeo

Ocean#

Provider data_source
Argo Program argo
Copernicus Marine Service cmems
NOAA ERDDAP erddap

Population & human settlement#

Provider data_source
European Commission Joint Research Centre (GHSL) ghsl
WorldPop worldpop

Precipitation & drought#

Provider data_source
Climate Hazards Center (UCSB) chc
Copernicus European Drought Observatory / NDMC drought

Renewable energy#

Provider data_source
Global Solar Atlas / Global Wind Atlas (World Bank/ESMAP) solar-wind-atlas
National Laboratory of the Rockies (formerly NREL) nrel
PVGIS (EU JRC) pvgis

SAR / radar imagery#

Provider data_source
Alaska Satellite Facility (ASF) asf

Soil#

Provider data_source
ISRIC SoilGrids soilgrids

Tropical cyclones#

Provider data_source
Tropycal tropycal

Vector basemaps & boundaries#

Provider data_source
OpenStreetMap osm
Overture Maps Foundation overture
geoBoundaries admin

Weather forecast (NWP)#

Provider data_source
Herbie (NWP archive access) nwp

Weather radar#

Provider data_source
NOAA NEXRAD radar

Logos are each provider's own mark, used only to identify which service a backend talks to (not an endorsement of earthlens by that provider) — see docs/_images/logos/ATTRIBUTION.md for sourcing and rights notes on every logo.

Quick examples#

A few backends' end-to-end usage, worked out in full below. Every other provider's own introduction page has the same kind of walkthrough.

ECMWF (Copernicus Climate Data Store)#

The ECMWF backend talks to the Copernicus Climate Data Store via cdsapi. ERA-Interim was retired in 2019 and the public-datasets endpoint that hosted it was decommissioned in 2023; ERA5 on CDS is the production successor and what every ECMWF retrieve in this package now hits. Set up your ~/.cdsapirc first (see Authentication) and accept the licence for the relevant ERA5 dataset on the CDS website.

source = "ecmwf"
path = "examples/data/era5"
# Variables are addressed by (CDS dataset short name, variable code).
variables = {
    "reanalysis-era5-single-levels": ["2m-temperature"],
}

earthlens = EarthLens(
    data_source=source,
    start=start,
    end=end,
    variables=variables,
    lat_lim=latlim,
    lon_lim=lonlim,
    temporal_resolution=temporal_resolution,
    path=path,
)
earthlens.download()

Expect to wait

client.retrieve() blocks until the request reaches the front of the CDS queue and the file is generated — typically minutes, occasionally longer for large requests. Pick a small bbox and date range to keep wait times bearable. In CI the cdsapi client is mocked; the live end-to-end suite is selected with pytest -m e2e.

CHC (CHIRPS / CHIRP / CHIRTS / …)#

source = "chc"
path = "examples/data/chirps"
variables = ["precipitation"]

earthlens = EarthLens(
    data_source=source,
    start=start,
    end=end,
    variables=variables,
    lat_lim=latlim,
    lon_lim=lonlim,
    temporal_resolution=temporal_resolution,
    path=path,
)
earthlens.download()

Parallel Download#

path = "examples/data/chirps-cores"

earthlens = EarthLens(
    data_source=source,
    start=start,
    end=end,
    variables=variables,
    lat_lim=latlim,
    lon_lim=lonlim,
    temporal_resolution=temporal_resolution,
    path=path,
)
earthlens.download(cores=4)

Amazon S3#

path = "examples/data/s3-backend"
source = "amazon-s3"
variables = ["precipitation"]

earthlens = EarthLens(
    data_source=source,
    start=start,
    end=end,
    variables=variables,
    temporal_resolution=temporal_resolution,
    path=path,
)
earthlens.download()