NWP polish features — catalog metadata & guards (no network)¶

A tour of the nwp-polish follow-on changes against the bundled NWP catalog:

1. Per-model licence — every shipped row now self-describes its licence so downstream redistribution honours it.
2. Per-model retention_days + RetentionWarning — short-retention providers (DWD ICON ~1 day, Météo-France 14 days, ECCC 30 days) emit a warning when the requested start falls outside the window — replacing the silent "empty results" failure that confused users on the older release.
3. grid_kind — the declarative grid-type flag the new _aggregate guard uses to refuse icosahedral DWD ICON rows.

All cells run offline — no network, no SDK, no eccodes — so this is the fastest way to inspect what the polish work added.

Setup¶

In [1]:

from collections import Counter

import warnings

from earthlens.nwp import Catalog, NWP, NWPModel, RetentionWarning

from collections import Counter import warnings from earthlens.nwp import Catalog, NWP, NWPModel, RetentionWarning

1 — Per-model licence¶

Every shipped NWP model row carries an SPDX-style licence identifier the provider publishes the data under. Never inferred from the URL — populated row-by-row from the provider's stated terms.

In [2]:

cat = Catalog()
by_license = Counter(m.license for m in cat.datasets.values())
for licence, n in sorted(by_license.items(), key=lambda kv: -kv[1]):
    print(f'{licence!s:18s} {n:2d} model(s)')

cat = Catalog() by_license = Counter(m.license for m in cat.datasets.values()) for licence, n in sorted(by_license.items(), key=lambda kv: -kv[1]): print(f'{licence!s:18s} {n:2d} model(s)')

PD-US-GOV          16 model(s)
CC-BY-4.0          11 model(s)
OGL-Canada-2.0      4 model(s)
Etalab-2.0          2 model(s)

Look up one model's licence directly:

In [3]:

for key in ('gfs', 'ifs-hres', 'icon-eu', 'gdps', 'arpege-world'):
    m = cat.datasets[key]
    print(f'{key:14s}  provider={m.provider:18s}  licence={m.license}')

for key in ('gfs', 'ifs-hres', 'icon-eu', 'gdps', 'arpege-world'): m = cat.datasets[key] print(f'{key:14s} provider={m.provider:18s} licence={m.license}')

gfs             provider=noaa-nodd           licence=PD-US-GOV
ifs-hres        provider=ecmwf-opendata      licence=CC-BY-4.0
icon-eu         provider=dwd-opendata        licence=CC-BY-4.0
gdps            provider=eccc-msc            licence=OGL-Canada-2.0
arpege-world    provider=meteofrance         licence=Etalab-2.0

2 — Retention windows and RetentionWarning¶

Short-retention providers (DWD ICON ~1 day, Météo-France 14, ECCC 30) now self-describe their rolling window. Models the provider keeps archived (NOAA NODD, ECMWF Open Data) leave retention_days = None.

In [4]:

by_retention = Counter(m.retention_days for m in cat.datasets.values())
for window, n in sorted(by_retention.items(), key=lambda kv: (kv[0] is None, kv[0])):
    label = 'archival / unspecified' if window is None else f'{window} day(s)'
    print(f'{label:25s} {n:2d} model(s)')

by_retention = Counter(m.retention_days for m in cat.datasets.values()) for window, n in sorted(by_retention.items(), key=lambda kv: (kv[0] is None, kv[0])): label = 'archival / unspecified' if window is None else f'{window} day(s)' print(f'{label:25s} {n:2d} model(s)')

1 day(s)                   6 model(s)
14 day(s)                  3 model(s)
30 day(s)                  3 model(s)
archival / unspecified    21 model(s)

When the request's start is older than now - retention_days, the NWP backend emits a RetentionWarning at construction. The cell below fires the warning intentionally with warnings.catch_warnings so the notebook record stays clean.

In [5]:

from earthlens.nwp import Catalog
from earthlens.nwp.catalog import NWPModel

demo_catalog = Catalog(datasets={
    'icon-demo': NWPModel(
        provider='dwd-opendata', backend='direct-https', cycles_utc=[0, 12],
        horizon_h=48, idx=False, mirrors=['origin'],
        url_template='https://example.test/{var}.bz2',
        bands={'t2m': 'T_2M'}, retention_days=1,
    ),
})
with warnings.catch_warnings(record=True) as captured:
    warnings.simplefilter('always')
    NWP(
        start='2020-01-01', end='2020-01-01',
        variables={'icon-demo': ['t2m']},
        lat_lim=[48, 52], lon_lim=[6, 12], path='out',
        catalog=demo_catalog,
    )
for w in captured:
    if issubclass(w.category, RetentionWarning):
        print(str(w.message))

from earthlens.nwp import Catalog from earthlens.nwp.catalog import NWPModel demo_catalog = Catalog(datasets={ 'icon-demo': NWPModel( provider='dwd-opendata', backend='direct-https', cycles_utc=[0, 12], horizon_h=48, idx=False, mirrors=['origin'], url_template='https://example.test/{var}.bz2', bands={'t2m': 'T_2M'}, retention_days=1, ), }) with warnings.catch_warnings(record=True) as captured: warnings.simplefilter('always') NWP( start='2020-01-01', end='2020-01-01', variables={'icon-demo': ['t2m']}, lat_lim=[48, 52], lon_lim=[6, 12], path='out', catalog=demo_catalog, ) for w in captured: if issubclass(w.category, RetentionWarning): print(str(w.message))

'icon-demo' retains ~1 day(s); requested start 2020-01-01T00 is older than the retention cutoff at 2026-06-16T20 UTC — expect empty results.

3 — grid_kind and the icosahedral aggregate guard¶

Five DWD ICON rows ship as grid_kind=icosahedral; everything else defaults to regular-latlon. The new icosahedral guard on NWP._aggregate reads this declarative field instead of pattern-matching the URL, so it stays correct even if DWD renames a directory.

In [6]:

by_grid = Counter(m.grid_kind for m in cat.datasets.values())
for kind, n in sorted(by_grid.items()):
    print(f'{kind:18s} {n:2d} model(s)')
print()
ico = [k for k, m in cat.datasets.items() if m.grid_kind == 'icosahedral']
print('icosahedral rows:', sorted(ico))

by_grid = Counter(m.grid_kind for m in cat.datasets.values()) for kind, n in sorted(by_grid.items()): print(f'{kind:18s} {n:2d} model(s)') print() ico = [k for k, m in cat.datasets.items() if m.grid_kind == 'icosahedral'] print('icosahedral rows:', sorted(ico))

icosahedral         5 model(s)
regular-latlon     28 model(s)

icosahedral rows: ['icon-d2', 'icon-d2-eps', 'icon-eps', 'icon-eu-eps', 'icon-global']

Calling NWP._aggregate(...) on an icosahedral row raises a clean NotImplementedError with the griddable alternatives in its body:

In [7]:

from earthlens.aggregate import AggregationConfig
from pathlib import Path

b = NWP(
    start='2024-06-01', end='2024-06-01',
    variables={'icon-global': ['temperature_2m']},
    lat_lim=[40, 50], lon_lim=[0, 10], path='out',
)
try:
    b._aggregate([Path('placeholder.tif')], AggregationConfig(freq='1D', op='mean'))
except NotImplementedError as exc:
    print(type(exc).__name__, '-', exc)

from earthlens.aggregate import AggregationConfig from pathlib import Path b = NWP( start='2024-06-01', end='2024-06-01', variables={'icon-global': ['temperature_2m']}, lat_lim=[40, 50], lon_lim=[0, 10], path='out', ) try: b._aggregate([Path('placeholder.tif')], AggregationConfig(freq='1D', op='mean')) except NotImplementedError as exc: print(type(exc).__name__, '-', exc)

NotImplementedError - NWP aggregate: 'icon-global' is on an icosahedral grid (not a regular lat/lon raster); aggregation is not supported. Request a griddable model (ICON-EU, or a regridded global feed) instead.

C:\gdrive\algorithms\remote-sensing\earthlens\.claude\worktrees\nwp-polish\src\earthlens\base\abstractdatasource.py:560: RetentionWarning: 'icon-global' retains ~1 day(s); requested start 2024-06-01T00 is older than the retention cutoff at 2026-06-16T20 UTC — expect empty results.
  orig(self, *args, **kw)

Recap¶

• The catalog now self-describes licence, retention, and grid-type — no callers have to infer those from the URL.
• Short-retention providers fail loud (warning) instead of silently empty.
• Icosahedral DWD ICON rows refuse aggregation via the declarative grid_kind flag — see eccc_gdps.ipynb for the other half of the polish work (the new ECCC MSC Datamart centre).