NWP polish features — catalog metadata & guards (no network)¶

A tour of the nwp-polish follow-on changes against the bundled NWP catalog:

1. Per-model licence — every shipped row now self-describes its licence so downstream redistribution honours it.
2. Per-model retention_days + RetentionWarning — short-retention providers (DWD ICON ~1 day, Météo-France 14 days, ECCC 30 days) emit a warning when the requested start falls outside the window — replacing the silent "empty results" failure that confused users on the older release.
3. grid_kind — the declarative grid-type flag the new _aggregate guard uses to refuse icosahedral DWD ICON rows.

All cells run offline — no network, no SDK, no eccodes — so this is the fastest way to inspect what the polish work added.

Setup¶

In [ ]:

import warnings
from collections import Counter

from earthlens.nwp import NWP, Catalog, NWPModel, RetentionWarning

import warnings from collections import Counter from earthlens.nwp import NWP, Catalog, NWPModel, RetentionWarning

1 — Per-model licence¶

Every shipped NWP model row carries an SPDX-style licence identifier the provider publishes the data under. Never inferred from the URL — populated row-by-row from the provider's stated terms.

In [ ]:

cat = Catalog()
by_license = Counter(m.license for m in cat.datasets.values())
for licence, n in sorted(by_license.items(), key=lambda kv: -kv[1]):
    print(f'{licence!s:18s} {n:2d} model(s)')

cat = Catalog() by_license = Counter(m.license for m in cat.datasets.values()) for licence, n in sorted(by_license.items(), key=lambda kv: -kv[1]): print(f'{licence!s:18s} {n:2d} model(s)')

Look up one model's licence directly:

In [ ]:

for key in ('gfs', 'ifs-hres', 'icon-eu', 'gdps', 'arpege-world'):
    m = cat.datasets[key]
    print(f'{key:14s}  provider={m.provider:18s}  licence={m.license}')

for key in ('gfs', 'ifs-hres', 'icon-eu', 'gdps', 'arpege-world'): m = cat.datasets[key] print(f'{key:14s} provider={m.provider:18s} licence={m.license}')

2 — Retention windows and RetentionWarning¶

Short-retention providers (DWD ICON ~1 day, Météo-France 14, ECCC 30) now self-describe their rolling window. Models the provider keeps archived (NOAA NODD, ECMWF Open Data) leave retention_days = None.

In [ ]:

by_retention = Counter(m.retention_days for m in cat.datasets.values())
for window, n in sorted(by_retention.items(), key=lambda kv: (kv[0] is None, kv[0])):
    label = 'archival / unspecified' if window is None else f'{window} day(s)'
    print(f'{label:25s} {n:2d} model(s)')

by_retention = Counter(m.retention_days for m in cat.datasets.values()) for window, n in sorted(by_retention.items(), key=lambda kv: (kv[0] is None, kv[0])): label = 'archival / unspecified' if window is None else f'{window} day(s)' print(f'{label:25s} {n:2d} model(s)')

When the request's start is older than now - retention_days, the NWP backend emits a RetentionWarning at construction. The cell below fires the warning intentionally with warnings.catch_warnings so the notebook record stays clean.

In [ ]:

from earthlens.nwp import Catalog

demo_catalog = Catalog(
    datasets={
        'icon-demo': NWPModel(
            provider='dwd-opendata',
            backend='direct-https',
            cycles_utc=[0, 12],
            horizon_h=48,
            idx=False,
            mirrors=['origin'],
            url_template='https://example.test/{var}.bz2',
            bands={'t2m': 'T_2M'},
            retention_days=1,
        ),
    }
)
with warnings.catch_warnings(record=True) as captured:
    warnings.simplefilter('always')
    NWP(
        start='2020-01-01',
        end='2020-01-01',
        variables={'icon-demo': ['t2m']},
        lat_lim=[48, 52],
        lon_lim=[6, 12],
        path='out',
        catalog=demo_catalog,
    )
for w in captured:
    if issubclass(w.category, RetentionWarning):
        print(str(w.message))

from earthlens.nwp import Catalog demo_catalog = Catalog( datasets={ 'icon-demo': NWPModel( provider='dwd-opendata', backend='direct-https', cycles_utc=[0, 12], horizon_h=48, idx=False, mirrors=['origin'], url_template='https://example.test/{var}.bz2', bands={'t2m': 'T_2M'}, retention_days=1, ), } ) with warnings.catch_warnings(record=True) as captured: warnings.simplefilter('always') NWP( start='2020-01-01', end='2020-01-01', variables={'icon-demo': ['t2m']}, lat_lim=[48, 52], lon_lim=[6, 12], path='out', catalog=demo_catalog, ) for w in captured: if issubclass(w.category, RetentionWarning): print(str(w.message))

3 — grid_kind and the icosahedral aggregate guard¶

Five DWD ICON rows ship as grid_kind=icosahedral; everything else defaults to regular-latlon. The new icosahedral guard on NWP._aggregate reads this declarative field instead of pattern-matching the URL, so it stays correct even if DWD renames a directory.

In [ ]:

by_grid = Counter(m.grid_kind for m in cat.datasets.values())
for kind, n in sorted(by_grid.items()):
    print(f'{kind:18s} {n:2d} model(s)')
print()
ico = [k for k, m in cat.datasets.items() if m.grid_kind == 'icosahedral']
print('icosahedral rows:', sorted(ico))

by_grid = Counter(m.grid_kind for m in cat.datasets.values()) for kind, n in sorted(by_grid.items()): print(f'{kind:18s} {n:2d} model(s)') print() ico = [k for k, m in cat.datasets.items() if m.grid_kind == 'icosahedral'] print('icosahedral rows:', sorted(ico))

Calling NWP._aggregate(...) on an icosahedral row raises a clean NotImplementedError with the griddable alternatives in its body:

In [ ]:

from pathlib import Path

from earthlens.aggregate import AggregationConfig

b = NWP(
    start='2024-06-01',
    end='2024-06-01',
    variables={'icon-global': ['temperature_2m']},
    lat_lim=[40, 50],
    lon_lim=[0, 10],
    path='out',
)
try:
    b._aggregate([Path('placeholder.tif')], AggregationConfig(freq='1D', op='mean'))
except NotImplementedError as exc:
    print(type(exc).__name__, '-', exc)

from pathlib import Path from earthlens.aggregate import AggregationConfig b = NWP( start='2024-06-01', end='2024-06-01', variables={'icon-global': ['temperature_2m']}, lat_lim=[40, 50], lon_lim=[0, 10], path='out', ) try: b._aggregate([Path('placeholder.tif')], AggregationConfig(freq='1D', op='mean')) except NotImplementedError as exc: print(type(exc).__name__, '-', exc)

Recap¶

• The catalog now self-describes licence, retention, and grid-type — no callers have to infer those from the URL.
• Short-retention providers fail loud (warning) instead of silently empty.
• Icosahedral DWD ICON rows refuse aggregation via the declarative grid_kind flag — see eccc_gdps.ipynb for the other half of the polish work (the new ECCC MSC Datamart centre).