NWP download mode — subset vs whole (no network)¶

The NWP backend downloads only the requested bands by default, using each model's .idx byte-range index where it has one — cutting >99 % of the bytes off the wire. The mode= constructor kwarg lets you override that:

mode=	behaviour
"subset" (default)	.idx byte-range subset where available, else whole-per-variable
"whole"	force a full-file download even for .idx-capable models, then crop
"zarr"	rejected — no NWP row carries a zarr_url (a documented follow-on)

This notebook shows the mode API offline — no bucket access. The live size-difference proof (whole file > subset file) runs in the gated e2e suite (pytest -m "nwp and e2e").

Setup¶

In [1]:

from earthlens.nwp import NWP

# A tiny request skeleton reused across the cells below.
REQ = dict(
    start='2024-06-01', end='2024-06-01',
    variables={'gfs': ['temperature_2m']},
    lat_lim=[40, 45], lon_lim=[-80, -75], path='out/mode-demo',
)

from earthlens.nwp import NWP # A tiny request skeleton reused across the cells below. REQ = dict( start='2024-06-01', end='2024-06-01', variables={'gfs': ['temperature_2m']}, lat_lim=[40, 45], lon_lim=[-80, -75], path='out/mode-demo', )

1 — subset is the default¶

Constructing without mode= selects the byte-range subset path. No download happens at construction — the mode is just recorded for the later .download().

In [2]:

backend = NWP(**REQ)
print('default mode:', backend._mode)

backend = NWP(**REQ) print('default mode:', backend._mode)

default mode: subset

2 — mode="whole" forces a full-file download¶

whole pulls the entire GRIB2 for each (cycle, step) even for .idx-capable models (the NOAA / Herbie ones), then crops to the bbox exactly like a subset. It only changes behaviour for the .idx centres — ECMWF Open Data, DWD, ECCC and Météo-France are already whole-per-variable, so they accept and ignore it.

In [3]:

backend = NWP(**REQ, mode='whole')
print('mode:', backend._mode)

backend = NWP(**REQ, mode='whole') print('mode:', backend._mode)

mode: whole

3 — mode="zarr" is rejected¶

No NWP catalog row carries a zarr_url, and Zarr sources (NWM, hrrrzarr) are separate backends — so zarr is rejected at construction with a clear message.

In [4]:

try:
    NWP(**REQ, mode='zarr')
except ValueError as exc:
    print('ValueError:', exc)

try: NWP(**REQ, mode='zarr') except ValueError as exc: print('ValueError:', exc)

ValueError: mode='zarr' is not supported: no NWP catalog row carries a `zarr_url`, and Zarr sources (NWM, hrrrzarr) are separate backends. Use mode='subset' (default) or mode='whole'.

4 — the facade forwards mode= too¶

The unified EarthLens facade passes any extra kwarg through to the backend, so mode= works the same way there — this is how you'd use it in practice (here we only construct; add .download() against the live bucket to fetch).

In [5]:

from earthlens.earthlens import EarthLens

lens = EarthLens(data_source='nwp', mode='whole', **REQ)
# the facade stores the constructed backend as `.datasource`
print('facade wired mode=whole ->', lens.datasource._mode)

from earthlens.earthlens import EarthLens lens = EarthLens(data_source='nwp', mode='whole', **REQ) # the facade stores the constructed backend as `.datasource` print('facade wired mode=whole ->', lens.datasource._mode)

facade wired mode=whole -> whole

Recap¶

• subset (default) → .idx byte-range subset (>99 % less bandwidth) where the model has an index, else whole-per-variable.
• whole → force the full file even for .idx models, then crop.
• zarr → rejected (documented follow-on, pending a zarr_url field).
• mode= only changes behaviour for the NOAA / Herbie centre; the other centres accept and ignore it.

See the NWP usage guide for the full write-up.