MSWEP / MSWX catalog explorer¶

This notebook needs no credentials and no network. It reads the bundled earthlens.mswep catalog to answer the questions that decide every MSWEP request: which products exist, how their Drive paths differ, which variant can serve a given date, and what a granule will be called.

The actual download does need access — GloH2O grants it per person via a request form. Once approved, the folder is link-shared, so any credential that reads Drive works, a service-account key included. See Authentication. The last section shows the download call without running it.

The two products¶

MSWEP is merged precipitation; MSWX is the companion meteorological forcing. They are catalogued together because they share a transport, but they do not share a path shape.

In [ ]:

from earthlens.mswep import Catalog

catalog = Catalog()
catalog.products()

from earthlens.mswep import Catalog catalog = Catalog() catalog.products()

In [ ]:

for key in catalog.products():
    product = catalog.get_product(key)
    print(f'{key:6} {product.path_template}')

for key in catalog.products(): product = catalog.get_product(key) print(f'{key:6} {product.path_template}')

MSWX carries a {variable} level that MSWEP does not. That single difference is why the template lives in the catalog rather than in the backend: one shared f-string would silently build MSWX paths that do not exist, and because a missing granule is logged and skipped, the request would come back empty instead of failing.

In [ ]:

mswep, mswx = catalog.get_product('mswep'), catalog.get_product('mswx')
print('mswep needs a variable folder:', mswep.needs_variable_folder)
print('mswx  needs a variable folder:', mswx.needs_variable_folder)

mswep, mswx = catalog.get_product('mswep'), catalog.get_product('mswx') print('mswep needs a variable folder:', mswep.needs_variable_folder) print('mswx needs a variable folder:', mswx.needs_variable_folder)

Versions and the share layout¶

GloH2O shares one folder per version, and the folder_id you are given is that version root (its children are the variants). The version selects catalog metadata, not the data. The trend caveat still matters: V3.15/V3.16 read artificially low over 2000-2015, and GloH2O recommends V2.80 for trend work.

In [ ]:

for version, row in sorted(mswep.versions.items()):
    flag = ' (provisional)' if row.provisional else ''
    print(f'{version:5} -> {row.root}{flag}')
    if row.description:
        print(f'        {row.description.strip()[:90]}')

for version, row in sorted(mswep.versions.items()): flag = ' (provisional)' if row.provisional else '' print(f'{version:5} -> {row.root}{flag}') if row.description: print(f' {row.description.strip()[:90]}')

The v3.16 folder is really named MSWEP_V316_test (the _test suffix, matching GloH2O's MSWEP_V315_test pattern) -- confirmed against the live share. Earlier releases of this backend flagged unverified values provisional and refused them; the share has since been walked, and every provisional flag has been dropped.

In [ ]:

mswep.versions['3.16'].root, mswep.versions['3.16'].provisional

mswep.versions['3.16'].root, mswep.versions['3.16'].provisional

Variants are chosen by date¶

Past and Past_nogauge cover 1979–2024; NRT starts in 2025 and runs to about two hours from real time. Omit variant= and each timestep routes itself, so a window crossing the boundary spans both.

In [ ]:

for name, row in mswep.variants.items():
    print(f'{name:13} {row.start} -> {row.end or "now"}')

for name, row in mswep.variants.items(): print(f'{name:13} {row.start} -> {row.end or "now"}')

In [ ]:

import datetime as dt

for day in ['2020-04-25', '2024-12-31', '2025-01-01', '2026-06-01']:
    date = dt.date.fromisoformat(day)
    print(f'{day}  ->  {mswep.variant_for(date)}')

import datetime as dt for day in ['2020-04-25', '2024-12-31', '2025-01-01', '2026-06-01']: date = dt.date.fromisoformat(day) print(f'{day} -> {mswep.variant_for(date)}')

What a granule will be called¶

File names follow YYYYDOY.HH for hourly and 3-hourly (HH is the accumulation's starting hour), YYYYDOY daily and YYYYMM monthly. Note the upstream's mixed folder casing — Hourly but 3hourly — which the catalog preserves verbatim rather than normalising.

In [ ]:

stamp = dt.datetime(2020, 4, 25, 18)

for key, row in mswep.resolutions.items():
    name = f'{stamp.strftime(row.stem)}.nc'
    print(f'{key:9} {row.folder:8} {name:16} {row.units}')

stamp = dt.datetime(2020, 4, 25, 18) for key, row in mswep.resolutions.items(): name = f'{stamp.strftime(row.stem)}.nc' print(f'{key:9} {row.folder:8} {name:16} {row.units}')

Putting it together, this is the Drive path a daily request resolves to — the example straight out of the MSWEP V3.16 documentation:

In [ ]:

variant = mswep.variant_for(stamp.date())
hourly = mswep.resolutions['hourly']
root = mswep.versions['3.15'].root

print(
    mswep.path_template.format(
        root=root,
        variant=variant,
        temporal=hourly.folder,
        stem=stamp.strftime(hourly.stem),
    )
)

variant = mswep.variant_for(stamp.date()) hourly = mswep.resolutions['hourly'] root = mswep.versions['3.15'].root print( mswep.path_template.format( root=root, variant=variant, temporal=hourly.folder, stem=stamp.strftime(hourly.stem), ) )

MSWX variables and forecast streams¶

For MSWX, variables= names a Drive folder. All ten are confirmed against the share. MSWX also has two ensemble forecast streams, the folders Mid and Long, fetchable via variant=/init=/members= (keyed by init time, member and lead).

In [ ]:

print('variables:', list(mswx.variables))
for name in ('Mid', 'Long'):
    row = mswx.variants[name]
    print(f'{name}: {row.members} {row.base_model} members, {row.horizon}')

print('variables:', list(mswx.variables)) for name in ('Mid', 'Long'): row = mswx.variants[name] print(f'{name}: {row.members} {row.base_model} members, {row.horizon}')

Licence¶

MSWEP and MSWX are CC BY-NC 4.0 — non-commercial only, attribution required. Every download() emits a LicenseWarning carrying this citation.

In [ ]:

print(catalog.license_id)
print()
print(catalog.attribution)

print(catalog.license_id) print() print(catalog.attribution)

Running an actual download¶

The cell below is not executed here — it needs an approved GloH2O share, which is granted per person. With access configured it returns the list of .nc paths written.

from earthlens.core import EarthLens

paths = EarthLens(
    'mswep',
    start='2020-04-25',
    end='2020-04-30',
    variables=['precipitation'],
    temporal_resolution='daily',
    path='out',
).download()

earthlens ships the granules raw and does not decode them — reading and clipping NetCDF is pyramids' job:

from pyramids.netcdf import NetCDF

NetCDF.read_file(paths[0]).subset(bounds=(-20, 0, 55, 40))

For a large window use rclone sync instead: an hourly year is ~8760 granules, and GloH2O asks non-commercial users to transfer in bulk that way.