Subsetting, the retrospective archive, and what's deferred¶

A plain operational request downloads whole-CONUS files. A subset (sites= / bbox) or the retrospective archive is read — not downloaded whole — through pyramids' LabeledDataset reader (pyramids >= 0.38.0). earthlens never imports xarray/zarr itself.

For the tabular products (chrtout, lakeout, coastal) the reader opens the store anonymously + lazily, slices, and writes a tidy feature_id x time Parquet table. This notebook does one live retrospective subset and shows what is still deferred.

Setup¶

A small set of imports: tempfile/Path for a scratch output directory, pandas to read the resulting Parquet table, and the unified EarthLens entry point.

In [ ]:

import tempfile
from pathlib import Path

import pandas as pd

from earthlens import EarthLens

import tempfile from pathlib import Path import pandas as pd from earthlens import EarthLens

Live: retrospective streamflow for three reaches¶

Opens the 1.4 TB retrospective chrtout.zarr anonymously, slices to three feature_ids and a two-day window, and writes a Parquet table — no whole-store download.

Build the request¶

Describe the retrospective subset up front — dataset, variable, date window, the three feature_ids via sites=, and a scratch path to write the Parquet table into.

In [ ]:

nwm = EarthLens(
    data_source="nwm",
    start='2010-06-01',
    end='2010-06-02',
    dataset='chrtout',
    variables=['streamflow'],
    aoi=[-180, -90, 180, 90],
    configuration='analysis_assim',
    mode='retrospective',
    sites=[101, 179, 181],
    path=tempfile.mkdtemp(prefix='nwm_retro_'),
)

nwm = EarthLens( data_source="nwm", start='2010-06-01', end='2010-06-02', dataset='chrtout', variables=['streamflow'], aoi=[-180, -90, 180, 90], configuration='analysis_assim', mode='retrospective', sites=[101, 179, 181], path=tempfile.mkdtemp(prefix='nwm_retro_'), )

Download the subset¶

download() opens the store lazily, slices to the requested reaches and window, and returns the list of written Parquet paths.

In [ ]:

paths = nwm.download(progress_bar=False)

paths = nwm.download(progress_bar=False)

Inspect the table¶

Read the first written Parquet file back and peek at its shape and head.

In [ ]:

df = pd.read_parquet(paths[0])
print(Path(paths[0]).name, '|', df.shape)
df.head()

df = pd.read_parquet(paths[0]) print(Path(paths[0]).name, '|', df.shape) df.head()

The result is a tidy table — one row per (feature_id, time) — with the streamflow plus the in-file latitude/longitude/gage_id coords.

In [ ]:

print('feature_ids:', sorted(df['feature_id'].unique().tolist()))
print('columns:', list(df.columns))

print('feature_ids:', sorted(df['feature_id'].unique().tolist())) print('columns:', list(df.columns))

USGS gage_id join¶

sites= also accepts USGS gage_id strings — earthlens routes those to the in-file gage_id coordinate join instead of feature_id.

Build a mixed sites= request¶

Mix an integer feature_id with a gage_id string in the same sites= list so we can see how each is routed (enumerated here, not downloaded).

In [ ]:

g = EarthLens(
    data_source="nwm",
    start='2010-06-01',
    end='2010-06-02',
    dataset='chrtout',
    variables=['streamflow'],
    aoi=[-180, -90, 180, 90],
    configuration='analysis_assim',
    mode='retrospective',
    sites=[101, '01010000'],
)

g = EarthLens( data_source="nwm", start='2010-06-01', end='2010-06-02', dataset='chrtout', variables=['streamflow'], aoi=[-180, -90, 180, 90], configuration='analysis_assim', mode='retrospective', sites=[101, '01010000'], )

Inspect the routing¶

_feature_ids() and _gage_ids() show the split: integer ids go to the feature_id index, strings to the gage_id coordinate join.

In [ ]:

print('feature_id ints:', g._feature_ids(), '| gage_id strings:', g._gage_ids())

print('feature_id ints:', g._feature_ids(), '| gage_id strings:', g._gage_ids())

Still deferred: subsetting the gridded products¶

ldasout / rtout / forcing are gridded; subsetting them (or reading their retrospective cube) needs a gridded cloud-cube reader pyramids does not yet expose, so it raises a clear NotImplementedError. Their whole-file operational download works.

A small error-reporting helper¶

show() runs a thunk and prints either success or the truncated NotImplementedError message, so the deferred cases read cleanly.

In [ ]:

def show(label, fn):
    try:
        fn()
        print(f'{label}: (no error)')
    except NotImplementedError as exc:
        print(f'{label}: NotImplementedError -> {str(exc)[:70]}...')

def show(label, fn): try: fn() print(f'{label}: (no error)') except NotImplementedError as exc: print(f'{label}: NotImplementedError -> {str(exc)[:70]}...')

Build a gridded retrospective request¶

Same shape as before but pointed at the gridded ldasout product with a sites= subset — the combination the reader cannot yet satisfy.

In [ ]:

grid = EarthLens(
    data_source="nwm",
    start='2010-06-01',
    end='2010-06-02',
    dataset='ldasout',
    variables=['SOIL_M'],
    aoi=[-180, -90, 180, 90],
    configuration='analysis_assim',
    mode='retrospective',
    sites=[101],
)

grid = EarthLens( data_source="nwm", start='2010-06-01', end='2010-06-02', dataset='ldasout', variables=['SOIL_M'], aoi=[-180, -90, 180, 90], configuration='analysis_assim', mode='retrospective', sites=[101], )

Confirm it is deferred¶

Attempting the gridded retrospective subset raises a clear NotImplementedError.

In [ ]:

show('gridded retrospective subset', lambda: grid.download(progress_bar=False))

show('gridded retrospective subset', lambda: grid.download(progress_bar=False))

aggregate= is rejected¶

chrtout is feature-id indexed (not griddable) and a gridded reduce needs a separate reader, so the temporal aggregator is unsupported.

Build a chrtout request¶

A plain feature-id-indexed chrtout request — no sites= subset — to exercise the aggregator path.

In [ ]:

agg = EarthLens(
    data_source="nwm",
    start='2010-06-01',
    end='2010-06-02',
    dataset='chrtout',
    variables=['streamflow'],
    aoi=[-180, -90, 180, 90],
    configuration='analysis_assim',
    mode='retrospective',
)

agg = EarthLens( data_source="nwm", start='2010-06-01', end='2010-06-02', dataset='chrtout', variables=['streamflow'], aoi=[-180, -90, 180, 90], configuration='analysis_assim', mode='retrospective', )

Confirm aggregate= is rejected¶

Passing any aggregate= value raises a clear NotImplementedError.

In [ ]:

show('aggregate=', lambda: agg.download(aggregate=object()))

show('aggregate=', lambda: agg.download(aggregate=object()))