Dask quickstart — NetCDF¶

Read one variable lazily, across time and across files. read_array(variable, chunks=...) returns a dask.array; open_mfdataset stacks many files into one. Kerchunk manifests (zero-copy byte-range indexes) need the [lazy] extra. For the full surface see lazy-netcdf-complete.

Setup¶

In [1]:

%matplotlib inline

from pathlib import Path

import dask.array as da
import numpy as np

DATA = Path('../../../examples/data')
DATA.is_dir()

%matplotlib inline from pathlib import Path import dask.array as da import numpy as np DATA = Path('../../../examples/data') DATA.is_dir()

Out[1]:

True

Eager vs lazy variable reads¶

In [2]:

from pyramids.netcdf import NetCDF

nc = NetCDF.read_file(DATA / 'netcdf' / 'cf__4v__1d3-3d1__proj__y-desc.nc')
nc.variable_names

from pyramids.netcdf import NetCDF nc = NetCDF.read_file(DATA / 'netcdf' / 'cf__4v__1d3-3d1__proj__y-desc.nc') nc.variable_names

2026-07-11 14:37:56 | INFO | pyramids.base.config | Logging is configured.

Out[2]:

['values']

nc is a concrete NetCDF. Plotting its values variable shows the cube that the lazy read_array(..., chunks=...) calls below read one step at a time.

In [3]:

nc.plot(variable='values', title="cf__4v__1d3-3d1__proj__y-desc — values")

nc.plot(variable='values', title="cf__4v__1d3-3d1__proj__y-desc — values")

Out[3]:

<cleopatra.array_glyph.ArrayGlyph at 0x7fa13291d2b0>

/home/runner/work/pyramids/pyramids/.pixi/envs/docs/lib/python3.14/site-packages/matplotlib/colors.py:816: RuntimeWarning: overflow encountered in multiply
  xa *= self.N

In [4]:

eager = nc.read_array('values')  # numpy
lazy = nc.read_array('values', chunks=(1, -1, -1))  # dask — one chunk per leading step
(type(eager).__name__, eager.shape), (type(lazy).__name__, lazy.chunks)

eager = nc.read_array('values') # numpy lazy = nc.read_array('values', chunks=(1, -1, -1)) # dask — one chunk per leading step (type(eager).__name__, eager.shape), (type(lazy).__name__, lazy.chunks)

Out[4]:

(('ndarray', (3, 13, 14)), ('Array', ((1, 1, 1), (13,), (14,))))

Reduce over the leading axis¶

In [5]:

# No I/O until compute().
per_cell_mean = lazy.mean(axis=0)
per_cell_mean.compute().shape

# No I/O until compute(). per_cell_mean = lazy.mean(axis=0) per_cell_mean.compute().shape

/home/runner/work/pyramids/pyramids/.pixi/envs/docs/lib/python3.14/site-packages/numpy/_core/_methods.py:49: RuntimeWarning: overflow encountered in reduce
  return umr_sum(a, axis, dtype, out, keepdims, initial, where)

Out[5]:

(13, 14)

Multi-file stacks — open_mfdataset¶

One variable at a time; returns a (n_files, *var_shape) dask array. Here we reuse one file three times to stand in for a real multi-file cube.

In [6]:

paths = [DATA / 'netcdf' / 'cf__4v__1d3-3d1__proj__y-desc.nc'] * 3
stack = NetCDF.open_mfdataset(paths, variable='values')
stack.shape, stack.mean(axis=0).compute().shape

paths = [DATA / 'netcdf' / 'cf__4v__1d3-3d1__proj__y-desc.nc'] * 3 stack = NetCDF.open_mfdataset(paths, variable='values') stack.shape, stack.mean(axis=0).compute().shape

/home/runner/work/pyramids/pyramids/.pixi/envs/docs/lib/python3.14/site-packages/numpy/_core/_methods.py:49: RuntimeWarning: overflow encountered in reduce
  return umr_sum(a, axis, dtype, out, keepdims, initial, where)

Out[6]:

((3, 3, 13, 14), (3, 13, 14))

Kerchunk manifests (optional)¶

to_kerchunk writes a JSON byte-range index so the file can be opened zero-copy by downstream tools. It needs the [lazy] extra — guard the call so the notebook still runs without it.

In [7]:

import tempfile

manifest = Path(tempfile.mkdtemp(prefix='pyramids-nc-')) / 'single.json'
try:
    refs = nc.to_kerchunk(manifest)
    outcome = 'wrote manifest:', manifest.exists()
except ImportError as exc:
    outcome = 'lazy extra missing:', str(exc)
outcome

import tempfile manifest = Path(tempfile.mkdtemp(prefix='pyramids-nc-')) / 'single.json' try: refs = nc.to_kerchunk(manifest) outcome = 'wrote manifest:', manifest.exists() except ImportError as exc: outcome = 'lazy extra missing:', str(exc) outcome

Out[7]:

('wrote manifest:', True)

Where to next¶

Three tiers for NetCDF + Dask:

• Complete cookbook (offline) — Lazy NetCDF: unpack, combined kerchunk manifests, handing arrays to xarray.
• Cloud (live data) — Dask on ERA5 (AWS): kerchunk over a decade of reanalysis.
• Narrative guide — the Lazy NetCDF tutorial.
• All classes at a glance — the overview.