Skip to content

Lazy raster reads#

Backs NetCDF.read_array(chunks=…). Returns a dask.array.Array instead of a NumPy array, without dask being imported until the call happens.

Dataset.read_array does not route here. It accepts chunks= and returns a dask.array.Array, but it is served by core's own in-core lazy raster path (pyramids.dataset.engines.IO.read_array, which never calls require_compute_backend). Every read_array_lazy call site in core lives under pyramids/netcdf/ (netcdf.py, engines/interop.py, ugrid/interop_xarray.py).

So a lazy Dataset read needs dask but not this package, whereas a lazy NetCDF read needs this package. Dataset's Zarr and kerchunk operations do come here — see Zarr (Dataset).

Backend method read_array_lazy
Module pyramids_lazy._lazy
Returns dask.array.Array
from pyramids.netcdf import NetCDF

cube = NetCDF.read_file("era5.nc")
arr = cube.read_array(variable="pr", chunks={"rows": 64, "cols": 64})
arr.mean().compute()

How the graph is built#

The array is assembled from a low-level task graph — one pickle-safe reader task per block — rather than from dask.array.from_array. That is what makes a distributed read work: each task carries the path and window, opens the dataset through a CachingFileManager, and reads only its own window. The manager is the reason a worker handling many blocks of one file opens it once, and the reason the handle is shared process-globally rather than per task.

chunks accepts "auto", a single int, a sequence per axis, or a dict keyed by rows / cols / columns / bands (case-insensitive) or by integer axis index; normalization to an explicit per-axis tuple happens here. The keys are axis roles, not the file's dimension names, so {"time": 24} or {"x": 512} raises ValueError. Spatial-dimension orientation (flips and axis moves needed to present a plane north-up) is applied lazily, inside the graph, so it costs nothing until compute.

Object-specific preparation stays in core#

The dataset argument exists for the Protocol's sake. Everything object-specific — resolving the path, the variable name, the spatial-dimension orientation, the manager hook and the cloud environment — is done by core and forwarded in kwargs, so this module never reaches back into a core object's internals.

API#

pyramids_lazy._lazy.build_lazy_array(path, variable_name, chunks, lock=None, manager_id=None, manager_hook=None, spatial_dims=None, flips=None, gdal_env=None, orient=True) #

Build a :class:dask.array.Array backed by MDArray chunk reads.

Parameters:

Name Type Description Default
path str

On-disk path to the NetCDF / HDF5 / Zarr file.

required
variable_name str

Name of the MDArray, group-qualified ("flight_03/CO") for a variable in a sub-group.

required
chunks Any

Raw user input — int, tuple, dict, or the string "auto". See :func:_normalize_chunks.

required
lock Any

Lock passed to :class:CachingFileManager; see :func:_resolve_lock.

None
manager_id Any

Optional stable id so two lazy reads of the same variable share a cache slot. Defaults to (path, variable_name) so repeated calls for the same variable de-duplicate the cached handle.

None
gdal_env dict[str, str] | None

The dataset's captured cloud config, carried into every chunk task so a signed remote store authenticates the re-open each task performs. A no-op when empty / None.

None
manager_hook Any

Optional callable invoked with the created CachingFileManager before the array is returned, so the owning object (e.g. the NetCDF container) can track it and release its handle from close() -- the second half of the

727 fix (release on the parent's close(), not only when#

the array is dropped). It should hold only a weak reference, so it does not defeat the drop-time finalizer.

None
spatial_dims tuple[int, int] | None

(x_index, y_index) of the raster plane the eager get_variable path resolved (explicit x_dim/y_dim or CF detection). When given, the resolved plane is moved to the trailing two axes so the lazy read is oriented on the same plane as the eager read; None (the default) keeps the historical trailing-two-axes normalization. See :func:_orient_lazy_plane (#728).

None
flips tuple[bool, bool] | None

(needs_y_flip, needs_x_flip) the eager path decided for the resolved plane. Used with spatial_dims; None falls back to the trailing-plane decision.

None
orient bool

When True (the default) the block is normalized to the raster convention (row 0 = north, col 0 = west) like the eager get_variable read. Pass False for a raw read in the file's native axis order (the raw interop read path, whose coordinate arrays are also read raw, so the data must not be flipped relative to them).

True

Returns:

Name Type Description
Any

dask.array.Array: Lazy array that computes chunk-by-chunk

through Any

func:_read_mdarray_chunk.

Raises:

Type Description
ImportError

If dask is not installed.

ValueError

For malformed chunks input or missing variable.

Source code in src/pyramids_lazy/_lazy.py
def build_lazy_array(
    path: str,
    variable_name: str,
    chunks: Any,
    lock: Any = None,
    manager_id: Any = None,
    manager_hook: Any = None,
    spatial_dims: tuple[int, int] | None = None,
    flips: tuple[bool, bool] | None = None,
    gdal_env: dict[str, str] | None = None,
    orient: bool = True,
) -> Any:
    """Build a :class:`dask.array.Array` backed by MDArray chunk reads.

    Args:
        path: On-disk path to the NetCDF / HDF5 / Zarr file.
        variable_name: Name of the MDArray, group-qualified
            (``"flight_03/CO"``) for a variable in a sub-group.
        chunks: Raw user input — `int`, `tuple`, `dict`, or
            the string `"auto"`. See :func:`_normalize_chunks`.
        lock: Lock passed to :class:`CachingFileManager`; see
            :func:`_resolve_lock`.
        manager_id: Optional stable id so two lazy reads of the same
            variable share a cache slot. Defaults to
            `(path, variable_name)` so repeated calls for the same
            variable de-duplicate the cached handle.
        gdal_env: The dataset's captured cloud config, carried into every
            chunk task so a signed remote store authenticates the re-open each
            task performs. A no-op when empty / `None`.
        manager_hook: Optional callable invoked with the created
            `CachingFileManager` before the array is returned, so the
            owning object (e.g. the `NetCDF` container) can track it and
            release its handle from `close()` -- the second half of the
            #727 fix (release on the parent's `close()`, not only when
            the array is dropped). It should hold only a weak reference,
            so it does not defeat the drop-time finalizer.
        spatial_dims: `(x_index, y_index)` of the raster plane the eager
            `get_variable` path resolved (explicit `x_dim`/`y_dim` or CF
            detection). When given, the resolved plane is moved to the
            trailing two axes so the lazy read is oriented on the same
            plane as the eager read; `None` (the default) keeps the
            historical trailing-two-axes normalization. See
            :func:`_orient_lazy_plane` (#728).
        flips: `(needs_y_flip, needs_x_flip)` the eager path decided for
            the resolved plane. Used with `spatial_dims`; `None` falls
            back to the trailing-plane decision.
        orient: When `True` (the default) the block is normalized to the
            raster convention (row 0 = north, col 0 = west) like the eager
            `get_variable` read. Pass `False` for a raw read in the file's
            native axis order (the raw interop read path, whose coordinate
            arrays are also read raw, so the data must not be flipped relative
            to them).

    Returns:
        dask.array.Array: Lazy array that computes chunk-by-chunk
        through :func:`_read_mdarray_chunk`.

    Raises:
        ImportError: If `dask` is not installed.
        ValueError: For malformed `chunks` input or missing
            variable.
    """
    import_dask(_DASK_MISSING_MESSAGE)
    import dask.array as da
    from dask.base import tokenize

    shape, dtype, block_size, flip_y, flip_x = _mdarray_shape_and_dtype(
        path,
        variable_name,
    )
    chunk_shape = _normalize_chunks(chunks, shape, block_size, dtype)
    resolved_lock = _resolve_lock(lock)
    key_id = manager_id if manager_id is not None else (path, variable_name)
    manager = CachingFileManager(
        gdal_mdarray_open,
        path,
        "read_only",
        {},
        lock=resolved_lock,
        manager_id=key_id,
        auto_release=True,
    )
    if manager_hook is not None:
        manager_hook(manager)
    chunks_per_axis = _expand_chunks(shape, chunk_shape)
    starts_per_axis = _chunk_starts(chunks_per_axis)
    # Deterministic, content-addressed graph name (path + variable + chunking + cache key) rather
    # than `id(manager)`, so two identical lazy reads share a name and dask can dedupe them
    # (common-subexpression elimination); `id()` gave every read a fresh name and defeated CSE
    # (ARC-76).
    name = f"pyramids-netcdf-read-{variable_name}-{tokenize(path, variable_name, chunk_shape, key_id)}"
    graph: dict[tuple, Any] = {}
    grid_shape = tuple(len(sizes) for sizes in chunks_per_axis)
    for index in np.ndindex(*grid_shape):
        counts = tuple(chunks_per_axis[axis][i] for axis, i in enumerate(index))
        starts = tuple(starts_per_axis[axis][i] for axis, i in enumerate(index))
        reader = _MDArrayChunkReader(
            manager,
            variable_name,
            dtype,
            starts,
            counts,
            gdal_env,
        )
        graph[(name,) + index] = (reader,)
    lazy = da.Array(graph, name, chunks_per_axis, dtype=dtype)
    # Normalize to the raster convention the eager path produces (row 0 = north, col 0 = west) on the
    # SAME plane the eager `get_variable` resolved -- moving a non-trailing plane to the trailing two
    # axes when `spatial_dims` is threaded through (#728). A trailing plane makes this a no-op, so the
    # ordinary `(time, lev, lat, lon)` case is unchanged. Skipped for a raw read (`orient=False`),
    # which must return the file's native axis order (ARC-48 raw interop read).
    if orient:
        lazy = _orient_lazy_plane(
            lazy, da, len(shape), spatial_dims, flips, (flip_y, flip_x)
        )
    # The parked FILE_CACHE handle is released deterministically when this `manager` is
    # garbage-collected -- the `CachingFileManager` registers a `weakref.finalize` on itself in
    # `__init__`. Because the manager is kept alive by the chunk readers in the graph (not by this
    # `lazy` object), that release still fires for derived arrays -- `unpack=True`, `open_mfdataset`,
    # `plot(chunks=)` -- whose graph keeps only the readers, not this wrapper (#727).
    return lazy

pyramids_lazy._lazy.mask_no_data(arr, fill) #

Return a lazy dask masked array with arr's no-data cells masked.

The lazy counterpart of the eager read_array(masked=True): cells equal to the (physical) fill — or NaN cells when fill is NaN — are masked, so computing the result yields a :class:numpy.ma.MaskedArray. fill=None (a variable with no declared no-data) yields an all-unmasked masked array, matching the eager path. fill is the sentinel in read units (physical, post-unpack); applied after apply_unpack, so it compares equal to the unpacked fill cells exactly, the same way the eager path's _read_no_data does.

Parameters:

Name Type Description Default
arr Any

A :class:dask.array.Array of (already unpacked) values.

required
fill Any

The physical no-data sentinel, or None.

required

Returns:

Type Description
Any

dask.array.Array: A dask masked array over arr.

Source code in src/pyramids_lazy/_lazy.py
def mask_no_data(arr: Any, fill: Any) -> Any:
    """Return a lazy dask masked array with `arr`'s no-data cells masked.

    The lazy counterpart of the eager `read_array(masked=True)`: cells equal to the (physical)
    `fill` — or NaN cells when `fill` is NaN — are masked, so computing the result yields a
    :class:`numpy.ma.MaskedArray`. `fill=None` (a variable with no declared no-data) yields an
    all-unmasked masked array, matching the eager path. `fill` is the sentinel **in read units**
    (physical, post-unpack); applied after `apply_unpack`, so it compares equal to the unpacked
    fill cells exactly, the same way the eager path's `_read_no_data` does.

    Args:
        arr: A :class:`dask.array.Array` of (already unpacked) values.
        fill: The physical no-data sentinel, or `None`.

    Returns:
        dask.array.Array: A dask masked array over `arr`.
    """
    import_dask(_DASK_MISSING_MESSAGE)
    import dask.array as da

    if fill is None:
        result = da.ma.masked_array(arr)
    elif np.isnan(fill):
        result = da.ma.masked_invalid(arr)
    else:
        result = da.ma.masked_equal(arr, fill)
    return result