Skip to content

Multi-file reads#

Backs NetCDF.open_mfdataset(). Opens many NetCDF files and stacks one named variable into a single lazy dask.array.Array of shape (n_files, *variable_shape).

Backend method open_mfdataset
Module pyramids_lazy._mfdataset
Returns dask.array.Array of shape (n_files, *variable_shape)

It returns the stacked dask array itself, not a NetCDF wrapping it — there is no cube to call read_array() on:

from pyramids.netcdf import NetCDF

arr = NetCDF.open_mfdataset("noah_2009*.nc", variable="precipitation")
arr.sum(axis=0).compute()

Deliberately narrow#

One variable at a time, no by_coords inference and no combine strategies. For the common hydrology and meteorology case — open 365 daily files, stack one gridded variable, reduce along time — that narrowness is the whole point: no metadata inference means no failure modes when one file in the set disagrees with the others.

The per-file reads are sequential, and the parallel= argument is deprecated and inert — passing it emits a DeprecationWarning and changes nothing. It needs to do nothing: every per-file read is already lazy, so the scheduler parallelises the per-chunk reads at compute time without an eager dask.delayed fan-out.

API#

pyramids_lazy._mfdataset.open_mfdataset(paths, variable, *, chunks=None, parallel=False, preprocess=None) #

Open many NetCDFs; stack variable into one lazy dask array.

Parameters:

Name Type Description Default
paths str | Sequence[str | Path]

Glob string ("noah_*.nc"), an explicit file path, or a sequence of paths. Glob inputs are expanded and sorted alphabetically so the stack order is deterministic.

required
variable str

Name of the NetCDF variable to extract from each file. Must exist in every file with the same spatial shape + dtype; mismatches produce a :class:ValueError from :func:dask.array.stack at graph-construction time.

required
chunks dict | str | None

Chunk specification forwarded to :meth:NetCDF.read_array when reading each file. None (the default) reads each file lazily with "auto" (native-ish) chunking so the files are not all materialised into RAM before stacking; pass an explicit spec to override.

None
parallel bool

Deprecated and inert; retained only for backward-compatible call signatures. Every per-file read returns a lazy dask array, so the scheduler already parallelises the per-chunk reads at compute time and there is no separate eager dask.delayed fan-out to enable. Passing a truthy value emits a DeprecationWarning.

False
preprocess Callable[[NetCDF], NetCDF] | None

Optional callable applied to each :class:NetCDF subset before its array is extracted — for example to unpack scale/offset, crop, or drop auxiliary variables.

None

Returns:

Type Description
Any

dask.array.Array: A stack of shape (n_files, *var_shape).

Any

Element [i,...] corresponds to paths[i] (after glob

Any

expansion + sort).

Raises:

Type Description
ImportError

When dask is not installed.

FileNotFoundError

When paths is an explicit path that does not exist.

Examples:

  • Stack a single file into a 1-element dask array:
    >>> from pyramids_lazy._mfdataset import open_mfdataset
    >>> path = "tests/data/netcdf/cf__4v__1d3-3d1__proj__y-desc.nc"
    >>> stack = open_mfdataset([path], "values")
    >>> stack.shape[0]
    1
    
Source code in src/pyramids_lazy/_mfdataset.py
def open_mfdataset(
    paths: str | Sequence[str | Path],
    variable: str,
    *,
    chunks: dict | str | None = None,
    parallel: bool = False,
    preprocess: Callable[[NetCDF], NetCDF] | None = None,
) -> Any:
    """Open many NetCDFs; stack `variable` into one lazy dask array.

    Args:
        paths: Glob string (`"noah_*.nc"`), an explicit file path, or
            a sequence of paths. Glob inputs are expanded and sorted
            alphabetically so the stack order is deterministic.
        variable: Name of the NetCDF variable to extract from each
            file. Must exist in every file with the same spatial
            shape + dtype; mismatches produce a :class:`ValueError`
            from :func:`dask.array.stack` at graph-construction time.
        chunks: Chunk specification forwarded to
            :meth:`NetCDF.read_array` when reading each file. `None`
            (the default) reads each file lazily with `"auto"`
            (native-ish) chunking so the files are not all materialised
            into RAM before stacking; pass an explicit spec to override.
        parallel: Deprecated and inert; retained only for backward-compatible
            call signatures. Every per-file read returns a lazy dask array, so
            the scheduler already parallelises the per-chunk reads at compute
            time and there is no separate eager `dask.delayed` fan-out to
            enable. Passing a truthy value emits a `DeprecationWarning`.
        preprocess: Optional callable applied to each
            :class:`NetCDF` subset before its array is extracted —
            for example to unpack scale/offset, crop, or drop
            auxiliary variables.

    Returns:
        dask.array.Array: A stack of shape `(n_files, *var_shape)`.
        Element `[i,...]` corresponds to `paths[i]` (after glob
        expansion + sort).

    Raises:
        ImportError: When dask is not installed.
        FileNotFoundError: When `paths` is an explicit path that
            does not exist.

    Examples:
        - Stack a single file into a 1-element dask array:
            ```python
            >>> from pyramids_lazy._mfdataset import open_mfdataset
            >>> path = "tests/data/netcdf/cf__4v__1d3-3d1__proj__y-desc.nc"
            >>> stack = open_mfdataset([path], "values")
            >>> stack.shape[0]
            1

            ```
    """
    try:
        import dask.array as da
    except ImportError as exc:
        raise ImportError(_LAZY_IMPORT_ERROR) from exc

    if parallel:
        warnings.warn(
            "open_mfdataset(parallel=...) is deprecated and has no effect: the default lazy "
            "per-file read already parallelises chunk reads at compute time. The argument is "
            "accepted for backward compatibility and will be removed in a future release.",
            DeprecationWarning,
            stacklevel=2,
        )

    resolved = _resolve_paths(paths)
    # Default to a lazy per-file read (native-ish `"auto"` chunking) so the stack does not
    # materialise every file into RAM up front -- opening hundreds of files used to load them all
    # eagerly before stacking (ARC-48). An explicit `chunks` is honoured as given.
    effective_chunks = "auto" if chunks is None else chunks

    # Each per-file read already returns a lazy dask array (chunks are never eager here), so the
    # stack is built directly and dask parallelises the per-chunk reads at compute time. The
    # `parallel` flag -- which used to wrap eager reads in `dask.delayed` -- is inert now that the
    # default read is lazy (wrapping a dask array in `from_delayed` would nest a synchronous inner
    # compute per file); it is retained only for backward-compatible call signatures (ARC-48).
    arrays = [
        _open_and_extract(p, variable, preprocess, effective_chunks) for p in resolved
    ]
    arrays = [
        a if hasattr(a, "dask") else da.from_array(np.asarray(a), chunks="auto")
        for a in arrays
    ]
    return da.stack(arrays, axis=0)