Skip to content

Dask dataframe view#

Backs NetCDF.to_dask_dataframe(). Builds a tidy dask.dataframe.DataFrame — dimension coordinates as columns beside the value columns, on a flat index — reading values lazily.

Backend method to_dask_dataframe
Module pyramids_lazy._to_dask_dataframe
Returns dask.dataframe.DataFrame
from pyramids.netcdf import NetCDF

cube = NetCDF.read_file("era5.nc")
df = cube.to_dask_dataframe(variables=["t2m"], dropna=True, chunks="auto")
df.groupby("time").t2m.mean().compute()

Relocated from core's Interop engine — this is the lazy half of it; the eager to_dataframe stays in core.

API#

pyramids_lazy._to_dask_dataframe.to_dask_dataframe(nc, *, variables=None, dropna=False, chunks='auto') #

Hand nc's cube to dask in tidy form — the lazy sibling of to_dataframe.

Returns a :class:dask.dataframe.DataFrame with the dimension coordinates as plain columns beside the value columns on a flat index. Values are read lazily through NetCDF.read_array.

Source code in src/pyramids_lazy/_to_dask_dataframe.py
def to_dask_dataframe(
    nc: Any, *, variables: Any = None, dropna: bool = False, chunks: Any = "auto"
) -> Any:
    """Hand ``nc``'s cube to dask in tidy form — the lazy sibling of ``to_dataframe``.

    Returns a :class:`dask.dataframe.DataFrame` with the dimension coordinates as plain columns
    beside the value columns on a flat index. Values are read lazily through ``NetCDF.read_array``.
    """
    import_dask(_DASK_MISSING)
    require_optional(
        "dask.dataframe",
        "to_dask_dataframe() needs dask.dataframe, which requires pyarrow — install the "
        "`parquet` extra (e.g. `pip install pyramids-lazy[parquet]`).",
    )
    import dask.array as da
    import dask.dataframe as dd

    names = _frame_variables(nc, variables)
    first = nc._require_raster_variable(names[0]) if _is_container(nc) else nc
    in_memory = _is_in_memory(nc)
    value_arrays = _lazy_value_columns(nc, names, first, in_memory, chunks, da)
    ref_chunks = next(iter(value_arrays.values())).chunks
    columns = _lazy_dimension_columns(first, ref_chunks, da)
    for name, array in value_arrays.items():
        columns[name] = array.rechunk(ref_chunks)
    frame = dd.concat(
        [dd.from_dask_array(col, columns=[label]) for label, col in columns.items()],
        axis=1,
    )
    return frame.dropna(subset=list(names), how="all") if dropna else frame