Skip to content

Lazy vector reads#

Backs FeatureCollection.read_file(backend="dask") and FeatureCollection.read_parquet(backend="dask"). Both return a LazyFeatureCollection.

Backend methods read_vector_lazy, read_vector_parquet_lazy
Module pyramids_lazy._vector_read
Needs read_file → dask-geopandas; read_parquet → dask-geopandas and pyarrow
Returns LazyFeatureCollection
from pyramids.feature import FeatureCollection

fc = FeatureCollection.read_parquet("parcels.parquet", backend="dask")
fc.npartitions
fc.geometry.area.mean().compute()

Partitioning#

Partitioning is resolved before the read, and the knobs are passed straight through to dask_geopandas rather than translated:

  • an explicit npartitions wins; failing that, an explicit chunksize is used;
  • with neither, the split is guessed — a remote path (/vsi, http(s), s3, gs, az) gets npartitions=1, and a local file gets ceil(filesize / target_bytes_per_partition), falling back to 1 if the size cannot be read.

So a default read of a local file is already partitioned by size; pass npartitions explicitly when you want a particular layout.

Filters are refused, not ignored#

read_file(backend="dask") raises ValueError if given bbox, mask, rows, columns, where or layer — dask_geopandas.read_file has no pushdown for them, and silently dropping a filter would return more rows than asked for. Filter after loading (.clip / .loc / .compute), or use read_parquet(backend="dask", filters=…).

These are the backend="dask" branches lifted out of core's pyramids.feature._read; the eager branches stay in core, which is why a minimal install never evaluates this module.

API#

pyramids_lazy._vector_read.read_file_dask(resolved, *, layer=None, bbox=None, mask=None, rows=None, columns=None, where=None, npartitions=None, chunksize=None) #

Dask backend for FeatureCollection.read_file: reject unsupported filters, wrap as LazyFC.

Source code in src/pyramids_lazy/_vector_read.py
def read_file_dask(
    resolved: str,
    *,
    layer: str | int | None = None,
    bbox: Any = None,
    mask: Any = None,
    rows: slice | int | None = None,
    columns: list[str] | None = None,
    where: str | None = None,
    npartitions: int | None = None,
    chunksize: int | None = None,
) -> LazyFeatureCollection:
    """Dask backend for ``FeatureCollection.read_file``: reject unsupported filters, wrap as LazyFC."""
    unsupported = {
        "bbox": bbox,
        "mask": mask,
        "rows": rows,
        "columns": columns,
        "where": where,
        "layer": layer,
    }
    supplied = [k for k, v in unsupported.items() if v is not None]
    if supplied:
        raise ValueError(
            f"backend='dask' does not support filter kwargs "
            f"{supplied}. dask_geopandas.read_file has no "
            "pushdown story for these. Either omit them and "
            "filter post-load via .clip / .loc / .compute, or "
            "switch to read_parquet(backend='dask', filters=...)"
        )
    dask_geopandas = import_dask_geopandas()
    partition_kwargs = _resolve_lazy_partitioning(resolved, npartitions, chunksize)
    dask_gdf = dask_geopandas.read_file(resolved, **partition_kwargs)
    return LazyFeatureCollection.from_dask_gdf(dask_gdf)

pyramids_lazy._vector_read.read_parquet_dask(resolved, *, columns=None, split_row_groups=None, filters=None, blocksize=None, storage_options=None, extra_kwargs=None) #

Dask backend for FeatureCollection.read_parquet: wrap dask_geopandas as a LazyFC.

Source code in src/pyramids_lazy/_vector_read.py
def read_parquet_dask(
    resolved: str,
    *,
    columns: list[str] | None = None,
    split_row_groups: bool | None = None,
    filters: list | None = None,
    blocksize: int | str | None = None,
    storage_options: dict | None = None,
    extra_kwargs: dict[str, Any] | None = None,
) -> LazyFeatureCollection:
    """Dask backend for ``FeatureCollection.read_parquet``: wrap dask_geopandas as a LazyFC."""
    dask_geopandas = import_dask_geopandas()
    dask_kwargs = _compact(
        {
            "columns": columns,
            "split_row_groups": split_row_groups,
            "filters": filters,
            "blocksize": blocksize,
            "storage_options": storage_options,
        }
    )
    dask_kwargs.update(extra_kwargs or {})
    _require_pyarrow()
    dask_gdf = dask_geopandas.read_parquet(resolved, **dask_kwargs)
    return LazyFeatureCollection.from_dask_gdf(dask_gdf)