Backend contract#
The object pyramids-core resolves and dispatches to. One instance exposing one method per lazy
operation, two class accessors (lazy_netcdf_type and lazy_feature_collection_type, which hand
core the classes it forwards its own public names to) and an api_version core checks before
using it.
| Entry-point group | pyramids.compute_backend |
| Entry point | lazy = pyramids_lazy.backend:backend |
| Fallback | pyramids_lazy.get_compute_backend() |
| Contract version | API_VERSION = 2 |
| Authoritative Protocol | pyramids.base._compute_backend.ComputeBackend |
Every method is a thin forwarder that imports its implementation module inside the method body,
so an operation's own module (_zarr, _kerchunk_facade, _vector_read, …) is loaded only when
that operation is first used.
That is a statement about the per-operation modules, not about dask. With the [parquet] extra
installed, importing pyramids_lazy pulls in dask (core, array and dataframe) along with
dask_geopandas, pyarrow, pandas, geopandas and scipy — though not distributed, zarr
or xarray, which stay out until an operation needs them. None of this is caused by anything in
this package. Nothing in pyramids_lazy, and nothing in pyramids-core, imports dask at module
scope. The chain is:
pyramids_lazy/__init__.pyre-exportsLazyNetCDFeagerly, which importspyramids.lazy_support;- that facade imports core subpackages, reaching
pyramids.feature.__init__; - which probes for
dask_geopandasat module scope — atry/except OptionalPackageDoesNotExistthat sets the_HAS_DASK_GEOPANDASflag core uses for dispatch; - and
dask_geopandasimports dask.
So the cost appears only when the optional dependency is present. Blocking dask_geopandas to
simulate a bare install, the probe fails, the flag goes False and no dask is imported:
| Imported | Modules | dask present |
|---|---|---|
pyramids |
306 | no |
pyramids.feature, with dask_geopandas blocked |
855 | no |
pyramids.feature |
1397 | yes |
pyramids.dataset |
1690 | yes |
pyramids_lazy |
1868 | yes |
Measured:
Measured in two separate interpreters:
$ python -c "import sys, pyramids; print(len(sys.modules), 'dask' in sys.modules)"
306 False
$ python -c "import sys, pyramids_lazy; print(len(sys.modules), 'dask' in sys.modules)"
1868 True
What the deferred imports buy is therefore narrower than "no dask on import": zarr stays out until a Zarr operation runs, and the per-operation modules stay unloaded until used.
Why a singleton#
The Protocol describes a single object rather than a class discovered per source, mirroring core's
plot-backend seam rather than its raster/vector driver seam. Both discovery paths resolve to the
same module-level backend instance.
That is a statement about which object they return, not about patching. Core caches the backend it
resolved, so a test that replaces pyramids_lazy.backend after resolution will not be seen by core
until the cache is cleared (clear_compute_backend_cache).
API#
pyramids_lazy.backend.LazyComputeBackend
#
Concrete pyramids.compute_backend backend for the dask/zarr lazy layer.
See pyramids.base._compute_backend.ComputeBackend for the authoritative method contract and
signatures; every Protocol method is implemented here one-for-one. classify_zarr_store is an
additional, duck-typed extension core reaches via getattr -- deliberately not a Protocol
member, so a backend predating it is not rejected by core's structural check.
Source code in src/pyramids_lazy/backend.py
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 | |
build_lazy_cube(netcdf, chunks)
#
Build the lazy cube view for NetCDF.chunk (returns a LazyNetCDF).
classify_zarr_store(store, storage_options=None)
#
Back core's Dataset.read_file Zarr routing: classify a .zarr store.
Returns "single" / "cube" / "other". Core applies the cheap suffix gate itself
and asks the backend to look inside a .zarr; opening the store needs the optional
zarr package, which lives here, so the classification does too. An unreadable or foreign
store is reported as "other" (never raises) so core falls back to GDAL.
Source code in src/pyramids_lazy/backend.py
combine_kerchunk(paths, output_path, **kwargs)
#
Back NetCDF.combine_kerchunk / DatasetCollection.to_kerchunk.
Source code in src/pyramids_lazy/backend.py
lazy_feature_collection_type()
#
Return the class the lazy vector reads produce.
Backs core's public pyramids.feature.LazyFeatureCollection and is_lazy_fc.
A method, NOT a property: runtime_checkable isinstance evaluates data members with
getattr, so as a property this body ran whenever core merely type-checked the backend.
Importing the module needs dask-geopandas (the [parquet] extra), so that made the
whole backend fail its structural check in every env without that extra -- and core, which
treats a failing backend as one to skip, then fell back to eager execution for raster and
cube work that never needed vectors at all. As a method the import is deferred to the call,
where the branded hint is the right answer.
Source code in src/pyramids_lazy/backend.py
lazy_mesh_type()
#
Return the class :meth:open_mesh_mfdataset produces (the lazy time-stacked mesh).
The mesh analogue of :meth:lazy_netcdf_type, provided so core can forward a public
pyramids.netcdf.ugrid.LazyMeshDataset name (and its isinstance checks) to this
without importing pyramids_lazy itself. No released core consumes it yet -- unlike
lazy_netcdf_type / lazy_feature_collection_type, which core does forward: core's
MeshDataset.open_mfdataset(chunks=...) reaches LazyMeshDataset only as the return of
:meth:open_mesh_mfdataset and never asks for its type. This accessor is therefore dead
until someone wires that forwarding; it is kept for parity and that future use. An additive,
duck-typed accessor core would reach via getattr -- see :meth:open_mesh_mfdataset.
Source code in src/pyramids_lazy/backend.py
lazy_netcdf_type()
#
Return the class :meth:build_lazy_cube produces.
Core forwards its public pyramids.netcdf.LazyNetCDF name (and the isinstance checks
that go with it) to this, instead of importing pyramids_lazy itself, so the entry-point
seam stays honest about which backend is installed.
Source code in src/pyramids_lazy/backend.py
open_mesh_mfdataset(paths, **kwargs)
#
Back MeshDataset.open_mfdataset(chunks=...) (dask-stacked multi-file mesh read).
An additive, duck-typed extension core reaches via getattr, like
:meth:classify_zarr_store and deliberately not a ComputeBackend Protocol member:
the Protocol is runtime_checkable and core drops any backend failing its structural
isinstance check, so adding a method there would reject every already-released plugin.
Returns a :class:~pyramids_lazy._mesh_mfdataset.LazyMeshDataset; see
serapeum-org/pyramids-core#1458.
Source code in src/pyramids_lazy/backend.py
open_mfdataset(paths, variable, **kwargs)
#
Back NetCDF.open_mfdataset (dask-stacked multi-file read).
Source code in src/pyramids_lazy/backend.py
read_array_lazy(dataset, *, chunks, lock=None, **kwargs)
#
Lazy dask-backed read of a raster/variable (Dataset/NetCDF read_array).
dataset is accepted for the Protocol; the object-specific prep (path, variable name,
spatial-dim orientation, manager hook, cloud env) is done by core and forwarded in
kwargs to :func:build_lazy_array.
Source code in src/pyramids_lazy/backend.py
read_collection_from_zarr(collection_cls, store, **kwargs)
#
Back DatasetCollection.from_zarr (returns a DatasetCollection).
Source code in src/pyramids_lazy/backend.py
read_dataset_from_zarr(store, **kwargs)
#
Back Dataset.from_zarr (returns a Dataset).
read_vector_lazy(source, **kwargs)
#
Back FeatureCollection.read_file(backend="dask").
read_vector_parquet_lazy(source, **kwargs)
#
Back FeatureCollection.read_parquet(backend="dask").
to_dask_dataframe(netcdf, *, variables, dropna, chunks)
#
Back NetCDF.to_dask_dataframe.
Source code in src/pyramids_lazy/backend.py
to_kerchunk(source, output_path, **kwargs)
#
Back NetCDF.to_kerchunk (single-file kerchunk manifest).
Source code in src/pyramids_lazy/backend.py
write_collection_to_zarr(collection, store, **kwargs)
#
Back DatasetCollection.to_zarr.
Source code in src/pyramids_lazy/backend.py
write_dataset_to_zarr(dataset, store, **kwargs)
#
Back Dataset.to_zarr.
pyramids_lazy.get_compute_backend()
#
Return the singleton compute backend (core's first-party fallback entry point).
pyramids.base._compute_backend.require_compute_backend calls this by name when no backend is
registered on the pyramids.compute_backend entry-point group, so the plugin is usable from a
bare checkout on PYTHONPATH as well as from an installed distribution.
Source code in src/pyramids_lazy/__init__.py
The core seam guard#
The package pins the facade contract at import time and fails loudly — telling the user to upgrade
pyramids-core — rather than later, on a missing attribute deep inside a lazy call.
pyramids_lazy._seam.require_core_seam()
#
Raise ImportError unless core's facade satisfies this backend's contract.
Raises:
| Type | Description |
|---|---|
ImportError
|
When |
Source code in src/pyramids_lazy/_seam.py
Optional-extra handling#
pyramids_lazy._optional
#
Branded guards for this backend's own optional dependencies.
Core's extra_hint composes hints for core's extras, so it names pyramids-core[parquet].
The lazy-vector path ships in THIS distribution's [parquet] extra, so its hints have to name
pyramids-lazy[parquet] instead. The guards live in their own module so a module-scope import can
be branded without _vector_read and _lazy_collection importing each other.
LazyExtraNotInstalled
#
Bases: ModuleNotFoundError
A dependency from one of this backend's optional extras is missing.
Subclasses :class:ModuleNotFoundError, not just :class:ImportError, deliberately. Tooling
treats the two differently: :func:pytest.importorskip skips on ModuleNotFoundError and
re-raises every other ImportError so a real breakage is never swallowed, and plenty of
library code guards optional imports with except ModuleNotFoundError. Raising a plain
ImportError here therefore turned "this extra is not installed" into a hard error for every
such caller -- it aborted collection across pyramids-core's whole matrix once. Being a
ModuleNotFoundError keeps those callers working while the message stays branded.
Source code in src/pyramids_lazy/_optional.py
import_dask_geopandas()
#
Import and return dask_geopandas, or raise a branded ImportError.
Called at module scope by every module that needs dask_geopandas, so the branded hint is
raised on a cold import too — not only by the call-time guards, which a warm sys.modules
is all that lets them be reached.
Returns:
| Type | Description |
|---|---|
ModuleType
|
The imported |
Raises:
| Type | Description |
|---|---|
LazyExtraNotInstalled
|
When |
Examples:
- The result is the
dask_geopandasmodule itself, so its API is reached straight off it: - Every call hands back the single module object
sys.modulesalready holds, so calling it at module scope and again inside a call-time guard costs one import:
Source code in src/pyramids_lazy/_optional.py
lazy_hint(lead, package)
#
Compose an install hint for one of this distribution's own runtime deps.
Unlike :func:parquet_hint (which names the [parquet] extra), these dependencies —
dask, zarr, kerchunk, h5py and friends — live in pyramids-lazy's plain
[project.dependencies]. A user who reaches one of these guards therefore already has
pyramids-lazy installed, so the actionable remedy is to install the one missing library
(pip install dask); pip install pyramids-lazy would not reinstall an already-satisfied
dependency, hence the --force-reinstall repair line as the fallback. No conda-forge line is
offered: pyramids-lazy is not on conda-forge, and the package of that name there is core's
old lazy extra, which carries no backend.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lead
|
str
|
The sentence naming the operation that needed the dependency, ending in a period. |
required |
package
|
str
|
The pip install target(s) for the missing dependency, e.g. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The lead followed by the two-line install block. |
Examples:
- The lead is kept verbatim at the start and the named library is the pip target:
>>> from pyramids_lazy._optional import lazy_hint >>> hint = lazy_hint("open_mfdataset requires the 'dask' dependency.", "dask") >>> hint.startswith("open_mfdataset requires the 'dask' dependency.") True >>> "pip install dask" in hint True >>> "pip install --force-reinstall pyramids-lazy" in hint True - A space-joined
packageinstalls several libraries in one command, and no core extra or conda line is ever named:
Source code in src/pyramids_lazy/_optional.py
parquet_hint(lead)
#
Compose the install hint for this distribution's [parquet] extra.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lead
|
str
|
The sentence naming the operation that needed the dependency, ending in a period. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The lead followed by the two-line install block. |