Lazy raster reads#
Backs NetCDF.read_array(chunks=…). Returns a dask.array.Array instead of a NumPy array,
without dask being imported until the call happens.
Dataset.read_array does not route here. It accepts chunks= and returns a
dask.array.Array, but it is served by core's own in-core lazy raster path
(pyramids.dataset.engines.IO.read_array, which never calls require_compute_backend). Every
read_array_lazy call site in core lives under pyramids/netcdf/ (netcdf.py,
engines/interop.py, ugrid/interop_xarray.py).
So a lazy Dataset read needs dask but not this package, whereas a lazy NetCDF read needs this
package. Dataset's Zarr and kerchunk operations do come here — see
Zarr (Dataset).
| Backend method | read_array_lazy |
| Module | pyramids_lazy._lazy |
| Returns | dask.array.Array |
from pyramids.netcdf import NetCDF
cube = NetCDF.read_file("era5.nc")
arr = cube.read_array(variable="pr", chunks={"rows": 64, "cols": 64})
arr.mean().compute()
How the graph is built#
The array is assembled from a low-level task graph — one pickle-safe reader task per block — rather
than from dask.array.from_array. That is what makes a distributed read work: each task carries
the path and window, opens the dataset through a CachingFileManager, and reads only its own
window. The manager is the reason a worker handling many blocks of one file opens it once, and the
reason the handle is shared process-globally rather than per task.
chunks accepts "auto", a single int, a sequence per axis, or a dict keyed by rows / cols /
columns / bands (case-insensitive) or by integer axis index; normalization to an explicit
per-axis tuple happens here. The keys are axis roles, not the file's dimension names, so
{"time": 24} or {"x": 512} raises ValueError. Spatial-dimension orientation
(flips and axis moves needed to present a plane north-up) is applied lazily, inside the graph,
so it costs nothing until compute.
Object-specific preparation stays in core#
The dataset argument exists for the Protocol's sake. Everything object-specific — resolving the
path, the variable name, the spatial-dimension orientation, the manager hook and the cloud
environment — is done by core and forwarded in kwargs, so this module never reaches back into a
core object's internals.
API#
pyramids_lazy._lazy.build_lazy_array(path, variable_name, chunks, lock=None, manager_id=None, manager_hook=None, spatial_dims=None, flips=None, gdal_env=None, orient=True)
#
Build a :class:dask.array.Array backed by MDArray chunk reads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
On-disk path to the NetCDF / HDF5 / Zarr file. |
required |
variable_name
|
str
|
Name of the MDArray, group-qualified
( |
required |
chunks
|
Any
|
Raw user input — |
required |
lock
|
Any
|
Lock passed to :class: |
None
|
manager_id
|
Any
|
Optional stable id so two lazy reads of the same
variable share a cache slot. Defaults to
|
None
|
gdal_env
|
dict[str, str] | None
|
The dataset's captured cloud config, carried into every
chunk task so a signed remote store authenticates the re-open each
task performs. A no-op when empty / |
None
|
manager_hook
|
Any
|
Optional callable invoked with the created
727 fix (release on the parent's
|
None
|
spatial_dims
|
tuple[int, int] | None
|
|
None
|
flips
|
tuple[bool, bool] | None
|
|
None
|
orient
|
bool
|
When |
True
|
Returns:
| Name | Type | Description |
|---|---|---|
Any
|
dask.array.Array: Lazy array that computes chunk-by-chunk |
|
through |
Any
|
func: |
Raises:
| Type | Description |
|---|---|
ImportError
|
If |
ValueError
|
For malformed |
Source code in src/pyramids_lazy/_lazy.py
737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 | |
pyramids_lazy._lazy.mask_no_data(arr, fill)
#
Return a lazy dask masked array with arr's no-data cells masked.
The lazy counterpart of the eager read_array(masked=True): cells equal to the (physical)
fill — or NaN cells when fill is NaN — are masked, so computing the result yields a
:class:numpy.ma.MaskedArray. fill=None (a variable with no declared no-data) yields an
all-unmasked masked array, matching the eager path. fill is the sentinel in read units
(physical, post-unpack); applied after apply_unpack, so it compares equal to the unpacked
fill cells exactly, the same way the eager path's _read_no_data does.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
arr
|
Any
|
A :class: |
required |
fill
|
Any
|
The physical no-data sentinel, or |
required |
Returns:
| Type | Description |
|---|---|
Any
|
dask.array.Array: A dask masked array over |