Zarr — Dataset#
Backs Dataset.to_zarr(), Dataset.from_zarr() and the GeoZarr routing read_file uses to
recognise a Zarr store.
| Backend methods | write_dataset_to_zarr, read_dataset_from_zarr, classify_zarr_store |
| Module | pyramids_lazy._zarr |
| Returns | a Dataset on read; None on write, or a dask.delayed.Delayed when compute=False |
from pyramids.dataset import Dataset
ds = Dataset.read_file("dem.tif")
ds.to_zarr("dem.zarr")
back = Dataset.from_zarr("dem.zarr")
The only fully-parallel raster write#
Zarr is the one raster output format where pyramids can write in parallel: each dask chunk lands in
an independent Zarr chunk file, so there is no single writer serialising the output and no GDAL
dataset handle to share. The geobox travels as a standard crs_wkt / GeoTransform attribute
convention, so the result round-trips through other Zarr raster readers rather than needing bespoke
glue.
Overview levels are written as their own arrays, with the geotransform scaled per level.
No-data sentinel agreement#
A Dataset carries one no-data value per band, while a Zarr array carries a single fill_value.
Writing therefore has to decide whether the bands agree. The logic here treats NaN as equal to
itself, checks that the candidate sentinel is representable in the array's dtype, and only then
commits it as the store's fill_value; disagreeing bands leave it unset rather than silently
adopting the first band's value.
API#
pyramids_lazy._zarr.write_dataset_to_zarr(ds, store, *, compute=True, mode='w', chunks='auto', storage_options=None, compressor='auto', overview_factors=None, overview_resampling='average')
#
Serialise ds to a Zarr store.
Writes the (bands, rows, cols) dask array to <store>/data
and persists the pyramids geobox metadata as attributes on both
the root group and the array. On compute=False the data write
and the attribute write are bundled into a single
:class:dask.delayed.Delayed so calling .compute() finalizes
everything atomically.
The data array is read with ds.read_array(chunks=...), so it holds
physical values: a CF-packed band is unpacked. The store has no channel for
scale_factor / add_offset, so such a source is written materialised --
float64 values, and a no-data (no_data_value, _FillValue and the
array's fill_value) in the same physical units; see :func:_metadata_dict.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ds
|
Dataset
|
Source :class: |
required |
store
|
str | Path | Any
|
Target store — a path, a fsspec URL ( |
required |
compute
|
bool
|
|
True
|
mode
|
str
|
Zarr open mode. |
'w'
|
chunks
|
Any
|
Chunk specification forwarded to
:meth: |
'auto'
|
storage_options
|
dict[str, Any] | None
|
fsspec options for cloud stores. |
None
|
compressor
|
Any
|
Zarr codec(s) for the |
'auto'
|
overview_factors
|
list[int] | None
|
Downsample factors (e.g. |
None
|
overview_resampling
|
str
|
GDAL resampling for those levels, |
'average'
|
Returns:
| Type | Description |
|---|---|
Any
|
|
Any
|
on |
Raises:
| Type | Description |
|---|---|
OverviewTargetError
|
|
ValueError
|
|
Examples:
- Round-trip a small Dataset through Zarr (requires zarr + dask,
pyramids-lazy dependencies):
>>> import tempfile # doctest: +SKIP >>> from pathlib import Path # doctest: +SKIP >>> import numpy as np # doctest: +SKIP >>> from pyramids.dataset import Dataset, GeoReference # doctest: +SKIP >>> from pyramids_lazy._zarr import write_dataset_to_zarr # doctest: +SKIP >>> arr = np.arange(16, dtype=np.float32).reshape(4, 4) # doctest: +SKIP >>> ds = Dataset.from_array( ... arr, ... geo_ref=GeoReference( ... top_left_corner=(0.0, 4.0), cell_size=1.0, epsg=4326 ... ), ... ) # doctest: +SKIP >>> store = Path(tempfile.mkdtemp()) / "ds.zarr" # doctest: +SKIP >>> write_dataset_to_zarr(ds, str(store)) is None # doctest: +SKIP True
Source code in src/pyramids_lazy/_zarr.py
508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 | |
pyramids_lazy._zarr.read_dataset_from_zarr(store, *, chunks=None, storage_options=None, level=1, data_name=None)
#
Open a pyramids-written Zarr store and materialise a :class:Dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
str | Path | Any
|
Input store — path / fsspec URL / :class: |
required |
level
|
int
|
Pyramid downsample factor to read (FR-7). |
1
|
data_name
|
str | None
|
Explicit name of the data array in the group. |
None
|
chunks
|
Any
|
When given (a 3-tuple |
None
|
storage_options
|
dict[str, Any] | None
|
fsspec storage options. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Dataset |
Dataset
|
The reconstructed dataset. |
Examples:
- Read a store that was written with :func:
write_dataset_to_zarrand check the recovered shape (requires zarr + dask, pyramids-lazy dependencies):>>> import tempfile # doctest: +SKIP >>> from pathlib import Path # doctest: +SKIP >>> import numpy as np # doctest: +SKIP >>> from pyramids.dataset import Dataset, GeoReference # doctest: +SKIP >>> from pyramids_lazy._zarr import ( ... read_dataset_from_zarr, write_dataset_to_zarr, ... ) # doctest: +SKIP >>> arr = np.arange(16, dtype=np.float32).reshape(4, 4) # doctest: +SKIP >>> src = Dataset.from_array( ... arr, ... geo_ref=GeoReference( ... top_left_corner=(0.0, 4.0), cell_size=1.0, epsg=4326 ... ), ... ) # doctest: +SKIP >>> store = Path(tempfile.mkdtemp()) / "ds.zarr" # doctest: +SKIP >>> write_dataset_to_zarr(src, str(store)) # doctest: +SKIP >>> recovered = read_dataset_from_zarr(str(store)) # doctest: +SKIP >>> (recovered.rows, recovered.columns) # doctest: +SKIP (4, 4)
Source code in src/pyramids_lazy/_zarr.py
865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 | |
pyramids_lazy._zarr.classify_zarr_store(store, storage_options=None)
#
Classify a .zarr store for :meth:~pyramids.dataset.Dataset.read_file routing.
Returns one of:
"single"-- a pyramids-written single-raster GeoZarr store (a 3-Ddataarray beside aspatial_refnode); route it to :meth:~pyramids.dataset.Dataset.from_zarr."cube"-- a pyramids-written temporal cube (a 4-Ddataarray beside aspatial_refnode); it is not a single raster."other"-- anything else: a foreign / xarray-written store, a plain file that merely ends in.zarr, a store that cannot be opened here (e.g. a remote store needing credentials this call was not given), or an install without the optionalzarrpackage. The caller should let GDAL handle it.
This is the backend side of core's read_file Zarr routing: Dataset.read_file first
applies its own cheap, backend-free suffix check and, only for a .zarr, asks the installed
compute backend to classify it here. Only
pyramids-written stores are claimed, so a foreign .zarr GDAL can already read is left to
GDAL rather than mis-routed. The store is opened read-only for metadata only; any failure to
open and confirm it -- including the optional zarr package being absent -- is reported as
"other" so the caller falls back to GDAL rather than surfacing an opaque error. This never
raises for a missing zarr: a zarr-less install reads a .zarr through GDAL exactly as
before this routing existed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
Any
|
A |
required |
storage_options
|
dict[str, Any] | None
|
fsspec options forwarded to :func: |
None
|
Returns:
| Type | Description |
|---|---|
str
|
|
Examples:
A pyramids single-raster store classifies as "single" and a cube as "cube";
a foreign or unreadable store as "other" (all need a real store, so illustrative):
>>> classify_zarr_store("elevation.zarr")
'single'
>>> classify_zarr_store("timeseries_cube.zarr")
'cube'
>>> classify_zarr_store("xarray_export.zarr")
'other'
Source code in src/pyramids_lazy/_zarr.py
962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 | |
pyramids_lazy._zarr.is_cube_store(store, storage_options=None)
#
Whether a pyramids Zarr store holds a temporal cube rather than a single raster.
Thin predicate over :func:classify_zarr_store: True only for a pyramids-written 4-D
(time, band, y, x) cube (as :meth:pyramids.dataset.DatasetCollection.to_zarr writes).
A single raster, a foreign store, or an unreadable path is False.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
Any
|
A |
required |
storage_options
|
dict[str, Any] | None
|
fsspec options forwarded to :func: |
None
|
Returns:
| Type | Description |
|---|---|
bool
|
|
bool
|
which cannot open the store and so cannot call it a cube). |
Examples:
A single-raster store is not a cube, while a cube store is (both need a real on-disk store, so these are illustrative):
See Also
classify_zarr_store: the three-way classification used for routing.