Kerchunk manifests#
Backs NetCDF.to_kerchunk() and NetCDF.combine_kerchunk() / DatasetCollection.to_kerchunk().
Serialises a NetCDF archive into a kerchunk JSON reference manifest so consumers can open it as a
lazy Zarr-backed cube with zero rewrite — the manifest holds byte-range pointers into the
source files; no pixel data is moved.
| Backend methods | to_kerchunk, combine_kerchunk |
| Modules | pyramids_lazy._kerchunk_facade, pyramids_lazy._kerchunk_builder |
| Returns | the manifest dict (also written to output_path) |
from pyramids.netcdf import NetCDF
NetCDF.read_file("era5.nc").to_kerchunk("era5.json")
NetCDF.combine_kerchunk(["2009.json", "2010.json"], "combined.json")
Two builders, and why#
The facade dispatches between kerchunk's own translators and a native builder in this package.
The native builder exists because of a deadlock: kerchunk.hdf drives zarr's synchronous v2 API,
which under zarr v3 submits work to a process-global event loop and hangs intermittently. The
builder walks the HDF5 container with h5py and emits zarr-v2 metadata plus byte-range chunk
references directly, never instantiating a live zarr group. Its output is the ordinary kerchunk
reference document, so the consumer side is unchanged.
Its scope is the feature surface GDAL's netCDF driver writes: fixed-width numeric and fixed-length
string dtypes, deflate/shuffle filters, and dimension scales. An unsupported HDF5 feature
raises a clear ValueError rather than producing a subtly wrong manifest.
API#
pyramids_lazy._kerchunk_facade.to_kerchunk(src_path, output_path, *, inline_threshold=500, vlen_encode='embed', backend='native')
#
Emit a single-file kerchunk reference manifest.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src_path
|
str | Path
|
Path or URL of the source NetCDF / HDF5 file. |
required |
output_path
|
str | Path
|
Path where the JSON manifest is written. |
required |
inline_threshold
|
int
|
Chunks smaller than this many bytes are embedded directly in the manifest rather than referenced by offset. |
500
|
vlen_encode
|
str
|
One of |
'embed'
|
backend
|
str
|
|
'native'
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The manifest dict that was written — useful for inspection |
dict[str, Any]
|
or further programmatic use. |
Raises:
| Type | Description |
|---|---|
ImportError
|
When the chosen backend's dependency is missing
(h5py for |
ValueError
|
When |
Examples:
- Emit a manifest for one NetCDF file (requires kerchunk, a pyramids-lazy dependency):
Source code in src/pyramids_lazy/_kerchunk_facade.py
pyramids_lazy._kerchunk_facade.combine_kerchunk(src_paths, output_path, *, concat_dims=('time',), identical_dims=('lat', 'lon'), inline_threshold=500, backend='native')
#
Emit a combined kerchunk manifest spanning many source files.
The default "native" backend scans each file with :func:build_single_manifest
(h5py, no live zarr group) and concatenates the per-file manifests along the
single concat_dims entry — every variable carrying that dimension is stacked,
the rest are carried from the first file. This avoids the zarr-v3 sync()
deadlock (#530). "legacy" forces the
SingleHdf5ToZarr + MultiZarrToZarr path; the native backend also falls back
to it for inputs it cannot handle (e.g. multiple concat dims, unsupported HDF5
features). "kerchunk" is a deprecated alias for "legacy".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src_paths
|
Sequence[str | Path]
|
Sequence of source NetCDF / HDF5 paths or URLs, in concatenation order. |
required |
output_path
|
str | Path
|
Path where the combined JSON manifest is written. |
required |
concat_dims
|
Sequence[str]
|
Dimension name(s) along which to concatenate. The native
backend supports exactly one; more than one falls back to kerchunk.
Default |
('time',)
|
identical_dims
|
Sequence[str]
|
Dimension name(s) expected to be identical across files
(e.g. shared |
('lat', 'lon')
|
inline_threshold
|
int
|
Same semantics as :func: |
500
|
backend
|
str
|
|
'native'
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The combined manifest dict that was written. |
Raises:
| Type | Description |
|---|---|
ImportError
|
When the chosen backend's dependency is missing. |
ValueError
|
When |
Examples:
- Combine a year's worth of daily NetCDFs into one manifest:
>>> from pathlib import Path # doctest: +SKIP >>> from pyramids_lazy._kerchunk_facade import combine_kerchunk # doctest: +SKIP >>> srcs = sorted(Path("/data/noah").glob("noah_*.nc")) # doctest: +SKIP >>> manifest = combine_kerchunk( ... srcs, "noah_combined.json", ... concat_dims=("time",), ... ) # doctest: +SKIP >>> "refs" in manifest or "version" in manifest # doctest: +SKIP True
Source code in src/pyramids_lazy/_kerchunk_facade.py
328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 | |
The native builder#
pyramids_lazy._kerchunk_builder.build_single_manifest(src_path, *, src_url=None, inline_threshold=500, vlen_encode='embed')
#
Build a kerchunk v1 reference manifest for one NetCDF4 / HDF5 file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src_path
|
str | Path
|
Local path used to read chunk byte ranges and HDF5 metadata. |
required |
src_url
|
str | None
|
URL written into byte-range references. Defaults to
|
None
|
inline_threshold
|
int
|
Chunks smaller than this many bytes are embedded directly (base64) rather than referenced by offset. |
500
|
vlen_encode
|
str
|
VLEN string handling mode (reserved; object dtypes are currently rejected with a clear error). |
'embed'
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The manifest dict |
Raises:
| Type | Description |
|---|---|
ImportError
|
When h5py is not installed. |
ValueError
|
When the file uses an unsupported HDF5 feature. |
Source code in src/pyramids_lazy/_kerchunk_builder.py
pyramids_lazy._kerchunk_builder.combine_manifests(per_file, *, concat_dim)
#
Concatenate per-file manifests along concat_dim into one manifest.
Every variable whose _ARRAY_DIMENSIONS contains concat_dim is stacked
along that axis (its concat-axis chunk indices are renumbered and its
shape summed); every other variable must be identical across files and is
carried from the first. No zarr group is created, so this cannot hit the
zarr-v3 sync() deadlock (#530).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
per_file
|
list[dict[str, Any]]
|
Per-file manifests, in concatenation order, as returned by
:func: |
required |
concat_dim
|
str
|
Dimension name to concatenate along. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The combined manifest dict |
Raises:
| Type | Description |
|---|---|
ValueError
|
When |