Skip to content

Kerchunk manifests#

Backs NetCDF.to_kerchunk() and NetCDF.combine_kerchunk() / DatasetCollection.to_kerchunk(). Serialises a NetCDF archive into a kerchunk JSON reference manifest so consumers can open it as a lazy Zarr-backed cube with zero rewrite — the manifest holds byte-range pointers into the source files; no pixel data is moved.

Backend methods to_kerchunk, combine_kerchunk
Modules pyramids_lazy._kerchunk_facade, pyramids_lazy._kerchunk_builder
Returns the manifest dict (also written to output_path)
from pyramids.netcdf import NetCDF

NetCDF.read_file("era5.nc").to_kerchunk("era5.json")
NetCDF.combine_kerchunk(["2009.json", "2010.json"], "combined.json")

Two builders, and why#

The facade dispatches between kerchunk's own translators and a native builder in this package.

The native builder exists because of a deadlock: kerchunk.hdf drives zarr's synchronous v2 API, which under zarr v3 submits work to a process-global event loop and hangs intermittently. The builder walks the HDF5 container with h5py and emits zarr-v2 metadata plus byte-range chunk references directly, never instantiating a live zarr group. Its output is the ordinary kerchunk reference document, so the consumer side is unchanged.

Its scope is the feature surface GDAL's netCDF driver writes: fixed-width numeric and fixed-length string dtypes, deflate/shuffle filters, and dimension scales. An unsupported HDF5 feature raises a clear ValueError rather than producing a subtly wrong manifest.

API#

pyramids_lazy._kerchunk_facade.to_kerchunk(src_path, output_path, *, inline_threshold=500, vlen_encode='embed', backend='native') #

Emit a single-file kerchunk reference manifest.

Parameters:

Name Type Description Default
src_path str | Path

Path or URL of the source NetCDF / HDF5 file.

required
output_path str | Path

Path where the JSON manifest is written.

required
inline_threshold int

Chunks smaller than this many bytes are embedded directly in the manifest rather than referenced by offset.

500
vlen_encode str

One of "embed" | "null" | "leave" | "encode". Controls how VLEN (variable-length) strings are handled. Default "embed" inlines string values; other modes trade compatibility vs fidelity — see the kerchunk docs.

'embed'
backend str

"native" (default) builds the manifest directly with h5py — no live zarr group, so it cannot hit the zarr-v3 sync() deadlock (#530). "legacy" forces the kerchunk.hdf.SingleHdf5ToZarr translator. The native backend falls back to the legacy translator for files using HDF5 features it does not support (e.g. vlen/compound dtypes). "kerchunk" is a deprecated alias for "legacy".

'native'

Returns:

Type Description
dict[str, Any]

The manifest dict that was written — useful for inspection

dict[str, Any]

or further programmatic use.

Raises:

Type Description
ImportError

When the chosen backend's dependency is missing (h5py for "native", kerchunk for "legacy").

ValueError

When backend is not "native" or "legacy".

Examples:

  • Emit a manifest for one NetCDF file (requires kerchunk, a pyramids-lazy dependency):
    >>> from pathlib import Path  # doctest: +SKIP
    >>> from pyramids_lazy._kerchunk_facade import to_kerchunk  # doctest: +SKIP
    >>> manifest = to_kerchunk(
    ...     "noah_20240101.nc", "noah_20240101.kerchunk.json",
    ... )  # doctest: +SKIP
    >>> "refs" in manifest or "version" in manifest  # doctest: +SKIP
    True
    
Source code in src/pyramids_lazy/_kerchunk_facade.py
def to_kerchunk(
    src_path: str | Path,
    output_path: str | Path,
    *,
    inline_threshold: int = 500,
    vlen_encode: str = "embed",
    backend: str = "native",
) -> dict[str, Any]:
    """Emit a single-file kerchunk reference manifest.

    Args:
        src_path: Path or URL of the source NetCDF / HDF5 file.
        output_path: Path where the JSON manifest is written.
        inline_threshold: Chunks smaller than this many bytes are
            embedded directly in the manifest rather than referenced
            by offset.
        vlen_encode: One of `"embed" | "null" | "leave" | "encode"`.
            Controls how VLEN (variable-length) strings are handled.
            Default `"embed"` inlines string values; other modes
            trade compatibility vs fidelity — see the kerchunk docs.
        backend: `"native"` (default) builds the manifest directly with
            h5py — no live zarr group, so it cannot hit the zarr-v3
            `sync()` deadlock (#530). `"legacy"` forces the
            `kerchunk.hdf.SingleHdf5ToZarr` translator. The native
            backend falls back to the legacy translator for files using
            HDF5 features it does not support (e.g. vlen/compound dtypes).
            `"kerchunk"` is a deprecated alias for `"legacy"`.

    Returns:
        The manifest dict that was written — useful for inspection
        or further programmatic use.

    Raises:
        ImportError: When the chosen backend's dependency is missing
            (h5py for `"native"`, kerchunk for `"legacy"`).
        ValueError: When `backend` is not `"native"` or `"legacy"`.

    Examples:
        - Emit a manifest for one NetCDF file (requires kerchunk, a
          pyramids-lazy dependency):
            ```python
            >>> from pathlib import Path  # doctest: +SKIP
            >>> from pyramids_lazy._kerchunk_facade import to_kerchunk  # doctest: +SKIP
            >>> manifest = to_kerchunk(
            ...     "noah_20240101.nc", "noah_20240101.kerchunk.json",
            ... )  # doctest: +SKIP
            >>> "refs" in manifest or "version" in manifest  # doctest: +SKIP
            True

            ```
    """
    backend = _normalize_backend(backend)
    src_str = str(src_path)

    def _legacy() -> dict[str, Any]:
        return _kerchunk_translate_single(
            src_str, inline_threshold=inline_threshold, vlen_encode=vlen_encode
        )

    if backend == "legacy":
        refs = _legacy()
    else:
        refs = _native_or_fallback(
            lambda: build_single_manifest(
                src_str, inline_threshold=inline_threshold, vlen_encode=vlen_encode
            ),
            _legacy,
            local_inputs_exist=os.path.exists(src_str),
            subject=f"native kerchunk builder could not handle {src_str!r}",
        )
    Path(output_path).write_text(json.dumps(refs))
    return refs

pyramids_lazy._kerchunk_facade.combine_kerchunk(src_paths, output_path, *, concat_dims=('time',), identical_dims=('lat', 'lon'), inline_threshold=500, backend='native') #

Emit a combined kerchunk manifest spanning many source files.

The default "native" backend scans each file with :func:build_single_manifest (h5py, no live zarr group) and concatenates the per-file manifests along the single concat_dims entry — every variable carrying that dimension is stacked, the rest are carried from the first file. This avoids the zarr-v3 sync() deadlock (#530). "legacy" forces the SingleHdf5ToZarr + MultiZarrToZarr path; the native backend also falls back to it for inputs it cannot handle (e.g. multiple concat dims, unsupported HDF5 features). "kerchunk" is a deprecated alias for "legacy".

Parameters:

Name Type Description Default
src_paths Sequence[str | Path]

Sequence of source NetCDF / HDF5 paths or URLs, in concatenation order.

required
output_path str | Path

Path where the combined JSON manifest is written.

required
concat_dims Sequence[str]

Dimension name(s) along which to concatenate. The native backend supports exactly one; more than one falls back to kerchunk. Default ("time",).

('time',)
identical_dims Sequence[str]

Dimension name(s) expected to be identical across files (e.g. shared lat/lon). Used by the kerchunk backend; the native backend treats every non-concat variable as identical implicitly. Default ("lat", "lon").

('lat', 'lon')
inline_threshold int

Same semantics as :func:to_kerchunk.

500
backend str

"native" (default) or "legacy" ("kerchunk" is a deprecated alias).

'native'

Returns:

Type Description
dict[str, Any]

The combined manifest dict that was written.

Raises:

Type Description
ImportError

When the chosen backend's dependency is missing.

ValueError

When backend is not "native" or "legacy".

Examples:

  • Combine a year's worth of daily NetCDFs into one manifest:
    >>> from pathlib import Path  # doctest: +SKIP
    >>> from pyramids_lazy._kerchunk_facade import combine_kerchunk  # doctest: +SKIP
    >>> srcs = sorted(Path("/data/noah").glob("noah_*.nc"))  # doctest: +SKIP
    >>> manifest = combine_kerchunk(
    ...     srcs, "noah_combined.json",
    ...     concat_dims=("time",),
    ... )  # doctest: +SKIP
    >>> "refs" in manifest or "version" in manifest  # doctest: +SKIP
    True
    
Source code in src/pyramids_lazy/_kerchunk_facade.py
def combine_kerchunk(
    src_paths: Sequence[str | Path],
    output_path: str | Path,
    *,
    concat_dims: Sequence[str] = ("time",),
    identical_dims: Sequence[str] = ("lat", "lon"),
    inline_threshold: int = 500,
    backend: str = "native",
) -> dict[str, Any]:
    """Emit a combined kerchunk manifest spanning many source files.

    The default `"native"` backend scans each file with :func:`build_single_manifest`
    (h5py, no live zarr group) and concatenates the per-file manifests along the
    single `concat_dims` entry — every variable carrying that dimension is stacked,
    the rest are carried from the first file. This avoids the zarr-v3 `sync()`
    deadlock (#530). `"legacy"` forces the
    `SingleHdf5ToZarr` + `MultiZarrToZarr` path; the native backend also falls back
    to it for inputs it cannot handle (e.g. multiple concat dims, unsupported HDF5
    features). `"kerchunk"` is a deprecated alias for `"legacy"`.

    Args:
        src_paths: Sequence of source NetCDF / HDF5 paths or URLs, in
            concatenation order.
        output_path: Path where the combined JSON manifest is written.
        concat_dims: Dimension name(s) along which to concatenate. The native
            backend supports exactly one; more than one falls back to kerchunk.
            Default `("time",)`.
        identical_dims: Dimension name(s) expected to be identical across files
            (e.g. shared `lat`/`lon`). Used by the kerchunk backend; the native
            backend treats every non-concat variable as identical implicitly.
            Default `("lat", "lon")`.
        inline_threshold: Same semantics as :func:`to_kerchunk`.
        backend: `"native"` (default) or `"legacy"` (`"kerchunk"` is a deprecated alias).

    Returns:
        The combined manifest dict that was written.

    Raises:
        ImportError: When the chosen backend's dependency is missing.
        ValueError: When `backend` is not `"native"` or `"legacy"`.

    Examples:
        - Combine a year's worth of daily NetCDFs into one manifest:
            ```python
            >>> from pathlib import Path  # doctest: +SKIP
            >>> from pyramids_lazy._kerchunk_facade import combine_kerchunk  # doctest: +SKIP
            >>> srcs = sorted(Path("/data/noah").glob("noah_*.nc"))  # doctest: +SKIP
            >>> manifest = combine_kerchunk(
            ...     srcs, "noah_combined.json",
            ...     concat_dims=("time",),
            ... )  # doctest: +SKIP
            >>> "refs" in manifest or "version" in manifest  # doctest: +SKIP
            True

            ```
    """
    backend = _normalize_backend(backend)

    def _legacy() -> dict[str, Any]:
        return _kerchunk_combine(
            src_paths,
            concat_dims=concat_dims,
            identical_dims=identical_dims,
            inline_threshold=inline_threshold,
        )

    def _native() -> dict[str, Any]:
        if len(concat_dims) != 1:
            raise ValueError(
                "native combine supports exactly one concat dimension; got "
                f"{tuple(concat_dims)}"
            )
        per_file = [
            build_single_manifest(str(path), inline_threshold=inline_threshold)
            for path in src_paths
        ]
        return combine_manifests(per_file, concat_dim=concat_dims[0])

    if backend == "legacy":
        combined = _legacy()
    else:
        combined = _native_or_fallback(
            _native,
            _legacy,
            local_inputs_exist=all(os.path.exists(str(path)) for path in src_paths),
            subject="native kerchunk combine could not handle these inputs",
        )
    Path(output_path).write_text(json.dumps(combined))
    return combined

The native builder#

pyramids_lazy._kerchunk_builder.build_single_manifest(src_path, *, src_url=None, inline_threshold=500, vlen_encode='embed') #

Build a kerchunk v1 reference manifest for one NetCDF4 / HDF5 file.

Parameters:

Name Type Description Default
src_path str | Path

Local path used to read chunk byte ranges and HDF5 metadata.

required
src_url str | None

URL written into byte-range references. Defaults to str(src_path) so the manifest points back at the source as named (matching kerchunk's behaviour for local paths).

None
inline_threshold int

Chunks smaller than this many bytes are embedded directly (base64) rather than referenced by offset.

500
vlen_encode str

VLEN string handling mode (reserved; object dtypes are currently rejected with a clear error).

'embed'

Returns:

Type Description
dict[str, Any]

The manifest dict {"version": 1, "refs": {...}}.

Raises:

Type Description
ImportError

When h5py is not installed.

ValueError

When the file uses an unsupported HDF5 feature.

Source code in src/pyramids_lazy/_kerchunk_builder.py
def build_single_manifest(
    src_path: str | Path,
    *,
    src_url: str | None = None,
    inline_threshold: int = 500,
    vlen_encode: str = "embed",
) -> dict[str, Any]:
    """Build a kerchunk v1 reference manifest for one NetCDF4 / HDF5 file.

    Args:
        src_path: Local path used to read chunk byte ranges and HDF5 metadata.
        src_url: URL written into byte-range references. Defaults to
            ``str(src_path)`` so the manifest points back at the source as
            named (matching kerchunk's behaviour for local paths).
        inline_threshold: Chunks smaller than this many bytes are embedded
            directly (base64) rather than referenced by offset.
        vlen_encode: VLEN string handling mode (reserved; object dtypes are
            currently rejected with a clear error).

    Returns:
        The manifest dict ``{"version": 1, "refs": {...}}``.

    Raises:
        ImportError: When h5py is not installed.
        ValueError: When the file uses an unsupported HDF5 feature.
    """
    h5py = _require_h5py()
    src_str = str(src_path)
    src_local = src_str if os.path.exists(src_str) else None
    if src_url is not None:
        url = src_url
    elif src_local is not None:
        # Absolutise local paths so the manifest resolves regardless of the
        # consumer's working directory, and use forward slashes (as_posix) so a
        # Windows-written manifest carries a portable path rather than back-slashes
        # that some fsspec/zarr consumers mishandle.
        url = Path(src_str).resolve().as_posix()
    else:
        url = src_str

    refs: dict[str, Any] = {}
    # One binary handle for the whole walk so inlined chunks don't reopen the file
    # per chunk; None for a remote source (those are always referenced, not inlined).
    src_handle = open(src_local, "rb") if src_local is not None else None
    try:
        with h5py.File(src_str, "r") as h5file:
            refs[_ZARR_GROUP] = json.dumps({"zarr_format": 2})
            refs[_ZARR_ATTRS] = json.dumps(_clean_attrs(h5file.attrs), allow_nan=False)

            def visit(name: str, obj: Any) -> None:
                if isinstance(obj, h5py.Dataset):
                    _emit_dataset(
                        refs,
                        h5file=h5file,
                        name=name,
                        dataset=obj,
                        src_url=url,
                        src_handle=src_handle,
                        inline_threshold=inline_threshold,
                        vlen_encode=vlen_encode,
                    )
                else:
                    refs[f"{name}/{_ZARR_GROUP}"] = json.dumps({"zarr_format": 2})
                    refs[f"{name}/{_ZARR_ATTRS}"] = json.dumps(
                        _clean_attrs(obj.attrs), allow_nan=False
                    )

            h5file.visititems(visit)
    finally:
        if src_handle is not None:
            src_handle.close()

    return {"version": 1, "refs": refs}

pyramids_lazy._kerchunk_builder.combine_manifests(per_file, *, concat_dim) #

Concatenate per-file manifests along concat_dim into one manifest.

Every variable whose _ARRAY_DIMENSIONS contains concat_dim is stacked along that axis (its concat-axis chunk indices are renumbered and its shape summed); every other variable must be identical across files and is carried from the first. No zarr group is created, so this cannot hit the zarr-v3 sync() deadlock (#530).

Parameters:

Name Type Description Default
per_file list[dict[str, Any]]

Per-file manifests, in concatenation order, as returned by :func:build_single_manifest.

required
concat_dim str

Dimension name to concatenate along.

required

Returns:

Type Description
dict[str, Any]

The combined manifest dict {"version": 1, "refs": {...}}.

Raises:

Type Description
ValueError

When per_file is empty, files disagree on the variable set, a concat variable has inconsistent chunking, or a non-concat variable is not identical across files.

Source code in src/pyramids_lazy/_kerchunk_builder.py
def combine_manifests(
    per_file: list[dict[str, Any]],
    *,
    concat_dim: str,
) -> dict[str, Any]:
    """Concatenate per-file manifests along ``concat_dim`` into one manifest.

    Every variable whose ``_ARRAY_DIMENSIONS`` contains ``concat_dim`` is stacked
    along that axis (its concat-axis chunk indices are renumbered and its
    ``shape`` summed); every other variable must be identical across files and is
    carried from the first. No zarr group is created, so this cannot hit the
    zarr-v3 ``sync()`` deadlock (#530).

    Args:
        per_file: Per-file manifests, in concatenation order, as returned by
            :func:`build_single_manifest`.
        concat_dim: Dimension name to concatenate along.

    Returns:
        The combined manifest dict ``{"version": 1, "refs": {...}}``.

    Raises:
        ValueError: When ``per_file`` is empty, files disagree on the variable
            set, a concat variable has inconsistent chunking, or a non-concat
            variable is not identical across files.
    """
    if not per_file:
        raise ValueError("combine_manifests requires at least one manifest.")
    refs_list = [_manifest_refs(m) for m in per_file]
    if len(refs_list) == 1:
        return {"version": 1, "refs": dict(refs_list[0])}

    var_sets = [set(_variable_names(refs)) for refs in refs_list]
    if any(names != var_sets[0] for names in var_sets[1:]):
        raise ValueError(
            "cannot combine manifests with differing variable sets: "
            f"{[sorted(names) for names in var_sets]}"
        )

    combined: dict[str, Any] = {}
    variables = set(_variable_names(refs_list[0]))
    for key, value in refs_list[0].items():
        is_root_meta = "/" not in key
        is_subgroup_meta = (
            key.endswith((_ZARR_GROUP, _ZARR_ATTRS))
            and key.rsplit("/", 1)[0] not in variables
        )
        # carry root metadata and sub-group metadata (not owned by a variable)
        if is_root_meta or is_subgroup_meta:
            combined[key] = value

    for name in _variable_names(refs_list[0]):
        dims = json.loads(refs_list[0][f"{name}/{_ZARR_ATTRS}"]).get(
            "_ARRAY_DIMENSIONS", []
        )
        if concat_dim in dims:
            combined.update(_concat_variable(name, refs_list, dims.index(concat_dim)))
        else:
            combined.update(_identical_variable(name, refs_list))

    return {"version": 1, "refs": combined}