The cube as a nested dict#
Export a cube's whole structure to plain Python objects and rebuild it from them. It needs a
multidimensional open — reopen with open_as_multi_dimensional=True if the cube was opened classically.
Every other export goes to a
file or a foreign object — to_file, to_zarr, to_kerchunk, to_dataframe, to_xarray, to_stac_item.
This is the one that goes to nothing but dict, list, str and numbers, which is what inspection, diffing two
cubes, JSON transport and asserting on structure in a test all want.
payload = cube.to_dict() # values included
structure = cube.to_dict(data=False) # structure only: no read_array call
rebuilt = NetCDF.from_dict(payload)
The schema is a superset of xarray's#
dims, coords, data_vars and attrs keep their xarray meanings, so a reader that knows only xarray's
schema finds what it expects. Everything GDAL needs rides under one extra pyramids key:
| key | what it carries |
|---|---|
schema |
the schema version, refused by from_dict if it is newer than this reader |
epsg |
the CRS, None only under data=False |
geotransform |
the six affine coefficients |
spatial_dims |
the (y, x) pair every variable in the payload shares |
no_data_value |
the sentinel per variable |
coords carries the band dimensions only — the spatial axes are the geotransform rather than coordinate
arrays, so an xarray-only reader of this payload gets a Dataset with no spatial coordinates. Under
data=False both variables and coordinates report dims/dtype/shape/attrs and no data.
xarray's schema has no slot for a CRS or a geotransform — a crs attribute survives there only because it
happens to ride in attrs. For a GDAL-backed cube the georeferencing is the object, so matching xarray exactly
would have made the round trip silently drop it.
The payload records pyramids' dimension layout — band axes first, (y, x) last — which is the shape
read_array returns. A store's own declared order is not always that: variable U of the CAM/CESM fixture in
this repo declares ['time', 'lat', 'lev', 'lon'], interleaving a spatial axis between two band axes.
What round-trips, and what does not#
Round-trips: dimensions and their order, a band dimension's coordinates and their CF attributes (so a time axis keeps its units and calendar), variable arrays and dtypes, the CRS, the geotransform, the no-data sentinels and the spatial dimension names.
When variables sit on different band axes, add_variable leaves the later ones' coordinate map empty upstream
of this module, so those axes arrive already unstamped and from_dict warns that it rebuilt them so.
Does not, in each case because from_array is the only constructor and takes neither:
- Per-variable attributes.
to_dictreports every one it finds — 32 on this repo'scf__5v__1d4-3d1__geog__y-desc.nc— and the rebuilt variable carries only thegrid_mappingthatfrom_arrayregenerates. There is no per-variable attribute setter to restore them through. - Arbitrary global attributes. Globals reach
from_arrayas aCFAttributes, a fixed set of CF fields rather than a free mapping, so the CF-recognised ones are restored and the rest dropped. - Non-gridded variables. They have no raster plane, so
from_arraycannot rebuild them;to_dictdrops them with a warning naming each. 12 of the 43 variables in the CAM/CESM fixture are of this kind.
A no-data sentinel also widens to a float for an integer cube (-1 becomes -1.0); the array dtype itself
round-trips exactly.
Because of the attribute losses above, to_dict(a) == to_dict(from_dict(to_dict(a))) is not an identity —
the two payloads differ in attrs and in each variable's attrs, while dims, coords, shape, dtype and
every value agree. Diff on those keys rather than on the whole dict.
JSON#
Python's json emits NaN and Infinity and reads them back, so a payload round-trips through
json.dumps/loads between Python processes. Those tokens are not RFC-8259 JSON, so JSON.parse,
serde_json and strict Python parsers reject them: a gappy raster's payload is Python-JSON, not wire-JSON.
Convert the non-finite cells before sending one somewhere strict.
API#
The cube as plain Python objects: NetCDF.to_dict out, NetCDF.from_dict back.
The methods are the public spelling. The module-level functions behind them are cube_to_dict
and cube_from_dict rather than to_dict / from_dict, because pyramids.netcdf already
exports a to_dict — metadata.to_dict(NetCDFMetadata), a different function taking a different
argument — and from pyramids.netcdf import to_dict binds that one. Two same-named functions in
one package is a trap for anyone following a docs page to the obvious import.
Every other export goes to a file or a foreign object — to_file, to_zarr, to_kerchunk,
to_dataframe, to_xarray, to_stac_item. This is the one that goes to nothing but dict,
list, str and numbers, which is what inspection, diffing two cubes, JSON transport and
asserting on structure in a test all want.
The schema is a superset of xarray's, and that is the whole design decision. xarray's
to_dict is keyed dims / coords / data_vars / attrs and has no slot for a CRS or a
geotransform — a crs attribute survives there only because it happens to ride in attrs. For
a GDAL-backed cube the georeferencing is the object, so matching xarray exactly would make the
round trip silently drop it. Instead those four keys keep their xarray meaning exactly, and
everything GDAL needs lives under one extra pyramids key. A reader that knows only xarray's
schema still finds what it expects.
One limit on that: coords carries the band dimensions only, because those are the axes a
cube stamps. The spatial axes are not coordinate arrays here — they are the geotransform, from
which y and x are derived — so an xarray-only reader of this payload gets a Dataset with no
spatial coordinates. xr.Dataset.from_dict accepts it, but the georeferencing is in the
pyramids block, which by definition such a reader is not looking at.
What round-trips: dimensions and their order, a band dimension's coordinates and their CF attributes (a time axis keeps its units and calendar), variable arrays and dtypes, the CRS, the geotransform, the no-data sentinels and the spatial dimension names.
One caveat on the coordinates, which is upstream of this module rather than in it: when variables
sit on different band axes, add_variable leaves the later ones' coordinate map empty, so those
axes arrive here already unstamped and from_dict warns that it rebuilt them so. The warning
names the payload because that is what it can see; the loss happened before to_dict was called.
What does not round-trip, in both cases because from_array is the only constructor and it
takes neither:
- Per-variable attributes.
to_dictreports every one it finds — 32 of them on this repo'scf__5v__1d4-3d1__geog__y-desc.nc— and the rebuilt variable carries exactly one,grid_mapping, whichfrom_arrayregenerates itself. There is no per-variable attribute setter on the class to restore them through. The payload is complete; the constructor is the narrow part, and until it widens this is a real asymmetry rather than a stated-and-harmless one. - Arbitrary global attributes.
from_arraytakes globals as aCFAttributes, a fixed set of CF fields rather than a free mapping, sofrom_dictrestores the CF ones it recognises and drops the rest.
One narrower asymmetry: a no-data sentinel comes back as a float even for an integer cube
(-1 becomes -1.0), because it travels through from_array's no_data_value. The array dtype
itself round-trips exactly.
to_dict(a) == to_dict(from_dict(to_dict(a))) is therefore not an identity, and an earlier
version of this note wrongly said it was. Measured on the COARDS sample this repo ships, the two
payloads differ in attrs and in every variable's attrs — the attribute losses above — while
dims, coords, shape, dtype and every value agree. Diff on those keys, not on the whole
dict.
A cube holding non-gridded variables is lossy for a third reason: they have no raster plane,
so from_array cannot rebuild them, and to_dict drops them with a warning naming each.
cube_to_dict(nc, *, data=True)
#
Export a cube's full structure as a nested dict.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
nc
|
NetCDF
|
The container or variable to export. |
required |
data
|
bool
|
Whether to include the values. |
True
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: The payload, keyed |
dict[str, Any]
|
meanings, plus |
dict[str, Any]
|
sentinels and schema version. |
Raises:
| Type | Description |
|---|---|
ValueError
|
The cube has no data variables, or declares no CRS, so the payload could only rebuild into an unreferenced cube. |
Examples:
- The four xarray keys, plus the one that carries the georeferencing:
>>> import numpy as np
>>> from pyramids.netcdf import ExtraDimensions, GeoReference, NetCDF
>>> cube = NetCDF.from_array(
... np.array([1.0, 2.0]).reshape(2, 1, 1),
... geo_ref=GeoReference(geo=(0.0, 1.0, 0.0, 1.0, 0.0, -1.0), epsg=4326),
... variable_name="t",
... dims=ExtraDimensions(name="time", values=[0.0, 6.0]),
... )
>>> payload = cube.to_dict()
>>> sorted(payload)
['attrs', 'coords', 'data_vars', 'dims', 'pyramids']
>>> payload["dims"]
{'time': 2, 'y': 1, 'x': 1}
>>> payload["coords"]["time"]["data"]
[0.0, 6.0]
pyramids block is what xarray's schema has nowhere to put:
>>> import numpy as np
>>> from pyramids.netcdf import ExtraDimensions, GeoReference, NetCDF
>>> cube = NetCDF.from_array(
... np.array([1.0, 2.0]).reshape(2, 1, 1),
... geo_ref=GeoReference(geo=(0.0, 1.0, 0.0, 1.0, 0.0, -1.0), epsg=4326),
... variable_name="t",
... dims=ExtraDimensions(name="time", values=[0.0, 6.0]),
... )
>>> block = cube.to_dict()["pyramids"]
>>> block["epsg"], block["spatial_dims"]
(4326, ['y', 'x'])
>>> block["geotransform"]
[0.0, 1.0, 0.0, 1.0, 0.0, -1.0]
data=False keeps the shape and drops the values, for inspecting or diffing:
>>> import numpy as np
>>> from pyramids.netcdf import ExtraDimensions, GeoReference, NetCDF
>>> cube = NetCDF.from_array(
... np.array([1.0, 2.0]).reshape(2, 1, 1),
... geo_ref=GeoReference(geo=(0.0, 1.0, 0.0, 1.0, 0.0, -1.0), epsg=4326),
... variable_name="t",
... dims=ExtraDimensions(name="time", values=[0.0, 6.0]),
... )
>>> entry = cube.to_dict(data=False)["data_vars"]["t"]
>>> entry["shape"], entry["dtype"]
([2, 1, 1], 'float64')
>>> "data" in entry
False
See Also
cube_from_dict: Rebuilds a cube from what this returns. pyramids.netcdf.NetCDF.to_dataframe: The cube as a pandas frame instead, which is a flat table rather than the structure.
Source code in src/pyramids/netcdf/dict_io.py
406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 | |
cube_from_dict(payload)
#
Rebuild a cube from a nested dict as :func:to_dict produces.
The acceptance criterion is the round trip, not the parse: to_dict then from_dict gives
back an equivalent cube, georeferencing included.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
payload
|
Any
|
The dict to rebuild from. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Container |
Container
|
The rebuilt cube. |
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
A required key is missing; |
Examples:
- The round trip, which is the contract: values and stamps come back:
>>> import numpy as np
>>> from pyramids.netcdf import ExtraDimensions, GeoReference, NetCDF
>>> cube = NetCDF.from_array(
... np.array([1.0, 2.0]).reshape(2, 1, 1),
... geo_ref=GeoReference(geo=(0.0, 1.0, 0.0, 1.0, 0.0, -1.0), epsg=4326),
... variable_name="t",
... dims=ExtraDimensions(name="time", values=[0.0, 6.0]),
... )
>>> rebuilt = NetCDF.from_dict(cube.to_dict())
>>> rebuilt.get_variable("t").read_array().ravel().tolist()
[1.0, 2.0]
>>> np.asarray(rebuilt.get_variable("t").coords["time"]).tolist()
[0.0, 6.0]
>>> import numpy as np
>>> from pyramids.netcdf import ExtraDimensions, GeoReference, NetCDF
>>> cube = NetCDF.from_array(
... np.array([1.0, 2.0]).reshape(2, 1, 1),
... geo_ref=GeoReference(geo=(0.0, 1.0, 0.0, 1.0, 0.0, -1.0), epsg=4326),
... variable_name="t",
... dims=ExtraDimensions(name="time", values=[0.0, 6.0]),
... )
>>> rebuilt = NetCDF.from_dict(cube.to_dict())
>>> rebuilt.epsg
4326
>>> tuple(rebuilt.geotransform)
(0.0, 1.0, 0, 1.0, 0, -1.0)
>>> import numpy as np
>>> from pyramids.netcdf import ExtraDimensions, GeoReference, NetCDF
>>> cube = NetCDF.from_array(
... np.array([1.0, 2.0]).reshape(2, 1, 1),
... geo_ref=GeoReference(geo=(0.0, 1.0, 0.0, 1.0, 0.0, -1.0), epsg=4326),
... variable_name="t",
... dims=ExtraDimensions(name="time", values=[0.0, 6.0]),
... )
>>> NetCDF.from_dict(cube.to_dict(data=False)) # doctest: +ELLIPSIS
Traceback (most recent call last):
...
ValueError: from_dict() needs values for 't', ... structure only and cannot be rebuilt.
See Also
cube_to_dict: Produces the payload this consumes. pyramids.netcdf.NetCDF.from_array: The constructor this builds on, and the reason only the CF-recognised global attributes are restored.
Source code in src/pyramids/netcdf/dict_io.py
693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 | |