NetCDF Utilities

Low-level GDAL helpers for attribute reading, no-data extraction, scale/offset handling, time conversion, and dtype utilities.

pyramids.netcdf.utils

cf_epoch_units(resolution)

The CF units string for offsets counted from the pyramids epoch.

The writers disagree, deliberately, on the resolution they count in: the collection axis uses nanoseconds because an int64 count at that scale round-trips datetime64[ns] exactly, while the xarray interop path uses fractional seconds because it must also carry NaT as NaN. What they must not disagree on is the epoch, which is why only that half is shared.

Parameters:

Name	Type	Description	Default
resolution	str	A CF time unit name, e.g. "nanoseconds" or "seconds".	required

Returns:

Name	Type	Description
str	str	"<resolution> since <epoch>".

Note

Both readers now parse every resolution this builds, and they preserve different amounts of it. :func:pyramids.netcdf.utils.decode_cf_time scales the offsets to datetime64[ns] in integer nanoseconds, so the "nanoseconds since ..." axis DatasetCollection.to_netcdf writes comes back exact -- which is the point of writing at that resolution. :func:pyramids.netcdf.utils.create_time_conversion_func formats through datetime, whose finest unit is the microsecond, so a nanosecond axis is rounded there; the residual is visible only in a %f format.

This is worth stating because it was wrong twice. The docstring first claimed decode_cf_time parsed all of them (it raised on nanoseconds, because it handed the string to cftime); the correction then recorded that gap as if it were the design. It was a defect: is_cf_time_units accepted the axis, so the writer produced a file the reader refused. The test that should have caught either version asserted cf_epoch_units(r) == f"{r} since {CF_EPOCH}" against a function whose body is that f-string -- so it could not fail, whatever the string was for.

Examples:

• The collection axis counts in nanoseconds:

  >>> from pyramids.base._cf_epoch import cf_epoch_units
  >>> cf_epoch_units("nanoseconds")
  'nanoseconds since 1970-01-01 00:00:00'
  

• The interop path counts in seconds, from the same epoch -- which is the whole reason the epoch is shared and the resolution is not:

  >>> from pyramids.base._cf_epoch import cf_epoch_units
  >>> cf_epoch_units("seconds")
  'seconds since 1970-01-01 00:00:00'
  

• Whatever the resolution, the epoch behind it is the same one, which is what lets two axes written at different scales be compared:

  >>> from pyramids.base._cf_epoch import cf_epoch_units
  >>> epochs = {
  ...     cf_epoch_units(res).split(" since ")[1]
  ...     for res in ("seconds", "nanoseconds", "days")
  ... }
  >>> sorted(epochs)
  ['1970-01-01 00:00:00']
  

See Also

pyramids.netcdf.utils.create_time_conversion_func: Formats any of these strings back to text, at microsecond resolution. pyramids.netcdf.utils.decode_cf_time: Decodes them to datetime64[ns], at the full nanosecond resolution -- see the note above.

Source code in src/pyramids/base/_cf_epoch.py

def cf_epoch_units(resolution: str) -> str:
    """The CF `units` string for offsets counted from the pyramids epoch.

    The writers disagree, deliberately, on the resolution they count in: the
    collection axis uses `nanoseconds` because an `int64` count at that scale
    round-trips `datetime64[ns]` exactly, while the xarray interop path uses
    fractional `seconds` because it must also carry `NaT` as `NaN`. What they
    must not disagree on is the epoch, which is why only that half is shared.

    Args:
        resolution: A CF time unit name, e.g. `"nanoseconds"` or `"seconds"`.

    Returns:
        str: `"<resolution> since <epoch>"`.

    Note:
        Both readers now parse every resolution this builds, and they preserve
        different amounts of it. :func:`pyramids.netcdf.utils.decode_cf_time`
        scales the offsets to `datetime64[ns]` in integer nanoseconds, so the
        `"nanoseconds since ..."` axis `DatasetCollection.to_netcdf` writes
        comes back exact -- which is the point of writing at that resolution.
        :func:`pyramids.netcdf.utils.create_time_conversion_func` formats
        through `datetime`, whose finest unit is the microsecond, so a
        nanosecond axis is rounded there; the residual is visible only in a
        `%f` format.

        This is worth stating because it was wrong twice. The docstring first
        claimed `decode_cf_time` parsed all of them (it raised on
        `nanoseconds`, because it handed the string to `cftime`); the
        correction then recorded that gap as if it were the design. It was a
        defect: `is_cf_time_units` accepted the axis, so the writer produced a
        file the reader refused. The test that should have caught either
        version asserted `cf_epoch_units(r) == f"{r} since {CF_EPOCH}"` against
        a function whose body is that f-string -- so it could not fail,
        whatever the string was for.

    Examples:
        - The collection axis counts in nanoseconds:
            ```python
            >>> from pyramids.base._cf_epoch import cf_epoch_units
            >>> cf_epoch_units("nanoseconds")
            'nanoseconds since 1970-01-01 00:00:00'

            ```
        - The interop path counts in seconds, from the same epoch -- which is
          the whole reason the epoch is shared and the resolution is not:
            ```python
            >>> from pyramids.base._cf_epoch import cf_epoch_units
            >>> cf_epoch_units("seconds")
            'seconds since 1970-01-01 00:00:00'

            ```
        - Whatever the resolution, the epoch behind it is the same one, which
          is what lets two axes written at different scales be compared:
            ```python
            >>> from pyramids.base._cf_epoch import cf_epoch_units
            >>> epochs = {
            ...     cf_epoch_units(res).split(" since ")[1]
            ...     for res in ("seconds", "nanoseconds", "days")
            ... }
            >>> sorted(epochs)
            ['1970-01-01 00:00:00']

            ```

    See Also:
        pyramids.netcdf.utils.create_time_conversion_func: Formats any of these
            strings back to text, at microsecond resolution.
        pyramids.netcdf.utils.decode_cf_time: Decodes them to `datetime64[ns]`,
            at the full nanosecond resolution -- see the note above.
    """
    return f"{resolution} since {CF_EPOCH}"

resolve_full_name(obj, group_full_name, fallback_name)

Full hierarchical name of a GDAL dimension / array, with a qualified fallback.

Tries obj.GetFullName(); when that raises (some drivers don't expose it on every object), composes the path from the parent group's full name and fallback_name — "<group>/<name>", or "/<name>" when the parent is the root group.

Parameters:

Name	Type	Description	Default
obj	Any	A GDAL dimension or MDArray.	required
group_full_name	str	Full name of the parent group (e.g. "/" or "/grp").	required
fallback_name	str	The object's short name, used to build the fallback path.	required

Returns:

Type	Description
str	The resolved full-name string.

Source code in src/pyramids/netcdf/utils.py

def resolve_full_name(obj: Any, group_full_name: str, fallback_name: str) -> str:
    """Full hierarchical name of a GDAL dimension / array, with a qualified fallback.

    Tries ``obj.GetFullName()``; when that raises (some drivers don't expose it on every
    object), composes the path from the parent group's full name and ``fallback_name`` —
    ``"<group>/<name>"``, or ``"/<name>"`` when the parent is the root group.

    Args:
        obj: A GDAL dimension or MDArray.
        group_full_name: Full name of the parent group (e.g. ``"/"`` or ``"/grp"``).
        fallback_name: The object's short name, used to build the fallback path.

    Returns:
        The resolved full-name string.
    """
    try:
        return cast("str", obj.GetFullName())
    except Exception:
        if group_full_name != "/":
            return f"{group_full_name}/{fallback_name}"
        return f"/{fallback_name}"

create_time_conversion_func(units, out_format='%Y-%m-%d %H:%M:%S', calendar='standard')

Create a converter that maps numeric CF time offsets to date strings.

Parses CF-compliant time unit strings (e.g., "days since 1979-01-01") and returns a callable that converts numeric offsets to formatted date strings.

The fast datetime + timedelta path is taken only for the Gregorian family (standard, gregorian, proleptic_gregorian) on an origin at or after the 1582-10-15 reform. Everything else is resolved by cftime.num2date(): every other calendar (julian, noleap, 365_day, all_leap, 366_day, 360_day), and — whatever the calendar is called, proleptic_gregorian included — any origin that predates the reform, since timedelta counts proleptic Gregorian days from it while a standard axis counts Julian ones. On "hours since 1-1-1 00:00:0.0" the offset 17549208 is 2003-01-01 under standard and 2003-01-03 under proleptic_gregorian; the two agree only from 1582-10-15 on.

Parameters:

Name	Type	Description	Default
units	str	CF time unit string in the format "<unit> since <origin>". Supported units are days, hours, minutes, seconds, milliseconds, microseconds, and nanoseconds. Sub-second units decode at microsecond resolution (Python datetime's finest), so a nanosecond axis is rounded to the nearest microsecond — exact for date/second output, and visible only in a %f (microsecond) out_format.	required
out_format	str	strftime format for the output strings. Defaults to "%Y-%m-%d %H:%M:%S".	'%Y-%m-%d %H:%M:%S'
calendar	str	CF calendar type. Defaults to "standard". Anything outside the Gregorian family is decoded with cftime (a core dependency), as is any calendar at all when the origin in units predates the 1582 reform.	'standard'

Returns:

Name	Type	Description
Callable	Callable	A function that takes a numeric value and returns a formatted date string.

Raises:

Type	Description
ValueError	If the unit string cannot be parsed or uses an unsupported time unit.

Examples:

• Convert day offsets from a 1979 origin:

  >>> from pyramids.netcdf.utils import (
  ...     create_time_conversion_func,
  ... )
  >>> convert = create_time_conversion_func(
  ...     "days since 1979-01-01"
  ... )
  >>> convert(0)
  '1979-01-01 00:00:00'
  >>> convert(365)
  '1980-01-01 00:00:00'
  

• Use hour-based units with a custom format:

  >>> convert = create_time_conversion_func(
  ...     "hours since 2000-01-01",
  ...     out_format="%Y-%m-%d",
  ... )
  >>> convert(24)
  '2000-01-02'
  >>> convert(0)
  '2000-01-01'
  

• A pre-1582 origin is resolved by calendar, not by timedelta, so the mixed standard calendar and the proleptic one answer two days apart:

  >>> mixed = create_time_conversion_func(
  ...     "hours since 1-1-1 00:00:0.0"
  ... )
  >>> mixed(17549208)
  '2003-01-01 00:00:00'
  >>> proleptic = create_time_conversion_func(
  ...     "hours since 1-1-1 00:00:0.0",
  ...     calendar="proleptic_gregorian",
  ... )
  >>> proleptic(17549208)
  '2003-01-03 00:00:00'
  

• A 360-day calendar counts twelve 30-day months, so day 360 is a year on:

  >>> convert = create_time_conversion_func(
  ...     "days since 2000-01-01",
  ...     calendar="360_day",
  ... )
  >>> convert(360)
  '2001-01-01 00:00:00'
  >>> convert(30)
  '2000-02-01 00:00:00'
  

See Also

_parse_units_origin: Parses the unit string. _origin_precedes_reform: Decides whether the origin needs cftime.

Source code in src/pyramids/netcdf/utils.py

def create_time_conversion_func(
    units: str,
    out_format: str = "%Y-%m-%d %H:%M:%S",
    calendar: str = "standard",
) -> Callable:
    """Create a converter that maps numeric CF time offsets to date strings.

    Parses CF-compliant time unit strings (e.g.,
    `"days since 1979-01-01"`) and returns a callable that
    converts numeric offsets to formatted date strings.

    The fast `datetime` + `timedelta` path is taken only for the Gregorian
    family (`standard`, `gregorian`, `proleptic_gregorian`) on an origin at or
    after the 1582-10-15 reform. Everything else is resolved by
    `cftime.num2date()`: every other calendar (`julian`, `noleap`, `365_day`,
    `all_leap`, `366_day`, `360_day`), and — whatever the calendar is called,
    `proleptic_gregorian` included — any origin that predates the reform, since
    `timedelta` counts proleptic Gregorian days from it while a `standard` axis
    counts Julian ones. On `"hours since 1-1-1 00:00:0.0"` the offset 17549208
    is `2003-01-01` under `standard` and `2003-01-03` under
    `proleptic_gregorian`; the two agree only from 1582-10-15 on.

    Args:
        units: CF time unit string in the format
            `"<unit> since <origin>"`. Supported units are days, hours,
            minutes, seconds, milliseconds, microseconds, and nanoseconds.
            Sub-second units decode at microsecond resolution (Python
            `datetime`'s finest), so a nanosecond axis is rounded to the
            nearest microsecond — exact for date/second output, and visible
            only in a `%f` (microsecond) `out_format`.
        out_format: strftime format for the output strings.
            Defaults to `"%Y-%m-%d %H:%M:%S"`.
        calendar: CF calendar type. Defaults to `"standard"`. Anything
            outside the Gregorian family is decoded with `cftime` (a core
            dependency), as is any calendar at all when the origin in `units`
            predates the 1582 reform.

    Returns:
        Callable: A function that takes a numeric value and
            returns a formatted date string.

    Raises:
        ValueError: If the unit string cannot be parsed or
            uses an unsupported time unit.

    Examples:
        - Convert day offsets from a 1979 origin:
            ```python
            >>> from pyramids.netcdf.utils import (
            ...     create_time_conversion_func,
            ... )
            >>> convert = create_time_conversion_func(
            ...     "days since 1979-01-01"
            ... )
            >>> convert(0)
            '1979-01-01 00:00:00'
            >>> convert(365)
            '1980-01-01 00:00:00'

            ```

        - Use hour-based units with a custom format:
            ```python
            >>> convert = create_time_conversion_func(
            ...     "hours since 2000-01-01",
            ...     out_format="%Y-%m-%d",
            ... )
            >>> convert(24)
            '2000-01-02'
            >>> convert(0)
            '2000-01-01'

            ```

        - A pre-1582 origin is resolved by calendar, not by `timedelta`, so
          the mixed `standard` calendar and the proleptic one answer two days
          apart:
            ```python
            >>> mixed = create_time_conversion_func(
            ...     "hours since 1-1-1 00:00:0.0"
            ... )
            >>> mixed(17549208)
            '2003-01-01 00:00:00'
            >>> proleptic = create_time_conversion_func(
            ...     "hours since 1-1-1 00:00:0.0",
            ...     calendar="proleptic_gregorian",
            ... )
            >>> proleptic(17549208)
            '2003-01-03 00:00:00'

            ```

        - A 360-day calendar counts twelve 30-day months, so day 360 is a
          year on:
            ```python
            >>> convert = create_time_conversion_func(
            ...     "days since 2000-01-01",
            ...     calendar="360_day",
            ... )
            >>> convert(360)
            '2001-01-01 00:00:00'
            >>> convert(30)
            '2000-02-01 00:00:00'

            ```

    See Also:
        _parse_units_origin: Parses the unit string.
        _origin_precedes_reform: Decides whether the origin needs `cftime`.
    """
    converter = None

    # A `standard` / `gregorian` axis is the *mixed* Julian-Gregorian calendar, and Python's
    # `datetime` is proleptic Gregorian — the two diverge before the 1582 reform, by ten days
    # at the cutover and more further back. So an origin that predates it has to go through
    # `cftime` as well, or `hours since 1-1-1` decodes two days late (#1140). The origin test
    # is calendar-blind on purpose: it diverts `proleptic_gregorian` too, which `cftime` then
    # answers proleptically — the same date `timedelta` would have given, by the one route
    # that is right for every calendar name. Origins from 1582-10-15 on, which is every real
    # store bar the year-one ones, keep the fast `timedelta` path.
    if not _is_standard_calendar(calendar) or _origin_precedes_reform(units):

        def convert_cftime(value):
            dt = cftime.num2date(value, units, calendar)
            return dt.strftime(out_format)

        converter = convert_cftime
    else:
        unit, origin = _parse_units_origin(units)

        # datetime/timedelta is microsecond-resolution, so the offset is resolved as
        # ``origin + timedelta(microseconds=value * factor)``: day/hour/minute/second
        # stay exact over any realistic date range, while the sub-second units (notably
        # the ``nanoseconds`` axis DatasetCollection.to_netcdf writes) round to the
        # nearest microsecond — exact for date/second output, with any residual only in
        # the microsecond digit of a ``%f`` format. The scale comes from the shared
        # nanosecond table so this reader and ``decode_cf_time`` cannot drift on which
        # unit names they accept.
        nanos = _ns_per_cf_unit(unit)
        if nanos is None:
            raise ValueError(f"Unsupported time unit: {unit!r}")
        factor = nanos / 1000.0

        def convert(value):
            dt = origin + timedelta(microseconds=float(value) * factor)
            return dt.strftime(out_format)

        converter = convert

    return converter

cf_units_text(units)

The units attribute as text, or None when it is not text at all.

A units read straight out of an HDF5 / netCDF attribute can arrive as bytes. That is the same unit, undecoded -- not a different kind of thing -- so both the predicate below and the decoder that gates on it resolve it here rather than each deciding for itself. Answering "not a time unit" for bytes put them back in the hole the predicate exists to fill; decoding them in the predicate alone would have let decode_cf_time hand the raw bytes to cftime, which raises.

Parameters:

Name	Type	Description	Default
units	Any	The CF units attribute -- a string, the bytes an undecoded attribute arrives as, None, or whatever a malformed file put there. GDAL normalises attributes into scalars or lists, so a units written as a one-element array is a real input.	required

Returns:

Type	Description
str | None	str | None: The text, or None for a non-text value and for bytes that are not valid UTF-8.

Examples:

• A string is itself, and bytes are decoded:

  >>> from pyramids.netcdf.utils import cf_units_text
  >>> cf_units_text("days since 1970-01-01")
  'days since 1970-01-01'
  >>> cf_units_text(b"days since 1970-01-01")
  'days since 1970-01-01'
  

• Anything that is not text at all answers None, rather than raising the way a bare " since " in unit would:

  >>> from pyramids.netcdf.utils import cf_units_text
  >>> print(cf_units_text(None), cf_units_text(1), cf_units_text(["a"]))
  None None None
  

See Also

is_cf_time_units: Asks whether that text names a CF time axis.

Source code in src/pyramids/netcdf/utils.py

def cf_units_text(units: Any) -> str | None:
    """The `units` attribute as text, or `None` when it is not text at all.

    A `units` read straight out of an HDF5 / netCDF attribute can arrive as
    `bytes`. That is the same unit, undecoded -- not a different kind of thing
    -- so both the predicate below and the decoder that gates on it resolve it
    here rather than each deciding for itself. Answering "not a time unit" for
    bytes put them back in the hole the predicate exists to fill; decoding them
    in the predicate alone would have let `decode_cf_time` hand the raw bytes
    to `cftime`, which raises.

    Args:
        units: The CF `units` attribute -- a string, the bytes an undecoded
            attribute arrives as, `None`, or whatever a malformed file put
            there. GDAL normalises attributes into scalars *or lists*, so a
            `units` written as a one-element array is a real input.

    Returns:
        str | None: The text, or `None` for a non-text value and for bytes
            that are not valid UTF-8.

    Examples:
        - A string is itself, and bytes are decoded:
            ```python
            >>> from pyramids.netcdf.utils import cf_units_text
            >>> cf_units_text("days since 1970-01-01")
            'days since 1970-01-01'
            >>> cf_units_text(b"days since 1970-01-01")
            'days since 1970-01-01'

            ```
        - Anything that is not text at all answers `None`, rather than
          raising the way a bare `" since " in unit` would:
            ```python
            >>> from pyramids.netcdf.utils import cf_units_text
            >>> print(cf_units_text(None), cf_units_text(1), cf_units_text(["a"]))
            None None None

            ```

    See Also:
        is_cf_time_units: Asks whether that text names a CF time axis.
    """
    if isinstance(units, str):
        text: str | None = units
    elif isinstance(units, (bytes, bytearray)):
        try:
            text = bytes(units).decode("utf-8")
        except UnicodeDecodeError:
            text = None
    else:
        text = None
    return text

is_cf_time_units(units)

True when units declares a CF time axis.

Four places asked this and answered differently. Axis detection matched a bare lowercased "since" substring; decoding required a case-sensitive " since ". A file whose units read "Days SINCE 1970-01-01" -- which cftime parses perfectly well -- was therefore reported as a time axis and then handed back as raw numbers, with nothing raised. This is the one rule both questions now use.

Total: anything that is not a string is not a time unit. GDAL attributes are normalised into scalars or lists, so a units written as a one-element array is a real input, and the CF compliance checker exists to report on malformed files rather than to crash on one.

Parameters:

Name	Type	Description	Default
units	str | bytes | None	The CF units attribute -- a string, the bytes an undecoded attribute arrives as, None, or whatever a malformed file put there.	required

Returns:

Name	Type	Description
bool	bool	True when units is a string with the <period> since <timestamp> shape.

Examples:

• The ordinary form, and the uppercase one that used to decode wrong:

  >>> from pyramids.netcdf.utils import is_cf_time_units
  >>> is_cf_time_units("days since 1970-01-01")
  True
  >>> is_cf_time_units("Days SINCE 1970-01-01")
  True
  

• A spatial unit is not a time axis, nor is a missing one:

  >>> from pyramids.netcdf.utils import is_cf_time_units
  >>> is_cf_time_units("degrees_north"), is_cf_time_units(None)
  (False, False)
  

• Neither is a units a malformed file wrote as a list or a number:

  >>> from pyramids.netcdf.utils import is_cf_time_units
  >>> is_cf_time_units(["days since 1970-01-01"])
  False
  >>> is_cf_time_units(1)
  False
  

• since has to be a word with a period and an epoch around it, so a unit that merely contains the letters is not matched:

  >>> from pyramids.netcdf.utils import is_cf_time_units
  >>> is_cf_time_units("sincerity"), is_cf_time_units("since")
  (False, False)
  

• Bytes are decoded first, so an attribute that arrived undecoded still names the axis it names:

  >>> from pyramids.netcdf.utils import is_cf_time_units
  >>> is_cf_time_units(b"days since 1970-01-01")
  True
  

See Also

decode_cf_time: Decodes the axes this identifies.

Source code in src/pyramids/netcdf/utils.py

def is_cf_time_units(units: str | bytes | None) -> bool:
    """True when `units` declares a CF time axis.

    Four places asked this and answered differently. Axis *detection* matched a
    bare lowercased `"since"` substring; *decoding* required a case-sensitive
    `" since "`. A file whose units read `"Days SINCE 1970-01-01"` -- which
    `cftime` parses perfectly well -- was therefore reported as a time axis and
    then handed back as raw numbers, with nothing raised. This is the one rule
    both questions now use.

    Total: anything that is not a string is not a time unit. GDAL attributes
    are normalised into scalars *or lists*, so a `units` written as a
    one-element array is a real input, and the CF compliance checker exists to
    *report* on malformed files rather than to crash on one.

    Args:
        units: The CF `units` attribute -- a string, the bytes an undecoded
            attribute arrives as, `None`, or whatever a malformed file put
            there.

    Returns:
        bool: True when `units` is a string with the `<period> since
            <timestamp>` shape.

    Examples:
        - The ordinary form, and the uppercase one that used to decode wrong:
            ```python
            >>> from pyramids.netcdf.utils import is_cf_time_units
            >>> is_cf_time_units("days since 1970-01-01")
            True
            >>> is_cf_time_units("Days SINCE 1970-01-01")
            True

            ```
        - A spatial unit is not a time axis, nor is a missing one:
            ```python
            >>> from pyramids.netcdf.utils import is_cf_time_units
            >>> is_cf_time_units("degrees_north"), is_cf_time_units(None)
            (False, False)

            ```
        - Neither is a `units` a malformed file wrote as a list or a number:
            ```python
            >>> from pyramids.netcdf.utils import is_cf_time_units
            >>> is_cf_time_units(["days since 1970-01-01"])
            False
            >>> is_cf_time_units(1)
            False

            ```
        - `since` has to be a word with a period and an epoch around it, so a
          unit that merely contains the letters is not matched:
            ```python
            >>> from pyramids.netcdf.utils import is_cf_time_units
            >>> is_cf_time_units("sincerity"), is_cf_time_units("since")
            (False, False)

            ```
        - Bytes are decoded first, so an attribute that arrived undecoded
          still names the axis it names:
            ```python
            >>> from pyramids.netcdf.utils import is_cf_time_units
            >>> is_cf_time_units(b"days since 1970-01-01")
            True

            ```

    See Also:
        decode_cf_time: Decodes the axes this identifies.
    """
    text = cf_units_text(units)
    return text is not None and _CF_TIME_UNITS.search(text) is not None

decode_cf_time(values, unit, calendar='standard', context=None)

Decode numeric CF time offsets to datetimes.

Standard / gregorian / proleptic_gregorian calendars yield datetime64[ns] when representable; non-standard calendars (360_day / noleap …) yield cftime objects. Arrays whose unit is not a "<interval> since <origin>" string are returned unchanged.

A Gregorian-family axis is decoded by adding its offsets to the origin in integer nanoseconds, rather than through cftime. Behaviour differs from earlier releases in two ways, both of them consequences of dropping cftime's microsecond floor: a "nanoseconds since …" axis decodes instead of raising, and a NaN offset decodes to NaT instead of to the origin. Both decode paths agree about missing values: the cftime fallback honours the mask cftime returns, so a NaN or an inf offset is missing there too rather than silently reading as the origin (#1116).

Anything the integer path cannot take exactly -- an unparseable origin, a period such as "months", an instant the integer nanosecond scale cannot reach, or a pre-1582 origin on a mixed Julian/Gregorian calendar -- still goes to cftime. That reach is not a fixed span of years: the gate sums the offset's magnitude and the origin's distance from 1970, so a far-from-1970 epoch shortens it (off days since 1900-01-01 it gives out in March 2115). That is a wider range than the integer path's, not a narrower one: a date cftime decodes is still cast to datetime64[ns] whenever it fits, so 2255-2262 -- past the integer scale but inside the type -- comes back as datetime64 all the same.

Parameters:

Name	Type	Description	Default
values	ndarray	The numeric values already read for the coordinate.	required
unit	str | bytes | None	The coordinate's CF unit string (e.g. "days since 1979-01-01"), or the bytes an undecoded attribute arrives as.	required
calendar	str	The CF calendar name. Defaults to "standard".	'standard'
context	str | None	Optional name of the axis being decoded, used only so the out-of-range warning can say which axis it is about. A store with several time axes otherwise warns with nothing but the units string to tell them apart.	None

Returns:

Type

Description

NDArray

np.ndarray: Decoded datetimes for a time axis, else values unchanged. For a time axis the dtype depends on the dates: datetime64[ns] when every one of them is representable in that type, and an object array otherwise -- either because the calendar is not a standard one, or because a date falls outside datetime64[ns]'s 1677-09-21 to 2262-04-11 range, which is warned about (#1087). The choice is array-wide, since one array has one dtype: a single out-of-range value keeps its in-range neighbours as objects too. A value cftime could not decode -- a NaN or an inf offset -- comes back as NaT in a datetime64 result and as None in an object one, since an object array of datetimes has no NaT (#1116). What that object array holds is cftime's own choice, not this function's, and it is made once per array from the units origin rather than per value: cftime.real_datetime (a datetime.datetime subclass, which pandas coerces to datetime64[us]) when the proleptic Gregorian rules already cover that origin -- a proleptic_gregorian calendar, or a mixed-calendar origin strictly after the 1582 reform -- and a true cftime datetime such as DatetimeGregorian when they do not, which means a mixed-calendar origin no later than the reform date itself (days since 1582-10-15 already gives DatetimeGregorian). Being an array-wide choice, such an axis stays DatetimeGregorian even in the offsets that land after the reform.

Raises:

Type

Description

ValueError

cftime cannot decode this axis -- most often a "months since …" axis on anything but 360_day, since a calendar month has no fixed length, and otherwise an unparseable origin or an offset landing outside datetime's year 1 to 9999. An offset so large the scale overflows a 64-bit integer arrives from cftime as an OverflowError and is normalised to this one type, so callers have a single failure to catch. Re-raised with the axis, units and calendar named, because cftime's own message identifies none of them (#1117).

Examples:

• A date past what datetime64[ns] holds keeps its real value, and says so:

  >>> import warnings
  >>> import numpy as np
  >>> from pyramids.netcdf.utils import decode_cf_time
  >>> with warnings.catch_warnings(record=True) as caught:
  ...     warnings.simplefilter("always")
  ...     decoded = decode_cf_time(
  ...         np.array([400_000]), "days since 1970-01-01", "standard"
  ...     )
  >>> decoded.dtype, decoded[0].year
  (dtype('O'), 3065)
  >>> caught[0].category.__name__
  'UserWarning'
  

• The resolution the collection writer counts in, read back with its sub-microsecond digits intact:

  >>> import numpy as np
  >>> from pyramids.netcdf.utils import cf_epoch_units, decode_cf_time
  >>> offsets = np.array([0, 1577836800123456789], dtype="int64")
  >>> decoded = decode_cf_time(
  ...     offsets, cf_epoch_units("nanoseconds"), "proleptic_gregorian"
  ... )
  >>> decoded.astype(str).tolist()
  ['1970-01-01T00:00:00.000000000', '2020-01-01T00:00:00.123456789']
  

See Also

is_cf_time_units: The predicate this gates on. It accepts every "<period> since <origin>" string, so a resolution it admits and this cannot decode is a defect -- which is what a nanoseconds axis was, accepted by the predicate and refused by cftime.

Source code in src/pyramids/netcdf/utils.py

def decode_cf_time(
    values: np.ndarray,
    unit: str | bytes | None,
    calendar: str = "standard",
    context: str | None = None,
) -> np.typing.NDArray:
    """Decode numeric CF time offsets to datetimes.

    Standard / gregorian / proleptic_gregorian calendars yield ``datetime64[ns]`` when
    representable; non-standard calendars (``360_day`` / ``noleap`` …) yield ``cftime``
    objects. Arrays whose ``unit`` is not a ``"<interval> since <origin>"`` string are
    returned unchanged.

    A Gregorian-family axis is decoded by adding its offsets to the origin in integer
    nanoseconds, rather than through ``cftime``. Behaviour differs from earlier releases
    in two ways, both of them consequences of dropping ``cftime``'s microsecond floor:
    a ``"nanoseconds since …"`` axis decodes instead of raising, and a ``NaN`` offset
    decodes to ``NaT`` instead of to the origin. Both decode paths agree about missing
    values: the ``cftime`` fallback honours the mask ``cftime`` returns, so a ``NaN`` or an
    ``inf`` offset is missing there too rather than silently reading as the origin (#1116).

    Anything the integer path cannot take exactly -- an unparseable origin, a period such as
    ``"months"``, an instant the integer nanosecond scale cannot reach, or a pre-1582 origin
    on a mixed Julian/Gregorian calendar -- still goes to ``cftime``. That reach is not a
    fixed span of years: the gate sums the offset's magnitude and the origin's distance from
    1970, so a far-from-1970 epoch shortens it (off ``days since 1900-01-01`` it gives out in
    March 2115). That is a wider range than the integer path's, not a narrower one: a date
    ``cftime`` decodes is still cast to ``datetime64[ns]`` whenever it fits, so 2255-2262 --
    past the integer scale but inside the type -- comes back as ``datetime64`` all the same.

    Args:
        values: The numeric values already read for the coordinate.
        unit: The coordinate's CF unit string (e.g. ``"days since 1979-01-01"``),
            or the bytes an undecoded attribute arrives as.
        calendar: The CF calendar name. Defaults to ``"standard"``.
        context: Optional name of the axis being decoded, used only so the out-of-range
            warning can say *which* axis it is about. A store with several time axes
            otherwise warns with nothing but the units string to tell them apart.

    Returns:
        np.ndarray: Decoded datetimes for a time axis, else ``values`` unchanged. For a
            time axis the dtype depends on the dates: ``datetime64[ns]`` when every one of
            them is representable in that type, and an object array otherwise -- either
            because the calendar is not a standard one, or because a date falls outside
            ``datetime64[ns]``'s 1677-09-21 to 2262-04-11 range, which is warned about
            (#1087). The choice is array-wide, since one array has one dtype: a single
            out-of-range value keeps its in-range neighbours as objects too. A value
            ``cftime`` could not decode -- a ``NaN`` or an ``inf`` offset -- comes back as
            ``NaT`` in a ``datetime64`` result and as ``None`` in an object one, since an
            object array of datetimes has no ``NaT`` (#1116).
            What that object array holds is ``cftime``'s own choice, not this
            function's, and it is made once per array from the units origin rather than per
            value: ``cftime.real_datetime`` (a ``datetime.datetime`` subclass, which
            ``pandas`` coerces to ``datetime64[us]``) when the proleptic Gregorian rules
            already cover that origin -- a ``proleptic_gregorian`` calendar, or a
            mixed-calendar origin strictly after the 1582 reform -- and a true ``cftime``
            datetime such as ``DatetimeGregorian`` when they do not, which means a
            mixed-calendar origin no later than the reform date itself
            (``days since 1582-10-15`` already gives ``DatetimeGregorian``). Being an
            array-wide choice, such an axis stays ``DatetimeGregorian`` even in the
            offsets that land after the reform.

    Raises:
        ValueError: ``cftime`` cannot decode this axis -- most often a ``"months since …"``
            axis on anything but ``360_day``, since a calendar month has no fixed length,
            and otherwise an unparseable origin or an offset landing outside ``datetime``'s
            year 1 to 9999.
            An offset so large the scale overflows a 64-bit integer arrives from ``cftime``
            as an ``OverflowError`` and is normalised to this one type, so callers have a
            single failure to catch. Re-raised with the axis, units and calendar named,
            because ``cftime``'s own message identifies none of them (#1117).

    Examples:
        - A date past what ``datetime64[ns]`` holds keeps its real value, and says so:
            ```python
            >>> import warnings
            >>> import numpy as np
            >>> from pyramids.netcdf.utils import decode_cf_time
            >>> with warnings.catch_warnings(record=True) as caught:
            ...     warnings.simplefilter("always")
            ...     decoded = decode_cf_time(
            ...         np.array([400_000]), "days since 1970-01-01", "standard"
            ...     )
            >>> decoded.dtype, decoded[0].year
            (dtype('O'), 3065)
            >>> caught[0].category.__name__
            'UserWarning'

            ```
        - The resolution the collection writer counts in, read back with its
          sub-microsecond digits intact:
            ```python
            >>> import numpy as np
            >>> from pyramids.netcdf.utils import cf_epoch_units, decode_cf_time
            >>> offsets = np.array([0, 1577836800123456789], dtype="int64")
            >>> decoded = decode_cf_time(
            ...     offsets, cf_epoch_units("nanoseconds"), "proleptic_gregorian"
            ... )
            >>> decoded.astype(str).tolist()
            ['1970-01-01T00:00:00.000000000', '2020-01-01T00:00:00.123456789']

            ```

    See Also:
        is_cf_time_units: The predicate this gates on. It accepts every
            ``"<period> since <origin>"`` string, so a resolution it admits and
            this cannot decode is a defect -- which is what a ``nanoseconds``
            axis was, accepted by the predicate and refused by ``cftime``.
    """
    # Through the same normaliser the predicate uses: `cftime` needs `str`, and
    # a `units` the predicate accepted as bytes would otherwise raise here.
    text = cf_units_text(unit)
    decoded: np.typing.NDArray
    if not is_cf_time_units(text):
        decoded = values
    else:
        standard = _is_standard_calendar(calendar)
        text = cast("str", text)
        exact = _decode_gregorian_ns(values, text, calendar) if standard else None
        if exact is not None:
            decoded = exact
        else:
            decoded = _decode_via_cftime(values, text, calendar, standard, context)
    return decoded

encode_cf_time(value, unit, calendar='standard')

Convert a date string / datetime to a coordinate's numeric CF scale.

The inverse of :func:decode_cf_time for a single value: used to translate a user-supplied selection bound back to the time axis's stored numbers.

Parameters:

Name	Type	Description	Default
value	Any	A date string, :class:datetime, or anything :class:pandas.Timestamp accepts.	required
unit	str	The CF unit string the numeric scale is expressed in.	required
calendar	str	The CF calendar name.	'standard'

Returns:

Name	Type	Description
float	float	value expressed in unit on the given calendar.

Source code in src/pyramids/netcdf/utils.py

def encode_cf_time(value: Any, unit: str, calendar: str = "standard") -> float:
    """Convert a date string / datetime to a coordinate's numeric CF scale.

    The inverse of :func:`decode_cf_time` for a single value: used to translate a
    user-supplied selection bound back to the time axis's stored numbers.

    Args:
        value: A date string, :class:`datetime`, or anything :class:`pandas.Timestamp`
            accepts.
        unit: The CF unit string the numeric scale is expressed in.
        calendar: The CF calendar name.

    Returns:
        float: ``value`` expressed in ``unit`` on the given ``calendar``.
    """
    year, month, day, hour, minute, second, microsecond = _datetime_components(value)
    dt = cftime.datetime(
        year,
        month,
        day,
        hour,
        minute,
        second,
        microsecond,
        calendar=calendar,
    )
    return float(cftime.date2num(dt, unit, calendar))

read_cf_attributes(obj)

Read a GDAL object's attributes the way a CF consumer needs them.

The CF reader for every consumer of units / calendar / standard_name / axis. Use this rather than :func:_read_attributes wherever the attributes are interpreted as CF: attributes alone omit units, because GDAL moves it to the object's unit slot, so a site that reads them raw silently loses it — which is how a time axis came back undecodable, CF axis detection by units never fired, and a UGRID variable's unit vanished on a round trip (#1078).

:func:_read_attributes remains the right call for a verbatim view of what the file declares (serialisation, round-trip writers); this one is for interpreting CF.

The slot is the CF units for the netCDF and HDF5 drivers, which is where this matters. Another multidim driver (Zarr, HDF-EOS) may fill it from a format-specific field, in which case the reported units is that driver's unit rather than a CF attribute the file literally declares — a reason to read attributes raw when the question is "what does this file say", not "what does this axis mean".

Parameters:

Name	Type	Description	Default
obj	Any	Any GDAL object exposing GetAttributes() and GetUnit().	required

Returns:

Type	Description
dict[str, AttributeValue]	dict[str, AttributeValue]: The object's attributes, including units.

Source code in src/pyramids/netcdf/utils.py

def read_cf_attributes(obj: Any) -> dict[str, AttributeValue]:
    """Read a GDAL object's attributes the way a CF consumer needs them.

    The CF reader for every consumer of ``units`` / ``calendar`` / ``standard_name`` /
    ``axis``. Use this rather than :func:`_read_attributes` wherever the attributes are
    interpreted as CF: attributes alone omit ``units``, because GDAL moves it to the object's
    unit slot, so a site that reads them raw silently loses it — which is how a time axis came
    back undecodable, CF axis detection by units never fired, and a UGRID variable's unit
    vanished on a round trip (#1078).

    :func:`_read_attributes` remains the right call for a verbatim view of what the file
    declares (serialisation, round-trip writers); this one is for interpreting CF.

    The slot is the CF ``units`` for the netCDF and HDF5 drivers, which is where this matters.
    Another multidim driver (Zarr, HDF-EOS) may fill it from a format-specific field, in which
    case the reported ``units`` is that driver's unit rather than a CF attribute the file
    literally declares — a reason to read attributes raw when the question is "what does this
    file say", not "what does this axis mean".

    Args:
        obj: Any GDAL object exposing ``GetAttributes()`` and ``GetUnit()``.

    Returns:
        dict[str, AttributeValue]: The object's attributes, including ``units``.
    """
    return _merge_unit(_read_attributes(obj), obj)