Aggregation API#
The temporal aggregator: reduce a downloaded stack into windowed composites (daily mean, monthly sum, …). For the guide with worked examples, see Temporal aggregation.
aggregate= is forwarded only when both conditions hold: the backend's OUTPUT_KIND is raster or mixed
— the shapes a gridded reduction is defined for — and the backend declares SUPPORTS_AGGREGATE. A vector /
tabular backend is refused because the aggregator has no meaning on GeoDataFrame / DataFrame rows; a raster
backend that has not wired the reducer is refused for that reason instead. Either way the refusal is a
NotImplementedError raised before the backend's download runs. See Base contracts.
AggregationConfig#
earthlens.core.AggregationConfig
#
Bases: BaseModel
Frozen request shape consumed by :func:aggregate_netcdf.
Carries the windowing frequency, reduction operator, and output
location. Frozen + extra="forbid" so a typo in a field name
(e.g. freqency=) fails loud at construction time rather than
silently using the default.
Attributes:
| Name | Type | Description |
|---|---|---|
freq |
str
|
Pandas offset alias defining the window. Examples:
|
op |
OperationLiteral
|
Reduction applied within each window. |
out_dir |
Path | None
|
Directory the per-window GeoTIFFs are written to.
Created (with parents) if absent. |
cell_size |
float
|
Pixel size in degrees, embedded in the output
filename as a metadata note. |
level |
int | float | None
|
When the NetCDF has a |
skipna |
bool
|
When |
min_count |
int | None
|
Minimum non-NaN samples required for a window to
produce a non-NaN value. Windows with fewer samples emit
NaN. |
Examples:
-
Daily-mean defaults — only
freqis required, the rest stays at sensible CDS-shaped defaults:- Monthly sum into an explicit output directory: - Pin a pressure level for 4-D inputs and require a minimum sample count per window:>>> from earthlens.aggregate import AggregationConfig >>> cfg = AggregationConfig(freq="1D") >>> cfg.op 'auto' >>> cfg.skipna True >>> cfg.cell_size 0.125 >>> cfg.out_dir is None True
Source code in libs/core/src/earthlens/aggregate.py
509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 | |
aggregate_netcdf#
earthlens.core.aggregate_netcdf(nc_path, var_info, config)
#
Slice a CDS-shaped NetCDF into per-window aggregated outputs.
Reads the NetCDF, groups its time axis by config.freq, reduces
each group with config.op, and (when config.out_dir is set)
writes one GeoTIFF per window. Returns the per-window arrays
alongside their timestamps and output paths so callers can chain
further processing without re-opening the files.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
nc_path
|
Path | str
|
Path to the NetCDF on disk. |
required |
var_info
|
Variable
|
Catalog row for the variable being aggregated. Used
to pick the variable from the NetCDF
( |
required |
config
|
AggregationConfig
|
Frozen :class: |
required |
Returns:
| Type | Description |
|---|---|
list[tuple[Timestamp, ndarray | None, Path | None]]
|
list[tuple[pd.Timestamp, np.ndarray | None, Path | None]]: One |
list[tuple[Timestamp, ndarray | None, Path | None]]
|
entry per window. The first item is the window's left-edge |
list[tuple[Timestamp, ndarray | None, Path | None]]
|
timestamp; the second is the reduced 2-D array — |
list[tuple[Timestamp, ndarray | None, Path | None]]
|
the request set |
list[tuple[Timestamp, ndarray | None, Path | None]]
|
disk; the third is the GeoTIFF path (or |
list[tuple[Timestamp, ndarray | None, Path | None]]
|
|
Raises:
| Type | Description |
|---|---|
KeyError
|
If the NetCDF has no recognised time variable
( |
ValueError
|
If |
See Also
- :class:
AggregationConfig: the frozen request payload. - :class:
earthlens.ecmwf.Catalog: resolves(dataset, code)pairs to the :class:earthlens.ecmwf.Variablerows that drivevar_info.is_fluxand the output filename. examples/post_process_ecmwf_netcdf.py: thin CLI demo of this function (after task L1).
Source code in libs/core/src/earthlens/aggregate.py
iter_aggregate_netcdf#
Streams one reduced window at a time instead of materialising the whole cube — this is what keeps memory bounded on a long time series.
earthlens.core.iter_aggregate_netcdf(nc_path, var_info, config)
#
Yield one :class:AggregatedWindow per time window, streaming.
The streaming counterpart to :func:aggregate_netcdf, and the
implementation it is built on. Two properties make it usable on cubes
that do not fit in memory:
- Only one window is resident at a time. The time steps for the
current window are read band by band and stacked; the whole
(time, y, x)cube is never materialised. A ten-year hourly ERA5 request over Europe at 0.25° is ~33.6 GB as one array but ~9 MB per daily window. - Windows are not accumulated. Each is yielded and then dropped, so
a caller that writes and discards holds nothing. Pair it with
keep_arrays=Falseto drop the reduced array too once it is on disk.
Every handle opened — the container, the variable subset, and the level-pinned view — is closed when the generator finishes or is abandoned, so the file can be deleted or overwritten straight after. Closing the container alone is not sufficient: the variable subset holds its own handle, and one left open keeps a Windows lock on the file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
nc_path
|
Path | str
|
Path to the NetCDF on disk. |
required |
var_info
|
Variable
|
Catalog row for the variable being aggregated. Used to
pick the variable from the NetCDF ( |
required |
config
|
AggregationConfig
|
Frozen :class: |
required |
Yields:
| Name | Type | Description |
|---|---|---|
AggregatedWindow |
AggregatedWindow
|
One per window, in time order. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If the NetCDF has no recognised time variable
( |
ValueError
|
If |
See Also
- :func:
aggregate_netcdf: the eagerlistform of this function. - :class:
AggregationConfig: the frozen request payload.
Source code in libs/core/src/earthlens/aggregate.py
720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 | |
AggregatedWindow#
earthlens.core.AggregatedWindow
dataclass
#
One reduced time window: its label, its array, and where it was written.
Yielded by :func:iter_aggregate_netcdf. array is None when the
request set keep_arrays=False and the window was written to disk — the
point of that mode is that a long run does not accumulate every window in
memory alongside the GeoTIFFs it has already produced.
Comparison is left as identity (eq=False). A generated __eq__ would
compare the array field with ==, which for a numpy array yields an
elementwise array and then raises ValueError: truth value ... ambiguous;
the matching __hash__ would raise TypeError because an ndarray is
unhashable. Compare the fields you actually care about instead.
Attributes:
| Name | Type | Description |
|---|---|---|
label |
Timestamp
|
The window's left-edge timestamp. |
array |
ndarray | None
|
The reduced 2-D array, or |
path |
Path | None
|
The written GeoTIFF, or |
Examples:
- A window keeps both its array and its path when it was written:
>>> import numpy as np >>> import pandas as pd >>> from pathlib import Path >>> from earthlens.aggregate import AggregatedWindow >>> window = AggregatedWindow( ... label=pd.Timestamp("2020-01-01"), ... array=np.array([[1.0, 2.0]]), ... path=Path("out/t2m_1D_20200101.tif"), ... ) >>> window.label.strftime("%Y-%m-%d") '2020-01-01' >>> float(window.array.mean()) 1.5 >>> window.path.name 't2m_1D_20200101.tif' - A discarded array leaves only the label and the path to read back:
>>> import pandas as pd >>> from pathlib import Path >>> from earthlens.aggregate import AggregatedWindow >>> window = AggregatedWindow( ... label=pd.Timestamp("2020-02-01"), ... array=None, ... path=Path("out/t2m_1D_20200201.tif"), ... ) >>> window.array is None True >>> window.path.name 't2m_1D_20200201.tif'
Source code in libs/core/src/earthlens/aggregate.py
Reduction helpers#
Public since 0.12.0 (previously _reduce and _window_groups).
earthlens.aggregate.reduce_time_axis(arr, op, skipna, min_count)
#
Reduce a (time, lat, lon) slice along axis 0 with the named op.
Dispatches op to the matching numpy reducer (np.nanmean etc.
when skipna=True, plain np.mean etc. when skipna=False),
then masks pixels whose non-NaN sample count falls below
min_count.
op="auto" is not accepted here — aggregate_netcdf resolves
auto to a concrete operator before calling this helper. Passing
"auto" raises KeyError to surface the mistake at the call site.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
arr
|
ndarray
|
Array to reduce. The first axis is collapsed; the
remaining axes pass through unchanged. Typically
|
required |
op
|
str
|
One of |
required |
skipna
|
bool
|
When |
required |
min_count
|
int | None
|
When set, pixels with fewer than this many non-NaN
samples along axis 0 emit NaN regardless of the reduction
result. |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
np.ndarray: Reduced array with axis 0 collapsed. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If |
Examples:
-
NaN-aware mean over the time axis:
- Strict mean propagates NaN when>>> import numpy as np >>> from earthlens.aggregate import reduce_time_axis >>> arr = np.array([[[1.0, 2.0]], [[3.0, np.nan]], [[5.0, 6.0]]]) >>> reduce_time_axis(arr, op="mean", skipna=True, min_count=None).tolist() [[3.0, 4.0]]skipna=False:->>> import numpy as np >>> from earthlens.aggregate import reduce_time_axis >>> arr = np.array([[[1.0, np.nan]], [[3.0, 4.0]]]) >>> result = reduce_time_axis(arr, op="mean", skipna=False, min_count=None) >>> bool(np.isnan(result[0, 1])), float(result[0, 0]) (True, 2.0)min_countmasks under-sampled pixels:
Source code in libs/core/src/earthlens/aggregate.py
336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 | |
earthlens.aggregate.window_groups(time_axis, freq)
#
Yield (window_label, mask) pairs that bucket time_axis by freq.
Builds a pandas.Series indexed by time_axis and groups it
with pandas.Grouper(freq=freq). Each group's index gives the
timestamps belonging to that window; the boolean mask is built
by membership against time_axis so callers can use it to slice
a numpy array along its first axis.
Empty groups (windows with no samples) are silently skipped —
aggregate_netcdf doesn't write a GeoTIFF for a window it has
no data for.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
time_axis
|
DatetimeIndex
|
Time coordinate as a :class: |
required |
freq
|
str
|
Pandas offset alias ( |
required |
Yields:
| Type | Description |
|---|---|
Timestamp
|
tuple[pd.Timestamp, np.ndarray]: For each non-empty window: |
ndarray
|
the group key (window's left-edge timestamp) paired with a |
tuple[Timestamp, ndarray]
|
boolean mask of length |
Examples:
-
Group four 6-hourly slots into one daily window:
- Group two days of 6-hourly samples into two daily windows:>>> import pandas as pd >>> from earthlens.aggregate import window_groups >>> idx = pd.date_range("2022-01-01", periods=4, freq="6h") >>> windows = list(window_groups(idx, "1D")) >>> len(windows) 1 >>> label, mask = windows[0] >>> label Timestamp('2022-01-01 00:00:00') >>> mask.tolist() [True, True, True, True]