Catchment attributes, basin polygons, and the base extension¶
The quickstart covers the common case: pick catchments, get a daily table. This notebook covers the four things it leaves out, each of which you reach for once the daily series alone stops being enough:
| capability | what it gives you | when you need it |
|---|---|---|
with_attributes=True |
static catchment properties joined onto every row | regression, clustering, anything comparing catchments |
with_geometry=True |
the basin polygons | mapping, spatial joins, area checks |
version="1.2" |
an older, range-readable base release |
reaching CAMELS/HYSETS cheaply |
allow_full_download=True |
the current base release |
you actually want all ~16,300 catchments |
It closes by opening the transport directly, so the "3 MB out of 8.84 GB" claim stops being a number in a README and becomes something you can watch happen.
Setup¶
Everything here is anonymous static HTTP — no credentials, no accounts.
import matplotlib.pyplot as plt
import pandas as pd
from earthlens.caravan import Caravan, Catalog
OUT = "outputs"
1. Static attributes¶
Every Caravan archive ships per-catchment attributes alongside the timeseries: where the gauge is, how big the basin is, and a set of derived climate indices (aridity, mean precipitation, snow fraction, seasonality).
with_attributes=True joins them onto every returned row. That is what turns a stack of hydrographs into something you can compare — you can only ask "do drier catchments respond differently?" once each row knows its own aridity.
Denmark is used here because it is the smallest archive (0.52 GB), so the notebook stays quick.
danish = Caravan(
start="2015-01-01",
end="2015-12-31",
variables=["streamflow", "total_precipitation"],
lat_lim=[-90, 90],
lon_lim=[-180, 180],
dataset="denmark",
country="Denmark",
with_attributes=True,
path=OUT,
).download(limit=2190)
The frame now carries the static columns next to the daily ones. Note they repeat per row — they describe the catchment, not the day.
danish[
[
"gauge_id",
"date",
"streamflow",
"gauge_name",
"area",
"aridity_ERA5_LAND",
"p_mean",
]
].head()
Using them: runoff ratio against aridity¶
With the attributes attached, a real question becomes a one-liner. The runoff ratio — the fraction of rainfall that leaves as streamflow — should fall as a catchment gets drier, because more water evaporates before it reaches the river.
Both streamflow and total_precipitation_sum are in mm/day, so the ratio is simply one summed over the other.
per_catchment = danish.groupby("gauge_id").agg(
runoff=("streamflow", "sum"),
rainfall=("total_precipitation_sum", "sum"),
aridity=("aridity_ERA5_LAND", "first"),
area=("area", "first"),
)
per_catchment["runoff_ratio"] = per_catchment["runoff"] / per_catchment["rainfall"]
per_catchment
fig, ax = plt.subplots(figsize=(7, 4.5))
ax.scatter(
per_catchment["aridity"],
per_catchment["runoff_ratio"],
s=per_catchment["area"] / 5,
alpha=0.75,
color="#1f6feb",
)
ax.set_xlabel("aridity index (ERA5-Land)")
ax.set_ylabel("runoff ratio (streamflow / precipitation)")
ax.set_title("Danish catchments, 2015 — marker area ∝ catchment area")
ax.grid(alpha=0.3)
fig.tight_layout()
plt.show()
A handful of catchments is far too few to claim a relationship — the point is that the question is now expressible. Run the same cell against dataset="grdc" with a wide bounding box and the scatter becomes a real climate gradient.
2. Basin polygons¶
with_geometry=True additionally reads the basin shapefiles. The polygons do not go into the frame — they would repeat once per row — so they are attached to the backend as .geometry, a pyramids FeatureCollection.
This is why the backend object is worth keeping rather than discarding after download().
source = Caravan(
start="2015-01-01",
end="2015-01-31",
variables=["streamflow"],
lat_lim=[-90, 90],
lon_lim=[-180, 180],
dataset="denmark",
country="Denmark",
with_geometry=True,
path=OUT,
)
source.download(limit=93)
basins = source.geometry
type(basins).__name__
The collection proxies its underlying GeoDataFrame, so it plots and measures directly. Every Danish basin is included — the shapefile is per source dataset, not per selected catchment.
fig, ax = plt.subplots(figsize=(6, 7))
basins.plot(ax=ax, facecolor="#cfe3ff", edgecolor="#1f6feb", linewidth=0.4)
ax.set_title(f"CAMELS-DK basin boundaries ({len(basins)} catchments)")
ax.set_xlabel("longitude")
ax.set_ylabel("latitude")
fig.tight_layout()
plt.show()
3. The base extension is guarded¶
base is where CAMELS-US, CAMELS-AUS, CAMELS-BR, CAMELS-CL, CAMELS-GB, HYSETS and LamaH-CE live — roughly two-thirds of all Caravan catchments. It is also the one release that is not range-readable: since v1.4 it ships as a single .tar.gz, a gzip stream with no directory, so reaching one catchment means transferring all 25–29 GB of it.
Requesting it therefore raises rather than quietly starting that download.
try:
Caravan(
start="2000-01-01",
end="2000-01-05",
variables=["streamflow"],
lat_lim=[-90, 90],
lon_lim=[-180, 180],
dataset="base",
gauge_ids=["camels_01022500"],
path=OUT,
)
except ValueError as exc:
print(exc)
The message names both ways forward. allow_full_download=True accepts the transfer; version="1.2" reaches CAMELS through the last release that was still a ZIP, and therefore still range-readable.
The cheap route is not a free lunch, and the catalog records why.
base = Catalog().get_extension("base")
pd.DataFrame(
[
{
"version": key,
"catchments": release.n_catchments,
"measured?": release.n_catchments_verified,
"period": release.data_period,
"columns": release.column_set,
"format": release.file_for("csv").archive_format,
"GB": round(release.file_for("csv").size / 1e9, 1),
}
for key, release in sorted(base.versions.items())
]
)
1.2 is a materially different dataset, not merely an older cut: 42% of the catchments, forcing ending in 2020 rather than 2023, and the legacy column set — one potential_evaporation_sum instead of the ERA5-Land/FAO pair the current releases split it into.
Ask for potential_evaporation against it and the backend resolves the legacy name for you.
camels = Caravan(
start="2000-01-01",
end="2000-01-10",
variables=["streamflow", "potential_evaporation"],
lat_lim=[-90, 90],
lon_lim=[-180, 180],
dataset="base",
version="1.2",
gauge_ids=["camels_01022500", "hysets_01010070"],
path=OUT,
).download()
camels
Two catchments from two different national datasets — CAMELS-US and HYSETS — in one frame with one schema. That interoperability is the whole reason Caravan exists.
4. Watching the transport¶
Everything above rests on one idea: a ZIP stores its file directory at the tail, so with HTTP range requests you can read that directory and then pull single members, without the rest of the archive ever crossing the network.
transfer_stats reports what a request actually cost, so the claim is checkable rather than asserted.
grdc = Caravan(
start="2010-01-01",
end="2010-12-31",
variables=["streamflow"],
lat_lim=[-90, 90],
lon_lim=[-180, 180],
dataset="grdc",
gauge_ids=["GRDC_1159100"],
path=OUT,
)
rows = grdc.download()
requests_made, transferred_mb = grdc.transfer_stats
archive_gb = grdc.archive_file.size / 1e9
print(f"{len(rows)} rows returned")
print(
f"{transferred_mb:.2f} MB in {requests_made} requests, from a {archive_gb:.2f} GB archive"
)
fig, ax = plt.subplots(figsize=(7, 2.6))
ax.barh(
["downloaded", "transferred"],
[archive_gb * 1000, transferred_mb],
color=["#d0d7de", "#1f6feb"],
)
ax.set_xscale("log")
ax.set_xlabel("megabytes (log scale)")
ax.set_title(
f"One catchment-year: {archive_gb * 1000 / transferred_mb:.0f}x less than the archive"
)
for index, value in enumerate([archive_gb * 1000, transferred_mb]):
ax.text(value, index, f" {value:,.0f} MB", va="center")
ax.grid(alpha=0.3, axis="x")
fig.tight_layout()
plt.show()
Closing the backend releases the archive handle and the HTTP session. download() deliberately does not do it for you, because the statistics above live on the archive.
grdc.close()
source.close()
Takeaway¶
with_attributes=Trueturns per-catchment series into something comparable — the runoff-ratio-against-aridity question needs it.with_geometry=Trueattaches basin polygons to.geometry, not to the frame, because they describe the catchment rather than the day.baseis gated, and the error names both escapes.version="1.2"is cheap but is a smaller, older, differently-columned dataset — check the version table before treating it as a substitute.transfer_statsmakes the range-read design auditable from your own session.
Next: the extension table for what each release contains, and usage for the full option surface.