Precipitation — CHIRPS daily rainfall¶
CHIRPS daily rainfall (5.5 km, 1981-present) over the Lake Victoria region, reduced to one monthly mean over June 2024.
Setup¶
The imports for the whole notebook. pyramids provides Dataset (GeoTIFF/NetCDF
reading + plotting); earthlens provides the unified EarthLens entry point and the
Earth Engine Catalog.
import os
from pathlib import Path
import ee
from pyramids.dataset import Dataset
from pyramids.plot import ColorBar
from earthlens.core import EarthLens
from earthlens.gee import Catalog, cancel_task, list_recent_tasks, wait_for_task_id
Output directory¶
Every written GeoTIFF lands under a per-notebook out/ directory (which is
.gitignored — re-running overwrites it).
OUT_DIR = Path('out') / 'precipitation'
OUT_DIR.mkdir(parents=True, exist_ok=True)
print(f'output directory: {OUT_DIR.resolve()}')
Credentials¶
The notebook reads the GEE service-account credentials from the GEE_SERVICE_ACCOUNT
/ GEE_SERVICE_KEY environment variables. Both must be set before running this cell.
SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT']
SERVICE_KEY = os.environ['GEE_SERVICE_KEY']
Inspect the catalog entry¶
Before downloading anything, look at what the bundled catalog knows about the asset — bands, cadence, license, provider.
cat = Catalog()
ds = cat.get_dataset('UCSB-CHG/CHIRPS/DAILY')
print(ds)
# The summary clips long text and shows only a band count, so an explorer
# notebook still wants the untruncated title and the fields it omits:
print(f'title (full): {ds.title}')
print(f'ee_type: {ds.ee_type}')
print(f'default_reducer: {ds.default_reducer}')
print(f'license: {ds.license}')
print(f'band ids (first 5): {list(ds.bands)[:5]}')
Download¶
Tiny AOI ([-2.0, 2.0] lat, [29.0, 33.0] lon) at 5566.0 m, monthly cadence — keeps
the synchronous download under EE's 32768-px per-axis cap. We build the request
first, then authenticate on its own line.
gee = EarthLens(
data_source="gee",
start='2024-06-01',
end='2024-06-30',
dataset='UCSB-CHG/CHIRPS/DAILY',
variables=['precipitation'],
aoi=[29.0, -2.0, 33.0, 2.0],
cadence='monthly',
path=OUT_DIR,
scale=5566.0,
reducer='mean',
)
gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
Run the synchronous download and report the GeoTIFF(s) written to disk.
paths = gee.download(progress_bar=False)
print(f'wrote {len(paths)} GeoTIFF(s):')
for p in paths:
print(f' {p} ({p.stat().st_size / 1024:.1f} KB)')
Quick preview¶
Load the first written GeoTIFF through pyramids and read its single band into an
array. (pyramids.dataset.Dataset is the project's GeoTIFF/NetCDF wrapper.) We also
mask the dataset's nodata so the colormap isn't pinned to it.
preview = Dataset.read_file(paths[0])
Render the band¶
Plot the precipitation band through pyramids and print its value range. The values are well spread across the window (p25 = 0.61, p50 = 1.37, p75 = 2.14 mm/day), so a linear ramp uses the colour range properly and needs no stretch.
glyph = preview.plot(
cmap='viridis',
colorbar=ColorBar(label='precipitation (mm/day)'),
title='CHIRPS daily precipitation — June 2024 mean',
)
glyph.ax.title.set_fontsize(11)
stats = preview.stats(approx_ok=False)
low, high = stats['min'].iloc[0], stats['max'].iloc[0]
print(f'value range: [{low:.4g}, {high:.4g}] mm/day')
Tracking submitted jobs (asynchronous export)¶
The download above uses export_via="url" — a synchronous getDownloadURL round-trip. Nothing was queued, so there's no Earth Engine job to track.
To track an export instead, switch to an asynchronous sink (drive / gcs / asset) and pass wait_for_export=False so .download() returns a TaskInfo at submission time rather than blocking until completion. The cells below submit the same (asset_id, band, AOI, scale) request as an export_via="asset" task into the service account's own asset folder, then walk the four jobs-API calls (list_recent_tasks → wait_for_task_id → ee.data.getAsset → ee.data.deleteAsset) to make the job finish and tidy up. See track-batch-exports.ipynb for a deeper worked example.
The demo asset folder¶
Name a Folder asset we own. GEE._export_via_batch writes the actual image at
<asset_id>/<prefix>, so asset_id here is the parent FOLDER (not the final image
path).
# The asset goes into a `Folder` asset that we own. `GEE._export_via_batch`
# writes the actual image at `<asset_id>/<prefix>`, so `asset_id` here is
# the parent FOLDER (not the final image path). Both must be cleaned up.
_proj = ee.data._get_projects_path().removeprefix('projects/')
PARENT = f'projects/{_proj}/assets'
DEMO_FOLDER = f'{PARENT}/earthlens-demo-precipitation'
print(f'demo folder: {DEMO_FOLDER}')
Prepare a clean folder¶
Listing the parent says whether a previous run left the folder behind, so the cleanup never has to swallow a "not found" from Earth Engine. Then create the folder Earth Engine requires before a child write.
# Listing the parent says whether a previous run left the folder behind, so the
# cleanup below never has to swallow a "not found" from Earth Engine.
listed = ee.data.listAssets({'parent': PARENT})
siblings = [asset['name'] for asset in listed.get('assets', [])]
if DEMO_FOLDER in siblings:
children = ee.data.listAssets({'parent': DEMO_FOLDER})
for child in children.get('assets', []):
ee.data.deleteAsset(child['name'])
print(f'cleared leftover child: {child["name"]}')
ee.data.deleteAsset(DEMO_FOLDER)
print(f'cleared leftover folder: {DEMO_FOLDER}')
# Create the parent folder — EE requires it to exist before a child write.
ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER)
print(f'created folder: {DEMO_FOLDER}')
Submit¶
Same (asset_id, band, AOI, scale) request as the sync download above, just routed
through export_via="asset" + wait_for_export=False. Build the request and
authenticate first.
async_gee = EarthLens(
data_source="gee",
start='2024-06-01',
end='2024-06-30',
dataset='UCSB-CHG/CHIRPS/DAILY',
variables=['precipitation'],
aoi=[29.0, -2.0, 33.0, 2.0],
cadence='monthly',
path=OUT_DIR,
scale=5566.0,
reducer='mean',
export_via='asset',
asset_id=DEMO_FOLDER,
wait_for_export=False,
)
async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
download() returns a TaskInfo per submitted bucket at the moment the task is
queued — no blocking.
submitted = async_gee.download(progress_bar=False)
task_info = submitted[0]
print(f'submitted: id={task_info.id} state={task_info.state}')
print(f' description={task_info.description}')
List submitted tasks¶
list_recent_tasks(description_prefix=...) returns every matching task across the
current project, so we can confirm ours is queued before waiting on it.
recent = list_recent_tasks(
description_prefix=task_info.description,
max_age_min=10,
)
print(f'list_recent_tasks matched {len(recent)} task(s):')
for t in recent:
print(f' {t.id} {t.state:<12} {t.description}')
Wait for completion¶
wait_for_task_id blocks until the task reaches a terminal state. A real workflow
would poll later from a separate process — the wait here exists so the notebook shows
the full success path end-to-end. On FAILED / CANCELLED it raises, and we
cancel-if-still-running so no in-flight task leaks on notebook restart.
final = None
try:
final = wait_for_task_id(
task_info.id,
poll_seconds=10,
progress_bar=False,
)
print(f'final state: {final.state}')
finally:
if final is None:
# The wait raises on FAILED / CANCELLED *and on timeout* — and a
# timeout leaves the export still running. Cancel it so an aborted
# notebook does not leave a live task behind; cancel_task is a no-op
# on an already-terminal task, and the original error still
# propagates out of this finally.
cancel_task(task_info.id)
print(f'cancelled {task_info.id} after the wait failed')
Verify + clean up¶
Confirm the produced asset exists on Earth Engine, then delete it (and the surrounding demo folder) so we don't leak storage between notebook runs. The backend wrote the image at <DEMO_FOLDER>/<task description>.
produced = f'{DEMO_FOLDER}/{task_info.description}'
meta = ee.data.getAsset(produced)
print(f'asset exists: type={meta.get("type")} name={meta.get("name")}')
ee.data.deleteAsset(produced)
print('asset deleted')
# Tear down the parent folder.
ee.data.deleteAsset(DEMO_FOLDER)
print(f'folder deleted: {DEMO_FOLDER}')
What's on disk¶
The written GeoTIFF is left under the per-notebook out/ directory for you to inspect. That directory is
.gitignored — re-running the notebook overwrites it.
for p in sorted(OUT_DIR.iterdir()) if OUT_DIR.exists() else []:
print(f'{p} ({p.stat().st_size / 1024:.1f} KB)')