Hydrology & water — JRC Global Surface Water (v1.4)¶
JRC's Global Surface Water occurrence band (1984-2021) over a small AOI near the Nile delta — a static ee.Image with no temporal cadence.
Setup¶
The imports for the whole notebook in one place: pyramids provides Dataset (reading + plotting) and earthlens provides the unified EarthLens entry point plus the bundled GEE Catalog.
import os
from pathlib import Path
from pyramids.dataset import Dataset
from pyramids.plot import ColorBar
from earthlens.core import EarthLens
from earthlens.gee import Catalog, cancel_task
Credentials and output directory¶
The notebook reads its GEE service-account credentials from the GEE_SERVICE_ACCOUNT / GEE_SERVICE_KEY environment variables — both must be set before running this cell. Each download is written under a per-notebook out/ directory.
OUT_DIR = Path('out') / 'hydrology-water'
OUT_DIR.mkdir(parents=True, exist_ok=True)
SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT']
SERVICE_KEY = os.environ['GEE_SERVICE_KEY']
print(f'output directory: {OUT_DIR.resolve()}')
Inspect the catalog entry¶
Before downloading anything, look at what the bundled catalog knows about the asset — bands, cadence, license, provider.
cat = Catalog()
ds = cat.get_dataset('JRC/GSW1_4/GlobalSurfaceWater')
Print the headline metadata for the asset so the request below is easy to reason about.
print(ds)
# The summary clips long text and shows only a band count, so an explorer
# notebook still wants the untruncated title and the fields it omits:
print(f'title (full): {ds.title}')
print(f'ee_type: {ds.ee_type}')
print(f'default_reducer: {ds.default_reducer}')
print(f'license: {ds.license}')
print(f'band ids (first 5): {list(ds.bands)[:5]}')
Download¶
Tiny AOI ([29.9, 30.1] lat, [31.1, 31.3] lon) at 90.0 m, raw cadence — keeps the synchronous download under EE's 32768-px per-axis cap. The construct → authenticate → download steps are kept on separate lines.
gee = EarthLens(
data_source="gee",
start='1984-03-16',
end='1984-03-17',
dataset='JRC/GSW1_4/GlobalSurfaceWater',
variables=['occurrence'],
aoi=[31.1, 29.9, 31.3, 30.1],
cadence='raw',
path=OUT_DIR,
scale=90.0,
reducer='mean',
)
gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
download() writes the GeoTIFF(s) to disk and returns the written paths.
paths = gee.download(progress_bar=False)
print(f'wrote {len(paths)} GeoTIFF(s):')
for p in paths:
print(f' {p} ({p.stat().st_size / 1024:.1f} KB)')
Quick preview¶
Load the first written GeoTIFF through pyramids. (pyramids.dataset.Dataset is the project's
GeoTIFF/NetCDF wrapper.)
preview = Dataset.read_file(paths[0], read_only=False)
print(f'dtype : {preview.dtype}')
print(f'declared : {preview.no_data_value}')
The band arrives declaring no nodata at all: the EEDAI reader fills the pixels Earth Engine had masked — here
everything that is not permanent water — but never stamps the fill into the band. occurrence is a percentage
in 0 … 100, so the int8 fill (-128) would otherwise be read as a real occurrence and pin the bottom of
the colour ramp. Declare it, and both the map and the statistics drop it.
Render the occurrence band and report its value range.
preview.no_data_value = [-128]
glyph = preview.plot(
cmap='Blues',
colorbar=ColorBar(label='surface-water occurrence (%)'),
title='Surface-water occurrence — the Nile south of Cairo',
)
glyph.ax.title.set_fontsize(11)
stats = preview.stats(approx_ok=False)
low, high = stats['min'].iloc[0], stats['max'].iloc[0]
print(f'value range: [{low:.4g}, {high:.4g}] %')
# Occurrence is a percentage. A declaration that failed to match the band is a
# silent no-op, so pin the range the masking must produce rather than trusting it.
assert 0 <= low <= high <= 100, (
f'occurrence should be a 0-100 percentage after masking; got [{low}, {high}]'
)
water = preview.count_domain_cells()
total_cells = preview.rows * preview.columns
print(f'water cells: {water:,} of {total_cells:,}')
# The handle was opened writable to declare the nodata; release it so the
# GDAL lock does not outlive the cell.
preview.close()
Tracking submitted jobs (asynchronous export)¶
The download above uses export_via="url" — a synchronous getDownloadURL round-trip. Nothing was queued, so there's no Earth Engine job to track.
To track an export instead, switch to an asynchronous sink (drive / gcs / asset) and pass wait_for_export=False so .download() returns a TaskInfo at submission time rather than blocking until completion. The cells below submit the same (asset_id, band, AOI, scale) request as an export_via="asset" task into the service account's own asset folder, then walk the four jobs-API calls (list_recent_tasks → wait_for_task_id → ee.data.getAsset → ee.data.deleteAsset) to make the job finish and tidy up. See track-batch-exports.ipynb for a deeper worked example.
Prepare the demo asset folder¶
The export writes into a Folder asset that we own; GEE._export_via_batch writes the image at <asset_id>/<prefix>, so asset_id here is the parent folder. Clear any leftover children from a previous run, then create the folder fresh.
import ee
from earthlens.gee import list_recent_tasks, wait_for_task_id
_proj = ee.data._get_projects_path().removeprefix('projects/')
PARENT = f'projects/{_proj}/assets'
DEMO_FOLDER = f'{PARENT}/earthlens-demo-hydrology-water'
print(f'demo folder: {DEMO_FOLDER}')
Listing the parent says whether a previous run left the folder behind, so the cleanup never has to swallow a "not found" from Earth Engine.
listed = ee.data.listAssets({'parent': PARENT})
siblings = [asset['name'] for asset in listed.get('assets', [])]
if DEMO_FOLDER in siblings:
children = ee.data.listAssets({'parent': DEMO_FOLDER})
for child in children.get('assets', []):
ee.data.deleteAsset(child['name'])
print(f'cleared leftover child: {child["name"]}')
ee.data.deleteAsset(DEMO_FOLDER)
print(f'cleared leftover folder: {DEMO_FOLDER}')
Create the parent folder — EE requires it to exist before a child write.
ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER)
print(f'created folder: {DEMO_FOLDER}')
Submit¶
Same (asset_id, band, AOI, scale) request as the sync download above, just routed through export_via="asset" + wait_for_export=False. The construct → authenticate steps are split onto their own lines.
async_gee = EarthLens(
data_source="gee",
start='1984-03-16',
end='1984-03-17',
dataset='JRC/GSW1_4/GlobalSurfaceWater',
variables=['occurrence'],
aoi=[31.1, 29.9, 31.3, 30.1],
cadence='raw',
path=OUT_DIR,
scale=90.0,
reducer='mean',
export_via='asset',
asset_id=DEMO_FOLDER,
wait_for_export=False,
)
async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
download() returns a TaskInfo per submitted bucket at the moment the task is queued — no blocking.
submitted = async_gee.download(progress_bar=False)
task_info = submitted[0]
print(f'submitted: id={task_info.id} state={task_info.state}')
print(f' description={task_info.description}')
List + wait¶
list_recent_tasks(description_prefix=...) returns every matching task across the current project. A real workflow would just poll later from a separate process — the wait here exists so the notebook shows the full success path end-to-end.
recent = list_recent_tasks(
description_prefix=task_info.description,
max_age_min=10,
)
print(f'list_recent_tasks matched {len(recent)} task(s):')
for t in recent:
print(f' {t.id} {t.state:<12} {t.description}')
wait_for_task_id blocks until the task reaches a terminal state. On a FAILED / CANCELLED result it raises RuntimeError; we cancel-if-still-running so we don't leak an in-flight task on notebook restart.
final = None
try:
final = wait_for_task_id(
task_info.id,
poll_seconds=10,
progress_bar=False,
)
print(f'final state: {final.state}')
finally:
if final is None:
# The wait raises on FAILED / CANCELLED *and on timeout* — and a
# timeout leaves the export still running. Cancel it so an aborted
# notebook does not leave a live task behind; cancel_task is a no-op
# on an already-terminal task, and the original error still
# propagates out of this finally.
cancel_task(task_info.id)
print(f'cancelled {task_info.id} after the wait failed')
Verify + clean up¶
Confirm the produced asset exists on Earth Engine, then delete it (and the surrounding demo folder) so we don't leak storage between notebook runs. The backend wrote the image at <DEMO_FOLDER>/<task description>.
produced = f'{DEMO_FOLDER}/{task_info.description}'
meta = ee.data.getAsset(produced)
print(f'asset exists: type={meta.get("type")} name={meta.get("name")}')
ee.data.deleteAsset(produced)
print('asset deleted')
Always try to tear down the parent folder, even if the verify/delete step above was skipped.
ee.data.deleteAsset(DEMO_FOLDER)
print(f'folder deleted: {DEMO_FOLDER}')
What's on disk¶
The GeoTIFF is left under the per-notebook out/ directory for you to inspect. That directory is .gitignored — re-running the notebook overwrites it.
for p in sorted(OUT_DIR.iterdir()) if OUT_DIR.exists() else []:
print(f'{p} ({p.stat().st_size / 1024:.1f} KB)')