Community — GLOBathy global lake bathymetry¶

GLOBathy is a global lake-bathymetry grid derived from HydroLAKES, published on Earth Engine as one of the user-contributed projects/... assets the bundled catalog carries. This notebook takes a window over Lake Nasser — the Aswan High Dam reservoir on the Egyptian/Sudanese Nile — and walks the whole path from catalog lookup to a written GeoTIFF.

What this notebook does¶

1. Catalog — inspect what the bundled GEE catalog knows about the GLOBathy asset (bands, resolution, license, provider).
2. Download — a synchronous getDownloadURL fetch of the depth band over a lake-sized AOI.
3. Preview — render the written GeoTIFF through pyramids.
4. Async export — submit the same request as an export_via="asset" batch task and walk the jobs API (list → wait → verify → clean up).

Needs GEE_SERVICE_ACCOUNT / GEE_SERVICE_KEY for the live Earth Engine cells.

Setup¶

First the imports. pyramids provides Dataset (reading + plotting); earthlens provides the unified EarthLens entry point plus the GEE Catalog and batch-task helpers.

In [ ]:

import os
from pathlib import Path

from pyramids.dataset import Dataset
from pyramids.plot import ColorBar, ColorScaling

from earthlens.core import EarthLens
from earthlens.gee import Catalog, cancel_task

import os from pathlib import Path from pyramids.dataset import Dataset from pyramids.plot import ColorBar, ColorScaling from earthlens.core import EarthLens from earthlens.gee import Catalog, cancel_task

Output directory and credentials¶

Written GeoTIFFs go under a per-notebook out/ directory. The service-account credentials are read from the GEE_SERVICE_ACCOUNT / GEE_SERVICE_KEY environment variables — both must be set before running this cell.

In [ ]:

OUT_DIR = Path('out') / 'community'
OUT_DIR.mkdir(parents=True, exist_ok=True)

SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT']
SERVICE_KEY = os.environ['GEE_SERVICE_KEY']
print(f'output directory: {OUT_DIR.resolve()}')

OUT_DIR = Path('out') / 'community' OUT_DIR.mkdir(parents=True, exist_ok=True) SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT'] SERVICE_KEY = os.environ['GEE_SERVICE_KEY'] print(f'output directory: {OUT_DIR.resolve()}')

Inspect the catalog entry¶

Before downloading anything, look at what the bundled catalog knows about the asset — bands, resolution, license, provider. The asset id and band are bound to names here so the download, the preview title and the batch export all refer to the same request.

In [ ]:

ASSET_ID = 'projects/sat-io/open-datasets/GLOBathy/GLOBathy_bathymetry'
BAND = 'b1'

cat = Catalog()
ds = cat.get_dataset(ASSET_ID)
print(ds)

# The summary clips long text and shows only a band count, so an explorer
# notebook still wants the untruncated title and the fields it omits:
print(f'title (full):        {ds.title}')
print(f'ee_type:             {ds.ee_type}')
print(f'default_reducer:     {ds.default_reducer}')
print(f'license:             {ds.license}')
print(f'provider:            {ds.provider}')
print(f'band ids:            {list(ds.bands)}')

band = ds.bands[BAND]
print(f'{BAND} description:      {band.description} [{band.units}]')

ASSET_ID = 'projects/sat-io/open-datasets/GLOBathy/GLOBathy_bathymetry' BAND = 'b1' cat = Catalog() ds = cat.get_dataset(ASSET_ID) print(ds) # The summary clips long text and shows only a band count, so an explorer # notebook still wants the untruncated title and the fields it omits: print(f'title (full): {ds.title}') print(f'ee_type: {ds.ee_type}') print(f'default_reducer: {ds.default_reducer}') print(f'license: {ds.license}') print(f'provider: {ds.provider}') print(f'band ids: {list(ds.bands)}') band = ds.bands[BAND] print(f'{BAND} description: {band.description} [{band.units}]')

Download¶

A window over Lake Nasser ([32.0, 22.5, 33.2, 23.7]) at 120 m — coarser than the 30 m native grid, which keeps the synchronous getDownloadURL payload small while still resolving the flooded Nile channel. GLOBathy is a static ee_type="image", so cadence='raw' reads it straight through and the date window only records the product epoch. We build the request, authenticate, and download as three separate steps.

In [ ]:

gee = EarthLens(
    data_source="gee",
    start='2022-01-01',
    end='2022-12-31',
    dataset=ASSET_ID,
    variables=[BAND],
    aoi=[32.0, 22.5, 33.2, 23.7],
    cadence='raw',
    path=OUT_DIR,
    scale=120.0,
    reducer='mosaic',
)
gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
paths = gee.download(progress_bar=False)
print(f'wrote {len(paths)} GeoTIFF(s):')
for p in paths:
    print(f'  {p}  ({p.stat().st_size / 1024:.1f} KB)')

gee = EarthLens( data_source="gee", start='2022-01-01', end='2022-12-31', dataset=ASSET_ID, variables=[BAND], aoi=[32.0, 22.5, 33.2, 23.7], cadence='raw', path=OUT_DIR, scale=120.0, reducer='mosaic', ) gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY) paths = gee.download(progress_bar=False) print(f'wrote {len(paths)} GeoTIFF(s):') for p in paths: print(f' {p} ({p.stat().st_size / 1024:.1f} KB)')

Quick preview¶

Load the written GeoTIFF through pyramids and render the depth band. (pyramids.dataset.Dataset is the project's GeoTIFF/NetCDF wrapper.) Only lake pixels carry a depth — everything outside a HydroLAKES polygon is fill.

Two things shape how this renders. The EEDAI reader fills those pixels with pyramids' own default_no_data_value (-9999) but does not stamp the value into the band, so the raster arrives declaring no nodata at all; declaring it is what keeps the fill out of the colour ramp and the statistics. And depth is skewed — 41% of the lake is shallower than 10 m while the drowned Nile channel reaches 130 m — so a linear ramp would bury the shelves. Class breaks give each depth band its own block, the way a bathymetric chart does.

In [ ]:

# Depth classes rather than a linear ramp, so the shallow shelves stay legible
# next to the 130 m channel.
DEPTH_BREAKS = [0, 2, 5, 10, 20, 40, 60, 90, 130]

preview = Dataset.read_file(paths[0], read_only=False)
preview.no_data_value = [Dataset.default_no_data_value]
print(f'declared nodata: {preview.no_data_value}')

glyph = preview.plot(
    cmap='Blues',
    color=ColorScaling.boundary(bounds=DEPTH_BREAKS),
    colorbar=ColorBar(label='depth (m)'),
    title='GLOBathy — Lake Nasser bathymetry',
)
glyph.ax.title.set_fontsize(11)
# Nodata is everything outside a lake polygon. A sand background reads as the
# surrounding desert instead of blending into the shallowest depth class.
glyph.ax.set_facecolor('#efe6d5')

stats = preview.stats(approx_ok=False)
low, high = stats['min'].iloc[0], stats['max'].iloc[0]
print(f'depth range: [{low:.4g}, {high:.4g}] m')
# The fill is -9999; a lake depth is positive. A minimum at or below zero would
# mean the declaration did not match the band and masked nothing.
assert low > 0, f'depth should be strictly positive after masking; got {low}'

lake = preview.count_domain_cells()
total_cells = preview.rows * preview.columns
print(f'lake cells:  {lake:,} of {total_cells:,}')
preview.close()

# Depth classes rather than a linear ramp, so the shallow shelves stay legible # next to the 130 m channel. DEPTH_BREAKS = [0, 2, 5, 10, 20, 40, 60, 90, 130] preview = Dataset.read_file(paths[0], read_only=False) preview.no_data_value = [Dataset.default_no_data_value] print(f'declared nodata: {preview.no_data_value}') glyph = preview.plot( cmap='Blues', color=ColorScaling.boundary(bounds=DEPTH_BREAKS), colorbar=ColorBar(label='depth (m)'), title='GLOBathy — Lake Nasser bathymetry', ) glyph.ax.title.set_fontsize(11) # Nodata is everything outside a lake polygon. A sand background reads as the # surrounding desert instead of blending into the shallowest depth class. glyph.ax.set_facecolor('#efe6d5') stats = preview.stats(approx_ok=False) low, high = stats['min'].iloc[0], stats['max'].iloc[0] print(f'depth range: [{low:.4g}, {high:.4g}] m') # The fill is -9999; a lake depth is positive. A minimum at or below zero would # mean the declaration did not match the band and masked nothing. assert low > 0, f'depth should be strictly positive after masking; got {low}' lake = preview.count_domain_cells() total_cells = preview.rows * preview.columns print(f'lake cells: {lake:,} of {total_cells:,}') preview.close()

Tracking submitted jobs (asynchronous export)¶

The download above uses export_via="url" — a synchronous getDownloadURL round-trip. Nothing was queued, so there's no Earth Engine job to track.

To track an export instead, switch to an asynchronous sink (drive / gcs / asset) and pass wait_for_export=False so .download() returns a TaskInfo at submission time rather than blocking until completion. The cells below submit the same (asset_id, band, AOI, scale) request as an export_via="asset" task into the service account's own asset folder, then walk the four jobs-API calls (list_recent_tasks → wait_for_task_id → ee.data.getAsset → ee.data.deleteAsset) to make the job finish and tidy up. See track-batch-exports.ipynb for a deeper worked example.

Prepare the demo asset folder¶

The batch export writes the image at <asset_id>/<prefix>, so asset_id is the parent folder. Earth Engine needs that folder to exist and to be empty before a child write, and createAsset fails if it is already there — so we list the parent to see which of those applies, and clear any leftovers from a previous run.

In [ ]:

import ee

from earthlens.gee import list_recent_tasks, wait_for_task_id

# The asset goes into a `Folder` asset that we own. `GEE._export_via_batch`
# writes the actual image at `<asset_id>/<prefix>`, so `asset_id` here is
# the parent FOLDER (not the final image path). Both must be cleaned up.
_proj = ee.data._get_projects_path().removeprefix('projects/')
PARENT = f'projects/{_proj}/assets'
DEMO_FOLDER = f'{PARENT}/earthlens-demo-community'
print(f'demo folder: {DEMO_FOLDER}')

# Listing the parent says whether a previous run left the folder behind, so the
# cleanup below never has to swallow a "not found" from Earth Engine.
listed = ee.data.listAssets({'parent': PARENT})
siblings = [asset['name'] for asset in listed.get('assets', [])]
if DEMO_FOLDER in siblings:
    children = ee.data.listAssets({'parent': DEMO_FOLDER})
    for child in children.get('assets', []):
        ee.data.deleteAsset(child['name'])
        print(f'cleared leftover child: {child["name"]}')
    ee.data.deleteAsset(DEMO_FOLDER)
    print(f'cleared leftover folder: {DEMO_FOLDER}')

ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER)
print(f'created folder: {DEMO_FOLDER}')

import ee from earthlens.gee import list_recent_tasks, wait_for_task_id # The asset goes into a `Folder` asset that we own. `GEE._export_via_batch` # writes the actual image at `<asset_id>/<prefix>`, so `asset_id` here is # the parent FOLDER (not the final image path). Both must be cleaned up. _proj = ee.data._get_projects_path().removeprefix('projects/') PARENT = f'projects/{_proj}/assets' DEMO_FOLDER = f'{PARENT}/earthlens-demo-community' print(f'demo folder: {DEMO_FOLDER}') # Listing the parent says whether a previous run left the folder behind, so the # cleanup below never has to swallow a "not found" from Earth Engine. listed = ee.data.listAssets({'parent': PARENT}) siblings = [asset['name'] for asset in listed.get('assets', [])] if DEMO_FOLDER in siblings: children = ee.data.listAssets({'parent': DEMO_FOLDER}) for child in children.get('assets', []): ee.data.deleteAsset(child['name']) print(f'cleared leftover child: {child["name"]}') ee.data.deleteAsset(DEMO_FOLDER) print(f'cleared leftover folder: {DEMO_FOLDER}') ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER) print(f'created folder: {DEMO_FOLDER}')

Submit¶

Same (asset_id, band, AOI, scale) request as the sync download above, just routed through export_via="asset" + wait_for_export=False. download() returns a TaskInfo per submitted bucket at the moment the task is queued — no blocking.

In [ ]:

async_gee = EarthLens(
    data_source="gee",
    start='2022-01-01',
    end='2022-12-31',
    dataset=ASSET_ID,
    variables=[BAND],
    aoi=[32.0, 22.5, 33.2, 23.7],
    cadence='raw',
    path=OUT_DIR,
    scale=120.0,
    reducer='mosaic',
    export_via='asset',
    asset_id=DEMO_FOLDER,
    wait_for_export=False,
)
async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
submitted = async_gee.download(progress_bar=False)
task_info = submitted[0]
print(f'submitted: id={task_info.id} state={task_info.state}')
print(f'           description={task_info.description}')

async_gee = EarthLens( data_source="gee", start='2022-01-01', end='2022-12-31', dataset=ASSET_ID, variables=[BAND], aoi=[32.0, 22.5, 33.2, 23.7], cadence='raw', path=OUT_DIR, scale=120.0, reducer='mosaic', export_via='asset', asset_id=DEMO_FOLDER, wait_for_export=False, ) async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY) submitted = async_gee.download(progress_bar=False) task_info = submitted[0] print(f'submitted: id={task_info.id} state={task_info.state}') print(f' description={task_info.description}')

List + wait¶

list_recent_tasks(description_prefix=...) returns every matching task across the current project; wait_for_task_id blocks until the one we care about reaches a terminal state. A real workflow would just poll later from a separate process — the wait here exists so the notebook shows the full success path end-to-end.

In [ ]:

recent = list_recent_tasks(
    description_prefix=task_info.description,
    max_age_min=10,
)
print(f'list_recent_tasks matched {len(recent)} task(s):')
for t in recent:
    print(f'  {t.id}  {t.state:<12} {t.description}')

final = None
try:
    final = wait_for_task_id(
        task_info.id,
        poll_seconds=10,
        progress_bar=False,
    )
    print(f'final state: {final.state}')
finally:
    if final is None:
        # The wait raises on FAILED / CANCELLED *and on timeout* — and a
        # timeout leaves the export still running. Cancel it so an aborted
        # notebook does not leave a live task behind; cancel_task is a no-op
        # on an already-terminal task, and the original error still
        # propagates out of this finally.
        cancel_task(task_info.id)
        print(f'cancelled {task_info.id} after the wait failed')

recent = list_recent_tasks( description_prefix=task_info.description, max_age_min=10, ) print(f'list_recent_tasks matched {len(recent)} task(s):') for t in recent: print(f' {t.id} {t.state:<12} {t.description}') final = None try: final = wait_for_task_id( task_info.id, poll_seconds=10, progress_bar=False, ) print(f'final state: {final.state}') finally: if final is None: # The wait raises on FAILED / CANCELLED *and on timeout* — and a # timeout leaves the export still running. Cancel it so an aborted # notebook does not leave a live task behind; cancel_task is a no-op # on an already-terminal task, and the original error still # propagates out of this finally. cancel_task(task_info.id) print(f'cancelled {task_info.id} after the wait failed')

Verify + clean up¶

Confirm the produced asset exists on Earth Engine, then delete it (and the surrounding demo folder) so we don't leak storage between notebook runs. The backend wrote the image at <DEMO_FOLDER>/<task description>.

In [ ]:

produced = f'{DEMO_FOLDER}/{task_info.description}'
meta = ee.data.getAsset(produced)
print(f'asset exists: type={meta.get("type")} name={meta.get("name")}')

ee.data.deleteAsset(produced)
print('asset deleted')
ee.data.deleteAsset(DEMO_FOLDER)
print(f'folder deleted: {DEMO_FOLDER}')

produced = f'{DEMO_FOLDER}/{task_info.description}' meta = ee.data.getAsset(produced) print(f'asset exists: type={meta.get("type")} name={meta.get("name")}') ee.data.deleteAsset(produced) print('asset deleted') ee.data.deleteAsset(DEMO_FOLDER) print(f'folder deleted: {DEMO_FOLDER}')

What's on disk¶

The written GeoTIFF is left under the per-notebook out/ directory for you to inspect. That directory is .gitignored — re-running the notebook overwrites it.

In [ ]:

for p in sorted(OUT_DIR.iterdir()):
    print(f'{p}  ({p.stat().st_size / 1024:.1f} KB)')

for p in sorted(OUT_DIR.iterdir()): print(f'{p} ({p.stat().st_size / 1024:.1f} KB)')