Tracking asynchronous Earth Engine exports¶

Demonstrates the earthlens.gee.jobs surface — the same job-tracking shape earthlens.ecmwf.jobs exposes for CDS, applied to GEE batch tasks.

Two tiny Drive exports are submitted with wait_for_export=False (so download() does NOT block), then:

1. list_recent_tasks enumerates them.
2. get_task_status fetches one by id.
3. cancel_task cancels both so they don't actually run.
4. wait_for_task_id polls one of them to its terminal CANCELLED state.

No Drive permission is needed — the tasks are cancelled before EE attempts to write.

Setup¶

Reads GEE_SERVICE_ACCOUNT / GEE_SERVICE_KEY from the environment.

In [1]:

import os
import time
from pathlib import Path

from earthlens.gee import (
    GEE,
    cancel_task,
    get_task_status,
    list_recent_tasks,
    wait_for_task_id,
)

OUT_DIR = Path('out') / 'track-batch-exports'
OUT_DIR.mkdir(parents=True, exist_ok=True)
SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT']
SERVICE_KEY = os.environ['GEE_SERVICE_KEY']
print(f'output directory: {OUT_DIR.resolve()}')

import os import time from pathlib import Path from earthlens.gee import ( GEE, cancel_task, get_task_status, list_recent_tasks, wait_for_task_id, ) OUT_DIR = Path('out') / 'track-batch-exports' OUT_DIR.mkdir(parents=True, exist_ok=True) SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT'] SERVICE_KEY = os.environ['GEE_SERVICE_KEY'] print(f'output directory: {OUT_DIR.resolve()}')

2026-05-18 00:20:44 | INFO | pyramids.base.config | Logging is configured.

output directory: C:\gdrive\algorithms\remote-sensing\earthlens\docs\examples\gee\out\track-batch-exports

Submit two tiny Drive exports (non-blocking)¶

wait_for_export=False returns a TaskInfo per submitted export at the moment the task is queued — no waiting for completion. We pick two short date windows so EE creates two separate buckets.

In [2]:

gee = GEE(
    start='2024-06-01',
    end='2024-06-02',
    variables={'UCSB-CHG/CHIRPS/DAILY': ['precipitation']},
    temporal_resolution='daily',
    lat_lim=[29.9, 30.1],
    lon_lim=[31.1, 31.3],
    path=str(OUT_DIR),
    service_account=SERVICE_ACCOUNT,
    service_key=SERVICE_KEY,
    scale=5566.0,
    export_via='drive',
    drive_folder='earthlens_track_demo',
    wait_for_export=False,  # <- the key knob
)
submitted = gee.download(progress_bar=False)
print(f'submitted {len(submitted)} task(s):')
for info in submitted:
    print(f'  {info.id:26} {info.state:14} {info.description}')

gee = GEE( start='2024-06-01', end='2024-06-02', variables={'UCSB-CHG/CHIRPS/DAILY': ['precipitation']}, temporal_resolution='daily', lat_lim=[29.9, 30.1], lon_lim=[31.1, 31.3], path=str(OUT_DIR), service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY, scale=5566.0, export_via='drive', drive_folder='earthlens_track_demo', wait_for_export=False, # <- the key knob ) submitted = gee.download(progress_bar=False) print(f'submitted {len(submitted)} task(s):') for info in submitted: print(f' {info.id:26} {info.state:14} {info.description}')

2026-05-18 00:20:54.622 | INFO     | earthlens.gee.backend:_export_via_batch:833 - Submitted drive export ZXRWVQPYU7XODMSD56MDZMYZ (UCSB-CHG_CHIRPS_DAILY_precipitation_20240601); track via earthlens.gee.jobs.

2026-05-18 00:20:55.846 | INFO     | earthlens.gee.backend:_export_via_batch:833 - Submitted drive export WSWS3M3JL6BP5B4KGDLLOFJO (UCSB-CHG_CHIRPS_DAILY_precipitation_20240602); track via earthlens.gee.jobs.

submitted 2 task(s):
  ZXRWVQPYU7XODMSD56MDZMYZ   READY          UCSB-CHG_CHIRPS_DAILY_precipitation_20240601
  WSWS3M3JL6BP5B4KGDLLOFJO   READY          UCSB-CHG_CHIRPS_DAILY_precipitation_20240602

List recent tasks on the project¶

list_recent_tasks wraps ee.data.listOperations() with client-side filtering by state / age / task type / description prefix. Filtering by our description_prefix= matches what GEE._api emits as the task description: <asset-slug>_<bands>_<YYYYMMDD>.

In [3]:

ours = list_recent_tasks(
    description_prefix='UCSB-CHG_CHIRPS_DAILY_precipitation',
    max_age_min=10,
)
print(f'{len(ours)} matching task(s):')
for t in ours:
    print(f'  {t.id:26} {t.state:14} {t.task_type:14} {t.description}')

ours = list_recent_tasks( description_prefix='UCSB-CHG_CHIRPS_DAILY_precipitation', max_age_min=10, ) print(f'{len(ours)} matching task(s):') for t in ours: print(f' {t.id:26} {t.state:14} {t.task_type:14} {t.description}')

3 matching task(s):
  WSWS3M3JL6BP5B4KGDLLOFJO   READY          EXPORT_IMAGE   UCSB-CHG_CHIRPS_DAILY_precipitation_20240602
  ZXRWVQPYU7XODMSD56MDZMYZ   READY          EXPORT_IMAGE   UCSB-CHG_CHIRPS_DAILY_precipitation_20240601
  CWC2CUOT6MNDJX5XSRTBJMNJ   COMPLETED      EXPORT_IMAGE   UCSB-CHG_CHIRPS_DAILY_precipitation_20240601

Inspect one task in detail¶

get_task_status(id) issues a single ee.data.getOperation call and returns the same normalised TaskInfo.

In [4]:

first = submitted[0]
detail = get_task_status(first.id)
print(detail.model_dump_json(indent=2))

first = submitted[0] detail = get_task_status(first.id) print(detail.model_dump_json(indent=2))

{
  "id": "ZXRWVQPYU7XODMSD56MDZMYZ",
  "operation_name": "projects/earth-engine-415620/operations/ZXRWVQPYU7XODMSD56MDZMYZ",
  "description": "UCSB-CHG_CHIRPS_DAILY_precipitation_20240601",
  "state": "READY",
  "task_type": "EXPORT_IMAGE",
  "create_time": "2026-05-17T22:20:53.834628",
  "update_time": "2026-05-17T22:20:53.834628",
  "start_time": "1970-01-01T00:00:00",
  "attempt": 1,
  "priority": 100,
  "destination_uris": [],
  "error_message": null,
  "done": false
}

Cancel both tasks¶

cancel_task(id) wraps the modern ee.data.cancelOperation (replacing the deprecated cancelTask). Cancellation is asynchronous — EE acknowledges the request immediately and the state transitions to CANCEL_REQUESTED then CANCELLED shortly after.

In [5]:

for info in submitted:
    cancel_task(info.id)
    print(f'cancel requested for {info.id}')

for info in submitted: cancel_task(info.id) print(f'cancel requested for {info.id}')

cancel requested for ZXRWVQPYU7XODMSD56MDZMYZ

cancel requested for WSWS3M3JL6BP5B4KGDLLOFJO

Wait for one task to reach a terminal state¶

wait_for_task_id(id) polls until the task reaches COMPLETED / FAILED / CANCELLED / CANCEL_REQUESTED. Since we cancelled the tasks above, the call below should return quickly with a RuntimeError reporting the cancelled state.

(The notebook catches that error so the cell still produces clean output — in a real script you'd let it propagate.)

In [6]:

try:
    final = wait_for_task_id(submitted[0].id, poll_seconds=3, progress_bar=False)
    print(f'finished: {final.state}')
except RuntimeError as exc:
    print(f'expected (we cancelled it): {exc}')

try: final = wait_for_task_id(submitted[0].id, poll_seconds=3, progress_bar=False) print(f'finished: {final.state}') except RuntimeError as exc: print(f'expected (we cancelled it): {exc}')

expected (we cancelled it): Earth Engine task ZXRWVQPYU7XODMSD56MDZMYZ ended CANCELLED: Cancelled.

What's next¶

• resolve_destination(info) returns the Drive / GCS / asset destination URIs of a completed task — for our cancelled ones, the destinations are empty.
• Catalog.list_recent_tasks / Catalog.get_task_status are the same calls bound on the catalog object, mirroring the ECMWF Catalog.list_recent_jobs ergonomic.
• The CLI python -m earthlens.gee.jobs {list,status,cancel,wait} exposes the same surface for shell scripts.