SAR / radar — Sentinel-1 GRD VV backscatter¶

Sentinel-1 Ground Range Detected, VV-polarisation, monthly mean over Giza. Sentinel-1 carries VV/VH over land and HH/HV over polar regions; VV is the universal pick.

This notebook downloads a single monthly composite, previews it, then demonstrates the asynchronous (batch) export path with full job tracking and cleanup.

Setup¶

First the imports and the output directory. pyramids provides Dataset (GeoTIFF/NetCDF reading); earthlens provides the unified EarthLens entry point and the Earth Engine Catalog.

In [ ]:

import os
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
from pyramids.dataset import Dataset as PyramidsDataset

from earthlens import EarthLens
from earthlens.gee import Catalog

OUT_DIR = Path('out') / 'sar-radar'
OUT_DIR.mkdir(parents=True, exist_ok=True)
print(f'output directory: {OUT_DIR.resolve()}')

import os from pathlib import Path import matplotlib.pyplot as plt import numpy as np from pyramids.dataset import Dataset as PyramidsDataset from earthlens import EarthLens from earthlens.gee import Catalog OUT_DIR = Path('out') / 'sar-radar' OUT_DIR.mkdir(parents=True, exist_ok=True) print(f'output directory: {OUT_DIR.resolve()}')

Credentials¶

The notebook reads the GEE service-account credentials from the GEE_SERVICE_ACCOUNT / GEE_SERVICE_KEY environment variables. Both must be set before running this cell.

In [ ]:

SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT']
SERVICE_KEY = os.environ['GEE_SERVICE_KEY']

SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT'] SERVICE_KEY = os.environ['GEE_SERVICE_KEY']

Inspect the catalog entry¶

Before downloading anything, look at what the bundled catalog knows about the asset — bands, cadence, license, provider.

In [ ]:

cat = Catalog()
ds = cat.get_dataset('COPERNICUS/S1_GRD')
print(f'title:               {ds.title}')
print(f'ee_type:             {ds.ee_type}')
print(f'spatial_resolution:  {ds.spatial_resolution} m')
print(f'extent.start_date:   {ds.extent.start_date}')
print(f'extent.end_date:     {ds.extent.end_date}')
print(f'default_reducer:     {ds.default_reducer}')
print(f'license:             {ds.license}')
print(f'provider:            {ds.provider}')
print(f'#bands:              {len(ds.bands)}')
print(f'band ids (first 5):  {list(ds.bands)[:5]}')

cat = Catalog() ds = cat.get_dataset('COPERNICUS/S1_GRD') print(f'title: {ds.title}') print(f'ee_type: {ds.ee_type}') print(f'spatial_resolution: {ds.spatial_resolution} m') print(f'extent.start_date: {ds.extent.start_date}') print(f'extent.end_date: {ds.extent.end_date}') print(f'default_reducer: {ds.default_reducer}') print(f'license: {ds.license}') print(f'provider: {ds.provider}') print(f'#bands: {len(ds.bands)}') print(f'band ids (first 5): {list(ds.bands)[:5]}')

Download¶

Build the request first: a tiny AOI ([29.95, 30.05] lat, [31.15, 31.25] lon) at 30.0 m, monthly cadence — small enough to keep the synchronous download under EE's 32768-px per-axis cap.

In [ ]:

gee = EarthLens(
    data_source="gee",
    start='2024-06-01',
    end='2024-06-30',
    dataset='COPERNICUS/S1_GRD',
    variables=['VV'],
    aoi=[31.15, 29.95, 31.25, 30.05],
    cadence='monthly',
    path=str(OUT_DIR),
    scale=30.0,
    reducer='mean',
)

gee = EarthLens( data_source="gee", start='2024-06-01', end='2024-06-30', dataset='COPERNICUS/S1_GRD', variables=['VV'], aoi=[31.15, 29.95, 31.25, 30.05], cadence='monthly', path=str(OUT_DIR), scale=30.0, reducer='mean', )

Authentication and download are kept as separate steps: authenticate() resolves the service-account credentials, then download() writes the composite to disk and returns the written paths.

In [ ]:

gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
paths = gee.download(progress_bar=False)
print(f'wrote {len(paths)} GeoTIFF(s):')
for p in paths:
    print(f'  {p}  ({p.stat().st_size / 1024:.1f} KB)')

gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY) paths = gee.download(progress_bar=False) print(f'wrote {len(paths)} GeoTIFF(s):') for p in paths: print(f' {p} ({p.stat().st_size / 1024:.1f} KB)')

Quick preview¶

Load the first written GeoTIFF through pyramids and pull out the single band as an array. (pyramids.dataset.Dataset is the project's GeoTIFF/NetCDF wrapper.)

In [ ]:

pds = PyramidsDataset.read_file(str(paths[0]))
arr = pds.read_array()
if arr.ndim == 3:
    arr = arr[0]

pds = PyramidsDataset.read_file(str(paths[0])) arr = pds.read_array() if arr.ndim == 3: arr = arr[0]

Mask the dataset's nodata value so the colormap doesn't get pinned to it, then render the band with a viridis colormap and report the value range.

In [ ]:

nodata = pds.no_data_value
if nodata is not None:
    try:
        arr = np.ma.masked_equal(
            arr, float(nodata[0] if isinstance(nodata, (list, tuple)) else nodata)
        )
    except (TypeError, ValueError):
        pass

fig, ax = plt.subplots(figsize=(6, 5))
im = ax.imshow(arr, cmap='viridis')
ax.set_title(f'{'COPERNICUS/S1_GRD'} / {'VV'}')
ax.set_xlabel('x (px)')
ax.set_ylabel('y (px)')
fig.colorbar(im, ax=ax, shrink=0.8)
plt.tight_layout()
plt.show()
print(f'value range: [{float(np.nanmin(arr)):.4g}, {float(np.nanmax(arr)):.4g}]')

nodata = pds.no_data_value if nodata is not None: try: arr = np.ma.masked_equal( arr, float(nodata[0] if isinstance(nodata, (list, tuple)) else nodata) ) except (TypeError, ValueError): pass fig, ax = plt.subplots(figsize=(6, 5)) im = ax.imshow(arr, cmap='viridis') ax.set_title(f'{'COPERNICUS/S1_GRD'} / {'VV'}') ax.set_xlabel('x (px)') ax.set_ylabel('y (px)') fig.colorbar(im, ax=ax, shrink=0.8) plt.tight_layout() plt.show() print(f'value range: [{float(np.nanmin(arr)):.4g}, {float(np.nanmax(arr)):.4g}]')

Tracking submitted jobs (asynchronous export)¶

The download above uses export_via="url" — a synchronous getDownloadURL round-trip. Nothing was queued, so there's no Earth Engine job to track.

To track an export instead, switch to an asynchronous sink (drive / gcs / asset) and pass wait_for_export=False so .download() returns a TaskInfo at submission time rather than blocking until completion. The cells below submit the same (asset_id, band, AOI, scale) request as an export_via="asset" task into the service account's own asset folder, then walk the four jobs-API calls (list_recent_tasks → wait_for_task_id → ee.data.getAsset → ee.data.deleteAsset) to make the job finish and tidy up. See track-batch-exports.ipynb for a deeper worked example.

Prepare the demo asset folder¶

GEE._export_via_batch writes the image at <asset_id>/<prefix>, so asset_id here is the parent Folder asset (not the final image path). We clear any leftovers from a previous run, then create a fresh empty folder that we own.

In [ ]:

import ee

from earthlens.gee import cancel_task, list_recent_tasks, wait_for_task_id

_proj = ee.data._get_projects_path().removeprefix('projects/')
DEMO_FOLDER = f'projects/{_proj}/assets/earthlens-demo-sar-radar'
print(f'demo folder: {DEMO_FOLDER}')

# Best-effort cleanup of leftover children from a previous run (so the
# folder is empty before we try to delete it below).
try:
    for child in ee.data.listAssets({'parent': DEMO_FOLDER}).get('assets', []):
        ee.data.deleteAsset(child['name'])
        print(f'cleared leftover child: {child["name"]}')
    ee.data.deleteAsset(DEMO_FOLDER)
    print(f'cleared leftover folder: {DEMO_FOLDER}')
except Exception:
    pass
# Create the parent folder — EE requires it to exist before a child write.
ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER)
print(f'created folder: {DEMO_FOLDER}')

import ee from earthlens.gee import cancel_task, list_recent_tasks, wait_for_task_id _proj = ee.data._get_projects_path().removeprefix('projects/') DEMO_FOLDER = f'projects/{_proj}/assets/earthlens-demo-sar-radar' print(f'demo folder: {DEMO_FOLDER}') # Best-effort cleanup of leftover children from a previous run (so the # folder is empty before we try to delete it below). try: for child in ee.data.listAssets({'parent': DEMO_FOLDER}).get('assets', []): ee.data.deleteAsset(child['name']) print(f'cleared leftover child: {child["name"]}') ee.data.deleteAsset(DEMO_FOLDER) print(f'cleared leftover folder: {DEMO_FOLDER}') except Exception: pass # Create the parent folder — EE requires it to exist before a child write. ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER) print(f'created folder: {DEMO_FOLDER}')

Submit¶

Same (asset_id, band, AOI, scale) request as the sync download above, just routed through export_via="asset" + wait_for_export=False. We build the request, authenticate, then submit — download() returns a TaskInfo per submitted bucket at the moment the task is queued, with no blocking.

In [ ]:

async_gee = EarthLens(
    data_source="gee",
    start='2024-06-01',
    end='2024-06-30',
    dataset='COPERNICUS/S1_GRD',
    variables=['VV'],
    aoi=[31.15, 29.95, 31.25, 30.05],
    cadence='monthly',
    path=str(OUT_DIR),
    scale=30.0,
    reducer='mean',
    export_via='asset',
    asset_id=DEMO_FOLDER,
    wait_for_export=False,
)

async_gee = EarthLens( data_source="gee", start='2024-06-01', end='2024-06-30', dataset='COPERNICUS/S1_GRD', variables=['VV'], aoi=[31.15, 29.95, 31.25, 30.05], cadence='monthly', path=str(OUT_DIR), scale=30.0, reducer='mean', export_via='asset', asset_id=DEMO_FOLDER, wait_for_export=False, )

Authenticate and submit the export. The returned TaskInfo carries the queued task's id and state.

In [ ]:

async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)
submitted = async_gee.download(progress_bar=False)
task_info = submitted[0]
print(f'submitted: id={task_info.id} state={task_info.state}')
print(f'           description={task_info.description}')

async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY) submitted = async_gee.download(progress_bar=False) task_info = submitted[0] print(f'submitted: id={task_info.id} state={task_info.state}') print(f' description={task_info.description}')

List + wait¶

list_recent_tasks(description_prefix=...) returns every matching task across the current project; wait_for_task_id blocks until the one we care about reaches a terminal state. A real workflow would just poll later from a separate process — the wait here exists so the notebook shows the full success path end-to-end.

In [ ]:

recent = list_recent_tasks(
    description_prefix=task_info.description,
    max_age_min=10,
)
print(f'list_recent_tasks matched {len(recent)} task(s):')
for t in recent:
    print(f'  {t.id}  {t.state:<12} {t.description}')

recent = list_recent_tasks( description_prefix=task_info.description, max_age_min=10, ) print(f'list_recent_tasks matched {len(recent)} task(s):') for t in recent: print(f' {t.id} {t.state:<12} {t.description}')

Now block on the specific task id until it reaches a terminal state. On FAILED / CANCELLED wait_for_task_id raises a RuntimeError; we cancel-if-still-running so we don't leak an in-flight task on notebook restart.

In [ ]:

try:
    final = wait_for_task_id(
        task_info.id,
        poll_seconds=10,
        progress_bar=False,
    )
    print(f'final state: {final.state}')
except RuntimeError as exc:
    print(f'wait_for_task_id raised: {exc}')
    try:
        cancel_task(task_info.id)
    except Exception:
        pass

try: final = wait_for_task_id( task_info.id, poll_seconds=10, progress_bar=False, ) print(f'final state: {final.state}') except RuntimeError as exc: print(f'wait_for_task_id raised: {exc}') try: cancel_task(task_info.id) except Exception: pass

Verify + clean up¶

Confirm the produced asset exists on Earth Engine, then delete it (and the surrounding demo folder) so we don't leak storage between notebook runs. The backend wrote the image at <DEMO_FOLDER>/<task description>.

In [ ]:

produced = f'{DEMO_FOLDER}/{task_info.description}'
try:
    meta = ee.data.getAsset(produced)
    print(f'asset exists: type={meta.get("type")} ' f'name={meta.get("name")}')
    ee.data.deleteAsset(produced)
    print('asset deleted')
except Exception as exc:
    print(f'verify/delete skipped: {exc}')
# Always try to tear down the parent folder.
try:
    ee.data.deleteAsset(DEMO_FOLDER)
    print(f'folder deleted: {DEMO_FOLDER}')
except Exception as exc:
    print(f'folder delete skipped: {exc}')

produced = f'{DEMO_FOLDER}/{task_info.description}' try: meta = ee.data.getAsset(produced) print(f'asset exists: type={meta.get("type")} ' f'name={meta.get("name")}') ee.data.deleteAsset(produced) print('asset deleted') except Exception as exc: print(f'verify/delete skipped: {exc}') # Always try to tear down the parent folder. try: ee.data.deleteAsset(DEMO_FOLDER) print(f'folder deleted: {DEMO_FOLDER}') except Exception as exc: print(f'folder delete skipped: {exc}')

What's on disk¶

The GeoTIFF is left under the per-notebook out/ directory for you to inspect. That directory is .gitignored — re-running the notebook overwrites it.

In [ ]:

for p in sorted(OUT_DIR.iterdir()) if OUT_DIR.exists() else []:
    print(f'{p}  ({p.stat().st_size / 1024:.1f} KB)')

for p in sorted(OUT_DIR.iterdir()) if OUT_DIR.exists() else []: print(f'{p} ({p.stat().st_size / 1024:.1f} KB)')