Atmosphere & chemistry — Sentinel-5P NO2¶

Tropospheric NO2 column density from TROPOMI on Sentinel-5P (near-real-time, ~7 km native pixel). A short window over Paris, reduced to one composite scene.

Setup¶

pyramids provides Dataset (GeoTIFF/NetCDF reading + plotting); earthlens provides the unified EarthLens entry point and the GEE Catalog. We also create a per-notebook out/ directory for the downloaded GeoTIFFs.

In [ ]:

import os
from pathlib import Path

from pyramids.dataset import Dataset
from pyramids.plot import ColorBar

from earthlens.core import EarthLens
from earthlens.gee import Catalog, cancel_task

OUT_DIR = Path('out') / 'atmosphere-chemistry'
OUT_DIR.mkdir(parents=True, exist_ok=True)
print(f'output directory: {OUT_DIR.resolve()}')

import os from pathlib import Path from pyramids.dataset import Dataset from pyramids.plot import ColorBar from earthlens.core import EarthLens from earthlens.gee import Catalog, cancel_task OUT_DIR = Path('out') / 'atmosphere-chemistry' OUT_DIR.mkdir(parents=True, exist_ok=True) print(f'output directory: {OUT_DIR.resolve()}')

Credentials¶

The notebook reads the GEE service-account credentials from the GEE_SERVICE_ACCOUNT / GEE_SERVICE_KEY environment variables. Both must be set before running this cell.

In [ ]:

SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT']
SERVICE_KEY = os.environ['GEE_SERVICE_KEY']

SERVICE_ACCOUNT = os.environ['GEE_SERVICE_ACCOUNT'] SERVICE_KEY = os.environ['GEE_SERVICE_KEY']

Inspect the catalog entry¶

Before downloading anything, look at what the bundled catalog knows about the asset — bands, cadence, license, provider.

In [ ]:

cat = Catalog()
ds = cat.get_dataset('COPERNICUS/S5P/NRTI/L3_NO2')
print(ds)

# The summary clips long text and shows only a band count, so an explorer
# notebook still wants the untruncated title and the fields it omits:
print(f'title (full):        {ds.title}')
print(f'ee_type:             {ds.ee_type}')
print(f'default_reducer:     {ds.default_reducer}')
print(f'license:             {ds.license}')
print(f'band ids (first 5):  {list(ds.bands)[:5]}')

cat = Catalog() ds = cat.get_dataset('COPERNICUS/S5P/NRTI/L3_NO2') print(ds) # The summary clips long text and shows only a band count, so an explorer # notebook still wants the untruncated title and the fields it omits: print(f'title (full): {ds.title}') print(f'ee_type: {ds.ee_type}') print(f'default_reducer: {ds.default_reducer}') print(f'license: {ds.license}') print(f'band ids (first 5): {list(ds.bands)[:5]}')

Download¶

Tiny AOI ([48.5, 49.5] lat, [2.0, 3.0] lon) at 7000.0 m, raw cadence — keeps the synchronous download under EE's 32768-px per-axis cap.

This request pins engine='ee'. Under the default engine='auto' the pixels are served by the EEDAI reader, which composites client-side without a nodata value — and a TROPOMI L3 scene is a narrow swath that is masked almost everywhere outside its own overpass. Averaging 13 such scenes unmasked propagates the masked cells into every pixel, and the whole window comes back empty. engine='ee' makes Earth Engine do the reduction with its own masking, which is what this notebook is illustrating.

Build the request and authenticate¶

Construct the EarthLens request first, then resolve the GEE service-account credentials with authenticate() on its own line so each step is easy to read and re-run.

In [ ]:

gee = EarthLens(
    data_source="gee",
    start='2024-01-05',
    end='2024-01-10',
    dataset='COPERNICUS/S5P/NRTI/L3_NO2',
    variables=['tropospheric_NO2_column_number_density'],
    aoi=[2.0, 48.5, 3.0, 49.5],
    cadence='raw',
    path=OUT_DIR,
    scale=7000.0,
    reducer='mean',
    engine='ee',
)
gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)

gee = EarthLens( data_source="gee", start='2024-01-05', end='2024-01-10', dataset='COPERNICUS/S5P/NRTI/L3_NO2', variables=['tropospheric_NO2_column_number_density'], aoi=[2.0, 48.5, 3.0, 49.5], cadence='raw', path=OUT_DIR, scale=7000.0, reducer='mean', engine='ee', ) gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)

Fetch the composite¶

download() writes the reduced scene to disk and returns the list of GeoTIFF paths.

In [ ]:

paths = gee.download(progress_bar=False)
print(f'wrote {len(paths)} GeoTIFF(s):')
for p in paths:
    print(f'  {p}  ({p.stat().st_size / 1024:.1f} KB)')

paths = gee.download(progress_bar=False) print(f'wrote {len(paths)} GeoTIFF(s):') for p in paths: print(f' {p} ({p.stat().st_size / 1024:.1f} KB)')

Quick preview¶

Load the first written GeoTIFF through pyramids and render the single band. (pyramids.dataset.Dataset is the project's GeoTIFF/NetCDF wrapper.)

Load and declare the fill¶

Read the written GeoTIFF through pyramids. Earth Engine writes the cells it had masked as 0, and does not declare that as the band's nodata — so the raster arrives claiming a tropospheric NO2 column of exactly zero, which is not a physical value (the smallest real reading here is about 9e-07 mol/m²). Declaring it keeps those cells out of the colour ramp and the statistics.

In [ ]:

preview = Dataset.read_file(paths[0], read_only=False)
print(f'shape (y, x) : {preview.rows} x {preview.columns}')
print(f'declared     : {preview.no_data_value}')

preview.no_data_value = [0.0]
print(f'after declare: {preview.no_data_value}')

preview = Dataset.read_file(paths[0], read_only=False) print(f'shape (y, x) : {preview.rows} x {preview.columns}') print(f'declared : {preview.no_data_value}') preview.no_data_value = [0.0] print(f'after declare: {preview.no_data_value}')

Render the NO2 column¶

A tropospheric NO2 column is on the order of 1e-4 mol/m², so plotting the band as delivered leaves the colour bar ticking through unreadable 0.0000… steps. apply() rescales it to micromoles per square metre — it is a single-raster operation, so the georeferencing and the declared nodata come through untouched and the masked cells stay masked.

The bright cells over the centre of the window are the Paris plume.

In [ ]:

# mol/m² -> µmol/m², so the colour bar reads 0.9 … 172 instead of 0.0000009 … 0.0001723.
column = preview.apply(lambda v: v * 1e6)

glyph = column.plot(
    cmap='inferno',
    colorbar=ColorBar(label='tropospheric NO₂ column (µmol/m²)'),
    title='Sentinel-5P NO₂ — Paris, 5–10 Jan 2024',
)
glyph.ax.title.set_fontsize(11)

stats = column.stats(approx_ok=False)
low, high = stats['min'].iloc[0], stats['max'].iloc[0]
print(f'value range: [{low:.4g}, {high:.4g}] µmol/m²')
# Earth Engine wrote its masked cells as 0, which is not a physical column.
# A minimum of exactly 0 would mean the declaration silently did nothing.
assert low > 0, f'NO2 column should be strictly positive after masking; got {low}'

retrieved = column.count_domain_cells()
total_cells = column.rows * column.columns
print(f'retrieved cells: {retrieved:,} of {total_cells:,}')

# The handle was opened writable to declare the nodata; release it so the
# GDAL lock does not outlive the cell.
column.close()
preview.close()

# mol/m² -> µmol/m², so the colour bar reads 0.9 … 172 instead of 0.0000009 … 0.0001723. column = preview.apply(lambda v: v * 1e6) glyph = column.plot( cmap='inferno', colorbar=ColorBar(label='tropospheric NO₂ column (µmol/m²)'), title='Sentinel-5P NO₂ — Paris, 5–10 Jan 2024', ) glyph.ax.title.set_fontsize(11) stats = column.stats(approx_ok=False) low, high = stats['min'].iloc[0], stats['max'].iloc[0] print(f'value range: [{low:.4g}, {high:.4g}] µmol/m²') # Earth Engine wrote its masked cells as 0, which is not a physical column. # A minimum of exactly 0 would mean the declaration silently did nothing. assert low > 0, f'NO2 column should be strictly positive after masking; got {low}' retrieved = column.count_domain_cells() total_cells = column.rows * column.columns print(f'retrieved cells: {retrieved:,} of {total_cells:,}') # The handle was opened writable to declare the nodata; release it so the # GDAL lock does not outlive the cell. column.close() preview.close()

Tracking submitted jobs (asynchronous export)¶

The download above uses export_via="url" — a synchronous getDownloadURL round-trip. Nothing was queued, so there's no Earth Engine job to track.

To track an export instead, switch to an asynchronous sink (drive / gcs / asset) and pass wait_for_export=False so .download() returns a TaskInfo at submission time rather than blocking until completion. The cells below submit the same (asset_id, band, AOI, scale) request as an export_via="asset" task into the service account's own asset folder, then walk the four jobs-API calls (list_recent_tasks → wait_for_task_id → ee.data.getAsset → ee.data.deleteAsset) to make the job finish and tidy up. See track-batch-exports.ipynb for a deeper worked example.

Imports and the demo asset folder¶

Bring in the jobs-API helpers and derive a Folder asset path under the current project. The backend writes the image at <asset_id>/<prefix>, so this asset_id is the parent folder, not the final image path.

In [ ]:

import ee

from earthlens.gee import list_recent_tasks, wait_for_task_id

# The asset goes into a `Folder` asset that we own. `GEE._export_via_batch`
# writes the actual image at `<asset_id>/<prefix>`, so `asset_id` here is
# the parent FOLDER (not the final image path). Both must be cleaned up.
_proj = ee.data._get_projects_path().removeprefix('projects/')
PARENT = f'projects/{_proj}/assets'
DEMO_FOLDER = f'{PARENT}/earthlens-demo-atmosphere-chemistry'
print(f'demo folder: {DEMO_FOLDER}')

import ee from earthlens.gee import list_recent_tasks, wait_for_task_id # The asset goes into a `Folder` asset that we own. `GEE._export_via_batch` # writes the actual image at `<asset_id>/<prefix>`, so `asset_id` here is # the parent FOLDER (not the final image path). Both must be cleaned up. _proj = ee.data._get_projects_path().removeprefix('projects/') PARENT = f'projects/{_proj}/assets' DEMO_FOLDER = f'{PARENT}/earthlens-demo-atmosphere-chemistry' print(f'demo folder: {DEMO_FOLDER}')

Prepare a clean folder¶

Listing the parent says whether a previous run left the folder behind, so the cleanup never has to swallow a "not found" from Earth Engine. Then create the folder Earth Engine requires before a child write.

In [ ]:

# Listing the parent says whether a previous run left the folder behind, so the
# cleanup below never has to swallow a "not found" from Earth Engine.
listed = ee.data.listAssets({'parent': PARENT})
siblings = [asset['name'] for asset in listed.get('assets', [])]
if DEMO_FOLDER in siblings:
    children = ee.data.listAssets({'parent': DEMO_FOLDER})
    for child in children.get('assets', []):
        ee.data.deleteAsset(child['name'])
        print(f'cleared leftover child: {child["name"]}')
    ee.data.deleteAsset(DEMO_FOLDER)
    print(f'cleared leftover folder: {DEMO_FOLDER}')
# Create the parent folder — EE requires it to exist before a child write.
ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER)
print(f'created folder: {DEMO_FOLDER}')

# Listing the parent says whether a previous run left the folder behind, so the # cleanup below never has to swallow a "not found" from Earth Engine. listed = ee.data.listAssets({'parent': PARENT}) siblings = [asset['name'] for asset in listed.get('assets', [])] if DEMO_FOLDER in siblings: children = ee.data.listAssets({'parent': DEMO_FOLDER}) for child in children.get('assets', []): ee.data.deleteAsset(child['name']) print(f'cleared leftover child: {child["name"]}') ee.data.deleteAsset(DEMO_FOLDER) print(f'cleared leftover folder: {DEMO_FOLDER}') # Create the parent folder — EE requires it to exist before a child write. ee.data.createAsset({'type': 'Folder'}, DEMO_FOLDER) print(f'created folder: {DEMO_FOLDER}')

Submit¶

Same (asset_id, band, AOI, scale) request as the sync download above, just routed through export_via="asset" + wait_for_export=False. download() returns a TaskInfo per submitted bucket at the moment the task is queued — no blocking.

Build the async request and authenticate¶

Same (asset_id, band, AOI, scale) request as the sync download, but routed through export_via="asset" + wait_for_export=False. Construct it first, then authenticate() on its own line.

In [ ]:

async_gee = EarthLens(
    data_source="gee",
    start='2024-01-05',
    end='2024-01-10',
    dataset='COPERNICUS/S5P/NRTI/L3_NO2',
    variables=['tropospheric_NO2_column_number_density'],
    aoi=[2.0, 48.5, 3.0, 49.5],
    cadence='raw',
    path=OUT_DIR,
    scale=7000.0,
    reducer='mean',
    export_via='asset',
    asset_id=DEMO_FOLDER,
    wait_for_export=False,
)
async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)

async_gee = EarthLens( data_source="gee", start='2024-01-05', end='2024-01-10', dataset='COPERNICUS/S5P/NRTI/L3_NO2', variables=['tropospheric_NO2_column_number_density'], aoi=[2.0, 48.5, 3.0, 49.5], cadence='raw', path=OUT_DIR, scale=7000.0, reducer='mean', export_via='asset', asset_id=DEMO_FOLDER, wait_for_export=False, ) async_gee.authenticate(service_account=SERVICE_ACCOUNT, service_key=SERVICE_KEY)

Submit the export¶

download() returns a TaskInfo per submitted bucket at the moment the task is queued — no blocking.

In [ ]:

submitted = async_gee.download(progress_bar=False)
task_info = submitted[0]
print(f'submitted: id={task_info.id} state={task_info.state}')
print(f'           description={task_info.description}')

submitted = async_gee.download(progress_bar=False) task_info = submitted[0] print(f'submitted: id={task_info.id} state={task_info.state}') print(f' description={task_info.description}')

List + wait¶

list_recent_tasks(description_prefix=...) returns every matching task across the current project; wait_for_task_id blocks until the one we care about reaches a terminal state. A real workflow would just poll later from a separate process — the wait here exists so the notebook shows the full success path end-to-end.

In [ ]:

recent = list_recent_tasks(
    description_prefix=task_info.description,
    max_age_min=10,
)
print(f'list_recent_tasks matched {len(recent)} task(s):')
for t in recent:
    print(f'  {t.id}  {t.state:<12} {t.description}')
final = None
try:
    final = wait_for_task_id(
        task_info.id,
        poll_seconds=10,
        progress_bar=False,
    )
    print(f'\nfinal state: {final.state}')
finally:
    if final is None:
        # The wait raises on FAILED / CANCELLED *and on timeout* — and a
        # timeout leaves the export still running. Cancel it so an aborted
        # notebook does not leave a live task behind; cancel_task is a no-op
        # on an already-terminal task, and the original error still
        # propagates out of this finally.
        cancel_task(task_info.id)
        print(f'cancelled {task_info.id} after the wait failed')

recent = list_recent_tasks( description_prefix=task_info.description, max_age_min=10, ) print(f'list_recent_tasks matched {len(recent)} task(s):') for t in recent: print(f' {t.id} {t.state:<12} {t.description}') final = None try: final = wait_for_task_id( task_info.id, poll_seconds=10, progress_bar=False, ) print(f'\nfinal state: {final.state}') finally: if final is None: # The wait raises on FAILED / CANCELLED *and on timeout* — and a # timeout leaves the export still running. Cancel it so an aborted # notebook does not leave a live task behind; cancel_task is a no-op # on an already-terminal task, and the original error still # propagates out of this finally. cancel_task(task_info.id) print(f'cancelled {task_info.id} after the wait failed')

Verify + clean up¶

Confirm the produced asset exists on Earth Engine, then delete it (and the surrounding demo folder) so we don't leak storage between notebook runs. The backend wrote the image at <DEMO_FOLDER>/<task description>.

In [ ]:

produced = f'{DEMO_FOLDER}/{task_info.description}'
meta = ee.data.getAsset(produced)
print(f'asset exists: type={meta.get("type")} name={meta.get("name")}')
ee.data.deleteAsset(produced)
print('asset deleted')
# Tear down the parent folder.
ee.data.deleteAsset(DEMO_FOLDER)
print(f'folder deleted: {DEMO_FOLDER}')

produced = f'{DEMO_FOLDER}/{task_info.description}' meta = ee.data.getAsset(produced) print(f'asset exists: type={meta.get("type")} name={meta.get("name")}') ee.data.deleteAsset(produced) print('asset deleted') # Tear down the parent folder. ee.data.deleteAsset(DEMO_FOLDER) print(f'folder deleted: {DEMO_FOLDER}')

What's on disk¶

The written GeoTIFF is left under the per-notebook out/ directory for you to inspect. That directory is .gitignored — re-running the notebook overwrites it.

In [ ]:

for p in sorted(OUT_DIR.iterdir()):
    print(f'{p}  ({p.stat().st_size / 1024:.1f} KB)')

for p in sorted(OUT_DIR.iterdir()): print(f'{p} ({p.stat().st_size / 1024:.1f} KB)')