ERDDAP catalog tooling — validate, audit coverage, curate¶

The ERDDAP backend ships a small curated catalog, but ERDDAP servers publish thousands of datasets. The earthlens datasets CLI gives a maintainer the loop to grow that catalog safely: validate the rows, audit --coverage to see what is worth curating next, curate a row straight from a dataset's metadata, and refresh the index. This notebook drives that loop through the same Python entry points the CLI uses, live against the public NOAA CoastWatch ERDDAP.

Setup¶

list_backends() resolves the erddap backend; the tooling functions take that BackendInfo and load the bundled catalog themselves.

In [1]:

from earthlens.cli.adapter import list_backends
from earthlens.cli.validate import validate_one
from earthlens.cli.refresh import coverage_one
from earthlens.cli.stanza import emit_stanza

info = next(b for b in list_backends() if b.provider == 'erddap')
print('provider:', info.provider)
print('module:  ', info.module)
print('aliases: ', info.aliases)

from earthlens.cli.adapter import list_backends from earthlens.cli.validate import validate_one from earthlens.cli.refresh import coverage_one from earthlens.cli.stanza import emit_stanza info = next(b for b in list_backends() if b.provider == 'erddap') print('provider:', info.provider) print('module: ', info.module) print('aliases: ', info.aliases)

provider: erddap
module:   earthlens.erddap
aliases:  ('erddap', 'ioos')

1. Validate the shipped catalog (offline)¶

validate runs an offline structural lint over every curated row — a non-http(s) server_url, a griddap row with empty dim_names, or a flux_variables entry that is not one of the row's variables. The shipped catalog lints clean.

In [2]:

result = validate_one(info)
print('status :', result.status)
print('checked:', result.checked, 'rows')
print('issues :', result.issues or 'none')

result = validate_one(info) print('status :', result.status) print('checked:', result.checked, 'rows') print('issues :', result.issues or 'none')

status : ok
checked: 4 rows
issues : none

2. Audit coverage — classify the server universe (live)¶

audit --coverage walks each curated server's allDatasets table and buckets every dataset by curation status and data structure: DONE (already curated), addressable (a griddap raster worth curating next), table (a tabledap dataset — the separate tabular track), thin (ERDDAP test/demo datasets), missing (an id in the bundled available_datasets: index the server isn't currently serving — a stale index, or, as here, a server mid-reload). todo is the addressable griddap backlog worth curating next.

In [3]:

coverage = coverage_one(info)
print('status:', coverage.status)
for bucket, count in coverage.counts.items():
    print(f'  {bucket:12s} {count:5d}')
print()
print(f'addressable griddap datasets not yet curated: {len(coverage.todo)}')
print('first 8 todo:', coverage.todo[:8])

coverage = coverage_one(info) print('status:', coverage.status) for bucket, count in coverage.counts.items(): print(f' {bucket:12s} {count:5d}') print() print(f'addressable griddap datasets not yet curated: {len(coverage.todo)}') print('first 8 todo:', coverage.todo[:8])

status: ok
  DONE             4
  addressable     36
  thin             4
  table          170
  missing       2850

addressable griddap datasets not yet curated: 36
first 8 todo: ['NOAA_DHW_Lon0360', 'NOAA_DHW_monthly', 'NOAA_DHW_monthly_Lon0360', 'erdATclddhday', 'erdATcldnhday', 'erdEtopo22SeafloorGradient', 'erdEtopo22SeafloorGradient_Atlantic', 'erdEtopoSeafloorGradient']

3. Curate a row straight from /info (live)¶

curate seeds a complete catalog row from a dataset's /info metadata: the grid dimensions decide protocol (griddap if dimensioned, else tabledap) and dim_names, the variable list becomes the default variables, and the NC_GLOBAL title / license fill the human fields. We pick the first dataset from the live todo backlog above, so the example always points at a dataset the server is serving right now.

In [4]:

import json

target = coverage.todo[0]
print('seeding a row for:', target)
seed = emit_stanza(info, target)
print('status:', seed.status)
print(json.dumps(seed.row, indent=2))

import json target = coverage.todo[0] print('seeding a row for:', target) seed = emit_stanza(info, target) print('status:', seed.status) print(json.dumps(seed.row, indent=2))

seeding a row for: NOAA_DHW_Lon0360

status: ok
{
  "server_url": "https://coastwatch.pfeg.noaa.gov/erddap",
  "dataset_id": "NOAA_DHW_Lon0360",
  "protocol": "griddap",
  "dim_names": [
    "time",
    "latitude",
    "longitude"
  ],
  "variables": [
    "CRW_BAA",
    "CRW_BAA_mask",
    "CRW_BAA_7D_MAX",
    "CRW_BAA_7D_MAX_mask",
    "CRW_DHW",
    "CRW_DHW_mask",
    "CRW_HOTSPOT",
    "CRW_HOTSPOT_mask",
    "CRW_SEAICE",
    "CRW_SST",
    "CRW_SSTANOMALY",
    "CRW_SSTANOMALY_mask"
  ],
  "title": "NOAA Coral Reef Watch Operational Daily Near-Real-Time Global 5-km Satellite Coral Bleaching Monitoring Products, Lon0360",
  "license_note": "OSTIA Usage Statement (1985-2002): IMPORTANT usage statement. Unless otherwise agreed in writing, these data may be used for pure academic research only, with no commercial or other application and all usage must meet the Met Office Standard Terms and Conditions, which may be found here: https://www.metoffice.gov.uk/corporate/legal/tandc.html. The data may be used for a maximum period of 5 years. Reproduction of the data is permitted provided the following copyright statement is included: (C) Crown Copyright 2010, published by the Met Office. You must submit a completed reproduction license application form (here https://www.metoffice.gov.uk/corporate/legal/repro_licence.html) before using the data. This only needs to be completed once for each user. WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value. GHRSST statement (2002-present): GHRSST protocol describes data use as free and open. Coral Reef Watch program statement: The data produced by Coral Reef Watch are available for use without restriction, but Coral Reef Watch relies on the ethics and integrity of the user to ensure that the source of the data and products is appropriately cited and credited. When using these data and products, credit and courtesy should be given to NOAA Coral Reef Watch. Please include the appropriate DOI associated with this dataset in the citation. For more information, visit the NOAA Coral Reef Watch website: https://coralreefwatch.noaa.gov. Recommendations for citing and providing credit are provided at https://coralreefwatch.noaa.gov/satellite/docs/recommendations_crw_citation.php. Users are referred to the footer section of Coral Reef Watch's website (https://coralreefwatch.noaa.gov/index.php) for disclaimers, policies, notices pertaining to the use of the data."
}

4. The seeded row is a valid catalog row¶

The emitted dict round-trips through the pydantic Dataset model (extra="forbid"), so it is a paste-ready row — the maintainer trims variables to the headline set, then commits it to a slice file.

In [5]:

from earthlens.erddap.catalog import Dataset

ds = Dataset(**seed.row)
print('protocol :', ds.protocol)
print('dim_names:', ds.dim_names)
print('variables:', ds.variables)
print('title    :', ds.title[:70])

from earthlens.erddap.catalog import Dataset ds = Dataset(**seed.row) print('protocol :', ds.protocol) print('dim_names:', ds.dim_names) print('variables:', ds.variables) print('title :', ds.title[:70])

protocol : griddap
dim_names: ['time', 'latitude', 'longitude']
variables: ['CRW_BAA', 'CRW_BAA_mask', 'CRW_BAA_7D_MAX', 'CRW_BAA_7D_MAX_mask', 'CRW_DHW', 'CRW_DHW_mask', 'CRW_HOTSPOT', 'CRW_HOTSPOT_mask', 'CRW_SEAICE', 'CRW_SST', 'CRW_SSTANOMALY', 'CRW_SSTANOMALY_mask']
title    : NOAA Coral Reef Watch Operational Daily Near-Real-Time Global 5-km Sat

Takeaway¶

The maintainer loop is four commands: validate erddap keeps the rows coherent, audit erddap --coverage surfaces the addressable backlog, curate erddap <id> seeds a row from metadata, and refresh erddap --write regenerates the available_datasets: index by crawling each server's allDatasets table. Every step is offline-safe except the live crawls, and a stale or broken row is caught before it ships.