ERDDAP catalog tooling — validate, audit coverage, curate¶

The ERDDAP backend ships a small curated catalog, but ERDDAP servers publish thousands of datasets. The earthlens datasets CLI gives a maintainer the loop to grow that catalog safely: validate the rows, audit --coverage to see what is worth curating next, curate a row straight from a dataset's metadata, and refresh the index. This notebook drives that loop through the same Python entry points the CLI uses, live against the public NOAA CoastWatch ERDDAP.

Setup¶

list_backends() resolves the erddap backend; the tooling functions take that BackendInfo and load the bundled catalog themselves.

In [ ]:

from earthlens.cli.adapter import list_backends
from earthlens.cli.refresh import coverage_one
from earthlens.cli.stanza import emit_stanza
from earthlens.cli.validate import validate_one

info = next(b for b in list_backends() if b.provider == 'erddap')
print('provider:', info.provider)
print('module:  ', info.module)
print('aliases: ', info.aliases)

from earthlens.cli.adapter import list_backends from earthlens.cli.refresh import coverage_one from earthlens.cli.stanza import emit_stanza from earthlens.cli.validate import validate_one info = next(b for b in list_backends() if b.provider == 'erddap') print('provider:', info.provider) print('module: ', info.module) print('aliases: ', info.aliases)

1. Validate the shipped catalog (offline)¶

validate runs an offline structural lint over every curated row — a non-http(s) server_url, a griddap row with empty dim_names, or a flux_variables entry that is not one of the row's variables. The shipped catalog lints clean.

In [ ]:

result = validate_one(info)
print('status :', result.status)
print('checked:', result.checked, 'rows')
print('issues :', result.issues or 'none')

result = validate_one(info) print('status :', result.status) print('checked:', result.checked, 'rows') print('issues :', result.issues or 'none')

2. Audit coverage — classify the server universe (live)¶

audit --coverage walks each curated server's allDatasets table and buckets every dataset by curation status and data structure: DONE (already curated), addressable (a griddap raster worth curating next), table (a tabledap dataset — the separate tabular track), thin (ERDDAP test/demo datasets), missing (an id in the bundled available_datasets: index the server isn't currently serving — a stale index, or, as here, a server mid-reload). todo is the addressable griddap backlog worth curating next.

In [ ]:

coverage = coverage_one(info)
print('status:', coverage.status)
for bucket, count in coverage.counts.items():
    print(f'  {bucket:12s} {count:5d}')
print()
print(f'addressable griddap datasets not yet curated: {len(coverage.todo)}')
print('first 8 todo:', coverage.todo[:8])

coverage = coverage_one(info) print('status:', coverage.status) for bucket, count in coverage.counts.items(): print(f' {bucket:12s} {count:5d}') print() print(f'addressable griddap datasets not yet curated: {len(coverage.todo)}') print('first 8 todo:', coverage.todo[:8])

3. Curate a row straight from /info (live)¶

curate seeds a complete catalog row from a dataset's /info metadata: the grid dimensions decide protocol (griddap if dimensioned, else tabledap) and dim_names, the variable list becomes the default variables, and the NC_GLOBAL title / license fill the human fields. We pick the first dataset from the live todo backlog above, so the example always points at a dataset the server is serving right now.

In [ ]:

import json

target = coverage.todo[0]
print('seeding a row for:', target)
seed = emit_stanza(info, target)
print('status:', seed.status)
print(json.dumps(seed.row, indent=2))

import json target = coverage.todo[0] print('seeding a row for:', target) seed = emit_stanza(info, target) print('status:', seed.status) print(json.dumps(seed.row, indent=2))

4. The seeded row is a valid catalog row¶

The emitted dict round-trips through the pydantic Dataset model (extra="forbid"), so it is a paste-ready row — the maintainer trims variables to the headline set, then commits it to a slice file.

In [ ]:

from earthlens.erddap.catalog import Dataset

ds = Dataset(**seed.row)
print('protocol :', ds.protocol)
print('dim_names:', ds.dim_names)
print('variables:', ds.variables)
print('title    :', ds.title[:70])

from earthlens.erddap.catalog import Dataset ds = Dataset(**seed.row) print('protocol :', ds.protocol) print('dim_names:', ds.dim_names) print('variables:', ds.variables) print('title :', ds.title[:70])

Takeaway¶

The maintainer loop is four commands: validate erddap keeps the rows coherent, audit erddap --coverage surfaces the addressable backlog, curate erddap <id> seeds a row from metadata, and refresh erddap --write regenerates the available_datasets: index by crawling each server's allDatasets table. Every step is offline-safe except the live crawls, and a stale or broken row is caught before it ships.