OSM — power user & all the knobs¶

The three download-type notebooks (quickstart, ohsome_history, pbf_bulk_extracts) cover the common path. This one gathers the cross-cutting capabilities of the earthlens.osm backend: exploring the named-query catalog, requesting several queries at once, choosing the output format, the bounding-box safety guard, and the two read engines compared side by side.

Every result is the same vector FeatureCollection (EPSG:4326) and emits the ODbL LicenseWarning.

Setup¶

In [ ]:

from pathlib import Path

import matplotlib.pyplot as plt

from earthlens.core import EarthLens
from earthlens.osm import Catalog

OUT_DIR = Path('osm_output')
OUT_DIR.mkdir(exist_ok=True)
LAT_LIM = [49.40, 49.42]
LON_LIM = [8.67, 8.71]
HOSPITALS = "live:hospitals"  # the named query reused across the examples below

from pathlib import Path import matplotlib.pyplot as plt from earthlens.core import EarthLens from earthlens.osm import Catalog OUT_DIR = Path('osm_output') OUT_DIR.mkdir(exist_ok=True) LAT_LIM = [49.40, 49.42] LON_LIM = [8.67, 8.71] HOSPITALS = "live:hospitals" # the named query reused across the examples below

Exploring the named-query catalog¶

The OSM backend is driven by a small curated catalog of <type>:<name> named queries. EarthLens.list_datasets('osm') lists them; the Catalog object exposes each row's dispatch detail — the Overpass QL template, the ohsome filter, or the pyrosm layer method.

In [ ]:

EarthLens.list_datasets('osm')

EarthLens.list_datasets('osm')

In [ ]:

cat = Catalog()
print('live:hospitals QL  :', cat.get(HOSPITALS).query_template)
print('history:buildings filter:', cat.get('history:buildings').ohsome_filter)
print(
    'bulk:roads method       :',
    cat.get('bulk:roads').pyrosm_method,
    '| network_type:',
    cat.get('bulk:roads').network_type,
)
print('bulk region keys        :', cat.region_ids())

cat = Catalog() print('live:hospitals QL :', cat.get(HOSPITALS).query_template) print('history:buildings filter:', cat.get('history:buildings').ohsome_filter) print( 'bulk:roads method :', cat.get('bulk:roads').pyrosm_method, '| network_type:', cat.get('bulk:roads').network_type, ) print('bulk region keys :', cat.region_ids())

Several named queries in one call¶

variables= takes a list, and download() concatenates the per-query results into one FeatureCollection (the union of their columns). Here hospitals and schools come back together.

In [ ]:

combined = EarthLens(
    data_source='osm',
    variables=[HOSPITALS, 'live:schools'],
    lat_lim=LAT_LIM,
    lon_lim=LON_LIM,
    path=OUT_DIR,
).download()

len(combined), [c for c in ['osm_id', 'amenity', 'geometry'] if c in combined.columns]

combined = EarthLens( data_source='osm', variables=[HOSPITALS, 'live:schools'], lat_lim=LAT_LIM, lon_lim=LON_LIM, path=OUT_DIR, ).download() len(combined), [c for c in ['osm_id', 'amenity', 'geometry'] if c in combined.columns]

Output format — GeoJSON or GeoPackage¶

Every download() also writes one vector file under path. The default is GeoJSON; pass file_format='gpkg' for a GeoPackage. The file is named osm_<query-ids>.<ext>, so the format only changes the extension and OGR driver.

In [ ]:

EarthLens(
    data_source='osm',
    variables=['live:cafes'],
    lat_lim=LAT_LIM,
    lon_lim=LON_LIM,
    file_format='gpkg',
    path=OUT_DIR,
).download()

sorted(p.name for p in OUT_DIR.glob('osm_live-cafes.*'))

EarthLens( data_source='osm', variables=['live:cafes'], lat_lim=LAT_LIM, lon_lim=LON_LIM, file_format='gpkg', path=OUT_DIR, ).download() sorted(p.name for p in OUT_DIR.glob('osm_live-cafes.*'))

The bounding-box guard¶

OSM live queries are for small areas. The backend rejects a bbox whose area exceeds max_bbox_deg2 (default 100 deg², about a large country) before issuing any request — in particular the whole-Earth box you would get by omitting lat_lim / lon_lim through the facade. Raise the cap with max_bbox_deg2= when you really mean it.

In [ ]:

# An oversized box is rejected up front — no request is sent:
try:
    EarthLens(
        data_source='osm',
        variables=[HOSPITALS],
        lat_lim=[-40, 40],
        lon_lim=[-100, 100],  # 80 x 200 deg = 16000 deg^2, far over the 100 cap
        path=OUT_DIR,
    ).download()
except ValueError as exc:
    print('rejected:', exc)

# An oversized box is rejected up front — no request is sent: try: EarthLens( data_source='osm', variables=[HOSPITALS], lat_lim=[-40, 40], lon_lim=[-100, 100], # 80 x 200 deg = 16000 deg^2, far over the 100 cap path=OUT_DIR, ).download() except ValueError as exc: print('rejected:', exc)

The two read engines, compared¶

The bulk type reads a .osm.pbf extract with one of two engines. pyosmium (the default) streams with bounded memory and ships with earthlens[osm]; it returns a slim osm_id / osm_type / geometry schema. pyrosm reads the whole extract in memory and returns the richer, exact per-layer schema (parsed tag columns), but is opt-in — it is in no earthlens extra, so it is installed on demand (it builds the sdist-only cykhash from source, which needs a C compiler).

The cell below installs it live, then reads Malta's buildings with each engine so you can compare the schemas.

In [ ]:

%pip install --quiet pyrosm

%pip install --quiet pyrosm

In [ ]:

# pbf reads a Geofabrik region; compare the engines on Malta's Valletta core
malta = {
    'region': 'malta',
    'lat_lim': [35.88, 35.94],
    'lon_lim': [14.49, 14.55],
    'path': OUT_DIR,
}

pyosmium_fc = EarthLens(
    data_source='osm', variables=['bulk:buildings'], engine='pyosmium', **malta
).download()
pyrosm_fc = EarthLens(
    data_source='osm', variables=['bulk:buildings'], engine='pyrosm', **malta
).download()

print('pyosmium:', len(pyosmium_fc), 'features | columns:', list(pyosmium_fc.columns))
print('pyrosm  :', len(pyrosm_fc), 'features | columns:', list(pyrosm_fc.columns)[:12])

# pbf reads a Geofabrik region; compare the engines on Malta's Valletta core malta = { 'region': 'malta', 'lat_lim': [35.88, 35.94], 'lon_lim': [14.49, 14.55], 'path': OUT_DIR, } pyosmium_fc = EarthLens( data_source='osm', variables=['bulk:buildings'], engine='pyosmium', **malta ).download() pyrosm_fc = EarthLens( data_source='osm', variables=['bulk:buildings'], engine='pyrosm', **malta ).download() print('pyosmium:', len(pyosmium_fc), 'features | columns:', list(pyosmium_fc.columns)) print('pyrosm :', len(pyrosm_fc), 'features | columns:', list(pyrosm_fc.columns)[:12])

In [ ]:

fig, axes = plt.subplots(1, 2, figsize=(12, 6))
pyosmium_fc.plot(ax=axes[0], markersize=2, color='#4457a4')
axes[0].set_title(
    f'pyosmium (slim) - {len(pyosmium_fc)} buildings, {len(pyosmium_fc.columns)} cols'
)
pyrosm_fc.plot(ax=axes[1], markersize=2, color='#c05a3c')
axes[1].set_title(
    f'pyrosm (rich) - {len(pyrosm_fc)} buildings, {len(pyrosm_fc.columns)} cols'
)
for ax in axes:
    ax.set_xlabel('longitude')
    ax.set_ylabel('latitude')
fig.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(12, 6)) pyosmium_fc.plot(ax=axes[0], markersize=2, color='#4457a4') axes[0].set_title( f'pyosmium (slim) - {len(pyosmium_fc)} buildings, {len(pyosmium_fc.columns)} cols' ) pyrosm_fc.plot(ax=axes[1], markersize=2, color='#c05a3c') axes[1].set_title( f'pyrosm (rich) - {len(pyrosm_fc)} buildings, {len(pyrosm_fc.columns)} cols' ) for ax in axes: ax.set_xlabel('longitude') ax.set_ylabel('latitude') fig.tight_layout() plt.show()

Takeaway¶

• The named-query catalog is introspectable: EarthLens.list_datasets('osm') and the Catalog rows (query_template / ohsome_filter / pyrosm_method).
• variables=[...] runs several queries in one download(), combined into a single FeatureCollection.
• file_format='gpkg' switches the written file to GeoPackage; max_bbox_deg2= tunes the live-query size guard.
• engine='pyosmium' (default, bounded memory) vs engine='pyrosm' (opt-in, richer schema) for bulk reads.
• Every result is ODbL — honour the attribution / share-alike obligation.