OpenStreetMap in bulk — Geofabrik .osm.pbf extracts (pbf)¶

This notebook teaches the pbf protocol of the earthlens.osm backend: how to read bulk / regional OpenStreetMap features — every building or road in a country — from a Geofabrik .osm.pbf extract, rather than the small, targeted live queries the overpass / ohsome protocols serve.

The backend downloads the regional extract once (cached on disk), reads a whole layer with pyrosm, and clips it to your bbox. Like the other OSM protocols it returns a pyramids FeatureCollection (a geopandas.GeoDataFrame subclass, CRS EPSG:4326) and emits an ODbL LicenseWarning — credit '© OpenStreetMap contributors' when you redistribute.

Setup¶

The pbf protocol needs the osm-pbf extra (pip install earthlens[osm-pbf] — pyrosm + osmium), which is kept out of [all] because it is heavy. EarthLens is the unified entry point; Catalog lists the available Geofabrik regions; matplotlib draws the maps.

In [ ]:

from pathlib import Path

import matplotlib.pyplot as plt

from earthlens.core import EarthLens
from earthlens.osm import Catalog

OUT_DIR = Path('osm_pbf_output')
OUT_DIR.mkdir(exist_ok=True)

from pathlib import Path import matplotlib.pyplot as plt from earthlens.core import EarthLens from earthlens.osm import Catalog OUT_DIR = Path('osm_pbf_output') OUT_DIR.mkdir(exist_ok=True)

Pick a region¶

A pbf:* query needs a region= — a Geofabrik region key (or a raw continent/region path). The catalog ships a handful of small keys; region_ids() lists them. We use Malta (~8.8 MB) so the download is quick.

In [ ]:

Catalog().region_ids()

Catalog().region_ids()

The area of interest¶

The request bbox clips the read. We zoom into the dense Valletta / Sliema core so the map is readable; omit lat_lim / lon_lim to read the whole extract instead.

In [ ]:

LAT_LIM = [35.88, 35.94]
LON_LIM = [14.48, 14.54]

LAT_LIM = [35.88, 35.94] LON_LIM = [14.48, 14.54]

Every building in the area¶

variables=['pbf:buildings'] maps to pyrosm's building reader. The first call downloads the Malta extract into the cross-run cache (osm_pbf/ under the shared earthlens cache directory); repeat calls reuse it.

In [ ]:

buildings = EarthLens(
    data_source='osm',
    variables=['pbf:buildings'],
    region='malta',
    lat_lim=LAT_LIM,
    lon_lim=LON_LIM,
    path=str(OUT_DIR),
).download()

len(buildings), type(buildings).__name__

buildings = EarthLens( data_source='osm', variables=['pbf:buildings'], region='malta', lat_lim=LAT_LIM, lon_lim=LON_LIM, path=str(OUT_DIR), ).download() len(buildings), type(buildings).__name__

Each row is one building footprint. The osm_id / osm_type columns identify it (pyrosm's native id is normalised to osm_id so it matches the other OSM protocols), alongside the OSM tags pyrosm parsed.

In [ ]:

cols = [
    c
    for c in ['osm_id', 'osm_type', 'building', 'name', 'geometry']
    if c in buildings.columns
]
buildings[cols].head()

cols = [ c for c in ['osm_id', 'osm_type', 'building', 'name', 'geometry'] if c in buildings.columns ] buildings[cols].head()

Map the footprints¶

A FeatureCollection is a GeoDataFrame, so .plot() maps it directly.

In [ ]:

ax = buildings.plot(figsize=(7, 7), color='#4457a4', edgecolor='white', linewidth=0.1)
ax.set_title('Building footprints — Valletta / Sliema (OSM via Geofabrik pbf)')
ax.set_xlabel('longitude')
ax.set_ylabel('latitude')
plt.show()

ax = buildings.plot(figsize=(7, 7), color='#4457a4', edgecolor='white', linewidth=0.1) ax.set_title('Building footprints — Valletta / Sliema (OSM via Geofabrik pbf)') ax.set_xlabel('longitude') ax.set_ylabel('latitude') plt.show()

The drivable road network¶

pbf:roads reads pyrosm's road network with network_type='driving' (from the catalog row). The cached Malta extract is reused — no second download.

In [ ]:

roads = EarthLens(
    data_source='osm',
    variables=['pbf:roads'],
    region='malta',
    lat_lim=LAT_LIM,
    lon_lim=LON_LIM,
    path=str(OUT_DIR),
).download()

len(roads), sorted(roads.geometry.geom_type.unique())

roads = EarthLens( data_source='osm', variables=['pbf:roads'], region='malta', lat_lim=LAT_LIM, lon_lim=LON_LIM, path=str(OUT_DIR), ).download() len(roads), sorted(roads.geometry.geom_type.unique())

In [ ]:

ax = roads.plot(figsize=(7, 7), color='#c05a3c', linewidth=0.6)
ax.set_title('Drivable roads — Valletta / Sliema (OSM via Geofabrik pbf)')
ax.set_xlabel('longitude')
ax.set_ylabel('latitude')
plt.show()

ax = roads.plot(figsize=(7, 7), color='#c05a3c', linewidth=0.6) ax.set_title('Drivable roads — Valletta / Sliema (OSM via Geofabrik pbf)') ax.set_xlabel('longitude') ax.set_ylabel('latitude') plt.show()

Engines — pyrosm (default) vs pyosmium¶

engine='pyrosm' (the default, used above) reads the whole extract in memory and gives the richest columns; it refuses a file over 4 GB. For a continent- or planet-scale extract too big to hold in memory, pass engine='pyosmium' to stream it with bounded memory — a coarser fallback (a slimmer osm_id / osm_type / geometry schema, one geometry kind per layer). Never load planet.osm with pyrosm.

Takeaway¶

• EarthLens('osm', variables=['pbf:<layer>'], region=…, lat_lim=…, lon_lim=…).download() reads a whole layer from a cached Geofabrik extract — the right tool for bulk asks that blow past Overpass's size limits.
• region= picks the extract (Catalog().region_ids(), or a raw continent/region path); the bbox clips the read; the extract is cached across runs.
• Use engine='pyosmium' for extracts too large for the in-memory pyrosm engine.
• Honour the ODbL attribution / share-alike obligation when you redistribute.