OpenStreetMap in bulk — Geofabrik .osm.pbf extracts (bulk)¶

This notebook teaches the bulk download type of the earthlens.osm backend: how to read bulk / regional OpenStreetMap features — every building or road in a country — from a Geofabrik .osm.pbf extract, rather than the small, targeted live queries the live / history download types serve.

The backend downloads the regional extract once (cached on disk), reads a whole layer with the default streaming pyosmium engine, and clips it to your bbox. Like the other OSM download types it returns a pyramids FeatureCollection (a geopandas.GeoDataFrame subclass, CRS EPSG:4326) and emits an ODbL LicenseWarning — credit '© OpenStreetMap contributors' when you redistribute.

Setup¶

The bulk type needs the osm extra (pip install earthlens[osm]), which ships the wheel-clean pyosmium engine (published on PyPI as osmium) and is part of [all]. The richer in-memory pyrosm engine is opt-in — pip install pyrosm (it builds the sdist-only cykhash from source) — and only needed when you pass engine='pyrosm'. EarthLens is the unified entry point; Catalog lists the available Geofabrik regions; matplotlib draws the maps.

In [ ]:

from pathlib import Path

import matplotlib.pyplot as plt

from earthlens.core import EarthLens
from earthlens.osm import Catalog

OUT_DIR = Path('osm_pbf_output')
OUT_DIR.mkdir(exist_ok=True)

from pathlib import Path import matplotlib.pyplot as plt from earthlens.core import EarthLens from earthlens.osm import Catalog OUT_DIR = Path('osm_pbf_output') OUT_DIR.mkdir(exist_ok=True)

Pick a region¶

A bulk:* query needs a region= — a Geofabrik region key (or a raw continent/region path). The catalog ships a handful of small keys; region_ids() lists them. We use Malta (~8.8 MB) so the download is quick.

In [ ]:

Catalog().region_ids()

Catalog().region_ids()

The area of interest¶

The request bbox clips the read. We zoom into the dense Valletta / Sliema core so the map is readable; omit lat_lim / lon_lim to read the whole extract instead.

In [ ]:

LAT_LIM = [35.88, 35.94]
LON_LIM = [14.48, 14.54]

LAT_LIM = [35.88, 35.94] LON_LIM = [14.48, 14.54]

Every building in the area¶

variables=['bulk:buildings'] reads the buildings layer (with the default pyosmium engine). The first call downloads the Malta extract into the cross-run cache (osm_pbf/ under the shared earthlens cache directory); repeat calls reuse it.

In [ ]:

buildings = EarthLens(
    data_source='osm',
    variables=['bulk:buildings'],
    region='malta',
    lat_lim=LAT_LIM,
    lon_lim=LON_LIM,
    path=OUT_DIR,
).download()

len(buildings), type(buildings).__name__

buildings = EarthLens( data_source='osm', variables=['bulk:buildings'], region='malta', lat_lim=LAT_LIM, lon_lim=LON_LIM, path=OUT_DIR, ).download() len(buildings), type(buildings).__name__

Each row is one building footprint. The osm_id / osm_type columns identify it (the native id is normalised to osm_id so it matches the other OSM download types). The default pyosmium engine returns this slim osm_id / osm_type / geometry schema; the opt-in pyrosm engine (engine='pyrosm') adds the parsed OSM tag columns (e.g. building, name).

In [ ]:

cols = [
    c
    for c in ['osm_id', 'osm_type', 'building', 'name', 'geometry']
    if c in buildings.columns
]
buildings[cols].head()

cols = [ c for c in ['osm_id', 'osm_type', 'building', 'name', 'geometry'] if c in buildings.columns ] buildings[cols].head()

Map the footprints¶

A FeatureCollection is a GeoDataFrame, so .plot() maps it directly.

In [ ]:

ax = buildings.plot(figsize=(7, 7), color='#4457a4', edgecolor='white', linewidth=0.1)
ax.set_title('Building footprints — Valletta / Sliema (OSM via Geofabrik pbf)')
ax.set_xlabel('longitude')
ax.set_ylabel('latitude')
plt.show()

ax = buildings.plot(figsize=(7, 7), color='#4457a4', edgecolor='white', linewidth=0.1) ax.set_title('Building footprints — Valletta / Sliema (OSM via Geofabrik pbf)') ax.set_xlabel('longitude') ax.set_ylabel('latitude') plt.show()

The drivable road network¶

bulk:roads reads the road network with network_type='driving' (from the catalog row). The cached Malta extract is reused — no second download. With the default pyosmium engine the driving filter is approximate (a highway-value allow-list); engine='pyrosm' reproduces pyrosm's exact drivable subset.

In [ ]:

roads = EarthLens(
    data_source='osm',
    variables=['bulk:roads'],
    region='malta',
    lat_lim=LAT_LIM,
    lon_lim=LON_LIM,
    path=OUT_DIR,
).download()

len(roads), sorted(roads.geometry.geom_type.unique())

roads = EarthLens( data_source='osm', variables=['bulk:roads'], region='malta', lat_lim=LAT_LIM, lon_lim=LON_LIM, path=OUT_DIR, ).download() len(roads), sorted(roads.geometry.geom_type.unique())

In [ ]:

ax = roads.plot(figsize=(7, 7), color='#c05a3c', linewidth=0.6)
ax.set_title('Drivable roads — Valletta / Sliema (OSM via Geofabrik pbf)')
ax.set_xlabel('longitude')
ax.set_ylabel('latitude')
plt.show()

ax = roads.plot(figsize=(7, 7), color='#c05a3c', linewidth=0.6) ax.set_title('Drivable roads — Valletta / Sliema (OSM via Geofabrik pbf)') ax.set_xlabel('longitude') ax.set_ylabel('latitude') plt.show()

The other bulk layers¶

Beside buildings and roads, the catalog maps four more Geofabrik layers: bulk:boundaries, bulk:landuse, bulk:natural, bulk:pois. They read from the same cached Malta extract — no second download — so sweeping all four is cheap. Each uses the default pyosmium engine.

In [ ]:

bulk_layers = ['bulk:boundaries', 'bulk:landuse', 'bulk:natural', 'bulk:pois']
bulk_results = {}
for layer in bulk_layers:
    bulk_results[layer] = EarthLens(
        data_source='osm',
        variables=[layer],
        region='malta',
        lat_lim=LAT_LIM,
        lon_lim=LON_LIM,
        path=OUT_DIR,
    ).download()

{layer: len(fc) for layer, fc in bulk_results.items()}

bulk_layers = ['bulk:boundaries', 'bulk:landuse', 'bulk:natural', 'bulk:pois'] bulk_results = {} for layer in bulk_layers: bulk_results[layer] = EarthLens( data_source='osm', variables=[layer], region='malta', lat_lim=LAT_LIM, lon_lim=LON_LIM, path=OUT_DIR, ).download() {layer: len(fc) for layer, fc in bulk_results.items()}

In [ ]:

fig, axes = plt.subplots(2, 2, figsize=(11, 10))
for ax, (layer, fc) in zip(axes.ravel(), bulk_results.items()):
    if len(fc):
        fc.plot(ax=ax, markersize=6)
    ax.set_title(f'{layer}  ({len(fc)} features)')
fig.tight_layout()
plt.show()

fig, axes = plt.subplots(2, 2, figsize=(11, 10)) for ax, (layer, fc) in zip(axes.ravel(), bulk_results.items()): if len(fc): fc.plot(ax=ax, markersize=6) ax.set_title(f'{layer} ({len(fc)} features)') fig.tight_layout() plt.show()

A region outside the catalog — a raw Geofabrik path¶

region= also accepts a raw continent/region path (any string with a /), so you are not limited to the curated keys. Here we read Monaco straight from its Geofabrik path; omitting lat_lim / lon_lim reads the whole (tiny) extract.

In [ ]:

monaco_buildings = EarthLens(
    data_source='osm',
    variables=['bulk:buildings'],
    region='europe/monaco',  # raw Geofabrik path, not a catalog key
    path=OUT_DIR,
).download()

len(monaco_buildings)

monaco_buildings = EarthLens( data_source='osm', variables=['bulk:buildings'], region='europe/monaco', # raw Geofabrik path, not a catalog key path=OUT_DIR, ).download() len(monaco_buildings)

Engines — pyosmium (default) vs pyrosm¶

engine='pyosmium' (the default, used above) streams the extract with bounded memory and ships with earthlens[osm], so it works out of the box and handles continent- or planet-scale extracts. It is the coarser reader: a slimmer osm_id / osm_type / geometry schema, one geometry kind per layer.

engine='pyrosm' reads the whole extract in memory and gives the richest, exact per-layer columns and mixed geometry; it refuses a file over 4 GB, so never load planet.osm with it. It is opt-in (pip install pyrosm); selecting it without pyrosm installed raises a clear ImportError naming the command.

Takeaway¶

• EarthLens('osm', variables=['bulk:<layer>'], region=…, lat_lim=…, lon_lim=…).download() reads a whole layer from a cached Geofabrik extract — the right tool for bulk asks that blow past Overpass's size limits.
• region= picks the extract (Catalog().region_ids(), or a raw continent/region path); the bbox clips the read; the extract is cached across runs.
• The default pyosmium engine streams with bounded memory and ships with [all]; switch to the opt-in engine='pyrosm' (pip install pyrosm) for the richest, exact per-layer output.
• Honour the ODbL attribution / share-alike obligation when you redistribute.