Skip to content

Parallel windowed reads#

For I/O-bound workloads — many windows of a large or remote raster — reading windows concurrently is a quick win. GDAL releases the GIL while it does I/O, so threads genuinely overlap. pyramids exposes this two ways: the low-level read_array(threadsafe=True) primitive, and the read_windows convenience that fans a list of windows across a thread pool for you.

read_windows — the one-liner#

from pyramids.dataset import Dataset, Window

ds = Dataset.read_file("big.tif")            # must be path-backed (disk or /vsimem/)
windows = list(ds.block_windows())           # or any list of Window objects
blocks = ds.read_windows(windows, threads=8) # one ndarray per window, in input order

read_windows submits each window to a concurrent.futures.ThreadPoolExecutor, reading every window through a per-thread GDAL handle (read_array(threadsafe=True)), and returns the results in the same order as the input windows. threads=1 is exactly the sequential path.

How the threading model works#

A single gdal.Dataset handle is not safe for concurrent calls. pyramids therefore opens one read-only handle per thread, keyed by the dataset's path (the ThreadLocalFileManager behind read_array(threadsafe=True)). That is why:

  • The dataset must be reopenable — path-backed on disk or under /vsimem/. A pure in-memory (MEM) dataset has no path to reopen per thread and is rejected.
  • Threads scale with I/O, not CPU. For CPU-bound post-processing, move the heavy work into the worker or use the Dask path (read_array(chunks=...)) instead.

Credentials and the pool#

The pool threads do not inherit thread-local GDAL config, which is exactly where credentials used to sit — so a remote read could fan out and authenticate with nothing. read_windows now scopes a CloudConfig onto the bucket it is reading for the duration of the call, which the pool threads can see:

from pyramids.base.remote import CloudConfig

with CloudConfig(extra={"AWS_S3_ENDPOINT": "minio.example",
                        "AWS_ACCESS_KEY_ID": "AK"}):
    blocks = ds.read_windows(windows, threads=8)

No path= is needed here: the read knows its own source. The same applies to a dataset opened from an object store inside such a block — it carries the credentials into its later reads, including these.

A bucket-scoped option is invisible to gdal.GetConfigOption; use gdal.GetPathSpecificOption(path, key) to inspect one.

With a custom endpoint#

One extra thing happens for a non-AWS store (CDSE, MinIO, Ceph). GDAL discovers a bucket's region through CPLGetConfigOption, which takes no path and so can never read the per-bucket table. For the pool threads to address the right host, the routing subset — AWS_S3_ENDPOINT, AWS_ENDPOINT_URL, AWS_REGION, AWS_DEFAULT_REGION, AWS_VIRTUAL_HOSTING and AWS_HTTPS — is published process-globally for the duration of the read, then restored.

Credentials are never published that way; they stay per-bucket. But while that read runs, another store that sets no endpoint of its own will inherit this one, on any thread. Two overlapping reads of different custom-endpoint stores share a single slot per key, so the later one wins for both until it finishes.

The COG tutorial has the full table of which option lands in which storage, who can read it, and for how long.

Sanity-check the speedup yourself#

This is illustrative, not a CI assertion — wall-clock numbers depend on the storage and the raster:

import time
from pyramids.dataset import Dataset

ds = Dataset.read_file("big.tif")
windows = list(ds.block_windows())

t0 = time.perf_counter()
_ = [ds.read_array(window=w) for w in windows]          # sequential
seq = time.perf_counter() - t0

t0 = time.perf_counter()
_ = ds.read_windows(windows, threads=8)                  # parallel
par = time.perf_counter() - t0

print(f"sequential {seq:.3f}s  parallel {par:.3f}s  speedup {seq / par:.1f}x")

On a large COG over /vsicurl/ the parallel path is typically several times faster; on a small local file the thread overhead can make them comparable — measure on your data.

See also#

  • Dataset.read_windows — the API reference.
  • read_array(chunks=...) — the lazy / Dask path for compute-heavy pipelines.