Creating the next generation of geospatial data tools for Python & the browser with GeoArrow, GeoParquet & GeoRust @developmentseed | he/him 🌈
Obstore v0.7 is out! The simplest Python cloud storage interface, powered by Rust
- Anonymous connections to GCS
- Obspec compatibility
- Improvements to NASA Earthdata credential provider
- Fixed creation of AzureStore from HTTPS URL
- Improved documentation
https://developmentseed.org/obstore/latest/blog/2025/06/25/releasing-obstore-07/
Introducing Obspec: A Python protocol for interfacing with object storage
Obspec is an alternative to fsspec. It defines a minimal Python interface to read, write, and modify data on object storage while acknowledging that object storage is *not* a filesystem.
Obspec formalizes the Obstore API. You can build around Obspec and be generic around backend implementation used at runtime. It also provides an avenue for caching to be built as a wrapper on top of Obspec.
@josi No, you need to read the post 😛. The geoarrow crate is deprecated in favor of the smaller namespace crates
Published version 0.4 of the GeoArrow Rust library! It's been fully rewritten from scratch, has a much more stable core, and supports all of the GeoArrow 0.2 specification. https://geoarrow.org/geoarrow-rs/rust/
@lnicola Haha I resisted for a while because I wanted to nudge people to PROJJSON. But alas if you want to read Shapefile or FlatGeobuf etc and you don't have access to PROJ, you have no way to propagate the information you do have access to.
Really excited for version 0.2 of GeoArrow! This has been years in the making and adds:
- Geometry, GeometryCollection, Box types. We can now represent mixed-type geometries in an array
- New ways to represent CRS.
See the release post for more information: https://geoarrow.org/news/release-0-2.html
https://mapstodon.space/@paleolimbot@fosstodon.org/114580749340718371
We’re chuffed to announce the 0.2 release of the GeoArrow specification! We released the first version in September, 2023 and have since worked with the community towards our vision that getting spatial data from one library to another should be fast, easy, and preserve important spatial properties like the CRS.
(Geo)Arrow + Rust :rust_ferris: + extendr :rstats: + PyO3 :python: June 4th!
What I think the future of scientific computing looks like in Rust—FREE!!
Abstract 👇
https://scientificcomputing.rs/2025/talks/parry.html
#rust #data #rstats #python
This benchmarking is done by Max Jones in https://github.com/maxrjones/zarr-obstore-performance
With today's Zarr Python release, switch to the new obstore-based backend for **3x faster** reading from S3, Google Cloud Storage, and Azure than the default fsspec backend.
https://zarr.readthedocs.io/en/stable/user-guide/storage.html#object-store
https://developmentseed.org/obstore/latest/
@josi @JensWiesehahn @djh Hilbert curves and z order curves are both instances of space filling curves, but the Hilbert curves gives better locality (it doesn't have those "jumps" at the end of the z), and most geospatial applications care about locality, so I don't think the z order curve is used often
@jgomezdans I don't know Microsoft's plans but I thought existing datasets were still getting updated
@mdsumner @josi @hughagraham @atsyplenkov
async-tiff is fully general and supports not just COG, but GeoTIFF and arbitrary TIFF files as well. Maybe the layer on top of async-tiff that presents a rasterio-like API should be named rusty-cogs
Obstore v0.6 brings simple, painless integration with Microsoft Planetary Computer!
- *Automatic token refresh*. No need to manually fetch SAS tokens and never think about token expiration times again.
- STAC API integration
https://developmentseed.org/obstore/latest/blog/2025/03/24/releasing-obstore-06/
Published v0.1 of async-tiff, a Rust crate and Python library for low-level, async reading of TIFF and GeoTIFF files.
There's high potential for improved performance when opening many TIFF files concurrently, especially in Python server environments.
Published obstore 0.5, the fastest Python interface to S3, GCS & Azure storage.
- Updated benchmarks: **2.8x higher throughput** than aioboto3 for many small, concurrent requests
- Custom credential providers, automatic token refresh
- Improved docs, type hints, fsspec integration
https://developmentseed.org/obstore/latest/blog/2025/03/17/releasing-obstore-05/
New release of obstore: the simplest, highest-throughput Python interface to S3, GCS & Azure Storage!
- Pickle support, to unblock usage in Dask and Zarr
- Easier one-line store creation
- Open remote objects as file-like readers or writers
- Improved docs
https://developmentseed.org/obstore/latest/blog/2025/02/10/releasing-obstore-04/
New obstore v0.3 for easy access to S3, GCS, and Azure Storage!
- Easy **streaming** downloads, uploads, and copies across object stores via async iterators
- Streaming list, with no pagination
- Easier configuration with improved type hints
- Fsspec integration
Spatial indexes, and RTrees specifically, are at the core of geospatial software.
But ever wondered how an RTree is actually implemented?
In this post we’ll dive into the implementation of Flatbush, an elegant, blazing-fast, memory-efficient RTree.
Releasing geo-index 0.2: Fast, memory-efficient, zero-copy spatial indexes (RTree & KDTree) for Python & Rust.
- Range, nearest-neighbor queries
- Persist indexes as bytes, reuse them later. Query operations work on external memory.
- Supports float32 for 2x memory savings
- Enables GeoParquet spatial partitioning (Python APIs to infer the groups for output partitions that are spatially grouped)
- Multi-tree join (find overlapping items in two RTrees)
