geoarrow / geoarrow-rs

GeoArrow in Rust, Python, and JavaScript (WebAssembly) with vectorized geometry operations

geoarrow-rs

A Rust implementation of the GeoArrow specification and bindings to GeoRust algorithms for efficient spatial operations on GeoArrow memory.

This repository also includes Python bindings and JavaScript (WebAssembly) bindings, wrapping the GeoArrow memory layout and offering vectorized geometry operations.

Documentation

• Use from Rust
• Use from Python
• Use from JavaScript
• Create your own Rust-JavaScript library with wasm-bindgen

Examples

• Rust examples

Install

Add this to your Cargo.toml:

geoarrow = "0.1"

References

• Prototyping GeoRust + GeoArrow in WebAssembly gives an early preview of the JavaScript API.
• GeoArrow and GeoParquet in deck.gl gives an overview of what GeoArrow's memory layout looks like under the hood, even though it's focused on how to render the data on a map.
• Thoughts on GEOS in WebAssembly introduces why I think GeoRust + GeoArrow on the web has significant potential.
• Zero-copy Apache Arrow with WebAssembly explains how the JavaScript bindings are able to move memory between JavaScript and WebAssembly so efficiently.

geoarrow-rs's Issues

check slicing

check that slicing is accurate. we should not be slicing the underlying arrays? when should we be slicing arrays vs slicing offsets?

JS: How to handle geometry overloading

E.g. for contains, I want to support contains between each geometry type to each other geometry type. In Rust, I can do that via traits, and so contains() will work on each combination of types automatically. But in JS it doesn't have overloading, so I need to do something like adding a GeometryArray class that can be any geometry type?

Finish implementing algorithm traits on both arrays and scalars

Continuation of #98. Ideally the Broadcastable* should never be called from the Rust code

typedoc plugins

• https://www.npmjs.com/package/typedoc-plugin-versions
• https://github.com/Gerrit0/typedoc-plugin-mdn-links

RectArray follow ups

• Add docstring. Be explicit that this is not included in the geoarrow specification.
• Add impl From<RectArray> for PolygonArray

Documentation on how to use `geoarrow-wasm` from a standalone rust crate

I.e. it should be possible for me to publish geoarrow-wasm to crates.io and give an example of using that with a custom operation.

pub use geoarrow_wasm::*;

• Use no-default-features on the geoarrow_wasm dependency... only bring in the core geoarrow data structures and bindings, without geo algorithms.
• Example of some custom operation on the data arrays

JS: Mutations of an affine transform

I.e. we should let people do

const transform = AffineTransform.identity().scale(1).skew(2); // etc

Then let people pass that into array.affine_transform().

Ideally these affine mutations should be vectorized as well.

Probably these should be by value rather than by reference so that you don't leak memory? And then that also allows you to mutate the array of affine transforms in place.

Call `shrink_to_fit` when converting mutable arrays into immutable ones

Mutable arrays may have had reserve called, which reserves extra space in the vec. We should probably call Vec::shrink_to_fit before coercing into the immutable representation. Not sure if this is already getting called by arrow2 under the hood.

Update for latest geometry access traits

Finish moving algorithms to traits

Write DEVELOP.md

Add developer and architecture information

Remove WKB from GeometryArray enum

Implement algorithm traits on geo scalars as well as geoarrow scalars

That'll lead to a really seamless API, and it'll be future compatible because both geo scalars and arrow scalars implement the same geometry access traits

Support casting between i32 and i64 offset arrays

Some From implementations are needed

Move rust code into rust/ folder

GeometryArrayTrait::is_sliced

Useful to know whether the array occupies the full underlying buffer or not/whether it's shared

Wrap `TriangulateEarcut` trait.

Earcut functionality would be useful to support.

• Does a TriangleArray make sense? It wouldn't be exactly aligned with (today's) formal geoarrow types, but it would be really high performance. A TriangleArray could naively be defined as a FixedSizeList[3]. We could have a to_polygon_array method on the TriangleArray struct, which should be high performance because we know the size of the output coordinate array and output offsets array at the start.
• earcut_triangles_raw is quite useful because the triangle indices are already in a flat array.
  • Is the order of vertices in the result guaranteed to be the same as the original vertices? They are in the example but you should look/ask to see whether this is always the case. If it is, then you can reuse the input coordinate array and just shift the indices of the output indices as needed to make it accurate in the full array.

Implement correct array/scalar equality

Right now I'm adding a derived PartialEq, which just checks that the constituent parts are the same, but we need to check that the equality on sliced arrays works as expected. If it just checks equality on the underlying buffers, then array equality will not be true whenever comparing a sliced array to a non-sliced array.

GeometryArrayTrait::value() should return an option

Should do a bounds check

Fix skipped tests

Several tests have

#[ignore = "This is failing on coordinate access"]

Good first issues

What is the correct implementation of an invalid point?

I.e. is a point with coordinates NaN, NaN and the validity bitmask set to true allowed? I.e. a point that exists but is unknown? This is subtly different than the bitmask set to false, which says the point is missing. WKT allows Point Empty I believe

This is probably something to ask on the geoarrow spec repo.

rename geoarrow crate to `geoarrow-wasm`

Don't implement GeometryArrayTrait for Coord arrays?

Validate broadcasting

add a function to validate that the broadcasted array has the same length as the other array it's being zipped onto

Clone-on-write for affine transformations

Since affine transformations don't change the size of the arrays, they're great candidates for COW

Add geospatial category to cargo.toml

georust/geo#974

Implement conversion from GEOS geometries to GeoArrow arrays

Finish up on #108

information on coord layout

• what is the current coord format
• convert from one coord format to the other

Consider using georust docker images in ci

https://github.com/georust/docker-images

This might be more useful once we have proj integration

Geozero: Return errors for invalid geometry type

e.g. make sure we're always erroring when trying to add a polygon into a point array. You should add tests for this as well

Finish geozero integration

CI tests for JS code

Algorithms on scalars should convert to geo only once

This converts to geo on every iteration of the loop instead of only once
https://github.com/kylebarron/geoarrow-rs/blob/09c5fc45456f48140a5dd3f482c5afa9a83ec99c/src/algorithm/geo/euclidean_distance.rs#L194

This is a good opportunity to set up benchmarks

geoarrow-wasm feature flags?

How many JS feature flags should there be?

Ideally it should be possible to take out most features to make geoarrow-wasm have a really really tiny bundle size. But that comes at a cost of build complexity.

WKB: fix parsing empty polygons

the code currently crashes on empty polygons because the trait requires that an exterior exists (ref: georust/geo#1039)

Generic over i32/i64 offsets

This would make it easier to use arrow js ffi in the browser where presumably we'll never have a need for i64 offsets

Tests with empty geometries

The format specifies that empty geometries (which are not the same as missing!) have an empty inner array.

We need to make sure that this works with the layout I have, and we should also make sure that is_empty is accurate for this!

Also an empty point is defined with NaN values, so we should validate that (and write a custom PartialEq probably)

Move wkb parsing code into top level io/ folder

Integration with arrow-rs

add an optional feature to convert arrays into arrow-rs

JS: how to handle scalars?

E.g. in algorithms like contains and euclidean_distance where you want to broadcast against a single scalar

Python CI

Create geoarrow arrays from iterators of anything that implements geometry access traits?

It would be really cool if the wkb types could implement geometry access traits directly, and the From implementation to create arrow arrays could be exactly the same as what it is from geo objects

Explore type-preserving operations

Right now we have operations like

pub fn area(array: GeometryArray) -> Result<PrimitiveArray<f64>> {}

We then need a verbose match on the array type. But when the area function is implemented on every geometry type, there should be some way to do

pub fn area<'a, T: GeometryArrayTrait<'a>>(array: T) -> Result<PrimitiveArray<f64>> {}

Then we wouldn't need a match statement with a lot of copying between match arms. The downsides there are:

• returning iterators from traits is not yet stable I think?
• We'd have to have to return a geo::Geometry I think, not a different geometry type per trait implementation? Or always have to convert the local geometry type into geo::Geometry?

Optional rayon integration?

Optimize affine transformations

Right now I'm implementing the simplest way, which is to go through geo objects, but I think it should be possible to get some speedups by using arrow arrays as much as possible. for example, Scale::scale() just takes the center of every geometry and then applies affine_transform. It might be more efficient if we first do center(geometry_array) and then pass that to affine_transform directly.

Especially if/once we implement COW apis for affine transforms

Owned scalar types

I.e. a Point that isn't a reference onto another array.

This could even be implemented as

pub struct OwnedPoint(PointArray)

where the invariant is that the underlying array only has one item

Coordinate/Coordinate Array trait to support XYZM

Should presumably also have a perf boost by eliminating lots of match statements

Implement `From` for arrow2 arrays to either i32 or i64 geometry arrays

Infer geometry type from wkb array