Performance of Range Query about sedona

[jiayuasu]

Hi @tiwanpunit ,

Thanks for asking this interesting question. In fact, we have done some research on this.

First, I am not sure whether you are using the proper format to do iterative range query. Here is the proper format to do iterative range query with cache. (The query envelope can be replaced by JTS Polygon)

    	PointRDD objectRDD = new PointRDD(sc, inputLocation, offset, splitter);
    	objectRDD.rawPointRDD.persist(StorageLevel.MEMORY_ONLY());
    	for(int i=0;i<loopTimes;i++)
    	{
    		long resultSize = RangeQuery.SpatialRangeQuery(objectRDD, queryEnvelope, 0).getRawRectangleRDD().count();
    		assert resultSize>-1;
    	}

PointRDD objectRDD = new PointRDD(sc, inputLocation, offset, splitter);
    	objectRDD.buildIndex("rtree");
    	for(int i=0;i<loopTimes;i++)
    	{
    		long resultSize = RangeQuery.SpatialRangeQueryUsingIndex(objectRDD, queryEnvelope, 0).getRawRectangleRDD().count();
    		assert resultSize>-1;
    	}

Second, I have tested GeoSpark range query with cache on super large data. The benefit of using spatial index in range query is not very obvious, it takes some additional time to build index in the first round execution.

Moreover, range query is actual "Map" operation. This is called Narrow Dependency in Spark and a bunch of same range queries can be pipelined together and executed without a stop. The pipeline execution is super fast and using index is just a little bit faster than it. You may not even notice it.

However, if your cluster computation power is quite limited, I think using index may still speed up your execution a lot.

Thanks,
Jia

[tiwanpunit]

Thank you for replying so fast.
So in my usecase "objectRDD" is the "point" and "queryEnvelope" is the "polygon" ? i.e. objectRDD contains the points that need's to be queried and queryEnvelope is the shape on which query is performed.
Did I understand your explanation correctly?

Also you have suggested to pass JTS Polygon in place of "queryEnvelope" but the method only accepts "Envelope" type of shape not Polygon.

[jiayuasu]

1. Yes. You understand me correctly.
2. Yes. You are right. Thanks for reminding me. Currently, it only supports Envelope type as the query window. I can add the support of polygon type query window in the upcoming version. It is very easy.

For now, it seems the only way to perform a polygon window range query is to do a join query on PolygonRDD(only 1 polygon inside) and PointRDD.

[jiayuasu]

Soloved in GeoSpark 0.4.0