Comments (4)
Hi @tiwanpunit ,
Thanks for asking this interesting question. In fact, we have done some research on this.
First, I am not sure whether you are using the proper format to do iterative range query. Here is the proper format to do iterative range query with cache. (The query envelope can be replaced by JTS Polygon)
PointRDD objectRDD = new PointRDD(sc, inputLocation, offset, splitter);
objectRDD.rawPointRDD.persist(StorageLevel.MEMORY_ONLY());
for(int i=0;i<loopTimes;i++)
{
long resultSize = RangeQuery.SpatialRangeQuery(objectRDD, queryEnvelope, 0).getRawRectangleRDD().count();
assert resultSize>-1;
}
PointRDD objectRDD = new PointRDD(sc, inputLocation, offset, splitter);
objectRDD.buildIndex("rtree");
for(int i=0;i<loopTimes;i++)
{
long resultSize = RangeQuery.SpatialRangeQueryUsingIndex(objectRDD, queryEnvelope, 0).getRawRectangleRDD().count();
assert resultSize>-1;
}
Second, I have tested GeoSpark range query with cache on super large data. The benefit of using spatial index in range query is not very obvious, it takes some additional time to build index in the first round execution.
Moreover, range query is actual "Map" operation. This is called Narrow Dependency in Spark and a bunch of same range queries can be pipelined together and executed without a stop. The pipeline execution is super fast and using index is just a little bit faster than it. You may not even notice it.
However, if your cluster computation power is quite limited, I think using index may still speed up your execution a lot.
Thanks,
Jia
from sedona.
Thank you for replying so fast.
So in my usecase "objectRDD" is the "point" and "queryEnvelope" is the "polygon" ? i.e. objectRDD contains the points that need's to be queried and queryEnvelope is the shape on which query is performed.
Did I understand your explanation correctly?
Also you have suggested to pass JTS Polygon in place of "queryEnvelope" but the method only accepts "Envelope" type of shape not Polygon.
from sedona.
- Yes. You understand me correctly.
- Yes. You are right. Thanks for reminding me. Currently, it only supports Envelope type as the query window. I can add the support of polygon type query window in the upcoming version. It is very easy.
For now, it seems the only way to perform a polygon window range query is to do a join query on PolygonRDD(only 1 polygon inside) and PointRDD.
from sedona.
Soloved in GeoSpark 0.4.0
from sedona.
Related Issues (20)
- Unable to use sedona.global.charset in ShapefileReader HOT 4
- Add `markdown-link-check` with `pre-commit` HOT 2
- Add documentation on `pre-commit` usage
- We should add `ruff-pre-commit` a `pre-commit` hook for `Ruff` for Python linting and formatting HOT 1
- Add some `pre-commit` hooks that target the `R` language
- Apache Sedona on Fabric, does not read parquet from Lakehouse HOT 7
- Images should have alternate text (alt text)
- Specify custom transformation parameters/wkt string from CoordinateSystem A til CoordinateSystem B. HOT 3
- AWS Glue official tutorial HOT 5
- Hidden requirement for geopandas in apache-sedona[spark] 1.5.2 HOT 4
- after latest update of sedona snow on snowflake side all functions are gone HOT 2
- datatype geometry is not supported when 'create table xxx (geom geometry)' HOT 3
- AttributeError: 'sedona' has no attribute 'read' HOT 2
- St_isempty(geometry) finds non null geometries but does not find null geometries. HOT 2
- ST_Snap example code does not work HOT 2
- Flink Sedona,geomTbl.execute().print() happen error: HOT 2
- Sedona fails to write Delta Lake on Databricks 15.3 Beta: ClassCastException HOT 7
- ST_IsPolygonCW, ST_IsPolygonCCW, ST_ForcePolygonCW and ST_ForcePolygonCCW fails on Polygons without interior ring
- Breaking change between 1.5.3 and 1.6.0 affecting RASTER functions java.lang.NoSuchMethodError: void org.geotools.coverage.grid.GridGeometry2D HOT 5
- There was garbled code when reading Chinese from shp file HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sedona.