We investigate three different systems that use different ways to support spatial operations based on NoSQL key-value stores: GeoMesa, Pyro, MD-HBase Since some systems do not open source their codes or their codes cannot run because of some version/dependency issues, we mainly focus on the evaluation of GeoMesa.
The hadoop version we used is 2.7.7; HBase version is 1.4.11
- geomesa: a open-source spatial-temporal tools based on HBase
- official website: https://www.geomesa.org/
- we use its open-source code to test it
- it uses Z-curve as spatial index and use a top-to-bottom algorithm to generate scan range for spatial query. (https://www.geomesa.org/documentation/stable/tutorials/geohash-substrings.html)
Uri of full dataset: https://drive.google.com/file/d/1yQbDj6a-a8xOk9osF8mB3ZdVKZ0vSVTP/view?usp=sharing
- src/main/java: evaluation codes
- src/main/python: code of generation of query workloads
- src/main/resources/dataset: only contains a subset of dataset, the full data set can be found in the above google driver uri.
- src/main/resources/query: the generated query workloads used in this project
- src/main/resources/result: the running result of query workloads
- HBase: 1.4.11
- Java: JDK1.8
- Maven
- Python: Python3, Numpy
- GeoMesa: 2.2.1
- Scala: 2.11
After installing all dependencies:
- The workloads used in the evaluation have been generated and is located at the src/main/resources/query
- (Optional) generate it by yourself, use the command in the root of project directory
python3 src/main/python/SyntheticWorkload.py // it needs 1~2 mintues because it needs to read pickup data file to generate data-related queries.
- Start HBase
cd /home/rogerguo/Tools/hbase-1.4.11/bin
./start-hbase.sh
- Insert data, it is handled by HBaseWriteClient.java (src/main/java/com/rogerguo/spatial/geomesa/client/HBaseWriteClient.java), you can specify input data file path in this code, and then run following commands in the root of project directory
mvn compile // it needs several mintues when run at the first time (maven needs to download all denpendencies at the first time)
mvn exec:java -Dexec.mainClass=com.rogerguo.spatial.geomesa.client.HBaseWriteClient -Dexec.args="--hbase.zookeepers 127.0.0.1 --hbase.catalog geomesa_csci5570_test" // it needs about 4~5 mintues to insert data
- Run range query workload, it is handled by HBaseClient.java, you can specify path of query workload and the path of output result, and then run following commands in the root of project directory, the response time of each query will be written in the output file.
mvn compile
mvn exec:java -Dexec.mainClass=com.rogerguo.spatial.geomesa.client.HBaseClient -Dexec.args="--hbase.zookeepers 127.0.0.1 --hbase.catalog geomesa_csci5570_test"
- Run k-NN query
mvn compile
mvn exec:java -Dexec.mainClass=com.rogerguo.spatial.geomesa.client.KNNQueryClient -Dexec.args="--hbase.zookeepers 127.0.0.1 --hbase.catalog geomesa_csci5570_test"
We have installed all dependencies, generated query workload and inserted data in a virtual machine, so you can directly run queries using above commands in the root of project directory after starting HBase.