amazon-archives / dynamodb-geo Goto Github PK

License: Apache License 2.0

Java 100.00%

dynamodb-geo's Introduction

Geo Library for Amazon DynamoDB

The Geo Library for Amazon DynamoDB enables Java developers to easily create and query geospatial data. The library takes care of managing the geohash indexes required for fast and efficient execution of location-based queries over a table of items representing points of interest - latitude/longitude pairs.

Along with this library we provide sample applications demonstrating usage of the library for a cloud-backed mobile app development scenario. You can get up and running quickly with a sample AWS ElasticBeanstalk application and a sample iOS project.

API Reference

Geo Library for Amazon DynamoDB Javadoc

Features

Box Queries: Return all of the items that fall within a pair of geo points that define a rectangle as projected onto a sphere.
Radius Queries: Return all of the items that are within a given radius of a geo point.
Basic CRUD Operations: Create, retrieve, update, and delete geospatial data items.
Easy Integration: Adds functionality to the AWS SDK for Java in your server application.
Customizable: Access to raw request and result objects from the AWS SDK for Java.

Getting Started

Setup Environment

Sign up for AWS - Before you begin, you need an AWS account. Please see the AWS Account and Credentials section of the developer guide for information about how to create an AWS account and retrieve your AWS credentials.
Minimum Java requirements - To run the SDK you will need Java 1.6+. We strongly recommend Java 6 Update 24 at the minimum to mitigate the parseDouble DoS attacks. For more information about the requirements and optimum settings for the SDK, please see the Java Development Environment section of the developer guide.
Minimum iOS requirements - The sample iOS app supports iOS 5 and above.
Download Geo Library for Amazon DynamoDB - To download the code from GitHub, simply clone the repository by typing: git clone https://github.com/awslabs/dynamodb-geo.

Run the Sample Server App

Go to AWS Management Console and select ElasticBeanstalk.
Click Create a New Application.
Specify Application name and click Create.
Make sure Launch a new environment running this application is selected.
Select Tomcat for Predefined configuration. Environment type can be either Loadbalancing, autoscaling or Single instance. Then click Continue.
Select Upload your own for Source. Click Browse and select /samples/dynamodb-geo-server/dynamodb-geo-server.war. Then click Continue.
Choose your Environment name and Environment URL. Then click Continue.
Make sure Create an RDS DB Instance with this environment is NOT checked. Then click Continue.
Accept the default values for Configuration Details and click Continue.
Review the information and click Create.
Wait until Health becomes Green. This may take a few minutes.
Select Configuration > Software Configuration.
Under Environment Properties, type your AWS_ACCESS_KEY_ID and AWS_SECRET_KEY.
Type your DynamoDB table name for PARAM1 (e.g. geo-test). Make sure you do NOT already have a table with the same name.
Type your DynamoDB region name for PARAM2 (e.g. us-west-2). For the full list of regions, please read Regions and Endpoints.
Click Save.
Click on Dashboard and wait until Health becomes Green. This may take a few minutes.
Open your URL (e.g. http://YOUR-ENDPOINT.elasticbeanstalk.com). NOTE: For security reasons, we strongly encourage developers to use https endpoints on ElasticBeanstalk. Please read Configuring HTTPS for your AWS Elastic Beanstalk Environment for more details.
The status on the website will change as follows: preparing to start > creating a table > inserting test data into the table > running.
Wait until the website says Congratulations! Geo Library for Amazon DynamoDB Sample Server is running. This may take 10 to 20 minutes.

Run the Sample iOS App

Open the Xcode project under samples/dynamodb-geo-ios/dynamodb-geo-ios.xcodeproj.
Open AWSConstants.m.
Update YOUR-ENVIRONMENT part of http://YOUR-ENVIRONMENT.elasticbeanstalk.com/dynamodb-geo with your actual environment URL.
Click Run on Xcode to run the sample iOS app.

Building From Source

Once you check out the code from GitHub, you can build it using Maven: mvn package

Limitations

Java focused

Internally, Geo Library uses the S2 Geometry Library for spherical math, and the library is available only in Java and C++. For now, we are focusing on Java, and we don't have short term plans to port Geo Library for other languages.

No composite key support

Currently, the library does not support composite keys. You may want to add tags such as restaurant, bar, and coffee shop, and search locations of a specific category; however, it is currently not possible. You need to create a table for each tag and store the items separately.

Queries retrieve all paginated data

Although low level DynamoDB Query requests return paginated results, this library automatically pages through the entire result set. When querying a large area with many points, a lot of Read Capacity Units may be consumed.

High memory consumption

Because all paginated Query results are loaded into memory and processed, it may consume substantial amounts of memory for large datasets.

The server is essential

Because Geo Library calls multiple DynamoDB Query requests and processes the results in memory, it is not suitable for mobile device use. You should maintain a Java server, and use the library on the server.

Dataset density limitation

The Geohash used in this library is roughly centimeter precision. Therefore, the library is not suitable if your dataset has much higher density.

Reference

Amazon DynamoDB

AWS ElasticBeanstalk

Mobile Development

Java Development

dynamodb-geo's People

Contributors

Stargazers

Watchers

dynamodb-geo's Issues

Rectangle that intersects with four children doesn't mean it covers all its children

This snippet of code is problematic:

if (children.size() == 1 || children.size() == 2) {
            for (S2CellId child : children) {
                if (child.isLeaf()) {
                    cellIds.add(child);
                } else {
                    queue.add(child);
                }
            }
        } else if (children.size() == 3) {
            cellIds.addAll(children);
        } else if (children.size() == 4) {
            cellIds.add(parent);
        } else {
            assert false; // This should not happen.
        }

Image the situation that the rectangle is located near the center of the parent but isn't large enough to cover it, then it can still intersect with all its children.

The right way to do is to first check whether the rectangle contains the parent, and add all its intersected children to the queue.

Example bounds: (46.95913069858797, -123.74038696289062, 48.055558949569566, -121.27670288085938). Queries also returns points in Spokane.

Why enforcing consistent reads?

Would be great to have an option to enable/disable consistent reads when triggering the queries.

Not a bug but it is for sure an improvement specially if you are searching a big area and want to save some read units. We can soon open a PR with this... I hope you consider.

Find the nearest item with given geo point

Is there a way to query the nearest item stored in dyanmoDB with a given geo point? Let's assume that we have 50k items of interests stored in dynamoDB and i need to find the closet one with the given geo point. What is the best way to accomplish this?

Consistent reads are not supported on global secondary indexes

Need some doc on how a table in dynamodb should be created. I'm able to save geopoints to my table, but when i do a queryRadius I get an error that the geo-hash index is missing. When i create the index i get this error:

Caused by: java.util.concurrent.ExecutionException: com.amazonaws.AmazonServiceException: Consistent reads are not supported on global secondary indexes (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: DM3EDOUN6B0783R7FQBPJBG1U3VV4KQNSO5AEMVJF66Q9ASUAAJG)
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:188)
    at com.amazonaws.geo.GeoDataManager.dispatchQueries(GeoDataManager.java:343)

I must be missing something in the docs on how to create the table in dynamodb.

Maven Repo

Does this project exist in a public maven repo somewhere? If not, could it be added to one?

Old Jackson versions break Android Studio.

Importing the dependencies of Jackson Core 1.x.x and Jackson Mapper 1.x.x to an Android Studio project causes a "duplicate file" error. And it cannot be built.

Would this be fixed by updating the dynamodb-geo.jar to reference to the latest 2.x.x. Jackson versions?

Here is a link to a related SO post on the topic I made.
http://stackoverflow.com/questions/29886227/java-lang-noclassdeffounderror-when-already-imported-jar-files-to-android-studio

What's the default cell width, is it possible to config this value?

Is there a version that supports AWS C++ SDK

Any Pointers or If there is enough interest to create this capability in other SDKs ?

Throwing errors in case of data is >= 1 Mb

Hi,
I have approx. 2000 records at a single area but when i increase the radius it throws exception. Can some one tell me how to paginate these results and recompile source if change in library.

Error :
24-Oct-2016 22:55:31.195 INFO [http-nio-8080-exec-1] org.apache.catalina.core.ApplicationContext.log GeoDynamoDBServlet: com.amazonaws.AmazonClientException: Querying Amazon DynamoDB failed.
at com.amazonaws.geo.GeoDataManager.dispatchQueries(GeoDataManager.java:348)
at com.amazonaws.geo.GeoDataManager.queryRadius(GeoDataManager.java:225)
at com.amazonaws.geo.server.GeoDynamoDBServlet.queryRadius(GeoDynamoDBServlet.java:347)
at com.amazonaws.geo.server.GeoDynamoDBServlet.doPost(GeoDynamoDBServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1100)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:687)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: AmazonServiceException: Status Code: 400, AWS Service: AmazonDynamoDBv2, AWS Request ID: HF7JTN010QMENVE28OTL656ASVVV4KQNSO5AEMVJF66Q9ASUAAJG, AWS Error Code: ValidationException, AWS Error Message: The provided starting key is invalid
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.amazonaws.geo.GeoDataManager.dispatchQueries(GeoDataManager.java:343)
... 26 more

Caused by: AmazonServiceException: Status Code: 400, AWS Service: AmazonDynamoDBv2, AWS Request ID: HF7JTN010QMENVE28OTL656ASVVV4KQNSO5AEMVJF66Q9ASUAAJG, AWS Error Code: ValidationException, AWS Error Message: The provided starting key is invalid

at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:679)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:350)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:202)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1217)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.query(AmazonDynamoDBClient.java:834)
at com.amazonaws.geo.dynamodb.internal.DynamoDBManager.queryGeohash(DynamoDBManager.java:95)
at com.amazonaws.geo.GeoDataManager$GeoQueryThread.run(GeoDataManager.java:418)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more

Geoquery at 0.0, 0.0 breaks library

Any queries happening at or near (0.0, 0.0) cause the Library to either end up in a infinite query loop or cause a DivisionByZero Exception at S2Manager : generateHashKey method.

Sort QueryRadiusRequest result by distance to the center point

Is there a way to sort result of QueryRadiusRequest bay the distance to the center point?

.NET support

Can we please get .NET support for the DynamoDB-Geo lib?

I have successfully ported the Java S2 Geo library to C# (all tests pass, etc.)

https://www.nuget.org/packages/S2Geometry
https://github.com/onovotny/s2-geometry-library-csharp

So that's no longer a blocker.

Thanks

Wich is the correct librery

My porm.xml file

    <dependency>
        <groupId>com.google.common.geometry</groupId>
        <artifactId>s2-geometry-java</artifactId>
        <version>1.0.0</version>
        <scope>system</scope>
        <systemPath>${basedir}/libs/s2-geometry-java.jar</systemPath>
    </dependency>

    <dependency>
        <groupId>com.google.common</groupId>
        <artifactId>guava-r09</artifactId>
        <version>1.0.0</version>
        <scope>system</scope>
        <systemPath>${basedir}/libs/guava-r09.jar</systemPath>
    </dependency>

    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-dynamodb</artifactId>
        <version>1.10.61</version>
    </dependency>

    <dependency>
        <groupId>com.amazonaws.geo</groupId>
        <artifactId>dynamodb-geo</artifactId>
        <version>1.0.0</version>
        <scope>system</scope>
        <systemPath>${basedir}/libs/dynamodb-geo-1.0.0.jar</systemPath>
    </dependency>

The error

Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper
at com.amazonaws.geo.util.GeoJsonMapper.(GeoJsonMapper.java:26)
at com.amazonaws.geo.dynamodb.internal.DynamoDBManager.putPoint(DynamoDBManager.java:125)
at com.amazonaws.geo.GeoDataManager.putPoint(GeoDataManager.java:132)
at com.manriqueweb.awsdynamodb.main.Main.main(Main.java:40)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.ObjectMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

org.codehaus.jackson.map.ObjectMapper

Incorrect GeoJson (according to spec)

Hi,

The data stored in the geoJson field is incorrect as in not valid GeoJSON; the spec says that coordinates should be Long, Lat (much to the confusion with the rest of GI systems!).

I am happy to fork the repo and reverse this, along with other changes related if you want?

Scaling

Possibility to use other than iOS client

Hi guys!

I'd like to test dynamodb-geo, but have no Apple device. Is there a way to use other client?

Node/javascript/typescript port

In case it's of interest to anyone else, I'm working on a typescript port for use in node projects.

This is a complete port with no known issues yet, but it's only a few hours old. There are some minor API differences, following the conventions of the AWS javascript SDK, and using promises where java uses threads.

All credit to @yosuke-matsuda, thanks for showing the way!

Can't get stable service

Hi,

I am using this open source for dynamo geo query.
And I am making request from iOS mobile application.
But I can't get stable service. I am using t2.small instance type.
By the way, even though I sent request less than 10 per minute, I have got a lot of timeout errors as the response.
And, since I have got issues, even though I did not send any request, its instance health is changing "Ok" -> "Warning" or "Degraded"

I'd like you to let me know what the problem is.
Thanks