whosonfirst / go-whosonfirst-pip-v2 Goto Github PK

View Code? Open in Web Editor NEW

11.0 10.0 5.0 22.76 MB

An in-memory point-in-polygon (reverse geocoding) package for GeoJSON data, principally Who's On First data.

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.09% Go 4.35% Shell 0.44% CSS 0.03% HTML 0.06% JavaScript 94.84% Dockerfile 0.19%

golang whosonfirst geo point-in-polygon

go-whosonfirst-pip-v2's Issues

Investigate geohashtree as index/engine

https://github.com/murphy214/geohashtree

Add a SQLite caching layer (for SPR)

A side-effect of all the sqlite/extras work is that I think I have all the pieces to use SQLite databases that are generated for extras for generating the standard SPR responses too which in principle means the overall memory requirements will come down.

I am not convinced the savings won't be eaten up entirely by disk and other I/O stuff but one thing at a time...

too many pip-extras files

Since the announcement of shutdown of MapZen I'm trying to move from WoF to extract of OSM for use with this pip-server. So this issue is not critical for me and just for a note for others. (we need to find out how to return local names in OSM data).

This is related to #16.

In a test, number of pip-extras files is too many (>255) and the pip-server could not accept connections from clients.

...
254
/private/var/folders/bz/7sjt8cp50735lkmcpr6ftpv00000gp/T/pip-extras578601435
255
/private/var/folders/bz/7sjt8cp50735lkmcpr6ftpv00000gp/T/pip-extras578601435

Add query logging to wof-pip-server

Out of date Docker & Makefile?

Hello,

This looks like a great service for some of our use cases, and would love to get a docker instance up but running into a couple of issues. Any guidance would be appreciated.

Both the Dockerfile (https://github.com/whosonfirst/go-whosonfirst-pip-v2/blob/master/Dockerfile#L33) and documentation make a reference to make bin but looking at the Makefile, there doesn't seem to be a bin directive. I switched this to tools, but then ran into some other issues with downloading dependences when building the Docker container.

I can however run locally using make tools (if that helps at all).

Add crosshairs and fixed pin when jumping to lat/lon

Update CLI opts for stand-alone wof-pip tool

Also: consider removing it altogether...

Add a -cache-all flag

Investigate a socket based server

Something without the HTTP overhead. Maybe use:

https://github.com/tidwall/shiny/blob/master/examples/echo-server/main.go

Include concordances in response?

Is there a way to include the concordances in the response?

Seems like this would probably need to happen during the index phase, not sure if there was an easy way to configure this or not.

Thanks!

Document all the CLI opts

Add > 2 cache chain-ing

Essentially multiple -cache flags where each one fails over to the next until there are no more

Add support for decorating responses from SQLite databases

Essentially support for an ?extras= parameter. See also:

#16

Rename -debug flag(s)

Allow a person to specify path for debug endpoint
Give -debug-as-geojson flag a less-stupid name

api-key support

-mapzen-api-key is used to access the MapZen site.

To protect the pip-server from invalid access, we need the -api-key option. Though I wrote a code for it and it is working, there are some problems.

My modification works only on intersectHandler, because I modified it and I hesitate to add the modification to other handlers.

Ideally, handlers were chainable middlewares. If so, we could write apiKeyHandlerMiddleware or something like that and it might be applicable to any middlewares.

Another problem is api-key exchange. That could be resolved if we would use a reverse proxy and replace pip-servers periodically. The reverse proxy might be needed to replace pip-servers with pip-servers with updated data.

switch data with keeping service?

It took 23 minutes to start after indexing country.db, region.db, county.db and locality.db.

Every time WoF-data is updated, we need to restart the pip-server and service stops for the period, right?

Is there any way to switch data while it is serving?

It occupies 4.38GB now on my iMac. I don't think double buffering is a good idea.

about the deprecation notice

@straup Hi, I read the soon-to-be-deprecated/superseded notice. Please leave this repo if you do so.

Thanks for your contribution.

extras=geometry does not work

This issue is related to #16.

I need local names (e.g. name:jpn_x_preferred) and geometry of a point so that I can render the place using geometry coordinates in a map. How do I get both?

./bin/wof-pip-server -allow-extras -allow-geojson -mode sqlite -port 8300 ~/whosonfirst-data-sqlite/whosonfirst-data-latest.db

format=geojson&extras=name:jpn_x_preferred returns geometry but not :jpn_x_preferred.

curl 'http://localhost:8300?format=geojson&latitude=36.1&longitude=140.08&extras=name:jpn_x_preferred' | python -mjson.tool

format=json&extras=name:jpn_x_preferred,geometry returns name:jpn_x_preferred but not geometry.

curl 'http://localhost:8300?format=json&latitude=36.1&longitude=140.08&extras=name:jpn_x_preferred,geometry' | python -mjson.tool

Add support for indexing points... maybe?

Really only useful for nearest neighbour, but maybe that's reason enough. In index/rtree.go invoke the ToRect method:

https://github.com/dhconnelly/rtreego/#storing-updating-and-deleting-objects

Sort out "must provide positive size" errors

166 ->./bin/wof-pip-server -cache lru -lru-cache-size 0 -mode feature-collection -not-wof -debug -debug-as-geojson map2.geojson
18:12:09.365810 [wof-pip-server][index] STATUS time to index feature collection '/Users/asc/Desktop/map2.geojson' 3.00181ms
18:12:09.365885 [wof-pip-server][index] STATUS time to index path 'map2.geojson' 3.091668ms
18:12:09.365890 [wof-pip-server][index] STATUS time to index paths (1) 3.099504ms
18:12:09.365912 [wof-pip-server] FATAL Failed to creation application cache, because Must provide a positive size
18:12:09.365916 [wof-pip-server] FATAL Failed to creation application cache, because Must provide a positive size

return name: yyy_x_preferred?

Hi,

I tried the server. It's working greatly!! I'm impressed with its speed.

How do I get locale names? The following response does not include name:yyy_x_preferred.
And I don't need all the fields. So it would be nice if we could specify which fields were needed.

EDIT: I think it should have region (85672801, Ibaraki), but not.

    curl 'http://localhost:8300?latitude=36.1&longitude=140.08' | python -mjson.tool    
     {
         "places": [
             {
                 "mz:is_ceased": -1,
                 "mz:is_current": 1,
                 "mz:is_deprecated": 0,
                 "mz:is_superseded": 0,
                 "mz:is_superseding": 0,
                 "mz:latitude": 36.096175,
                 "mz:longitude": 140.072981,
                 "mz:max_latitude": 36.2361335754395,
                 "mz:max_longitude": 140.17239379882795,
                 "mz:min_latitude": 35.94746398925778,
                 "mz:min_longitude": 139.99589538574196,
                 "mz:uri": "https://whosonfirst.mapzen.com/data/890/519/957/890519957.geojson",
                 "wof:country": "JP",
                 "wof:id": 890519957,
                 "wof:lastmodified": 1505433065,
                 "wof:name": "Tsukuba Shi",
                 "wof:parent_id": 85672801,
                 "wof:path": "890/519/957/890519957.geojson",
                 "wof:placetype": "county",
                 "wof:repo": "whosonfirst-data",
                 "wof:superseded_by": [],
                 "wof:supersedes": []
             },
             {
                 "mz:is_ceased": -1,
                 "mz:is_current": -1,
                 "mz:is_deprecated": 0,
                 "mz:is_superseded": 0,
                 "mz:is_superseding": 0,
                 "mz:latitude": 36.101144,
                 "mz:longitude": 140.087585,
                 "mz:max_latitude": 36.11459552940025,
                 "mz:max_longitude": 140.09841285397465,
                 "mz:min_latitude": 36.0890603976255,
                 "mz:min_longitude": 140.07392334000025,
                 "mz:uri": "https://whosonfirst.mapzen.com/data/859/090/39/85909039.geojson",
                 "wof:country": "JP",
                 "wof:id": 85909039,
                 "wof:lastmodified": 1497309907,
                 "wof:name": "\uff14\u4e01\u76ee",
                 "wof:parent_id": 102031903,
                 "wof:path": "859/090/39/85909039.geojson",
                 "wof:placetype": "neighbourhood",
                 "wof:repo": "whosonfirst-data",
                 "wof:superseded_by": [],
                 "wof:supersedes": []
             },
             {
                 "mz:is_ceased": -1,
                 "mz:is_current": -1,
                 "mz:is_deprecated": 0,
                 "mz:is_superseded": 0,
                 "mz:is_superseding": 0,
                 "mz:latitude": 0,
                 "mz:longitude": 0,
                 "mz:max_latitude": 90,
                 "mz:max_longitude": 180,
                 "mz:min_latitude": -90,
                 "mz:min_longitude": -180,
                 "mz:uri": "https://whosonfirst.mapzen.com/data/0/0.geojson",
                 "wof:country": "",
                 "wof:id": 0,
                 "wof:lastmodified": 1501272888,
                 "wof:name": "Earth",
                 "wof:parent_id": -1,
                 "wof:path": "0/0.geojson",
                 "wof:placetype": "planet",
                 "wof:repo": "whosonfirst-data",
                 "wof:superseded_by": [],
                 "wof:supersedes": []
             },
             {
                 "mz:is_ceased": -1,
                 "mz:is_current": -1,
                 "mz:is_deprecated": 0,
                 "mz:is_superseded": 0,
                 "mz:is_superseding": 0,
                 "mz:latitude": 36.129991,
                 "mz:longitude": 138.445432,
                 "mz:max_latitude": 45.52314,
                 "mz:max_longitude": 153.985229,
                 "mz:min_latitude": 24.036472,
                 "mz:min_longitude": 122.93853,
                 "mz:uri": "https://whosonfirst.mapzen.com/data/102/047/657/102047657.geojson",
                 "wof:country": "",
                 "wof:id": 102047657,
                 "wof:lastmodified": 1470254944,
                 "wof:name": "Asia/Tokyo",
                 "wof:parent_id": 85632429,
                 "wof:path": "102/047/657/102047657.geojson",
                 "wof:placetype": "timezone",
                 "wof:repo": "whosonfirst-data",
                 "wof:superseded_by": [],
                 "wof:supersedes": []
             },
             {
                 "mz:is_ceased": -1,
                 "mz:is_current": 1,
                 "mz:is_deprecated": 0,
                 "mz:is_superseded": 0,
                 "mz:is_superseding": 0,
                 "mz:latitude": 36.13793,
                 "mz:longitude": 138.441899,
                 "mz:max_latitude": 45.52288818400007,
                 "mz:max_longitude": 153.98706054700006,
                 "mz:min_latitude": 20.42288970900006,
                 "mz:min_longitude": 122.93458557100007,
                 "mz:uri": "https://whosonfirst.mapzen.com/data/856/324/29/85632429.geojson",
                 "wof:country": "JP",
                 "wof:id": 85632429,
                 "wof:lastmodified": 1509142777,
                 "wof:name": "Japan",
                 "wof:parent_id": 102191569,
                 "wof:path": "856/324/29/85632429.geojson",
                 "wof:placetype": "country",
                 "wof:repo": "whosonfirst-data",
                 "wof:superseded_by": [],
                 "wof:supersedes": []
             }
         ]
     }

Replace go-http-mapzenjs

Update to use:

For example:

https://github.com/sfomuseum/go-placeholder-client-www

Add support for updating records on the fly

https://github.com/dhconnelly/rtreego/#storing-updating-and-deleting-objects

Mostly just a re-jiggering of the code in IndexFeature in to share methods and various locks and flags (indexing) to keep everything in order.

error: invalid character 'v' looking for beginning of value

I would like to report that I got the following error during indexing.

./bin/wof-pip-server -port 8300 -mode directory ~/whosonfirst-data/data
12:25:55.267986 [wof-pip-server] STATUS listening on localhost:8300
12:25:56.272296 [wof-pip-server] STATUS indexing 3023 records indexed
12:25:57.271904 [wof-pip-server] STATUS indexing 6554 records indexed
12:25:58.273400 [wof-pip-server] STATUS indexing 10200 records indexed
12:25:59.276565 [wof-pip-server] STATUS indexing 13596 records indexed
......
12:34:36.252812 [wof-pip-server] STATUS indexing 778120 records indexed
12:34:37.251508 [wof-pip-server] STATUS indexing 778120 records indexed
error: invalid character 'v' looking for beginning of value
12:34:37.980572 [wof-pip-server] STATUS finished indexing

I used the latest data and pip-server-v2.
whosonfirst-data commit 1fa48fd5b9b875fa016a86729d7c99a85e916a95
go-whosonfirst-pip-v2 commit d646dc1

OS: Mac OS X 10.12.6
go: go version go1.9.2 darwin/amd64

Add jump to lat,lon input box

Add a local/custom db.Table definition for properties

So that we are only storing/retrieving properties rather than entire features (and their geometries)

The other alternative is to update the geojson table schema in go-whosonfirst-sqlite to store geometries and properties as separate columns/blobs but then it is always left as an exercise to the user to reconstruct the feature...

Use with OpenStreetMap GeoJSON data

GeoJSON data made from OpenStreetMap could be used. However tags are different.
And osm_id: 3795064 in the geojson was replaced with "spr:id":"wyp70xyep6m0". It is impossible to use with other OSM based tools and servers. Maybe the current implementation for the extras parameter is for sqlite mode only.

whosonfirst / go-whosonfirst-pip-v2 Goto Github PK

go-whosonfirst-pip-v2's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs