koopjs / koop Goto Github PK

FeatureCollections are "dumb blobs" of JSON features. Using Koop to have FeatureService similar access, query and analyze would make them much more useful.

Everything cannot be in master or in the same repo. We need to establish a best practice for writing providers and having them included into the code base upon a deployment of koop.

Thoughts:

use npm as a way to install a provider
use configurable grunt tasks
use git submodules?

git clone https://github.com/Esri/koop.git
cd koop
npm i

yields....

npm http GET https://api.github.com/repos/chelm/Terraformer/tarball/master
npm http GET https://registry.npmjs.org/nodetiles-core/0.0.3
npm http 304 https://registry.npmjs.org/nodetiles-core/0.0.3
npm http 200 https://api.github.com/repos/chelm/Terraformer/tarball/master
npm http GET https://registry.npmjs.org/async
npm http GET https://registry.npmjs.org/lodash
npm http GET https://registry.npmjs.org/proj4js
npm http GET https://registry.npmjs.org/canvas
npm http GET https://registry.npmjs.org/proj4js-defs
npm http GET https://registry.npmjs.org/carto
npm http GET https://github.com/yuletide/node-shp/tarball/master
npm http 304 https://registry.npmjs.org/async
npm http 304 https://registry.npmjs.org/lodash
npm http 304 https://registry.npmjs.org/canvas
npm http 304 https://registry.npmjs.org/proj4js
npm http 304 https://registry.npmjs.org/proj4js-defs
npm http 304 https://registry.npmjs.org/carto
npm http 200 https://github.com/yuletide/node-shp/tarball/master
npm WARN engine [email protected]: wanted: {"node":">=0.6.x","npm":"1.1.x"} (current: {"node":"v0.10.25","npm":"1.3.26"})
npm WARN engine [email protected]: wanted: {"node":">=0.6","npm":"~1.2.0"} (current: {"node":"v0.10.25","npm":"1.3.26"})

> [email protected] install /Users/daniel/dev/ext/koop/node_modules/nodetiles-core/node_modules/canvas
> node-gyp rebuild

Package xcb-shm was not found in the pkg-config search path.
Perhaps you should add the directory containing `xcb-shm.pc'
to the PKG_CONFIG_PATH environment variable
Package 'xcb-shm', required by 'cairo', not found
gyp: Call to './util/has_cairo_freetype.sh' returned exit status 0.
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/Users/daniel/.nvm/v0.10.25/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:337:16)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:797:12)
gyp ERR! System Darwin 13.0.0
gyp ERR! command "node" "/Users/daniel/.nvm/v0.10.25/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/daniel/dev/ext/koop/node_modules/nodetiles-core/node_modules/canvas
gyp ERR! node -v v0.10.25
gyp ERR! node-gyp -v v0.12.2
gyp ERR! not ok 
npm http GET https://registry.npmjs.org/underscore
npm http GET https://registry.npmjs.org/mapnik-reference
npm http GET https://registry.npmjs.org/optimist
npm http GET https://registry.npmjs.org/xml2js
npm http GET https://registry.npmjs.org/iconv-lite
npm http 304 https://registry.npmjs.org/underscore
npm http 304 https://registry.npmjs.org/optimist
npm http 304 https://registry.npmjs.org/xml2js
npm http 304 https://registry.npmjs.org/mapnik-reference
npm http 304 https://registry.npmjs.org/iconv-lite
npm http GET https://registry.npmjs.org/sax
npm http GET https://registry.npmjs.org/wordwrap
npm http GET https://registry.npmjs.org/minimist
npm http 304 https://registry.npmjs.org/sax
npm http 304 https://registry.npmjs.org/wordwrap
npm http 304 https://registry.npmjs.org/minimist
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the canvas package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls canvas
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 13.0.0
npm ERR! command "node" "/Users/daniel/.nvm/v0.10.25/bin/npm" "i"
npm ERR! cwd /Users/daniel/dev/ext/koop
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.26
npm ERR! code ELIFECYCLE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/daniel/dev/ext/koop/npm-debug.log
npm ERR! not ok code 0

Both #45 and #46 will need to scale cleanly so we need to understand the bet route for queueing tasks for data retrieval and data conversion.

Previoulsy we've used ZeroMQ to handling scaling to many workers. This task will explore that as an option and make sure its a valid solution.

Koop should be able to access a FeatureService layer, get all its data (#45), and convert it features into the following formats:

• kml
• csv
• geojson
• shapefile

Doing this involves wrapping calls to ogr2ogr from javascript via a child process. Some thought needs to be put into using a worker / queue system to support many requests at once.

Tried mapping https://github.com/cambridgegis/cambridgegis_data/blob/master/Demographics/Census_2010/2010_Tracts/DEMOGRAPHICS_Tracts2010.geojson

koop URL: https://koop.dc.esri.com/github/cambridgegis/cambridgegis_data/Demographics::Census_2010::2010_Tracts::DEMOGRAPHICS_Tracts2010/FeatureServer/0

Results in just points

Utfgrid is super slick and has a direct friend in koop's png tile support. Koop should also provide utfgrid endpoint for clients that support utfgrid+tiles

I would like to build new input Data provider adapters, ala Node Geoservices Adaptor. For example I want to connect to the DC Metro API or Capital Bikeshare data.

This is supposedly possible but it's unclear how to properly add this. Can you provide an example template data provider adapter?

Terraformer has this, should be easy to add

hit this once, then refresh: http://koop.herokuapp.com/geojson/snow/FeatureServer

every refresh adds another layer.

we should be caching requests using something like: https://npmjs.org/package/express-view-cache

or whatever host

In order for this to work the latest gdal code needs to be installed. Do so is proving a bit difficult so far.

For both gists and repos the geohub modules returns a list of geojson objects (per file).

Koop needs to handle accessing these as layers

I think something is wrong with the link in the demo section. If you visit http://koop.dc.esri.com you will get a "502 Bad Gateway" error msg.

Right now we simple serialize json into an object.

We should be staching data in a persistent store and checking for file updates on github or gists to wipe.

Simple change to the views to use the minified assets when in prod.

need a solid example

having this working would be the minimum for this release, very important.

Users are experimenting with storing large quantity of features (1m+) in ElasticSearch in order to have fast, dynamic, spatial queries.

They would like to use the Esri JS API to visualize these features. So Koop connecting to ElasticSearch would be valuable. These requests should be sub-second with gridded queries.

We need to access the same filters as feature services on raw geojson. Should be able to apply objectIds, where, etc to a geojson endpoint

This can be essentially copied from @nixta and his node ags stuff

Each endpoint should follow the typically layout and structure of an AGS instance.

Right now we touch the features arrays a lot, this blocks node and impact performance. We need to be queuing filters and processing them async if possible.

These should be pulled in dynamically as an example of how to dynamically amend the routes and controllers via a plugin type arch.

Currently things are geared towards points, lines, and polygons need to have specific renders.

Currently just using the "Sails" favicon. I would suggest making a simple Koop (or esri) favicon instead.

Would be awesome to provide geo spatial where clauses to raw geojson output too, not just esri json

This involves a refactor of the filesystem methods in various controllers and models to all call a helper class that can be configured at startup to write to either the filesystem or s3 (or some other place down the road).

Right now the controllers are doing too much logic for retrieving gists and github data. They should use a model to get data and then the controller should simply format and route it to the correct view

APIs like ArcGIS online have the aiblity to be deployed in any place. Users of koop should be able register an arcgis online endpoint and reference item ids.

The same goes for any instance of an API that can be deployed to different servers / orgs. This holds true for things like geocommons.

POST to koop..com/ { :type } /register -> { host: 'http://...' }
-> return :ID

Also going to /:type should list registered provider id and hosts

Everything should be operating on node streams, this will improve performance and will allow better chaining of filter processes.

LinkedData all the things

• dumpgj JSON-LD from @sgillies
• example of Roman ampitheaters

The potential is to make default display, as well as downstream analysis, easier.

currently outSR only works with points

Add support for /z/x/y.png Tile output from a Koop endpoint. By default these would be simple styling - blue dots or blue polygons.

Additionally, it would be useful to reference the URL to a CartoCSS file or other styling that could be dynamically applied for these tiles.

By having this defined at Query time it would permit a user to load data from a third-party but apply their own personal style.

Ensure we can support pointing to URL or File based versions of GeoCommons GeoJSON files -

http://geocommons.com/overlays/1.json?include_features=1&geometry_format=geojson

set supportStats to true and return stats on requests with outStatistics defined (also accepts post reqs for stat calls...)

We need to support accessing large feature services in Koop. Or rather we need to treat featureservices that exceed the maxRecordCount as single entities so we can support data exports.

This task includes:

• making id only requests first and checking the size of the data
• building a series of request tasks for getting pages of features
• combining the data into a one set of features
• returning all the data to the request

This is important so we have our custom data providers loaded up in the global space so our controllers have access.

Koop should support "outStatistics" as defined here: http://resources.arcgis.com/en/help/rest/apiref/

example usage:

[
{
"statisticType": "<count | sum | min | max | avg | stddev | var>",
"onStatisticField": "Field1",
"outStatisticFieldName": "Out_Field_Name1"
},
{
"statisticType": "<count | sum | min | max | avg | stddev | var>",
"onStatisticField": "Field2",
"outStatisticFieldName": "Out_Field_Name2"
}
]

To support current and future needs for geoprocessing and geometry based selection Koop should be using PostGIS. Mongo is great for simply adding and retrieving data much like a simple json cache, but as soon as Koop needs to rely on subsetting data directly from its DB then mongo really falls short in terms of polygon intersection / bbox selection.

All geospatial filters and selections should pass to the database where koop caches data.

I receive an error when I try to start with "node app.js".
Sails loads successfully and I get the information how to start on port: 1337 but in the end there is something wrong with the Redis connection to 127.0.0.1:6379.
The redis db comes with the project?

I use nodejs 0.10 on win2008 server x64.

Waiting...
verbose: Binding route :: get /gist/:id/FeatureServer/:layer/:method
verbose: Binding route :: get /gist/:id/FeatureServer/:layer
verbose: Binding route :: get /gist/:id/FeatureServer
verbose: Binding route :: get /gist/:id
verbose: Binding route :: get /gist/raw/:id
verbose: Binding route :: get /gist/raw/:id/:layer
verbose: Binding route :: get /gist
verbose: Binding route :: get /gist/:id/preview
verbose: Binding route :: get /github/
verbose: Binding route :: get /github/:user
verbose: Binding route :: get /github/:user/:repo
verbose: Binding route :: get /github/:user/:repo/FeatureServer
verbose: Binding route :: get /github/:user/:repo/FeatureServer/:layer
verbose: Binding route :: get /github/:user/:repo/FeatureServer/:layer/:method
verbose: Binding route :: get /github/:user/:repo/:file
verbose: Binding route :: get /github/:user/:repo/:file/FeatureServer
verbose: Binding route :: get /github/:user/:repo/:file/FeatureServer/:layer
verbose: Binding route :: get /github/:user/:repo/:file/FeatureServer/:layer/:method
verbose: Binding route :: get /github/:user/:repo/:file/preview
verbose: Binding route :: get /sample
debug: Lowering sails...
verbose: Sent kill signal to grunt child process...

events.js:72
throw er; // Unhandled 'error' event
^
Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
at RedisClient.on_error (C:\Users\Administrator\Documents\GitHub\koop\node_modules\redis\index.js:163:24)
at Socket. (C:\Users\Administrator\Documents\GitHub\koop\node_modules\redis\index.js:86:14)
at Socket.EventEmitter.emit (events.js:95:17)
at net.js:441:14
at process._tickDomainCallback (node.js:459:13)

Add more coverage of the spec inside koop

Esri JSON doesnt support geom collections so we need a way to support rendering each geometry type.

This can be used to quickly index and access geometry. Should be able to cache indexes and stores into a db of some sort.

Would be awesome to come up with a postgis or mongo store too.

Our current parser doesnt work with the arcgis rest spec's sql

the features should be accessible via urls: http:///

http://arcgis.dtsagile.com/arcgis/sdk/rest/index.html?catalog.html

Stefan's setup some docket servers that koops needs to be dployed within.

dont make people have to access to github to see simple docs. makes it accessible out of esri

We should support different database adaptations like postGIS, mongodb and couchdb.

This requires cleaning up the PostGIS code and abstracting out the config params a bit.