pelias-deprecated / fences-builder Goto Github PK

Implement a CLI to allow easy access to all the Fences related functionality

Current extracts (geofabrik as well as mapzen) can't be used to generate complete admin boundary data set for the given region. Firstly, the data doesn't contain the lowest levels (i.e. national borders and continents). Second, the data has many missing members for admin boundary relations.

Proposed solution is to filter and extract admin boundaries using the full planet. Then implement an extract generator that takes a bounding box and generates extracts that contain any area that overlaps that box. Therefore, this extract generation process would result in a full stack of admin levels for given area.

1. convert planet pbf to o5m (~16 minutes)
   osmconvert planet-latest.osm.pbf -o=planet-latest.o5m
2. filter planet o5m to leave admin boundaries only (~50 minutes)
   osmfilter planet-latest.o5m --keep="boundary=administrative" > planet-latest-admin-boundaries.osm
3. convert filtered results back to pbf (~5 minutes)
   osmconvert planet-latest-admin-boundaries.osm -o=planet-latest-admin-boundaries.osm.pbf
4. feed pbf through openstreetmap-polygons (~3.5 hours)
   npm start
5. run geojson-slicer for a list of bounding boxes and generate extracts for each region
   TBD

Export ways and relations tagged with highway and see the quality of the data. Determine if the way data is enough or if we need to expand import pipeline to support relations.

Related to pelias/pelias#52

• Remove unused dependencies from package.json
• Change test input file from Philadelphia to something smaller that still has several admin levels

Instead of doing

var val = nconf.get('key');

for each item, just do

var config = nconf.get();

see
https://github.com/pelias/fences-builder/blob/master/bin/fences-builder#L16

Would be nice to provide a short README.md in each extract with details about the org and structure of the extracted data.

Cleanup and normalize admin boundary properties.

• Set name to name:en, where available
• Remove uncommon/unimportant tags
• Provide stats per admin level (err, areas, average number of tags)
• Identify and decide what to do with admin_level values in "other"

During the extract process, generate simplified polygons and store them in a single file. Add admin_level tag to each for easy filtering. (use ogr2ogr)

Test case:

• attempt to process the planet with max memory limit set very low (e.g. 1024)
• process will fail to successfully output any extract data, but on exit querying the exit code (echo $?) shows success: 0.

Process should exit with non-zero exit status on failure.

Admin levels with problematic characters are causing the process to choke.

Here's what happened when admin_level was set to Village/Desa (it tried to create a new directory in output dir)

stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
Error: ENOENT, open '/mnt/poly/output/indonesia-latest.osm.pbf-level-Village/Desa.geojson'

There are many errors about name tags missing for coastline areas. Maybe just set name to "coastline"? These could also go into a separate file, if it makes sense to break them out since they aren't administrative, but rather natural.

{
  "message": "Area is missing a name tag",
  "data": {
    "type":"Feature",
    "properties": {
      "natural":"coastline",
      "boundary":"administrative",
      "admin_level":"4"
    }
  }
}

There are many errors that show up in the extract for missing name tag for TIGER data. Implement a way to specify alternate name tags, ordered by priority.

example:

{
  "message": "Area is missing a name tag",
  "data": {
  "type": "Feature",
    "properties": {
      "is_in": "USA, Alabama",
      "place":"town",
      "source": "TIGER/Line® 2008 Place Shapefiles (http://www.census.gov/geo/www/tiger/)",
      "boundary": "administrative",
      "tiger:CPI": "N",
      "wikipedia": "Somerville,_Alabama",
      "created_by":"polyshp2osm-multipoly",
      "place_name": "Somerville",
      "tiger:LSAD": "43",
      "tiger:NAME": "Somerville",
      "admin_level": "8",
      "border_type":"town",
      "is_in:state": "Alabama",
      "tiger:MTFCC": "G4110",
      "is_in:country": "USA",
      "tiger:CLASSFP": "C1",
      "tiger:PCICBSA":"N",
      "tiger:PLACEFP": "71496",
      "tiger:PLACENS": "02407365",
      "tiger:PLCIDFP": "0171496",
      "tiger:STATEFP": "01",
      "tiger:FUNCSTAT":"A",
      "tiger:NAMELSAD": "Somerville town",
      "tiger:PCINECTA": "N",
      "tiger:reviewed": "no",
      "is_in:iso_3166_2": "US:AL",
      "is_in:state_code":"AL",
      "is_in:country_code": "US"
    }
  }
}  

use "tiger:NAME": "Somerville" as a replacement for name.

Use admin_level tag values to split the geojson output into several files.

Figure out why the national border for US is missing. When filtering with osmfilter and then converting to geojson using ogr2ogr, I get an error about too many tags on that object.

1. Add sanity checks for errors, like looking up which relations/ways became actual areas and why the others failed to.
2. Expose the errors, along with some explanation as to why they failed.

Potential things to add:

• Timezones
• Driving side
• Metric / Imperial
• Language
• Name of region in all languages
• Population
• Currency