project-epic / epic-osm Goto Github PK

The new libosmium library maintained by osmcode.org (same author, joto) does all the PBF parsing we need, and does it much more intelligently. As such, PBF import tasks should be ported to use pyosmium, the Python bindings for libosmium so we can take advantage of the speed and caching for geographic points -- these can be fed into bulk imports to PyMongo, creating our preferred Mongo datastructure with much less hassle.

Default to :end_date = Time.now for 'on-going events'

Explain the project

Maintaining up-to-date history information for given areas of interest (analysis windows) requires a constantly running, stable database import script and a source.

Wondering about strategy for addressing missing_nodes.

This became apparent in analysis with incomplete polylines in the GeoJSON generated for users (check user "Thod" for a clear example.In http://mikelmaron.github.io/osm-history/users/ for one example). There are roads which appear "broken". Besides appearances, this makes analysis like length of roads edited inaccurate. I found this strange, because these nodes should be within the study area; it's not due to doing a hard cut.

Checking the resulting pbf file from the osm-history-splitter cut, missing nodes are actually present in the file. But they don't get imported into mongo. Looking at OSMPBF::parse_to_collection, I can't see why this would happen.

Is this a known problem? Or, anything I can do to help debug?

This is mostly done, but a sample implementation would be nice

Deal with them. What kind of questions are we interested in?

• Popular tags over different time frames?
• Resolving tag discrepancies with tools like tagwatcher ... does this have an API?

The GMLAuthor needs to re-encode to html character codes for all latin-1 characters, per the official GML format.

if bucketing the metrics by time

Package into a gem and make bin executables?

The current back and forth is creating data duplication and large time lags in the jekyll build, that said -- it works and I don't want to F with it right now.

right now we import EVERYTHING up to the end date so we have nodes pre analysis window -- but should we update to allow a smaller import?

we need a 'created_at' index on nearly all collections.

bucket out list of coordinates
run through https://github.com/ericfischer/datamaps

Pull them in! Should be a relatively simple API call with the boundingbox geometry. Save to a new mongo collection and create a new domain object for handling them. The rest should fall into place.

The paths for the Jekyll build need to be updated to reflect the relative locations to root

More exhaustive specs are needed for the pbf import to confirm this is behaving as expected. This will become more important as we subject the framework to larger files.

the to_mongo function should be catching this -- will look into it this week