jbinto / bikeways Goto Github PK
View Code? Open in Web Editor NEWAn exploration of the City of Toronto Bikeways open data set, using Rails & PostGIS
License: MIT License
An exploration of the City of Toronto Bikeways open data set, using Rails & PostGIS
License: MIT License
The initialization process should use some sort of heuristic (SHA hash, size of file, etc.) to determine whether or not to download the file.
When I imported the shapefiles manually using shp2pgsql
, it created a geometry(MultiLineString,4326)
column.
Using the RGeo::Shapefile::Reader``importer, it's stored as a
geometry(Geometry,4326)`.
Determine if there's any real impact to this.
Also investigate why records are stored as MULTILINESTRING
in the original data when they are all just individual LINESTRING
s.
See output from rake opendata:stats:walk_all_routes
Bayview Ave (lfn_id=24); id=1987; geoid=1903; 16 segments in db; 12 segments found by walking; BAD! counts DO NOT MATCH!
If you look at the data, you'll see it picks up geoid=1903
. If you walk manually on the website from here, you see 12 segments on the Bayview Extension.
Where are the unaccounted 4? They are on Bayview Av near Lawrence, about 10 km away. They're not contiguous with the arbitrary segment we tried to walk.
This means that we can't just group by linear feature ID, because it could result in wonky data (e.g. two small bike lanes on opposite ends of town not at all connected).
If we do incorporate the SegmentWalker
into the import process, it would need to be robust enough to not only handle non-contiguous paths, but forks in the road too.
Object cache of KML output. Make sure e-tags is used. Compress as much as possible.
Look at e.g. http://www.canadatrailguide.com/map_results.php for inspiration
The postgresql user bikeways_staging
(et al) has SUPERUSER
privilege. This is necessary to create the PostGIS extension.
This violates least privilege. We could fix this using the approach suggested in GeoRails:
You'll set up two users in the database. One is the normal database user that your Rails app will connect as. The other needs to be a superuser, and is used to create the databases initially (i.e. using rake db:create). Creating a PostGIS database requires superuser privileges unless you set up a special template for it, and for simplicity, we won't do that right now.
Curious about the template hint there. Not sure exactly what it means, but perhaps we could create a PostGIS template in the Ansible playbook.
I ripped out turbolinks
in an attempt to get things going quickly, but it was a hack around the fact that I'm loading the Google Maps scripts synchronously.
There's many advantages to loading the script async. (n.b. perhaps the following is an unrelated tangent, but:) We should be able to dynamically tell the map container to reload its KML without having to reload the page.
References:
As a result of #5, where we've been re-importing the Open Data dump over and over again, our IDs are incrementing. Previous links no longer work.
We should use a gem like friendly_id or sluggi to make slugs, so at least the URLs stable.
We could use the city_geo_id
, or a hash of the geometry itself as a "stable" ID.
A little background:
The city supplies ESRI shapefiles in the following formats:
The former is SRID 2019 (Toronto/S. Ont. specific), the latter SRID 4326 (Plate Carree projection).
When using the SRID 2019 data and using PostGIS st_transform
to convert to WGS84/4326, we get "reasonable" results but the misalignment to Google Maps is unacceptable (see #1).
When using the SRID 4326 data directly as a geometry
(as opposed to a geography
), we have fairly good matching with Google Maps. Some routes appear a little off, but closer examination with the satellite view proves the city's data more accurate than Google's. (n.b. this is an n=1 conclusion, further research needed)
However, the third option we haven't tried yet is storing lat/lng natively in a geography
. According to Daniel Azuma on his GeoRails series:
You might be tempted to store latitude and longitude in a geometry type column. That is, to set up your PostGIS column with a geometry type, but use SRID=4326 (which is the EPSG number for WGS 84 latitude and longitude).
Don't do this.
I did this a few times in my naive youth, and it came back to bite me. What you're really doing here is employing a particular projection called Plate Carree, which simply maps latitude and longitude directly to x and y on the plane. Remember, any time you use geometry rather than geography, you are working with a flat coordinate system, and thus a projection. You might think you're working with latitude and longitude, but you're actually not.
Plate Carree is not a particularly useful projection (except that it is trivial to compute). It doesn't preserve distances, angles, directions, areas, or any other cartographically useful properties, and its distortion in polar regions is severe. In almost all cases, you can do much better with a different projection.
He suggests using a Mercator (EPSG:3785) projection that is used by Google Maps.
But here's the problem: My data is fixed. I am not creating any data myself. While I could transform the city's data to whatever projection I want, I currently only have two uses for the data:
In hindsight, it seems obvious that KML was intended for Google Earth, a 3D application. That Google Maps accepts KML is surely a result of the synergies between the two, but it was easy to forget that Maps is a 2D application, despite accepting 3D coordinates.
I do not see any benefit to storing my data in EPSG:3785 when that is none of the following: the form my input data is in, the form my output data must be, nor the most efficient/accurate way to perform distance calculations.
This issue will remain open until I can wrap my head around the tradeoffs between these projections.
It's trivial to disable the scroll wheel in google maps.
This would allow us to put maps near the top.
Everything is a perfect 10 or-so-metres west on the map. Consider using the WGS84 data instead of the local "MTM" (SRID 2019) data..
See http://staging.416.bike/bikeway_segments/1715
Or screenshot:
The thrill of having a map that shows all the routes without chrome crashing has faded.
While the .kml approach works, it's slow and cumbersome.
Look into the following:
Not sure how I'm supposed to import the intersection data from the city's opendata site:
http://www1.toronto.ca/wps/portal/contentonly?vgnextoid=e659522373c20410VgnVCM10000071d60f89RCRD&vgnextchannel=1a66e03bb8d1e310VgnVCM10000071d60f89RCRD
I expected that the intersection table would look like this:
But instead, it's basically a duplicate of the existing bikeway data:
The only difference appears to be the existence of the fields fcode
and fcode_desc
(which I believe I deliberately omitted from the bikeways).
None of the city's geoid
, rid
, object_id
, etc. match up on either table.
It's almost less than useless. Mainly because there doesn't appear to be any cross street information.
At this point I don't see any difference between the intersection
and bikeway_segment
tables.
For whatever reason, the Bikeways data file is not being detected, and the load process is happening on each deploy.
Perhaps vendor/opendata
needs to be a shared (symlinked) directory on Cap deploys.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.