eebbesen / freerider_api

Track rideshare usage in your area

Freerider API

This project will read vehicle location data from Car2Go and persist it to

• a file in Dropbox
• a relational database like PostgreSQL

This project will read Car2Go vehicle location data from a Dropbox file and persist it to a relational database like PostgreSQL.

The reason for offering Dropbox as an intermediary is to allow me to avoid paying for a hosted PostgreSQL instance :).

Note that times are by default in UTC.

System dependencies

• Ruby 2.2.0

Configuration

I'm including Heroku and Heroku add-on information here, but you can deploy where you'd like.

Config variables to set in deployment app or cofig/environments/development.rb

ENV['CONSUMER_KEY'] = '<your_car2go_consumer_key>'
ENV['NEW_RELIC_LICENSE_KEY'] = '<your_new_relic_license_key>'
ENV['DROPBOX_OAUTH_BEARER'] = '<your_dropbox_access_token>'

Database initialization

rake db:migrate

Tests

rake test

Deployment instructions

Heroku

To run a rake task which will

• push code to Heroku
• run migrations on Heroku
• restart the application on Heroku
• run poll_and_persist_vehicles['austin'] against the Heroku instance (as a validation -- you should monitor the output)

rake heroku:deploy_and_run

Jobs

poll_and_dropbox_vehicles

This task polls for and places one file per run per city in a dropbox location.

rake poll_and_dropbox_vehicles

Or for just one city

rake poll_and_dropbox_vehicles['austin']

poll_and_persist_vehicles

This task polls for and persists the vehicle/locations for all cities into the database.

rake poll_and_persist_vehicles

Or for just one city

rake poll_and_persist_vehicles['austin']

consume_dropbox_data

This task persists Dropbox file data into a local database and deletes processed files from Dropbox

RAILS_ENV=production bundle exec rake consume_dropbox_data

I cron it:

42 * * * * /bin/bash -l -c 'cd /home/username/projects/freerider_api && rvm use ruby-2.2.2 && bundle install \
&& RAILS_ENV=production bundle exec rake consume_dropbox_data'

converter:add_vehicle_location_time

The filename column gives you an idea of when a vehicle location was recorded, but that doesn't work in the context of tools like Carto. This task adds a timestamp on the end of each row in a new CSV it creates from the one you pass in

rake converter:add_vehicle_location_time['<csv_to_convert>']

locations

This task returns the URI-ready city names where Car2Go operates

rake locations

create_map

This task will take the last five days' location data for a given city and create a CSV of the data in Dropbox

rake create_map['austin']

freerider_api's Issues

Consumer process stops when unexpected file in dropbox

A VCR cassette brought the whole thing to its knees.

Some exception handling (and maybe notification) would be great.

As an application owner I want to be able to turn on debug-level logging in Heroku deployment

• configure logging
• add logging to app as needed

For example, api calls returned 500 because migrations had yet to be run, but heroku logs was not showing errors I'd expect to see to tell me what was going on

utilize vcr for car2go and dropbox

As a developer I want poll_and_persist_locations to run after every repository master build and every Heroku deployment

Trigger in Heroku after every successful deployment

batch inserts into database

Inserting one record at a time is not fast

Throw charts up on WordPress or some such

upgrade rails

We are getting duplicate vehicle locations per file

For example:

99106220,AB8510,-93.31421,44.99523,2016-12-20 15:55:54.443353,twincities,GOOD,GOOD,WMEEJ3BA8EK755985,twincities-20161217_180115
99106221,AB8529,-93.27318,44.97135,2016-12-20 15:55:54.445088,twincities,GOOD,GOOD,WMEEJ3BA8EK752844,twincities-20161217_180115
99106207,AB8510,-93.31421,44.99523,2016-12-20 15:55:54.425077,twincities,GOOD,GOOD,WMEEJ3BA8EK755985,twincities-20161217_180115
99106209,AB8529,-93.27318,44.97135,2016-12-20 15:55:54.42687,twincities,GOOD,GOOD,WMEEJ3BA8EK752844,twincities-20161217_180115

As an application owner I want New Relic instrumentation in the Heroku instance

Tweet when lots of cars available to refuel

Figure out what number of vehicles it makes sense to tweet for and/or if we want a radius.

Maybe do DMs to subscribers for specific areas?

Will require Twilio.

decommission application

API is going away so we'll want to shut down stuff.

Dear car2go Open API User, 
As a user of our Open API you know that for several years car2go has made our vehicle positions available via an API that can be accessed via a simple authentication.    You are likely also aware that  we are consistently trying to improve our members experience and evolve our business model.    Therefore, I am communicating to you a planned change regarding the access to our vehicle positions via our car2go Open API.  
First we plan to deprecate the GET vehicles interface as of end of this year, documentation found here (https://github.com/car2go/openAPI/wiki/v2.1_vehicles).    The interface will be replaced with a new interface that will allow for the retrieval of the vehicles real time and not with a delay as was the case with the Open API.  Due to past abuses where data was used in ways that violate the terms of use for the Open API; we will issue credentials for this new interface more selectively in the past and will require you to reapply for access to the vehicle interfaces.  As a pre-notification, it is unlikely commercial applications and those that aggregate transportation data from multiple sources will be approved to receive access to the new interface; however, access to the other end-points available via the Open API will not currently change.   To apply please send an email to [email protected] and include a the information below.   Additionally, if you no longer need or desire access to the API, please notify us so we can invalidate your current API key(s).
·       Your full name

·       Your e-mail

·       Country you reside in

·       Country(s) the project will used / distributed in

·       Type of project and will the project be used in a commercial application: Do you develop in private/for education or studies/for a company/...

·       Project or product name

·       Project description (what's the intend and goal, which functionality is focused)

·        Target Platform/Technology (Web Application/Mobile App/iOS/Android/Nokia Sxyz/Blackberry/Palm/RichClient/...)

 I know this may not be welcome news to you, but it is a necessary change at car2go and I appreciate your understanding.  If there are further questions you are also welcome to address those to me directly.
Kind regards,

Create script to convert filename to timestamp

When file data is persisted the timestamp created_at is the time the record is persisted -- this value is not useful when attempting to determine when in time the vehicle was at the location. However, we can extrapolate this info from the filename.

99106220,AB8510,-93.31421,44.99523,2016-12-20 15:55:54.443353,twincities,GOOD,GOOD,WMEEJ3BA8EK755985,twincities-20161217_180115

Above is an example where the consuming system was not run for several days, so the time the vehicle locations were persisted is much later than when they were collected.

CircleCI test metadata collection instrumentation

https://circleci.com/docs/test-metadata#metadata-collection-in-custom-test-steps

gem 'minitest-ci', :git => '[email protected]:circleci/minitest-ci.git'

As a developer I want to persist JSON in files in a better format

Currently JSON data from Car2Go is just saved to a file as a string that requires a fair amount of massaging upon read to get to the data structure I want.

[{"address"=>"W 4th St 90, 55102 Saint Paul", "coordinates"=>[-93.097112, 44.944025, 0], "engineType"=>"CE", "exterior"=>"GOOD", "fuel"=>36, "interior"=>"GOOD", "name"=>"AB5102", "smartPhoneRequired"=>false, "vin"=>"HAPPYGOFUNTIME000"}, {"address"=>"Marshal Ave 1831, 55104 Saint Paul", "coordinates"=>[-93.177645, 44.949533, 0], "engineType"=>"CE", "exterior"=>"GOOD", "fuel"=>66, "interior"=>"GOOD", "name"=>"AB5104", "smartPhoneRequired"=>false, "vin"=>"HAPPYGOFUNTIME001"}, {"address"=>"Ford Pkway 1974, 55116 Saint Paul", "coordinates"=>[-93.183036, 44.917769, 0], "engineType"=>"CE", "exterior"=>"GOOD", "fuel"=>18, "interior"=>"GOOD", "name"=>"AB7740", "smartPhoneRequired"=>false, "vin"=>"HAPPYGOFUNTIME002"}]

• Investigate the best format to write the data (to_json?)
• Make sure the read/parser can handle both formats OR (probably better) just make sure to have a downtime release during which all files of the old format are consumed

Parsing from Dropbox fails when compressed file encountered

A zip file got synched to the Dropbox directory causing the process to (somewhat silently) fail. Nothing in the logs, but got a /var/mail/username entry from the job with the stack trace below.

• Figure out why (and fix) lack of logging when error encountered
• Recover and continue processing when error thrown from file parsing

rake aborted!
JSON::ParserError: 795: unexpected token at 'PK^C^D^T'
/home/username/.rvm/gems/ruby-2.2.2/gems/json-1.8.3/lib/json/common.rb:155:in `parse'
/home/username/.rvm/gems/ruby-2.2.2/gems/json-1.8.3/lib/json/common.rb:155:in `parse'
/home/username/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/json/decoding.rb:26:in `decode'
/home/username/projects/freerider_api/app/controllers/concerns/dropbox_persistence.rb:50:in `get_file_data'
/home/username/projects/freerider_api/app/controllers/vehicle_locations_controller.rb:60:in `block (2 levels) in save_from_dropbox'
/home/username/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/home/username/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/home/username/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/home/username/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/transactions.rb:220:in `transaction'
/home/username/projects/freerider_api/app/controllers/vehicle_locations_controller.rb:58:in `block in save_from_dropbox'
/home/username/projects/freerider_api/app/controllers/vehicle_locations_controller.rb:57:in `each'
/home/username/projects/freerider_api/app/controllers/vehicle_locations_controller.rb:57:in `save_from_dropbox'
/home/username/projects/freerider_api/lib/tasks/scheduler.rake:27:in `block in <top (required)>'
/home/username/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `eval'
/home/username/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => consume_dropbox_data
(See full trace by running task with --trace)

As an application owner I want to implement proper logging instead of `puts`ing

As an application owner I want to Import from files into a local database

Since the free Heroku PostgreSQL instance is too small to handle the amount of rows generated I am saving JSON responses from Car2Go in files on Dropbox.

• Import unprocessed files from Dropbox
• Persist data to preferred database
• Mark processed file(s) as processed
• Remove files from importing filesystem

As a developer I want to test cover my rake tasks

http://eftimov.net/test-rake-tasks/

Integrate with IFTT

Looks like I'll need to contact ifttt:
https://docs.google.com/forms/d/1dlH6aA2nXQ4EWfQbxHKueHvfyulFPmInkjk7PabUxF8/viewform

As an analyst I want to use VIN to help ensure that I don't lose track of a car if the license plate changes

VINs are immutable but license plate numbers can change for a vehicle over time.

• Add VIN column to VEHICLE_LOCATIONS
• Add indexes for VIN and VEHICLE columns

Integrate with Zapier

Just for the taste of it: https://zapier.com/developer/

As a cheapskate I want to JSONify my database records, store them somewhere and delete them from the free PostgreSQL instance on Heroku

• There are close to 500 Car2Go cars in the Twin Cities, and I'm polling every hour.
• The free PostgreSQL instance on Heroku only allows 10K records, so that limit is reached within one day.

I still want the data available to me but don't want to pay $9.00/month for a PostgreSQL database with more records (yet).

How to export

https://devcenter.heroku.com/articles/heroku-postgres-import-export

Free hosted PostgreSQL

• https://www.elephantsql.com/plans.html

Requirements

Create a system where:

• The database has ~ 9K records at any time
• Records are deleted from the database after being exported (PostgreSQL export, CSV, JSON, whatever)
• Records are backed up and purged FIFO

As a user I want to poll vehicles once an hour for a given city and persist values

• One job per hour per city
• Persist in VEHICLE_LOCATIONS
• Figure out scheduled jobs on Heroku

As a cheapskate I want to store data in Dropbox

Instead of persisting to DB and then sending to Dropbox, just save JSON in Dropbox, then load from there into local database and do reporting from there.

• Create app account
• Create 'user' account for app to use
• Save results directly to file
• Put files in dropbox

change to dropbox v2 -- v1 no longer live

upgrade ruby

https://www.ruby-lang.org/en/downloads/releases/

Create functional tests for API

https://github.com/brooklynDev/airborne

City names with non-ASCII characters cause failures in Car2Go API calls

København and München have characters that need to be encoded for API calls

Finished processing 711 vehicles for Hamburg
Processing vehicle locations for K\xC3\xB8benhavn
rake aborted!
OpenURI::HTTPError: 400 Bad Request
/Users/eebbesen/.rvm/gems/ruby-2.2.2@frapi/gems/caruby2go-0.0.2/lib/caruby2go.rb:39:in `issue_get'
/Users/eebbesen/.rvm/gems/ruby-2.2.2@frapi/gems/caruby2go-0.0.2/lib/caruby2go.rb:28:in `vehicles'
/Users/eebbesen/proj3/fr_api/app/controllers/vehicle_locations_controller.rb:52:in `poll_and_persist'
/Users/eebbesen/proj3/fr_api/lib/tasks/scheduler.rake:10:in `block (2 levels) in <top (required)>'
/Users/eebbesen/proj3/fr_api/lib/tasks/scheduler.rake:5:in `each'
/Users/eebbesen/proj3/fr_api/lib/tasks/scheduler.rake:5:in `block in <top (required)>'
Tasks: TOP => poll_and_persist_vehicles
(See full trace by running task with --trace)

Multithread processing

Provide ability to multithread processing of files. Need a way to not try and grab the same file from dropbox.

• start at end of list and start of list
• thread on location name
• other?

A day/week/month/year in the life of a car

Map/graph activity for a specific car over a given time period

• car of the week
  ** cleanliness
  ** most used
  ** farthest travelled
  *** sp and mpls

Add a 'riding dirty' tracker

Identify which cars have internal/external cleanliness issues. Try to figure out if there's a pattern.

• add exterior and interior to model
• provide endpoint for report
• allow caller to specify interior, exterior or both
• allow caller to specify time range (e.g. week, month, year)

$ bin/caruby2go v twincities | grep terior | sort | uniq
exterior:: GOOD
exterior:: UNACCEPTABLE
interior:: GOOD
interior:: UNACCEPTABLE

Facebook integration

• Create Facebook account/page
• Post to Facebook when
  • lots of cars ready to fuel
  • select a car for 'week in the life' and post map/stats/etc.

As an application owner I want to minimize database usage

10K limit (free PostgreSQL instance on Heroku) fills up quickly

• de-dupe VEHICLE_LOCATIONS by keeping earliest record and modifying updated_at for cars that don't change location
• export VEHICLE_LOCATIONS records (dropbox, etc.?)
• just run rake task for one location
• migrate to hobby-basic plan on Heroku (https://elements.heroku.com/addons/heroku-postgresql)

refactor for testability

Especially for vehicle locations controller, need to refactor for ease of testing. It's a mess...

As a user I want a descriptive README

Implement monitoring

Currently neither process (Car2Go to Dropbox, Dropbox to database) alerts me when it fails.

Something like Nagios/PagerDuty/etc.

• Not sure how to do alerting from Heroku -- NewRelic?
• Dropbox to database lives on a local machine and can probably be changed so that errors are emailed to an actual address instead of localhost:/var/mail/username

Integrate with Urthcast?

https://www.urthecast.com/developers

Anything interesting in Urthcast API?

Create a Google Now card to display local refuelable cars

Add mixpanel

https://mixpanel.com/free/