blahblahblah- / goodservice-v2

New York City Subway Status Page driven by real-time feeds

goodservice-v2

This is a Rails app that generates live route maps, detects headway discrepancies, track delays and compare runtimes to the schedule on the New York City Subway system by comparing the data for live countdown clocks with the static schedule data provided by the MTA.

goodservice-v2 is a re-write of the original goodservice codebase. The goal for this re-write was 1) reduce the time to process feed data to be under 30 seconds at all times, so that data can be refreshed in shorter intervals instead of every minute (data is refreshed every 15 seconds in production), 2) use rolling average of past trips between each pair of stops to predict ETAs of each train to each station, and 3) instead of extracting arrival times in the past for headway calculations, we are keeping track of trips in relation each other in order to figure out the time between trains. With these changes, the app no longer uses the concept of lines, and train arrival times are now available to view via lookup by station.

The biggest change in technology use is using Redis as the primary persistence source, relying on Sidekiq to process data asynchronously, using sidekiq-cron to schedule jobs and a custom written Heroku Autoscaler to scale horizontally when there are more trains running and the job queue is getting too large. Postgres is still used to store static schedule info.

See it live at https://www.goodservice.io. The same set of APIs are used to power https://www.theweekendest.com

Running locally

To run locally, you'll need a couple things. First, the app requires Ruby 3.1.3 and Rails 7.1. We suggest managing this with rbenv. It also depends on Redis, Postgres, Yarn, and Semantic UI React. If you are on a Mac with Homebrew installed, you can get all these requirements with the following commands:

# Ruby dependencies
brew install rbenv
brew install ruby-build
rbenv install 3.1.3
gem install bundler

# Other dependencies
brew install postgresql
brew install redis
brew install node
npm install -g yarn

Next, you'll need to sign up for a developer account with the MTA. To do so, go to https://api.mta.info. You'll get an API key and set it as MTA_KEY env variable.

Finally, you'll need to download the current static schedules from the MTA. Go to https://web.mta.info/developers/developer-data-terms.html, agree to the terms, and then download the data for New York City Transit. (Ctrl+F for "GTFS".). Put this into the import folder and unzip it. From the same site, download the Station Locations file (Stations.csv), and save it into the same directory.

Finally, to run the app locally, do

export MTA_KEY=<<YOUR API KEY>>
bundle install
yarn install
initdb
rails db:reset  # This will take a *very* long time on its first run
rails db:seed
rails db:migrate
foreman start -f Procfile

A brief tour of the code

This is a Rails app that uses React as a view manager. As such, there are a lot of moving components. We briefly explain how data flows and how the code is laid out.

Server side: Rails

Regularly occuring jobs are scheduled with sidekiq-cron, and are defined in the Sidekiq initializer.

The order of operation to process the feeds within the every 15 seconds cycle is: FeedRetrieverSpawningXWorker > FeedRetrieverWorker > FeedProcessorWorker > FeedProcessor > RouteProcessor > RouteAnalyzer

Static schedule data is stored in Postgres and their associated ActiveRecord classes are in the Scheduled namespace. No other data is currently stored in Postgres.

The primary persistence source is Redis. The class RedisStore provides the interfaces used to interact with the Redis client.

Client side: React

The client side view libraries are a React app that is compiled by the webpacker gem. For more information on webpacker, you can see their repository. But the basic summary is that all entry points (in React lingo) live in /app/javascript/packs and are compiled to the /public directory. As with all Rails apps, this is driven by the views in /app/views, which are basically the bare minimum to get the compiled React to appear.

In the middle: An API

The React front end is fed by an API that Rails serves. The routes are specified in the /app/controllers/api directory. Specifically, dynamically-generated routes are specified in /config/routes.rb. There are only 2 endpoints for now, and they're both in the Api::RoutesController class, which serve the data as /api/routes

Supported external services

Twitter

The TwitterDelaysNotifierWorker job to used to check if there are currently delays and TwitterServiceChangesNotifierWorker is used to check for service changes. If so, notifications are sent on Twitter. Make sure the env variables TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_TOKEN_SECRET, TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET are populated.

Alexa

See goodservice-ask.

Other resources

• MTA's GTFS-realtime manual
• Most up-to-date MTA static schedule data

Inspirations

• DC MetroHero
• NYC Subway Stringlines

Subway Route Symbols ®: Metropolitan Transportation Authority. Used with permission.

goodservice-v2's Issues

Public API

I love this project. Nice work!!

Is there any way I can query the live data you're using (to feed goodservice.io) programmatically?

I was hoping for something like api.goodservice.io, where I can get accurate train arrival times for my local station.

Thank you

Type Error breaks trains pages

As of today, the page displaying info on some trains no longer loads for me. Instead I am presented with a solid gray page, and the console outputs the errors included below. This does not affect all trains, and it does not affect page displaying a trains general status (https://goodservice.io/trains/<train_line>/), only pages under the level of the train with direction specified (https://goodservice.io/trains/<train_line>/[NS]/*). The train lines affected are the 4, 5, A, B, D, N, Q, and SIR. I have gotten this error across three browsers and two systems. The log is from Firefox.

Encountered on:

• Debian 10.13
  • Mozilla Firefox 102.14.0esr
  • Chromium Version 90.0.4430.212 (Developer Build) built on Debian 10.9, running on Debian 10.13 (64-bit)
• iOS 15.7.8
  • DuckDuckGo 7.87.0.0

TypeError: n.toReversed is not a function
    o trainModalDirectionPane.jsx:485
    a trainModalDirectionPane.jsx:480
    value trainModalDirectionPane.jsx:477
    value trainModalDirectionPane.jsx:906
    React 8
    unstable_runWithPriority scheduler.production.min.js:291
    React 6
    value trainModal.jsx:57
react-dom.production.min.js:5311:12
    React 5
    unstable_runWithPriority scheduler.production.min.js:291
    React 4
    unstable_runWithPriority scheduler.production.min.js:291
    React 6
    value trainModal.jsx:57
Uncaught (in promise) TypeError: n.toReversed is not a function
    o trainModalDirectionPane.jsx:485
    a trainModalDirectionPane.jsx:480
    value trainModalDirectionPane.jsx:477
    value trainModalDirectionPane.jsx:906
    React 8
    unstable_runWithPriority scheduler.production.min.js:291
    React 6
    value trainModal.jsx:57
trainModalDirectionPane.jsx:485:32