trynmaps / metrics-mvp Goto Github PK

Working on it :)

we can also get rid of the jquery then

/(home): splashscreen with "basic details" like route, state, stop
/route/{route}: optional advanced filtering with graphics
/route/myroute?/{token}: specifically tailored route with graphics, filtering

install react router

Currently, in metrics-api.py, the /metrics endpoint is an unholy giant function. Can we separate that into separate smaller functions (or at least pull out some code into helper functions)?

• modify the event payload to identify what the option belongs to (something like {route: 14} or {routes: [8, 9]})
• use e.target.value instead of e.target.textContent
• clear/reset options
• remove other options when one option is selected (or maybe not for select lists that allow for multiple options to be selected)

react/graphql/node stack

• working state with scaffolding (hello world)
• basic component structure (App, Control Panel, output component)
• query builder with updating/refetching
• displaying the json from the query in the output component

get to a state where json from relevant data will be ported to the output component.

for example, the control panel will have a bunch of options/filters that will be used to narrow a search.
given a search, return the json associated with the constraints.

ability to have more than one date and add time ranges

This data (like cached point-to-point travel times) could be useful in this MVP.

it runs locally but we arent sure if it works on heroku yet

The files are large so it might help to ignore them. But if our server needs it, it might be more convenient to keep tracking.

See the mockup here: https://docs.google.com/document/d/1E6rPTom6rkc9QKQCriGeYL2Aqoo4dan6eCNMxYMENyI/edit#

Should make these time ranges a constant

Nothing to see here folks

Handle state with Redux and Redux-First Router

look up the actual timetable stop ID via the route config since it may not always be "1" + Nextbus stop tag
See https://github.com/trynmaps/metrics-mvp/pull/57/files/e4bea26eee9511c887baccb90cf6f3f4fa3401cb#diff-bb76665c29abb732e3d6a1d469444ca1

React 16.8 should be coming out soon which should include Hooks.
I think it would be worth having the team write new components with hooks
especially since we're building a SPA without too much state.

Currently, you have to re-run yarn build each time you make a change to React (because on starting the server, you run yarn build to make a static folder). Not ideal. Can we make yarn auto-rebuild each time you change react?

When pulling stop information for a route from nextbus, there will sometimes be stops with no associated direction. For example, on route KT, stop 7058 has no associated direction:

http://webservices.nextbus.com/service/publicJSONFeed?command=routeConfig&a=sf-muni&r=KT&t=0&terse

Searching for 7058 finds the stop info (name, lat/lon, etc) for stop 7058 and 17058 (the same stop, but for different directions) but those stops don't show up in the list of stops for each direction. This will happen for stops with only one direction as well - stop 4509 on route L is an example:

http://webservices.nextbus.com/service/publicJSONFeed?command=routeConfig&a=sf-muni&r=L&t=0&terse

I don't know the exact number of stops where this happens, but I'd estimate that 30% of routes have at least one stop with no associated direction; I can find the exact number later.

This will make development much easier. Explain the desired inputs and outputs.

Borrow @ckrajewski's stop-selection-on-a-map and draw the speed in the lines between stops.

In #111, we replaced the favicon with the OpenTransit logo. But it looks really bad when scaled down to 16x16. We should make a proper 16x16 logo with a pixel art maker. (Same with 32x32.)

Maybe that will help them play together more nicely.

The 311.org API only provides timetables for a portion of the stops on each route. Without a timetable to compare actual arrival times to, it will be more difficult to evaluate how close buses are keeping on schedule at a particular stop. If we can't get this information from another source, here are a few possible workarounds:

• for each route, SFMTA gives projected service frequencies, as well as the first and last trip for each day of the week (ex/ https://www.sfmta.com/routes/10-townsend). For any stop whose timetable isn't on 311, we can just use this information to generate a projected timetable. This would be simple, but I suspect it wouldn't be very accurate.

• given the timetables that we already have access to, we could interpolate the timetables for the other stops by adding the projected time between stops to the stop with a known timetable. Since stops are different distances from each other, this feels like it would produce more accurate timetables, but I don't know how we would obtain the projected time between stops - using the data we have now doesn't make sense since what we want to do is compare that data to what is projected, as opposed to comparing the data to itself.

@jtanquil What repo is that json in? Or is it from the API?

The mockup shows a bar chart (and the same data in a line chart with time of day on the x axis) of how wait time and travel time vary across different parts (am rush/midday/pm rush, etc.) of a day. Since we're currently showing data for one time range of one day, the next incremental step is to make a chart of multiple time ranges for one day. It is a slightly different workflow from what we have now, though, so we may need UI design changes to support this.

Currently we show the grade for a route given the to and from stops. But people keep asking us, "which is the best/worst route?" So we need to prototype out a way to give a grade for a route without having to specify the to and from stops.

I imagine this would be based on just the average speed and wait time for the route, across all stops, across all trips of the day.

Let's put this as a card in the MVP (so that in addition to the card about the grade for the specific stops, we also have a card about the grade for the route overall).

• Flask "Hello, World" app
• Simple demo RESTful API endpoint
• Homepage queries this API endpoint and shows information on screen

Gitpod currently defaults to using Python 2.7 but a lot of our Python code uses Python 3. How do we make Gitpod default to using Python 3?

Noticed that metrics-mvp doesn't specify an open source license yet. Other projects in the trynmaps organization use Apache 2.0 or MIT license. Either of those seems fine to me. Anyone have a preference?

For most lines (e.g., the 12 or 27), the travel time from the first stop to any other stop (e.g., the second stop) is not accurate. The histograms show a wide spread of travel times when in actuality the spread should be narrow (as it is from the second stop to the third stop).

Don't just dump users on the main page. Have some explanation of who we are or some system-level, high-level metrics (like www.busstat.nyc and la.railstats.org).

This will make it much easier to build APIs going forward. Right now the APIs are mixed in with app initialization code and some business logic. Ideally we would move the business logic (namely, everything that /metrics uses) and the business logic code out somewhere else.

fyi @exxonvaldez @vicky11z

While we're waiting on #89 to get merged, let's start making graphs (like those on www.busstat.nyc) that show how the scheduled travel times stack up to the actual travel times. Use fake data to make these graphs and let the PR sit until #89 is merged; then we can plug in the real data.

Add support for showing different route info like:

-Speed

Need to add agency and any test dates (i.e. date(2019, 4, 8)) to constants.py

Muni timepoint data (time vehicle passed by a stop) can be found here: s3://muni-timepoint-avl-data/muni_timepoint_data_fall_2018.zip

Each month (total of 3) is about 100 MB uncompressed CSV.

It's the same data Muni uses for some of their metrics here: https://www.sfmta.com/performance-metrics

Need to draw lines through circles (stops) on map

As Jose mentioned in #89 , we need to archive GTFS data because it updates regularly.

For example, between May 24 and June 11, the F outbound direction id changed from F____O_E00 to F____O_F00, with a new terminal ("Castro" instead of "17th and Noe") and new last stop id (33311). So attempts to show anything about F outbound trips will not find any data . Even if we add logic to fall back on older direction ids, we still have the problem of the stop list being out of sync also.

As Jesse suggested in #132 : For production, we should probably have nginx in front of flask to gzip and cache responses to reduce bandwidth and reduce load on the flask backend. Don't need to implement that now though. Vickie H also suggests Envoy (https://www.loggly.com/blog/benchmarking-5-popular-load-balancers-nginx-haproxy-envoy-traefik-and-alb/)

Also, if the changes for #132 have no issues and can become permanent, then there is some refactoring and cleanup as described there. Can clone this issue as desired for that.

See bit.ly/opentransit-mvp.

should we enable cors?

this involves:

• flask cors being functional
• js cors being functional
• security concerns (which methods should we make public and stuff)

i noticed a few things with the procfile and at first glance there might be some bugs introduced here

the app is running on a flask dev server serving the static files - maybe we can use nginx to do this for us? we can also feed heroku nginx configs to fine tune the location rules. also, we can choose ports to listen on etc.

npm start: i don't know if this is ideal on a production server. i could be mistaken but isn't this script used for development? as in part of your workflow: change a file and the hot reload job reflects the change. an alternative could be to npm build -> let the web server know where the files are hosted (i think like /app/www/<static_folder>)

Same chart in React-Vis vs react-d3-components, trying to match NYC bus stats as closely as possible just as a test:

In creating a new "phases of the day" chart, I came across these issues with react-d3-components:

• Tooltips when hovering over charts do not work.
  • Possibly due to conflict with react-bootstrap CSS.
  • Could not find a good fix, seems hard to override third party packages.
• Documentation is not available, but there are good set of examples.
• Customization options are limited.
  • Example: cannot remove axis lines.
  • The original (non-React) D3 API allows clients to remove and modify svg elements it creates, but here we cannot do that.
• No built in support for chart legends.
• Last pull request merged June 2018, project is looking for maintainers.

I looked for articles about popular React charting packages:

• https://medium.com/dailyjs/data-visualization-libraries-for-react-developers-in-2019-a2b9c01262f8
  • (Victory, React-Vis by Uber)
• http://voidcanvas.com/top-10-react-graph-chart-libraries-with-demo/
  • (ReCharts, Victory, VX, React-Vis, Nivo, ChartJS, Rumble, highCharts, Easy Chart, fusionCharts)
• https://www.overloop.io/blog/top-5-react-chart-libraries
  • (Nivo, React-vis, Victory, ReCharts, ChartJS)

Of these, React-Vis looked good visually. It offered more customization options (and correspondingly more complex component hierarchy). It uses D3-inspired scales.

I tried reimplementing a chart in React-Vis. Worked pretty well. Issues:

• Documentation is sometimes out of date.
• Link to view source of examples is sometimes missing.
• Some bugs. For example, height of legend lines is not actually settable due to some hardcoded css. My mistake, included wrong version of react-vis css file.