mapbox / speed-percentile

Compute percentile speed from speed histogram

speed-percentile

A JavaScript utility to compute percentile speed from speed histogram.

Install

npm install @mapbox/speed-percentile

Test

cd to speed-percentile folder then run

npm test

Use

v1.x

var o1 = percentile(P1, P2, P3)

Computes the p-th precentile speed from a sparse hash speed histogram.

Inputs:

param	data type	description
P1	associative array with integer keys	speed histogram hash {<speed>:<count>}
P2	number or array	one or more percentiles in decimal
P3	string	algorithm flag (optional): 'R4', 'R5' (default)

Algorithms:

• R4 – R's sample quantile Type 4, but with linearly interpolated lower tail
• R5 – R's sample quantile Type 5, but with both tails linearly interpolated

Outputs:

output	data type	description
o1	number or array	corresponding speed(s) sorted in descending order

v2.x

v2.x only has R5 algorithm.

var PercentileInterpolator = require('@mapbox/speed-percentile');

var hist = {10: 2, 30: 3, 40: 4};
var pi = new PercentileInterpolator(hist);  

// find speed from percentile
var speed = pi.getSpeed(0.7);
var speeds = pi.getSpeed([0.1, 0.7]);  // irrespective of input order, output is always in descending order

// find percentile from speed
var p = pi.getPercentile(30);
var ps = pi.getPercentile([20, 30]);   // irrespective of input order, output is always in descending order

speed-percentile's Issues

Make this utility more general

Make this utility more general for computing percentile from any histogram.

Quick explanation of how speed-percentile R4, R5 works

One day I'll put these in words... one day...
_(Edited 2016/07/25: Moved this to wiki.)_

/cc @morganherlocker

Lower tail is not extrapolated.

Compare:

https://github.com/mapbox/speed-percentile/wiki/How-speed-percentile-works

with

> PI = require("@mapbox/speed-percentile")
[Function: PercentileInterpolator]
> h = {20:1, 30:3, 40:2, 50:5, 60:3}
{ '20': 1, '30': 3, '40': 2, '50': 5, '60': 3 }
> pi = new PI(h);
PercentileInterpolator {
  histogram: { '20': 1, '30': 3, '40': 2, '50': 5, '60': 3 },
  n: 14,
  dist:
   { cdf:
      { '20': 0,
        '30': 0.17857142857142858,
        '40': 0.35714285714285715,
        '50': 0.6071428571428571,
        '60': 0.8928571428571429,
        '64': 1 },
     icdf:
      { '0': 20,
        '35714': 20,
        '178571': 30,
        '357143': 40,
        '607143': 50,
        '892857': 60,
        '1000000': 64 } },
  keys:
   { cdf: [ 20, 30, 40, 50, 60, 64 ],
     icdf: [ 0, 35714, 178571, 357143, 607143, 892857, 1000000 ] },
  midpt: { cdf: 50, icdf: 357143 } }
> pi.getSpeed(0)
20
> pi.getSpeed(1)
64

The lower tail of the histogram is not extrapolated. This means that speed-estimator will never output a speed lower than the lowest observed speed. If there is sampling bias and we do not observe low speeds, this may give speed estimates a positive bias. However, the target percentiles for which this behavior would apply are very low, probably lower in most cases than the target percentiles used in production. Therefore, it's unclear how much effect on production speeds this bug has. It can only affect speed estimates that are equal to the lowest observed speed, so we can measure the potential scope of this issue directly.

cc @RuggeroAltair @camilosanin @angelanavarro @s-heisler

Computing speed percentile from daily increment histograms

Goal

Compute the p-th percentile speed for an edge from its speed histogram.

Input Data

Increment JSON files outputted by speedracer.

Example:

{"a":11111,"b":11111,"increments":{"27":1,"28":1,"29":1,"40":1,"42":1,"47":1,"51":1,"56":1,"58":1,"59":2,"60":1}}
{"a":22222,"b":2222,"increments":{"26":1,"28":1,"29":1,"41":1,"50":1,"51":1,"59":2,"60":1,"63":1}}

Output

CSV files of the p-th percentile speed for each edge.

Schema:

column	description
nodeA	starting node ID
nodeB	ending node ID
speed	p-th percentile speed [kph]

To-Do

• for a single JSON file: parse histogram
• confirm with @morganherlocker: histograms from speedracer are in buckets of 1 kph
• compute percentiles
  • for a single edge: compute the p-th percentile
• write out profile CSV

/cc: @morganherlocker