Optimize end to end flow of enabling telenav live traffic for OSRM about osrm-backend

[CodeBear801]

Meeting memo

Topic

Optimize OSRM traffic end-to-end time cost

Summary

• Current osrm-traffic solution need 15 minutes to enable live traffic
  generate traffic.csv(1.5 minutes) -> osrm-customization(10 minutes) -> routed-init(3 minutes)
  Our target is optimize the overall time cost

• Generate traffic takes 15 seconds for communication with traffic server and another 60 seconds to generate traffic.csv needed by osrm. The file contains 75M lines and physical size is 2.6GB

• osrm-customization need 1 minutes to load traffic.csv, 3 minutes to load other data(e2n/n2e/partition/cost), 6 minutes to do custmization

• routed-init need 3 minutes to load data(e2n/n2e/partition/cost)

• We want to use delete strategy to do customization, which means routing service will only take updated speed file except the initial customization. This would decrease time in generate traffic.csv and osrm-customize(load traffic.csv and customization step)

• Keep common data in memory to avoid redundant loading(e2n/n2e/partition)

Actions

[wangyoucao577]

Designed user story mapping, breakdowned to sub issues.

graph original link: https://www.processon.com/diagraming/5d436a87e4b07c4cf2ff3142

[CodeBear801]

Problem description

After #22, now we generating a 2.5GB csv file with 71M records, based on current OSRM's strategy and profile, I think it might need 15 minutes ~20 minutes for a full round of customization, which could be settled for prof of context server but not for production.

It means, we should consider delta strategy from beginning to the end. For OSRM's customization step, a general thinking is, for initial round it will take all the traffic data and generate cost matrix, but for following steps only do patch modification on previous version.

We should do some bench mark on the overall flow then decide how to move forward.

Target

Live traffic update should be in the level of minutes in OSRM part

[wangyoucao577]

Have filed a issue to OSRM team: Project-OSRM#5503

[wangyoucao577]

$ osrm-routed -h
osrm-routed <base.osrm> [<options>]:

Options:
  -v [ --version ]                      Show version
  -h [ --help ]                         Show this help message
  -l [ --verbosity ] arg (=INFO)        Log verbosity level: NONE, ERROR, 
                                        WARNING, INFO, DEBUG
  --trial [=arg(=1)]                    Quit after initialization

Configuration:
  -i [ --ip ] arg (=0.0.0.0)            IP address
  -p [ --port ] arg (=5000)             TCP/IP port
  -t [ --threads ] arg (=8)             Number of threads to use
  -s [ --shared-memory ] [=arg(=1)] (=0)
                                        Load data from shared memory
  --memory_file arg                     DEPRECATED: Will behave the same as 
                                        --mmap.
  -m [ --mmap ] [=arg(=1)] (=0)         Map datafiles directly, do not use any 
                                        additional memory.
  --dataset-name arg                    Name of the shared memory dataset to 
                                        connect to.
  -a [ --algorithm ] arg (=CH)          Algorithm to use for the data. Can be 
                                        CH, CoreCH, MLD.
  --max-viaroute-size arg (=500)        Max. locations supported in viaroute 
                                        query
  --max-trip-size arg (=100)            Max. locations supported in trip query
  --max-table-size arg (=100)           Max. locations supported in distance 
                                        table query
  --max-matching-size arg (=100)        Max. locations supported in map 
                                        matching query
  --max-nearest-size arg (=100)         Max. results supported in nearest query
  --max-alternatives arg (=3)           Max. number of alternatives supported 
                                        in the MLD route query
  --max-matching-radius arg (=-1)       Max. radius size supported in map 
                                        matching query. Default: unlimited.

OSRM already supports both --mmap and --shared-memory.
See Configuring and using Shared Memory for shared memory part.

[wangyoucao577]

We have created new project https://github.com/Telenav/osrm-backend/projects/3 to break down and track issues from this one. Some general discussion will still in this original issue.

[wangyoucao577]

Created #49, #50, #51 below actions after review codes and discuss together:

• action item. profile OSRM
• action item. review customize's part code

[AlexKaravaev]

Hi there!
Found your repo, while was searching for possibilities of integrating real-time traffic into OSRM.
Have you had any luck doing this? Or work stopped at designing stage?

Thank's in advance.

[wangyoucao577]

@AlexKaravaev We have implemented 3 ways to integrate OSRM with traffic:

• customize full region traffic into OSRM data
• customize blocking-only traffic into OSRM data
• alternatives ranking

Here's the design document for them: https://github.com/Telenav/osrm-backend/blob/master-telenav/integration/doc/osrm-with-telenav-traffic.md

We implemented them mostly by Golang, all codes are under https://github.com/Telenav/osrm-backend/tree/master-telenav/integration, you'll be able to read them easily if you're also familiar with Golang. Several binaries have been created by these codes: osrm-traffic-updater, osrm-ranking, etc. You can find these binaries from our built docker: docker pull telenavmap/osrm-backend. You can also build them by yourself if you're interested, all of them can be built by docker: https://github.com/Telenav/osrm-backend/tree/master-telenav/docker-orchestration

We integrate our traffic from an separate traffic-proxy process which is not part of open source. But the protocol is here: https://github.com/Telenav/osrm-backend/blob/master-telenav/integration/proxy.proto. If you can provide similar process with same protocol, then these binaries will work as expect.

[AlexKaravaev]

@wangyoucao577 Thank you very much for this enormous amount of useful information. I will take a really close look on it and look up all the details.
Last question, regarding traffic-proxy and real-time traffic. Is is possible somehow to just feed the osrm stream of live data(like {uuid, lan, lot}) and from this stream get the information about traffic and make routing based on this information?

[wangyoucao577]

@AlexKaravaev Sorry, I'm not very sure about your last question. Do the stream include traffic speeds or not? Could you please make it more clear?

[AlexKaravaev]

@wangyoucao577
Let me elaborate more clearly on my thoughts. I meant, that suppose we have live gps signal incoming. Like the one going from telephone etc. It comes only with a timestamp and Lat,Lon fields available, and we can suppose, that we have speed of the device itself, not the whole street, on which it located.
Is it possible now to make routing based on this kind of data? Or for now, the OSRM fork, that you are guys making needs an input in a form of edge_id,speed(speed of the whole edge, not single device) pairs?

[wangyoucao577]

@AlexKaravaev No. OSRM calculate route based on graph(route networks), the traffic data represents ways' speeds on the graph actually. From the data you said 'gps signal', you have to convert them to ways' speeds, e.g. match the gps location to a way, calcualte speed of the gps signals, then you'll get the way's speed. If you have many many devices, you can calculate speeds of the ways these devices on, which means you'll calculate your own traffic data from these devices. Then you can take these traffic data as OSRM's input, or what I said traffic-proxy's input.

[AlexKaravaev]

@wangyoucao577
Okay,now I completely got you. Thank you very much for clearing it out and for the work you have done, I will definitely check it out!