map matching becomes very slow when the trajectory is long (contains many points) about fmm

[cyang-kth]

@zhuang-hao-ming
I have pushed the parallel computing version as ubodt_gen_omp. Currently only csv format is supported.

[cyang-kth]

@zhuang-hao-ming
I have never encountered a problem like this. In principle, the number of points within a trajectory should not affect the performance so I guess the reason could be something else.

Would you mind sending me the network and gps data you tested so I can check it further?
My email is [email protected]

[zhuang-hao-ming]

@cyang-kth
Thank you for your reply.
I have sent my data to your email.
By the way, I once implemented a version that uses postgis to select candidate points and networkx to do the shortest distance calculation (with cutoff). The efficiency is similar to the result on the screenshot.
So I expect fmm to have good performance on my data and look forward to your good news.

[cyang-kth]

@zhuang-hao-ming

I think that I have figured out the problem. There are two reasons.

(1) Hashtable problem
The primary one is the nhash configuration setting in your mm_config,xml file.
It defines the number of buckets in the UBODT. In your setting it is 5000.

In your log, there should be one line below

    Number of rows read 34114078
    Estimated load factor #elements/#tablebuckets 6822.82
    *** Warning, load factor is too large.

It implies that each bucket contains 6800 elements on average so each search need to iterate through an array in size of 6800, which is extremely inefficient.

That value should be set as a large prime number.

After I change it to 5178049, the speed is reported as

=============================
MM process finished
Time takes 30.76
Finish map match total points 2276 and points matched 2087
Matched percentage: 0.91696
Point match speed:67.8479pt/s
Clean UBODT
Clean UBODT finished
Cleaning network
Cleaning network finished
------------    Program finished     ------------

With 5000 of nhash, the log on my desktop is

=============================
MM process finished
Time takes 52.565
Finish map match total points 2276 and points matched 2087
Matched percentage: 0.91696
Point match speed:39.7032pt/s
Clean UBODT
Clean UBODT finished
Cleaning network
Cleaning network finished
------------    Program finished     ------------

The speed increases but is still slow because of the second reason.

(2) Large amount of time spent in reading UBODT
Almost all the 30 seconds are spent in reading UBODT, after I report the time after reading input, the log is

=============================
MM process finished
Time takes 30.85
Time takes excluding input 0.07
Finish map match total points 2276 and points matched 2087
Matched percentage: 0.91696
Point match speed:67.6499pt/s
Point match speed (excluding input): 29814.3pt/s
Clean UBODT
Clean UBODT finished
Cleaning network
Cleaning network finished
------------    Program finished     ------------

I think we may need to have a correction on the performance report to really report the matching speed.

[cyang-kth]

Some notes on improving the performance:

1. Try binary ubodt instead of csv, which takes much shorter time to read input
2. Run parallel program postfixed with _omp which is likely to substaintally improve the speed of precomputation and map matching.

The parallel map matching function is recently added to the repo.

I have also implemented a parallel ubodt precomputation program but I need to test and document it. I will upload the codes in the near future.

[zhuang-hao-ming]

Thank you very much for your efforts.
What a detailed explanation.
I will experiment on my own and then feed back the results.

[zhuang-hao-ming]

Thank you very much for your help.
After changing nhash, performance has improved dramatically.

MM process finished
Time takes 89.843
Finish map match total points 414949 and points matched 396610
Matched percentage: 0.955804
Point match speed:4414.48pt/s
Clean UBODT
Clean UBODT finished
Cleaning network
Cleaning network finished

However, my speed is not as fast as yours.
I think this may be due to my compiler under the windows operating system.

But this speed is already acceptable.
😃