Segfault when running fmm about fmm

[cyang-kth]

it may be a degenerate case. Can you check the last line of the mr.txt to find out which trajectory causes that problem? Then they can be visualized on a map to figure out the problem.

[skinkie]

Created a small patch to overcome the segfault.

diff.txt

These are the top lines from my patched code:

id;o_path;c_path;m_geom
368979;115382466,115382466,115382466,115382466,115382466,115886653;;
891358;115908089,115906316,115906316,115910026,115906316,115906316,115906997;;
892581;125970392,115946449,115946449,115946424,115947401,115947401,115947401,115947401,115947630;;
903710;115941752,115939752;;
967974;116121972,116120447,116120469,116120447,116120469,116120469,116120469,116123171,116123171;;
202038;116808621,116790308,116790308,116790308,116790308,116790308,116790308,116790308,116790308,116790308,116840318;116808621,116790308,116840318;010200000011000000B8B0B411840DF9407AEA7EDE8D7D1D4196438B6C910DF9403BDF4F0D8C7D1D41FCA9F1D2A70DF940759318048D7D1D4148E17A14CE0DF9408FC2F5A88E7D1D41CBA145B6E70DF940068195C38F7D1D41355EBA49FE0DF94046B6F3FD8E7D1D41986E1283060EF940CBA145B68E7D1D4108AC1C5A2C0EF9405A643BDF8B7D1D4146B6F3FDF40EF940C3F5285C717D1D41B6F3FDD41A0FF9405839B4C8697D1D41C74B3789310FF9405A643B5F5F7D1D41D7A3703D480FF940F0A7C6CB577D1D411F85EB516A0FF9400E2DB21D527D1D41000000001010F940DF4F8D973B7D1D41DBF97E6A1810F940E7FBA9713A7D1D410C022B873C10F94096438BEC337D1D41A6BF0DAA3E10F940E590B989337D1D41

[cyang-kth]

As the order of edges in the network shp file could be different from mine and yours, I would suggest you run

# find out the number of lines 
wc -l mr.txt

# find the feature that cause the problem, fid starting from 0
ogrinfo wegvakken_renumber.shp -fid NUM_LINES -al

It should export the WVK_ID of the trajectory that causes the problem.

[skinkie]

Shall I just provide my working directory so you can reproduce it?

[cyang-kth]

What is the size of your working directory? You can send it to me by email. cyang(at)kth.se

[skinkie]

Was the data any useful?

[cyang-kth]

I think the problem is that the gps error you used is too small as you use 0.5 but now the unit is meter. It assumes a normal distribution with standard deviation of 0.5 meter but in case that the GPS point is far away from the road, the probability is rather small and results in underflow in the optimal path inference in fmm.

<?xml version="1.0" encoding="utf-8"?>
<fmm_config>
    <parameters>
        <k>4</k>
        <r>100</r>
        <pf>0</pf>
        <gps_error>0.5</gps_error>
    </parameters>
</fmm_config>

I have extracted that first problematic trajectory and the debug shows with 0.5 meter gps error

Viterbi start!
step;from;to;sp;eu;tran_prob;e_prob;cumu_prob
0;127747071;127747071;43.3789;32.2959;0.744508;9.27885e-103;6.90818e-103
0;127747071;127747037;987.845;32.2959;0.0326933;0;0
0;127747071;124480907;66.6335;32.2959;0.48468;0;0
0;127747071;127746785;899.334;32.2959;0.0359109;0;0
0;127746940;127747071;5000;32.2959;0.00645919;9.27885e-103;5.99339e-105
0;127746940;127747037;270.198;32.2959;0.119527;0;0
0;127746940;124480907;273.288;32.2959;0.118175;0;0
0;127746940;127746785;181.687;32.2959;0.177756;0;0
0;127747037;127747071;5000;32.2959;0.00645919;9.27885e-103;5.99339e-105
0;127747037;127747037;18.1256;32.2959;0.561233;0;0
0;127747037;124480907;21.2156;32.2959;0.656912;0;0
0;127747037;127746785;853.916;32.2959;0.037821;0;0
0;124480907;127747071;5000;32.2959;0.00645919;9.27885e-103;5.99339e-105
0;124480907;127747037;921.212;32.2959;0.0350581;0;0
0;124480907;124480907;0;32.2959;0;0;0
0;124480907;127746785;832.7;32.2959;0.0387846;0;0
1;127747071;127747071;11.4288;9.76362;0.854299;6.64191e-204;5.67418e-204
1;127747071;127747037;936.997;9.76362;0.0104201;1.13868e-279;1.18652e-281
1;127747071;124480907;23.2546;9.76362;0.419858;0;0
1;127747071;127746785;861.258;9.76362;0.0113365;0;0
1;127747037;127747071;5000;9.76362;0.00195272;6.64191e-204;1.29698e-206
1;127747037;127747037;916.833;9.76362;0.0106493;1.13868e-279;1.21261e-281
1;127747037;124480907;3.09004;9.76362;0.316485;0;0
1;127747037;127746785;841.094;9.76362;0.0116082;0;0
1;124480907;127747071;5000;9.76362;0.00195272;6.64191e-204;1.29698e-206
1;124480907;127747037;913.742;9.76362;0.0106853;1.13868e-279;1.21672e-281
1;124480907;124480907;0;9.76362;0;0;0
1;124480907;127746785;838.003;9.76362;0.0116511;0;0
1;127746785;127747071;5000;9.76362;0.00195272;6.64191e-204;1.29698e-206
1;127746785;127747037;81.0416;9.76362;0.120477;1.13868e-279;1.37184e-280
1;127746785;124480907;91.6008;9.76362;0.106589;0;0
1;127746785;127746785;5.30283;9.76362;0.543121;0;0
Find last optimal candidate
Back tracking
Segmentation fault (core dumped)

After setting the GPS error to 50 meters, the problem is addressed

Viterbi start!
step;from;to;sp;eu;tran_prob;e_prob;cumu_prob
0;127747071;127747071;43.3789;32.2959;0.744508;0.97678;1.69252
0;127747071;127747037;987.845;32.2959;0.0326933;0.889398;0.99438
0;127747071;124480907;66.6335;32.2959;0.48468;0.884968;1.39423
0;127747071;127746785;899.334;32.2959;0.0359109;0.323589;0.976923
0;127746940;127747071;5000;32.2959;0.00645919;0.97678;0.606027
0;127746940;127747037;270.198;32.2959;0.119527;0.889398;0.706024
0;127746940;124480907;273.288;32.2959;0.118175;0.884968;0.704299
0;127746940;127746785;181.687;32.2959;0.177756;0.323589;0.657237
0;127747037;127747071;5000;32.2959;0.00645919;0.97678;0.601456
0;127747037;127747037;18.1256;32.2959;0.561233;0.889398;1.09431
0;127747037;124480907;21.2156;32.2959;0.656912;0.884968;1.17649
0;127747037;127746785;853.916;32.2959;0.037821;0.323589;0.607386
0;124480907;127747071;5000;32.2959;0.00645919;0.97678;0.546786
0;124480907;127747037;921.212;32.2959;0.0350581;0.889398;0.571658
0;124480907;124480907;0;32.2959;0;0.884968;0.540477
0;124480907;127746785;832.7;32.2959;0.0387846;0.323589;0.553027
1;127747071;127747071;11.4288;9.76362;0.854299;0.954294;2.50778
1;127747071;127747037;936.997;9.76362;0.0104201;0.93779;1.70229
1;127747071;124480907;23.2546;9.76362;0.419858;0.91508;2.07673
1;127747071;127746785;861.258;9.76362;0.0113365;0.408435;1.69715
1;127747037;127747071;5000;9.76362;0.00195272;0.954294;1.09617
1;127747037;127747037;916.833;9.76362;0.0106493;0.93779;1.10429
1;127747037;124480907;3.09004;9.76362;0.316485;0.91508;1.38392
1;127747037;127746785;841.094;9.76362;0.0116082;0.408435;1.09905
1;124480907;127747071;5000;9.76362;0.00195272;0.954294;1.39609
1;124480907;127747037;913.742;9.76362;0.0106853;0.93779;1.40425
1;124480907;124480907;0;9.76362;0;0.91508;1.39423
1;124480907;127746785;838.003;9.76362;0.0116511;0.408435;1.39899
1;127746785;127747071;5000;9.76362;0.00195272;0.954294;0.978787
1;127746785;127747037;81.0416;9.76362;0.120477;0.93779;1.0899
1;127746785;124480907;91.6008;9.76362;0.106589;0.91508;1.07446
1;127746785;127746785;5.30283;9.76362;0.543121;0.408435;1.19875
Find last optimal candidate
Back tracking
Optimal Path candidate index 2 edge id 127747071
Optimal Path candidate index 1 edge_id 127747071
Optimal Path candidate index 0 edge_id 127747071
Viterbi ends

[cyang-kth]

=============================
MM process finished
Time takes 155.044
Finish map match total points 4999688 and points matched 2724955
Matched percentage: 0.545025
Point match speed:17575.4pt/s
Clean UBODT
Clean UBODT finished
Cleaning network
Cleaning network finished
------------    Program finished     ------------

I also have a rather low matching percentage, I will check the result further.

[skinkie]

I do hope you agree that the code should be guarded not to produce segfaults ;)

[cyang-kth]

I have tried with 3000 meters delta but the matching percentage is still about 60%. I have visualized the result on a map, it seems that in most unmatched cases, the road direction is reverse to the movement. The original information about road direction is not complete.

[skinkie]

Would it significantly improve if you would add all the reverse directions in TOP10NL like I did first? So just add the union with x=y y=x and st_reverse(geom)

[cyang-kth]

In principle, it should work. But now I am busy with some of my own work. I will have a look at it in my spare time.

[cyang-kth]

@skinkie
I have tried with precomputing the reversed network with delta of 2000 meters. The generated ubodt file is about 38GB (903 million rows). I am afraid that it is too big to fit into my memory. I verified the result on a map. It shows that after adding reverse edges, a lot of previously unreached nodes get reached. I am afraid that the algorithm does have limitations to scale to that large road network in a country level.

In practice, I would suggest dividing a large road network into city level to perform map matching separately.

[skinkie]

I have actually done the same. Starting with delta 1200 (89% match), now computing for 2000. I changed the GPS error to 50m. I am running fmm on a 32GB machine, and even for that it is too big. For practical purposes I could run it at a machine of 128GB, but still... you should really rethink how it internally works. I would strongy suggest to look into RoutingKit either use it as inspiration, or join forces :)

[cyang-kth]

The size of UBODT is closely related to the density of road network. The figure below shows that about 2000 nodes are within a distance of 2 km from the center. If it is the average count for all nodes, given a network with 1.8 million nodes, the ubodt will contain 3600 million rows. The precomputation approach cannot do too much if the result is that big.

Actually, the reason to perform precomputation is to avoid repeated routing queries in map matching, which can degrade the performance significantly. In a small or middle network (the one I evaluated in Stockholm may still be small), the precomputation outperforms the others. However, in case of an extremely large network, creating an index on the graph where the RoutingKit may be used. or adding dummy nodes might be a better solution. Further evaluation is still needed.

[cyang-kth]

In my paper of fmm, I have performed some evaluation of ubodt integrated with Dijkstra routing. Since no graph index is used and the network is small, the result may not apply to your case. It could be an interesting topic to explore in the future.