Some question about "Val14" evaluation about tuplan_garage

[DanielDauner]

Hi @soyaCat, the scenarios fail during metric computation. We observed this previously in this issue, where the parameter worker=single_machine_thread_pool caused the problem. Have you tried potential fixes for the ray problem, i.e. described in this comment?

The final score is significantly higher. Could you send the scenario tokens where the simulation failed?

[soyaCat]

Thank you for your response
The failed token I found in the terminal window is as follows:

devkit/nuplan/planning/simulation/runner/executor.py:134}  Failed simulations [log, token]:
[2021.07.24.18.06.35_veh-35_00016_03642, 0942e763807c536b] 
[2021.07.24.19.10.14_veh-37_02937_03698, 0f218605b46855a3] 
[2021.06.08.18.59.48_veh-12_03766_03974, 12384a61bce55dfd] 
[2021.07.24.23.50.16_veh-17_06539_06969, 2acb1cd62bbc5f5d] 
[2021.06.08.14.35.24_veh-26_02246_02541, 2ae1a98a4abb5634] 
[2021.07.24.19.10.14_veh-37_00015_01108, 341f2720ef215146] 
[2021.07.24.23.59.52_veh-12_04767_05924, 35c34962ceb55ae1] 
[2021.06.08.18.59.48_veh-12_03122_03677, 392b41b3af975424] 
[2021.07.24.23.50.16_veh-17_00010_00554, 3c2dc4fa5cea5da2] 
[2021.06.08.12.54.54_veh-26_02994_03970, 3c7fc272a0eb5a74] 
[2021.07.24.23.50.16_veh-17_06539_06969, 4a1cedb052d35575] 
[2021.06.07.13.53.57_veh-35_01195_01572, 501e6ffbc107514b] 
[2021.06.07.18.29.03_veh-16_04252_04622, 53f040fc7b815331] 
[2021.07.24.22.53.21_veh-47_01429_03205, 552c8befa3c45662] 
[2021.06.08.17.36.50_veh-26_05134_05378, 5badc98899c35e10] 
[2021.06.08.17.36.50_veh-26_05134_05378, 5fd2c4ba12af5433] 
[2021.07.24.00.58.02_veh-12_01105_01810, 605598ad9db955c1] 
[2021.07.24.22.53.21_veh-47_01429_03205, 6aaaedc401ee5107] 
[2021.07.24.16.51.13_veh-26_00404_00941, 882b5af1243757f1] 
[2021.08.24.14.35.46_veh-45_00715_01404, 8c3efbd29dac55de] 
[2021.07.24.16.07.03_veh-35_00244_01628, 93455e89334950b2] 
[2021.09.14.20.42.30_veh-45_01265_01584, 993b14add9b3537a] 
[2021.06.07.17.49.04_veh-47_00561_01239, 9be7b4ef51ad537b] 
[2021.07.24.19.10.14_veh-37_00015_01108, a64188586ebc5d4b] 
[2021.07.24.23.59.52_veh-12_02884_03403, b09528b71a5451cf] 
[2021.07.24.00.58.02_veh-12_01831_03390, b3e8728bd5ce5340] 
[2021.06.07.18.29.03_veh-16_02679_03723, c3afbe8d5dfc5ec7] 
[2021.09.23.06.47.56_veh-53_00016_00621, c455ca6714a5529d] 
[2021.07.24.23.50.16_veh-17_04580_05245, c7468120b03a56e7] 
[2021.06.07.13.53.57_veh-35_00428_00678, c766017cb6c6548c] 
[2021.07.24.23.59.52_veh-12_07598_08663, c80486444a075315] 
[2021.06.08.12.54.54_veh-26_04829_05317, cea6d0cc7fae5b64] 
[2021.06.07.12.42.11_veh-38_04779_06284, d5fb322ec0b758d3] 
[2021.07.24.16.51.13_veh-26_01887_04395, d9e89ee9ac10598d] 
[2021.06.07.18.29.03_veh-16_02679_03723, ed8c0ae24032517e] 
[2021.06.08.14.35.24_veh-26_01356_01914, f1f8af6bf7685fa3] 
[2021.07.24.20.58.00_veh-35_00016_00776, f47d8b7828245583] 
[2021.07.24.19.10.14_veh-37_00015_01108, fa82f3a838b15805] 

Here is the extracted 'token' from the results obtained in the terminal window:

['0942e763807c536b', '0f218605b46855a3', '12384a61bce55dfd',
       '2acb1cd62bbc5f5d', '2ae1a98a4abb5634', '341f2720ef215146',
       '35c34962ceb55ae1', '392b41b3af975424', '3c2dc4fa5cea5da2',
       '3c7fc272a0eb5a74', '4a1cedb052d35575', '501e6ffbc107514b',
       '53f040fc7b815331', '552c8befa3c45662', '5badc98899c35e10',
       '5fd2c4ba12af5433', '605598ad9db955c1', '6aaaedc401ee5107',
       '882b5af1243757f1', '8c3efbd29dac55de', '93455e89334950b2',
       '993b14add9b3537a', '9be7b4ef51ad537b', 'a64188586ebc5d4b',
       'b09528b71a5451cf', 'b3e8728bd5ce5340', 'c3afbe8d5dfc5ec7',
       'c455ca6714a5529d', 'c7468120b03a56e7', 'c766017cb6c6548c',
       'c80486444a075315', 'cea6d0cc7fae5b64', 'd5fb322ec0b758d3',
       'd9e89ee9ac10598d', 'ed8c0ae24032517e', 'f1f8af6bf7685fa3',
       'f47d8b7828245583', 'fa82f3a838b15805']

[DanielDauner]

Hi @soyaCat, the score of Urban Driver in the reactive simulation is 50.4 with all 1118 samples and 50.6 when excluding the failed scenarios. Thus, the higher score is not only related to the 38 failed scenarios.

However, since the issue originates from nuPlan, there isn’t anything we can fix in our code. As @mh0797 mentioned before, consider opening an issue for the nuplan-devkit.

Sorry that we can’t offer a fix. We will write a follow-up if we find a solution.

I am closing this issue for now.