alessiogambi / asfault

Evolutionary test case generation for simulation testing of autonomous vehicles.

AsFault

WIP

AsFault is a tool to generate test cases for self-driving cars. The tests are executed in a simulator called BeamNG. Currently, the test cases aim at testing the "lane keeping" functionality of the self-driving cars, that is, the ability of the self-driving car of driving on its own lane. Each test consists of one or more roads on a fixed-size map (whose size can be specified), and the goal of the self-driving AI agent is to follow a given and predetermined path from an initial position to a goal position. A good test is one that creates multiple situations where the self-driving car drives off the lane. We call such situations out-of-bounds examples (OBE).

The tests are designed and executed in different phases. The tests are first designed using poly-lines. Then AsFault creates the code needed for the simulator, BeamNG, to simulate the driving situations, execute the tests and collect, report and visualise the results (that is, the number of times the car drove off the lane).

There are several details that need to be explained. In particular, we need to specify how the poly-lines are exactly generated, which path is chosen for the AI to follow and how to create effective test cases (that is, test cases that cause the self-driving car to drive off its lane).

Prerequisites

• Windows
• SVN
• Python 3
• pip
• virtualenv (or Anaconda)

BeamNG.research Installation

Given that AsFault uses the simulator BeamNG.research, we first need to download it. To do it, we can use the command: svn checkout https://projects.beamng.com/svn/research (which thus requires SVN to be installed on your system). I recommend that you checkout this repository inside C:\Users\<You>\Documents. Then you need to set the environment variable BNG_HOME to point to the folder containing the executable of BeamNG.research. To do that, you can follow the instructions in one of the answers to this question https://superuser.com/q/949560. For example, if you downloaded BeamNG.research to C:\Users\<You>Documents\beamng\research, where research is the name of the downloaded repository, then you should have the environment variable BNG_HOME that points to this path.

AsFault Installation

You first need to clone AsFault using e.g. https://github.com/Signaltonsalat/AsFault.git. Then I recommend that you install AsFault in a virtual environment so that you do not pollute your global Python environment. In the following instructions, I assume that you're inside a virtual environment.

If you're using virtualenv (rather than Anaconda), you will likely encounter a problem while installing the package AsFault in your virtual environment. Specifically, it is possible you will encounter a problem related to the installation of the required (by AsFault) package "shapely", something like "OSError: [WinError 126] module could not be found" or "Command "python setup.py egg_info" failed with error code 1 in ...shapely". In that case, I followed the solution provided in this answer https://gis.stackexchange.com/a/62931. Essentially, you should download the wheel file of shapely that corresponds to your installed version of Python from https://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely, then you should install shapely using e.g. pip install Shapely‑1.6.4.post2‑cp36‑cp36m‑win32.whl (where the file Shapely‑1.6.4.post2‑cp36‑cp36m‑win32.whl should be under the current directory), where, in my case, I am using Python 3.6 (a 32-bit version), but, of course, you will download and install the version of shapely that corresponds to your Python version. If you don't install the version of shapely that matches your Python distribution, you will likely encounter an incompatibility error.

If you're using an Anaconda environment, you should be able to install shapely using conda install shapely.

You can now install the AsFault package. Go inside the cloned repo, and then type pip install -e ., which essentially installs AsFault in your current environment in an editable mode.

Example Execution

You can run AsFault using e.g. asfault evolve bng --seed 1234 --budget 2 --show --render. You should run the mentioned command from inside the AsFault folder (otherwise you will get errors). See the file src/asfault/app.py for more details regarding these options. The options are specified using the Python module click.

asfault's Issues

AsFault should compute OBE Intensity using Polygons

At the moment we use an approximation based on triangles, but we can do the following instead:
Create a list of coordinate pairs and pass that to the Polygon constructor, then Polygon.area is the area.

Finding those coordinates should be: 1) take OBE trace 2) intersect with lane edge 3) take OBE trace points between intersection points 4) take lane edge points between intersection points 5) make list of the resulting OBE trace points, an intersection point, the lane edge points, the last intersection point.

More at: https://shapely.readthedocs.io/en/stable/manual.html#polygons

AsFault should not create roads that cross the map boundaries and come back into the map

Instead, it should clip the roads to the map or define a buffer around the map.

AsFault should use Logical/Simulation time to trigger test timeouts

We should check the logical time not the real-time while testing external AIs

About running the code

Hello, I am Professor Joshua Garcia's student who is trying to run AsFault tool. But I met a problem and hope to get your help. When I run "asfault evolve bng --seed 1234 --budget 2 --show --render". It said no --budget options to use. May I know how to solve this problem?

How to run the code?

Please explain in readme how to run the code in discrete steps. That would help a lot.

Error in running the "Example Execution" the last step to run the setup:::

The new concise and user-friendly installation guide for AsFault should be released here as soon as possible, it seems that the error is encountered in 2019 as discussed in #27, and is still there.
we also encounter the error when running
asfault evolve bng --seed 1234 --budget 2 --show --render
as well as
asfault evolve bng --seed 1234 --generations 2 --show --render
looking forward to the new installation guide
Regards

Road Creating with the Specific Lane Segments

I want to create the Road of 3 lane segments. Can somebody please tell me in which file i could add this information.

'info.json' file is missing,how to fix it?

First， I use the command of ’asfault evolve mock --generations 2 ‘, which generates map of the road.
Then execute the 'asfault replay run-test C:\Users\2020.asfaultenv\output\tests\00000026.json' command, the BeamNG console prompts that the info.json file is missing, I try to copy the 'asfault.json' file according to the prompt information put it under 'levels/asfault/' and rename it to 'info.json', where {{prefab}} is already the correct paththe，but result shows 'info.json' parse error.
So I want to know why the 'info.json' file is missing and how to fix it?
Thank you all for your help!

AsFault should automatically listen for incoming connections from BeamNGpy

When using the ext configuration AsFault should configure BeamNG to listen for external commands.

AsFault should use the simulation time to trigger timeout

AsFault should consider the simulation time for triggering the timeout during test execution, otherwise, in case sync simulation is in place, the timeout will trigger too soon.

Cannot start BeamNG

The code of https://github.com/billbos/CPS-SORTER requires AsFault at version 430fe9ba0a4dafb174616bbe0babefb3f0e4aca3. If I run AsFault with asfault evolve bng --seed 1234 --time-limit 2 --show --render then it fails.

My system:

• Windows 10 Education
• Python 3.7.9
• BeamNG.research v1.7.0.1
• $BNG_HOME points to BeamNG folder without whitespaces in path according to (see last line of shell output)

I am wondering if this might be due to some version incompatibilities between AsFault, BeamNG.research, BeamNGpy etc.

Any ideas how to solve this issue?

Below you find my shell output:

Issue with BeamNG when using AsFault as library

When using AsFault as a library the method test_from_network makes problems because of the line beam_test = generate_test_prefab(test). Commenting this out works as a workaround.

Crossover and Mutation

I analyze the cross over and mutation files but unable to grasp the following concept. I don't understand how crossover and mutation of real numbers performed by AsFault.

For instance
chromosome A: 2|5|3|9|6
chromosome B: 1|4|3|8|6

1. How do you restrict the value of each gene between 0 and 9?
2. how did you configure out the crossover and mutation probability.

Segment long DecalRoads into multiple to prevent groundplane clipping

AsFault should not enable any specific search operator by default

Currently, the used of the merge join operator is hardcoded in the code, while this should be configurable (with associated probabilities)

Enable the collection of more data during the experiments

For example, in some cases accessing GForces might be useful. Maybe it is worth considering using the same abstractions provided by BeamNGpy inside asfault. Definitively the asfault.lua code needs to be revisited

Same velocity and position states

AsFault is writing the same value for the x,y,z velocity and position states into the json test case.

write_scenario_prefab does not create the scenarios folder and this causes a problem

In the function write_scenario_prefab (in the file beamer.py) does not create the folder that will contain the scenarios, but this causes a problem when attempting to render the template at the line

prefab = TEMPLATE_ENV.get_template(PREFAB_FILE).render(...)

given that write_scenario_prefab is called before write_scenario_description, which, on the other hand, attempts to create the folder scenarios (under the user directory) using

    if not os.path.exists(scenarios_dir):
        os.makedirs(scenarios_dir)

The function bng in app.py is not correctly passing the command-line arguments to the experiment function

In the function bng (in app.py), the arguments are NOT being passed to the correct parameters of the experiment function. In fact, the show argument is being passed to the render parameter and the render argument to the time_limit parameter.

In fact, because of this bug, no plots are showing during the experiments.