bethesirius / chosuntruck

Euro Truck Simulator 2 autonomous driving solution

ChosunTruck

Introduction

ChosunTruck is an autonomous driving solution for Euro Truck Simulator 2. Recently, autonomous driving technology has become a big issue and as a result we have been studying technology that incorporates this. It is being developed in a simulated environment called Euro Truck Simulator 2 to allow us to study it using vehicles. We chose Euro Truck Simulator 2 because this simulator provides a good test environment that is similar to the real road.

Features

• You can drive a vehicle without handling it yourself.
• You can understand the principles of autonomous driving.
• (Experimental/Linux only) You can detect where other vehicles are.

How To Run It

Windows

Dependencies

• OS: Windows 7, 10 (64bit)

• IDE: Visual Studio 2013, 2015

• OpenCV version: >= 3.1

• Cuda Toolkit 7.5 (Note: Do an ADVANCED INSTALLATION. ONLY install the Toolkit + Integration to Visual Studio. Do NOT install the drivers + other stuff it would normally give you. Once installed, your project properties should look like this: https://i.imgur.com/e7IRtjy.png)

• If you have a problem during installation, look at our Windows Installation wiki page

Required to allow input to work in Windows:

• Go to C:\Users\YOURUSERNAME\Documents\Euro Truck Simulator 2\profiles and edit controls.sii from

config_lines[0]: "device keyboard `di8.keyboard`"
config_lines[1]: "device mouse `fusion.mouse`"

to

config_lines[0]: "device keyboard `sys.keyboard`"
config_lines[1]: "device mouse `sys.mouse`"

(thanks Komat!)

• While you are in controls.sii, make sure your sensitivity is set to:

 config_lines[33]: "constant c_rsteersens 0.775000"
 config_lines[34]: "constant c_asteersens 4.650000"

Then:

• Set controls.sii to read-only
• Open the visual studio project and build it.
• Run ETS2 in windowed mode and set resolution to 1024 * 768.(It will work properly with 1920 * 1080 screen resolution and 1024 * 768 window mode ETS2.)

Linux

Dependencies

• OS: Ubuntu 16.04 LTS

• OpenCV version: >= 3.1

• (Optional) Tensorflow version: >= 0.12.1

Build the source code with the following command (inside the linux directory).

make

If you want the car detection function then:

make Drive

Then:

• Run ETS2 in windowed mode and set its resolution to 1024 * 768. (It will work properly with 1920 * 1080 screen resolution and 1024 * 768 windowed mode ETS2)
• It cannot find the ETS2 window automatically. Move the ETS2 window to the right-down corner to fix this.
• In ETS2 Options, set controls to 'Keyboard + Mouse Steering', 'left click' to acclerate, and 'right click' to brake.
• Go to a highway and set the truck's speed to 40~60km/h. (I recommend you turn on cruise mode to set the speed easily)
• Run this program!

To enable car detection mode, add -D or --Car_Detection.

./ChosunTruck [-D|--Car_Detection]

Troubleshooting

See Our wiki page.

If you have some problems running this project, reference the demo video below. Or, open a issue to contact our team.

Demo Video

Lane Detection (Youtube link)

Lane Detection + Vehicle Detection (Youtube link)

Todo

• For better detection performance, Change the Tensorbox to YOLO2.
• The information from in-game screen have Restrictions. Read ETS2 process memory to collect more driving environment data.

Founders

• Chiwan Song, [email protected]

• JaeCheol Sim, [email protected]

• Seongjoon Chu, [email protected]

Contributors

• zappybiby

How To Contribute

Anyone who is interested in this project is welcome! Just fork it and pull requests!

License

ChosunTruck, Euro Truck Simulator 2 auto driving solution Copyright (C) 2017 chi3236, bethesirius, uoyssim

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

chosuntruck's Issues

Should improve lane detection algorithm especially at the curvy lanes

Our lane detection code does not work properly at the very curvy lanes like highway entrance or exit.
I'm reading some papers which can improve our lane detection algorithm.
https://arxiv.org/pdf/1501.03124.pdf
http://research.ijcaonline.org/volume122/number13/pxc3905011.pdf

Linux build error with opencv2

nils@gwendolene:~/ChosunTruck/linux$ make Drive pip install runcython Requirement already satisfied: runcython in /home/nils/.local/lib/python2.7/site-packages (0.2.5) Requirement already satisfied: Cython>=0.10 in /home/nils/.local/lib/python2.7/site-packages (from runcython) (0.29.2) makecython++ tensorbox/utils/stitch_wrapper.pyx "" "tensorbox/utils/stitch_rects.cpp tensorbox/utils/hungarian/hungarian.cpp" /home/nils/.local/lib/python2.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/nils/ChosunTruck/linux/tensorbox/utils/stitch_wrapper.pyx tree = Parsing.p_module(s, pxd, full_module_name) nils@gwendolene:~/ChosunTruck/linux$ ^C nils@gwendolene:~/ChosunTruck/linux$ cd / nils@gwendolene:/$ ^C nils@gwendolene:/$ cd /home/nils/ChosunTruck/linux nils@gwendolene:~/ChosunTruck/linux$ make mkdir -p build g++ -c src/getScreen_linux.cc pkg-config opencv --cflags --libs-std=c++11 -lX11 -Wall -fopenmp -O3 -march=native -o build/getScreen_linux.o Package opencv was not found in the pkg-config search path. Perhaps you should add the directory containingopencv.pc'
to the PKG_CONFIG_PATH environment variable
No package 'opencv' found
mkdir -p build
g++ -c src/main2.cc pkg-config opencv --cflags --libs -std=c++11 -lX11 -Wall -fopenmp -O3 -march=native -o build/main2.o
Package opencv was not found in the pkg-config search path.
Perhaps you should add the directory containing opencv.pc' to the PKG_CONFIG_PATH environment variable No package 'opencv' found src/main2.cc:8:10: fatal error: opencv2/core/core.hpp: Datei oder Verzeichnis nicht gefunden #include <opencv2/core/core.hpp> ^~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. make: *** [Makefile:18: build/main2.o] Fehler 1

it looks like it tries to find a file that doesn't exist in src so can anybody help me please?

LG Revilum

video

Can u make a video of how to install on windows. Thank you

[master] Assertion failure when running in Debug mode

This error occurs at startup while in Debug mode:

OpenCV Error: Assertion failed (src.checkVector(2, CV_32F) == 4 && dst.checkVect or(2, CV_32F) == 4) in cv::getPerspectiveTransform, file ...\opencv_sources\modules\imgproc\src\imgwarp.cpp, line 6353

This makes the program not run in Debug mode. This is a big problem for me, as Debug mode would allow for better debugging.

The program runs fine in Release mode.

Reducing extraneous lines in ChosunTruck [windows]

I have been manually adjusting the findLines parameters and have come up with the following:
One of the most frequent issues I encounter with ChosunTruck's current implementation is the extraneous lines created by the guardrails along the road. Here are my observations and thoughts about this:

1. Changing the deltaRho value had a significant impact on the quality of lines detected. I found the optimal value of deltaRho to be around 0.75-0.80. However, changing this by itself would not be sufficient as the lines would typically disappear when the truck accelerates and the IPM window 'zooms out'. While deltaRho's default value of 1 is sufficient in most cases, it may be worth trying to reduce it to 0.90 - 0.95 to see if that reduces the extraneous lines.

• Consideration: Implementing a version of adaptive IPM like the one found in the linux version may make changing the deltaRho value a viable option.

2. Changing the minLength and maxGap parameters had a small effect on line detection. However, I do believe our current values of 20 for minLength and 120 for maxGap may be sub-optimal. Through my own personal expirementation, I found the optimal range for minLength to be from 5-35 and the optimal range for maxGap to be from 70-75.
3. Changing setMinVote had a significant impact on line detection. It is important to note that changing setMinVote won't be effective by itself. This parameter is entirely dependent on the quality of the pre-proccessed image. I found the optimal value to be between 60 and 65.
4. Guassian blur - I tried replacing cv::blur(gray, blur, cv::Size(10, 10)); with cv2.GaussianBlur(gray, blur, (17,17),0) and I believe it might be more effective at reducing noise. I am still experimenting with this.

I will be creating a related issue regarding the use of the detected lines' angles to devise a way to filter the near-horizontal lines created by the guardrails. I'm also looking at calculating the average slope of the lines to implement cv::fitLine to extrapolate the lines detected. Implementing these features to our project may make the following suggestions above unnecessary.

Switching to adaptiveThreshold

I have found that using adaptiveThreshold works better for lane detection than the standard threshold being used here:

ChosunTruck/windows/src/main.cpp

Line 117 in 8896443

cv::threshold(sobel, contours, 145, 255, CV_THRESH_BINARY);

I used the following code to replace that line:
cv::adaptiveThreshold(sobel, contours, 255, ADAPTIVE_THRESH_GAUSSIAN_C, CV_THRESH_BINARY, 21, -3);

A comparison between my implementation of adaptivethreshold and the standard threshold can be viewed here:
https://www.youtube.com/watch?v=GT3MJb495Ak&feature=youtu.be

• The left output window shows lines being drawn on the contours created by adaptiveThreshold while the right output window shows the standard threshold that we are using currently.
• adaptiveThreshold works in a variety of lighting settings. It even works even when driving at night with the headlights off!
• with the headlights on during the night, adaptiveThreshold can create 'double lines' (which usually doesn't affect steering). This can be adjusted by changing the blockSize and C parameters

Error open No such file run the ./ChosunTruck [OpenCV error message]

Hi, basically this error is a continuation of error issue #17 posted by @gotope.
Previously, I have successfully ran the ChosunTruck on Windows.
Now on Ubuntu 16.04, I had the same problem as @gotope's issue #17 , and ended up here:
~/ChosunTruck/linux$ sudo ./ChosunTruck
OpenCV Error: Assertion failed (0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows) in Mat, file /home/aqmar/OpenCV/modules/core/src/matrix.cpp, line 508
terminate called after throwing an instance of 'cv::Exception'
what(): /home/aqmar/OpenCV/modules/core/src/matrix.cpp:508: error: (-215) 0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows in function Mat

My suspicion right now is in the ETS2 windowed mode size is not appropriate. (I have put the window in the right bottom though).
This is the screenshot:

But I am not sure how to fix this yet. I it possible that my monitor resolution has a role on the error?

[linux] Buffer Overflow Detected

When running "sudo ./auto_drive", a buffer overflow is detected:

==19523== Command: ./auto_drive
==19523== 
*** buffer overflow detected ***: ./auto_drive terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x89fa7e5]==19523== Command: ./auto_drive
==19523== 
*** buffer overflow detected ***: ./auto_drive terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x89fa7e5]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x8a9b56c]
/lib/x86_64-linux-gnu/libc.so.6(+0x116570)[0x8a99570]
/lib/x86_64-linux-gnu/libc.so.6(__strcat_chk+0x5d)[0x8a9887d]
./auto_drive[0x40797a]
./auto_drive[0x402863]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x89a3830]
./auto_drive[0x404189]

Valgrind Info:

Process terminating with default action of signal 6 (SIGABRT)
==19523==    at 0x89B8428: raise (raise.c:54)
==19523==    by 0x89BA029: abort (abort.c:89)
==19523==    by 0x89FA7E9: __libc_message (libc_fatal.c:175)
==19523==    by 0x8A9B56B: __fortify_fail (fortify_fail.c:37)
==19523==    by 0x8A9956F: __chk_fail (chk_fail.c:28)
==19523==    by 0x8A9887C: __strcat_chk (strcat_chk.c:34)
==19523==    by 0x407979: setPath(char*) (in /home/brian/Desktop/ChosunTruck/auto_drive)
==19523==    by 0x402862: main (in /home/brian/Desktop/ChosunTruck/auto_drive)

[master] Need help testing mouse input

Finally got mouse input to (kind of) work. Still needs revisions, but it is something we can work with for now.

See my changes: ajchili@dc021d4

Linux input for reference: https://github.com/bethesirius/ChosunTruck/blob/master/linux/src/main2.cc#L237

SendInput was incredibly difficult to implement so I'm avoiding that for now. SetCursorPos should do what we need it to do.

Noob help

Hey! I am a noob needing help with this project. First, this project is awesome and is teaching me all sorts of stuff w/ OpenCV. You are awesome!!

I am wondering if CUDA 8.0 will work with this project, as that is what supports Ubuntu 16.04. Otherwise I can try to work-around for 7.5

Also, the ubuntu command references files aren't included in the source directory. "g++ -o auto_drive main2.cpp IPM.cpp lineinder.cpp uinput.c" <- those .cpp files aren't included.

Update OpenCV nuget packages

Currently our wiki links to the vc12 (VS2013) nuget package for opencv

https://github.com/bethesirius/ChosunTruck/wiki/Windows-Installation

Use Nuget Package Manager to grab these two packages:

https://www.nuget.org/packages/opencvcuda-release/
https://www.nuget.org/packages/opencvcuda-debug/

However, neither of these packages will work for VS2015 or VS2017. To be compatible with VS2017, we will need to use https://www.nuget.org/packages/opencv.win.native/320.1.1-vs141. The developer of that package claims this will work for VS2015, although I have not tested that specifically. I have used this package to successfully run Chosuntruck on VS2017.

So I'd recommend we drop support for VS2013 and use https://www.nuget.org/packages/opencv.win.native/320.1.1-vs141 from now on.

zlibd1.dll missing

Hi I did a Local Windows debugger on the program in VS 2017 it start but I get the error that the zlibd1.dll is missing and that I need to reinstall the program. I installed de zlibd1.dll in system32 and SysWOW64 but iI still get the error help me :)
This is for windows not linux

Assertion error occurs ONLY at the first run after build

I changed IPM area code at windows/src/main.cpp
This area setting works well in Linux without an assertion error, but in Windows, assertion error occurs.

OpenCV Error: Assertion failed (ssize.area() > 0) in cv::remapBilinear, file ...\opencv\sources\modules\imgproc\src\imgwarp.cpp, line 3868 

The error looks like this. It only occurs at the first execution after the build, and it does not occur after the second execution. The error can be ignored. Although the error occurs with the program stopped window, program runs well after a few seconds. Any ideas to deal with it?

error appeared: "This project references NuGet packages(s) that are missing......"

I was strictly following the instrctions provided in wiki. But an error appeared when I tried to build.

And the options are set to update packages automatically.

please note that I have used the .vcxproj file that founders offered us. And I have set all the required configuration correctly.
What's the reason of this error? How can I fix it?

Cannot build solution different LNKXXXX errors everytime

Hi this is to continue what was happening in the bad magic number in .pyc issue.
I cannot build the solution because i get FATAL ERROR all the time saying different LNKXXXX issues but now it is coming up with different ones that i had before.

Error open No such file run the ./ChosunTruck

Hello,
The compile is OK, but blocked when execute the file.

debian:~/ChosunTruck-master/linux$ ls
build ChosunTruck Makefile src tensorbox

debian:~/ChosunTruck-master/linux$ ./ChosunTruck
error: open: No such file or directory

[master] Buffer Overflow Detected

See the results.txt

I used a program called "Dr. Memory" to find the errors. Lines 140 and 145 are causing the buffer overflow for me. (Note: Results.txt were produced before some changes to code, so line numbers don't match up. However 140 and 145 are definitely the ones causing issues from what I am seeing)

[master] Freezing after input is used in Windows

With the new code suggested in #10, input works and the program can now move the truck. However, it is sending keys too fast, making it hard to accelerate or use other keys while the program is running. I tried adding Sleep() in different locations inside the code, however that only made the program freeze (crash / slow down significantly).

I also tried removing any commands that would be sent when the truck is going straight. This helped some, but didn't fix the problem.

If you want to help me test this, remember to go to C:\Users\YOURUSERNAME\Documents\Euro Truck Simulator 2\profiles and edit controls.sii from di8.keyboard/fusion.mouse to sys.keyboard/sys.mouse

Windows-based ChosunTruck Steering Problem

Hi, thank you for the amazing work!

I have compiled and run the program in Windows, and it can show the output result on the terminal window, Road image output and Lines image output, as you can see on the image I show below:

I think it can produce the output of the steering value (+/- pixel's unit), but it seems the steering output does not control the truck's steer, I stil have use my mouse to control the truck.

I have altered the "control.ssi" file, resized the window size, and set the control setting into keyboard+mouse.
If you know what else do I miss, I will be very happy to hear it.

Thanks!

Virtual key input is blocked in Windows

We used WM_KEYINPUT function and keybd_event funtion to drive the truck automatically, but thevirtual key input with those functions is blocked. Specifically, virtual key input is working on other programs, such as notepad or ms word, but we could not handle the truck with virtual key input.

RuntimeError: Bad magic number in .pyc file

Hi i am using Windows 10 64bit but when i run the .pyc file it comes up with this error all the time but i had to record it and slow it down to 0.05x speed so i could see it but that only just managed to show it so i screenshotted it. It is below. Thanks!

Linux Make Issue

I am getting stumped by this error. Log provided below.
~/ChosunTruck/linux$ make mkdir -p build g++ -c src/main2.cc pkg-config opencv --cflags --libs -std=c++11 -lX11 -Wall -fopenmp -O3 -march=native -o build/main2.o src/main2.cc: In function ‘int main(int, char**)’: src/main2.cc:128:32: error: ‘CV_RGBA2RGB’ was not declared in this scope cv::cvtColor(image, sendImg, CV_RGBA2RGB); ^~~~~~~~~~~ src/main2.cc:128:32: note: suggested alternative: ‘CV_RGB’ cv::cvtColor(image, sendImg, CV_RGBA2RGB); ^~~~~~~~~~~ CV_RGB src/main2.cc:182:44: error: ‘CV_THRESH_BINARY’ was not declared in this scope cv::threshold(sobel, contours, 145, 255, CV_THRESH_BINARY); ^~~~~~~~~~~~~~~~ src/main2.cc:240:8: warning: unused variable ‘degree’ [-Wunused-variable] int degree = atan2 (last_centerline - first_centerline, count_centerline) * 180 / PI; ^~~~~~ src/main2.cc:42:7: warning: unused variable ‘python_on’ [-Wunused-variable] bool python_on = false; ^~~~~~~~~ src/main2.cc:76:10: warning: ignoring return value of ‘int system(const char*)’, declared with attribute warn_unused_result [-Wunused-result] system("python tensorbox/Car_detection.py"); ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Am I doing something wrong here? I followed the steps and correctly installed opencv and tensorflow.

If the log isn't readable, here's a ghostbin link: https://ghostbin.com/paste/9p8h4/raw

Program using your imgwarp.cpp

When running the release x64 exe, I ran into this error:

OpenCV Error: Assertion failed (ssize.area() > 0) in cv::remapBilinear, file C:\Users\bishnois.VITECNET\Downloads\opencv-master\modules\imgproc\src\imgwarp.cpp, line 3832

So I removed line 3832 (because it will run without it) in my imgwarp.cpp, however the error still appears. I don't know where it's referencing your imgwarp and I don't know how to change it so that it uses my imgwarp.cpp instead of yours.

Still working ? Can i run it on Visual Studio 2019 ?

Hy, is this still working ? if yes can i have an example how to build it and run ? i'm not very good with c++ ! Please ? Sorry if sound weird, but i really need it !

Permission to work on this project of yours can pass a readme where to start!!

Hi, is this still working? if so can i have an example of how to build it and run it? For purposes of learning I want to be able to improve or try