This repository is home for the required packages and instructions to run VT&R in AirSim
Top-level Contents
- Repository Overview
- Install Unreal Engine
- Install AirSim
- Install AirSim ROS Wrapper
- Setup Python Client
- Setup Unreal Environment
- Install AirSim Interface
- Pre-launch Setup
- Run VT&R in AirSim
- Reference Material
This is a high-level overview of all the contents of this repository.
Pluginsa backup version of the Plugins folder with M600 Implementation, the folder is required to run airsim with Unreal editorairsim_vtr_interfacea catkin package for interfacing airsim with VT&Rscriptscontains some useful bash scriptstmuxpcontains the yaml files required for running VT&R with airsimAirsim_Changes.mda document containing all the changes made to airsim's master branch to implement the DJI M600 in itAirSim_Stereo_Gimbal.mda document containing all the details regarding the custom stereo gimbalsettings.jsonthe required airsim settings file for running VT&Rstereo.yamlthe required parameter file for babelfish_robochunk_translator to accept the airsim topics
To install Unreal Engine, clone the below repository and build it. You'll have to first create an EpicGames Account to be able to access the repository.
cd ~
git clone -b 4.24 https://github.com/EpicGames/UnrealEngine.git
cd UnrealEngine
./Setup.sh
./GenerateProjectFiles.sh
make
To launch Unreal Editor to test installation:
cd ~/UnrealEngine/Engine/Binaries/Linux/
./UE4Editor
The new project Window should open, you can create an empty project to test that the editor works.
Note that Unreal Engine uses Vulkan Drivers, if you are using Ubuntu 16.04 they might not be installed by default. If you receive a warning message "opengl is deprecated please use vulkan" upon launching Unreal Engine, then you need to install vulkan using the below command.
sudo apt install -y libvulkan1 vulkan-utils
Install AirSim from the following forked repository, the forked repository has changes are not available on AirSim's master branch. All the changes are presented in the doc AirSim_Changes.md for reference. The changes are primarily adapting the simulator to work with the hexacopter model of the M600. If you would like to make any changes to the M600 dynamics model, please refer to that doc. Run the following commands to install AirSim.
cd ~
git clone https://github.com/Omar-Elmofty/AirSim.git
cd AirSim
./setup.sh
./build.sh
Once AirSim is built, a "Plugins" Folder will be created under ~/AirSim/Unreal, this folder will be later attached to unreal projects in order to run AirSim with Unreal editor. Note that a backup version of the "Plugins" folder is attached to this repository for reference only.
You'll need gcc >= 8.0.0, check the version of gcc using gcc --version. Run the following commands to install gcc 8 if you don't have it:
sudo apt-get install gcc-8 g++-8
If you encounter issues running the above command, you might need to add the ppa repository as shown below.
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-8 g++-8
You'll also need C-make >= 3.10.0, which is not installed by default with VT&R.
To install the latest C-make version follow the below instructions
Visit https://cmake.org/download/ and download the latest binaries
I used cmake-3.17.3-Linux-x86_64.sh, copy the binary to /opt/
run chmod +x /opt/cmake-3.17.3-Linux-x86_64.sh
run sudo bash /opt/cmake-3.17.3-Linux-x86_64.sh* (you'll need to press y twice)
The script installs to /opt/cmake-3.17.3-Linux-x86_64* so in order to get the cmake command, make a symbolic link:
sudo ln -s /opt/cmake-3.17.3-Linux-x86_64/bin/* /usr/local/bin
Test your results with cmake --version
If the cmake version does not change, then you might need to export paths as shown below
export PATH=/opt/cmake-3.17.2-Linux-x86_64/bin:$PATH
export CMAKE_PREFIX_PATH=/opt/cmake-3.17.2-Linux-x86_64:$CMAKE_PREFIX_PATH
Finally, you'll also need tf2 sensor and mavros packages, install them using the below command:
sudo apt-get install ros-kinetic-tf2-sensor-msgs ros-kinetic-mavros*
To build Ros Wrapper Open a new terminal, export the cmake paths if you need to, then build ros package
cd ~AirSim/ros
catkin build -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8
If you encounter any issues in finding the directories of some of the dependencies, you could set the directories in the CMakeLists.txt of the airsim_ros_pkgs and airsim_tutorial_pkgs packages. Below are some examples of the directories that needed to be set at the time of writing this document, the below lines were added to CMakeLists.txt of both packages, before find_package command.
set(mavros_msgs_DIR "/opt/ros/kinetic/share/mavros_msgs/cmake")
set(geographic_msgs_DIR "/opt/ros/kinetic/share/geographic_msgs/cmake")
set(uuid_msgs_DIR "/opt/ros/kinetic/share/uuid_msgs/cmake")
To setup the AirSim python package run the following
cd ~/AirSim/PythonClient
sudo python setup.py install
Now import airsim should work in a python script.
The different functions available in the python api can be found in ~/AirSim/pythonClient/airsim/client.py
Different Examples of the python client usage can be found in ~/Airsim_Forked/PythonClient/multirotor
To setup an unreal environment, you'll have to download Epic Launcher and Unreal Engine 4.24 on a Windows machine, then transfer the project to Ubuntu. Instructions for downloading Unreal in windows can be found here, note that you don't have to install AirSim in Windows, only Epic Launcher and Unreal Engine are needed.
This video along with this guide have instructions on how to setup the environment in Windows, we'll be following a slightly modified approach since we will be building the AirSim Plugin in Ubuntu rather than Windows. Although it is good to watch the video to gain some background on the setup process.
Once you have download Epic Launch and Unreal Engine 4.24, follow the below steps to setup the environment.
- Open Unreal Editor and create new project.
- Select "Games" then click next.
- Select "Blank" then click next.
- Select "C++" project, and "No Starter Content" as shown in the below image
- Open Epic Launcher
- Navigate to Unreal Engine MarketPlace
- Browse for Free Environments (could use the price filter)
- Select the environment you like, then checkout cart, you'll then see an option called "Add to Project", click on that and select the project you just created. This is going to download the new environment in the project. After the download is done, you should the maps and assets under the "Content" folder of the project.
Navigate to the "Config" folder in your project, and open DefaultEngine.ini, add the following highlighted lines in the below picture. These lines will load the map of the environment you downloaded once the Unreal Editor is launched. Note that the path to the .umap will be different for your project.
- Open the visual studio solution file in your project
- Select "DebugGame Editor" and "Win64" build configuration
- Click "Run" (or press F5), the building process will start and might take several minutes. After the environment is built, the Unreal Editor should automatically open and you should see the Environment loaded (note that shaders might take a while to compile, so you might not see the environment rendered properly right away).
Now you should copy the project over to your Ubuntu machine
- Copy and paste the "Plugins" folder from AirSim (
~/AirSim/Unreal/Plugins), and paste it into your project - Open the
PROJECT_NAME.uprojectfile using a text editor - Add the "Additional Dependencies" and "Plugins" fields, as shown below
- Build your project by double clicking your
PROJECT_NAME.uprojectfile, you should get prompted to whether you want to build your project and the airsim plugin, select yes. The build process will start and might take several minutes, so be patient with it.
- Select "AirSimGameMode" as shown below
- Select a new player start position as shown in the video tutorial
In order to avoid Unreal Engine from dropping in performance while running VT&R, ensure it can run in the background. Go to "Edit -> Editor Preferences"; in the "Search" box, type "CPU" and ensure that the "Use Less CPU when in Background" is unchecked. If this step is not completed, the camera frame rate might drop significantly while Unreal Editor is running in the background, which will create issues while running VT&R.
When downloading some environments, you might get an error saying that "Lighting needs to be re-built". If you get that error, click the Build drop down menu on the main toolbar and click "Build Lighting only".
The airsim_vtr_interface package is a custom Package meant to interface AirSim with VT&R. The package is available in this repository.
Clone the repository
cd ~
git clone https://github.com/Omar-Elmofty/vtr_in_airsim.git
Create a catkin workspace
mkdir -p ~/airsim_interface/src
cd ~/airsim_interface/
catkin_make
Copy and paste package to airsim_vtr_interface into new workspace
cp -R ~/vtr_in_airsim/airsim_vtr_interface ~/airsim_interface/src
Build workspace
cd ~/airsim_interface/
catkin_make
Important Note: the airsim_vtr_interface package depends on the DJI SDK ros wrapper dji_osdk_ros, depending on which version of the sdk you have, the actual ros wrapper name might differ from dji_osdk_ros. If the package name is different, you'll have to change its name in airsim_gimbal_controller.py, airsim_command_drone.py and package.xml.
Also, additional documentation on the custom gimbal which is implemented in airsim_gimbal_controller.py can be found in the doc AirSim_Stereo_Gimbal.md.
Copy stereo.yaml available in this repo and paste in in the babelfish_robochunk_translator package. stereo.yaml contains all the correct names of all the ros topics published from the airsim interface.
cp ~/vtr_in_airsim/stereo.yaml ~/charlottetown/utiasASRL/vtr2/build/deps/robochunk_babelfish_generator/translator/robochunk/src/babelfish_robochunk_translator/param/ros_to_rig_images
Finally, you'll have to copy the settings.json file in this repo to your Documents folder
mkdir ~/Documents/AirSim
cp ~/vtr_in_airsim/settings.json ~/Documents/AirSim
The settings.json contains all the settings required by AirSim to load the drone and its sensors, the file you just copied contains basics needed to run VT&R, however a lot more customization could be applied to the drone. Check AirSim Settings Documentation for more details.
Open m600_backyard.yaml
gedit ~/charlottetown/utiasASRL/vtr2/src/asrl__navigation/param/scenarios/m600_backyard.yaml
Ensure you are using gray-scale images, you could also use color constant images
converter/extraction/conversions: ["RGB_TO_GRAYSCALE"]
converter/extraction/extractor/channels: ["grayscale"]
Open airsim_interface.launch in airsim_vtr_interface, adjust the ros parameters in the file to change the gimbal joint speed, and initial joint angles.
To edit all the pid gains for the drone, open airsim_command_drone.py in airsim_vtr_interface package, and adjust the gains in functions update_anglerate_gains, update_angle_gains, update_velocity_gains, and update_position_gains. Note that the gains are already tuned for the DJI M600.
Open m600_backyard.yaml under ~/charlottetown/utiasASRL/vtr2/src/asrl__navigation/param/scenarios and change the data_dir field to a suitable one.
Open vtr_m600_airsim.yaml under ~/vtr_in_airsim/tmuxp and change the data directory (on line 34) to be the same one you added before. This step will make sure that the directory is emptied out before running vt&r, if you don't want the directory to be emptied out, comment out this line in vtr_m600_airsim.yaml.
The desired image publishing rate from AirSim is 15 Hz, to simulate the output from the ZED camera used on the M600. In order to ensure you get such a frequency on the hardware you have, you might need to slow down the simulation. First you should check the publishing rate of any of the camera topics using the below command once you launch the AirSim interface (steps on how to launch it are explained in the following section):
rostopic hz /airsim_node/drone/front_left_custom/Scene
This command will give you the publishing rate of images from AirSim. Let's say this publishing rate is 5 Hz (which is lower than 15 Hz), then the simulation speed should be set to 0.3, hence the resulting images published per a simulation second will be 15. To change simulation speed, open settings.json and change the field "ClockSpeed".
If the simulation speed (which is set in settings.json) is not set to 1. Then the return speed which is set in dji_baseline_path_tracker.yaml under ~/charlottetown/utiasASRL/vtr2/src/asrl__path_tracker/params will need to be scaled accordingly. For example if the simulation speed is set to 0.25, and if the desired return speed is 2m/s, then the speed set in dji_baseline_path_tracker.yaml must be 0.5 m/s (2 x 0.25). This will results in the vehicle moving at 2 m/s in AirSim. That's because the AirSim Simulator clock will be running 4 times slower than the VT&R clock, hence the VT&R follow speed will have to be scaled down to cope with the slow AirSim clock.
Load the Unreal environment by either double clicking the .uproject file, or by running it from command line as follows:
cd ~/UnrealEngine/Engine/Binaries/Linux/
./UE4Editor PATH_TO_PROJECT/PROJECT_NAME.uproject
Once the Unreal Editor opens, click the Play button on the top bar of the Editor.
Confirm that you are not receiving any errors in the editor, errors will be highlighted in red.
Also, note that the display in the unreal editor is turned off, that is to help decrease the computational load, and hence increase the camera publishing frequency from AirSim. The live feed from the camera will be displayed using rqt_image_view package later on. To turn on display again, you could change the ViewMode in settings.json, check the AirSim settings documentation for more details.
Load airsim.yaml using tmuxp. This will initiate the ros wrapper, the stereo images publishers, the AirSim gimbal controller, and the rqt image viewer.
cd ~/vtr_in_airsim/tmuxp
tmuxp load airsim.yaml
You should now see a window pop up with the live camera feed from airsim.
Launch VT&R by running the following commands
cd ~/vtr_in_airsim/tmuxp
tmuxp load vtr2_m600_airsim.yaml
The vtr2_m600_airsim.yaml is a slightly modified version of the vtr2_m600_backyard.yaml file in asrl__navigation package.
After the file is loaded you should see three panes
- The left pane is running VT&R
- The top right pane is for controlling the drone in airsim by running
./airsim_command_drone.sh - The bottom right pane is for initiating learn or return by running
./learn.shor./return.sh
Start the teach pass, first you'll have to start moving the drone. Navigate to the top right pane as described in step 3, and run the following command:
./airsim_command_drone.sh
The drone now should take off vertically first, then start moving in the horizontal plane in an arc.
After the drone finishes taking off, you can start the teach phase by navigating to the bottom right pane, and running the following
./learn.sh
If you would like to create a different mission plan, you could do that by editing airsim_command_drone.py in airsim_vtr_interface package, there are a handful of functions in there that could be used or edited for commanding the drone. Also, you could check the AirSim API documentation for more details on how to the move the drone.
Once the top right pane (running airsim_command_drone.py) prints the message Initiating Return Phase Control Loop, this indicates that the drone has completed its mission and is ready for return, you could then start the return phase by running the following in the bottom right pane:
./return.sh
Once the top right pane prints Reached End, Hovering this indicates the end of the repeat run.
Follow this tutorial for rendering a hexacopter in AirSim.
AirSim Installation Instructions for Linux
AirSim ROS Wrapper Documentation
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent