It is important to work on a simulated environment rather than going for on-field testing directly to avoid any unwanted damage to property especially when working with aerial robots. The challenges are:

• To decide the appropriate simulation software based on the type of environment.

• To fake the sensor data which is comparable to the real data to a great extent and hence the need to determine the algorithm of Intel Realsense t265 and d435 cams.

• To prepare the appropriate 3D environment or importing existing ones from online databases.

Add a companion computer (RasPi) to the Pixhawk and control the drone manuallyusing commands sent from a ground station joystick connected to a computer.

The drone needs to do object detection to have any real utility. That means finding the 3D coordinates of nearby objects(as many as possible). These detections will then be used to aid in Path planning, since the motion of the detected obstacles can be used to model and predict their state in the future which will help plan safer trajectories.

Since the bot is equipped with a lidar and a stereo camera as well, we decided to go ahead with just stereo cameras for now. This is mainly to make the pipeline as streamlined and fast as possible. The process of getting the depth information from stereo image pairs is called stereopsis. We and all other 2-eyed animals do it all the time. Checkout OpenCV: Depth Map from Stereo Images

The current plan is to directly apply the object detector on the stereo images, without calculating the disparity map or the point cloud. Again, this is done to speed things up; Disparity calculation takes time(checkout this link at computerphile: Stereo 3D Vision to get a feel.
Still, there are existing ros packages like stereo_image_proc - ROS Wiki that do it, if one wants to try.

What’s done and what needs to be:

• Research and find the approaches that give a good mix of accuracy and speed. Here’s an non-exhaustive list: vision comparison.
  The candidates under consideration are either RCNN(e.g. Stereo-RCNN) or YOLO based (e,g. Complex YOLO)
  RCNN based algorithms are more accurate but slower; and ones like YOLO which directly output in a single pass are faster but less accurate. Currently trying to improve on both aspects.
• [WIP] Recreate/Modify and implement
  Implement the DL pipeline.
  The current network uses HRNet as the base, and the KITTI stereo dataset to train.
• Integrate with our quadcopter
  The current pipeline uses the camera calibration data from KITTI, which is going to be different for our model. So make necessary changes and create a ROS node that reads data from the stereo camera and outputs the Object points.

Micro-controllers currently compiled:

Jetson TX2/ Jetson Nano/ UDOO X86/ UDOO X86 II/ ODROID H2 / ODROID N2

Detailed information about all the comparisions (click here)

This will give a brief review:

My Personal Recommendation is : UDOO X86 II with a graphics card such as GTX 1060/ RX 570 (approx. cost 30K)

PS: We went ahead with Jetson Xavier NX after a detailed comparision.

Integration of Intel Realsense T265 and D435/D435i for position identification, hold and mapping

Hardware for the drone to achieve at least 15 minutes time of flight with ~600g payload

We decided to use a combination of the Extended Kalman Filter (EKF) and the Particle Filter (PF) for localization of the quadcopter as they complement each other well and have already been used in SLAM algorithms like FastSLAM

Brief Info:-

This issue will help not only the State Estimation subsystem but also the Motion Planning subsystem. I will cover the State Estimation aspect of the issue here.

First, head over to this link and read the short article to understand what State Estimation means and why it is important.

You are encouraged to read this wonderful roadmap prepared by Mehul.

From the doc you just read, we might not need Gmapping, SLAM Toolbox, Google Cartographer, Navigator 2, or Hector SLAM. So there is no need to explore these immediately for the quadcopter project. One of the most important takeaways from it, though, is that you WILL HAVE to learn C++. All the SLAM packages are written in C++, and for understanding the code as well as modifying it later, we will need good C++ skills.

What we have done so for:-

• Till now, we have explored a lot of algorithms for State Estimation. You can read about Kalman Filter, Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), Particle Filter (PF), etc. We will obviously provide resources for these, but more on that later.
• We had earlier tried to implement a combination of EKF and PF by scratch, but seeing the huge challenges beyond coding the basic algorithm, we have now decided to use the existing SOTA packages built for this.
• We are currently exploring the fusion of RTAB-Map and the robot_localization package. RTAB-Map has a localization stack which we will use for this purpose. robot_localization has two algorithms, EKF and UKF. We will be using EKF as it is computationally less expensive and provides almost equal estimates as compared to UKF. We might also use the AMCL package with this combination. After testing each algorithm individually and their fusion, we will think about the potential modifications to the codes to improve the state estimates.

What we are planning to do next:-

Now coming to the immediate subtask for the subsystem. This subtask will be of great help to not only the State Estimation Subsystem but also the Motion Planning Subsystem.

• Our current quadcopter model, i.e. a modified version of the if750a has been written as a .sdf file. We need a .urdf file of the same, as only the .urdf file type is compatible with some of the packages we have chosen to use.
• One can easily convert a urdf file to a sdf file using gz sdf -p MODEL_URDF.urdf > MODEL_SDF.sdf. Unfortunately, one cannot go the other way round. So you will have to convert each line of the sdf file into the urdf compatible syntax. Once you have converted one part of the sdf file to urdf (e.g., some link, or some joint, etc.), you can check if the conversion is valid by running gz sdf -p MODEL.urdf. The output of this command should be the same as the corresponding code from your sdf file.
• You can find a lot of tutorials on Gazebo’s and ROS's site. These will help you understand how to write the urdf and sdf files. Googling will go a looong way.
• You can even find a partially completed urdf file in the repository. Your job is to complete that implementation and make a working .urdf file of the robot.

After that, we will be able to use robot_localization and MoveIt! (a package for motion planning) on our quadcopter. More details will be provided on how that can be done after the completion of this subtask.

• Localisation of the drone in the room using the map as it's reference
• Hardware needed for holding position
• Hardware for localizing its position in the map
• Software to support both of these (Suggested Possibility - ORB SLAM)

This is not a task to be completed but an issue dedicated to discussing and recording all the relevant resources that one can find useful on the internet.
This issue has been opened so that those resources that we come across don't get lost away 😄
Feel free to post all links as comments below 😃

Brief info

• We need to create two kinds of map using the autonomous drone, the local and global map.
• The former for immediate maneuvers and the latter for global path planning and future reference.
• This issue is in continuation to #2 and details what we have done, are currently working on and need to achieve finally.

Packages, algorithms and tools used so far:-

• Canonical Scan Matcher - The C(anonical) Scan Matcher (CSM) is a pure C implementation of a very fast variation of ICP using a point-to-line metric optimized for range-finder scan matching.
  • Limitation: PLICP algorithm is very fast and easy to integrate however it has been strictly written to work with LaserScan. Hence to use it in our project we needed to convert PointCloud2 data to LaserScan. However, since we want the final output to be in 3d and not in 2d, we had to drop it.
• OctoMap - The OctoMap library implements a 3D occupancy grid mapping approach, providing data structures and mapping algorithms in C++. There are multiple advantages in storing the map as an occupancy grid and not as pointcloud, chief ammongst them is the less storage.
• GMapping - This package can create a 2-D occupancy grid map (like a building floorplan) from laser and pose data collected by a mobile robot.
• PointCloud library - It is not possible to move ahead with mapping without having sufficient understanding of PCL library. To start with it C++ basics and tutorials would come in handy.
• RTABMAP - RTAB-Map (Real-Time Appearance-Based Mapping) is a RGB-D Graph SLAM approach based on a global Bayesian loop closure detector. The loop closure detector uses a bag-of-words approach to determinate how likely a new image comes from a previous location or a new location.
  • Strengths: It is a very well maintained library with very active community, tutorials and lots of flexibilty.

Current Tech Stack (for mapping)

RTABMAP (primarily for mapping) + PCL (handling pointcloud data) + RViz (for visualisation) + OctoMap (for storing the occupancy grid).

Things that need to be done:-

• Using RTABMAP (which by default uses ICP), generate local and global maps - This is currently under progress, a quick way to reach the current setup would be to use the docker image (root password is hardik).
• These maps need to be in OctTree format (binary format would do) - should be straight forward as rtabmap provides built-in support to occupancy grid.
• Both maps should be visualizable from RViZ
• Global Map should be saved while local map should be published to another topic (to be used by other subsystem).
• The steps needed to generate octomap using this repo's packages need to be documented in the README.md.
• The steps for setup needs to be well documented.

Additionally

• An exhaustive comparision of many SLAM techniques available is done by Mathieu Labb ́e at al.

• Mapping a room and storing the data in the companion computer through ROS
• Extracting the data through ROS and processing for further use.

The aim of the motion planning subsystem is to calculate the motion of Quadcopter in 3D.
This task of motion planning is subdivided into the following sub-parts:

Global Path/Motion Planner

Given an Octomap, start position, and goal position, we need to calculate the ideal trajectory that the drone should follow to reach the goal. This trajectory would be planned before the drone starts its mission. Note that this is an ideal trajectory that is most probably going to change due to unpredicted obstacles in the way, inaccuracies in the initial map, and drone slightly deviating from its path while maneuvering. Hence, the need for a Local Motion Planner arises.

Local Motion Planner

The Local Motion Planner would receive

1. The current state of Drone from the State Estimation Subsystem.
2. The ideally desired path from Global Path Planner.
3. Local Octomap from Visual Perception Subsystem.

and then, based on the surroundings and deviation from the ideal path, it should be able to recalculate its trajectory as well as motion.

Surveying Unknown Environments

Since we also want to survey unknown environments to generate the maps for the first time, we need to work on a surveying algorithm as well for unknown environments. There has been really good work by HKUST in this field. Fast Planner and FUEL are some of the examples that we were looking to test. Needless to say, these might also be useful in understanding more about how to go about for Local Planner.

Sending Commands to Bot

The commands generated by Local Motion Planner need to be converted into ones that are understandable by the PX4 and then can be sent over to the bot. For this MAVROS is to be used.

Current Progress

Since it's a bit difficult to code up everything by hand, and we don't want to reinvent the wheel, We are in the process of trying out MoveIt! for Global Planner.
MoveIt! already has abstract 3D planners, collision checking algos, etc. implemented in the context of robots (robotic arms to be precise). We can just drop the URDF of our bot in it, give it our world's Octomap and specify algo with which we want to calculate the trajectory/motion.

As we were trying out MoveIt! we realized that we need the URDF (Unified Robot Description Format) of our bot to move any further, but we have only SDF (Simulation Description Format) for now (as it was being used in the gazebo).
And hence, the next immediate step is to convert the SDF into URDF (which mostly needs to be done manually).
For MoveIt! the implementation we are referring to is Wil Selby 3D Mapping and Navigation on Drone using MoveIt!

Thoughts on Local Planner

1. The receding horizon controller with dynamic window approach looks promising but there's time till we get there. FAST and FUEL probably use this approach too.
2. The maps generated by Local Planner should also be merged with existing maps so as to increase their accuracy over time.

Other stuff

1. Sampling-based Algorithms looked better for our use case.
2. MoveIt! was easier to use and advance than ROS Navigation for 3D Navigation.
3. Octomaps are an efficient way to store 3D worlds. They are based on Octrees.
4. Behavioural Planner needs to be implemented too, to take into account the predicted trajectory of moving obstacles by Visual Perception Subsystem.
5. For Model Predictive Controller and the Receding horizon Controller for Local Planner a good starting point will be to refer to Research Papers by Kostas Alexis

Aim : To achieve around 4:1 thrust : weight ratio.

Even if you only just planned to fly a slow and stable aerial photography rig, you should aim at somewhere between 3:1 and 4:1 ratios. This does not only give you better control, but also provides room for extra payload in the future.
Source : Oscar Liang

What we know :

• The quad will itself weigh around 850 gram without battery.
• 4s 5200mAh battery weighs around 500 gram.
• The extra hardware for SLAM is expected to weigh around 600 gram.
• We can't use Propeller more than 10inch in Size for our current frame