cmusatyalab / steeleagle

Automated drone flights for visual inspection tasks

SteelEagle: Edge-Enabled Drone Autonomy

Introduction

SteelEagle is a software suite that transforms commercial-off-the-shelf (COTS) drones into fully-autonomous, beyond-visual-line-of-sight (BVLOS) UAVs. This allows users to develop complex autonomous UAV applications without needing to purchase or configure expensive aircraft hardware. SteelEagle drones are easy to deploy and importantly, are drone agnostic. This means that SteelEagle can be adapted to work with any drone control API, as long as it supports control over WiFi. Of particular interest to us are extremely lightweight drones because of the simplified regulatory approval process.

Democratizing Drone Autonomy Via Edge Computing published in ACM SEC 2023

Quickstart

Documentation

License

Unless otherwise stated in the table below, all source code and documentation are under the GNU Public License, Version 2.0. A copy of this license is reproduced in the LICENSE file.

Portions from the following third party sources have been modified and are included in this repository. These portions are noted in the source files and are copyright their respective authors with the licenses listed.

Project	Modified	License
cmusatyalab/openscout	Yes	Apache 2.0
bytedeco/javacv	Yes	Apache 2.0
xianglic/droneDSL	Yes	GPL 3.0

Design

SteelEagle is separated into three distinct parts: the local commander client, the cloudlet server, and the onboard software. The commander client is intended to run on a personal computer close to the RPIC (Remote Pilot-in-Command) with an internet connection. It gives an interface to receive telemetry and upload a mission script to the drone. It also provides tools to assume manual control of the drone while it is in-flight. The cloudlet server is the bridge between the onboard drone software and the commander client. It relays messages between the two and also publicly hosts flight scripts. Additionally, the server runs compute engines for the drone which will be executed on the offloaded sensor data/video stream. Finally, the onboard software consists of an app that runs on the drone-mounted Android or router device. This app relays telemetry and offloads sensor data/video frames to the cloudlet server. In the Android case, it is also responsible for running the mission script and directly sending control commands to the drone. In the router case, the cloudlet runs the mission script and sends control commands over the network to the drone.

Architecture

Workflow

1. A planner utilizes Google MyMaps to define the mission in a graphical UI by creating tasks. A task is created by drawing a polygon/marker, naming it, and defining the actions associated with that task by adding these to the description textbox.

2. Once defined, the planner exports the mission as a KML file.

3. The planner generates a mission script (.ms file) from the KML file using the Hermes compiler.

4. The pilot powers on the drone, starts the onboard software, connects to the drone through the commander, and sends the .ms file.

5. The drone executes its mission.

steeleagle's Issues

Add mission planning to Streamlit commander

Add a mission planning page that consists of a Folium Draw element along with a modified Barfi component for drawing states/transitions.

Missing Python packages from CnC Python client

The requirements.txt file is missing Python packages customtkinter and tkintermapview. We also need to install python3-tk, python3-pil, and python3-pil.imagetk using apt. These are required to run the commander_client.py script in cnc/python_client.

Flight Scripts don't execute when they are not named classes.dex

We attempted to run to different flight plans at once, so we named one harpy.dex and the other buzzard.dex by using the -o option in hermes. However, when we selected the file in the commander, the log said the file was uploaded, but the drone never took off.

Change marker as drone moves

Currently we update the marker on the map as the drone's GPS location is updated. We should leave behind a different marker to show the locations that the drone has previously held during the flight.

Alternatively, we could look at over laying the polygons from the KML document (or whatever hermes builds that contains the gps locations) to show the proposed flight path and then the marker should fly along (or at least near) the proposed plan.

Altitude is not a parameter to any tasks

Currently we only have a fixed altitude of 15m (from hermes.py) for all flight plans. We should consider adding an altitude param to each task definition and then apply that to the moveTos for all points in that polygon. Or we could consider a list of points, but then we would have to check that the list length matches the number of vertices in the polygon etc.

DroneBrain needs to use openscout_pb2

DroneBrain is sending both the heartbeat messages and frames from the stream to the CNC server. OpenScout is configured to receive both of these, but is only interested in the image frames. However, the protocol buffer for OpenScout varies from the CNC one. At the moment, the DroneBrain is using the CNC protobuf when it sends both the TEXT and the IMAGE payloads, but it should actually be using the OpenScout protobuf when it sends images. An example of this can be seen in drone_adpater.py.

Land doesn't work

Though we have implemented land in ParrotAnafi.kt, it doesn't work properly. The drone remains hovering and doesn't enter the canLand() state.

Make user executed code run in a separate asyncio event loop

It appears that the command handler of supervisor.py is being starved enough, such that imperative commands like manual kill and return to home, sometimes do not get processed. We need to make the user code run in a separate asyncio event loop so that this method is unaffected by greedy user code.

cnc_pb2 vs openscout_pb2

When an image is sent from a DroneBrain client, both the command cognitive engine and the OpenScout engines (for object detection/face recognition) need to receive it. The command engine will store it so it can be sent back to commander clients and OpenScout will attempt to detect/recognize things. However each requires a slightly different protobuf definition at the moment. The Extras for each both include a Location object and an identifier, however in cnc it is called drone_id and in openscout, client_id. Furthermore, current OpenScout is configured such that it expects client_id to be a UUID formatted string and will fail to parse and insert the data into ELK if it does not match the UUID regex. This results in objects being detected by failing to insert either because the client_id is not a UUID or because the location is not unpackable because it is not the protobuf it was expecting.

I believe the only way to solve this is to create a slightly modified version of OpenScout that will accept a cnc_pb2 instead and change the type of the client_id that gets parsed by logstash so that it is not a UUID. We could put these modifications into a branch in the OpenScout repo and publish a Docker image with the steel-eagle tag to differentiate it from the typical stable image for OpenScout.

Incorrect branch name in documentation

The documentation for the backend setup states:

git clone [email protected]:cmusatyalab/openscout.git
cd /path/to/openscout
# Checkout the SteelEagle branch of OpenScout
git checkout steeleagle

The branch name should be steel-eagle.

Secure zmq control plane with curve

See the following example for how to add curve based security to the ZMQ control plane.

https://github.com/zeromq/pyzmq/blob/main/examples/security/stonehouse.py

Incorrect OpenScout tag in template.env file

The template.env file has the following tag for OpenScout:

#OpenScout variables
OPENSCOUT_TAG=stable

We should instead initialize this to steeleagle as the documentation suggests using the following tag for the image:

# Build the OpenScout backend
docker build . -t cmusatyalab/openscout:steeleagle

requirements.txt is no longer generated and included in the .ms zip file

In hermes, we used pipreqs to generate a requirements.txt file that was then packed into the .ms (zip) file. This no longer appears to be happening in Compiler.java.

            # Generate requirements.txt inside the python directory
            os.system("cd ./python; pipreqs . --force")
            
            # Add requirements.txt to the zip, under the new directory
            zf.write("./python/requirements.txt", arcname=f"./requirements.txt")

Gimbal actuation, change heading, and offset moves do not work

Change licensing to GPL-v2

Initially we had the license as Apache, but it should be GPLv2. In addition to the LICENSE file, we will also need to update the boilerplate text on source files using something like REUSE (SPDX Id: GPL-2.0-only)

Documentation refers to docker-compose, not docker compose

Docker has switched to Compose V2 (see here) and the recommended command line syntax is now docker compose instead of docker-compose.

Affected documentation.