shaunmulligan / radr

An AI powered rear facing radar for your bike. Instead of using mmWave radar tech, this project uses an inexpensive raspberry pi zero 2, a camera and Coral.ai TPU accelerator to detect objects behind you.

Radr

An smart rear facing radar for your bike. Instead of using mmWave radar tech, this project uses an inexpensive raspberry pi zero 2, a camera and Coral.ai TPU accelerator to detect objects behind you.

MVP features:

• Detect cars, cyclist (person + bicycle) and people behind you.
  • Use the Coral.ai TPU accelerator to run the model, hopefully at ~ 25 FPS.
  • Track multiple objects at once.
• Implement the radar BLE Gatt service to send alerts to a cycling computer.
• 3D print a case for the pi zero 2, camera, TPU accelerator and battery that slips in under the saddle.

Future features:

• Add neopixel LEDs controllable via BLE Gatt service.
• brighten LEDs when an object is detected.
• Add ability to record video of the ride.
• periodically save images + detection while riding and upload to cloud
• Use the cloud to train a custom model on the images. Using something like v7Labs, roboflow or ClearML.
• Have Models be versioned and deployed to the device via wifi when device is charging.

Usage

1. Build the container:

cd radr && sudo docker build -t radr-v1 .

2. Run the container:

sudo docker run --rm -it --privileged radr-v1

The --privileged flag is required to access the camera and USB accelerator. 3. You should now be able to see the video stream on http://<RPI_IP_ADDRESS>:8080/video

Testing with video file.

If a video file with .mp4 extension is found in the root directory, the model will run on the video file instead of the camera. This offers a nice way to consistently test the model on a known input.

Example of the current model output:

radr's Issues

Estimate distance and speed of objects

depends on #7

Add ability to record and download video of the rides

Users might want to record their rides and view them later.

Periodically snapshot image and prediction during rides and upload to MLOps backend

We want to continually learn and improve the model, so it would be great to grab some snaps during rides and upload them when the device is charging and connected to wifi.

MLOps platforms to consider:

• v7Labs
• roboflow
• clearML
• edgeImpulse?
• others?

Optionally have a way to view the video stream with detections and tracking data overlayed

Option 1 - On screen:
Easy to do in opencv but won't be super useful when we have the device mounted on the bike with no screen (VNC maybe). This might be harder in docker as well as we need to pass in the x-org display but will be helpful while debugging and testing things.

Option 2 - http stream:
Ideally, we can have the image feed served via fastAPI and then just connect to it and view with the laptop. Have a look at https://github.com/zauberzeug/nicegui/blob/main/examples/opencv_webcam/main.py

Test efficientDet-lite3 model

Deploy on balena

Run testing on power consumption

We want to know how much power each part of the system uses and how much it uses when its running at full speed. Once we have these numbers we can figure out how to quantify and optimise battery usage.

Fine tune the model using our custom dataset

Initially we can try do this in google colabs similar to this example: https://colab.research.google.com/github/google-coral/tutorials/blob/master/retrain_efficientdet_model_maker_tf2.ipynb#scrollTo=Gb7qyhNL1yWt

Only detect the object classes we care about

We probably don't care if we detect thinks like a sofa or cup on the road and definitely don't want them pinging our head unit saying there is something behind us.

Only run inference when we are moving

Assuming running inference uses a lot of power, we could add an IMU sensor and detect when we are in motion and only actually do inference while we are moving.

Only turn on wifi when device is charging

Need some additional hardware to detect charging state

Add controllable neopixel LEDs

1. Should be able to switch on with a button
2. Would be amazing to implement BLE light protocol to make them controllable from head unit.

Implement a multi-object tracking algorithm

Something like Bot-Sort or just a standard SORT, this article looks like a good place to start: https://medium.com/geekculture/large-scale-object-detection-tracking-with-yolov5-package-c8eca66b80b6

Create a training set of use case specific data

Maybe do some rides with a goPro mounted at the back and then apply synthlabel project to auto label those images and do some category fusion so we can have a cyclist rather than bike + person detection.

Brighten LEDs a objects get closer

Depends on #15 and #13

Figure out how to run yolov8 on TPU

Figure out how to run yolov5 on TPU

Since we can't run yolov8 successfully yet (see: #2 ), lets try get v5 working

Add BLE gatt server with radar profile

Add a BLE gatt server that implements the radar profile and when we get detections notify clients. We should be able to reverse engineer how to create the profile and what to send using:

https://github.com/zacharyedwardbull/pycycling/blob/master/examples/rear_view_radar_example.py

Design and 3D print a case which secures everything under the saddle

Improve detection for small objects

Apparently yolov5 struggles to do small object detection, so cars and cyclists that are far away will be less likely to be seen. We might be able to do something like the following to improve that: https://github.com/google-coral/pycoral/blob/master/examples/small_object_detection.py