paulbozans / rpi-urban-mobility-tracker

The easiest way to count pedestrians, cyclists, and vehicles on edge computing devices or live video feeds.

Raspberry Pi Urban Mobility Tracker (DeepSORT + MobileNet)

The Raspberry Pi Urban Mobility Tracker is the simplest way to track and count pedestrians, cyclists, scooters, and vehicles. For more information, see the original blog post [here].

Hardware

Primary Components

1. Raspberry Pi (ideally v4-b)
2. Raspberry Pi camera (ideally v2)
3. Google Coral Accelerator (Not required, but strongly encouraged)

Secondary Components

1. Ballhead mount: https://www.amazon.com/gp/product/B00DA38C3G
2. Clear lens: https://www.amazon.com/gp/product/B079JW114G
3. Weatherproof enclosure: https://www.amazon.com/gp/product/B005UPAN0W
4. 30000mAh battery: https://www.amazon.com/gp/product/B01M5LKV4T

Notes

• The mounts located in geometry/ are currently represented as stl files which are 3d printer ready. I don't currently have a 3d printer so I used the crowd sourced printing service https://printathing.com/ which yielded great results (kind of a sales pitch, but not really. I just like the service).
• The original FreeCAD file is also included just in case you want to modify the geometry.
• The only cutting necessary is through the plastic case to allow for the lens. This joint should then be sealed using silicone caulk to prevent any moisture from entering.
• All the secondary components listed are just suggestions which worked well for my build. Feel free to use what ever you want.

3D printed mounts	mounts with attached hardware

Final setup (open)	Front (closed)

Install (Raspberry Pi)

1. UMT has been dockerized in order to minimize installation friction. Start off by installing Docker on your Raspbery Pi or what ever you plan on using. The instructions below assume a Raspberry Pi v4 with Raspberry Pi OS 2020-12-02. This is also a good time to add non-root users to the Docker user group. As an example, to add the Raspberry pi default user pi:

sudo usermod -aG docker pi

2. Open a terminal and create a directory for the UMT output:

UMT_DIR=${HOME}/umt_output && mkdir -p ${UMT_DIR}

3. Move into the new directory:

cd ${UMT_DIR}

4. Download the Dockerfile and build it:

wget https://raw.githubusercontent.com/paulbozans/rpi-urban-mobility-tracker/master/Dockerfile

docker build . -t umt

5. Start the Docker container:

docker run --rm -it --privileged --mount type=bind,src=${UMT_DIR},dst=/root umt

or with display

docker run --network=host -e DISPLAY=$DISPLAY --rm -it --privileged --mount type=bind,src=${UMT_DIR},dst=/root umt

or with display and tpu

docker run --network=host -e DISPLAY=$DISPLAY --rm -it -v /dev/bus/usb:/dev/bus/usb --privileged --mount type=bind,src=${UMT_DIR},dst=/root umt

fix KeyError: "The name 'net/images:0' refers to a Tensor which does not exist. The operation, 'net/images', does not exist in the graph." if neded

nano /usr/local/lib/python3.7/dist-packages/deep_sort_tools/generate_detections.py

Change this two lines form :

self.input_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{net/input_name}:0")
self.output_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{net/output_name}:0")

to this:

self.input_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{input_name}:0")
self.output_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{output_name}:0")

6. Test install by downloading a video and running the tracker:

wget https://github.com/paulbozans/rpi-urban-mobility-tracker/raw/master/data/videos/highway_01.mp4

umt -video highway_01.mp4

If everything worked correctly, you should see a directory labeled output filled with 10 annotated video frames.

Install (Ubuntu)

First, create a new virtualenv,

python3 -m venv umt

initialize it,

source umt/bin/activate

install whell

pip3 install wheel

then install the TensorFlow Lite runtime package for Python:

pip3 install --extra-index-url https://google-coral.github.io/py-repo/ tflite_runtime

Then finish with the following:

pip install git+https://github.com/paulbozans/rpi-urban-mobility-tracker

fix KeyError: "The name 'net/images:0' refers to a Tensor which does not exist. The operation, 'net/images', does not exist in the graph." if neded

nano umt/lib/python3.8/site-packages/deep_sort_tools/generate_detections.py

Change this two lines form :

self.input_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{net/input_name}:0")
self.output_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{net/output_name}:0")

to this:

self.input_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{input_name}:0")
self.output_var = tf.compat.v1.get_default_graph().get_tensor_by_name(
     f"{output_name}:0")

Lastly, test the install by running step #6 from the Raspberry Pi install instructions above.

Model Choice

The default deep learning model is the MobileNet v1 which has been trained on the COCO dataset and quantized for faster performance on edge deployments. Another good model choice is PedNet which is also a quantized MobileNet v1 however, it's been optimized specifically for pedestrians, cyclsts, and vehicles. To use PedNet, simply download it from its repo here: https://github.com/nathanrooy/ped-net or clone it.

git clone https://github.com/paulbozans/ped-net

Once the model and labels have been downloaded, simply use the modelpath and labelmap flags to specify a non-default model setup. As an example:

umt -camera -modelpath pednet_20200326_tflite_graph.tflite -labelmap labels.txt

If you got an error

Usage

Since this code is configured as a cli, everything is accessible via the umt command on your terminal. To run while using the Raspberry Pi camera (or laptop camera) data source run the following:

umt -camera

To run the tracker on an image sequence, append the -imageseq flag followed by a path to the images. Included in this repo are the first 300 frames from the MOT (Multiple Object Tracking Benchmark) Challenge PETS09-S2L1 video. To use them, simply download/clone this repo and cd into the main directory.

umt -imageseq data/images/PETS09-S2L1/

To view the bounding boxes and tracking ability of the system, append the -display flag to view a live feed. Note that this will greatly slow down the fps and is only recommended for testing purposes.

umt -imageseq data/images/PETS09-S2L1/ -display

By default, only the first 10 frames will be processed. To increase or decrease this value, append the -nframes flag followed by an integer value.

umt -imageseq data/images/PETS09-S2L1/ -display -nframes 20

To persist the image frames and detections, use the -save flag. Saved images are then available in the output/ directory.

umt -imageseq data/images/PETS09-S2L1/ -save -nframes 20

To run the tracker using a video file input, append the -video flag followed by a path to the video file. Included in this repo are two video clips of vehicle traffic.

umt -video data/videos/highway_01.mp4

In certain instances, you may want to override the default object detection threshold (default=0.5). To accompish this, append the -threshold flag followed by a float value in the range of [0,1]. A value closer to one will yield fewer detections with higher certainty while a value closer to zero will result in more detections with lower certainty. It's usually better to error on the side of lower certainty since these objects can always be filtered out during post processing.

umt -video data/videos/highway_01.mp4 -display -nframes 100 -threshold 0.4

To get the highest fps possible, append the -tpu flag to use the Coral USB Accelerator for inferencing.

umt -imageseq data/images/PETS09-S2L1/ -tpu

References

@inproceedings{Wojke2017simple,
  title={Simple Online and Realtime Tracking with a Deep Association Metric},
  author={Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich},
  booktitle={2017 IEEE International Conference on Image Processing (ICIP)},
  year={2017},
  pages={3645--3649},
  organization={IEEE},
  doi={10.1109/ICIP.2017.8296962}
}

@inproceedings{Wojke2018deep,
  title={Deep Cosine Metric Learning for Person Re-identification},
  author={Wojke, Nicolai and Bewley, Alex},
  booktitle={2018 IEEE Winter Conference on Applications of Computer Vision (WACV)},
  year={2018},
  pages={748--756},
  organization={IEEE},
  doi={10.1109/WACV.2018.00087}
}