putvision / thermo-hardware-benchmark

The official repository of the paper Efficient People Counting in Thermal Images: The Benchmark of Resource-Constrained Hardware

Hardware Benchmark of Neural Network Method for Presence Monitoring in Thermal Images

This is the official repository of the paper: Efficient People Counting in Thermal Images: The Benchmark of Resource-Constrained Hardware.

The repository includes the code for reproducing experiments and obtaining results from the aforementioned publication. This repository is organized as follows:

• thermo directory contains code and scripts related to model training, optimization, and quantization phases,
• evaluation folder consists of hardware deployment, inference, and evaluation tools.

Table of Contents

• Citation
• Dataset
• Benchmark Hardware
• Neural Network Architecture
• Results

Citation

@ARTICLE{9964383,
  author={Piechocki, Mateusz and Kraft, Marek and Pajchrowski, Tomasz and Aszkowski, Przemyslaw and Pieczynski, Dominik},
  journal={IEEE Access}, 
  title={Efficient People Counting in Thermal Images: The Benchmark of Resource-Constrained Hardware}, 
  year={2022},
  volume={10},
  number={},
  pages={124835-124847},
  doi={10.1109/ACCESS.2022.3225233}
}

Dataset

The benchmark utilizes Thermo Presence dataset that consists of 13 644 low-resolution thermal images recorded in office spaces. The dataset distribution is shown in the table below. The training, validation, and test split was chosen according to the authors' choice which is described in their project repository.

	0	1	2	3	4	5	Total
Training	99	105	2984	3217	1953	114	8472
Validation	0	139	631	1691	225	139	2825
Test	162	83	211	341	1235	315	2347

Benchmark Hardware

Device	Target Hardware	Max Clock Frequency	FLASH	SRAM	Evaluation Framework
Arduino Nano 33 BLE Sense	nRF52840	64 MHz	1 MB	256 KB	TFLite Micro
Arduino Portenta H7	STM32H747	480 MHz	2 MB	1 MB	TFLite Micro
LOLIN32	ESP32-WROOM-32	240 MHz	4 MB	520 KB	TFLite Micro with ESP-NN support
STM32 F429ZI Nucleo-144	STM32F429	180 MHz	2 MB	256 KB	STM32 Cube AI
STM32 H745ZI Nucleo-144	STM32H745	480 MHz	2 MB	1 MB	STM32 Cube AI
Raspberry Pi 4B	Quad Core Cortex-A72	1.5 GHz	*	2 GB	TensorFlow Lite
Coral USB Accelerator	Google Edge TPU	500 MHz	*	8 MB	TensorFlow Lite
Intel Neural Compute Stick 2	Intel Movidius Myriad X VPU	700 MHz	*	4 GB	OpenVINO

Neural Network Architecture

The neural network is based on U-Net architecture with shallow structure, single-channel input, and output. The proposed model has only 46 577 parameters.

Results

Neural Network Metrics

Metric Name	MAE	MSE	Counting MAE	Counting MSE	Counting MRAPE [%]	Accuracy	F1 Score	No. of parameters
Results	0.1057	0.0332	0.0226	0.0234	0.81	0.9778	0.9782	46 577

Hardware Performance

Device	Data Type	Avg. Inference Time [ms]	Counting MAE	Counting MSE	Counting MRAPE [%]
Arduino Nano 33 BLE Sense	INT8	1430.125 ±1.143	0.023	0.024	0.82
Arduino Portenta H7	INT8	137.494 ±0.500	0.023	0.024	0.82
LOLIN32	INT8	840.442 ±0.021	0.023	0.023	0.81
STM32 F429ZI Nucleo-144	INT8	230.939 ±0.100	0.036	0.038	1.14
STM32 H745ZI Nucleo-144	FP32	165.983 ±0.100	0.023	0.023	0.81
	INT8	53.260 ±0.100	0.036	0.038	1.14
Raspberry Pi 4B	FP32	7.707 ±0.591	0.023	0.023	0.81
	FP16	7.690 ±0.558	0.023	0.023	0.81
	INT8	4.194 ±0.052	0.038	0.039	1.18
Raspberry Pi 4B + Coral USB Accelerator (std)	INT8	0.605 ±0.044	0.037	0.038	1.16
Raspberry Pi 4B + Coral USB Accelerator (max)	INT8	0.570 ±0.060	0.037	0.038	1.16
Raspberry Pi 4B + Intel Neural Compute Stick 2	FP32	2.630 ±0.159	0.028	0.029	0.92
	FP16	2.300 ±0.100	0.028	0.029	0.91

The figure above consists of current consumption characteristics measured on benchmarked hardware. The graphs are limited to 180 seconds in order to highlight peaks related to the prediction stage.

thermo-hardware-benchmark's Issues

KeyError: 'raw_frame'

When executing the Modek Training I get the following Error:

Exception has occurred: KeyError

'raw_frame'  
  File "thermo/utils/model_utils.py", line 12, in check_model_prediction  
    raw_frame = np.array(config['dataset']["raw_frame"])  
  File "train.py", line 136, in train  
    count, output_frame = check_model_prediction(model, config)  
  File "train.py", line 170, in <module>  
    train()  
KeyError: 'raw_frame'

When calling the function count, output_frame = check_model_prediction(model, config) there appears to be no key raw_frawe in config.