kaylorchen / rk3588-yolo-demo Goto Github PK

The project is a multi-threaded inference demo of Yolo running on the RK3588 platform, which has been adapted for reading video files and camera feeds. The demo uses the Yolov8n model for file inference, with a maximum inference frame rate of up to 100 frames per second.

License: MIT License

CMake 1.21% C++ 97.43% C 1.37%

rk3588-yolo-demo's Introduction

Yolov8/v10 Demo for RK3588

The project is a multi-threaded inference demo of Yolov8 running on the RK3588 platform, which has been adapted for reading video files and camera feeds. The demo uses the Yolov8n model for file inference, with a maximum inference frame rate of up to 100 frames per second.

If you want to test yolov8n with ros2 for yourself kit, click the link

Model

Download Model File

you can find the model file in the 'src/yolov8/model', and some large files: Link: https://pan.baidu.com/s/1zfSVzR1G7mb-EQvs6A6ZYw?pwd=gmcs Password: gmcs
Google Drive: https://drive.google.com/drive/folders/1FYluJpdaL-680pipgIQ1zsqqRvNbruEp?usp=sharing

Model pt --> onnx

For Yolov8

go to my blog --> blog.kaylordut.com

For Yolov10

go to my another repository --> yolov10
download pt model and export:

# End-to-End ONNX
yolo export model=yolov10n/s/m/b/l/x.pt format=onnx opset=13 simplify

Model onnx --> rknn

go to my blog --> blog.kaylordut.com

TIPS: (Yolov10)

rknn-toolkit2(release:1.6.0) does not support some operators about attention, so it runs attention steps with CPU, leading to increased inference time.

rknn-toolkit2(beta:2.0.0b12) has the attention operators for 3588, so I build a docker image, you can pull it from kaylor/rknn_onnx2rknn:beta

Inference Time

Please refer to the spreadsheet '8vs10.xlsx' for details.

V8l-2.0.0	V8l-1.6.0	V10l-2.0.0	V10l-1.6.0	V8n-2.0.0	V8n-1.6.0	V10n-2.0.0	V10n-1.6.0
133.07572815534	133.834951456311	122.992233009709	204.471844660194	17.8990291262136	18.3300970873786	21.3009708737864	49.9883495145631

Demo Video and Guideline

https://space.bilibili.com/327258623?spm_id_from=333.999.0.0
QQ group: 957577822

Prepare

Build the Cross-Compilation Environment

Set up a cross-compilation environment based on the following link.

Install Runtime Libraries in Your RK3588 Target Board

cat << 'EOF' | sudo tee /etc/apt/sources.list.d/kaylordut.list 
deb [signed-by=/etc/apt/keyrings/kaylor-keyring.gpg] http://apt.kaylordut.cn/kaylordut/ kaylordut main
EOF
sudo mkdir /etc/apt/keyrings -pv
sudo wget -O /etc/apt/keyrings/kaylor-keyring.gpg http://apt.kaylordut.cn/kaylor-keyring.gpg
sudo apt update
sudo apt install kaylordut-dev libbytetrack

If your OS is not Ubuntu22.04, and find kaylordut-dev and libbytetrack sources in my github.

Build the Project for Your RK3588

Compile

git clone https://github.com/kaylorchen/rk3588-yolo-demo.git 
cd rk3588-yolo-demo/src/yolov8
mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=/path/to/toolchain-aarch64.cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
make

/path/to/toolchain-aarch64.cmake is .cmake file absolute path

Usage: ./videofile_demo [--model_path|-m model_path] [--input_filename|-i input_filename] [--threads|-t thread_count] [--framerate|-f framerate] [--label_path|-l label_path]  

Usage: ./camera_demo [--model_path|-m model_path] [--camera_index|-i index] [--width|-w width] [--height|-h height][--threads|-t thread_count] [--fps|-f framerate] [--label_path|-l label_path]

Usage: ./imagefile_demo [--model_path|-m model_path] [--input_filename|-i input_filename] [--label_path|-l label_path]

you can run the above command in your rk3588

rk3588-yolo-demo's People

Contributors

Stargazers

Watchers

rk3588-yolo-demo's Issues

两次推理同一个模型帧数不一致

模型、线程数都一样，两次推理一个帧数4npu占用率8%左右，一个21npu占用率55-60

when I execute "make",some error apper

collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/videofile_demo.dir/build.make:210: videofile_demo] Error 1
make[1]: *** [CMakeFiles/Makefile2:102: CMakeFiles/videofile_demo.dir/all] Error 2

为啥需要延迟函数

run_once_with_delay(func, std::chrono::milliseconds(delay));
这个函数的作用是啥？我注释掉之后程序跑是能跑，但是好像是偷懒跑完的（就是不给结果直接结束）。

./videofile_demo.cpp: line 12: struct: command not found

I've been setting up a cross-compilation environment for the RK3588 platform using the following steps:

Installed the cross-compilation toolchain using the commands:

sudo apt update
sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

Built a cross-compilation environment rootfs by cloning the repository:

git clone https://github.com/kaylorchen/rk3588_dev_rootfs.git
cd rk3588_dev_rootfs
bash ./build-rootfs.sh

Compiled my code using the cross-compilation environment with the following commands:

git clone https://github.com/kaylorchen/rk3588-yolo-demo.git 
cd rk3588-yolo-demo/src/yolov8
mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=/home/orangepi/rk3588_dev_rootfs/toolchain-aarch64.cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
make

However, when I attempt to run the videofile_demo.cpp script as follows:

./videofile_demo.cpp --model_path model/yolov8n.rknn --input_filename 720p60hz.mp4 --threads 4 --framerate 60 --label_path model/coco_80_labels_list.txt

I encounter the following errors:

./videofile_demo.cpp: line 12: struct: command not found
./videofile_demo.cpp: line 13: std::string: command not found
./videofile_demo.cpp: line 14: std::string: command not found
./videofile_demo.cpp: line 15: std::string: command not found
./videofile_demo.cpp: line 16: int: command not found
./videofile_demo.cpp: line 17: double: command not found
./videofile_demo.cpp: line 18: syntax error near unexpected token `}'
./videofile_demo.cpp: line 18: `};'

It seems like the script is being interpreted as shell commands instead of C++ code. Any guidance on how to resolve this issue would be greatly appreciated. Thank you!

yolov10在Model pt --> onnx环节有问题

我用了你的yolov10仓库,导出了onnx模型,但是转换为rknn模型时报错,用了你仓库的onnx模型转rknn却是成功的

(toolkit2_py3.8) book@100ask:~/Desktop/onnx->rknn/rk3588-convert-to-rknn-master$ python convert.py yolov10s.onnx rk3566 i8 yolov10.rknn
I rknn-toolkit2 version: 2.0.0b0+9bab5682
--> Config model
done
--> Loading model
I It is recommended onnx opset 19, but your onnx model opset is 13!
I Model converted from pytorch, 'opset_version' should be set 19 in torch.onnx.export for successful convert!
I Loading : 0%| | 0/I Loading : 100%|██████████████████████████████████████████████| 193/193 [00:00<00:00, 24905.41it/s]
done
--> Building model
E build: The input 0 of TopK('/model.23/TopK') need to be constant! It will cause the graph to be a dynamic graph!
Remove TopK('/model.23/TopK') manually and try again!
I ===================== WARN(0) =====================
E rknn-toolkit2 version: 2.0.0b0+9bab5682
E build: Catch exception when building RKNN model!
E build: Traceback (most recent call last):
E build: File "rknn/api/rknn_base.py", line 1977, in rknn.api.rknn_base.RKNNBase.build
E build: File "rknn/api/graph_optimizer.py", line 814, in rknn.api.graph_optimizer.GraphOptimizer.fold_constant
E build: File "rknn/api/graph_optimizer.py", line 361, in rknn.api.graph_optimizer._dynamic_check
E build: File "rknn/api/rknn_log.py", line 92, in rknn.api.rknn_log.RKNNLog.e
E build: ValueError: The input 0 of TopK('/model.23/TopK') need to be constant! It will cause the graph to be a dynamic graph!
E build: Remove TopK('/model.23/TopK') manually and try again!
W If you can't handle this error, please try updating to the latest version of the toolkit2 and runtime from:
https://console.zbox.filez.com/l/I00fc3 (Pwd: rknn) Path: RKNPU2_SDK / 2.X.X / develop /
If the error still exists in the latest version, please collect the corresponding error logs and the model,
convert script, and input data that can reproduce the problem, and then submit an issue on:
https://redmine.rock-chips.com (Please consult our sales or FAE for the redmine account)

The files in baidu netDisk are not accessible

Hi, first of all, thanks a lot for your efforts. I just wanted to mention that the files that you put in the Baidu netdisk are not accessible since it wants registration which is not available with my phone number.

./camera_demo: error while loading shared libraries: libspdlog.so.1: cannot open shared object file: No such file or directory

(base) firefly@firefly:~/Documents$ ./camera_demo
./camera_demo: error while loading shared libraries: libspdlog.so.1: cannot open shared object file: No such file or directory

详细的 to be specific

(base) firefly@firefly:~/Documents$ ldd camera_demo
./camera_demo: /lib/aarch64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by ./camera_demo) ./camera_demo: /lib/aarch64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by ./camera_demo)
./camera_demo: /lib/aarch64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.30' not found (required by ./camera_demo) ./camera_demo: /lib/aarch64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by ./camera_demo)
./camera_demo: /lib/aarch64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by ./camera_demo)
linux-vdso.so.1 (0x0000007f8c781000)
libspdlog.so.1 => not found
libsystemd.so.0 => /lib/aarch64-linux-gnu/libsystemd.so.0 (0x0000007f8c641000)
libbytetrack.so.3 => not found
libfmt.so.8 => not found
libopencv_highgui.so.4.5d => not found
libopencv_videoio.so.4.5d => not found
libopencv_imgproc.so.4.5d => not found
libopencv_core.so.4.5d => not found
librknnrt.so => /lib/librknnrt.so (0x0000007f8c2c9000)
libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f8c0e4000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f8c039000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f8c015000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8bea2000)
/lib/ld-linux-aarch64.so.1 (0x0000007f8c751000)
librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f8be8a000)
liblzma.so.5 => /lib/aarch64-linux-gnu/liblzma.so.5 (0x0000007f8be56000)
liblz4.so.1 => /lib/aarch64-linux-gnu/liblz4.so.1 (0x0000007f8be28000)
libgcrypt.so.20 => /lib/aarch64-linux-gnu/libgcrypt.so.20 (0x0000007f8bd5c000)
libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f8bd2b000)
libgpg-error.so.0 => /lib/aarch64-linux-gnu/libgpg-error.so.0 (0x0000007f8bcfb000)

我根据提示在rknn部署了运行环境具体的就是以下步骤

Install Runtime Libraries in Your RK3588 Target Board
cat << 'EOF' | sudo tee /etc/apt/sources.list.d/kaylordut.list
deb [signed-by=/etc/apt/keyrings/kaylor-keyring.gpg] http://apt.kaylordut.cn/kaylordut/ kaylordut main
EOF
sudo mkdir /etc/apt/keyrings -pv
sudo wget -O /etc/apt/keyrings/kaylor-keyring.gpg http://apt.kaylordut.cn/kaylor-keyring.gpg
sudo apt update
sudo apt install kaylordut-dev libbytetrack
If your OS is not Ubuntu22.04, and find kaylordut-dev and libbytetrack sources in my github.
（我编译的pc机是 22.04 但是我rknn设备的是20.04）

20.04版本的arm板子在哪安装kaylordut-dev，bytetrack

您好，@kaylorchen 20.04版本的arm板子在哪安装kaylordut-dev，bytetrack

关于模型转换问题

请问一下，这个onnx是如何转换得到的，能否给个示例或出个教程呢

请问您推较大的模型时是否会发生内存泄漏问题。

我在参考您的方案修改yolov5的demo，发现循环时设置较小的延时，并且推理较大的模型（如yolov5m），设置线程数为3时可能会发生内存泄漏问题，请问您有没有遇到类似的情况。

代码仓库里缺少头文件和库

主文件包含的
#include "getopt.h"
#include "kaylordut/log/logger.h"
#include "kaylordut/time/time_duration.h"
#include "kaylordut/time/timeout.h"
这几个头文件没有，对应的库文件也没有，能补一下吗？

opencv lib

Could you please provide the opencv library you used in this project? Thank you very much!

Memory leak if setting framerate too high.

When setting framerate too high (more than the file's fps), the memory would leak.
In other word, if I delay a very small tick in run_once_with_delay(func, std::chrono::milliseconds(delay));, it would cuase memory leakage.

延迟问题

不知道作者这个程序有没有遇到就是原始图片计数开始到一开始第六帧的时候，结果第一帧才处理完，然后就是会一直相差那么五帧的延迟，但是实时性感觉不会卡

如果想使用多摄像头进行推理，有什么好的建议吗？

如果分辨率变成19201080的话会出现目标框和物体不在同一个位置，会有偏移，分辨率640480的话是正常的

Add capability for another input image size

Hi, thank you for your work
I noticed that current version support only 640*640 input image size. Can you share what to do to use that repo for another image size (what yolo8 was trained on)

And can you give some explanation why this values are 160?:

#define PROTO_HEIGHT  (160)
#define PROTO_WEIGHT (160)

感谢作者提供的源码，已在此项目进行二次开发，根据pose网络输出的关键点坐标进行人体行为分类，在3588上单路视频推理帧率可达110帧

compile error 编译失败

请问我为啥编译失败啊 Why am I couldn't compile it
(base) liable@liable-G3:~/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/build$ cmake -DCMAKE_TOOLCHAIN_FILE=/home/liable/Documents/RKNN/rk3588_dev_rootfs/toolchain-aarch64.cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..

-- The C compiler identification is GNU 6.3.1
-- The CXX compiler identification is GNU 6.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/liable/Documents/RKNN/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/liable/Documents/RKNN/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenCV: /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr (found version "4.5.4")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/build

(base) liable@liable-G3:~/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/build$ make

[ 7%] Building CXX object utils/CMakeFiles/yolov8-kaylordut.dir/camera.cpp.o
In file included from /home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/camera.cpp:5:0:
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/include/camera.h: In constructor ‘Camera::Camera(uint16_t, cv::Size, double)’:
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/include/camera.h:16:20: warning: ‘Camera::capture_’ will be initialized after [-Wreorder]
cv::VideoCapture capture_;
^~~~~~~~
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/include/camera.h:15:12: warning: ‘cv::Size Camera::size_’ [-Wreorder]
cv::Size size_;
^~~~~
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/camera.cpp:10:1: warning: when initialized here [-Wreorder]
Camera::Camera(uint16_t index, cv::Size size, double framerate)
^~~~~~
[ 15%] Building CXX object utils/CMakeFiles/yolov8-kaylordut.dir/image_process.cpp.o
In file included from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/Core:162:0,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/dataType.h:6,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/kalmanFilter.h:3,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/STrack.h:4,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/BYTETracker.h:3,
from /home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/include/image_process.h:6,
from /home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/image_process.cpp:5:
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:504:25: error: ‘invoke_result’ in namespace ‘std’ does not name a template type
typedef typename std::invoke_result<F, ArgTypes...>::type type1;
^~~~~~~~~~~~~
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:504:38: error: expected unqualified-id before ‘<’ token
typedef typename std::invoke_result<F, ArgTypes...>::type type1;
^
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:505:31: error: ‘type1’ was not declared in this scope
typedef typename remove_all::type type;
^~~~~
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:505:36: error: template argument 1 is invalid
typedef typename remove_all::type type;
^
In file included from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/Core:162:0,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/dataType.h:6,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/kalmanFilter.h:3,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/STrack.h:4,
from /home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/BYTETracker.h:3,
from /home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/include/image_process.h:6,
from /home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/image_process.cpp:5:
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:616:25: error: ‘invoke_result’ in namespace ‘std’ does not name a template type
typedef typename std::invoke_result<F, ArgTypes...>::type type1;
^~~~~~~~~~~~~
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:616:38: error: expected unqualified-id before ‘<’ token
typedef typename std::invoke_result<F, ArgTypes...>::type type1;
^
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:617:31: error: ‘type1’ was not declared in this scope
typedef typename remove_all::type type;
^~~~~
/home/liable/Documents/RKNN/rk3588_dev_rootfs/rootfs/usr/include/eigen3/Eigen/src/Core/util/Meta.h:617:36: error: template argument 1 is invalid
typedef typename remove_all::type type;
^
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/image_process.cpp: In member function ‘std::unique_ptrcv::Mat ImageProcess::Convert(const cv::Mat&)’:
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/image_process.cpp:51:12: warning: the compiler can assume that the address of ‘src’ will never be NULL [-Waddress]
if (&src == nullptr) {
~~~~~^~~~~~~~~~
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/image_process.cpp: In member function ‘void ImageProcess::ImagePostProcess(cv::Mat&, object_detect_result_list&)’:
/home/liable/Documents/YoloV8Pose/rk3588-yolo-demo/src/yolov8/utils/image_process.cpp:73:9: warning: unused variable ‘cls_id’ [-Wunused-variable]
int cls_id = od_results.results[0].cls_id;
^~~~~~
make[2]: *** [utils/CMakeFiles/yolov8-kaylordut.dir/build.make:90: utils/CMakeFiles/yolov8-kaylordut.dir/image_process.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:182: utils/CMakeFiles/yolov8-kaylordut.dir/all] Error 2
make: *** [Makefile:91: all] Error 2