baxtrax / real-time-yolov8-dashboard

A Realtime YOLOv8 Dashboard which is built to be explainable and easy to use.

Real-time YOLOv8 Dashboard

A real-time YOLOv8 dashboard, was done as a small senior project to dive more into React using Next.js and real-time communication protocols. Allows the use of YOLOv8 (and technically later models like YOLOv9) lightweight object detection models in real-time including real-time Segmentation, Pose-Estimation, Heatmaps, and other configurable settings in real-time during detection. Program was built to be as efficient as possible and as fast as possible, achieving inference and display speeds near what were benchmarked by the YOLOv8 creators, while showing off more features and a dynamic, responsive UI.

Since this was done as a project, there are also project documents available, such as requirements, test deliverables, and design documents about the entirety of the program.

Software Design Document (SDD)

Requirements Document

Framerate of gif is locked at 7 FPS. Actual app performance is much faster (watch the Information panel on the demo for actual speeds up to 120 fps!)

Table of Contents

1. About The Code and Folders
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
   • Backend Launch
   • Frontend Launch

About The Code and Folders

The program uses a Server-Client Architecture (Alothough closer to MVC). The backend (Python, Flask, Pytorch, Swagger-ui) and the frontend (React, Next.js, MUI) have their own folders which contain each project.

A breakdown of the important frontend code folders is below. Frontend UI is available at http://127.0.0.1:5001.

• frontend/src/components/input
  • Holds all base input custom components used to create the UI; most components extend off the base.tsx component.
• frontend/src/components/output
  • Holds all components used for output. Many of these are fully custom due to the real-time nature of the outputs and styling to make them fit the theme.
• frontend/src/components/panel
  • Holds the panels that create the full UI on the frontend. This allows for easy responsive UI via flex-box grid.
• frontend/src/contexts
  • Holds the context providers which orchestrate the overall dynamic logic of the program

A breakdown of the important backend code folders is below. Backend Swagger UI is available at http://127.0.0.1:5001.

• backend/apis
  • Holds all of the API namespaces hosted by Flask and displayed on the Swagger UI
• backend/core
  • Holds all core functionality logic such as prediction, as well as utilities and other streaming-related logic.

(back to top)

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

You will need to have some sort of python. You will also need nodejs and react.

(Optional)

• Setup a python environment (conda or pyenv) to keep your development space tidy.
• A GPU. This will make generation much, much, quicker

Installation

1. Clone the repo

   git clone [email protected]:baxtrax/Realtime-YOLOv8-Dashboard.git

2. Install the required libraries

   cd frontend && pip install -r requirements.txt && cd ../backend && npm install

(back to top)

Usage

Backend Launch

1. Open a console and run the backend/app.py python file to launch the backend.

   cd backend && python app.py

2. (Optional) Open the backend swagger UI and test keepalive. Swagger UI is hosted at http://127.0.0.1:5001

Frontend Launch

1. Open a console and run the React project

   cd frontend && npm run dev

2. Open the frontend webpage. Its hosted at http://127.0.0.1:5001
3. Select a Model source. Thats it!
4. (Optional) Play around with all the settings, all can be changed in real-time with no delay, even swapping model sizes!

The website should try to match your system preferences for dark or light mode. If not, you can specify dark mode and light mode by clicking the sun icon above the Webcam panel.

(back to top)

real-time-yolov8-dashboard's Issues

Create Feature Settings Panel

Test Procedures

Load the main page completely. This test passes if the feature settings panel renders and functions per T.F.

Implement Settings for each feature

Test Procedures

Load the main page completely. Open the Dev Tools console. Enable a specific setting. This test passes if changing the feature specific settings for the enabled feature affects the functionality of the enabled feature.

Adjust UI to support mobile devices

Test Procedures

Load the main page completely. Open Dev Tools and toggle device emulation. Change dimensions to Iphone SE, Ipad Air, and Responsive with 1920x1080 and 1080x720. This test passes if all sub panels are loaded and scale without clipping.

Send post processing metrics

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if T.I.1 passes and the post processing sub panel renders and updates.

Setup real-time stream of webcam

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if the webcam stream connects in the Dev Tools console and the webcam stream is displayed in real time on the main page.

Add Queueing to popups

Test Procedures

Trigger multiple popups via different methods (connection errors, invalid inputs etc.) This test passes if the popup count is displayed and accurately depicts the number of current pop ups in the queue.

Send class filter

Test Procedures

Load the main page completely. Select a model size. This test passes if the changed classes filter is sent to the backend and verified via the Dev Tools console. If the Webcam is implemented, this can also be passed by verifying that only the classes specified are shown in object detections.

Create Top Prediction Panel

Test Procedures

Load the main page completely. This test passes if the top prediction panel renders and functions per T.L.

Create main page with responsive CSS

Test Procedures

Load the main page completely. And use the Dev Tools panel to scale the web app. This test has passed if all sub panels are loaded and scale without clipping.

Setup hotswap model size change

Test Procedures

Load the main page completely. Select a model size. This test passes if the changed model size is sent to the backend and verified via the Dev Tools console.

Integrate class and confidence into bbox detection

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if T.W.2 passes and the webcam stream has class and confidence displayed on the bounding boxes.

Add swappable IOU filter

Test Procedures

Load the main page completely. Select a model size. This test passes if the changed IOU filter is sent to the backend and verified via the Dev Tools console. If the Webcam is implemented, this can also be passed by sliding the IOU filter and verifying that more boxes appear as the IOU filter goes to 0.

Implement source grabbing and selection

Test Procedures

Load the main page completely. Select a source in the model settings. This test passes if the source is selected and successfully grabs frames from the source verified via the Dev Tools console.

Create Model Specific Settings Panel

Test Procedures

Load the main page completely. This test passes if the model specific settings panel renders and functions per T.P

Display top predictions

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if the webcam stream connects in the Dev Tools console and the predictions displayed are accurate and are in the top (ascending) order of highest confidence score.

Implement socket / API for metrics

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if the webcam stream connects in the Dev Tools console and the information sub panels update in real-time.

Define different popup levels and styles

Test Procedures

Trigger different level popups. This test passes if the color and level of the popup changes respectively to their level.

Send inference metrics

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if T.I.1 passes and the inference time ms and fps sub panels render and update.

Create Main Page UI with CSS Flexbox principles

Create Webcam Panel

Test Procedures

Load the main page completely. This test passes if the model output panel renders and functions per T.W.

Create popup/notification functionality

Test Procedures

Load the main page completely. Disconnect the frontend from the backend. This test passes if an error warning popup is displayed during a failed connection to the backend.

Restrict popups to one at a time

Trigger multiple popups. This test passes if only one popup is displayed at one time.

Create overall Main Page layout

Test Procedures

Load the main page completely. All sub panels are present.

Create Information Panel

Test Procedures

Load the main page completely. This test passes if the information panel renders and functions per T.I.

Setup Default Theme logic

Test Procedures

Load the main page completely. This test passes if the palette loaded is the dark mode palette.

Add features and bind to enable

Test Procedures

Load the main page completely. Open the Dev Tools console. This test passes if enabling each feature results in a broadcast settings change in the Dev Tools console and each implemented features enabled as intended.

Setup OpenCV loop and bbox detection

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if T.W.1 passes and the webcam stream has visible bounding boxes around objects.

Add swappable confidence filter

Test Procedures

Load the main page completely. Select a model size. This test passes if the changed confidence filter is sent to the backend and verified via the Dev Tools console. If the Webcam is implemented, this can also be passed by sliding the confidence filter and verifying only objects with that confidence or above are displayed.

Setup dark mode and light mode functionality

Test Procedures

Load the main page completely. Verify that both light mode and dark mode change the web app color palette via the dark mode and light mode toggle. This test passes if the color palette changes in both modes.

Create Feature Specific Settings Panel

Test Procedures

Load the main page completely. This test passes if the feature specific settings panel renders and functions per T.S.

Send top N predictions in real-time

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if the webcam stream connects in the Dev Tools console and the prediction panel updates in real time with valid current predictions.

Create Toggle Button for Theme

Test Procedures

Click the dark mode light mode toggle button. This test passes if the toggle button changes the web app palette accordingly.

Send # of objects

Test Procedures

Select a source and model size in the model settings panel if T.P passed. This test passes if T.I.1 passes and the number of objects sub panel renders and updates.

Add autoclear to popups queue

Test Procedures

Trigger multiple popups. This test passes if each popup is displayed in sequence and disappears after a set time.