Adding an external feature input to head (concatenate with first convolution layers of head) of YOLOv8 object detector and finetune for custom data. about ultralytics

[github-actions]

👋 Hello @billalkuet07, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

Hello! It's great to hear about your plans to experiment with enhancing YOLOv8 by adding an external feature input. For incorporating additional inputs into the model's architecture and tuning it for custom data, both options you've considered have their merits. However, the second option might give you a more straightforward path by leveraging the existing infrastructure of YOLOv8 for data handling, training, and testing workflows.

Here's a brief guide on how you might proceed with option 2:

• Model Architecture Modification: You'll likely need to adjust the architecture to concatenate the external features. Look into the .yaml file defining the model structure, possibly modifying it to introduce your changes at specified layers.
• Custom Dataset Loader: Since you have additional numerical data alongside the YOLO formatted annotations, you would need to modify the dataset loading mechanism. Extending datasets.py to incorporate the loading and preprocessing of your additional inputs will be required.
• Finetuning: Utilize the pre-trained weights for YOLOv8 and commence training with your custom dataset ensuring your modified data loader is effectively used. The training script, possibly train.py, may need minimal adjustments here, mostly to ensure it properly calls your modified dataset loader.

Here's a very basic example snippet illustrating how you could potentially modify the dataset loader to include your external features:

# Pseudo-code for custom data loading with external features

class CustomDataset(Dataset):
    def __init__(self, path, img_size=640, augment=False, ...):
        self.path = path
        ...
        # Load your additional data here
        self.external_features = load_external_features(path)

    def __getitem__(self, index):
        ...
        # Normal image loading
        img, labels = load_image_and_labels()

        # Example to get the corresponding external features for the image
        ext_features = self.external_features[index]

        return img, labels, ext_features

This just scratches the surface, but it should give you a starting point. Modifications in models/common.py might also become necessary, depending on how you plan to integrate the external features into the YOLOv8 architecture.

I recommend deeply studying the specific files (models/yolo.py, utils/datasets.py, etc.) to better understand where and how to implement your changes efficiently.

Kudos on taking on such a project, and we're excited to see what enhancements you can bring to YOLOv8! Remember, experimenting is key—don't hesitate to test different approaches to find what works best for your needs. Good luck! 🚀

[billalkuet07]

Thanks for your prompt response. This is very helpful.

[glenn-jocher]

You're welcome! I'm glad to hear that you found the information useful. If you have any more questions or need further clarification as you progress with your project, don't hesitate to reach out. Happy coding! 😊