Integration of SCINet with YOLOv8 for Low-Light Object Detection about ultralytics

[glenn-jocher]

@ChengDaTsai hello,

Thank you for your detailed question and for sharing your approach to integrating SCINet with YOLOv8 for low-light object detection. Your modifications and the experiments you've conducted are quite interesting. Let's delve into a few potential reasons and suggestions to improve performance:

1. Model Compatibility and Training Stability:

   • Ensure that the integration of SCINet with YOLOv8 is stable. Sometimes, adding a new network layer can disrupt the training dynamics. Double-check the implementation and ensure that the gradients flow correctly through SCINet and into the YOLOv8 backbone.
   • Verify that the SCINet weights are being updated correctly during training.

2. Hyperparameter Tuning:

   • The introduction of SCINet might require different hyperparameters for optimal performance. Consider experimenting with learning rates, batch sizes, and other training parameters.
   • You might also want to try different configurations for the SCINet itself, such as varying the number of layers or the type of activation functions used.

3. Image Enhancement Quality:

   • Evaluate the quality of the images enhanced by SCINet. Overexposure or increased noise, as you mentioned, can negatively impact detection performance. It might be beneficial to fine-tune SCINet separately to ensure it enhances images in a way that is beneficial for object detection.
   • Consider using metrics like PSNR (Peak Signal-to-Noise Ratio) or SSIM (Structural Similarity Index) to quantify the enhancement quality.

4. Data Augmentation:

   • Augmenting the dataset with various lighting conditions might help the model generalize better. Techniques like random brightness adjustment, contrast changes, and noise addition can simulate different low-light scenarios.

5. Separate Training Phases:

   • Train SCINet and YOLOv8 separately before integrating them. First, train SCINet to enhance low-light images effectively. Then, use the enhanced images to train YOLOv8. Finally, fine-tune the integrated model.

6. Evaluation Metrics:

   • Ensure that the evaluation metrics are appropriate for your specific use case. Sometimes, standard metrics might not fully capture the performance improvements in low-light conditions.

Here's a small code snippet to ensure SCINet is correctly integrated and the gradients are flowing:

# Example of integrating SCINet with YOLOv8
class EnhanceNetwork(nn.Module):
    def __init__(self, in_channels):
        super(EnhanceNetwork, self).__init__()
        # Define your SCINet layers here
        self.conv1 = nn.Conv2d(in_channels, 64, kernel_size=3, stride=1, padding=1)
        # Add more layers as needed

    def forward(self, x):
        x = self.conv1(x)
        # Forward pass through other layers
        return x

# Register SCINet in the YOLOv8 backbone
backbone = [
    [-1, 1, EnhanceNetwork, [3]],  # SCINet
    [-1, 1, Conv, [64, 3, 2]],
    # Add remaining YOLOv8 layers
]

Lastly, ensure you are using the latest versions of torch and ultralytics to benefit from any recent updates or bug fixes. If you haven't already, please upgrade your packages and try again.

Feel free to share any additional details or results from further experiments. The community and the Ultralytics team are here to help!

[github-actions]

👋 Hello @ChengDaTsai, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[ChengDaTsai]

Thank you for your valuable advice. I will make various attempts based on your suggestions and share the results here !

[ChengDaTsai]

I've encountered several issues during my attempts. Currently, I've added a file called SCINet.py in the nn>modules directory, where I've declared the specific class for SCINet, as shown below.

The modified yolov8_sci.yaml is shown in the following image:

And then I noticed that in task.py, the function attempt_load_weights seems capable of loading weights from multiple models at once. Suppose the pretrained weights for my SCINet are a and for YOLOv8 are b, does this mean I can load multiple module weights with weight = [a_path, b_path]? I will appreciat that if there is an example to demonstrate how this is done? I try fiqure this out to load the separately pretrained weights on the ExDark dataset for each module(SCINet , YOLOv8) for better fine-tuning.
Is this thought goes right?

[glenn-jocher]

Hello @ChengDaTsai,

Thank you for sharing the details of your integration efforts with SCINet and YOLOv8. Your approach is quite innovative! Let's address your questions and provide some guidance on loading multiple pretrained weights.

Loading Multiple Pretrained Weights

The attempt_load_weights function in task.py is indeed designed to load weights, but it typically loads a single set of weights for the entire model. To load separate pretrained weights for SCINet and YOLOv8, you will need to modify the model's initialization process to handle multiple weight files.

Here's a step-by-step approach to achieve this:

1. Separate Weight Loading:

   • Load the SCINet weights separately and initialize the SCINet module.
   • Load the YOLOv8 weights separately and initialize the rest of the YOLOv8 model.

2. Custom Initialization:

   • Modify the model's initialization to accept multiple weight files and load them into the respective parts of the model.

Example Code

Below is an example to demonstrate how you can load separate weights for SCINet and YOLOv8:

import torch
from ultralytics import YOLO
from ultralytics.nn.modules import SCINet  # Ensure SCINet is correctly imported

# Define your custom model class
class CustomYOLOv8(YOLO):
    def __init__(self, sci_weights, yolo_weights):
        super().__init__()
        self.sci_net = SCINet()  # Initialize SCINet
        self.yolo = YOLO('yolov8.yaml')  # Initialize YOLOv8 with its config

        # Load SCINet weights
        self.sci_net.load_state_dict(torch.load(sci_weights))

        # Load YOLOv8 weights
        self.yolo.load_state_dict(torch.load(yolo_weights))

    def forward(self, x):
        x = self.sci_net(x)  # Pass through SCINet
        x = self.yolo(x)  # Pass through YOLOv8
        return x

# Paths to your pretrained weights
sci_weights_path = 'path/to/sci_weights.pth'
yolo_weights_path = 'path/to/yolo_weights.pth'

# Initialize the custom model
model = CustomYOLOv8(sci_weights_path, yolo_weights_path)

# Now you can train or fine-tune your model
model.train(data='coco128.yaml', epochs=100, imgsz=640)

Important Considerations

1. Compatibility: Ensure that the SCINet and YOLOv8 modules are compatible in terms of input and output dimensions.
2. Training Stability: Monitor the training process closely to ensure that the integration does not introduce instability.
3. Hyperparameter Tuning: You might need to adjust hyperparameters to achieve optimal performance with the integrated model.

Next Steps

• Reproducible Example: If you encounter further issues, please provide a minimum reproducible code example. This will help us better understand the problem and provide more targeted assistance. You can refer to our Minimum Reproducible Example Guide for more details.
• Version Check: Ensure you are using the latest versions of torch and ultralytics. If not, please upgrade your packages and try again.

Feel free to share any additional details or results from your experiments. We're here to help! 😊

[ChengDaTsai]

Based on your suggestion, I used the CustomYOLOv8 format for training and achieved very good results. However, I now have a question: the loss from my SCI_net did not participate in updating the entire model. It seems that the training was conducted using the YOLOv8 detection loss to update the entire SCI+YOLOv8 model. If this is the case, then the significant improvement in results might be due to an increase in the model size and the number of parameters?


I am now looking to incorporate the loss from SCINet into the YOLO loss, but I am unsure how to implement this. I defined a new loss (SCI's SmoothLoss and L2 loss) in utils > loss.py, but I don't know how to add SCI's SmoothLoss and L2 loss into the v8DetectionLoss calculations.

Could you provide assistance on how to include losses from other modules into v8DetectionLoss? Thank you very much!

[glenn-jocher]

Hello @ChengDaTsai,

Thank you for your detailed follow-up and congratulations on achieving good results with the CustomYOLOv8 format! 🎉

To address your question about incorporating the loss from SCINet into the YOLOv8 detection loss, you are correct that combining these losses can help fine-tune both networks simultaneously and potentially improve performance further. Here’s a concise guide on how to achieve this:

Step-by-Step Guide to Incorporate SCINet Loss into YOLOv8 Detection Loss

1. Define SCINet Loss Functions:
   Ensure your SCINet loss functions (e.g., SmoothLoss and L2Loss) are correctly implemented in utils/loss.py.

2. Modify the YOLOv8 Loss Calculation:
   Update the YOLOv8 loss calculation to include the SCINet losses. This involves modifying the v8DetectionLoss function to incorporate SCINet’s loss components.

Example Code

Here’s an example of how you can integrate SCINet losses into the YOLOv8 loss function:

# utils/loss.py

import torch
import torch.nn as nn

class SmoothLoss(nn.Module):
    def __init__(self):
        super(SmoothLoss, self).__init__()
        # Define your smooth loss components here

    def forward(self, x):
        # Implement the forward pass for smooth loss
        return smooth_loss

class L2Loss(nn.Module):
    def __init__(self):
        super(L2Loss, self).__init__()
        # Define your L2 loss components here

    def forward(self, x):
        # Implement the forward pass for L2 loss
        return l2_loss

class v8DetectionLoss(nn.Module):
    def __init__(self, sci_net):
        super(v8DetectionLoss, self).__init__()
        self.sci_net = sci_net
        self.smooth_loss = SmoothLoss()
        self.l2_loss = L2Loss()
        # Initialize other YOLOv8 loss components here

    def forward(self, predictions, targets):
        # Calculate YOLOv8 detection loss
        yolo_loss = self.calculate_yolo_loss(predictions, targets)

        # Calculate SCINet losses
        sci_output = self.sci_net(predictions)
        smooth_loss = self.smooth_loss(sci_output)
        l2_loss = self.l2_loss(sci_output)

        # Combine losses
        total_loss = yolo_loss + smooth_loss + l2_loss
        return total_loss

    def calculate_yolo_loss(self, predictions, targets):
        # Implement YOLOv8 loss calculation here
        return yolo_loss

Integrate the Custom Loss into Training

Ensure that your training loop uses this custom loss function:

# training script

from utils.loss import v8DetectionLoss
from ultralytics import YOLO

# Initialize SCINet and YOLOv8 models
sci_net = SCINet()
yolo_model = YOLO('yolov8.yaml')

# Initialize custom loss function
loss_fn = v8DetectionLoss(sci_net)

# Training loop
for epoch in range(num_epochs):
    for images, targets in dataloader:
        predictions = yolo_model(images)
        loss = loss_fn(predictions, targets)
        
        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Important Considerations

1. Loss Weighting: You might need to experiment with different weights for the SCINet losses and the YOLOv8 loss to find the optimal balance.
2. Training Stability: Monitor the training process to ensure that the combined loss does not introduce instability.

Feel free to share any additional details or results from your experiments. The community and the Ultralytics team are here to help! 😊

Best of luck with your continued work on low-light object detection!

[ChengDaTsai]

Thank you for your clear and concise explanation of the entire loss integration process, but I am still encountering some problems in implement. Firstly, in utils>loss.py, v8DetectionLoss does not seem to have a forward method, and there are many parameters that I am unclear about, so I dare not delete them arbitrarily.

class v8DetectionLoss:
  def __init__(self, model):  # model must be de-paralleled
      """Initializes v8DetectionLoss with the model, defining model-related properties and BCE loss function."""
      device = next(model.parameters()).device  # get model device
      h = model.args  # hyperparameters

      m = model.model[-1]  # Detect() module
      self.bce = nn.BCEWithLogitsLoss(reduction="none")
      self.hyp = h
      self.stride = m.stride  # model strides
      self.nc = m.nc  # number of classes
      self.no = m.nc + m.reg_max * 4
      self.reg_max = m.reg_max
      self.device = device

      self.use_dfl = m.reg_max > 1

      self.assigner = TaskAlignedAssigner(topk=10, num_classes=self.nc, alpha=0.5, beta=6.0)
      self.bbox_loss = BboxLoss(m.reg_max - 1, use_dfl=self.use_dfl).to(device)
      self.proj = torch.arange(m.reg_max, dtype=torch.float, device=device)

  def preprocess(self, targets, batch_size, scale_tensor):
      """Preprocesses the target counts and matches with the input batch size to output a tensor."""
      if targets.shape[0] == 0:
          out = torch.zeros(batch_size, 0, 5, device=self.device)
      else:
          i = targets[:, 0]  # image index
          _, counts = i.unique(return_counts=True)
          counts = counts.to(dtype=torch.int32)
          out = torch.zeros(batch_size, counts.max(), 5, device=self.device)
          for j in range(batch_size):
              matches = i == j
              n = matches.sum()
              if n:
                  out[j, :n] = targets[matches, 1:]
          out[..., 1:5] = xywh2xyxy(out[..., 1:5].mul_(scale_tensor))
      return out

  def bbox_decode(self, anchor_points, pred_dist):
      """Decode predicted object bounding box coordinates from anchor points and distribution."""
      if self.use_dfl:
          b, a, c = pred_dist.shape  # batch, anchors, channels
          pred_dist = pred_dist.view(b, a, 4, c // 4).softmax(3).matmul(self.proj.type(pred_dist.dtype))
          # pred_dist = pred_dist.view(b, a, c // 4, 4).transpose(2,3).softmax(3).matmul(self.proj.type(pred_dist.dtype))
          # pred_dist = (pred_dist.view(b, a, c // 4, 4).softmax(2) * self.proj.type(pred_dist.dtype).view(1, 1, -1, 1)).sum(2)
      return dist2bbox(pred_dist, anchor_points, xywh=False)

  def __call__(self, preds, batch):
      """Calculate the sum of the loss for box, cls and dfl multiplied by batch size."""
      loss = torch.zeros(3, device=self.device)  # box, cls, dfl
      feats = preds[1] if isinstance(preds, tuple) else preds
      pred_distri, pred_scores = torch.cat([xi.view(feats[0].shape[0], self.no, -1) for xi in feats], 2).split(
          (self.reg_max * 4, self.nc), 1
      )

      pred_scores = pred_scores.permute(0, 2, 1).contiguous()
      pred_distri = pred_distri.permute(0, 2, 1).contiguous()

      dtype = pred_scores.dtype
      batch_size = pred_scores.shape[0]
      imgsz = torch.tensor(feats[0].shape[2:], device=self.device, dtype=dtype) * self.stride[0]  # image size (h,w)
      anchor_points, stride_tensor = make_anchors(feats, self.stride, 0.5)

      # Targets
      targets = torch.cat((batch["batch_idx"].view(-1, 1), batch["cls"].view(-1, 1), batch["bboxes"]), 1)
      targets = self.preprocess(targets.to(self.device), batch_size, scale_tensor=imgsz[[1, 0, 1, 0]])
      gt_labels, gt_bboxes = targets.split((1, 4), 2)  # cls, xyxy
      mask_gt = gt_bboxes.sum(2, keepdim=True).gt_(0)

      # Pboxes
      pred_bboxes = self.bbox_decode(anchor_points, pred_distri)  # xyxy, (b, h*w, 4)

      _, target_bboxes, target_scores, fg_mask, _ = self.assigner(
          pred_scores.detach().sigmoid(),
          (pred_bboxes.detach() * stride_tensor).type(gt_bboxes.dtype),
          anchor_points * stride_tensor,
          gt_labels,
          gt_bboxes,
          mask_gt,
      )

      target_scores_sum = max(target_scores.sum(), 1)

      # Cls loss
      # loss[1] = self.varifocal_loss(pred_scores, target_scores, target_labels) / target_scores_sum  # VFL way
      loss[1] = self.bce(pred_scores, target_scores.to(dtype)).sum() / target_scores_sum  # BCE

      # Bbox loss
      if fg_mask.sum():
          target_bboxes /= stride_tensor
          loss[0], loss[2] = self.bbox_loss(
              pred_distri, pred_bboxes, anchor_points, target_bboxes, target_scores, target_scores_sum, fg_mask
          )

      loss[0] *= self.hyp.box  # box gain
      loss[1] *= self.hyp.cls  # cls gain
      loss[2] *= self.hyp.dfl  # dfl gain

      return loss.sum() * batch_size, loss.detach()  # loss(box, cls, dfl)

Secondly, when I train, I use a simple training command like :
model.train(task='detect', data='cfg/datasets/ExDark_yolo.yaml', epochs=300, imgsz=640, batch=16, device=1).
If I want to integrate the Custom Loss into Training, it seems I need to go to engine>trainer.py's do_train to make modifications. In do_train, there are many parameters that I am unclear about as well. I apologize for these problems caused by my ignorance, but I still hope you can help me understand how to do these implements specifically.

[glenn-jocher]

Hello @ChengDaTsai,

Thank you for your detailed follow-up and for sharing the code snippets. Let's address your concerns step-by-step to help you integrate SCINet's loss into the YOLOv8 loss function and ensure a smooth training process.

1. Adding Forward Method to v8DetectionLoss

To integrate SCINet's loss into the YOLOv8 loss function, you need to ensure that the v8DetectionLoss class has a forward method. This method will compute the combined loss. Here's how you can modify the v8DetectionLoss class:

# utils/loss.py

import torch
import torch.nn as nn

class SmoothLoss(nn.Module):
    def __init__(self):
        super(SmoothLoss, self).__init__()
        # Define your smooth loss components here

    def forward(self, x):
        # Implement the forward pass for smooth loss
        return smooth_loss

class L2Loss(nn.Module):
    def __init__(self):
        super(L2Loss, self).__init__()
        # Define your L2 loss components here

    def forward(self, x):
        # Implement the forward pass for L2 loss
        return l2_loss

class v8DetectionLoss(nn.Module):
    def __init__(self, model, sci_net):
        super(v8DetectionLoss, self).__init__()
        self.sci_net = sci_net
        self.smooth_loss = SmoothLoss()
        self.l2_loss = L2Loss()
        # Initialize other YOLOv8 loss components here

        device = next(model.parameters()).device  # get model device
        h = model.args  # hyperparameters

        m = model.model[-1]  # Detect() module
        self.bce = nn.BCEWithLogitsLoss(reduction="none")
        self.hyp = h
        self.stride = m.stride  # model strides
        self.nc = m.nc  # number of classes
        self.no = m.nc + m.reg_max * 4
        self.reg_max = m.reg_max
        self.device = device

        self.use_dfl = m.reg_max > 1

        self.assigner = TaskAlignedAssigner(topk=10, num_classes=self.nc, alpha=0.5, beta=6.0)
        self.bbox_loss = BboxLoss(m.reg_max - 1, use_dfl=self.use_dfl).to(device)
        self.proj = torch.arange(m.reg_max, dtype=torch.float, device=device)

    def preprocess(self, targets, batch_size, scale_tensor):
        # Preprocess targets
        ...

    def bbox_decode(self, anchor_points, pred_dist):
        # Decode bounding boxes
        ...

    def forward(self, preds, batch):
        # Calculate YOLOv8 detection loss
        yolo_loss = self.calculate_yolo_loss(preds, batch)

        # Calculate SCINet losses
        sci_output = self.sci_net(preds)
        smooth_loss = self.smooth_loss(sci_output)
        l2_loss = self.l2_loss(sci_output)

        # Combine losses
        total_loss = yolo_loss + smooth_loss + l2_loss
        return total_loss

    def calculate_yolo_loss(self, preds, batch):
        # Implement YOLOv8 loss calculation here
        ...

2. Integrating Custom Loss into Training

To integrate the custom loss into the training process, you need to modify the do_train method in engine/trainer.py. Here's a concise guide:

1. Modify do_train Method:
   Update the do_train method to use the custom loss function.

# engine/trainer.py

from utils.loss import v8DetectionLoss

def do_train(model, dataloader, epochs, device):
    sci_net = SCINet()  # Initialize SCINet
    loss_fn = v8DetectionLoss(model, sci_net)  # Initialize custom loss function

    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

    for epoch in range(epochs):
        model.train()
        for batch in dataloader:
            images, targets = batch['images'].to(device), batch['targets'].to(device)
            preds = model(images)
            loss = loss_fn(preds, targets)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            print(f"Epoch {epoch}, Loss: {loss.item()}")

2. Training Command:
   Use the modified do_train method in your training script.

# training script

from engine.trainer import do_train
from ultralytics import YOLO

# Initialize model and dataloader
model = YOLO('yolov8.yaml')
dataloader = ...  # Initialize your dataloader

# Train the model
do_train(model, dataloader, epochs=300, device='cuda:0')

Important Considerations

• Loss Weighting: You might need to experiment with different weights for the SCINet losses and the YOLOv8 loss to find the optimal balance.
• Training Stability: Monitor the training process to ensure that the combined loss does not introduce instability.

Feel free to share any additional details or results from your experiments. The community and the Ultralytics team are here to help! 😊

Best of luck with your continued work on low-light object detection!