Performance still changes even after layers freezed during training about ultralytics

[github-actions]

👋 Hello @zvant, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

Hello,

It's intriguing that you're seeing changes in the pose estimation performance even after freezing the layers. Despite the logs confirming the layers were frozen, there could be a few things going on here.

One possibility might involve subtle interactions between the layers not accounted for by simply freezing, especially considering the complex multi-head architecture you're working with. It could be beneficial to double-check that no unexpected updates are being made to parameters or states outside of the detection head during the training.

Also, ensure that the training regime (learning rates, batches, data augmentation) remains consistent across the different training sessions as inconsistencies here might indirectly affect the model's behavior even if the layers are nominally frozen.

If you haven't already done so, a thorough comparison of pre- and post-training activations for the frozen layers could reveal if they are indeed unchanged.

Regarding your workaround using deepcopy, it's a clever approach to ensure absolute consistency in non-trained parts of the model, although it ideally shouldn't be necessary if freezing works as intended.

Feel free to share any further observations or code snippets, and I'm certain we can dive deeper into this issue together. Keep experimenting! 🚀

[zvant]

@glenn-jocher Thanks for the reply!
I also confirm that the VRAM usage when training with freezed layers is significantly lower than training whole network. And it trains much faster. So I am pretty sure gradients are not calculated for the freezed layers. So there should be some mechanism changing the model's performance that I am not aware of, maybe some sort of EMA or precision conversion.

But the workaround should be good enough for me, for now.

[glenn-jocher]

Hello,

Great to hear that the VRAM usage and training speed observations align with the layers being frozen correctly! It sounds like you're on the right track. The changes in performance might indeed be related to factors like exponential moving averages (EMA) or precision conversions that aren't immediately obvious.

Your workaround is a smart move to ensure consistency while you explore the underlying cause. If you need to delve deeper into this, checking any involved EMA updates or precision settings during training could provide more insights.

Keep up the good work, and don't hesitate to reach out if you have more questions or updates! 🌟