[Q] Stuck Iteration and Sync Issue with wandb sweeps for YOLOv7 Hyperparameter Tuning about wandb

[thanos-wandb]

Hi @Achyar-CN thank you for writing in. While we don't know how the train.py script structure looks like, it seems that way of integrating Sweeps with your code, it can cause issues. You're starting a wandb run, and inside that run's singleton you're calling an external script. Are you starting another run in that script?

For your use case, it would be most recommended that you define the sweep hyperparameter in a yaml file, and you add in the file the python script and its arguments using a similar format to this one:
https://docs.wandb.ai/guides/sweeps/faq#how-do-i-use-custom-cli-commands-with-sweeps

You will then only have to execute those two CLI commands:

wandb sweep --project sweep-demo-cli config.yaml
wandb agent --count $NUM your-entity/sweep-demo-cli/sweepID

Please let me know if that would work for you, or if you needed further assistance with this.

[Achyar-CN]

Hello @thanos-wandb, thank you for your response. I'm not entirely sure if it was calling an external script and initiating another run. From my understanding, there already exists a wandb, run within the YOLOv7 training process by default. I believe that when I run the sweeps, it also executes the internal wandb scripts.

Btw, I followed your suggestion regarding the CLI commands, and it worked perfectly. Many thanks for your assistance 👍👍

[Achyar-CN]

Hi @elon1992 , here are the steps I've taken for YOLOv7, which I believe also apply to YOLOv9:

0. Ensure that you're in the active directory path for YOLOv7.
1. Create a new 'config.yaml' file, for example:

program: train.py
method: bayes
metric:
  goal: maximize
  name: metrics/mAP_0.5
parameters:
  epochs:
    values: [2, 3]
command:
  - ${env}
  - python
  - ${program}
  - ${args}
  - "--batch-size=32"
  - "--img=800"
  - "--device=0"
  - "--workers=4"
  - "--data=/content/gdrive/MyDrive/YOLOv7_tiny/yolov7/data/coco.yaml"
  - "--weights=/content/gdrive/MyDrive/YOLOv7_tiny/yolov7/yolov7-tiny.pt"
  - "--cfg=/content/gdrive/MyDrive/YOLOv7_tiny/yolov7/cfg/training/yolov7-tiny.yaml"

2. Run the first CLI command in the terminal, or a Jupyter notebook (by starting with %%bash).

%%bash
wandb sweep --project your_sweep_project_name config.yaml

3. From the output of the second step, you will receive the code for the second CLI command including the sweep_id, and next you can copy that code and add the number of hyperparameter tuning iterations you want. For example, the command for conducting hyperparameter tuning with 3 iterations could be:

%%bash
wandb agent --count 3 your-entity/sweep-demo-cli/sweepID

I hope this helps! Let me know if you have any questions.

[elon1992]

Hello @thanos-wandb, @Achyar-CN Could you please explain me how did you solve this with some more detailed approach. Thanks

[thanos-wandb]

Great, glad to hear this is now fixed, and thank you for the code snippet, @Achyar-CN - much appreciated!

[elon1992]

Hello @Achyar-CN , Thank you so much for the code snippet. But i have questions like the hyperparameter you chose for tuning is just epch what if i have to use batch-size which is used in args, and what if i want to try different optimizers or lr0 or anything. Will inserting the hyperparameter name and value works same as epoch?
Thank you

[Achyar-CN]

Hi @elon1992, from my understanding, there are two types of hyperparameters in YOLOv7: direct parser hyperparameters (such as epoch, batch-size, img, etc.) and hyperparameters included within hyp.scratch.tiny.yaml (like lr0, momentum, etc.). For instance, you could directly add both as shown below:

program: train.py
method: bayes
metric:
  goal: maximize
  name: metrics/mAP_0.5
parameters:
  epochs:
    values: [2, 3]
  batch-size:
    values: [8, 16, 32]
hyp:
  parameters:
    momentum: 
      values: [0.8]
    lr0:
        max: 0.02
        min: 0.005
        distribution: uniform
command:
  - ${env}
  - python
  - ${program}
  - ${args}
  - "--img=800"
  - "--device=0"
  - "--workers=4"
  - "--data=/content/gdrive/MyDrive/YOLOv7_tiny/yolov7/data/pothole.yaml"
  - "--weights=/content/gdrive/MyDrive/YOLOv7_tiny/yolov7/yolov7-tiny.pt"
  - "--cfg=/content/gdrive/MyDrive/YOLOv7_tiny/yolov7/cfg/training/yolov7_pothole-tiny.yaml"