ControlLoRA Version 3: A Lightweight Neural Network To Control Stable Diffusion Spatial Information Version 3
ControlLoRA Version 3 is a neural network structure extended from ControlNet to control diffusion models by adding extra conditions.
Inspired by ControlLoRA, control-lora-v2 and script train_controlnet.py from diffusers, control-lora-v3 does not add new features, but provides a PEFT implement of ControlLoRA.
- Jun. 08, 2024. Norm layer is trainable.
- May. 19, 2024. Add DoRA.
To train ControlLoRA, you should have image-conditioning_image-text datasets. Of course you can hardly train on LAION-5B dataset in direct like Stable Diffusion. Here are some:
- fusing/fill50k. I do not suggest you to train ControlLoRA seriously as it is simple and lack of diversity.
- HighCWu/diffusiondb_2m_first_5k_canny. A small canny dataset. Here is poloclub/diffusiondb dataset. For canny condition, you can easily generate your own dataset.
- Nahrawy/VIDIT-Depth-ControlNet. Depth map? Heat map? But it is good!
- SaffalPoosh/scribble_controlnet_dataset. Many duplicate images. I suggest you synthesize your dataset.
Stable Diffusion v1-5 is the base model.
Stable Diffusion v1-4, Stable Diffusion v2-1 need to be vertified.
Stable Diffusion XL needs to be vertified, but probably does not work.
You can train either ControlNet or ControlLoRA using script train_control_lora.py.
By observation, training 50000 steps with batch size of 4 is the balance between image quality, control ability and time.
accelerate launch train_control_lora.py \
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
--output_dir="controlnet-model" \
--dataset_name="fusing/fill50k" \
--resolution=512 \
--learning_rate=1e-5 \
--train_batch_size=4 \
--max_train_steps=100000 \
--tracker_project_name="controlnet" \
--checkpointing_steps=5000 \
--validation_steps=5000 \
--report_to wandb
To train ControlLoRA, add --use_lora
in start command to activate it.
accelerate launch train_control_lora.py \
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
--output_dir="control_lora-model" \
--dataset_name="fusing/fill50k" \
--resolution=512 \
--learning_rate=1e-4 \
--train_batch_size=4 \
--max_train_steps=100000 \
--tracker_project_name="control_lora" \
--checkpointing_steps=5000 \
--validation_steps=5000 \
--report_to wandb \
--use_lora \
--lora_r=32
You can also train ControlLoRA / ControlNet with your own dataset.
accelerate launch train_control_lora.py \
--pretrained_model_name_or_path="stable-diffusion-v1-5" \
--output_dir="control_lora-model" \
--conditioning_image_column="hint" \
--image_column="jpg" \
--caption_column="txt" \
--resolution=512 \
--learning_rate=1e-4 \
--train_batch_size=4 \
--num_train_epochs=3 \
--max_train_steps=100000 \
--tracker_project_name="control_lora" \
--checkpointing_steps=5000 \
--validation_steps=5000 \
--report_to wandb \
--use_lora \
--lora_r=32 \
--custom_dataset="fill50k"
If you want to train ControlNet, you have already got it. If you got a lora, merge it!
python merge_lora.py
Original image:
Output:
@software{lavinal7122024controllorav3,
author = {lavinal712},
month = {5},
title = {{ControlLoRA Version 3: A Lightweight Neural Network To Control Stable Diffusion Spatial Information Version 3}},
url = {https://github.com/lavinal712/control-lora-v3},
version = {1.0.0},
year = {2024}
}