Comments (6)
Hi, thanks a lot for the report of this problem.
There is a args web_data_root
at the config file, e.g., https://github.com/Wangt-CN/DisCo/blob/main/config/ref_attn_clip_combine_controlnet_imgspecific_ft/webtan_S256L16_xformers_upsquare.py#L26C5-L26C101. Please change it to your own local dev path.
You may also need to modity the ft_idx
here (https://github.com/Wangt-CN/DisCo/blob/main/dataset/tiktok_controlnet_t2i_imagevar_combine_specifcimg_web_upsquare.py#L109), which is the specific dir name of the specific video frames.
Sorry that I am not at the workstation now so that can not refine this human-specific ft dataloader. If you finally work it out, you are also welcome to submit a pull request.
Best,
Tan
from disco.
Hi, thanks for your great interests.
May I first confirm the stage you want to ask is "fine-tuning" or "human-specific fine-tuning"?
We use tiktok data for fine-tuning, but for human-specific fine-tuning, actually you can use any data you want.
from disco.
The training process is: 1) Pre-training --> 2) General fine-tuning (on tiktok data).--> 3) Human-specific fine-tuning (optional, and you can expect a much better results if you have a human-specific training data, e.g, anime).
from disco.
Thanks for the quick response:)
Yes exactly, I'm interested in performing human specific finetuning on my data.
from disco.
Generally, you can follow this setup: https://github.com/Wangt-CN/DisCo#human-specific-fine-tuning.
But actually, we do not use Tiktok data in human-specific ft. Would you please provide the running command and the error message?
For your own data, you need to first get the human mask and the openpose annotation. You can choose to process by your own or use the toolkit provided by us.
Best,
Tan
from disco.
Indeed I am following the instructions provided, I prepared the masks and poses as well.
The command is cmd = f'{python_path} {script_path} \ --cf {config_path} --do_train --root_dir {project_dir} \ --local_train_batch_size 32 --local_eval_batch_size 32 --log_dir {project_dir} \ --epochs 20 --deepspeed --eval_step 500 --save_step 500 --gradient_accumulate_steps 1 \ --learning_rate 1e-3 --fix_dist_seed --loss_target "noise" \ --unet_unfreeze_type "crossattn" \ --refer_sdvae --ref_null_caption False --combine_clip_local --combine_use_mask --conds "poses" "masks" \ --freeze_pose True --freeze_background False \ --pretrained_model {base_model_ckpt} \ --ft_iters 500 --ft_one_ref_image False --ft_idx {hf_data_dir} --strong_aug_stage1 True --strong_rand_stage2 True'
and the error is:
────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /home/ubuntu/dataspan/dataspan_research/dataspan_research/external_repos/DisCo/finetune_sdm_yaml │ │ .py:209 in <module> │ │ │ │ 206 │ # main_worker(parsed_args) │ │ 207 │ from utils.args import sharedArgs │ │ 208 │ parsed_args = sharedArgs.parse_args() │ │ ❱ 209 │ main_worker(parsed_args) │ │ 210 │ │ │ │ /home/ubuntu/dataspan/dataspan_research/dataspan_research/external_repos/DisCo/finetune_sdm_yaml │ │ .py:99 in main_worker │ │ │ │ 96 │ │ │ train_dataset = BaseDataset(args, args.train_yaml, split='train', preprocess │ │ 97 │ │ │ eval_dataset = BaseDataset(args, args.val_yaml, split='val', preprocesser=mo │ │ 98 │ │ else: │ │ ❱ 99 │ │ │ train_dataset = BaseDataset(args, args.train_yaml, split='train') │ │ 100 │ │ │ eval_dataset = BaseDataset(args, args.val_yaml, split='val') │ │ 101 │ │ │ │ 102 │ │ train_info = get_loader_info(args, args.local_train_batch_size, │ │ │ │ /home/ubuntu/dataspan/dataspan_research/dataspan_research/external_repos/DisCo/dataset/tiktok_co │ │ ntrolnet_t2i_imagevar_combine_specifcimg_web_upsquare.py:133 in __init__ │ │ │ │ 130 │ │ │ │ │ image_files_idx = list(open(os.path.join(folder_path, 'image_list.tx │ │ 131 │ │ │ │ │ image_files_idx = [file.strip() for file in image_files_idx] │ │ 132 │ │ │ │ else: │ │ ❱ 133 │ │ │ │ │ image_files_idx = os.listdir(folder_path) │ │ 134 │ │ │ │ │ image_files_idx = [file for file in image_files_idx if file.endswith │ │ 135 │ │ │ else: │ │ 136 │ │ │ │ assert split == 'val' │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ FileNotFoundError: [Errno 2] No such file or directory: './blob_dir/debug_output/video_sythesis/dataset/Lindsey_0504_youtube/frames_tan//home/ubuntu/dataspan/common_data/human_specific_finetune'
I assume that some paths are coded into the scripts pointing to data that I don't have?
Thanks!
from disco.
Related Issues (20)
- Error when mpirun -np 3? HOT 2
- how to caculate fvd metrics? HOT 1
- Where can I get the 10K tiktok style test split?
- Questions about image size
- [BUG] a bug in the dataset/tiktok_video_dataset.py
- How can I get "More TikTok-Style Training Data" please? HOT 1
- the code for computing PSNR is wrong HOT 4
- None
- huggingface demo broken HOT 1
- How were these files generated? For validation and training.
- jax and jaxlib latest version problem
- None
- Jimmy HOT 1
- How to use DeepSpeed for multi-GPU training instead of using mpirun?
- Is the TikTok dataset you provide the MoreTikTok dataset?
- 能不能提供多机多卡的运行脚本
- drop_pose_ratio is not defined (temporal module finetuning)
- Typo
- solved
- A problem about diffusers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from disco.