GithubHelp home page GithubHelp logo

williamyang1991 / fresco Goto Github PK

View Code? Open in Web Editor NEW
602.0 10.0 58.0 10.18 MB

[CVPR 2024] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation

License: Other

Python 5.85% Jupyter Notebook 89.07% MATLAB 0.01% Shell 0.15% Batchfile 0.03% C 3.03% C++ 1.35% Cuda 0.51%
controlnet diffusion video-processing

fresco's Introduction

FRESCO - Official PyTorch Implementation

FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
Shuai Yang, Yifan Zhou, Ziwei Liu and Chen Change Loy
in CVPR 2024
Project Page | Paper | Supplementary Video | Input Data and Video Results

Web Demo

Abstract: The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains. Zero-shot methods seek to extend image diffusion models to videos without necessitating model training. Recent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency. In this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos. Extensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.

Features:

  • Temporal consistency: use intra-and inter-frame constraint with better consistency and coverage than optical flow alone.
    • Compared with our previous work Rerender-A-Video, FRESCO is more robust to large and quick motion.
  • Zero-shot: no training or fine-tuning required.
  • Flexibility: compatible with off-the-shelf models (e.g., ControlNet, LoRA) for customized translation.
teasers.mp4

Updates

  • [04/2024] Integrated to ๐Ÿค— Hugging Face. Enjoy the web demo!
  • [03/2024] Paper is released.
  • [03/2024] Code is released.
  • [03/2024] This website is created.

TODO

  • Integrate into Diffusers
  • Add Huggingface web demo
  • Add webUI.
  • Update readme
  • Upload paper to arXiv, release related material

Installation

  1. Clone the repository.
git clone https://github.com/williamyang1991/FRESCO.git
cd FRESCO

  1. You can simply set up the environment with pip based on requirements.txt

    • Create a conda environment and install torch >= 2.0.0. Here is an example script to install torch 2.0.0 + CUDA 11.8 :
    conda create --name diffusers python==3.8.5
conda activate diffusers
pip install torch==2.0.0 torchvision==0.15.1 --index-url https://download.pytorch.org/whl/cu118
    • Run pip install -r requirements.txt in an environment where torch is installed.
    • We have tested on torch 2.0.0/2.1.0 and diffusers 0.19.3
    • If you use new versions of diffusers, you need to modify my_forward()

  2. Run the installation script. The required models will be downloaded in ./model, ./src/ControlNet/annotator and ./src/ebsynth/deps/ebsynth/bin.

    • Requires access to huggingface.co
python install.py
  1. You can run the demo with run_fresco.py
python run_fresco.py ./config/config_music.yaml
  1. For issues with Ebsynth, please refer to issues

(1) Inference

WebUI (recommended)

python webUI.py

The Gradio app also allows you to flexibly change the inference options. Just try it for more details.

Upload your video, input the prompt, select the model and seed, and hit:

  • Run Key Frames: detect keyframes, translate all keyframes.
  • Run Propagation: propagate the keyframes to other frames for full video translation
  • Run All: Run Key Frames and Run Propagation

Select the model:

  • Base model: base Stable Diffusion model (SD 1.5)

overview

We provide abundant advanced options to play with

Advanced options for single frame processing
  1. Frame resolution: resize the short side of the video to 512.
  2. ControlNet related:
    • ControlNet strength: how well the output matches the input control edges
    • Control type: HED edge, Canny edge, Depth map
    • Canny low/high threshold: low values for more edge details
  3. SDEdit related:
    • Denoising strength: repaint degree (low value to make the output look more like the original video)
    • Preserve color: preserve the color of the original video
  4. SD related:
    • Steps: denoising step
    • CFG scale: how well the output matches the prompt
    • Added prompt/Negative prompt: supplementary prompts
  5. FreeU related:
    • FreeU first/second-stage backbone factor: =1 do nothing; >1 enhance output color and details
    • FreeU first/second-stage skip factor: =1 do nothing; <1 enhance output color and details
Advanced options for FRESCO constraints
  1. Keyframe related
    • Number of frames: Total frames to be translated
    • Number of frames in a batch: To avoid out-of-memory, use small batch size
    • Min keyframe interval (s_min): The keyframes will be detected at least every s_min frames
    • Max keyframe interval (s_max): The keyframes will be detected at most every s_max frames
  2. FRESCO constraints
    • FRESCO-guided Attention:
      • spatial-guided attention: Check to enable spatial-guided attention
      • cross-frame attention: Check to enable efficient cross-frame attention
      • temporal-guided attention: Check to enable temporal-guided attention
    • FRESCO-guided optimization:
      • spatial-guided optimization: Check to enable spatial-guided optimization
      • temporal-guided optimization: Check to enable temporal-guided optimization
  3. Background smoothing: Check to enable background smoothing (best for static background)
Advanced options for the full video translation
  1. Gradient blending: apply Poisson Blending to reduce ghosting artifacts. May slow the process and increase flickers.
  2. Number of parallel processes: multiprocessing to speed up the process. Large value (4) is recommended.

option

Command Line

We provide a flexible script run_fresco.py to run our method.

Set the options via a config file. For example,

python run_fresco.py ./config/config_music.yaml

We provide some examples of the config in config directory. Most options in the config is the same as those in WebUI. Please check the explanations in the WebUI section.

We provide a separate Ebsynth python script video_blend.py with the temporal blending algorithm introduced in Stylizing Video by Example for interpolating style between key frames. It can work on your own stylized key frames independently of our FRESCO algorithm. For the details, please refer to our previous work Rerender-A-Video

(2) Results

Key frame translation

a red car turns in the winter an African American boxer wearing black boxing gloves punches towards the camera, cartoon style a cartoon spiderman in black suit, black shoes and white gloves is dancing a beautiful woman holding her glasses in CG style

Full video translation

more_result_1.mp4

Citation

If you find this work useful for your research, please consider citing our paper:

@inproceedings{yang2024fresco,
โ€ƒtitle = {FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation},
โ€ƒauthor = {Yang, Shuai and Zhou, Yifan and Liu, Ziwei and and Loy, Chen Change},
 booktitle = {CVPR},
โ€ƒyear = {2024},
}

Acknowledgments

The code is mainly developed based on Rerender-A-Video, ControlNet, Stable Diffusion, GMFlow and Ebsynth.

fresco's People

Contributors

boxabirds avatar efwfe avatar jinwyp avatar singlezombie avatar williamyang1991 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

techthiyanes kerwinstyle enestatli camenduru joachim508 jinwyp gfanslee blizaine moxmoussa philtsip mylysddp superjili bleach4ever teloei efwfe startrekor hhy5277 silingyuan0 abdallah-amr-ali navezjt tonyabracadabra rakib1129 xiaoxianboy chenboos5 leetesla xymfei boxabirds yanhn majiajue gioannides g-force78 yomaser wangqi-xxxx xsoway singlezombie duya123 kamjin1996 jags111 strategist922 skymsg kevinfsz mingxta jackeyxiang gotuntun magicwang1111 tonghengcheng opensorceycw cocktailpeanut souvikm99 jinwook-shim zhangyuqiang aiforworlds happysharegithub happyxy masterhow whitefu joy1792

fresco's Issues

No video output after run the script

Hi:
First of all, this project is great.

After run the script

python run_fresco.py ./config/config_music.yaml

Everythings seems going well but there is no video output.
Is it necessary to manually execute the script below in order to generate the video?

python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75  --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps

The full log list below:


(venv) G:\AI\FRESCO>python run_fresco.py ./config/config_music.yaml
====================================================================================================
loading configuration...
batch_size: 8
cond_scale: 0.7
controlnet_type: hed
end_opt_step: 15
file_path: ./data/car-turn.mp4
gmflow_path: ./model/gmflow_sintel-0c07dcb3.pth
max_process: 4
maxinterv: 5
mininterv: 5
num_inference_steps: 20
num_warmup_steps: 5
prompt: a red car turns in the winter
run_ebsynth: False
save_path: ./output/car-turn/
sd_path: SG161222/Realistic_Vision_V2.0
seed: 0
sod_path: ./model/epoch_resnet.pth
use_controlnet: True
use_freeu: False
use_salinecy: True

====================================================================================================
creating models...
create optical flow estimation model successfully!
create saliency detection model successfully!
config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 920/920 [00:00<?, ?B/s]
diffusion_pytorch_model.safetensors: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1.45G/1.45G [01:49<00:00, 13.2MB/s]
create controlnet model-hed successfully!
config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 547/547 [00:00<?, ?B/s]
diffusion_pytorch_model.safetensors: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 335M/335M [00:26<00:00, 12.8MB/s]
model_index.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 577/577 [00:00<00:00, 576kB/s]
safety_checker\model.safetensors not found
scheduler/scheduler_config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 341/341 [00:00<?, ?B/s]
text_encoder/config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 612/612 [00:00<?, ?B/s]
(โ€ฆ)ature_extractor/preprocessor_config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 518/518 [00:00<00:00, 518kB/s]
tokenizer/special_tokens_map.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 472/472 [00:00<00:00, 472kB/s]]
safety_checker/config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 4.89k/4.89k [00:00<?, ?B/s]
unet/config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 901/901 [00:00<?, ?B/s]
tokenizer/tokenizer_config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 806/806 [00:00<?, ?B/s]
vae/config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 548/548 [00:00<?, ?B/s]
tokenizer/merges.txt: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 525k/525k [00:00<00:00, 793kB/s]
tokenizer/vocab.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1.06M/1.06M [00:00<00:00, 1.18MB/s]
text_encoder/pytorch_model.bin: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 492M/492M [00:48<00:00, 10.1MB/s]
safety_checker/pytorch_model.bin: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1.22G/1.22G [01:38<00:00, 12.4MB/s]
unet/diffusion_pytorch_model.bin: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 3.44G/3.44G [04:35<00:00, 12.5MB/s]
Fetching 14 files: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 14/14 [04:38<00:00, 19.86s/it]
Loading pipeline components...: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 7/7 [00:03<00:00,  2.05it/s]
create diffusion model SG161222/Realistic_Vision_V2.0 successfully!

====================================================================================================
key frame selection for "./data/car-turn.mp4"...
processing 3 batches:
keyframe indexes [[0, 5, 10, 15, 20, 25, 30, 35], [40, 45, 50, 55, 60], [65, 70, 75]]

====================================================================================================
video to video translation...
processing batch [1/3] with 8 frames
G:\AI\FRESCO\venv\lib\site-packages\torch\functional.py:507: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:3550.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
G:\AI\FRESCO\venv\lib\site-packages\diffusers\models\attention_processor.py:1117: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
  hidden_states = F.scaled_dot_product_attention(
100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 15/15 [06:36<00:00, 26.40s/it]
processing batch [2/3] with 5 frames
100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 15/15 [03:35<00:00, 14.35s/it]
processing batch [3/3] with 3 frames
100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 15/15 [00:26<00:00,  1.80s/it]

====================================================================================================
to translate full video with ebsynth, install ebsynth and run:

python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps


====================================================================================================
Done

imageio version?

Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 1437, in process_api
result = await self.call_function(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 1109, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\utils.py", line 650, in wrapper
response = f(*args, **kwargs)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Desktop\FRESCO\webUI.py", line 312, in process1
writer.append_data(curImg)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\core\format.py", line 590, in append_data
return self._append_data(im, total_meta)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\plugins\ffmpeg.py", line 587, in _append_data
self._initialize()
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\plugins\ffmpeg.py", line 630, in _initialize
self._write_gen = self._ffmpeg_api.write_frames(
TypeError: write_frames() got an unexpected keyword argument 'audio_path'

[Run Key Frames] sucess, but [Run Propagation (Ebsynth)] failed: FileNotFoundError: [Errno 2] No such file or directory: 'output/480x480/blend.mp4'

ๆˆชๅฑ2024-03-24 14 46 51

Running on local URL: http://127.0.0.1:7860/
Running on public URL: https://26363563e97e3055dc.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
100% 15/15 [01:10<00:00, 4.71s/it]
100% 15/15 [00:45<00:00, 3.01s/it]
python video_blend.py output/3 seconds --key keys --key_ind 0 12 18 27 32 52 57 64 74 79 90 --output output/3 seconds/blend.mp4 --fps 30 --n_proc 4 -ps
usage: video_blend.py [-h] [--output OUTPUT] [--fps FPS] [--key_ind KEY_IND [KEY_IND ...]]
[--key KEY] [--n_proc N_PROC] [-ps] [-ne] [-tmp]
name
video_blend.py: error: unrecognized arguments: seconds seconds/blend.mp4
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1440, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1341, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 281, in postprocess
processed_files = (self.format_video(y), None)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 355, in format_video
video = self.make_temp_copy_if_needed(video)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 226, in make_temp_copy_if_needed
temp_dir = self.hash_file(file_path)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 190, in hash_file
with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/3 seconds/blend.mp4'
/content/drive/MyDrive/video_gen/FRESCO/src/free_lunch_utils.py:38: UserWarning: ComplexHalf support is experimental and many operators don't support it yet. (Triggered internally at ../aten/src/ATen/EmptyTensor.cpp:31.)
x_freq = fft.fftn(x, dim=(-2, -1))
100% 15/15 [00:29<00:00, 1.96s/it]
100% 15/15 [00:19<00:00, 1.32s/it]
python video_blend.py output/480x480 --key keys --key_ind 0 12 18 27 32 52 57 64 74 79 90 --output output/480x480/blend.mp4 --fps 30 --n_proc 4 -ps
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_74/0075.jpg'): can't open/read file: check file path/integrity
Process Process-4:
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_57/0058.jpg'): can't open/read file: check file path/integrity
Process Process-3:
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_27/0028.jpg'): can't open/read file: check file path/integrity
Process Process-2:
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 8.934192419052124
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0012.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0011.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 308, in
main(args)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 268, in main
process_seq(video_sequence, i, blend_histogram, blend_gradient)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 200, in process_seq
dist1s.append(load_error(bin_a, img_shape))
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 161, in load_error
with open(bin_path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'output/480x480/out_0/0001.bin'
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1440, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1341, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 281, in postprocess
processed_files = (self._format_video(y), None)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 355, in _format_video
video = self.make_temp_copy_if_needed(video)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 226, in make_temp_copy_if_needed
temp_dir = self.hash_file(file_path)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 190, in hash_file
with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/480x480/blend.mp4'

Error in saving the video

Getting error on selecting Run All
using the given example dog video

python video_blend.py output/dog --key keys --key_ind 0 11 23 49 72 99  --output output/dog/blend.mp4 --fps 24 --n_proc 4 -ps
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_49/0050.jpg'): can't open/read file: check file path/integrity
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_23/0024.jpg'): can't open/read file: check file path/integrity
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_72/0073.jpg'): can't open/read file: check file path/integrity
Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 118.52607154846191
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0010.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
  File "/workspace/FRESCO/video_blend.py", line 308, in <module>
    main(args)
  File "/workspace/FRESCO/video_blend.py", line 268, in main
    process_seq(video_sequence, i, blend_histogram, blend_gradient)
  File "/workspace/FRESCO/video_blend.py", line 200, in process_seq
    dist1s.append(load_error(bin_a, img_shape))
  File "/workspace/FRESCO/video_blend.py", line 161, in load_error
    with open(bin_path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'output/dog/out_0/0001.bin'
Traceback (most recent call last):
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/queueing.py", line 388, in call_prediction
    output = await route_utils.call_process_api(
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/route_utils.py", line 219, in call_process_api
    output = await app.get_blocks().process_api(
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1440, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1341, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/video.py", line 281, in postprocess
    processed_files = (self._format_video(y), None)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/video.py", line 355, in _format_video
    video = self.make_temp_copy_if_needed(video)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/base.py", line 226, in make_temp_copy_if_needed
    temp_dir = self.hash_file(file_path)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/base.py", line 190, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/dog/blend.mp4'```

run on Colab with A100 GPU and 40G GPU RAM still got Error

run webUI.py on Colab with A100 GPU and 40G GPU RAM still got Error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.25 GiB (GPU 0; 39.56 GiB total capacity; 21.83 GiB already allocated; 10.45 GiB free; 22.58 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

ๆˆชๅฑ2024-03-24 14 23 53

update_sd_model 'pipe' is not defined

when switching models other than SG161222/Realistic_Vision_V2.0

File "E:\FRESCO\webUI.py", line 149, in update_sd_model
self.pipe.scheduler = DDPMScheduler.from_config(pipe.scheduler.config)
NameError: name 'pipe' is not defined

Diffusers

I noticed that the new update has been integrated into Diffusers. Please tell me how to use it.

Meaning of unet_chunk_size in FRESCOAttnProcessor2_0 codes๏ผŸ

The 'video_len' in the source code is equal to 'batch_size // unet_chunk_size'. But if 8 frames are fed in at once, isn't 'batch_size' equal to 'video_len'?

        '''for efficient cross-frame attention'''
        if self.controller and self.controller.use_cfattn and (not crossattn):
            video_length = key.size()[0] // self.unet_chunk_size  # BC // C = B
            former_frame_index = [0] * video_length  # pervious frame index
            attn_mask = None  # get attn_mask from controller
            if self.controller.attn_mask is not None:
                for m in self.controller.attn_mask:
                    if m.shape[1] == key.shape[1]:
                        attn_mask = m
            # get key and value in former_frame_index range
            # BC * HW * 8D --> B * C * HW * 8D
            key = rearrange(key, "(b f) d c -> b f d c", f=video_length)
            # B * C * HW * 8D --> B * C * HW * 8D
            if attn_mask is None:
                key = key[:, former_frame_index]
            else:
                key = repeat(key[:, attn_mask], "b d c -> b f d c", f=video_length)
            # B * C * HW * 8D --> BC * HW * 8D 
            key = rearrange(key, "b f d c -> (b f) d c").detach()
            value = rearrange(value, "(b f) d c -> b f d c", f=video_length)
            if attn_mask is None:
                value = value[:, former_frame_index]
            else:
                value = repeat(value[:, attn_mask], "b d c -> b f d c", f=video_length)              
            value = rearrange(value, "b f d c -> (b f) d c").detach()
        
        # BC * HW * 8D --> BC * HW * 8 * D --> BC * 8 * HW * D
        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
        # BC * 8 * HW2 * D
        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
        # BC * 8 * HW2 * D2
        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)

blender not found

how to install blender package in video_blend.py file? I use "pip install blender", when import blender, has "ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory"

Out of memory: RTX 4090 / 24GB

Hi no matter what movie size I choose -- 5fps, 640x480 I get this error below. nvtop shows that the webUI triggers pre-allocation of 21.5GB but then ... it's not used?

/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 0%| | 0/15 [00:01<?, ?it/s] Traceback (most recent call last): File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/queueing.py", line 388, in call_prediction output = await route_utils.call_process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/route_utils.py", line 219, in call_process_api output = await app.get_blocks().process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/blocks.py", line 1437, in process_api result = await self.call_function( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/blocks.py", line 1109, in call_function prediction = await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/utils.py", line 650, in wrapper response = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/webUI.py", line 159, in process keypath = process1(*args) ^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/webUI.py", line 280, in process1 latents = inference(global_state.pipe, global_state.controlnet, global_state.frescoProc, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/pipe_FRESCO.py", line 201, in inference noise_pred = pipe.unet( ^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/diffusion_hacked.py", line 776, in forward sample = optimize_feature(sample, flows, occs, correlation_matrix, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/diffusion_hacked.py", line 485, in optimize_feature optimizer.step(closure) File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/optim/optimizer.py", line 373, in wrapper out = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/optim/optimizer.py", line 76, in _use_grad ret = func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/optim/adam.py", line 143, in step loss = closure() ^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/diffusion_hacked.py", line 478, in closure loss.backward() File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/autograd/__init__.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.06 GiB. GPU 0 has a total capacty of 23.65 GiB of which 525.94 MiB is free. Process 15481 has 1.27 GiB memory in use. Including non-PyTorch memory, this process has 21.00 GiB memory in use. Of the allocated memory 20.11 GiB is allocated by PyTorch, and 416.71 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

ImportError: cannot import name 'logger' from 'diffusers.models.unet_2d_condition' (/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py)

2024-03-23 09:12:10.007036: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-23 09:12:10.007276: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-23 09:12:10.144748: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-23 09:12:13.261790: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py:20: FutureWarning: UNet2DConditionOutput is deprecated and will be removed in version 0.29. Importing UNet2DConditionOutput from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionOutput, instead.
deprecate("UNet2DConditionOutput", "0.29", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py:25: FutureWarning: UNet2DConditionModel is deprecated and will be removed in version 0.29. Importing UNet2DConditionModel from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel, instead.
deprecate("UNet2DConditionModel", "0.29", deprecation_message)
Traceback (most recent call last):
File "/content/FRESCO/webUI.py", line 25, in
from src.free_lunch_utils import apply_freeu
File "/content/FRESCO/src/free_lunch_utils.py", line 6, in
from diffusers.models.unet_2d_condition import logger as logger2d
ImportError: cannot import name 'logger' from 'diffusers.models.unet_2d_condition' (/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py)

run webUI.py with a 3 seconds video of 1280x720, got Error

Running on local URL: http://127.0.0.1:7860/
Running on public URL: https://26363563e97e3055dc.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
100% 15/15 [01:10<00:00, 4.71s/it]
100% 15/15 [00:45<00:00, 3.01s/it]
python video_blend.py output/3 seconds --key keys --key_ind 0 12 18 27 32 52 57 64 74 79 90 --output output/3 seconds/blend.mp4 --fps 30 --n_proc 4 -ps
usage: video_blend.py [-h] [--output OUTPUT] [--fps FPS] [--key_ind KEY_IND [KEY_IND ...]]
[--key KEY] [--n_proc N_PROC] [-ps] [-ne] [-tmp]
name
video_blend.py: error: unrecognized arguments: seconds seconds/blend.mp4
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1440, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1341, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 281, in postprocess
processed_files = (self._format_video(y), None)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 355, in _format_video
video = self.make_temp_copy_if_needed(video)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 226, in make_temp_copy_if_needed
temp_dir = self.hash_file(file_path)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 190, in hash_file
with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/3 seconds/blend.mp4'

Is there any way to speed it up?

Great work. Thank you very much.
And Is there any way to speed it up?
3090 GPU, deal with about 100frames,need 40 min+ ,got a 3 sec video.
I'm not sure if it's my setup or something?

add --listen

Can you add the --listen startup parameter to enable access within the LAN?

can not find target file while executing process_seq(video_sequence, i, blend_histogram, blend_gradient) in video_blend.py

After I configured parameter "run_ebsynth" as True to generate video occurred the following error

I am wondering is the files were not generated or is just given the wrong target file path ๏ผŸ

And why the target file is not there (and there are no *.bin files generated in output directory):
./output/boxer-punching-towards-camera/out_0/0001.jpg
./output/boxer-punching-towards-camera/out_2/0001.jpg
image

is there something wrong in code or the related file is not generated correctly?

  • this is the parameter config in yaml file

image

  • this is the error log

image

notebook doesn't run

Notebooks are a great tool to help people walk through a demo of something step by step. The notebook provided has some prerequisites listed at the top that have to be manually figured out. It'd be great if the notebook was standalone and had cells for all the steps required to run.

Which directory should the SD1.5 model be placed in?

Hi:

After run the script

python run_fresco.py ./config/config_music.yaml

But I didn't found any SD1.5 model in the project.

  1. Is it necessary to manually download the SD1.5 model?
  2. Which directory should the SD1.5 model be placed in?

BTW : the default path on windows 10 is C:\Users\xxxx\.cache\huggingface\hub

python video_blend.py Cannot run

I ran the command below
"python video_blend.py ./output/music/ --key keys --key_ind 0 26 36 54 72 87 107 122 132 159 173 183 194 206 217 229 239 --output ./output/music/blend.mp4 --fps 24 --n_proc 4 -ps"
which is get by command "python run_fresco.py ./config/config_music.yaml"
But I get the error output below:

/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
self.stylized_imgs[i - 1]: ./output/music/out_194/0194.jpg
self.stylized_imgs[i - 1]: ./output/music/out_194/0195.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_194/0195.jpg'): can't open/read file: check file path/integrity
Process Process-4:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
self.stylized_imgs[i - 1]: ./output/music/out_72/0072.jpg
self.stylized_imgs[i - 1]: ./output/music/out_72/0073.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_72/0073.jpg'): can't open/read file: check file path/integrity
Process Process-2:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
self.stylized_imgs[i - 1]: ./output/music/out_0/0000.jpg
self.stylized_imgs[i - 1]: ./output/music/out_0/0001.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
self.stylized_imgs[i - 1]: ./output/music/out_132/0132.jpg
self.stylized_imgs[i - 1]: ./output/music/out_132/0133.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_132/0133.jpg'): can't open/read file: check file path/integrity
Process Process-3:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 9.930143356323242
run_ebsynth done!
video_sequence.get_key_img(i): ./output/music/keys/0000.png
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0012.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0013.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0014.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0015.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0016.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0017.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0018.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0019.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0020.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0021.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0022.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0023.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0024.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0025.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0026.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0012.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0013.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0014.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0015.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0016.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0017.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0018.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0019.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0020.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0021.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0022.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0023.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0024.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0025.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 319, in
main(args)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 279, in main
process_seq(video_sequence, i, blend_histogram, blend_gradient)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 209, in process_seq
dist1s.append(load_error(bin_a, img_shape))
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 168, in load_error
with open(bin_path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: './output/music/out_0/0001.bin'

I am extremely confused.....................
Since I get the output images by official command, i didn't get corresponding images as supposed.

pull request requirements.txt

sorry still havent figured out pull requests
in requirements.txt

pip install torchvision=0.16.0
should be
pip install torchvision==0.16.0

im on windows annaconda so maybe its different on other systems

ebsynth generation fails

Both the web ui and directly run_fresco.py are failing at the same point. Here's my output. It suggests the run_fresco.pyfile is not producing the expected files.

(FRESCO) gruntus FRESCO % python ./run_fresco.py ./config/config_carturn.yaml
====================================================================================================
loading configuration...
batch_size: 8
cond_scale: 0.7
controlnet_type: hed
end_opt_step: 15
file_path: ./data/car-turn.mp4
gmflow_path: ./model/gmflow_sintel-0c07dcb3.pth
max_process: 4
maxinterv: 5
mininterv: 5
num_inference_steps: 20
num_warmup_steps: 5
prompt: a red car turns in the winter
run_ebsynth: False
save_path: ./output/car-turn/
sd_path: SG161222/Realistic_Vision_V2.0
seed: 0
sod_path: ./model/epoch_resnet.pth
use_controlnet: True
use_freeu: False
use_salinecy: True

====================================================================================================
creating models...
create optical flow estimation model successfully!
create saliency detection model successfully!
create controlnet model-hed successfully!
unet/diffusion_pytorch_model.safetensors not found
Loading pipeline components...:  57%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–Œ      | 4/7 [00:01<00:01,  2.57it/s]/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading pipeline components...: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 7/7 [00:01<00:00,  3.77it/s]
create diffusion model SG161222/Realistic_Vision_V2.0 successfully!

====================================================================================================
key frame selection for "./data/car-turn.mp4"...
processing 3 batches:
keyframe indexes [[0, 5, 10, 15, 20, 25, 30, 35], [40, 45, 50, 55, 60], [65, 70, 75]]

====================================================================================================
video to video translation...
processing batch [1/3] with 8 frames
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 15/15 [00:26<00:00,  1.73s/it]
processing batch [2/3] with 5 frames
100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 15/15 [00:22<00:00,  1.51s/it]
processing batch [3/3] with 3 frames
100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 15/15 [00:16<00:00,  1.11s/it]

====================================================================================================
to translate full video with ebsynth, install ebsynth and run:

python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps


====================================================================================================
Done
(FRESCO) gruntus FRESCO % python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75  --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([0, 1, 2, 3], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([4, 5, 6, 7], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([8, 9, 10, 11],
           <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([12, 13, 14], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_20/0021.jpg'): can't open/read file: check file path/integrity
Process Process-2:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_40/0041.jpg'): can't open/read file: check file path/integrity
Process Process-3:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_60/0061.jpg'): can't open/read file: check file path/integrity
Process Process-4:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 12.073577880859375
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0004.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 311, in <module>
    main(args)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 271, in main
    process_seq(video_sequence, i, blend_histogram, blend_gradient)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 203, in process_seq
    dist1s.append(load_error(bin_a, img_shape))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 164, in load_error
    with open(bin_path, 'rb') as fp:
         ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './output/car-turn/out_0/0001.bin'
``

่ง†้ข‘ๅคงไบŽ1M ๏ผŒ24GPUไผš็ˆ†ๅ—๏ผŸ่ฟ˜ๆ˜ฏๅชๆœ‰ๆˆ‘ๆ˜ฏ่ฟ™ๆ ท็š„๏ผŸ

Traceback (most recent call last):
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\blocks.py", line 1437, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\blocks.py", line 1109, in call_function
prediction = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\utils.py", line 650, in wrapper
response = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\webUI.py", line 159, in process
keypath = process1(*args)
^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\webUI.py", line 280, in process1
latents = inference(global_state.pipe, global_state.controlnet, global_state.frescoProc,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\pipe_FRESCO.py", line 201, in inference
noise_pred = pipe.unet(
^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\diffusion_hacked.py", line 787, in forward
sample = upsample_block(
^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\free_lunch_utils.py", line 346, in forward
hidden_states = attn(
^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\diffusers\models\transformer_2d.py", line 292, in forward
hidden_states = block(
^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\diffusers\models\attention.py", line 155, in forward
attn_output = self.attn1(
^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 322, in forward
return self.processor(
^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\diffusion_hacked.py", line 281, in call
query = F.scaled_dot_product_attention(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.66 GiB (GPU 0; 23.99 GiB total capacity; 15.29 GiB already allocated; 2.85 GiB free; 17.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Error when running python webUI.py

(diffusers) PS D:\FRESCO> python webUI.py
safety_checker\model.safetensors not found
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]D:\miniconda3\envs\diffusers\lib\site-packages\torch_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Loading pipeline components...: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 7/7 [00:04<00:00, 1.55it/s]
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
safety_checker\model.safetensors not found
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]D:\miniconda3\envs\diffusers\lib\site-packages\torch_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Loading pipeline components...: 29%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–Š | 2/7 [00:03<00:08, 1.67s/it]D:\miniconda3\envs\diffusers\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
warnings.warn(
Loading pipeline components...: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 7/7 [00:04<00:00, 1.54it/s]
Traceback (most recent call last):
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\blocks.py", line 1437, in process_api
result = await self.call_function(
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\blocks.py", line 1109, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\miniconda3\envs\diffusers\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "D:\miniconda3\envs\diffusers\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "D:\miniconda3\envs\diffusers\lib\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\utils.py", line 650, in wrapper
response = f(*args, **kwargs)
File "D:\miniconda3\envs\diffusers\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "webUI.py", line 159, in process
keypath = process1(*args)
File "D:\miniconda3\envs\diffusers\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "webUI.py", line 172, in process1
global_state.update_sd_model(sd_model)
File "webUI.py", line 149, in update_sd_model
self.pipe.scheduler = DDPMScheduler.from_config(pipe.scheduler.config)
NameError: name 'pipe' is not defined

Got RuntimeError:: element 0 of tensors does not require grad and does not have a grad_fn

video to video translation...
processing batch [1/7] with 8 frames
FutureWarning: _encode_prompt() is deprecated and it will be removed in a future version. Use encode_prompt() instead. Also, be aware that the output format changed from a concatenated tensor to a tuple.

0%| | 0/15 [00:00<?, ?it/s]
0%| | 0/15 [00:00<?, ?it/s]
!!! Exception during processing !!!
Traceback (most recent call last):
File "/data/comfy-ui/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/data/comfy-ui/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/data/comfy-ui/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/nodes.py", line 68, in stylize
FrescoService().stylize(video_file_path,output_dir,prompt)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco_service.py", line 39, in stylize
keys = run_keyframe_translation(config)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/run_fresco.py", line 239, in run_keyframe_translation
latents = inference(pipe, controlnet, frescoProc,
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/pipe_FRESCO.py", line 202, in inference
noise_pred = pipe.unet(
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 778, in forward
sample = optimize_feature(sample, flows, occs, correlation_matrix,
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 487, in optimize_feature
optimizer.step(closure)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper
out = func(*args, **kwargs)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 76, in _use_grad
ret = func(self, *args, **kwargs)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/adam.py", line 143, in step
loss = closure()
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 480, in closure
loss.backward()
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.