yrcong / flatten

Pytorch Implementation of FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024)

Here is Yuren Cong!

flatten's Issues

Batching/Sliding Window

Hi all,

Thank you for publishing this code and researching this amazing tech.

Is there a recommended way to perform inference on batches of a video? e.g. process a video in chunks?

In the code there's references to inter_frame and get_slide_window_indices, but they're never used.

Again, thank you for publishing this code. I've implemented it into ComfyUI a diffusion pipeline used by artists and creators. I'd like to share it with others, but without batching large GPUs can only handle ~40 frames.
https://github.com/logtd/ComfyUI-FLATTEN

Also, please let me know if I'm missing any citations, contributions, etc. Happy to add them as your research is what made that possible!

Hi, thank you for your work. I have no idea why this error occurs

Traceback (most recent call last):
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 270, in hf_raise_for_status
response.raise_for_status()
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/checkpoints/stable-diffusion-2-1-base/resolve/main/tokenizer/tokenizer_config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/transformers/utils/hub.py", line 430, in cached_file
resolved_file = hf_hub_download(
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1374, in hf_hub_download
raise head_call_error
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1247, in hf_hub_download
metadata = get_hf_file_metadata(
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1624, in get_hf_file_metadata
r = _request_wrapper(
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 402, in _request_wrapper
response = _request_wrapper(
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 426, in _request_wrapper
hf_raise_for_status(response)
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 320, in hf_raise_for_status
raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-65ef52c7-40b3d3317d5897a76a3b514d;7b602761-da5b-451b-9a6f-8b425e3bbf99)

Repository Not Found for url: https://huggingface.co/checkpoints/stable-diffusion-2-1-base/resolve/main/tokenizer/tokenizer_config.json.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "inference.py", line 42, in
tokenizer = CLIPTokenizer.from_pretrained(args.sd_path, subfolder="tokenizer")
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1947, in from_pretrained
resolved_config_file = cached_file(
File "/opt/conda/envs/flatten/lib/python3.8/site-packages/transformers/utils/hub.py", line 451, in cached_file
raise EnvironmentError(
OSError: checkpoints/stable-diffusion-2-1-base is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with huggingface-cli login or by passing token=<your_token>

can you check it? thanks

code release