camenduru / text-to-video-synthesis-colab

Text To Video Synthesis Colab

🐣 Please follow me for new updates https://twitter.com/camenduru
🔥 Please join our discord server https://discord.gg/k5BwmmvJJU
🥳 Please join my patreon community https://patreon.com/camenduru

🦒 Colab

Colab	Type
	longscope_text_to_video_colab polyware-ai/longscope (Thanks to @polyware_ai ❤ for the model)
	zeroscope_v2_xl_text_to_video_colab cerspense/zeroscope_v2_XL (Thanks to @cerspense ❤ for the model)
	zeroscope_v2_576w_text_to_video_colab cerspense/zeroscope_v2_576w (Thanks to @cerspense ❤ for the model)
	potat1_exp1_text_to_video_colab (Thanks to lambda labs ❤ for the gpu)
	zeroscope_v2_dark_text_to_video_colab cerspense/zeroscope_v2_dark_30x448x256 (Thanks to @cerspense ❤ for the model)
	zeroscope_v2_30x448x256_text_to_video_colab cerspense/zeroscope_v2_30x448x256 (Thanks to @cerspense ❤ for the model)
	zeroscope_v1_1_320s_text_to_video_colab cerspense/zeroscope_v1-1_320s (Thanks to @cerspense ❤ for the model)
	potat1_text_to_video_colab camenduru/potat1 (Thanks to lambda labs ❤ for the gpu)
	zeroscope_v1_320s_text_to_video_colab cerspense/zeroscope_v1_320s (Thanks to @cerspense ❤ for the model)
	ms_1_7b_text_to_video_colab damo-vilab/text-to-video-ms-1.7b (Thanks to damo ❤ damo-vilab ❤ for the model)
	animov_512x_text_to_video_colab strangeman3107/animov-512x (Thanks to @bRuefiRe ❤ for the model)
	animov_0_1_1_text_to_video_colab strangeman3107/animov-0.1.1 (Thanks to @bRuefiRe ❤ for the model)
	animov_0_1_text_to_video_colab strangeman3107/animov-0.1.1 (Thanks to @bRuefiRe ❤ for the model)
	Watermark Remover
	text_to_video_synthesis (Thanks to damo ❤ damo-vilab ❤ for the model)
	text_to_video_synthesis_diffusers (Thanks to damo ❤ damo-vilab ❤ for the model)

🦒 Community Colab

Colab	Function	Author	Readme
	potat1_text_to_video_colab_ALM (Thanks to snydjos ❤)	snydjos AILostMedia	Readme

Tutorials

https://www.youtube.com/watch?v=WHcjD-uraUs (for zeroscope_video_webui_colab and zeroscope_upscale_video_webui_colab)
https://www.youtube.com/watch?v=E45AO1w9gRU (for Potat 1️⃣ and Zeroscope)
https://www.youtube.com/watch?v=b8D4am73e6I

Main Repo

https://www.modelscope.cn/models/damo/text-to-video-synthesis/summary
https://github.com/modelscope/modelscope
https://github.com/huggingface/diffusers

Paper

https://arxiv.org/abs/2308.06571

Models License

Apache License 2.0

Examples

Potat 1️⃣

out.mp4

out2.mp4

out3.mp4

out4.mp4

out27.mp4

waterfall.eb62844e.mp4

Base Model

A giraffe underneath a microwave.	A goldendoodle playing in a park by a lake.	A panda bear driving a car.
A teddy bear running in New York City.	Drone flythrough of a fast food restaurant on a dystopian alien planet.	A dog wearing a Superhero outfit with red cape flying through the sky.
Monkey learning to play the piano.	A litter of puppies running through the yard.	Robot dancing in times square.

Related Colab

https://github.com/camenduru/text2video-zero-colab

text-to-video-synthesis-colab's Issues

Could not find TensorRT

Good morning, I get this error when I launch the second cell.

14:36:39.040164: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

How can I solve it?
Thank you in advance

README licence is outdated

zeroscope doesnt work

when i want to generate something i always get

/content/Text-To-Video-Finetuning
seed is 834037899790305933
Loading pipeline components...: 0% 0/5 [00:00<?, ?it/s]2024-01-26 10:00:10.608979: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-26 10:00:10.609030: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-26 10:00:10.610281: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-26 10:00:11.660957: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Loading pipeline components...: 100% 5/5 [00:02<00:00, 2.24it/s]
100% 50/50 [05:50<00:00, 7.02s/it]
Traceback (most recent call last):
File "/content/Text-To-Video-Finetuning/inference.py", line 232, in
export_to_video(video, f"{out_stem} {str(uuid4())[:8]}.mp4", fps)
File "/content/Text-To-Video-Finetuning/train.py", line 118, in export_to_video
img = cv2.cvtColor(video_frames[i], cv2.COLOR_RGB2GRAY)
cv2.error: OpenCV(4.8.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3, 4>; VDcn = cv::impl::{anonymous}::Set<1>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = cv::impl::::NONE; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'

Invalid number of channels in input image:
'VScn::contains(scn)'
where
'scn' is 40

is there a way to fix that

Runtime environment crushes after RAM usage goes over 12GB

I tired to generate anything using this colab but each time it crushes after system RAM goes over 12GB. Is there any way to mitigate this problem via controlling number of frames generated or the size of the frame?

Watermark remover image not found

in0.jpg
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 401, in run_predict
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1302, in process_api
result = await self.call_function(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1025, in call_function
prediction = await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "", line 80, in infer
lama_frame = Image.open(lama_frame)
File "/usr/local/lib/python3.10/dist-packages/PIL/Image.py", line 2975, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/content/lama/outdir/image_mask.png'

[Google Colab] Format of output video cannot play in some contexts

The current export_to_video function from HuggingFace encodes the output video with MPEG-4 Visual format, which doesn't play in some contexts such as Desktop Discord. (They can play in normal video players and mobile Discord.) Looking at the HuggingFace Space version, they are using imageio to export, which results in a video file with AVC format.

Could the colab be adjusted so the output format is functional in more contexts?

Watermark remover - uses GPU?

Colab warns me that I'm connected to a GPU but not using it. Is this expected behaviour? (Asking 'cos it seems pretty slow)
Cheers!

Notebook produce strange results

Something is broken, when trying V2V STEP 3: V2V with Potat1 (Will Need Interpolating). The function does not convert the supplied video into a prompt-compliant image,

can't get the path to the model right

hi i'm trying to set this up locally and followed the script from google colab.

however, every time i try to run it, i'm told it can't find the model. i'm pointing to it just like in colab and the file structure is the same. what am i doing wrong?

python inference.py -m "/content/zeroscope_v1-1_320s" -p "ducks in a lake" -W 320 -H 320 -o /content/outputs -d cuda -x -s 33 -g 23 -f 30 -T 24
Traceback (most recent call last):
  File "C:\Users\mberm\miniconda3\envs\vid\lib\site-packages\diffusers\configuration_utils.py", line 358, in load_config
    config_file = hf_hub_download(
  File "C:\Users\mberm\miniconda3\envs\vid\lib\site-packages\huggingface_hub\utils\_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "C:\Users\mberm\miniconda3\envs\vid\lib\site-packages\huggingface_hub\utils\_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/content/zeroscope_v1-1_320s'. Use `repo_type` argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\mberm\OneDrive\Desktop\content\Text-To-Video-Finetuning\inference.py", line 194, in <module>
    videos = inference(**args)
  File "C:\Users\mberm\miniconda3\envs\vid\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\mberm\OneDrive\Desktop\content\Text-To-Video-Finetuning\inference.py", line 122, in inference
    pipeline = initialize_pipeline(model, device, xformers, sdp)
  File "C:\Users\mberm\OneDrive\Desktop\content\Text-To-Video-Finetuning\inference.py", line 21, in initialize_pipeline
    scheduler, tokenizer, text_encoder, vae, _unet = load_primary_models(model)
  File "C:\Users\mberm\OneDrive\Desktop\content\Text-To-Video-Finetuning\train.py", line 132, in load_primary_models
    noise_scheduler = DDPMScheduler.from_pretrained(pretrained_model_path, subfolder="scheduler")
  File "C:\Users\mberm\miniconda3\envs\vid\lib\site-packages\diffusers\schedulers\scheduling_utils.py", line 140, in from_pretrained
    config, kwargs, commit_hash = cls.load_config(
  File "C:\Users\mberm\miniconda3\envs\vid\lib\site-packages\diffusers\configuration_utils.py", line 394, in load_config
    raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co' to load this model, couldn't find it in the cached files and it looks like /content/zeroscope_v1-1_320s is not the path to a directory containing a scheduler_config.json file.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/diffusers/installation#offline-mode'.

here's the absolute path to the content folder:

C:\Users\mberm\OneDrive\Desktop\content

Notebook not working.

I keep getting this error in the notebook and I can't generate anything from it.

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or
flax_model.msgpack found in directory /content/potat1.

Can somebody make a tutorial or write the steps for installing potat1 locally?

text_to_video_synthesis error log

I ran text_to_video_synthesis again just now received an app crash. markdown error log stated:

Timestamp	Level	Message
Apr 5, 2023, 5:29:23 PM	WARNING	WARNING:root:kernel fe7a616c-0131-49f8-8976-380c2d7012db restarted
Apr 5, 2023, 5:29:23 PM	INFO	KernelRestarter: restarting kernel (1/5), keep random ports
Apr 5, 2023, 5:24:35 PM	INFO	Kernel started: fe7a616c-0131-49f8-8976-380c2d7012db, name: python3
Apr 5, 2023, 5:23:51 PM	INFO	Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Apr 5, 2023, 5:23:51 PM	INFO	http://172\.28\.0\.12:9000/
Apr 5, 2023, 5:23:51 PM	INFO	Jupyter Notebook 6.4.8 is running at:
Apr 5, 2023, 5:23:51 PM	INFO	Serving notebooks from local directory: /
Apr 5, 2023, 5:23:51 PM	INFO	Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Apr 5, 2023, 5:23:51 PM	INFO	http://172\.28\.0\.2:9000/
Apr 5, 2023, 5:23:51 PM	INFO	Jupyter Notebook 6.3.0 is running at:
Apr 5, 2023, 5:23:51 PM	INFO	Serving notebooks from local directory: /
Apr 5, 2023, 5:23:51 PM	WARNING	ModuleNotFoundError: No module named 'jupyter_server'
Apr 5, 2023, 5:23:51 PM	WARNING	from jupyter_server.base.handlers import JupyterHandler
Apr 5, 2023, 5:23:51 PM	WARNING	File "/usr/local/lib/python3.9/dist-packages/panel/io/jupyter_server_extension.py", line 53, in <module>
Apr 5, 2023, 5:23:51 PM	WARNING	File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
Apr 5, 2023, 5:23:51 PM	WARNING	File "<frozen importlib._bootstrap_external>", line 850, in exec_module
Apr 5, 2023, 5:23:51 PM	WARNING	File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
Apr 5, 2023, 5:23:51 PM	WARNING	File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
Apr 5, 2023, 5:23:51 PM	WARNING	File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
Apr 5, 2023, 5:23:51 PM	WARNING	File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
Apr 5, 2023, 5:23:51 PM	WARNING	return _bootstrap._gcd_import(name[level:], package, level)
Apr 5, 2023, 5:23:51 PM	WARNING	File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
Apr 5, 2023, 5:23:51 PM	WARNING	mod = importlib.import_module(modulename)
Apr 5, 2023, 5:23:51 PM	WARNING	File "/usr/local/lib/python3.9/dist-packages/notebook/notebookapp.py", line 2033, in init_server_extensions
Apr 5, 2023, 5:23:51 PM	WARNING	Traceback (most recent call last):
Apr 5, 2023, 5:23:51 PM	WARNING	Error loading server extension panel.io.jupyter_server_extension
Apr 5, 2023, 5:23:46 PM	INFO	google.colab serverextension initialized.
Apr 5, 2023, 5:23:46 PM	INFO	Authentication of /metrics is OFF, since other authentication is disabled.
Apr 5, 2023, 5:23:46 PM	INFO	Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
Apr 5, 2023, 5:23:46 PM	WARNING	/root/.jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/root/.local/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/usr/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json
Apr 5, 2023, 5:23:46 PM	WARNING	/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	INFO	google.colab serverextension initialized.
Apr 5, 2023, 5:23:46 PM	INFO	Authentication of /metrics is OFF, since other authentication is disabled.
Apr 5, 2023, 5:23:46 PM	INFO	Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
Apr 5, 2023, 5:23:46 PM	WARNING	/root/.jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/root/.local/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/usr/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json
Apr 5, 2023, 5:23:46 PM	WARNING	/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:46 PM	WARNING	[D 22:23:46.007 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py
Apr 5, 2023, 5:23:46 PM	WARNING	[D 22:23:46.003 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter
Apr 5, 2023, 5:23:46 PM	WARNING	[D 22:23:46.003 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter
Apr 5, 2023, 5:23:46 PM	WARNING	[D 22:23:46.002 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter
Apr 5, 2023, 5:23:46 PM	WARNING	[D 22:23:46.002 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.997 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.996 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.994 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.993 NotebookApp] Looking for jupyter_config in /root/.jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.990 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.990 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.972 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.970 NotebookApp] Looking for jupyter_config in /etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.962 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.888 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.884 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.884 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.884 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.884 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.882 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.881 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.880 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.877 NotebookApp] Looking for jupyter_config in /root/.jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.877 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.877 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.860 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.858 NotebookApp] Looking for jupyter_config in /etc/jupyter
Apr 5, 2023, 5:23:45 PM	WARNING	[D 22:23:45.858 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files

Launching Web UI with arguments: --listen --xformers --enable-insecure-extension-access --theme dark --gradio-queue --multiple --disable-safe-unpickle

Launching Web UI with arguments: --listen --xformers --enable-insecure-extension-access --theme dark --gradio-queue --multiple --disable-safe-unpickle
src/tcmalloc.cc:283] Attempt to free invalid pointer 0x5bdd20789740

potat1_text_to_video_colab.ipynb won’t generate video

I am running this notebook in Google Colab. It took three goes for the dependencies to install properly. When I try to actually run the model, I get this:

/content/Text-To-Video-Finetuning
2024-04-01 18:39:22.405606: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-04-01 18:39:22.405655: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-04-01 18:39:22.407052: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-04-01 18:39:23.575649: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
  File "/content/Text-To-Video-Finetuning/inference.py", line 13, in <module>
    from train import export_to_video, handle_memory_attention, load_primary_models
  File "/content/Text-To-Video-Finetuning/train.py", line 18, in <module>
    import torchvision.transforms as T
  File "/usr/local/lib/python3.10/dist-packages/torchvision/__init__.py", line 6, in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
  File "/usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py", line 164, in <module>
    def meta_nms(dets, scores, iou_threshold):
  File "/usr/local/lib/python3.10/dist-packages/torch/_custom_ops.py", line 253, in inner
    custom_op = _find_custom_op(qualname, also_check_torch_library=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py", line 1076, in _find_custom_op
    overload = get_op(qualname)
  File "/usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py", line 1062, in get_op
    error_not_found()
  File "/usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py", line 1052, in error_not_found
    raise ValueError(
ValueError: Could not find the operator torchvision::nms. Please make sure you have already registered the operator and (if registered from C++) loaded it via torch.ops.load_library.

This is clearly not supposed to happen. Does anyone know what’s going on?

Where is the generated video supposed to be saved?

Hi!

Where is the generated video supposed to be saved?

After running the last cell a window of a 0.00 second video appears with the text: /tmp/tmpgv746osq.mp4 but it cannot be played or downloaded.

Thanks.

Text to Video

Watermark remover

Won't run. Crashes like this:

`2023-06-01 15:36:38 (205 MB/s) - ‘/content/lama/indir/image_mask.png’ saved [8636/8636]

/content/lama
WARNING:py.warnings:/usr/local/lib/python3.10/dist-packages/gradio/deprecation.py:40: UserWarning: The 'type' parameter has been deprecated. Use the Number component instead.
warnings.warn(value)

---

KeyError Traceback (most recent call last)
in <cell line: 88>()
86 return final_vid
87
---> 88 with gr.Blocks() as demo:
89 with gr.Column(elem_id="col-container"):
90 with gr.Row():

2 frames
/usr/local/lib/python3.10/dist-packages/gradio_client/serializing.py in input_api_info(self)
39 def input_api_info(self) -> tuple[str, str]:
40 api_info = self.api_info()
---> 41 return (api_info["serialized_input"][0], api_info["serialized_input"][1])
42
43 # For backwards compatibility

KeyError: 'serialized_input'`

How to make video that is longer than 30 frames?

Every time I set max frames more than 30 I get distorted video. This is also true for txt2vid extension in webui when running stable_diffusion_1_5_video_webui_colab.
Is this the limit on free google colab?

text_to_video_synthesis_diffuser rendering error.

Sometimes, my video output comes out looking like, the output in the image I just uploaded. I was trying to create video based on a prompt about an anime character, can this be synthesized? Or does the AI not recognize this information.

when zeroscope 2 colab?

when

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. pandas-gbq 0.17.9 requires pyarrow<10.0dev,>=3.0.0, but you have pyarrow 11.0.0 which is incompatible.

Attempted to run text_to_video_synthesis.ipynb

I have been getting this error: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pandas-gbq 0.17.9 requires pyarrow<10.0dev,>=3.0.0, but you have pyarrow 11.0.0 which is incompatible.

How would I downgrade the pyarrow from 11.0.0 to pyarrow<10.0dev,>=3.0.0 can someone help me with this error.

Your session crashed for an unknown reason.

getting this error after recent update to the .ipynb file

Your session crashed for an unknown reason.

Sufficient Colab tier/GPU for the "original" notebook?

I'm using the "original" notebook. At first it ran just fine and dandy for a few days, but since 2 days ago or so it's giving me almost always "CUDA OUT OF MEMORY". Other times it doesn't, which makes it even weirder, and I can't find anything that would have changed within the past 2 days.

I'm on Colab Pro tier (i.e. T4), didn't at any point have Premium GPU selected. Is Premium GPU required/recommended, or have you any idea what might be causing this behaviour or how to prevent it? I'm out of ideas myself...

Thanks for the great work btw!

Assets

![7](https://user-images.githubusercontent.com/54370274/226195706-281c959a-7

b6e-4091-a9e2-1362a1d3

4e48.gif)

Still doesnt work

eed is 2629834275745367308
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling transformers.utils.move_cache().
0it [00:00, ?it/s]
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformer_temporal.py:24: FutureWarning: TransformerTemporalModelOutput is deprecated and will be removed in version 0.29. Importing TransformerTemporalModelOutput from diffusers.models.transformer_temporal is deprecated and this will be removed in a future version. Please use from diffusers.models.transformers.tranformer_temporal import TransformerTemporalModelOutput, instead.
deprecate("TransformerTemporalModelOutput", "0.29", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformer_temporal.py:29: FutureWarning: TransformerTemporalModel is deprecated and will be removed in version 0.29. Importing TransformerTemporalModel from diffusers.models.transformer_temporal is deprecated and this will be removed in a future version. Please use from diffusers.models.transformers.tranformer_temporal import TransformerTemporalModel, instead.
deprecate("TransformerTemporalModel", "0.29", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformer_temporal.py:34: FutureWarning: TransformerTemporalModelOutput is deprecated and will be removed in version 0.29. Importing TransformerSpatioTemporalModel from diffusers.models.transformer_temporal is deprecated and this will be removed in a future version. Please use from diffusers.models.transformers.tranformer_temporal import TransformerSpatioTemporalModel, instead.
deprecate("TransformerTemporalModelOutput", "0.29", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformer_2d.py:20: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 0.29. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.transformers.transformer_2d import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "0.29", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformer_2d.py:25: FutureWarning: Transformer2DModel is deprecated and will be removed in version 0.29. Importing Transformer2DModel from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.transformers.transformer_2d import Transformer2DModel, instead.
deprecate("Transformer2DModel", "0.29", deprecation_message)
Loading pipeline components...: 0% 0/5 [00:00<?, ?it/s]2024-01-30 20:29:01.373400: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-30 20:29:01.405383: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-30 20:29:01.553697: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-30 20:29:04.513037: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Loading pipeline components...: 100% 5/5 [00:08<00:00, 1.76s/it]
100% 25/25 [02:57<00:00, 7.11s/it]
Traceback (most recent call last):
File "/content/Text-To-Video-Finetuning/inference.py", line 232, in
export_to_video(video, f"{out_stem} {str(uuid4())[:8]}.mp4", fps)
File "/content/Text-To-Video-Finetuning/train.py", line 118, in export_to_video
img = cv2.cvtColor(video_frames[i], cv2.COLOR_RGB2BGR)
cv2.error: OpenCV(4.8.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3, 4>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = cv::impl::::NONE; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'

Invalid number of channels in input image:
'VScn::contains(scn)'
where
'scn' is 40

STILL GETTING THIS ON EVERY NOTEBOOK that has something to do with Text to Video