picsart-ai-research / videoinr-continuous-space-time-super-resolution Goto Github PK

I noticed that in section 4.1 of the paper, the center frame is referred to as 1 4 9. This is causing confusion due to its asymmetry, particularly with the first frame being subscripted as 1 instead of 0.

While trying to locate your corresponding code, I came across the sequence 0 4 8, which clarified my query. To confirm, should it be as 1 5 9 in the paper?

https://github.com/Picsart-AI-Research/VideoINR-Continuous-Space-Time-Super-Resolution/blob/810854719afb657cc2c2feb796b7b7199023ebb9/data/Adobe_dataset.py#L191C1-L192C48

Hello,

MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

• Video Upscalers Benchmark: Quality Enhancement determines the best upscaling methods for increasing video resolution and improving visual quality.
• Super-Resolution for Video Compression benchmark aims to test Super-Resolution methods on compressed videos and select the best model for each video codec standard.

If you are interested in participating, you can add your algorithm following the submission steps:

• Submit for Video Upscalers Benchmark: Quality Enhancement
• Submit for Super-Resolution for Video Compression benchmark

We would be grateful for your feedback on our work!

I am having a hard time running the demo due to the following reasons:

• This module seems to be missing import models.modules.STVSR as STVSR
• This module also seems to be missing Sakuya_arch_test
• The DCNv2 module is not compatible with newer GPUs, I can't compile it

I did the following int your installation guid.
It seems to require cuda arch 8.0 but this torch version (1.4.0) doesn't support that version.

In cpp_extension.py there is not 8.0 valid arch for cuda.

Any help?

Thanks in advance.

I saw in the code that it says X2-downsampling the original image

@zychen-ustc
Could you please share your script in testing the pre-trained model on the Gopro dataset?
I'm trying to use the pre-trained model to input low-resolution 1st and 5th frames and generate high-resolution 2nd to 4th frames, but the frames I'm generating are only ~20dB on PSNR, which is much less than ~30dB stated in your paper.

Thanks for sharing this interesting code.
Can you show the processing code of data preparation? I follow TMNet to prepare it but it seems different with the format of this dataloader.

great work! But I have trouble understanding the process of motion flow field warping, I'd appreciate it very much if you are willing to give some hints.
In the last of LunaTokis.decoing_test(), there are some variances, namely "q_feat1, q_feat2, q_feat3, q_feat4, q_img1, q_img2". And these variances are concated and decoded to RGBs? What does these do?

1.请问为什么GT需要先imresize为原图的1/2再做crop 而不是直接在原分辨率上crop？
2.使用pre-train model做测试时，我选取一个视频读取帧中的第1帧，第9帧，第17帧,...,作为输入，但得到的结果完全不对，不知道是否我的图像输入有问题
3.请问特征aggregation部分的详细说明请问在哪里能获得？
4.训练和测试阶段的解码为何有较大区别？测试阶段的解码部分为何pipeline上有3个feature，拆开再做cat,最后预测也是三个feature分别做三个通道的预测？
非常感谢您的工作！期待您的回复！

Firstly, your research has been incredibly insightful, and I am truly grateful for the knowledge and inspiration the VideoINR paper has provided.

I recently had the opportunity to explore the code implementation accompanying your paper.
While I found it to be a valuable resource, there is one aspect that I believe requires further clarification and improvement.
Specifically, I noticed a lack of guidance regarding the evaluation metrics, PSNR and SSIM.

Additionally, I encountered a scenario where the video sequences in Adobe and GoPro contained varying numbers of scenes.
In such cases, I was uncertain whether the averaging of performance metrics was conducted at the level of the entire scene or if individual scene averages were computed before obtaining an overall average.
If you could kindly shed some light on this matter, it would significantly enhance our understanding and enable us to effectively evaluate the performance of the Video INR method.

I am also curious if you utilized commonly used libraries such as skimage or cv2 for measuring PSNR and SSIM.

Moreover, I am wondering if it would be possible for you to share the evaluation code used in your research.
Having access to the actual code would immensely help in understanding the implementation details.

Thank you!

Hi, thank you for your great work!

I have one question, what is the license of this code?
In README.md, you say:

Our code is built on Zooming-Slow-Mo-CVPR-2020 and LIIF. Thank the authors for sharing their codes!

It looks like Zooming-Slow-Mo-CVPR-2020 has GPL-3.0 and LIIF has BSD-3-Clause license. GPL-3.0 has the copyleft feature, so probably this code is under GPL-3.0 as well?

Thank you!

不好意思，请问表一中两阶段模型的测试有公开的代码吗，新手小白非常想要学习一下，能提供就太好了。

Hello, do you encounter this problem during training? Neither my code nor the dataset has changed. I will be very grateful if you can solve my problem

Thanks for your great job!
Could you provide the scripts for testing, which can reproduce the performances (PSNR and SSIM) reported in the paper?
It would be very helpful. Thanks again!

Thank you for your great work.
Regarding the evaluation part, I have a question: are the PSNR and SSIM metrics in the paper obtained on the Y channel?
Looking forward to your rely~

你好，请问在win10系统下是否一定要求CUDA版本为10.1？
目前Python 3.6，CUDA版本11.1，GPU RTX3070Ti
起初根据文档内容配置相关环境，执行python setup.py install的结果为ValueError: Unknown CUDA arch (8.6) or GPU not supported
尝试更换新版本的pytorch=1.8.0，torchvision=0.9继续根据文档进行默认配置，执行以下语句
‘’‘
conda create -n videoinr python=3.6
conda activate videoinr
conda install pytorch=1.4 torchvision -c pytorch

pip install opencv-python pillow tqdm
cd models/modules/DCNv2/
python setup.py install
’‘’
输出信息为
‘’‘
running install_lib
running build_ext
C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\utils\cpp_extension.py:304: UserWarning: Error checking compiler version for cl: [WinError 2] 系统找不到指定的文件。
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
......
FAILED: E:/ResourceCode/VideoINR-Continuous-Space-Time-Super-Resolution/models/modules/DCNv2/build/temp.win-amd64-3.6/Release/ResourceCode/VideoINR-Continuous-Space-Time-Super-Resolution/models/modules/DCNv2/src/cuda/dcn_v2_cuda.obj
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output E:\ResourceCode\VideoINR-Continuous-Space-Time-Super-Resolution\models\modules\DCNv2\build\temp.win-amd64-3.6\Release\ResourceCode\VideoINR-Continuous-Space-Time-Super-Resolution\models\modules\DCNv2\src\cuda\dcn_v2_cuda.obj.d --use-local-env -Xcompiler /MD -Xcompiler /wd4819 -Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -Xcompiler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -DWITH_CUDA -IE:\ResourceCode\VideoINR-Continuous-Space-Time-Super-Resolution\models\modules\DCNv2\src -IC:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\include -IC:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\include\torch\csrc\api\include -IC:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\include\TH -IC:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\DevelopmentProgram\anaconda3\envs\videoinr11.1\include -IC:\DevelopmentProgram\anaconda3\envs\videoinr11.1\include "-IE:\Program\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IE:\Program\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -c E:\ResourceCode\VideoINR-Continuous-Space-Time-Super-Resolution\models\modules\DCNv2\src\cuda\dcn_v2_cuda.cu -o E:\ResourceCode\VideoINR-Continuous-Space-Time-Super-Resolution\models\modules\DCNv2\build\temp.win-amd64-3.6\Release\ResourceCode\VideoINR-Continuous-Space-Time-Super-Resolution\models\modules\DCNv2\src\cuda\dcn_v2_cuda.obj -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75
......
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\utils\cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "setup.py", line 70, in
cmdclass={"build_ext": torch.utils.cpp_extension.BuildExtension},
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\setuptools\command\install.py", line 67, in run
self.do_egg_install()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\setuptools\command\install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\setuptools\command\bdist_egg.py", line 169, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\setuptools\command\bdist_egg.py", line 155, in call_command
self.run_command(cmdname)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\command\install_lib.py", line 107, in build
self.run_command('build_ext')
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\setuptools\command\build_ext.py", line 75, in run
_build_ext.run(self)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\utils\cpp_extension.py", line 708, in build_extensions
build_ext.build_extensions(self)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\command\build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\command\build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\setuptools\command\build_ext.py", line 196, in build_extension
_build_ext.build_extension(self, ext)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\distutils\command\build_ext.py", line 533, in build_extension
depends=ext.depends)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\utils\cpp_extension.py", line 690, in win_wrap_ninja_compile
with_cuda=with_cuda)
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\utils\cpp_extension.py", line 1359, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "C:\DevelopmentProgram\anaconda3\envs\videoinr11.1\lib\site-packages\torch\utils\cpp_extension.py", line 1683, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
’‘’
根据输出结构在命令行输入ninja -v得到ninja: error: loading 'build.ninja': 系统找不到指定的文件
查看了其他issues，是否无法在pytorch>1.5.0的环境上运行

Is there any way i can try to upscale from higher resolution? The resolution of this model is 160x90 pix to 4x.
Is it possible to start on a higher resolution like 960x640?

Hi there,

Thanks for your great work. Can I ask what is the training time, and GPU to train the model.

BW