icandle / camixersr

CAMixerSR: Only Details Need More “Attention” (CVPR 2024)

CAMixerSR: Only Details Need More “Attention”

Yan Wang^1,2, Yi Liu^1*, Shijie Zhao^1*†, Junlin Li¹, Li Zhang¹

¹ByteDance, ²Nankai University

Overview: We propose CAMixerSR, a new approach integrating content-aware accelerating framework and token mixer design, to pursue more efficient SR inference via assigning convolution for simple regions but window-attention for complex textures. It exhibits excellent generality and attains competitive results among state-of-the-art models with better complexity-performance trade-offs on large-image SR, lightweight SR, and omnidirectional-image SR.

This repository contains PyTorch implementation for CAMixerSR (CVPR 2024).

Table of contents

1. Requirements
2. Datasets
3. Test
4. Results
5. Acknowledgments
6. Citation

---

⚙️ Requirements

Dependencies

• PyTorch >= 1.7
• BasicSR == 1.4.2
• einops

Installation

pip install -r requirements.txt

🎈 Datasets

Large-Image SR

Training: DIV2K.

Testing: F2K, Test2K, Test4K, Test8K (Google Drive/Baidu Netdisk).

Lightweight SR

Training: DIV2K or DF2K.

Testing: Set5, Set14, BSD100, Urban100, Manga109 (Google Drive/Baidu Netdisk).

Preparing: Please refer to the Dataset Preparation of BasicSR.

Omni-Directional-Image SR

Training/Testing: lau dataset (Google Drive/Baidu Netdisk).

Preparing: Please refer to the Step 1&2&3 of OSRT.

▶️ How to Test

Clone this repository and change the directory to ./codes.

git clone https://github.com/icandle/CAMixerSR.git
cd codes

Large-Image SR

Testing: Change the dataset path of example option to your datasets and test with the command:

# 2K
python basicsr/test.py -opt options/test/test_2K.yml
# 4K/8K
python basicsr/test.py -opt options/test/test_8K.yml

Note: We use TileModel with Tile 64x64 and Overlap 4 to constrain the calculations.

Lightweight SR

Testing: Change the dataset path of example option to your datasets and test with the command:

# x2
python basicsr/test.py -opt options/test/test_x2.yml
# x4
python basicsr/test.py -opt options/test/test_x4.yml

✨ Results

Visual comparison (click me)

Large-Image SR

Lightweight SR

Omni-Directional-Image SR

💖 Acknowledgments

We would thank BasicSR, ClassSR, and OSRT for their enlightening work!

🎓 Citation

@article{wang2024camixersr,
  title={CAMixerSR: Only Details Need More ``Attention"},
  author={Wang, Yan and Liu, Yi and Zhao, Shijie and Li, Junlin and Zhang, Li},
  journal={arXiv preprint arXiv:2402.19289},
  year={2024}
}

camixersr's Issues

train_example.yml中的meta_info_file是什么？

How do I go through the training process?

How do I go through the training process?

How to test with my own local imgs?

when i try to modify the dataset path in the yml file
i encountered this error :

could you help me ?

AssertionError：bird_HR_x4.png is not in lq_paths

您好，当我执行test_x4.yml文件进行训练时，出现了这样的问题，请问该如何解决呢

AssertionError: 41033_HR_x4.png is not in lq_paths.

您好，当我运行训练代码时，出现如标题所示的bug。我想请问HR不应该是在HR里吗，为什么要检测这个HR图片不在lq_paths里呢？我该如何解决这个问题呢？谢谢

请问作者有尝试过在这个算法中用GAN吗

请问作者,有尝试过对结果再用GAN做些对抗训练吗?就类似REAL-ESRNET & REAL-ESRGAN那样

Concerning the addition of three conditions in the article, I have some doubts and don't understand them.

Hello, author. Could you please explain why these three conditions are added? Why are they added in this way? Why can we achieve both global and local effects with just a few convolutions?

How can I use pretrained models?

Excuse me, I wonder how can I use .pth files in the folder 'pretrained models', and how can I use your model to generate a high-resolution picture?

找不到basicsr.utils

尝试使用Readme中的文件运行然后报错了
code/basicsr文件夹下并没有utils诶

Traceback (most recent call last):
File "/mnt/CAMixerSR/codes/basicsr/test.py", line 4, in
import archs
File "/mnt/CAMixerSR/codes/basicsr/archs/init.py", line 5, in
from basicsr.utils import get_root_logger, scandir
ModuleNotFoundError: No module named 'basicsr'

Flops

Why is it that the model I got after changing [1,3,32,32] to [1,3,320,180] on the original code is much more computationally intensive than in the paper?

how to run demo?

i can not find test_pipeline in the code

How to infer on real-world images?

Dear developer,

It seems that you only release the code for inference on the test set.

How to infer on real-world images?

Best wishes.

关于在训练时使用gumble softmax的细节

您好！我对您的工作非常感兴趣，想询问您为什么要在训练时使用gumble softmax，具体是怎么实现的。

Cant find the file

The flow_warp file in the basicsr. archs. arch_util directory does not exist

difference of these pretrained models

thanks for your share, could you please tell us what the diffrence of these model in pretrained_models folders, thank

the result of net is something wrong

thanks for the shared codes.
i test the pretrained models using test2K dataset but the output is wrong just like the pic blow, i can't find the reason, do you know what happened?

The mask of CAMixer, Visualization.

主要work的是CAMixer，但基本上来说，这篇工作依赖于对于复杂和简单区域采用不同复杂的的计算获得性能改善。如果区域分别不出来的话，可能需要其他指标。因为CAMixer是用图像区域的复杂度进行分类，这个指标也可以换成其他的。如果不要mask那就是相当于一个普通的WSA和卷积的结合，效果类似但计算量比较大。

我试图复现文章中的mask的作用，文章中图4展现了一些图像不同复杂度对应的概率，但是在复现过程中我发现可视化出的mask十分随机，并不能表现出图像的复杂程度，mask的上一步的pred_score也没有我想要的表现。我的问题是想知道作者在可视化图4中的概率和mask与图片复杂度对应关系上使用的是哪一部分的值？这个值应该从整个SR模型中的第几个predictor中得到？Looking forward to your reply, thanks.

File missing

In the code, there are many files that cannot be found, such as arch_util, registry,sr_model, etc.

How the offsets work?

作者你好，我觉得你们的工作成果相当出色！但是我对论文里的偏移图（offsets）的原理感到不解，为什么Predictor经过几层卷积后得到的offsets和原图扭曲就能得到具有更多信息的窗口？k = x + flow_warp(x, offsets.permute(0, 2, 3, 1), interp_mode='bilinear', padding_mode='border') 另外，在flow_warp()的实现中提到了optical flow，这里的光流要怎么理解，据我所知光流的获取需要相邻帧的图像信息，这里把offsets看作是光流要怎么理解？期待你的解答！

核心work的模块是啥

请问一下，您这个工程中最主要能work的模块是啥，我这里的需要可能跟你不一样，你的是能分出空白的和需要注意力的区域，我这里区分不出来，在空白的区域可能也需要恢复出东西来，怎么解决呢，有推荐地论文吗

Mask Visualization

我试图复现文章中的mask的作用，文章中图4展现了一些图像不同复杂度对应的概率，但是在复现过程中我发现可视化出的mask十分随机，并不能表现出图像的复杂程度，mask的上一步的pred_score也没有我想要的表现。我的问题是想知道作者在可视化图4中的概率和mask与图片复杂度对应关系上使用的是哪一部分的值？这个值应该从整个SR模型中的第几个predictor中得到？Looking forward to your reply, thanks.