chaofengc / iqa-pytorch Goto Github PK

Hello! I create the AHIQ model, and want to propagate gradients:
model = pyiqa.create_metric('ahiq', as_loss=True, device=DEVICE)

However, gradient of the image tensor image.grad is None. I removed lines in AHIQ where gradient is disabled, and it works
https://github.com/chaofengc/IQA-PyTorch/blob/main/pyiqa/archs/ahiq_arch.py#L175
https://github.com/chaofengc/IQA-PyTorch/blob/main/pyiqa/archs/ahiq_arch.py#L191

There is an issue in NIQE model, I passed a tensor list of images in form [8,3,32,32] but it shows following error:

-> 4381 return torch._C._nn.replication_pad2d(input, pad)
4382 elif mode == "circular":
4383 return _pad_circular(input, pad)
RuntimeError: Expected 3D or 4D (batch mode) tensor with possibly 0 batch size and other non-zero dimensions for input, but got: [8, 1, 0, 0]

Hi，I want to test the prediction accuracy of MUSIQ and calculate plcc and srocc. Where can I get'./datasets/train_split_info/koniq10k_official.pkl'?

Does this method support CosineAnnealingRestartLR？

Thank you for your contribution！
I want to use BID datasets to train some networks in the warehouse. Do you have any relevant meta info files and train split files?https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/data_info_files.tgz

look forward to your reply.

Now it's use torch to predict, can you have plan to support deployment it on onnx.
With out torch, it will reduce image size and speed up model predict in production

在使用tid2008训练DBCNN时，在使用reference images与distorted images训练时，(https://github.com/chaofengc/IQA-PyTorch/tree/main/pyiqa/models)/general_iqa_model.py 第85行 return net(self.img_input, self.ref_input) 报错TypeError: forward() takes 2 positional arguments but 3 were given。请问有什么解决办法吗

@chaofengc

Hi, thanks for your work. I tried using maniqa, but every time I run it with the same image, the result is different, is this normal?

Observed behavior

Since CLIP dependency was added, the following error occurs:

$ python setup.py develop
>>> error in pyiqa setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Parse error at "'+https:/'": Expected string_end

The same problem occurs for PyPi installation:

However, installation from the previous commit leads to successful installation:

$ pip install git+https://github.com/chaofengc/IQA-PyTorch.git@4c5314cf45345c95ee879f1a4cf4d9c210436739
>>> Successfully built pyiqa
>>> Installing collected packages: pyiqa
>>> Successfully installed pyiqa-0.1.5

Expected behavior

Both git clone and pip install commands lead to successful installation.

Is line 253-254 of /pyiqa/archs/fid_arch.py supposed to be if verbose: print("compute FID between two folders")? When I set verbose = False, I don't want a detailed output print, but line 253-254 now is if not verbose: print("compute FID between two folders"), it printed and caused my tqdm to not display properly, Thanks!

Hi, I release the newest MANIQA checkpoints.

Thanks for a great library and your effort.

I think that you need add clip to requirements.txt.

CLIP-IQA metric requires a clip package (this one).

Otherwise import pyiqa will fail:

~/IQA-PyTorch/pyiqa/archs/clipiqa_arch.py:23
ModuleNotFoundError: No module named 'clip'

import numpy as np
from torchvision.transforms.functional import to_tensor
import torch
import imageio
import pyiqa
from PIL import Image
import cv2

if __name__ == "__main__":
    img = np.random.randint(200, 255, (250, 400)).astype(np.uint8)
    # print(img.shape)
    # cv2.imshow("0", img)
    # cv2.waitKey(2000)
    img = to_tensor(img).unsqueeze(0)
    niqe = pyiqa.create_metric('niqe', device="cpu")
    print(niqe(img))

If a generated numpy array which has close values was sent to niqe, an error would occur like this:

invcov_param = torch.linalg.pinv((cov_pris_param + cov_distparam) / 2)
torch._C._LinAlgError: linalg.svd: (Batch element 0): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 16).

Nan appears to be the problem, as it exists in cov_distparam. And some light pic will also encounter the same problem.

Would you please fix it or tell me how to deal with the problem? Thank you!

It seems that the DEFAULT_CONFIGS doesn't include the HYPERIQA. Is this metric incomplete?

我使用 python inference_iqa.py -m NRQM -i img_root，貌似不能满足我的需求

Hello, I'm trying to do BRISQUE training, however I'm not successful.
I tried to use options/train/train_general_iqa.yml as a base because there is no BRISQUE training example in the repository, but when trying to run it, I get this error "ValueError: optimizer got an empty parameter list", I would like to understand if I need to change something in the training settings so that the script can capture the optim_params.

My log:

IQA-PyTorch >>> python pyiqa/train.py -opt options/train/BRISQUE/train_brisque_livec.yml (1) (*main+254) 14:11:51
Disable distributed.
Path already exists. Rename it to /home/bilekig/Documents/Projects/IQA-PyTorch/experiments/debug_BRISQUE_NR_IQA_archived_20221215_141200
2022-12-15 14:12:00,387 INFO:
Version Information:
PyTorch: 1.13.0+cu117
TorchVision: 0.14.0+cu117
2022-12-15 14:12:00,387 INFO:
name: debug_BRISQUE_NR_IQA
model_type: GeneralIQAModel
num_gpu: 1
manual_seed: 123
save_final_results_path: ./experiments/BRISQUE_results.txt
datasets:[
train:[
name: livechallenge
type: LIVEChallengeDataset
dataroot_target: ./datasets/LIVEC/
meta_info_file: ./datasets/meta_info/meta_info_LIVEChallengeDataset.csv
split_file: ./datasets/train_split_info/livechallenge_seed123.pkl
split_index: 2
augment:[
hflip: True
random_crop: 448
]
img_range: 1
use_shuffle: True
num_worker_per_gpu: 4
batch_size_per_gpu: 8
dataset_enlarge_ratio: 1
prefetch_mode: None
phase: train
]
val:[
name: livechallenge
type: LIVEChallengeDataset
dataroot_target: ./datasets/LIVEC
meta_info_file: ./datasets/meta_info/meta_info_LIVEChallengeDataset.csv
split_file: ./datasets/train_split_info/livechallenge_seed123.pkl
split_index: 2
phase: val
]
]
network:[
type: BRISQUE
]
path:[
pretrain_network_g: None
strict_load_g: True
resume_state: None
experiments_root: /home/bilekig/Documents/Projects/IQA-PyTorch/experiments/debug_BRISQUE_NR_IQA
models: /home/bilekig/Documents/Projects/IQA-PyTorch/experiments/debug_BRISQUE_NR_IQA/models
training_states: /home/bilekig/Documents/Projects/IQA-PyTorch/experiments/debug_BRISQUE_NR_IQA/training_states
log: /home/bilekig/Documents/Projects/IQA-PyTorch/experiments/debug_BRISQUE_NR_IQA
visualization: /home/bilekig/Documents/Projects/IQA-PyTorch/experiments/debug_BRISQUE_NR_IQA/visualization
]
train:[
optim:[
type: SGD
lr: 0.001
momentum: 0.9
weight_decay: 0.0005
]
optim_finetune:[
type: Adam
lr: 1e-05
weight_decay: 0.0005
]
scheduler:[
type: MultiStepLR
milestones: [1000]
gamma: 1
]
scheduler_finetune:[
type: MultiStepLR
milestones: [1000]
gamma: 1
]
total_iter: 12000
finetune_start_iter: 6000
warmup_iter: -1
mos_loss_opt:[
type: MSELoss
loss_weight: 1.0
reduction: mean
]
]
val:[
val_freq: 7
save_img: False
pbar: True
key_metric: srcc
metrics:[
srcc:[
type: calculate_srcc
]
plcc:[
type: calculate_plcc
]
krcc:[
type: calculate_krcc
]
]
]
logger:[
print_freq: 1
save_checkpoint_freq: 7
save_latest_freq: 500.0
use_tb_logger: True
wandb:[
project: None
resume_id: None
]
]
dist_params:[
backend: nccl
port: 29500
]
dist: False
rank: 0
world_size: 1
auto_resume: False
is_train: True
root_path: /home/bilekig/Documents/Projects/IQA-PyTorch

2022-12-15 14:12:00,389 INFO: Dataset [LIVEChallengeDataset] - livechallenge is built.
2022-12-15 14:12:00,390 INFO: Training statistics:
Number of train images: 930
Dataset enlarge ratio: 1
Batch size per gpu: 8
World size (gpu number): 1
Require iter number per epoch: 117
Total epochs: 103; iters: 12000.
2022-12-15 14:12:00,392 INFO: Dataset [LIVEChallengeDataset] - livechallenge is built.
2022-12-15 14:12:00,392 INFO: Number of val images/folders in livechallenge: 232
2022-12-15 14:12:00,392 INFO: Network [BRISQUE] is created.
2022-12-15 14:12:00,392 INFO: Network: BRISQUE, with parameters: 0
2022-12-15 14:12:00,392 INFO: BRISQUE()
2022-12-15 14:12:00,392 INFO: Network [BRISQUE] is created.
2022-12-15 14:12:00,392 INFO: Loss [MSELoss] is created.
Traceback (most recent call last):
File "pyiqa/train.py", line 242, in
train_pipeline(root_path)
File "pyiqa/train.py", line 135, in train_pipeline
model = build_model(opt)
File "/home/bilekig/Documents/Projects/IQA-PyTorch/pyiqa/models/init.py", line 27, in build_model
model = MODEL_REGISTRY.get(opt['model_type'])(opt)
File "/home/bilekig/Documents/Projects/QA-PyTorch/pyiqa/models/general_iqa_model.py", line 33, in init
self.init_training_settings()
File "/home/bilekig/Documents/Projects/IQA-PyTorch/pyiqa/models/general_iqa_model.py", line 54, in init_training_settings
self.setup_optimizers()
File "/home/bilekig/Documents/Projects/IQA-PyTorch/pyiqa/models/general_iqa_model.py", line 69, in setup_optimizers
self.optimizer = self.get_optimizer(optim_type, optim_params, **train_opt['optim'])
File "/home/bilekig/Documents/Projects/IQA-PyTorch/pyiqa/models/base_model.py", line 122, in get_optimizer
optimizer = optim_class(params, lr, **kwargs)
File "/home/bilekig/.virtualenvs/test_pyiqa/lib/python3.8/site-packages/torch/optim/sgd.py", line 109, in init
super(SGD, self).init(params, defaults)
File "/home/bilekig/.virtualenvs/test_pyiqa/lib/python3.8/site-packages/torch/optim/optimizer.py", line 61, in init
raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list

I am trying to run some NR metrics (musiq as an example) on images of pixel format I420. The frame size 1920x1080. I get this error
shape '[-1, 3, 32, 32]' is invalid for input of size 3331072.

def proecss_frame(frame): 
   frame = TF.to_tensor(frame)
   frame = bt_frame.unsqueeze(0)
   return frame

What is the reason for this error? How can this be solved ?

Hi, I want to use pyiqa to train and test on new dataset, but I met some problems. Except adding a new code about new dataset in ./pyiqa/data/, Is there anywhere I need to modify? Thanks a lot!

Traceback (most recent call last):
File "pyiqa/train.py", line 242, in
train_pipeline(root_path)
File "pyiqa/train.py", line 131, in train_pipeline
result = create_train_val_dataloader(opt, logger)
File "pyiqa/train.py", line 36, in create_train_val_dataloader
train_set = build_dataset(dataset_opt)
File "/data1/xutong/anaconda3/envs/IQA/lib/python3.8/site-packages/pyiqa/data/init.py", line 34, in build_dataset
dataset = DATASET_REGISTRY.get(dataset_opt['type'])(dataset_opt)
File "/data1/xutong/anaconda3/envs/IQA/lib/python3.8/site-packages/pyiqa/utils/registry.py", line 65, in get
raise KeyError(f"No object named '{name}' found in '{self._name}' registry!")
KeyError: "No object named 'UWIQADataset' found in 'dataset' registry!"

hey, I wanted to test the train script from your amazing repo. But I always encounter the error of read_meta_info_file is not defined. I guess the import is missing in live challenge_dataset.py

It seems that the path of the img_dir, ref_dir and results_path on line 12,13,35 in tests/test_metrics.py is written incorrectly, it should be ../ but now is ./.
In addition, it seems that diff calculation method on line 68-72 works incorrectly when test metric name is dists, should the org_score be summed and then multiplied by our_score by 5 to find the difference? Now it takes the absolute difference between the mean of our_score and each number of the org_score and then calculate relative error, when the mode is DISTS, this calculation using absolute values seems to magnify the error a lot. Thanks!

I have tried to look into some parameter for psnr. However, it is much more difficult to check. I wonder if API reference is possible so that we can use the metric will revealed parameter setting.

Last but not least, I would suggest that self.device = 'cpu' if not specified when initialing the metric. It is important when using metric in DDP. cuda() will always put model and weight to gpu 0.

I have the following bugs as depicted. I wonder if .to(device) do not work for pyiqa metric.

Thanks for your awesome work.

I find some problems when I use pyiqa to calculate the NIQE score. It seems the result calculated by pyiqa is different from that of Matlab.

import matlab.engine
import matlab

import torch
import pyiqa

im = cv2.imread("img.png")


## calculate via matlab
eng = matlab.engine.start_matlab()
im_m = matlab.uint8(im.tolist())
niqe_matlab = eng.niqe(im_m)

## calculate via pyiqa
im_t = torch.FloatTensor(im.transpose(2,0,1)[None]).cuda() / 255.0
niqe_fn = pyiqa.create_metric("niqe")
niqe_pyiqa = niqe_fn(im_t)

Hi, thanks for your work. I want to learn about fast IQA methods in NR Method. do you have any recommendations?
Among the methods supported by your current project, is there a faster method?
Looking forward to your reply.

SNR (Wiki) https://en.wikipedia.org/wiki/Signal-to-noise_ratio

Thanks. Avesome IQA toolbox

Hi :) it's a nice library! i have a question about paq-2-piq IQA as a Loss function, it returns 2 values like:
tensor([[59.6956], [73.4137]])
i think it should returns just one value , i.e. one score.
what are these two values?

hi，thanks for your project!
I'd wonder why the project reports an error when i'm calculating pi and nrqm.
I changed np.complex() to np.comlex_() because of numpy version, and then it reports "TypeError: function takes at most 1 argument (2 given)".
Here is the error.
python inference_iqa.py -m nrqm -i ./ResultsCalibra/002_bicubic_x4.png 0%| | 0/1 [00:00<?, ?image/s]Traceback (most recent call last): File "/Users/limco/Downloads/IQA-PyTorch-main/inference_iqa.py", line 73, in <module> main() File "/Users/limco/Downloads/IQA-PyTorch-main/inference_iqa.py", line 52, in main score = iqa_model(img_path, ref_img_path).cpu().item() File "/Users/limco/Downloads/IQA-PyTorch-main/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/Users/limco/Downloads/IQA-PyTorch-main/pyiqa/models/inference_model.py", line 75, in forward output = self.net(target.to(self.device)) File "/Users/limco/Downloads/IQA-PyTorch-main/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/Users/limco/Downloads/IQA-PyTorch-main/pyiqa/archs/nrqm_arch.py", line 443, in forward score = calculate_nrqm(X, self.crop_border, self.test_y_channel, self.pretrained_model_path, self.color_space) File "/Users/limco/Downloads/IQA-PyTorch-main/pyiqa/archs/nrqm_arch.py", line 399, in calculate_nrqm nrqm_result = nrqm(img, linear_param, rf_params_list) File "/Users/limco/Downloads/IQA-PyTorch-main/pyiqa/archs/nrqm_arch.py", line 343, in nrqm f2 = global_gsm(img) File "/Users/limco/Downloads/IQA-PyTorch-main/pyiqa/archs/nrqm_arch.py", line 247, in global_gsm pyr = SCFpyr_PyTorch(height=2, nbands=num_bands, device=img.device).build(img) File "/Users/limco/Downloads/IQA-PyTorch-main/pyiqa/matlab_utils/scfpyr_util.py", line 47, in __init__ self.complex_fact_construct = np.power(np.complex_(0, -1), self.nbands - 1) TypeError: function takes at most 1 argument (2 given)

Thank you for your great work! I would like to know the differences among musiq-ava, musiq-koniq, musiq-paq2piq, and musiq-spaq (https://github.com/chaofengc/IQA-PyTorch/blob/main/tests/NR_benchmark_results.csv). I think that they were trained on the musiq model using different training datasets. Is it right?

Hello,

Is there any chance to get weights for pretrained models ?

Thank you in advance.

Hey there.
First of all, I would like to acknowledge the contribution of this project to the field of Image Quality Assessment. It is important for people to get access to open source implementations of various metrics and measures.
I would like to ask you whether you considered to join the piq project. I see quite some code inheritance and, in some cases, even duplication that brings limited additional value. From the other hand, your contribution to the already well established library would be appreciated and could potentially bring more impact rather than development of a very similar product in parallel. If there are some reasons for not to join other than a wish to develop your own library, could you share them?

Hi Chaofeng,

Thanks for your great work. I meet some problems when using the NIQE metric.
It seems that the NIQE results evaluated on JPG and PNG have a large difference and are different from Matlab.

In addition, the NIQE values of the same image evaluated on CPU and GPU also has a large difference.

import pyiqa
from PIL import Image
import torchvision.transforms.functional as TF
import torch
from torchvision.utils import save_image

niqe_metric = pyiqa.create_metric('niqe')

# img_jpg = TF.to_tensor(Image.open("37.jpg"))

img_tensor = torch.load("output_img.pt",map_location=torch.device('cpu'))
save_image(img_tensor, "37.png")
save_image(img_tensor, "37.jpg")

img_png = TF.to_tensor(Image.open("37.png"))
img_jpg = TF.to_tensor(Image.open("37.jpg"))

print(niqe_metric(img_tensor))
print(niqe_metric(img_jpg.unsqueeze(0)))
print(niqe_metric(img_png.unsqueeze(0)))

The output is

tensor(3.1253)
tensor(2.2823)
tensor(3.1208)

I also test the NIQE of saved JPG and PNG images using MATLAB, and the NIQE are 8.3841 and 8.6096.

In addition, it seems that the results of the evaluation on CPU and GPU also have a huge difference.

niqe_metric(img_jpg.unsqueeze(0).cuda())
tensor(4.9318, device='cuda:0')
niqe_metric(img_jpg.unsqueeze(0))
tensor(2.9255)

niqe_metric(img_png.unsqueeze(0))
tensor(3.8962)
niqe_metric(img_png.unsqueeze(0).cuda())
tensor(11.9320, device='cuda:0')

Do you have any advice on it? Thank you!

Hi @chaofengc,

Thanks for open-sourcing this repo!

Do you have MANIQA for retraining? I can see the architecture in archs.py but nothing in the options/train folder.

hi,chaofeng,i try to retrain musiq-spaq model，but the result on the test set is quite different from the pretrain model pth you gave.Can you tell me some tricks or details when training musiq...🥹？such as learning rate? Or are you training the musiq-spaq model on top of other pretrained models?

Hi!
I had this problem.
ModuleNotFoundError: No module named 'pyiqa.version'
How can I solve it?
Thank you！

Can I batch predict images?

iqa_metric = pyiqa.create_metric('nima',device= torch.device('cuda'),as_loss=False)
scores = iqa_metric(image_list_tensor).item()
image_list_tensor --> torch.Size([2, 3, 467, 700])

Line 221 is "cnn_dis = self.deform_net(cnn_dis, vit_ref)", but I think here is "cnn_dis = self.deform_net(cnn_dis, vit_dis)" to get the fused feature.

Hello, when I use NIQE metric, it reported the error as follows:

python inference_iqa.py -i /media/gpu/HS/SCGAN-main/test_results_pre/results_Widerface_96/ --metric_mode NR -m niqe --save_file /home/gpu/Documents/HS/niqe.txt

0%| | 0/2000 [00:00<?, ?image/s]Traceback (most recent call last):
File "inference_iqa.py", line 73, in
main()
File "inference_iqa.py", line 52, in main
score = iqa_model(img_path, ref_img_path).cpu().item()
File "/home/gpu/anaconda3/envs/pulse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/gpu/Documents/HS/IQA-PyTorch/pyiqa/models/inference_model.py", line 75, in forward
output = self.net(target.to(self.device), **kwargs)
File "/home/gpu/anaconda3/envs/pulse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/gpu/Documents/HS/IQA-PyTorch/pyiqa/archs/niqe_arch.py", line 446, in forward
score = calculate_niqe(X, self.crop_border, self.test_y_channel, self.pretrained_model_path, self.color_space)
File "/home/gpu/Documents/HS/IQA-PyTorch/pyiqa/archs/niqe_arch.py", line 179, in calculate_niqe
niqe_result = niqe(img, mu_pris_param, cov_pris_param)
File "/home/gpu/Documents/HS/IQA-PyTorch/pyiqa/archs/niqe_arch.py", line 133, in niqe
cov_distparam = nancov(distparam)
File "/home/gpu/Documents/HS/IQA-PyTorch/pyiqa/matlab_utils/functions.py", line 187, in nancov
cov_x.append(cov(x_no_nan, rowvar=False))
File "/home/gpu/Documents/HS/IQA-PyTorch/pyiqa/matlab_utils/functions.py", line 167, in cov
factor = 1 / (tensor.shape[-1] - int(not bool(bias)))
ZeroDivisionError: division by zero
0%| | 0/2000 [00:04<?, ?image/s]

I want to know what caused that? Thank you very much!

Thank you for your contribution！
I want to use BID datasets to train some networks in the warehouse. Do you have any relevant meta info files and train split files?https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/data_info_files.tgz

look forward to your reply.

Hi,
Thank you so much for your awesome work.
I have a question about the pretrained CNNIQA in this repo. As I understood, this model was trained on KonIQ in this repo, and the output range should be in [0, 100] as the zMOS provided in the dataset. However, when I used the pretrained metric in this repo, I see that some score outputs are negative, for examples: -4.12, or -3.33.
Could you please give the specific output range for this pretrained model?
Thank you so much!

您好，请问我用您的代码测试NIQE指标，PI指标和用Pirm-2018测出的结果有较大的差异，是否代码存在问题？

I see that Maniqa is listed as one of the available metrics. Yet, when I call pyiqaa.list_models(), I get

['brisque',
 'ckdn',
 'cw_ssim',
 'dbcnn',
 'dists',
 'fid',
 'fsim',
 'gmsd',
 'ilniqe',
 'lpips',
 'lpips-vgg',
 'mad',
 'ms_ssim',
 'musiq',
 'musiq-ava',
 'musiq-koniq',
 'musiq-paq2piq',
 'musiq-spaq',
 'nima',
 'niqe',
 'nlpd',
 'nrqm',
 'paq2piq',
 'pi',
 'pieapp',
 'psnr',
 'ssim',
 'vif',
 'vsi']

even though I have installed the latest version with pip install git+https://github.com/chaofengc/IQA-PyTorch.git.

Is there a way to find the missing models?

would like to check what the model architecture for the musiq-paq2piq method is. Is this based on the MUSIQ architecture? With paq2piq as the training dataset? thanks!

Hi. Is it possible or is there any way to pass images as a tensors instead of passing them as paths or filenames to perform inference

Whether this method supports “CosineAnnealingRestartLR”

I guess it should be like: split_file: ./datasets/train_split_info/livechallenge_seed123.pkl

Another question regarding the pertained models. Where can I get them since the folder is not found '.experiments/pretrained_models...'

Thanks to the author for providing such an excellent iqa model code. Can the author upload the yml file of musiq-spaq? Thanks to the author 🙏

Hi chaofeng, can you share the yml file for training MUSIQ ? Thank you!

Hi, how to train Model paq2piq？no yaml file found