You can easily calculate the following video quality metrics:
- FVD: Frechét Video Distance
- PSNR: peak-signal-to-noise ratio
- SSIM: structural similarity index measure
- LPIPS: learned perceptual image patch similarity
As for FVD, the code refers to MVCD and other websites and projects, I've just extracted the part of it that's relevant to the calculation. This code can be used to evaluate FVD scores for generative or predictive models.
- This project supports grayscale and RGB videos.
- This project supports Ubuntu, but maybe something is wrong with Windows. If you can solve it, welcome any PR.
- If the project cannot run correctly, please give me an issue or PR~
- use
scipy==1.7.3/1.9.3
, if you use 1.11.3, you will calculate a WRONG FVD VALUE!!! For more details see below Notice.
8 videos of a batch, 10 frames, 3 channels, 64x64 size.
import torch
from calculate_fvd import calculate_fvd
from calculate_psnr import calculate_psnr
from calculate_ssim import calculate_ssim
from calculate_lpips import calculate_lpips
NUMBER_OF_VIDEOS = 8
VIDEO_LENGTH = 30
CHANNEL = 3
SIZE = 64
videos1 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)
videos2 = torch.ones(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)
device = torch.device("cuda")
device = torch.device("cpu")
import json
result = {}
result['fvd'] = calculate_fvd(videos1, videos2, device)
result['ssim'] = calculate_ssim(videos1, videos2)
result['psnr'] = calculate_psnr(videos1, videos2)
result['lpips'] = calculate_lpips(videos1, videos2, device)
print(json.dumps(result, indent=4))
It means we calculate:
FVD-frames[:10]
,FVD-frames[:11]
, ...,FVD-frames[:30]
avg-PSNR/SSIM/LPIPS-frame[0]
,avg-PSNR/SSIM/LPIPS-frame[1]
, ...,avg-PSNR/SSIM/LPIPS-frame[:30]
, and their std.
We cannot calculate FVD-frames[:8]
, and it will pass when calculating, see ps.6.
The result shows: a all-zero matrix and a all-one matrix, their FVD-30 (FVD[:30]) is 152.15. We also calculate their standard deviation. Other metrics are the same.
{
"fvd": {
"value": {
"10": 570.07320378183,
"11": 486.1906542471159,
"12": 552.3373915075898,
"13": 146.6242330185728,
"14": 172.57268402948895,
"15": 133.88932632116126,
"16": 153.11023578170108,
"17": 357.56400892781204,
"18": 382.1335612721498,
"19": 306.7100176942531,
"20": 338.18221898178774,
"21": 77.95587603163293,
"22": 82.49997632357349,
"23": 64.41624523513073,
"24": 66.08097153313875,
"25": 314.4341061962642,
"26": 316.8616746151064,
"27": 288.884418528541,
"28": 287.8192683223724,
"29": 152.15076552354864
},
"video_setting": [
8,
3,
30,
64,
64
],
"video_setting_name": "batch_size, channel, time, heigth, width"
},
"ssim": {
"value": {
"0": 9.999000099990664e-05,
...,
"29": 9.999000099990664e-05
},
"value_std": {
"0": 0.0,
...,
"29": 0.0
},
"video_setting": [
30,
3,
64,
64
],
"video_setting_name": "time, channel, heigth, width"
},
"psnr": {
"value": {
"0": 0.0,
...,
"29": 0.0
},
"value_std": {
"0": 0.0,
...,
"29": 0.0
},
"video_setting": [
30,
3,
64,
64
],
"video_setting_name": "time, channel, heigth, width"
},
"lpips": {
"value": {
"0": 0.8140146732330322,
...,
"29": 0.8140146732330322
},
"value_std": {
"0": 0.0,
...,
"29": 0.0
},
"video_setting": [
30,
3,
64,
64
],
"video_setting_name": "time, channel, heigth, width"
}
}
- You should
pip install lpips
first. - Make sure the pixel value of videos should be in [0, 1].
- If you have something wrong with downloading FVD pre-trained model, you should manually download any of the following and put it into FVD folder.
- For grayscale videos, we multiply to 3 channels as it says.
- We average SSIM when images have 3 channels, ssim is the only metric extremely sensitive to gray being compared to b/w.
- Since
frames_num
should > 10 when calculating FVD, so FVD calculation begins from 10-th frame, like upper example. - You had better use
scipy==1.7.3/1.9.3
, if you use 1.11.3, **you will calculate a WRONG FVD VALUE!!! **
python calculate_fid.py 需要设定两个视频目录,一个是生成的,另一个是GT. 视频目录的视频数量要是4的倍数(原因未知,可能与torchscript这种模型读取运行方式有关). 运行流程:
def load_feature():
videos_feature = []
for every video:
feature = I3D(video)
videos_feature.append(feature)
return videos_feature
features_predict = load_feature(gt_videos)
features_gt = load_feature(predict_videos)
calculate_fid(features_predict,features_gt) # 这步和图像的计算是一样的