[ Common ] x264 FMV Range Horkage about ffnx

[ChthonVII]

I put in a PR. Unfortunately, I don't have a suitable Windows build environment readily available, so it's totally untested.

I noticed some other colorimetry issues along the way. I'll try to write them up.

[ChthonVII]

Assuming I understand the math correctly, it appears that using a decoding gamma of ~2.2 instead of the bt601/709 transfer curve gives wrong results that are very close to the observed wrong results. So my current working hypothesis is that something in the pipeline is ignoring the "transfer" metadata set by x264 and using something near 2.2 instead.

[ChthonVII]

Yep, it looks like that's it exactly. In /src/video/movies.cpp, ffmpeg is never asked to fetch the transfer metadata. And then, in misc/FFNx.common.sh, the srgb transfer curve is used to decode gamma. Which is wrong because the gamma encoding was almost certainly done with the SMPTE170M transfer curve (shared by rec601 and rec709) in the first instance.

So the fix is:

Add this function to misc/FFNx.common.sh:

vec3 toLinearSMPTE170M(vec3 _rgb)
{
	bvec3 cutoff = lessThan(_rgb.rgb, vec3_splat(0.081));
	vec3 higher = pow((_rgb.rgb + vec3_splat(0.099)) / vec3_splat(1.099), (vec3_splat(1.0) / vec3_splat(0.45)));
	vec3 lower = _rgb.rgb / vec3_splat(4.5);

	return mix(higher, lower, cutoff);
}

Then on line 96 of /misc/FFNx.frag, replace
color.rgb = toLinear(color.rgb);
with

if (isMovie){
	color.rgb = toLinearSMPTE170M(color.rgb);
}
else {
	color.rgb = toLinear(color.rgb);
}

And then make the same change on line 116 of misc/FFNx.lighting.frag.

[julianxhokaxhiu]

Thanks for the research and knowledge share. Care to make a PR? :) I'll be happy to review, thank you!

[julianxhokaxhiu]

Thanks a lot

[julianxhokaxhiu]

Closing in favor of #536