Add nclx->icc colour management to heifload about libvips

[jcupitt]

Let's make this into an enhancement issue and make a test PR.

[jcupitt]

Oh, that's a good find! It looks like they have this code:

https://github.com/mnaydenov/FreeImage-Sidecar/blob/master/src/PluginHEIF.cpp#L98-L169

Which is pretty simple and should be easy to add to libvips.

Testing would be a PITA though. I have a mac mini, which has quite good HDR support, but of course it's plugged into a non-HDR display. Perhaps I can persuade libvips.org to buy me a HDR monitor ...

[Starbix]

I've started coding a bit and wanted to share my findings.

Setting the correct ICC profile (Display P3; SMPTE ST 2084 PQ) using this in vips_foreign_load_heif_set_header works great and you get a JPG that looks very close to how it should. I used the ICC profile Apple provides.

vips_image_set_blob(out, VIPS_META_ICC_NAME,
			(VipsCallbackFn) vips_area_free_cb, data, length);

Thus generating an ICC profile from the NCLX data seems to be the way to go. colorist does exactly that. Simply using colorist convert in.avif out.jpg results in a veeeery dark image, due to forcing the max luminance to 10'000 nits (only for PQ, not sure why). However force-omitting it (or a lower value around 100 nits) results in very good results.
EDIT: according to this presentation, 203 nits should be used

libheif presents a nice API to read out the relevant information of the nclx triplet. The values of this triplet are defined in H.273.

I can look into implementing this more cleanly than what I have right now.
The code in already looks really good, but it's not handling HDR cases (PQ or HLG curves). For these, colorist has the curves in the source code as binary data. Either we copy this (I'm not quite sure where those curves come from initially) or we use a parametric curve like in the Apple profile. However the Apple profile also includes "Intent-0, device to PCS conversion table" tags which may be required in conjunction with the simple parametric tone response curve and I don't know how to replicate those with LCMS2.

I've included a zip file with some relevant results and ICC profiles.
nclx_shenanigans.zip

EDIT: I've emailed the author of colorist and his response was very helpful to understanding more about the choices colorist made. The curves definitely need to be in the repo. He created them himself from the spec.

[jcupitt]

Hi @mertalev,

You two images above ^^ look identical on firefox on linux, but different in firefox on macos, interestingly.

libvips uses ICC profiles for colour management, but your image has no ICC profile. I would guess it is using nclx for encoding HDR, and that is (as far as I know) only supported by heif. I tried in imagemagick6 and 7, and they both gave the same result as libvips.

We could add nclx support, but that would be a huge amount of work. Maybe you could save your image with an ICC profile? It might at least be less work to support.

[jcupitt]

... you're right, we could also add an sdr (for example) option to heifload which enabled convert_hdr_to_8bit. That might be a simple fix.

[mertalev]

It does apparently use nclx:

Color Representation            : nclx 12 16 6

A flag to enable convert_hdr_to_8bit sounds good! The only thing this doesn't handle is HDR -> HDR, but we can avoid converting in that case and just strip metadata.

[mertalev]

Oh, but would this still let us use a Display P3 profile in the output image? Or would it have to be sRGB?

[jcupitt]

It'd become a regular SDR image, yes. All the extra HDR bits would be removed.

I think you'd need to switch to an ICC workflow and encode with 10 or 12 bits to get HDR with a P3 profile. Your test image has no profile attached.

Though I'm a bit unclear on the details -- I've never actually tried this. I ought to get an HDR display.

[mertalev]

For context, this is for a photo backup / library app where users import images that get compressed, resized and stripped to serve more quickly. On our end, the goal is to make the most of what we're given when converting these images.

Looking into this a bit more, NCLX seems to be popular for this format - all the online examples for HDR HEIC/AVIF images I could find use it. I wonder if there's a straightforward way to convert NCLX to ICC? I could find this that uses liblcms2, but it's beyond my knowledge haha

[mertalev]

If it helps, I'm happy to test a branch with the changes!

[jcupitt]

Oh very nice @Starbix, I'll try to have a look this weekend!