All EXIF data selected after resetting history in Batch Edit about rawtherapee

[Lawrence37]

I can reproduce the ISOSpeedRatings tag warning, but not the rest. Can you provide the pp3(s) if you made any modifications to the metadata in RawTherapee and also the associated source image(s)? If you do not want to share the images, provide the output of exiv2 -pa your_image_here.raw for each source image and tif produced by RawTherapee.

[hujuice]

Yes, I am understanding something more.
I will add what you are asking for in the following comment.

I encountered the fatal error with the following pp3
20240222124633_4234.CR2.pp3.txt
(I manually added a .txt suffix because of GitHub, everything happens without that suffix)

With this pp3, I have the fatal "Sanity check on directory count failed" with just one file.

1. Generate the TIF file
2. Use mogrify

$ mogrify -resize 1920x1920 *.tif
mogrify: Incorrect count for "MakerNote"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/958.
mogrify: Wrong data type 3 for "PixelXDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
mogrify: Wrong data type 3 for "PixelYDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
mogrify: Unknown field with tag 40965 (0xa005) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
mogrify: Sanity check on directory count failed, zero tag directories not supported. `TIFFFetchDirectory' @ error/tiff.c/TIFFErrors/599.
mogrify: Failed to read directory at offset 73692. `TIFFReadDirectory' @ error/tiff.c/TIFFErrors/599.

The pp3 file has been generated using a simple partial custom profile, via RawTherapee.
Right click -> Processing profile operation -> Apply -> My profiles -> credits
where credits.pp3 is

[MetaData]
Mode=1

[Exif]
Artist=Choreography: Christodoulidou Elena. Dancers: Elina Karakosta, Anna Nicolaou, Ivi Hadjivasileiou, Maria Masonou. Photos: Sergio Vaccaro.
Copyright=CC-BY-NC-SA
ImageDescription=Between the memories/Through time and space
Exif.UserComment=Egomio Cultural Centre

This is what I was use to do until RawTherapee 5.9.
The result, with RawTherapee 5.10, is to have all the tags selected. When I apply my partial profile now RawTherapee adds ExifKeys=*; in the [MetaData] section.
And here is the failure.

If I do everything manually (no custom profile), the "Basic" section only is activated.

[MetaData]
Mode=1
ExifKeys=Exif.Image.Copyright;Exif.Image.Artist;Exif.Image.ImageDescription;Exif.Photo.UserComment;Exif.Image.Make;Exif.Image.Model;Exif.Photo.LensModel;Exif.Photo.FNumber;Exif.Photo.ExposureTime;Exif.Photo.FocalLength;Exif.Photo.ISOSpeedRatings;Exif.Photo.ExposureBiasValue;Exif.Photo.Flash;Exif.Photo.DateTimeOriginal;Exif.Image.XResolution;Exif.Image.YResolution;

[Exif]
Artist=Choreography: Christodoulidou Elena. Dancers: Elina Karakosta, Anna Nicolaou, Ivi Hadjivasileiou, Maria Masonou. Photos: Sergio Vaccaro.
Copyright=CC-BY-NC-SA
ImageDescription=Between the memories/Through time and space
Exif.UserComment=Egomio Cultural Centre

In this case I only have the ISOSpeed warning.

$ mogrify -resize 1920x1920 *.tif
mogrify: Wrong data type 9 for "ISOSpeedRatings"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.

I know that everything is a bit jammed.

Changing method is not a problem for me.
There are still, anyway, a couple of issues:

1. Backward compatibility: the pp3 files generated in the previous years are wrong, I should perform a massive sed correction.
2. The error comes apparently when ExifKeys=*;, so there's something bad in another sections.

I'm sorry for the long comment, I think it was necessary.

Thank you,
HUjuice

[hujuice]

The raw file:
https://www.sergiovaccarophoto.com/20240222124633_4234.CR2
https://www.sergiovaccarophoto.com/20240222124633_4234.CR2.pp3

The generated TIF file:
https://www.sergiovaccarophoto.com/20240222124633_4234.tif

exiv2 -pa 20240222124633_4234.CR2:
20240222124633_4234.CR2-exif.txt

exiv2 -pa 20240222124633_4234.tif:
20240222124633_4234.tif-exif.txt

Further note:

$ exiv2 -pa 20240222124633_4234.tif > /dev/null
Warning: Directory Image3, entry 0x0111: Strip 0 is outside of the data area; ignored.

It looks like there are multiple images (thumbnails, layers) in the TIF file.

Further note: there are no backward compatibilities. The "old" pp3 files without any ExifKeys= field work as expected.

[Lawrence37]

The fatal error is due to the copying of tags that should not be in the final tif. Many of the tags are specific to the raw file. Instead of keeping all the metadata tags, only select the ones you need. In general, the ones in the Basic group are safe to copy. Your raw file also has a GPSInfo group which I think is also safe. Some tags in Photo are also safe, but most or all of that data is already in the Basic group.

The wrong data type warnings are definitely a RawTherapee bug. The copyright warning is potentially a bug too, but I have not managed to reproduce it.

[hujuice]

Yes, I understood this. That's why I find that RT shouldn't add ExifKeys=*;, whatever my behavior was.
Exif data are a plethora of tags. I think RT should copy the modified ones only.

[Lawrence37]

I created a partial profile with the contents you supplied and applied it to your image in v5.10, but I could not reproduce the issue. The tags in the Basic group remain selected while all other tags remain deselected. The copyright, artist, etc. tags are modified as expected.

[hujuice]

I understand that you are dealing with a plethora of things.
It's difficult, for me too, to provide organised information.

I am playing with my command line right now. Look at this:

sergio@box ~/foto/spool $ /usr/bin/rawtherapee-cli -Y -o "/home/sergio/foto/spool" -s -t -c /home/sergio/foto/raw/selected/2018/Limisso/20180702213000_4417.CR2
RawTherapee, version 5.10, command line.
Output is 16-bit integer.
Processing: /home/sergio/foto/raw/selected/2018/Limisso/20180702213000_4417.CR2
Warning: Directory Canon, entry 0x0000 has unknown Exif (TIFF) type 0; setting type size 1.
  Merging sidecar procparams.

sergio@box ~/foto/spool $ /usr/bin/rawtherapee-cli -Y -o "/home/sergio/foto/spool" -s -j -c /home/sergio/foto/raw/selected/2018/Limisso/20180702213000_4417.CR2
RawTherapee, version 5.10, command line.
Output is 8-bit integer.
Processing: /home/sergio/foto/raw/selected/2018/Limisso/20180702213000_4417.CR2
Warning: Directory Canon, entry 0x0000 has unknown Exif (TIFF) type 0; setting type size 1.
  Merging sidecar procparams.
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.Canon.CRWParam not encoded

sergio@box ~/foto/spool $ identify -format '%w %h' 20180702213000_4417.tif
identify: Incorrect count for "MakerNote"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/958.
identify: Wrong data type 3 for "PixelXDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
identify: Wrong data type 3 for "PixelYDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
identify: Unknown field with tag 40965 (0xa005) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
identify: Sanity check on directory count failed, zero tag directories not supported. `TIFFFetchDirectory' @ error/tiff.c/TIFFErrors/599.
identify: Failed to read directory at offset 72950. `TIFFReadDirectory' @ error/tiff.c/TIFFErrors/599.

sergio@box ~/foto/spool $ identify -format '%w %h' 20180702213000_4417.jpg 
3000 2001

(identify is part of the imagemagick toolkit)

These issues don't appear with every file, but in this file I really don't find anything strange. Selected Exif tags are from the "Basic" group only.

I am tempted to go back to RT 5.9, to verify if the same issues happen. If it is helpful for you, I will do it.
Please, let me know.

[hujuice]

20180702213000_4417.CR2.pp3.txt

[Lawrence37]

There should be no errors (only warnings) if you select only the tags in the Basic group. In the pp3 you just provided, there are other tags selected. Same with the pp3s you used in the CLI. Review your processing profiles. There is something in them that is adding the extra tags.

[hujuice]

There's nothing outside Basic there. Also, mode is 0.

[MetaData]
Mode=0
ExifKeys=Exif.Image.Artist;Exif.Image.Copyright;Exif.Image.ImageDescription;Exif.Image.Make;Exif.Image.Model;Exif.Image.XResolution;Exif.Image.YResolution;Exif.Photo.DateTimeOriginal;Exif.Photo.ExposureBiasValue;Exif.Photo.ExposureTime;Exif.Photo.FNumber;Exif.Photo.Flash;Exif.Photo.FocalLength;Exif.Photo.ISOSpeedRatings;Exif.Photo.LensModel;Exif.Photo.UserComment;

I plaid with IPTC.

Another file, check this. Here RT goes fine, but imagemagick goes fatal.

sergio@box ~/foto/spool $ /usr/bin/rawtherapee-cli -Y -o "/home/sergio/foto/spool" -s -t -c /home/sergio/foto/raw/selected/2020/Nicosia/03/20200308173738_7004.CR2
RawTherapee, version 5.10, command line.
Output is 16-bit integer.
Processing: /home/sergio/foto/raw/selected/2020/Nicosia/03/20200308173738_7004.CR2
  Merging sidecar procparams.

sergio@box ~/foto/spool $ identify -format '%w %h' 20200308173738_7004.tif 
identify: Wrong data type 3 for "PixelXDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
identify: Wrong data type 3 for "PixelYDimension"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
identify: Unknown field with tag 40965 (0xa005) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
identify: Sanity check on directory count failed, zero tag directories not supported. `TIFFFetchDirectory' @ error/tiff.c/TIFFErrors/599.
identify: Failed to read directory at offset 46552. `TIFFReadDirectory' @ error/tiff.c/TIFFErrors/599.

20200308173738_7004.CR2.pp3.txt

I left my IT life for photography, RT is a column of my workflow, now. I'll do my best to help.

[Lawrence37]

In that pp3, you have the Metadata copy mode set to Copy unchanged. That will copy all the tags which is why imagemagick cannot process the image.

[hujuice]

Well, this is unclear to me.
"Copy unchanged" sounds like the Exif tags are the ones provided by my Canon camera (vs "Apply modifications").

Anyway, you are right, if I change this setting ("Apply modifications") the fatal error is resolved.

sergio@box ~/foto/spool $ /usr/bin/rawtherapee-cli -Y -o "/home/sergio/foto/spool" -s -t -c /home/sergio/foto/raw/selected/2020/Nicosia/03/20200308173738_7004.CR2
RawTherapee, version 5.10, command line.
Output is 16-bit integer.
Processing: /home/sergio/foto/raw/selected/2020/Nicosia/03/20200308173738_7004.CR2
  Merging sidecar procparams.

sergio@box ~/foto/spool $ identify -format '%w %h' 20200308173738_7004.tif
identify: Wrong data type 9 for "ISOSpeedRatings"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/958.
3000 2000

Tomorrow (it's 2am here) I'll verify with more files.

Thank you

[hujuice]

One of the errors above doesn't depend on RT, apparently.
It appears also while handling the CR2 files.

$ exiv2 20190611140432_0859.CR2 1>/dev/null 
Warning: Directory Canon, entry 0x0000 has unknown Exif (TIFF) type 0; setting type size 1.

This happens with files created by my old Canon 700D.

[Lawrence37]

"Copy unchanged" sounds like the Exif tags are the ones provided by my Canon camera

That is exactly right and that's why the tif metadata causes problems. Many of the tags in the raw file are specific to the raw and should not be copied to the tif.

[hujuice]

"Copy unchanged" sounds like the Exif tags are the ones provided by my Canon camera

That is exactly right and that's why the tif metadata causes problems. Many of the tags in the raw file are specific to the raw and should not be copied to the tif.

I didn't know it.

So the main issue is that ExifKeys=*; is added when I load my partial profile.
If I remove that line, I find my custom fields selected, as expected.
Out of this, there's just the ISOSpeedRatings issue, only if ExifKeys explicitly requires it.

[Lawrence37]

What are the exact steps to reproduce the ExifKeys=*; problem? It doesn't do that when I tried applying the partial profile.

[hujuice]

I often prepare a snippet of pp3 to apply license/credits to large groups of photos.
For example:

sergio@box ~/.config/RawTherapee/profiles/credits $ cat BetweenTheMemories.pp3 
[MetaData]
Mode=1

[Exif]
Artist=Choreography: Christodoulidou Elena. Dancers: Elina Karakosta, Anna Nicolaou, Ivi Hadjivasileiou, Maria Masonou. Photos: Sergio Vaccaro.
Copyright=CC-BY-NC-SA
ImageDescription=Between the memories/Through time and space
Exif.UserComment=Egomio Cultural Centre

(forget the Mode=0 cases: I had it rarely, probably because of manual actions)
Then I apply it with right-click -> Proessing profile operations -> Apply -> My profiles -> credits -> BetweenTheMemories on the group of files.
I've been doing it for many years/versions.

Now, with RT-5.10, after this action I find ExifKeys=*; under Mode=1.
On the UI side, I find all the Exif tags selected, in all groups.
According to what you told before, this is a problem.

--

As a workaround, if in my partial profile I add
ExifKeys=Exif.Image.Copyright;Exif.Image.Artist;Exif.Image.ImageDescription;Exif.Photo.UserComment;
(or more in the Basic group) I find my needed tags selected only, and the problem is resolved.

About the past, today I made heavy sed action, where I granted in all my (thousands of) pp3 files Mode=0 and no ExifTags line.
I am processing hundreds of files for printing and web purposes. Everything works fine until now (except warnings).

[Lawrence37]

I don't know what is going on. I created a partial profile with the contents you showed and saved it to my profiles directory. Then I opened RawTherapee, right-clicked an image, and applied the profile as you described. Opening the image in the editor and looking at the metadata tab, I can see the metadata has been successfully applied and only the tags in the Basic group are selected. Viewing the pp3 shows that ExifKeys is a list of the dozen or so selected tags, not *.

There are two ways I can coerce RawTherapee to select all tags.

1. Apply a pp3 that selects all the tags before applying the partial profile.
2. Add a Dynamic Profile Rule with a pp3 that selects all the tags and apply the partial profile using "Apply - partial" instead of "Apply".

Neither of these sound like what you are doing, and in both of these ways, it is expected behavior to have all tags selected.

[hujuice]

You know? I also cannot reproduce.
I did my operations again with a copy and I didn't have the same issue.

Please, consider that I had the ExifKeys=*; issue with two groups of photos, not just one time. And consider that I didn't know the ExifKey existence before.
For sure I didn't change anything manually. Selecting all tags is not believable.
What did I do?

So, I am sorry for your time, I cannot provide more informations.
If I grep in my backup I still have many pp3 with ExifKeys=*;, belonging to recent groups of files (after I upgraded to RT-5.10, but it could be a coincidence at this point).

I suggest to drop the issue (except the warnings), at least for the moment.
Thank you for your work.

[hujuice]

Just to make you know, today I took a photo collection from my past. Things working smoothly in the past.
I exported a photo in TIF and I have found layers.
So I opened the pp3 and I've found ExifKeys=*; (all tags selected in the UI).

I tried with other photos and I've found in some case the same, in other cases ExifKeys=Exif.Image.Artist;Exif.Image.Copyright;Exif.Image.ImageDescription;Exif.Image.Make;Exif.Image.Model;Exif.Image.XResolution;Exif.Image.YResolution;Exif.Photo.DateTimeOriginal;Exif.Photo.ExposureBiasValue;Exif.Photo.ExposureTime;Exif.Photo.FNumber;Exif.Photo.Flash;Exif.Photo.FocalLength;Exif.Photo.ISOSpeedRatings;Exif.Photo.LensModel;Exif.Photo.UserComment;
Grepping the pp3 files, only the photos I opened with RawTherapee 5.10 have this. All the other photos - the photos I never opened with RT-5.10 - are "clean": no ExifKeys= and eventually custom tags in the [Exif] section (Artist=blabla, Copyright=blabla, ImageDescription=blabla, Exif.UserComment=blabla).

RT-5.10 adds this setting, ExifKeys=, but it doesn't do it correctly (and it shouldn't do it, imo, like before).

[hujuice]

Ok, I've found something I can reproduce.

1. Take a photo, apply default settings from the File Browser (Right click -> Processing profile operations -> Reset to default). Now there is a ExifKeys setting under [MetaData] with all the tags of the Basic group listed. (This is already excessive, for what I think.)
2. Select the photo in the File Browser, change something using the Batch Edit tab (I selected the "Resize" button, nothing else) and revert the history. ExifKeys didn't change.
3. Open the photo: now ALL the tags in ALL groups are selected, but the pp3 is still at the Basic list.
4. Close the photo. Now ExifKeys=*;.

Please, try to reproduce it.
Thank you.

[Lawrence37]

Thanks for finding a way to reproduce it. I can see that the issue happens if you reset the history to the first entry in the batch edit tab. It does not happen in the editor and it does not happen unless you go back to the first history entry.

[hujuice]

Yes, I do this very often. Here is why the issue is frequent for me.