Ability to reindex an exhibit keeps throwing an error about exhibits

[jcoyne]

I fixed this one sidecar by doing the following:

sidecar = Spotlight::SolrDocumentSidecar.find_by(exhibit_id: 128, resource_id: 25244)
sidecar.update_column(:data, {})

exhibit = Spotlight::Exhibit.find(128)
data = Spotlight::Resources::IiifService.parse('https://gallica.bnf.fr/iiif/ark:/12148/btv1b550020492/manifest.json').first
data.to_solr(exhibit: exhibit)

[corylown]

I'm seeing in the logs:

Oct 19 13:51:42 exhibits-worker-prod-a sidekiq[2873046]: 2023-10-19T20:51:42.434Z pid=2873046 tid=1mory class=Spotlight::ReindexJob jid=c9842cd958d14d0d0d8aa169 ERROR: Caught exception Tried to load unspecified class: IIIF::OrderedHash

[jcoyne]

@corylown this might be a missing require for iiif-presentation:

./bin/rails c
Loading development environment (Rails 7.0.8)
irb(main):001> IIIF::OrderedHash
(irb):1:in `<main>': uninitialized constant IIIF::OrderedHash (NameError)

IIIF::OrderedHash
    ^^^^^^^^^^^^^
irb(main):002> require 'iiif/presentation'
=> true
irb(main):003> IIIF::OrderedHash
=> IIIF::OrderedHash

That is required in https://github.com/projectblacklight/spotlight/blob/main/app/models/spotlight/resources/iiif_harvester.rb#L3 and
https://github.com/projectblacklight/spotlight/blob/main/app/models/spotlight/resources/iiif_service.rb#L3

Ah! It looks like this is a Psych error: https://app.honeybadger.io/projects/49092/faults/101531481

Psych::DisallowedClass: Tried to load unspecified class: IIIF::OrderedHash

So, I suspect this is not in an AllowList of classes that Psych will safely deserialize. We'd need to add this to config.active_record.yaml_column_permitted_classes, but I wonder why that isn't in here: https://github.com/projectblacklight/spotlight/blob/0f5a47eefe43dafddd1702869a9f57aea0d4e6e4/lib/spotlight/engine.rb#L338-L341

3.1.0 :014 > sidecar = Spotlight::SolrDocumentSidecar.find_by(exhibit_id: 128, resource_id: 25244)
(Object doesn't support #inspect)
 =>
3.1.0 :015 > sidecar.attributes_before_type_cast
 =>
{"id"=>851735,
 "exhibit_id"=>128,
 "public"=>1,
 "data"=>
  "---\nreadonly_repository_tesim:\n- Bibliothèque nationale de France\nreadonly_digitised-by_tesim:\n- Bibliothèque nationale de France\nreadonly_source-images_tesim:\n- https://gallica.bnf.fr/ark:/12148/btv1b550020492\nreadonly_metadata-source_tesim:\n- http://oai.bnf.fr/oai2/OAIHandler?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:bnf.fr:gallica/ark:/12148/btv1b550020492\nreadonly_shelfmark_tesim:\n- Bibliothèque nationale de France, département Musique, MS-79\nreadonly_title_tesim:\n- \"[Ah ! perfido, op. 65 : fragment] (manuscrit autographe) / Ludwig van Beethoven]\"\nreadonly_date_tesim:\n- '1796'\nreadonly_format_tesim:\n- !ruby/hash:IIIF::OrderedHash\n  \"@value\": 1 partition (2 f.) ; 21 x 32 cm\n- !ruby/hash:IIIF::OrderedHash\n  \"@value\": application/pdf\n- !ruby/hash:IIIF::OrderedHash\n  \"@value\": 'Nombre total de vues :  4'\nreadonly_creator_tesim:\n- !ruby/hash:IIIF::OrderedHash\n  \"@value\": Beethoven, Ludwig van (1770-1827). Compositeur\n- !ruby/hash:IIIF::OrderedHash\n  \"@value\": Beethoven, Ludwig van (1770-1827). Compositeur\nreadonly_relation_tesim:\n- 'Notice d’oeuvre : http://catalogue.bnf.fr/ark:/12148/cb14794427w'\n- 'Notice du catalogue : http://catalogue.bnf.fr/ark:/12148/cb424513623'\nreadonly_type_tesim:\n- Partition\nreadonly_attribution_tesim:\n- Bibliothèque nationale de France\nreadonly_description_tesim:\n- \"[Ah ! perfido, op. 65 : fragment] (manuscrit autographe) / Ludwig van Beethoven]\"\nreadonly_license_tesim:\n- https://gallica.bnf.fr/html/und/conditions-dutilisation-des-contenus-de-gallica\n",
 "created_at"=>2018-07-26 16:22:32 UTC,
 "updated_at"=>2018-12-21 17:02:17 UTC,
 "document_id"=>"12f18e4d7d59a1a0178deca3e295f1f6",
 "document_type"=>"SolrDocument",
 "resource_id"=>25244,
 "resource_type"=>nil,
 "index_status"=>nil,
 "tag_list"=>nil}

It looks like we're not properly handling this sort of IIIF manifest: https://gallica.bnf.fr/iiif/ark:/12148/btv1b550020492/manifest.json where metadata.value is an Object not a String

This seems like a spotlight bug with this particular IIIF manifest: projectblacklight/spotlight#2985

[caaster]

Once I realized that a IIIF resource was in this test exhibit, this morning I reindexed a couple of additional test exhibits and had 100% success. So this reindexing problem appears to be isolated to IIIF only. We have IIIF resources in only a few exhibits (~3-4?), so this makes this reindexing problem less impactful -- especially if we discover it is perhaps isolated to this single IIIF resource?

[caaster]

@jcoyne reports: I think this may just have been old data in the database. I cleared out that sidecar and re-indexed it from the manifest and I think its’ working now.