Uploaded images not displaying in several widgets about exhibits

[hudajkhan]

@caaster, @cbeer finally tracked down what the problem appeared to be. I'll explain further in comments below. In the meantime, you should be able to see the images populate the widgets (carousels, item grids, etc.) properly if you edit the page, remove the image you had added previously and then lookup that image again in the autocomplete selection and add it again. If this does not work, please let us know.

[hudajkhan]

An explanation of what was happening:

a) The cause: The uploads directory on the exhibits-worker boxes (where Sidekiq indexing jobs run) was not linked to the right directory location. It should have pointed to /data/uploads but it did not. We updated the directory link. @cbeer reindexed the items so that their Solr documents are now updated with the correct information required for the images to display in the embedded widgets.

b) Background: When one first uploads an item, the code adds certain information to the Solr index. There are background jobs that are designated to do so in sidekiq. Eventually, these jobs will call the "to_solr" method on the resource upload class (https://github.com/projectblacklight/spotlight/blob/v3.5.0.2/app/models/spotlight/resources/upload.rb#L43). If the file for the upload is recognized as existing, the code adds the "iiif_manifest_url_ssi" field to the Solr document.

When one then goes to edit a page and add this uploaded item to a SirTrevor widget, the code will use the information in the Solr document to generate what will be saved for the widget in the database. The page generates a serialized string that will be stored as is to the Spotlighted Featured Pages tables. If all works correctly, this content will include a field entitled "iiif_tilesource".

When one then goes to view the page they just edited, the information stored in the table is sent along to the widgets. (https://github.com/projectblacklight/spotlight/blob/v3.5.0.2/app/views/spotlight/sir_trevor/blocks/_solr_documents_carousel_block.html.erb#L13) shows the code for the carousel checking to see if the "iiif_tilesource" field is present. This value is then used to generate the image tag source for that item in the carousel.

c) What happened as a result of the uploads directory not being linked to the correct location: On the web app server, the uploads were saved in the location recognized by the app. In the case of the exhibits workers boxes, which are separate and different from the servers for the web application, the images were uploaded to the wrong location. When the indexing jobs ran, they encountered this line (https://github.com/projectblacklight/spotlight/blob/main/app/models/spotlight/resources/upload.rb#L44) which thought the file was not present and returned an empty struct for the to_solr method. Because of this empty value, the Solr document for the uploaded image was not updated with the iiif_manifest value. Because of this, iiif_tilesource was not populated correctly. The content returned to the SirTrevor widgets had the string "undefined" for the iiif_tilesource field. This string was used to generate the image tag, which resulted in the image not being visible.

d) There are other steps in the middle which I would still like to review, but the above captures some of what is occurring behind the scenes.

[hudajkhan]

A little more information:
When editing the page and selecting an image to include in, for example, a carousel, the following JavaScript code comes into effect:
a) https://github.com/projectblacklight/spotlight/blob/v3.5.0.2/app/assets/javascripts/spotlight/admin/blocks/solr_documents_base_block.js#L91 tries to retrieve the array of images to be used for selection.
b) That method calls the imagesArray method in the iiif class here https://github.com/projectblacklight/spotlight/blob/v3.5.0.2/app/assets/javascripts/spotlight/admin/iiif.es6#L51 .
c) The iiif code generates various values for fields, including tilesource, and sends them back in the images array returned to the SirTrevor code: https://github.com/projectblacklight/spotlight/blob/v3.5.0.2/app/assets/javascripts/spotlight/admin/iiif.es6#L40 .
d) When an item is selected, all this information is used to set the values of hidden inputs on the page which will then be used to generate the content for the SirTrevor widget that will be stored in the database and retrieved for display later: https://github.com/projectblacklight/spotlight/blob/v3.5.0.2/app/assets/javascripts/spotlight/admin/blocks/solr_documents_base_block.js#L70

[caaster]

The fix is working, thank you so much to all. @hudajkhan your documentations/explanation are particularly valuable.

An important note, when I went to my test exhibit page to verify that the fix was working, here is what happened:

• For the first widget on the page, the item carousel widget, I first removed both images in the widget (edit mode)
• Then I hit SAVE
• Then I re-entered page edit mode, and added the 2 same images back to the widget
• Then I hit SAVE
• The images still appeared as broken in view mode
• Then I re-entered page edit mode, and deleted the item carousel widget from the page
• Then I hit SAVE
• Then I re-entered page edit mode, and added the item carousel widget back, and added the 2 same images to the widget
• Then I hit SAVE
• Voila! Now the images display in the item carousel widget in page view mode

[jcoyne]

@caaster are you able to duplicate this on the stage servers (https://exhibits-stage.stanford.edu/)? It would be a great help. Then we can make some code changes without risk to the production site.

[caaster]

@jcoyne here you go: https://exhibits-stage.stanford.edu/cathy-test/feature/uploaded-image-test-page

[jcoyne]

Thank you @caaster. Do you know of any other exhibits where uploaded files have been used successfully in these contexts? I'm trying to compare this broken cases with what it looks like in a working case.

[caaster]

@jcoyne there are many exhibits that use uploaded images -- do you need me to root around for a lot of them? This would take me a lot of time potentially. I can tell you that this exhibit has ALL uploaded images: https://exhibits.stanford.edu/panofsky-papers

[caaster]

@jcoyne here is an uploaded image that uses the item row widget (kid on the globe): https://exhibits.stanford.edu/maps101

[caaster]

@jcoyne I could go on & on but here is another: https://exhibits.stanford.edu/remapping. I think the biggest challenge would be if you require an example from a different, published exhibit for each widget that is not working for me & Kristen Valenti...

[jcoyne]

@caaster no, that's plenty. We're trying to figure out approximately when this stopped working, so we can look at the dates on those images to say "Must have worked before X"

[caaster]

Fabulous! Thank you @hudajkhan @cbeer. I'll test it on my test site first to re-validate before sharing the news with Kristen. I'll give a shout if anything seems amiss.