Write a search class to fetch library website results about sul-bento-app

[drh-stanford]

Some notes on possible alternatives to searching the Solr index directly:

• There is a Drupal Search API, but it requires (significant) configuration so the library website admins likely haven't set it up.

• The HTML they return is structured with classes, so perhaps we could use the regular search interface and extract the title, description, and url from the HTML. For example, there's ol.search-results container and div.views-field-title and div.views-field-body-value for the result. They do not, however, have facet support or return the total number of results AFAICT, so this approach might be moot.

<ol class="search-results">

...

<li class="views-row views-row-2 views-row-even">  
  <div class="views-field views-field-title">        
    <span class="field-content"><a href="/blogs/digital-library-blog/2016/02/inaugural-geo4libcamp-forges-new-collaborations">Inaugural Geo4LibCamp forges new collaborations</a></span>  </div>  
  <div class="views-field views-field-body-value">        
    <span class="field-content">From January 25th to 29th, we hosted Geo4LibCamp 2016 at the Hartley Conference Center and Branner Library. Inspired by the success of LDCX 2015,...</span>  </div>

<div class="result-breadcrumbs">
  <span class="raquo"> » </span> <a href="/">Home</a> › <a href="/about">About</a> › <a href="/blogs">Blogs</a> › <a href="http://library.stanford.edu/blogs/digital-library-blog">Digital Library Blog</a></div>
</li>

...
</ol>

[drh-stanford]

@jkeck I looked in https://github.com/sul-dlss/sul-solr-configs but didn't see a core that might be connected to the library website. Do you know where the core is?

[jkeck]

@drh-stanford see: sul-cms-*

[drh-stanford]

Looks like the relevant fields are:

• tm_title
• ss_url (or ss_search_api_url)
• tm_body:value (yes with a colon)
• tm_search_api_viewed (the text snippet untokenized)
• label (unformatted title of node, but not present)
• spell (full text index which works)

I can issue a search for tm_title and it works fine, but searching for 'tm_body:value':* doesn't find anything. That body field is supposed to be the full text of the document. The default search handler doesn't appear to have a default field that it searches, so we might be able to look at the Solr logs to see exactly how it's passing queries to Solr. Or, I might be missing something in the query syntax...

See https://www.drupal.org/project/search_api_solr

[drh-stanford]

Also see https://www.drupal.org/docs/7/modules/search-api/developer-documentation/basic-architecture-and-component-interaction

[drh-stanford]

From https://www.drupal.org/node/1999386

Advanced » Retrieve result data from Solr
This option allows you to retrieve the result data (the fields of the result items) directly from Solr instead of loading it from the database. This might have advantages regarding performance (though benchmarks indicate this isn't usually the case) or security (as access checks will always be made against the data that will also be shown).

However, due to technical restrictions this won't work for functionality which needs the complete entity to work. This includes showing Field API fields in Views, rendering an entity in a certain view mode (with Views or Pages), linking a displayed field in Views to its entity (you can use a custom link rewrite with the item ID to circumvent this) or using the "Additional access checks on result entities" option in Views.

So, if we do need to scrape the HTML from the library search form interaction directly, then we can use CSS classes like .search-results, .view-field-title, .view-field-body-value, .result-breadcrumbs, etc.

[drh-stanford]

Requirements

We need to be able to issue a simple search (q=my+query) against the library website search interface, and get the following results for the top 3 relevant documents:

• Title of document
• URL to document
• Description (brief snippet is ok)
• Breadcrumbs (e.g., Home › About › Blogs › Digital Library Blog)
• Facet values and counts for resource type (e.g., Blog (4), Person (2))
• Total number of hits

Approach

1. We initially thought we could implement this by querying the Drupal Solr index directly (e.g., sul_cms_prod in the sul-solr cluster). But the index appears to lack the data requirements above -- see this comment above.

2. Our next approach would be to find a JSON API that we could query the Drupal server with that will return the required data in a structured format. Something like http://library.stanford.edu/search/website?search=hardy&format=json (which doesn't work), for example. Perhaps there's a Drupal plugin that provides a structured data API for search?

3. Finally, our last resort approach is to "screen scrape" the search from the user-interface version of the Drupal server (e.g., http://library.stanford.edu/search/website?search=hardy).

[drh-stanford]

Here's an example of a screen scrape for the hits (from WIP branch 3-scraper):

    def results
      doc.css('.search-results .views-row').collect do |hit|
        result = AbstractSearchService::Result.new
        result.title = hit.at_css('.views-field-title .field-content a').text
        result.link = hit.at_css('.views-field-title .field-content a')['href']
        result.description = hit.at_css('.views-field-body-value .field-content').text
        result.breadcrumbs = hit.at_css('.result-breadcrumbs span').text
        result
      end
    end

[tbfisher]

(2) is pretty straightforward, there is a module for building such endpoints (https://www.drupal.org/project/services) but it is probably overkill for this, since we could write up a fairly simple module to execute the search via solr and return the desired data.