wellcomecollection / catalogue-pipeline Goto Github PK

Ref #817

On the bib records:

In order to save on running costs and enable opportunities for handling logs more effectively we want to log to an ELK stack.

There is no logstash service running in the Catalogue API cluster and consequently the API cannot log to elasticsearch through it.

Currently IDs can carry weird phrases such as External Reference from Nuffield department of anaethestics, Oxford on https://wellcomecollection.org/works/hkqtbg3k.

The current way we index and query means that query=Oxford matches, and shouldn't.

Index as lowercase keywords will solve most of these, but might eradicate the ability to search for multiple IDs and IDs as part of queries, but that might not be an issue.

• Add ascii folding to concepts
• Boost contributors to more than title, but title above other concepts

Known errors:

Yokai

Terms
Yōkai works
Yokai does not work

Result
Pandemonium and parade [electronic resource] : Japanese monsters and the culture of Yōkai

https://wellcomecollection.org/works/x2zbjrgp

## Two monsoons

Result is #15 which seems way too far down.

Terms
Two monsoons, Theon Wilkinson

• Result
  https://wellcomecollection.org/works/hgcbf8q9

(from standup) - search problems for known (or approximately known) titles

1. My known item was this - https://wellcomecollection.org/works/km3uczhf
   My query was the transformations of insects
   Only one result that looked even vaguely relevant on p1. This book appears halfway through p2.
2. My known item was this - https://wellcomecollection.org/works/ysmqsfhg
   My query was the biological basis of medicine (which in this case is its exact title)
   This result appears towards the end of all the results on page 3
   There are many works that appear in both sets of results. I wondered if that's because both queries are the xxx of yyy but the result titles don't support that. (edited)
   11:24
   (why was I searching... to find the work IDs to use in this comment - wellcomecollection/platform#4603 (comment)) (edited)

wellcomecollection/platform#4603 (comment)

Researchers can search for IDs in a variety of different way.

Ref #898

After a deployment to the stage API, we should have some tests run against

• /works
• /works?query={query}
• /works/{id}
• /images
• /images?query={query}
• /images/{id}

On the front end, the following paths should be checked:

• /visit-us
• /whats-on
• /stories
• /collections
• /what-we-do

To make sure we get 200s.

• Do we know that know item searching is better and we can move on?
• Analysis / rank_eval

Searches that we would expect the first result to be a known item, but isn't.

query: twenty one things
expected: https://wellcomecollection.org/works/fjbxucnf

query: maurice wilkins archive
expected: https://wellcomecollection.org/works/rq5g8r9g
when searching phrase maurice wilkins archive, I get the item and series-level stuff as expected, but no obvious appearance of the collection-level on first page of results; when filtering down to Archive collection, it shows me the Francis Crick collection (@jennpb)

Could we surface example queries of the different intentions we are aware of.

As a way for users to find and explore collections, we want to add top level collections to the search results.

We can do this quite easily by just adding archive-collection to the list of current workTypes we allow through.

This would add 1173 results to the list, which is insignificant in terms of affecting the tf/idf.

We will be adding and working via the rank_eval tests to ensure they don't skew the current query.

The main issue however that we also have the filter of items.locations.locationType=iiif-image,iiif-presentation which none of the archive-collections have, so they will not be returned.

Ping @harrisonpim @alicerichmond @taceybadgerbrook

We were querying on the wrong fields, this made more of the tiers work as intended.

#338

https://github.com/wellcomecollection/catalogue/pull/521/files#r409647521

i think we can still improve the way we're scoring contributors - we should be looking for the best matches within each contributor in the list, not the overall match across the list of contributors

eg. when searching for william blake,
{id: "12345", "contributors": ["william blake", "someone else"]}
should be scored much higher than
{id: "67890", "contributors": ["william someone", "blake else"]}

i'm seeing instances of the second example being given the same score as the first

See this PR for an example:
#419

Shake your body

?items.locations.type=OpenShelves,ClosedStores,DigitalResource

Our two prod image inferrers (features, palettes) share loads of functionality, and therefore loads of code. A quick look suggests that ~75% of their code is shared, and we're now starting to duplicate even more in the concepts store.

It makes sense to pull that shared functionality out into a common library, which could then be installed with pypi.

We currently store Sierra location codes in the locationType of a Location.

This isn't useful, and is potentially a data leak into where and how we store rare materials in our closed stores.

We could split these into ClosedStores and OpenShelves and map those from the locations codes from Sierra.

Here's an example of some before and afters.

It has been said that you can infer this from the prefix on the Sierra location code being wg meaning it is on the open shelves.

We should leave the DigitalLocations as is (IIIFPresentation and IIIFImage) for backwards compatibility.

Ref #898

Write a script that can compare the deployment:label tag of a service against the tasks running within that service to determine that a service has successfully deployed.

This can then be added to the release to stage step of the build as a "completed" check.

• audit

metrics

• searches
• sessions
• click through
• click position
• no results
• unique sessions with clicks / unique sessions

filters

• works/images
• page of results
• number of tokens in query

https://wellcome.slack.com/archives/C016NQB58N4/p1597935859065100

Currently different members across the team have different smoke tests they run before deploying to prod.

One set of tests is running the experience app against the staging API. It would be good to have these Dockerised and accessible to the whole team to help with a smoother, more confident API deployment.

This will add confidence to our releases as we add and transform a whole load of new data into the API.

This will then be available to CI where we're starting to deploy the API via CI to stage 🥳 .

After we have run some post-stage tests, we will be closer to be full Agile™ and continuously deploy to prod.

Add location filters to the API.

Locations

We're removing the current locationType(iiif-presentation, iiif-image) and will add filtering on the above locations via the items.locations.type query parameter. e.g. ?items.locations.type=OpenShelves,ClosedStores.

These will be aggregated so available to the frontend with the above labels and counts of how many are in each aggregation the same way we do with workType. e.g.

Note: I've just nabbed the interface with what is live, it will follow whatever filter pattern we have at the time of implementation.

Ref #898

weco-deploy needs to tag the service with deployment:label: deploymentID on the deployed service.

This will then allow us to check tasks against this ID to determine when a service has finished being deployed.

following conversation with @jamesgorrie about adding fuzziness to queries. While this is a clear intention, it should really be applied to all queries, rather than adding a new one to the tiers. Things like this should sit in a set of principles, eg

Principles

• we should try to match all tokens in a query
• all tokens should have fuzziness applied to them
• etc