saaay71 / solr-vector-scoring

Vector Plugin for Solr: calculate dot product / cosine similarity on documents

Vector Scoring Plugin for Solr : Dot Product and Cosine Similarity

With this plugin you can query documents with vectors and score them based on dot product or cosine similarity. This plugin is the same as Vector Scoring Plugin for Elasticsearch.

Plugin installation

The plugin was developed and tested on Solr 6.6.0.

1. Copy VectorPlugin.jar to {solr.install.dir}/dist/plugins/
2. Add the library to solrconfig.xml file:

<lib dir="${solr.install.dir:../../../..}/dist/plugins/" regex=".*\.jar" />

3. Add the plugin Query parser to solrconfig.xml:

<queryParser name="vp" class="com.github.saaay71.solr.VectorQParserPlugin" />

4. Add the fieldType VectorField to schema file(managed-schema):

  <fieldType name="VectorField" class="solr.TextField" indexed="true" termOffsets="true" stored="true" termPayloads="true" termPositions="true" termVectors="true" storeOffsetsWithPositions="true">
    <analyzer>
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.DelimitedPayloadTokenFilterFactory" encoder="float"/>
    </analyzer>
  </fieldType>

5. Add the field vector to schema file:

<field name="vector" type="VectorField" indexed="true" termOffsets="true" stored="true" termPositions="true" termVectors="true" multiValued="true"/>

6. Start Solr!

Example

Add example documents

curl -X POST -H "Content-Type: application/json" http://localhost:8983/solr/{your-collection-name}/update?commit=true  --data-binary '
[
    {"name":"example 0", "vector":"0|1.55 1|3.53 2|2.3 3|0.7 4|3.44 5|2.33 "},
    {"name":"example 1", "vector":"0|3.54 1|0.4 2|4.16 3|4.88 4|4.28 5|4.25 "},
    {"name":"example 2", "vector":"0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "},
    {"name":"example 3", "vector":"0|0.06 1|4.73 2|0.29 3|1.27 4|0.69 5|3.9 "},
    {"name":"example 4", "vector":"0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "},
    {"name":"example 5", "vector":"0|0.64 1|3.95 2|1.03 3|1.65 4|0.99 5|0.09 "}
]'

Query documents

Open your browser and copy the links

Query 1

http://localhost:8983/solr/{your-collection-name}/query?fl=name,score,vector&q={!vp f=vector vector="0.1,4.75,0.3,1.2,0.7,4.0"}

You should see the following result:

{
  "responseHeader":{
    "status":0,
    "QTime":1,
    "params":{
      "q":"{!myqp f=vector vector=\"0.1,4.75,0.3,1.2,0.7,4.0\"}",
      "fl":"name,score,vector"}},
  "response":{"numFound":6,"start":0,"maxScore":0.99984086,"docs":[
      {
        "name":["example 3"],
        "vector":["0|0.06 1|4.73 2|0.29 3|1.27 4|0.69 5|3.9 "],
        "score":0.99984086},
      {
        "name":["example 0"],
        "vector":["0|1.55 1|3.53 2|2.3 3|0.7 4|3.44 5|2.33 "],
        "score":0.7693964},
      {
        "name":["example 5"],
        "vector":["0|0.64 1|3.95 2|1.03 3|1.65 4|0.99 5|0.09 "],
        "score":0.76322395},
      {
        "name":["example 4"],
        "vector":["0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "],
        "score":0.5328145},
      {
        "name":["example 1"],
        "vector":["0|3.54 1|0.4 2|4.16 3|4.88 4|4.28 5|4.25 "],
        "score":0.48513117},
      {
        "name":["example 2"],
        "vector":["0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "],
        "score":0.44909418}]
  }}

Query 2

Adding the parameter cosine=false calculates the dot product

http://localhost:8983/solr/{your-collection-name}/query?fl=name,score,vector&q={!vp f=vector vector="0.1,4.75,0.3,1.2,0.7,4.0" cosine=false}

result of query 2:

{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"{!myqp f=vector vector=\"0.1,4.75,0.3,1.2,0.7,4.0\" cosine=false}",
      "fl":"name,score,vector"}},
  "response":{"numFound":6,"start":0,"maxScore":40.1675,"docs":[
      {
        "name":["example 3"],
        "vector":["0|0.06 1|4.73 2|0.29 3|1.27 4|0.69 5|3.9 "],
        "score":40.1675},
      {
        "name":["example 0"],
        "vector":["0|1.55 1|3.53 2|2.3 3|0.7 4|3.44 5|2.33 "],
        "score":30.180502},
      {
        "name":["example 1"],
        "vector":["0|3.54 1|0.4 2|4.16 3|4.88 4|4.28 5|4.25 "],
        "score":29.354},
      {
        "name":["example"],
        "vector":["0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "],
        "score":24.923502},
      {
        "name":["example"],
        "vector":["0|0.64 1|3.95 2|1.03 3|1.65 4|0.99 5|0.09 "],
        "score":22.1685},
      {
        "name":["example"],
        "vector":["0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "],
        "score":11.867001}]
  }}

Query 3

Quering on other fields and with vector scoring.

http://localhost:8983/solr/{your-collection-name}/query?fl=name,score,vector&q={!vp f=vector vector="0.1,4.75,0.3,1.2,0.7,4.0" cosine=false}name="example 2","example 4"

result of query 3:

{
  "responseHeader":{
    "status":0,
    "QTime":1,
    "params":{
      "q":"{!myqp f=vector vector=\"0.1,4.75,0.3,1.2,0.7,4.0\" cosine=false}name=\"example 2\",\"example 4\"",
      "fl":"name,score,vector"}},
  "response":{"numFound":2,"start":0,"maxScore":24.923502,"docs":[
      {
        "name":["example 4"],
        "vector":["0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "],
        "score":24.923502},
      {
        "name":["example 2"],
        "vector":["0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "],
        "score":11.867001}]
  }}

solr-vector-scoring's Issues

How to handle queries containing large vector embed.

Hi,

In the given example you took embedding of size 5.. in case of solr 8.5, vector embed of length 512 or greater when we send request for querying. it gives error 414.. url too large... how this issue can be resolved any idea about it?

Thanks

Solr 7 createWeight Exception

When following your installation procedure and executing your Examples, I ran into the following error with Query 1:

java.lang.UnsupportedOperationException: Query {! type=vp f=vector vector=0.1,4.75,0.3,1.2,0.7,4.0 v=} does not implement createWeight

Integrating with solr 8.5

Any thoughts on how to get this running on solr8.5? CustomScoreQuery and FunctionScoreQuery both seem to be unavailable in solr8.5.

NullPointerException while querying

I am facing NullPointerException at https://github.com/saaay71/solr-vector-scoring/blob/master/src/com/github/saaay71/solr/VectorScoreQuery.java#L48 line number.

The term vector is null and when checking the size it throws NullPointerException. I tried to debug this issue but unable to find what is the cause of terms vectors being null.

It may be possible that for all the documents in solr vectors are not available and for those documents the vector field will not even be indexed and hence no term vector.

There should be a null check and if term vectors are null score should be 0.

Solr Search is very slow with vector-scoring. Takes around 50 seconds.

After implementing custom scorer, search became too slow.
My index has around 4000000 docs. Querying takes around 50 seconds which is too slow.
Any solution on how to make in faster and search in ms ?

Don't use term vectors

Lucene Term Vectors are a bit heavy to use in the way this plugin does. And why encode/decode the vector ordinal numbers as terms at all? Instead I propose as follows:

Add a new special text field that has payloads enabled. No Term vectors. This field will only ever index one nominal term, say the empty-string or one letter 'X' -- it doesn't matter. Each vector ordinal 0 thru 5 or however long it is becomes a term position of this term for a document. The payload encodes the number -- a 4-byte float. The home page of this plugin shows the numbers as dense but this approach (and the term vec one) could easily be sparse. This would be somewhat slower than a custom BinaryDocValues (another implementation path) but it leverages Lucene more and is less custom, for whatever benefit that is (e.g. easier debug-ability).

Ideally a FieldType would be added which could be used to enclose the implementation details of analysis, and it could even be used to query without the addition of any other top level classes / plugins, since a FieldType works with most query parsers, including the default/standard/lucene one and you can do some neat things this way. e.g. q=vecField:"0.1,4.75,0.3,1.2,0.7,4.0" (taken from the example)