Whitespace Analyzer about laravel-scout-elasticsearch

[matchish]

Hey!

You should configure mappings https://www.elastic.co/guide/en/elasticsearch/reference/current/search-analyzer.html

[matchish]

If you don't sure how to do it properly, I recommend you open a question on StackOverflow. To get more views on the question, the question should be related only to ElasticSearch and not to the package.

Then you can config the package with proper mappings https://github.com/matchish/laravel-scout-elasticsearch#index-settings-and-mappings

[matchish]

Feel free to post a link to StackOverflow question here

[jmverges]

Hey @matchish !

Thanks for your quick response.

I'm not an ElasticSearch expert but as far as I understood the solution you are giving me is for altering the way properties are indexed into the index. I would say what I'm looking for is how they are searched once the are indexed. However my point of view can be wrong.

The problem I'm experimenting is with characters like t - shirt (0 results) and L / S (broken query)

As I understood here https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-whitespace-analyzer.html that is a way to configure it for all the properties, am I wrong?

We are having the usual multifacets with count filter to avoid 0 results which works like a charm except for this little detail.

So I suppose my first StackOverflow question could be: There is a way to change the default analyser for the ElasticSearch instance / an all the indexes?

I would expect something like:

"settings": {
        "analysis": {
            "analyzer": 'whitespace'
        }
}

that would become in the config.php

return [
    'host' => env('ELASTICSEARCH_HOST'),
    'indices' => [
        'settings' => [
            'default' => [
                'analysis' => [
                    'analyzer' => 'whitespace'
                ]
            ],
        ],
    ],
];

If this is not the right path to follow and I should change the way that is indexed each property in each index, I suspect I would need to look for each property that is string based and then doing something like

return [
    'host' => env('ELASTICSEARCH_HOST'),
    'indices' => [
        'mappings' => [
                'products' => [
                    'properties' => [
                        'category_name' => [
                            'type' => 'keyword',
                            'analyzer'        => 'whitespace',
                            'search_analyzer' => 'whitespace'
                        ],
                    ],
                ],
        ],
        'settings' => [
            'default' => [
                'number_of_shards' => 1,
                'number_of_replicas' => 0,
            ],
        ],
    ],
];

Didn't try for now nor one or the other but I will try all the examples as soon as I'm at my computer. In the meanwhile, if you would like to point me to the right direction, that would be great help!

[matchish]

You can see here an example of how to apply an analyzer to one field. You even can apply different analyzers to the same field.

[jmverges]

@matchish thanks. However I don't see how to fit it here via config/elasticsearch.php neither with ongr/elasticsearch-dsl in search time.

[matchish]

What is your plain elasticsearch query?

[jmverges]

{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "*t-shirt*"
          }
        },
        {
          "term": {
            "active": 1
          }
        },
        {
          "term": {
            "show_on_website": 1
          }
        }
      ]
    }
  },
  "aggregations": {
    "all_products": {
      "global": {},
      "aggregations": {
        "category": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "active": 1
                  }
                },
                {
                  "term": {
                    "show_on_website": 1
                  }
                },
                {
                  "query_string": {
                    "query": "*t-shirt*"
                  }
                }
              ]
            }
          },
          "aggregations": {
            "filtered_categories": {
              "terms": {
                "field": "category",
                "size": 100
              }
            }
          }
        },
        "brand": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "active": 1
                  }
                },
                {
                  "term": {
                    "show_on_website": 1
                  }
                },
                {
                  "query_string": {
                    "query": "*t-shirt*"
                  }
                }
              ]
            }
          },
          "aggregations": {
            "filtered_brands": {
              "terms": {
                "field": "brand",
                "size": 100
              }
            }
          }
        },
        "sizes_categories": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "active": 1
                  }
                },
                {
                  "term": {
                    "show_on_website": 1
                  }
                },
                {
                  "query_string": {
                    "query": "*t-shirt*"
                  }
                }
              ]
            }
          },
          "aggregations": {
            "filtered_sizes_categories": {
              "terms": {
                "field": "sizes_categories",
                "size": 100
              }
            }
          }
        },
        "color_families": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "active": 1
                  }
                },
                {
                  "term": {
                    "show_on_website": 1
                  }
                },
                {
                  "query_string": {
                    "query": "*t-shirt*"
                  }
                }
              ]
            }
          },
          "aggregations": {
            "filtered_color_families": {
              "terms": {
                "field": "color_families",
                "size": 100
              }
            }
          }
        },
        "services": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "active": 1
                  }
                },
                {
                  "term": {
                    "show_on_website": 1
                  }
                },
                {
                  "query_string": {
                    "query": "*t-shirt*"
                  }
                }
              ]
            }
          },
          "aggregations": {
            "filtered_services": {
              "terms": {
                "field": "services",
                "size": 100
              }
            }
          }
        }
      }
    }
  },
  "sort": [
    {
      "price": {
        "order": "desc"
      }
    },
    {
      "name.keyword": {
        "order": "desc"
      }
    }
  ],
  "from": 0,
  "size": 25
}

[matchish]

You can add parameter analyzer to query_string using ongr package

[jmverges]

thanks, I was able to add analyzer to query string reseting the $body paramenter in the callback and redoing the query like:

$query = Product::search($keywords, function($client, Search $body) use ($keywords) {
$body = new Search(); //Reset of original query and add it again with whitespace analyzer
$queryString = new QueryStringQuery($keywords);
$queryString->addParameter('analyzer', 'whitespace');
$body->addQuery($queryString);

I though it would be enough, but I was still getting 0 results for 't-shirt' so I approached using mappings like this

<?php

return [
    'host'    => env('ELASTICSEARCH_HOST'),
    'indices' => [
        'mappings' => [
            'products'       => [
                'properties' => [
                    'name'                => [
                        'type'            => 'text',
                        'analyzer'        => 'edge_ngram_analyzer',
                        'search_analyzer' => 'edge_ngram_analyzer',
                    ],
                    'style'               => [
                        'type'            => 'text',
                        'analyzer'        => 'edge_ngram_analyzer',
                        'search_analyzer' => 'edge_ngram_analyzer',
                    ],
                    'category_name'       => [
                        'type'            => 'text',
                        'analyzer'        => 'edge_ngram_analyzer',
                        'search_analyzer' => 'edge_ngram_analyzer',
                    ],
                    'brand_name'          => [
                        'type'            => 'text',
                        'analyzer'        => 'edge_ngram_analyzer',
                        'search_analyzer' => 'edge_ngram_analyzer',
                    ],
                    'color_families_name' => [
                        'type'            => 'text',
                        'analyzer'        => 'edge_ngram_analyzer',
                        'search_analyzer' => 'edge_ngram_analyzer',
                    ],
                    'tags'                => [
                        'type'            => 'text',
                        'analyzer'        => 'edge_ngram_analyzer',
                        'search_analyzer' => 'edge_ngram_analyzer',
                    ]
                ],
            ],

        ],
        'settings' => [
            'default' => [
                'number_of_shards'   => 1,
                'number_of_replicas' => 0,
                'analysis'           => [
                    'analyzer'  => [
                        'edge_ngram_analyzer' => [
                            'tokenizer' => 'custom_tokenizer',
                            'filter'    => [
                                'lowercase'
                            ]
                        ]
                    ],
                    'tokenizer' => [
                        'custom_tokenizer' => [
                            'type'        => 'edge_ngram',
                            'min_gram'    => 3,
                            'max_gram'    => 18,
                            'token_chars' => [
                                'letter',
                                'digit',
                                'punctuation',
                            ]
                        ]
                    ]
                ]

            ],
        ],
    ],
];

And don't using $queryString->addParameter('analyzer', 'whitespace');

I put it here, in case someone came here looking for the solution to a similar problem