algolia / scout-extended

Scout Extended: The Full Power of Algolia in Laravel

Documentation • Community Forum • Stack Overflow • Report a bug • FAQ • Support

To dig right in, visit the Scout Extended documentation.

Scout Extended provides the perfect starting point to integrate Algolia into your Laravel application. It is carefully crafted to simplify the usage of Algolia within the Laravel Framework.

It’s built on top of the latest release of Laravel Scout, the official Laravel search package. Using Scout Extended, you are free to take advantage of all of Laravel Scout’s great features, and at the same time leverage the complete set of Algolia’s search experience.

✨ Features

• Zero downtime reimports - it’s a breeze to import data in production.
• Implement site-wide search amongst multiple models with aggregators.
• Automatically improve the relevancy of your search results.
• Beautiful overview of your application indexes.
• A useful collection of macros, facades, and much more.

💡 Documentation

For full documentation, visit the Scout Extended documentation.

Note: This package adds functionalities to Laravel Scout, and for this reason, we encourage you to read the Scout documentation first. Documentation for Scout can be found on the Laravel website.

Because everyone should be able to build great search, you can use Algolia's basic Community Plan. It's free up to a certain number of records and operations.

Troubleshooting

Encountering an issue? Before reaching out to support, we recommend heading to our FAQ where you will find answers for the most common issues and gotchas with the package.

Use the Dockerfile

If you want to contribute to this project without installing all its dependencies, you can use our Docker image. Please check our dedicated guide to learn more.

🆓 License

Scout Extended is an open-sourced software licensed under the MIT license.

scout-extended's Issues

Searching for date time does not work

Hello everybody,

I got an record with this value:

publish_on:	2019-01-30 17:40:11

Unfortunately I can not search for this:

App\Posts::search('Title')->where('publish_on', '<', now())->get();

App\Posts::search('Title')->where('publish_on', '<', now()->toDateTimeString())->get();

does not work, too.

This result is always empty.

Question: Indices per ENV

Hi,

I was wondering if is possible to have different indices per environment and configure which index to use for a given environment via the .env file ?

Thank you.

Aggregates unable to rehyrdate models if scout key is set

I've not fully investigated the source of this yet, but it appears that if the scoutKey() feature is used, aggregates can't rehydrate themselves correctly. Mostly leaving this here as a reminder to come back to it.

Using multiple Algolia Accounts

Explain how to use multiple Algolia Accounts with a tutorial or add this feature on the code itself.

aggregators indexing fails if SCOUT_QUEUE is true

MakeSearchable job doesn't understand what to do with an Aggregator

Index configuration file per environment

Let's assume we have an articles index and on our local machine we've set SCOUT_PREFIX=dev_. This is to prevent screwing up the indexes for our production environment. On production, we might have set the prefix to prod_ or aren't using a prefix at all.

So to add our index configuration to version control we run php artisan scout:optimize and a scout-dev-articles.php configuration file will be generated.

However, this will not work for our production environment, which will effectively require a scout-prod-articles.php configuration file.

In response to: https://twitter.com/enunomaduro/status/1088459313964699655

How many models an aggregate can consist of ?

Hi,
I was wondering if there is a limit for the number of models an aggregate can consist of ?
Thanks.

Map model names

Would be nice if you could map model names similar to how morphMap works.
https://laravel.com/docs/5.7/eloquent-relationships#custom-polymorphic-types

Very helpful if you need to move a model or not wanting to expose the path.

Aggregate searchableAs equivalent

Is there are searchableAs equivalent for aggregates so we can separate local vs production like we do with models?

eg:
local_All
production_All

Where condition on multidimensional arrray or nested object in index

Is it possible to do a search with where condition on a model that has multidimensional array or nested object?

Here is an example of Student index's structure stored in Algolia:

{
    'name': 'John Doe',
    'score': [
        {'subject': 'math', 'score': '85'},
        {'subject': 'spanish', 'score': '75'},
        ...
    ],
}

From above example, I need to do search everyone who has score more than 75 in math.
Thanks!

Search behavior difference from Scout

I noticed another issue which would be a breaking change when switching from scout to scout-extended.

Scenario:

The items index has 10 records.

Issue:

With laravel/scout:

count(Item::search('')->get())   // === 10
count(Item::search(null)->get()) // === 10

With algolia/scout-extended:

count(Item::search('')->get()),  // === 10
count(Item::search(null)->get()) // === 0

This would be an issue because of Laravel's ConvertEmptyStringsToNull middleware. Like in my case, if someone is trying to do Item::search($request->query)->get(), they wouldn't get any results.

Index record does not exist

I get 'Index record does not exist' when using multiple models in one index (Aggregators). But searching directly with model works fine. Followed exactly as mention on the documentation with new laravel instance.

Aggregator Class

use Algolia\ScoutExtended\Searchable\Aggregator;
use App\Accommodation;
use App\Activity;
use App\TravelTip;

class Record extends Aggregator
{
    /**
     * The names of the models that should be aggregated.
     *
     * @var string[]
     */
    protected $models = [
        Accommodation::class,
        Activity::class,
        TravelTip::class,
    ];


}

AppServiceProvider

    public function boot()
    {
        Record::bootSearchable();
    }

In Controller

$records = \App\Search\Record::search()->raw();
dd($records);

Undefined property: Illuminate\Cache\Events\KeyWritten::$minutes

Hi,
I am using Laravel 5.8 and Scout-extended 1.6.

My Sentry reports shows that this line of code

api-key=""

Generates this error

Notice: Undefined property: Illuminate\Cache\Events\KeyWritten::$minutes

Have I miss something in the packages configuration ?

Thank you very much.

$model->searchable() does not update aggregates

I'm not sure if this is just a limitation of this package, but when calling searchable() or unsearchable() directly on a Model, any aggregators that the Model is part of do not get updated.

Not sure if there is an easy fix for this, but for now we just do the following...

$user = User::find(16);

$user->searchable();

\App\Search\UserAggregate::create($user)->searchable();

Collection::unsearchable() does not delete objects from my index.

Hi @nunomaduro

The Algolia\ScoutExtended\Jobs\DeleteJob uses deleteBy internally to make a Collection unsearchable.

$result = $index->deleteBy([
    'tagFilters' => $this->searchables->map(function ($searchable) {
        return ObjectIdEncrypter::encrypt($searchable);
    })->toArray(),
]);

However, based on the documentation, this will use the conjunction AND to find the items, whereas we are probably looking for OR.

I just gave this a try by sending a batch delete job and it couldn't delete anything because it can't find records that have these filters: App\Movie:1, App::Movie:2... which makes sense.

This is what I tried:

$movies = \App\Movie::get();
$movies->unsearchable();

Edit: I was able to fix the issue by using a nested array for Tag Filters. See this. I am not entirely sure what are the implications for doing this, to be honest. If you feel comfortable with this, I'd happily submit a PR for it.

where() method not filtering query

I'm trying to integrate algolia in a personal project. The table I'm indexing is the patterns table that has the user_id as FK and is included as one of the fileds I upload to algolia.

class SearchController extends Controller
{
    public function show()
    {
        $search = request('q');
        $patterns = Pattern::search($search)->where('user_id', '=', Auth::user()->id)->get();
        // dd(Auth::user()->id);
        return view('search.index', ['patterns' => $patterns]);
    }
}

If I dump the user_id I get 1. Intentionally I'm searching for something that doesn't belong to that user expecting to get nothing from algoia but I still get the record back. The record that I get back has the user_id 3.

I'm not sure if I'm missing some setting in the algolia dashboard or the code is missing something.

I'm searching for the word test and is the only record that contains that word.

Aggregate is casting numbers?

I have a model where we have a field, "line_id" the db type is string..
This looks like a number but it's not as we have e.g. "831", but we also have "062", the leading zero is important.

Our models toSearchableArray looks like this (Model is DDI):

public function toSearchableArray()
{
  $data = $this->toArray();
  $data['display_name'] = $data['name'];
  $data['model'] = 'ddi';
  $this->transform($data);

  return array_only($data, [
    'id',
    'line_id',
    'number',
    'name',
    'display_name',
    'model',
    'created_at',
    'updated_at'
  ]);
}

scout:import against the model works fine and in the index we see

We also have an aggregate defined which looks like this

class GlobalEntities extends Aggregator
{

  protected $models = [
    Client::class,
    DDI::class,
    Contact::class,
  ];
}

And that DDI that's imported looks like this in the search index

So why would the aggregate be casting line_id to a number hence dropping the leading zero and how can I fix it?

When there is no data a import, scout:reimport will not replace the destination index

Hello,

I have a model for which, on purpose I define the shouldBeSearchable method so that to always returns false. Despite this, the model is imported in my index when I run php artisan scout:reimport.

Please note that for the same model I overwritten the toSearchableArray method.

Thank you.

_getRankingInfo is not added to the models

Hi,

Thanks for the great work on the scout-extended package.

It seems like when passing 'getRankingInfo' => true using with(), the _rankingInfo is not added to the model. When using the regular api client, it is. Is this a bug?

Thanks!

Add compatibility with Laravel 5.8

• we may need to pass an date interval on every cache usage.
• update composer.json to support 5.8.* illuminate components.

No results returned when using with Laravel Nova

When using Scout Extended with Laravel Nova, I'm not getting any results searching on a model resource page for a specific model.

Using laravel/scout and algolia/algoliasearch-client-php:^1.27 per the Scout docs everything works out of the box.

With algolia/scout-extended I get nothing—everything else is the same, I just uninstalled the two packages above and installed Scout Extended instead.

I can see the queries hitting Algolia, and they look the same, the only difference I can see in the responses is that with scout and algoliasearch-client-php, Algolia stores the objectID property as the primary key of the model, something like "9". Scout Extended appears to store the objectID as the model name and primary key, like "App\Model::9".

Is this because of some change between algoliasearch-client-php 1.x and 2? Or is it possibly an issue with Nova?

Thanks!

Documentation

Hi,

On the following page
https://www.algolia.com/doc/framework-integration/laravel/searching/client-side-search/

I believe this code will not work

@{{ article.title }}

@{{ article.summary }}

You should probably use

@{{ result.title }}

@{{ result.summary }}

Cheers,

Refactoring needed in class `Algolia\ScoutExtended\Settings\LocalFactory::class`

The class Algolia\ScoutExtended\Settings\LocalFactory::class is responsable to detect settings out of Laravel models. We should consider a re-write this logic, and split it into multiple classes.

Invalid object Algolia\AlgoliaSearch\Exceptions\BadRequestException

Hi,

Each time I add this line in my blade file
api-key="{{ Algolia\ScoutExtended\Facades\Algolia::searchKey(App\MyModel::class) }}"

I get the following error
Algolia\AlgoliaSearch\Exceptions\BadRequestException

Invalid object

Thanks

Create a playground

This goal is perform a pull request on the https://github.com/algolia/api-clients-playground repository inside the laravel-scout-extended folder an Laravel Application with Scout Extended Installed on it.

People should be able to clone the project locally and start playing with Scout Extended easily.

Add method to delete remote index

Hello Algolia Team,

Overview
Currently there is a static method for making all Aggregated Models Searchable, but there isn't a method for making them all unsearchable.

Use Case
For unit tests with the Algolia scout driver, we create dummy records that are searchable. At the end of the test, we make the Model unsearchable. Unless there is a way, we are unable to make Aggregator unsearchable through a static method.

i.e. Aggregator:: makeAllUnsearchable(); or Aggregator::unsearchable();

Thank you!

Synchronize replicas's settings locally with scout:sync

We can define the index settings in its dedicated settings file, but there is no way to define replicas's settings. (at least not in documentation).

This following is example of Article Model setting which was generated by php artisan scout:optimize. The following settings are synced with remote configuration when I run php artisan scout:sync.

// conifg/scout-articles.php
<?php
return [
    'searchableAttributes' => [ 'title',  'content', ],
    'customRanking' => ['desc(published_at)'],
    'removeStopWords' => null,
    'disableTypoToleranceOnAttributes' => ['title', 'description'],
    'attributesForFaceting' => null,
    'unretrievableAttributes' => null,
    'ignorePlurals' => null,
    'queryLanguages' => null,
    'distinct' => null,
    'attributeForDistinct' => null,
    'replicas' => ['articles_asc', 'articles_desc'],
];

It would be nice to have similar settings for replicas locally.

@nunomaduro

Composer Install Error

When attempting to install I am getting an error with one of the required packages.

"laravel/scout dev-feat/upgrade-algolia-engine -> no matching package found"

Is there a work around for this? I am not seeing that branch on the laravel/scout repository.

Forwarding client IP address

We currently have a very simple aggregate keyword search that works something like this...

$result = KeywordSearch::search($this->request->get('query'))
    ->with([
        'filters' => $this->buildFilter(),
    ])
    ->paginate(80);

We noticed that in the Algolia statistics, it looks there there is only one user of the search, but it turns out this is because we need to forward the users IP address.

The API docs say that you should do something like the following to forward the IP address, but how does this fit with our code above?

$ip = '94.228.178.246';
$client->setExtraHeader('X-Forwarded-For', $ip);

$results = $index->search('query', [
    'analytics' => true
]);

Thanks

Data sorting dynamically by parameters send by user

Is it possible to do data sorting based on parameters being sent to Algolia?

For example, I have an e-commerce app that sell books. Sometimes I need to sort the data by popularity, sometimes by sales, sometimes by newest or oldest.

Can't find anything about this in the official Algolia docs.
Thanks!

Syncing models to multiple indexes

Ola!

Was hoping you could shed some light on the possibilities of syncing a model to multiple indexes?

Our example.

We have a marketing website which uses various indexes to provide results to the general public.

We also have an internal CRM which searches the same indexes so staff can find the same information.

HOWEVER

This creates a problem with the Algolia analytics - we can no longer see which searches are from the internal users and which are from the public.

I imagine this could be resolved by having a duplicate index, one which internal use, one which the public use - that way we can see the difference?

Does this make sense?

Thanks

Support for simplePaginate?

Hello,

Are there any plans to support simplePaginate?

Algolia\ScoutExtended\Builder::simplePaginate does not exist.

With the following:

$videos = Video::search(request('q'))->simplePaginate(12);

[Bug] scout:reimport moves index too soon, multiple race conditions

If you are using a queue to reimport into algolia that isn't FIFO eg AWS SQS, there is a very high chance of hitting a race condition where you move the temp index to the real index without pushing all the data into the new index. I've seen examples of 5-10% of the records missing.

Also depending on how long the reimport takes and how frequently you are updating records it seems like there is a high chance the reimport will import stale data by using this approach timeline eg:

1. start reimport
2. records gets pushed into new temp index
3. record gets updated and pushed into old index
4. temp index gets moved to main index.

You now have stale data in the newly moved index

Unsure is this is related but you seem to reference config('scout.synchronous', false) in the code but I can't seem to figure out where or how that is defined ?

PHP Code Quality Tool: PHPMD

PHPMD will display the possible bugs and misuses of the language in our package. We should require-dev it.

Goal: Study this tool and perform a Pull Request including this tool on require-dev and updating travis.yml if possible.

Current master branch sends '_highlightResult', '_rankingInfo' to Algolia in updateJob

This bug was introduced on #147. An UpdateJob will try to attach '_highlightResult', '_rankingInfo' to an record. Code to reproduce:

    $models = Article::search($query)->get();
    $models->searchable();

Goal: We need to think about attaching this information somewhere else.

Config sync is working wrong

Looks like config sync is working wrong.

For the config/scout-records.php like this:

'searchableAttributes' => [
    'ordered(make)',
    'title'
],

After syncing index settings with php artisan scout:sync I'm getting the following response in algolia backend search form - https://nimb.ws/ub1wrT

So in this case "make" field is not searchable, which I believe is wrong.

---

For the config/scout-records.php like this:

'searchableAttributes' => [
    'make',
    'title'
],

After syncing index settings with php artisan scout:sync I'm getting the following response in algolia backend search form - https://nimb.ws/oHhZQs

In this case "make" field is searchable.

In both cases "make" in "Searchable Attributes" section appears as "ordered".

Can't make it work - Throwing ErrorException

Versions:
Laravel Framework: 5.7.20
PHP: 7.2.9
Scout-extended: 1.1

I had a laravel/scout working fine but decided to upgrade and use scout-extended.

I followed the instructions and wasnt able to use it nor install it, it said there was an error because I already had laravel/scout installed and it used algolia/algoliasearch-client-php:^1.27 and scout-extended needed ^2.0.0.

I uninstalled laravel/scout and algolia/algoliasearch-client-php:^1.27, then installed algolia/scout-extended (which installed the required dependecies), then published the configuration file (which is the same that I had) and added the use Laravel\Scout\Searchable; to the models required.

Now everything should be working fine, but when I run php artisan scout:reimport I get the following error

ErrorException: include(/home/vagrant/code/mysite/vendor/composer/../algolia/algoliasearch-client-php/src/AlgoliaSearch/Version.php): failed to open stream: No such file or directory in /home/vagrant/code/mysite/vendor/composer/ClassLoader.php:444

Could you help me get this working? Thanks!

Add support to Lumen

Scout Extended was never tested on Lumen - The Micro Framework by Laravel. This may be a first good issue. List of tasks:

• Install Scout Extended locally with Lumen : https://www.algolia.com/doc/framework-integration/laravel/getting-started/installation/
• Check everything works as expected
• If does'nt work as expected, adapt the source code of Scout Extended and perform a Pull Request
• Eventually, if needed, adapt tests

Undefined offset: 1 scout-extended with extended & Record too big

I keep getting an undefined offset: 1 with scout extended at this line:
https://github.com/algolia/scout-extended/blob/master/src/Searchable/ObjectIdEncrypter.php#L77

Maybe due to Algolia search client incompatibility if you see here:
laravel/scout#328

Deep analyse the `HtmlSplitter::class`

The goal is study how useful the current implementation of the HtmlSplitter can be. Taking real articles writing in HTML and analyse if they can be splitted correctly using our HtmlSplitter.

After discussion, you may want to perform a Pull Request improving the current implementation.

Note: Consider @julienbourdeau is implementation for wordpress (https://gist.github.com/julienbourdeau/ad9e34a8b65613aaf27ed5f46600eafc#file-HtmlSplitter-php).

What is the purpose of searchable() on the Aggregator class?

Hi @nunomaduro

This is more of a question. I stumbled upon this on Algolia\ScoutExtended\Searchable\Aggregator.

/**
* {@inheritdoc}
*/
public function searchable(): void
{
    AggregatorCollection::make([$this])->searchable();
}

I am assuming it is used to make an Aggregator searchable. However, it can't be called directly as it throws an exception. I also can't seem to find where this is being used.

On the other hand, it would seem like we already have a makeAllSearchable() static method that works with the underlying models of the Aggregator.

Eager loading for model resolution

When I switched from laravel/scout to algolia/scout-extended I noticed the model resolution is no longer eager loaded.

Item::search('term')->get();

If the above search results in 20 items, there are 20 database queries. In laravel/scout, it would do one where in (...).

Support of more complex Where is search queries

Currently Scout allows you to add simple "where" clauses to your search queries, as commented in the documentation:

https://laravel.com/docs/5.7/scout#where-clauses

It would be great if we could add more complex Where clauses in an easy way, such as:

$orders = App\Order::search('Star Trek')->where('user_id', '!=','1')->get();

Package requirements and compatibility

Currently, the project requires at least Laravel 5.7 (the latest). We need to define what version we want to be compatible with.

We need to fix the version of Scout we want to support (which will also define the PHP version).

Things to consider:

• Laravel Scout was release with Laravel 5.4
• Laravel 5.4 and Scout < 4.0 support php 5.6
• getScoutKey was introduced in 4.x (great feature)

Performance issues when indexing records that contain data from relationships

Summary

When searchableArray contains data from a relationship, the import will perform an extra query for each record in order to get it.

Scenario

A project belongsTo a customer. In search results, I want to display the Customer name under the Project name. The record might look like this:

{
    id: 1001,                    // $this->id
    name: 'Test Project',        // $this->name
    customer: {
        'id': 2002,              // $this->customer->id
        'name': 'ABC Company',   // $this->customer->name
    }
}

Problem

When running scout:import, it would perform N+1 queries to get the Projects and then load the Customer for each. It was very slow and would not work on large data sets.

In reality, I have multiple indexes where the records contain data from 3 different relationships, so the problem is magnified.

My Solution

The solution that I ended up at was to create a custom console command for importing records. It would chunk() through the records in batches of 500, ->load() the relationships, then I used the Algolia PHP client library directly (not Scout) to push the records to Algolia.

One note: I'm aware of but don't use protected $with = ['customer']; on my Project model. I don't always want to load the Customer whenever I have a Project instance.

When I explained this to @nunomaduro he requested I open an issue here.

Problem with import to scout

Hi there,

i have an Aggregator:

class Calendar extends Aggregator
{
    /**
     * The names of the models that should be aggregated.
     *
     * @var string[]
     */
    protected $models = [
        \App\Calendar::class,
        \App\Restaurant::class,
        \App\Meal::class
    ];
}

For Example, in the Calendar Class i have an shouldBeSearchable() and a toSearchableArray() Function:

public function shouldBeSearchable()
{
    return $this->restaurant->is_searchable && $this->meal && $this->date >= Carbon::now()->toDateString();
}

public function toSearchableArray()
{
    $calendar            = [
        'id'            => $this->id,
        'restaurant_id' => $this->restaurant_id,
        'meal_id'       => $this->meal->id,
        'date'          => $this->date,
        'title'         => $this->title,
    ];
    $calendar['_geoloc'] = [
        'lat' => $this->restaurant->lat,
        'lng' => $this->restaurant->long
    ];

    return $calendar;
}

The artisan command scout:status give me the following information:

 --------------------- ---------- -------------- --------------- ---------------- 
  Searchable            Index      Settings       Local records   Remote records  
 --------------------- ---------- -------------- --------------- ---------------- 
  App\Search\Calendar   calendar   Synchronized   71              0               
 --------------------- ---------- -------------- --------------- ---------------- 

And when i run the scout:import command, i get an output like this:

Imported [App\Search\Calendar] models up to ID: App\Calendar::53
Imported [App\Search\Calendar] models up to ID: App\Restaurant::10
Imported [App\Search\Calendar] models up to ID: App\Meal::8

But i have no records in Algolia, only the index calendars. What do i wrong?

Cheers
Ralf

Support of Laravel Nova

Yeah it will be awesome if we can manage index, its status, statistics etc directly from Nova admin panel.

If dedicated tool like telescope is there it will be fine too. Yeah its easy to see from algolia but we cannot give access to everybody right like in admin panel we can have our own restrictions etc..

Option to set path for index-settings

Personally I'd like to put the scout configs somewhere other than the config folder. Would be nice to have an optional setting for this in config/scout.php.

Add support to Laravel Scout 7.0

We are currently depending on "laravel/scout": "^6.1.1". Would be great if we could add support also to Laravel Scout 7.0.

Normally, we just have to edit the composer.json to add support to Laravel Scout 6 and Laravel Scout 7 at the same time.

Issues Installing 'Composer Require'

Problem 1
- Installation request for algolia/scout-extended dev-master -> satisfiable by algolia/scout-extended[dev-master].
- algolia/scout-extended dev-master requires laravel/scout dev-feat/upgrade-algolia-engine -> no matching package found.