Exception when searching large datasets for common query using paginate about scout

[karakhanyans]

@razvaniacob working on it. 👌

[jasonbosco]

@driesvints Typesense has a max of 250 hits per page, and beyond that we'd have to use the page parameter to fetch additional pages. It looks like we're not managing this limit automatically within the Scout driver's pagination mechanism.

@karakhanyans is looking into this.

[driesvints]

@jasonbosco do you maybe know more about this?

[karakhanyans]

Hi @razvaniacob

I did setup a fresh project and could not reproduce the issue you are having.

1. I have setup typesense driver
2. Added Searchable to User model
3. Configured schema
4. Imported 10K users with factories
5. And did search with pagination ( User::search('m')->paginate(10);

The results worked fine.

Here is the repo where I did all that. The results are returned with /users endpoint.

https://github.com/karakhanyans/laravel-scout-typesense

Could you please fork the repo, and add the steps that you did and push them so I can reproduce error?

Thanks.

[razvaniacob]

Thanks @karakhanyans,

I've tested with your code and it works.

So then I started experimenting with what I have so when I do something like this:

return ImportedProperty::search('pipera')->paginate(10)->onEachSide(1)->withQueryString()
    ->through(fn ($obj) => [
        'name' => $obj->source_id,
    ]);

It works, but if I do it like this:

return ImportedProperty::search('pipera')
    ->query(fn (Builder $query) => $query->with(['imported_district', 'neighbourhood']))
    ->paginate(10)->onEachSide(1)->withQueryString()
    ->through(fn ($obj) => [
        'name' => $obj->source_id,
        'district' => $obj->imported_district?->name ?? ($obj->neighbourhood?->name ?? ''),
    ]);

It fails with the error

Maybe it has something to do with the
->query(fn (Builder $query) => $query->with(['imported_district', 'neighbourhood']))
line?

Any thoughts?

[karakhanyans]

@razvaniacob have you tried this line ->query(fn (Builder $query) => $query->with(['imported_district', 'neighbourhood'])) like this $query->with(['imported_district', 'neighbourhood']), without putting it inside query, as it's just a ->with.

[razvaniacob]

I followed the documentation found here Laravel Scout Documentation

If I do just ->with... I get this error
Method Laravel\Scout\Builder::with does not exist.

[thannaske]

This is an issue I also encountered, see: typesense/laravel-scout-typesense-driver#86

[errand]

Hey guys,
im getting the same issue when using queyr Builder

$posts = Post::search($text)
                ->query(fn (Builder $query) => $query->with(['images:id,name,path,extension']))

Removing it fixed the issue

[github-actions]

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

[davidstoker]

I just ran into this as well when using paginate() with query(). Looks to me like the root cause is how getTotalCount skips using the engine's result if a queryCallback exists. That causes it to call take with the $totalCount result if there is no limit. That ends up calling Typesense with a per_page value of $totalCount triggering the error.

scout/src/Builder.php

Lines 480 to 509 in 6e5b47d

	/**
	* Get the total number of results from the Scout engine, or fallback to query builder.
	*
	* @param mixed $results
	* @return int
	*/
	protected function getTotalCount($results)
	{
	$engine = $this->engine();
	$totalCount = $engine->getTotalCount($results);
	if (is_null($this->queryCallback)) {
	return $totalCount;
	}
	$ids = $engine->mapIdsFrom($results, $this->model->getScoutKeyName())->all();
	if (count($ids) < $totalCount) {
	$ids = $engine->keys(tap(clone $this, function ($builder) use ($totalCount) {
	$builder->take(
	is_null($this->limit) ? $totalCount : min($this->limit, $totalCount)
	);
	}))->all();
	}
	return $this->model->queryScoutModelsByIds(
	$this, $ids
	)->toBase()->getCountForPagination();
	}

As a workaround, because of the is_null($this->limit) ? $totalCount : min($this->limit, $totalCount) check, setting a limit in parallel to the paginate call results in a per_page value that's controlled.

This should workaround it for example and not affect pagination since it doesn't look at the $limit value.

$items = Item::search($request-input('query', ''))
    ->take(10)
    ->paginate(10);

It's not clear to me why the existence of the queryCallback should ignore the $totalCount returned by the engine? Why should it force falling back to the query builder for count if the query callback's purpose is to be "invoked after the relevant models have already been retrieved from your application's search engine" as described in docs.

[karakhanyans]

@davidstoker thanks for your input, this helped a lot with solving this.
@razvaniacob

cc: @driesvints @jasonbosco

#817