spatie / laravel-site-search

Create a full-text search index by crawling your site

Create a full-text search index by crawling your site

This package can crawl and index your entire site. You can think of it as private Google search. What gets crawled and indexed can be highly customized. Under the hood Meilisearch is used to provide blazing fast search speeds.

When crawling your site, multiple concurrent connections are used to speed up the crawling process.

Support us

We invest a lot of resources into creating best in class open source packages. You can support us by buying one of our paid products.

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on our contact page. We publish all received postcards on our virtual postcard wall.

Documentation

You will find full documentation on the dedicated documentions site.

Testing

Before running the test, make sure you are running Meilisearch locally.

composer test

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

• Freek Van der Herten
• All Contributors

License

The MIT License (MIT). Please see License File for more information.

laravel-site-search's Issues

No query has been set

work with liveware

report : No query has been set

Error exception being thrown after running php artisan view:cache

I've got an error saying directory does not exist in this package vendor directory after running php artisan view:cache. The log looks like the following:-

[2022-01-18 14:46:42] local.ERROR: The "/home/sail/example-app/vendor/spatie/laravel-site-search/src/../resources/views" directory does not exist. {"exception":"[object] (Symfony\\Component\\Finder\\Exception\\DirectoryNotFoundException(code: 0): The \"/home/sail/example-app/vendor/spatie/laravel-site-search/src/../resources/views\" directory does not exist. at /home/sail/example-app/vendor/symfony/finder/Finder.php:592) [stacktrace] #0 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Foundation/Console/ViewCacheCommand.php(67): Symfony\\Component\\Finder\\Finder->in() #1 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Foundation/Console/ViewCacheCommand.php(36): Illuminate\\Foundation\\Console\\ViewCacheCommand->bladeFilesIn() #2 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Collections/Traits/EnumeratesValues.php(242): Illuminate\\Foundation\\Console\\ViewCacheCommand->Illuminate\\Foundation\\Console\\{closure}() #3 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Foundation/Console/ViewCacheCommand.php(37): Illuminate\\Support\\Collection->each() #4 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Foundation\\Console\\ViewCacheCommand->handle() #5 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}() #6 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure() #7 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod() #8 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Container/Container.php(653): Illuminate\\Container\\BoundMethod::call() #9 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call() #10 /home/sail/example-app/vendor/symfony/console/Command/Command.php(298): Illuminate\\Console\\Command->execute() #11 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run() #12 /home/sail/example-app/vendor/symfony/console/Application.php(1005): Illuminate\\Console\\Command->run() #13 /home/sail/example-app/vendor/symfony/console/Application.php(299): Symfony\\Component\\Console\\Application->doRunCommand() #14 /home/sail/example-app/vendor/symfony/console/Application.php(171): Symfony\\Component\\Console\\Application->doRun() #15 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Console/Application.php(94): Symfony\\Component\\Console\\Application->run() #16 /home/sail/example-app/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run() #17 /home/sail/example-app/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle() #18 {main} "}

Markdown rendering on docs page

Hi,

this is not related to this package, as the markdown looks ok in installation-setup.md, but it does not render as a list on the docs page.

`highlightedSnippet()` does not work

When rendering the search results and trying to print the $hit->highlightedSnippet() nothing gets returned but $hit->snippet() does return something (unhighlighted of course).

Am I missing something?

Issue links are dead

On the New Issue Page, the links to Ask a question and Request a feature seem to be dead.

No records being created when crawling site

I have Meilisearch installed in a Docker container (on my staging server running Ubuntu) and can populate an index using CURL from the command line. But when I try index the site using laravel-site-search, everything looks like it worked fine in the terminal but when I check the index, no documents have been created. Have you had any other reports of this? Is there something I need to do with my docker set-up that I am not aware of? The funny thing is, it works fine locally on my Mac.

I am running Laravel 10 and PHP 8.1

After running the crawl command, If I run this:

curl -X GET http://172.19.0.22:7700/indexes/test-ATgwEZrpYOdOIYTd/documents

It results in this:

{"results":[],"offset":0,"limit":20,"total":0}

If I run

curl -X GET http://172.19.0.22:7700/indexes/movies/documents

I get

{"results":[{"id":287947,"title":"Shazam ⚡️","genres":"comedy"},{"id":287949,"title":"Thor ⚡️","genres":"action"},{"id":287948,"title":"Thor2 ⚡️","genres":"action"},{"id":2879455,"title":"Thor copy ⚡️","genres":"action"}],"offset":0,"limit":20,"total":4}

In a Laravel 9.19 sail install, meilisearch throws cURL error 7 (Connection refused)

I'm not sure why it's trying to request 127.0.0.1:7700

My .env file has MEILISEARCH_HOST=http://meilisearch:7700

In a sail shell, curl http://meilisearch:7700 works correctly.
curl http://127.0.0.1:7700 does not.

I can also access the Meilisearch UI in my browser at http://localhost:7700.

Dispatching job to crawl `http://localhost:8085`

   MeiliSearch\Exceptions\CommunicationException 

  cURL error 7: Failed to connect to 127.0.0.1 port 7700 after 0 ms: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://127.0.0.1:7700/indexes

  at vendor/meilisearch/meilisearch-php/src/Http/Client.php:164
    160▕ 
    161▕         try {
    162▕             return $this->parseResponse($this->http->sendRequest($request));
    163▕         } catch (NetworkExceptionInterface $e) {
  ➜ 164▕             throw new CommunicationException($e->getMessage(), $e->getCode(), $e);
    165▕         }
    166▕     }
    167▕ 
    168▕     private function buildQueryString(array $queryParams = []): string

Issues running first crawl

I seem to be having some trouble getting my first index crawled. I've installed meilisearch in an azure cloud deployment. In my local laravel app, I've required this package and meilisearch-php sdk package.

I created the index without issue and updated the table row to include {"meilisearch":{"url":"https:\/\/[SUBDOMAIN INSERTED].azurewebsites.net","apiKey":"[ADMIN KEY INSERTED]"}} where the two variables are populated with the correct information for my configuration. I have tested the Admin API key successfully in meilisearch on the web interface.

Initially when I ran php artisan site-search:crawl I received an error about casting (an array was expected for array_map but one of the functions was returning an array as contract model).

Doing some quick work to see if I could temporarily solve the problem with a simple cast, the issues just kept cascading down into other lines of code, leading me to believe that my installed versions were not meeting requirements.

I had installed the latest version of meilisearch, but I noticed a suggested version much lower. So I reverted back to meilisearch-php v. 0.17.2 and the casting issues went away.

However, now the crawl command is still throwing an error, saying that the Authorization header is missing. I'm at a loss at this point. I believe I've implemented all as instructed. Do you have any recommendations for me on where things might be going wrong?

UPDATE:
Poking at this a little further and I can see that this is still due to the meilisearch-php compatibility. v0.17.2 uses the old authorization header which is deprecated in the docker instance I deployed. If I edit the meilisearch-php vendor files directly to update the header then the error pushes on to another compatibility issue.

So I guess what I need to know is for laravel-site-search v.1.2.0, what version of meilisearch-php is required? Because the requirements spec'd on the documentation site don't give any information on versioning, but there are clearly some compatibility issues that need to be resolved. Or perhaps you can update to allow compatibility with the newest meilisearch-php release?

Can not open docs (too many redirects)

Hi, trying to read the docs but I'm receiving this error.
I've tried on other devices (and safari, chrome), as well on a other network and deleted cookies.

Fail to fetch the robots.txt file stops the crawler

If for some reason fetching the robots.txt throws and exception the crawler stops. Setting ignoreRobots has no effect because fetching the file happens before checking if we want to ignore it or not.

Maybe we can catch any exception when trying to fetch the robots.txt file and then check if we want to honor it and it is not null?

What do you think @freekmurze?

Still no docs

Can not open docs (too many redirects)