The algoliasearch-netlify's discuss from algolia

Algolia Crawler did not populate an index

Good day!

I tried installing the Algolia Plugin in netlify and I noticed that it didn't automatically populate an index.

When I tried entering the indices, it was blank, and prompted me to make a new index.

I am actually using Hugo for my blog website, and I wanted to add a search feature for it.

Is it because I am using Hugo that's why the plugin didn't work?

Please help.

Thank you again for your time.

Could not reach Algolia's Crawler, got: Internal Server Error {}

I am facing this issue from past few builds.

Here is the deploy log

3:40:58 PM: Could not reach algolia Error: Internal Server Error {}
3:40:58 PM: at onSuccess (/opt/build/repo/.netlify/plugins/node_modules/@algolia/netlify-plugin-crawler/dist/index.js:107:19)
3:40:58 PM: at processTicksAndRejections (internal/process/task_queues.js:97:5)
3:40:58 PM: at async Object.run (/opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins/child/run.js:20:3)
3:40:58 PM: at async handleEvent (/opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins/child/main.js:36:38)
3:40:58 PM: at async process. (/opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins/ipc.js:103:16)
3:40:58 PM:
3:40:58 PM: ────────────────────────────────────────────────────────────────
3:40:58 PM: Plugin "@algolia/netlify-plugin-crawler" failed
3:40:58 PM: ────────────────────────────────────────────────────────────────
3:40:58 PM:
3:40:58 PM: Error message
3:40:58 PM: Error: Could not reach Algolia's Crawler, got: Internal Server Error {}

3:40:58 PM: Plugin details
3:40:58 PM: Package: @algolia/netlify-plugin-crawler
3:40:58 PM: Version: 1.0.0
3:40:58 PM: Repository: git+https://github.com/algolia/algoliasearch-netlify.git
3:40:58 PM: npm link: https://www.npmjs.com/package/@algolia/netlify-plugin-crawler
3:40:58 PM: Report issues: https://github.com/algolia/algoliasearch-netlify/issues
3:40:58 PM:
3:40:58 PM: Error location
3:40:58 PM: In "onSuccess" event in "@algolia/netlify-plugin-crawler" from Netlify app
3:40:58 PM: at onSuccess (/.netlify/plugins/node_modules/@algolia/netlify-plugin-crawler/dist/index.js:112:21)
3:40:58 PM:
3:40:58 PM: Resolved config
3:40:58 PM: build:
3:40:58 PM: base: /opt/build/repo
3:40:58 PM: command: hugo --minify
3:40:58 PM: commandOrigin: ui
3:40:58 PM: environment:
3:40:58 PM: - ALGOLIA_API_KEY
3:40:58 PM: - ALGOLIA_BASE_URL
3:40:58 PM: publish: /opt/build/repo/public
3:40:58 PM: publishOrigin: ui
3:40:58 PM: plugins:
3:40:58 PM: - inputs: {}
3:40:58 PM: origin: ui
3:40:58 PM: package: '@algolia/netlify-plugin-crawler'
3:40:58 PM:

Crawler not triggered in manual deploys

The crawler does not seem to be triggering for manual deploys. Any thoughts on how to get it to trigger? I'm not seeing the crawler item in the deploy summary, but the plugin is showing up in the plugins list.

We run a docker process in Gitlab to build a static site, then use the netlify-cli npm package to deploy the files.

gitlab-ci.yml

- npm run build
- npm i -g netlify-cli
- netlify deploy --dir build --prod

netlify.toml

[[plugins]]
package = "@algolia/netlify-plugin-crawler"
  [plugins.inputs]
  branches = ['production']
  renderJavaScript = true

Netlify deploy details

Algolia Netlify frontend script no-ops

After configuring the Algolia Netlify Build Plugin and I embedded your script as suggested at https://github.com/algolia/algoliasearch-netlify/tree/master/frontend, but it doesn't do anything. Here is the relevant HTML from my sample page:

    <footer>
      <p>I'm a footer!  A copyright lives here, or something.</p>
      <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@0/dist/algoliasearchNetlify.css" />
      <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@0/dist/algoliasearchNetlify.js"></script>
      <script type="text/javascript">
        algoliasearchNetlify({
          appId: 'ABC123',
          apiKey: '987654321',
          siteId: '123456789',
          branch: 'main',
          analytics: true,
          autocomplete: {
            hitsPerPage: 5,
            inputSelector: 'input[type=search]',
          },
          color: '#3c4fe0',
          debug: false,
          poweredBy: true,
        });
      </script>
    </footer>

I wasn't sure if I was just missing a manual input box, but that didn't help either.

    <footer>
      <p>I'm a footer!  A copyright lives here, or something.</p>
      <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@0/dist/algoliasearchNetlify.css" />
      <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@0/dist/algoliasearchNetlify.js"></script>
      <script type="text/javascript">
        algoliasearchNetlify({
          appId: 'ABC123',
          apiKey: '987654321',
          siteId: '123456789',
          branch: 'main',
          analytics: true,
          autocomplete: {
            hitsPerPage: 5,
            inputSelector: 'input[type=search]',
          },
          color: '#3c4fe0',
          debug: false,
          poweredBy: true,
        });
      </script>
      <label for="site-search">Search the site:</label>
      <input type="search" id="site-search" name="q" aria-label="Search through site content">
      <button>Search</button>
	  <p>End of footer</p>
    </footer>

Is there something broken, did I misunderstand your getting started directions, is there a typo in the directions, etc?

Note that my Netlify build DID index my site -- I can see content from my site at https://www.algolia.com/apps/ABC123/explorer/browse/netlify_123456789_main_all

Error: This plugin does not work locally, please deploy to a branch to test it.

Hi, I try to run this plugin on my gitlab pipeline with netlify-cli, but this error is showing up. I've installed @algolia/[email protected] into my devDependencies because of problems with 1.0.14 related to this #712

$ sdk_run netlify deploy --build --prod
## /usr/bin/buildah run -t --add-history sdk-249368 -- netlify deploy --build --prod
(node:1) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)

┌─────────────────────────────┐
│        Netlify Build        │
└─────────────────────────────┘

❯ Version
  @netlify/build 4.1.3

❯ Flags
  mode: cli

❯ Current directory
  /data/builds/dovera/mojezdravie/design-system/.release

❯ Config file
  /data/builds/dovera/mojezdravie/design-system/.release/netlify.toml

❯ Context
  production

❯ Loading plugins
   - @algolia/[email protected] from netlify.toml and package.json

┌───────────────────────────────────────────────────────────┐
│ 1. onSuccess command from @algolia/netlify-plugin-crawler │
└───────────────────────────────────────────────────────────┘

Algolia Netlify plugin started

┌─────────────────────────────────────────────────┐
│ Plugin "@algolia/netlify-plugin-crawler" failed │
└─────────────────────────────────────────────────┘

  Error message
  Error: This plugin does not work locally, please deploy to a branch to test it.

  Plugin details
  Package:        @algolia/netlify-plugin-crawler
  Version:        1.0.12
  Repository:     git+https://github.com/algolia/algoliasearch-netlify.git
  npm link:       https://www.npmjs.com/package/@algolia/netlify-plugin-crawler
  Report issues:  https://github.com/algolia/algoliasearch-netlify/issues

  Error location
  In "onSuccess" event in "@algolia/netlify-plugin-crawler" from netlify.toml and package.json
      at onSuccess (/node_modules/@algolia/netlify-plugin-crawler/dist/index.js:55:28)

  Resolved config
  plugins:
    - inputs:
        renderJavaScript: true
      origin: config
      package: '@algolia/netlify-plugin-crawler'

┌─────────────────────────────┐
│   Netlify Build Complete    │
└─────────────────────────────┘

(Netlify Build completed in 460ms)
Please provide a publish directory (e.g. "public" or "dist" or "."):

netlify.toml

[[plugins]]
package = "@algolia/netlify-plugin-crawler"
  [plugins.inputs]
  branches = ['master']
  renderJavaScript = true

Error: "sh" is not a valid Prism.js language for eleventy-plugin-syntaxhighlight

hierarchical search doesn't work

Hello Algolia Netlify team,

I have an issue that's a bit inconsistent, i.e. it doesn't work but it doesn't not work all the time. Here goes:

I have set the plugin "template" setting in the config .toml file to "hierarchical". Now, my understanding is that this tells Algolia to extract multiple records per page, one record per HTML header tag, and if these header tags contain an ID search results can display and jump to the specific header. I obtained a JSON export of the Algolia records that seems to confirm that at least the extraction part is working as expected. However, the actual search experience does not bear up to this.

Here's my main website and github.

Hierarchical search works for some pages, but not others. For example, it works for items in these pages here and here. Try searching for "Alina" or "Biological communities". But it doesn't work for headings in these pages here or here. Try searching for "Dunton" or "policies".

Note that the search result still returns the correct page, and also jumps to the correct header when selected, but does not display the heading. Search results also used to display the breadcrumbs trail "H1 heading > H2 heading > H3 heading" etc, but now no longer do, even for the pages that "work".

I'd appreciate insight as to why this is happening. Thanks a lot!

Execute Javascript on webpage

Some Netlify websites requires javascript to be fully executed before we extract information.
Even though it's usually recommended to have SSR and/or static webpage,
you can easily plug any API to load the content in the frontend; making the crawler loose some info.

The crawler does support javascript rendering but it's not exposed in the plugin.
Should we do it?

Crawler finds sitemap but does not find links in it

Hello Algolia team,

I have a netlify website that is deployed using Netlify at https://ble.lternet.edu/, and would like to try Algolia for its search function. I made a fork of the underlying Github repository to test it out. The fork deploy URL is https://stoic-elion-ec8918.netlify.app/.

I installed and made changes to the Algolia plugin config in netlify.toml according to your recommendations. The crawler seems to be able to find sitemap.xml, but no links within it (see screenshot).

My hunch is that it's got something to do with the links in sitemap.xml being absolute URLs pointing to https://ble.lternet.edu/, and tried to set the customDomain parameter to match, but no dice.

Any ideas? Thanks for your work!

Add messages to the build logs

The following lines are shown in the deploy summary:

Warning when ALGOLIA_DISABLED or disabled is used
Warning when the branch is not part of the inputs.branches
Success crawler URL

Would it be useful to print those in the build logs as well using console.log()?

I was trying out the plugin myself and was wondering why the logs were only showing Algolia Netlify plugin started. After some time, I remembered that the deploy summary might have some information, and it did (I was on the wrong branch). In that case, seeing the information in the build logs might have helped. What do you think?

`postinstall` script

tsc is currently run on postinstall:

algoliasearch-netlify/plugin/package.json

Line 17 in d545cc0

"postinstall": "npm run build",

However, it is also run before yarn publish:

algoliasearch-netlify/plugin/package.json

Line 18 in d545cc0

"release": "yarn build && yarn publish"

Should the postinstall script be removed?

Use `utils.build.failPlugin()` instead of `utils.build.failBuild()`

This plugins runs onSuccess. onSuccess event handlers cannot make the build fail (see documentation) because it is run after the site has already been deployed.

Because of this, onSuccess event handlers should use utils.build.failPlugin() (which make the plugin fail, but not the build) instead of utils.build.failBuild() (which make both the plugin and the build fail).

When an onSuccess event handler uses utils.build.failBuild(), it is automatically converted to utils.build.failPlugin() under the hood, but the following error message is printed:

5:33:29 PM: ┌───────────────────────────────────────────────────────────┐
5:33:29 PM: │ 6. onSuccess command from @algolia/netlify-plugin-crawler │
5:33:29 PM: └───────────────────────────────────────────────────────────┘
5:33:29 PM: 
5:33:29 PM: Algolia Netlify plugin started
5:33:29 PM: Plugin error: since "onSuccess" happens after deploy, the build has already succeeded and cannot fail anymore. This plugin should either:
5:33:29 PM: - use utils.build.failPlugin() instead of utils.build.failBuild() to clarify that the plugin failed, but not the build.
5:33:29 PM: - use "onPostBuild" instead of "onSuccess" if the plugin failure should make the build fail too. Please note that "onPostBuild" (unlike "onSuccess") happens before deploy.
5:33:29 PM: 
5:33:29 PM: ┌─────────────────────────────────────────────────┐
5:33:29 PM: │ Plugin "@algolia/netlify-plugin-crawler" failed │
5:33:29 PM: └─────────────────────────────────────────────────┘

In order to remove this warning message, failBuild should be replaced with failPlugin.

front end module accessibility errors

Hi folks, just FYI, when I run my website which has the Algolia front-end bundle through a web accessibility checker, it flags two things in the front end that I can't change but perhaps would be easy enough changes for you:

The form with class "aa-Label" is empty
The button with class "aa-SubmitButton" is empty or has no value text

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

WARN: Using npm packages for Renovate presets is now deprecated. Please migrate to repository-based presets instead.

Warning

These dependencies are deprecated:

Datasource	Name	Replacement PR?
npm	`eslint-plugin-node`
npm	`eslint-plugin-standard`
npm	`node-sass`

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

chore(deps): update dependency terser-webpack-plugin to v5

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

npm

frontend/package.json

@algolia/autocomplete-js 1.5.6

@algolia/autocomplete-preset-algolia 1.5.6

@algolia/autocomplete-theme-classic 1.5.6

@algolia/transporter 4.13.0

@babel/core 7.17.9

@babel/preset-env 7.16.11

@types/react 17.0.44

algoliasearch 4.13.0

babel-loader 8.2.4

clean-webpack-plugin 4.0.0

core-js 3.21.1

css-loader 6.7.1

fork-ts-checker-webpack-plugin 6.5.1

mini-css-extract-plugin 2.6.0

mustache 4.2.0

node-sass 7.0.1

postcss 8.4.12

postcss-loader 6.2.1

postcss-preset-env 7.4.3

preact 10.7.1

sass-loader 12.6.0

terser-webpack-plugin 4.2.3

ts-loader 9.2.8

webpack 5.72.0

webpack-cli 4.9.2

webpack-dev-server 4.8.1

package.json

@netlify/build 26.5.2

@types/jest 27.4.1

@types/node 16.11.26

@typescript-eslint/eslint-plugin 5.18.0

@typescript-eslint/parser 5.18.0

concurrently 7.1.0

conventional-changelog 3.1.25

conventional-changelog-cli 2.2.2

eslint 8.12.0

eslint-config-algolia 20.0.0

eslint-config-prettier 8.5.0

eslint-config-standard 16.0.3

eslint-plugin-algolia 2.0.0

eslint-plugin-eslint-comments 3.2.0

eslint-plugin-import 2.26.0

eslint-plugin-jest 25.7.0

eslint-plugin-jsdoc 37.9.7

eslint-plugin-jsx-a11y 6.6.0

eslint-plugin-node 11.1.0

eslint-plugin-prettier 4.0.0

eslint-plugin-promise 6.0.0

eslint-plugin-react 7.29.4

eslint-plugin-react-hooks 4.4.0

eslint-plugin-standard 4.1.0

http-server 14.1.0

jest 27.5.1

json 11.0.0

netlify-cli 8.19.3

nodemon 2.0.15

prettier 2.7.1

renovate-config-algolia 2.1.9

ts-jest 27.1.4

typescript 4.6.3

node 16.*

yarn 3.2.0

plugin/package.json

node-fetch 2.6.7

@types/node-fetch 2.6.1

dotenv 16.0.0

nvm

.nvmrc

node 16.14.2

Check this box to trigger a request for Renovate to run again on this repository

`manifest.yml` `name`

The name in manifest.yml should match the npm package name (link to documentation).

Support for search via HTML text input form?

Thank you for bringing best-in-class search to Netlify sites.

Would it be possible to add support for searching via a standard text input form with results on a new page?

My homepage is currently less than 3KB unminified, with a DDG site search box.

Replacing it with the Netlify plugin's minified CSS/JS and related HTML adds over 144KB.

Would it be possible for the plugin to support using a basic HTML form instead with the results on a new page, much like HN does? The search box on their homepage consists simply of:

<form method="get" action="//hn.algolia.com/">
Search: <input type="text" name="q" value="" size="17" autocorrect="off" spellcheck="false" autocapitalize="off" autocomplete="false">
</form>

[feature request] Single Index for Multiple Netlify Apps

We're trying to use multiple Netlify apps to update the data in a single place. We have a usecase to display the search hits from multiple apps. So, we're trying to combine all the data from multiple apps into a single index.
It would be great if there is a feature to use a single index to crawl through multiple netlify apps. It will be great if there is an option to add more netlify apps into the inputs for a single crawler.
[plugins.inputs]
SITE_ID = ["xxxx"]

Crawler uses title over og:title

The docs state:

   * The title of the page
   * - og:title
   * - head > title
  title?: string;

Which i assume the og:title should be checked and if it’s not present, the title tag should be used.

Apparently the crawler seems to always use the title tag.

Our system is build with nuxtjs, even though i tried it with static .html pages and it doesn't seem to be working.

The site for reference: https://623c76049d958000095994ef--venerable-rabanadas-59494e.netlify.app/

[feature request]add pathPrefix to inputs

Gatsby generate urls with sub path when enables pathPrefix.

I'm using this feature to deploy all blogs in Netlify to https://xxx.com/blog/ without https://xxx.com which caused algolia's indexing approach failed.

Would you please consider adding an option pathPrefix to inputs to enable the sub directory deployment?

Improved logging

The utils.status.show() method is a convenient way to surface information to users (link to the documentation). As opposed to console.log() which prints to the build logs (where the user might not look), utils.status.show() shows information in the deploy summary, which is more visible.

The plugin result is probably a good candidate for this utility:

algoliasearch-netlify/plugin/src/index.ts

Lines 62 to 65 in d545cc0

 console.log({ 

 status: response.status, 

 text: await response.text(), 

 });

Example:

utils.status.show({
  summary: await response.text(),
  text: `HTTP status: ${response.status}`,
})

Search form doesn't work on a result page with hierarchical template

Hello Algolia team,

I've run into another issue while testing Algolia on our website. https://stoic-elion-ec8918.netlify.app/index.html is the demo. I've set the search template in netlify.toml to "hierarchical" and added IDs to <h1> elements, so that when users click on a result the page jumps to that specific section. What I've noticed is that once the user is there, the search form doesn't work again. E.g. typing in queries does nothing.

To replicate this behavior, you can try typing in "carbon" on the demo above, then go to the first result. From that page, type in carbon again and behold, nothing pops up.

Screenshots:

Searching from homepage:

Searching from result page:

From what I recall, this wasn't an issue when the indexing template was set to default.

No warning when no input matches

When no input matches the inputSelector, we should probably output a warning to the console, which would help debug the issue.

1.0.13 breaks build

Getting an error during build:

Error message
Could not import plugin:
/opt/build/repo/node_modules/@algolia/netlify-plugin-crawler/src/index.ts:1
import type { NetlifyPluginOptions, NetlifyPluginUtils } from '@netlify/build';
^^^^^^

SyntaxError: Cannot use import statement outside a module

I think upgrading node-fetch to a version >2 led to this problem since version >=3 doesn't support CJS. We had to downgrade to 1.0.12.

The version should either but adjusted accordingly (not a minor version) or node-fetch should be downgraded.

Sensitive values in the logs

Does the /api/1/netlify/crawl response include any secure/sensitive values?

The response might be printed by one of the following statements:

algoliasearch-netlify/plugin/src/index.ts

Line 81 in 601d32e

console.warn(response);

algoliasearch-netlify/plugin/src/index.ts

Line 91 in 601d32e

JSON.stringify(json)

algoliasearch-netlify/plugin/src/index.ts

Line 99 in 601d32e

console.error('Could not reach algolia', error);

Netlify build logs are sometimes public, in which case there would be a risk for those sensitive values to be made public as well. However, if this endpoint does not respond with any sensitive values, then this is not a concern. I am raising this up just to be 100% sure :)

User error vs plugin error

User errors should be reported using utils.build.failBuild() (link to the documentation), as opposed to throw new Error(), which are reported as plugin errors.

This plugin validates that some inputs are not missing. This would indicate a user error, i.e. using utils.build.failBuild() will indicate to the user that they should fix something in their configuration, as opposed to creating a bug report.

Hitting Enter/Return on search form does nothing

Hello Algolia team,

It seems to me that the Algolia front-end bundle does not generate a conventional search results page, and replaces it with the instant search feature. I'm a fan. However, hitting Enter/Return after typing in keywords on search form does nothing, which I suspect might be un-intuitive to users. Is this the desired behavior? For example, I imagine hitting enter could (1) go to the first result or (2) leave a "no results" message.

Thanks!

Plugin name convention

Plugin names follow a convention where they start with netlify-plugin- or @scope/netlify-plugin- (link to documentation).
This helps users discover and recognize Netlify plugins.

However, the plugin npm package is currently:

algoliasearch-netlify/plugin/package.json

Line 2 in d545cc0

"name": "@algolia/crawler-netlify-plugin",

Rendering issues on mobile

Hi,

I integrated this plugin with my Netlify site last week. I am also using the algoliasearch-netlify-frontend bundle to render my search box.

The search box renders fine on the desktop version of the site but not on mobile. Please see the following screenshot from my phone.

URL: ketanvijayvargiya.com/search/

(I would have liked to put the search box on the website header, so that it renders on every page, but my CSS-fu is so poor that I couldn't make that happen. So, for now, I am just putting a <div id="search"/> on a separate page.)

I am using Hugo as my SSG, in case that matters.

I'd appreciate it if someone can help me on this.

Inputs validation

The SITE_ID constant and the BRANCH environment variables are always defined, including in CLI builds. Therefore, the following validation checks can probably be safely removed:

algoliasearch-netlify/plugin/src/index.ts

Lines 45 to 46 in d545cc0

 if (!siteId) throw new Error('Missing SITE_ID'); 

 if (!branch) throw new Error('Missing BRANCH');

Please note that the SITE_NAME environment variable is always defined in production builds, but not in CLI builds yet. However, this is a bug that is currently being fixed (see netlify/build#1864).

algoliasearch-netlify/plugin/src/index.ts

Line 48 in d545cc0

if (!siteName) throw new Error('Missing or invalid SITE_NAME');

Incorrect error message when `ALGOLIA_BASE_URL` is missing

When ALGOLIA_BASE_URL is missing, the following error message is printed:

Error: Missing ALGOLIA_BASE_URL, please go to undefined/admin/netlify to complete your installation.

This is because algoliaBaseUrl is undefined in:

algoliasearch-netlify/plugin/src/index.ts

Lines 94 to 98 in 09c288b

 const missingEnvMessage = (key: string) => 

 `Missing ${key}, please go to ${algoliaBaseUrl}/admin/netlify to complete your installation.`; 

 if (!algoliaBaseUrl) { 

 return utils.build.failBuild(missingEnvMessage('ALGOLIA_BASE_URL')); 

 }

`NODE_ENV=development` behavior

When NODE_ENV=development:

siteName is crawler-netlify-plugin
deployPrimeUrl is https://master--crawler-netlify-plugin.netlify.app
the API key is unused

I wanted to confirm: is this temporary while this plugin is under development? The reason I am wondering is because some users have NODE_ENV set to development in their builds, so they would inherit this behavior, which might not be intended.

Debug statements

This plugins prints the following values inside the build logs:

algoliasearch-netlify/plugin/src/index.ts

Lines 27 to 29 in d545cc0

 // Debug 

 console.log(JSON.stringify(params, null, 2)); 

 console.log(JSON.stringify(process.env, null, 2));

Those values might be more useful for debugging, i.e. should they be printed?
Printing process.env might be a security concern as well since environment variables are likely to contain secret tokens, and build logs are often public.

Netlify CLI builds

Netlify users can run builds using the Netlify CLI command netlify build.
Local builds do not create deploys. Therefore, the DEPLOY_PRIME_URL environment variable is never defined in local builds.
However, this plugin requires this variable:

algoliasearch-netlify/plugin/src/index.ts

Line 49 in d545cc0

if (!deployPrimeUrl) throw new Error('Missing DEPLOY_PRIME_URL');

Would there be a way for this plugin to work without the DEPLOY_PRIME_URL? This would allow CLI users to use this plugin.

In the case a deploy URL is required for this plugin to work, this plugin should probably be successfully skipped (as opposed to make the build fail) when a local build is run:

if (constants.IS_LOCAL) {
  console.log('This plugin does not work in CLI builds');
  return;
}

What do you think?

Multiple selectors

I want to place the search box in the menu on the website. However, I have two version of the menu - one for desktop and one for mobile. I can't figure out how to initiate Algolia search in both of them. I tried using a class but that doesn't help. It is initiated only once for the desktop version and never shows in the mobile menu.

selector: 'div.algolia-search'

Unable to manage plugin through netlify.toml

Hi,

I enabled this plugin for my Netlify site by following the documentation.

However, I like to manage my plugins and environment variables through my netlify.toml file. So, I removed those environment variables and plugin from the Netlify website and added the following to netlify.toml:

...
ALGOLIA_API_KEY = "..."
ALGOLIA_BASE_URL = "https://crawler.algolia.com"

[[plugins]]
package = "@algolia/netlify-plugin-crawler"

On doing so, Algolia removed the crawler after sometime with the following error: Algolia can no longer access your Netlify Site "...". This shouldn't have happened, right? Or did I do something wrong?

Support for links' hostname being different than prime url

Each deploy have a unique DEPLOY_PRIME_URL (e.g: https://5b243e66dd6a547b4fee73ae--petsof.netlify.app).
The problem is that some website are not configured to have relative URL everywhere and we sometimes ends up crawling nothing because of a canonical URL or redirection that point to the prod domain.

Relatives URL are also not always a recommendation from Google, it goes against the standard but people trust Google's opinion.

Temp fix

Use relative URL everywhere in your website

Long Term Fix

Should we support this? and how?

Option to prevent PRs from indexing

It seems that PR previews will trigger a full index if they have the same branch name as those defined in plugins.inputs.

Is there a way to skip the index when a repository, not just the branch, is specified?

Nothing is showed in front-end

Hello, I have an issue similar to #82. The following is my code, which appears as grey (unactive) in my developer tools, and I can't figure out why.

This is my code:

<div id="search" class="row">
	<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@0/dist/algoliasearchNetlify.js"></script>
	<script type="text/javascript">
		algoliasearchNetlify({
			appId: 'UNJ7U6R9NG',
			apiKey: '2496a32ec63dd08b35a3************',
			siteId: 'c7f3a969-424a-450b-8636-2d477af82e76',
			branch: 'master',
		});
	</script>
</div>

I censored the last part of my API key but, if necessary, I can share it. I have no experience with JavaScript or APIs at all, I really need some help, here.

Note: there's no CSS since I embedded it in my website's CSS file to improve performance.

Thanks in advance!

Best,

Tommi

Algolia search works only in Jekyll's _posts folder

When I perform a search on my website, results are returned only from my Jekyll _posts folder. What could be the reason?

Add `main` to default `branches`

Since October 1st, GitHub new repositories' default branch will be main.

Should main be added to the default branches, together with master?

algoliasearch-netlify/plugin/manifest.yml

Line 8 in 8a2c881

default: ['master']

Source map warning

Firefox shows the following source map warning:

Source map error: Error: Invalid URL: webpack://[name]/./src/autocomplete-theme-classic.scss
Resource URL: https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@1/dist/algoliasearchNetlify.css
Source Map URL: algoliasearchNetlify.css.map

Crawler exclude pattern configuration

Hello all!

Not sure if it's me configuring something wrong re: exclude patterns, or if I understood the documentation wrong.

I configured the Allow setting in my robots.txt to only crawl a specific section of test site. But still the crawler puts all sites in its index, like this one. Which then in return leads to weird search results, as the paginated pages are part of the search query.

From my understanding setting Allow is kind of a allowlist mechanism, and should be sufficient. Or is it that I have to set Disallow explicity, with an appropriate pattern?

Thanks!

Support for custom meta robots

The crawler should support this meta by default
<meta name="Algolia crawler" content="noindex">

Plugin "@algolia/netlify-plugin-crawler" internal error

I have been using this plugin for almost a year with no issues, but this afternoon my deploys on netlify have been failing with the following error message:

Plugin "@algolia/netlify-plugin-crawler" internal error

Could not import plugin:
/opt/build/repo/web/.netlify/plugins/node_modules/@algolia/netlify-plugin-crawler/src/index.ts:1
import type { NetlifyPluginOptions, NetlifyPluginUtils } from '@netlify/build';
^^^^^^

SyntaxError: Cannot use import statement outside a module

While loading "@algolia/netlify-plugin-crawler" from Netlify app at Object.compileFunction (node:vm:352:18) at wrapSafe (node:internal/modules/cjs/loader:1031:15) at Module._compile (node:internal/modules/cjs/loader:1065:27) at Module._extensions..js (node:internal/modules/cjs/loader:1153:10) at Object.require.extensions. [as .ts] (/opt/buildhome/node-deps/node_modules/ts-node/src/index.ts:1361:43) at Module.load (node:internal/modules/cjs/loader:981:32) at Function.Module._load (node:internal/modules/cjs/loader:822:12) at Module.require (node:internal/modules/cjs/loader:1005:19) at require (node:internal/modules/cjs/helpers:102:18) at importLogic (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins/child/logic.js:23:14)

Does anyone have any ideas what could be causing this or what I can do to fix it?

Migrate to autocomplete.js v1

Currently we use the outdated autocomplete.js v0.
The decision was made for lack of better alternative and because we knew it correctly.
As the release of the refacto is coming soon and Docsearch using it, we could think about also migrating to v1.

`package.json` `keywords`

The package.json should contain the following keywords: netlify and netlify-plugin (link to documentation).
This helps users discover Netlify plugins on npmjs.com.
The same applies to GitHub topics.
Thanks!

`bugs` field

The bugs field should be defined in package.json. This is displayed to users when an error occurs inside your plugin. (link to documentation)

Option to don't overwrite new crawler code in online editor

On every deploy, the plugin overwrites the code in the online editor and I would like to keep it to customise the recordExtractor. I tried disabled = true in the netlify.toml file but it doesn't trigger the reindex and seems like the rest API is not available for the Netlify crawlers.

What options do I have to do not overwrite the code in the online editor and run the index?

new Crawler({
  appId: "...",
  apiKey: "...",
   ...
})

Thank you

Search bar disappears on completion of page load

The website is

website
Git repo:
repo

I used wget to clone the static html version of my website for which the following search bar is functional:

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@1/dist/algoliasearchNetlify.css" />
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/@algolia/algoliasearch-netlify-frontend@1/dist/algoliasearchNetlify.js"></script>
<script type="text/javascript">
  algoliasearchNetlify({
    appId: 'O6OT9G9U8Z',
    apiKey: 'b0cb3d390176e1ff4bd1ecb12fc15730',
    siteId: 'a1b7ee1a-11a7-4bd2-a341-2260656e216f',
    branch: 'master',
    selector: 'div#search',
  });
</script>

... ... ... insert html

<div id="search"></div>

```html

Support for password protected website

Netlify have a way to protect website behind a password.
Algolia Crawler support authentication but it's not exposed in the plugin yet.

Should we support this?
If so, how?

	console.log({
	status: response.status,
	text: await response.text(),
	});

	if (!siteId) throw new Error('Missing SITE_ID');
	if (!branch) throw new Error('Missing BRANCH');

	const missingEnvMessage = (key: string) =>
	`Missing ${key}, please go to ${algoliaBaseUrl}/admin/netlify to complete your installation.`;
	if (!algoliaBaseUrl) {
	return utils.build.failBuild(missingEnvMessage('ALGOLIA_BASE_URL'));
	}

	// Debug
	console.log(JSON.stringify(params, null, 2));
	console.log(JSON.stringify(process.env, null, 2));

algolia / algoliasearch-netlify Goto Github PK

algoliasearch-netlify's Issues

Repository problems

Rate-Limited

Edited/Blocked

Open

Detected dependencies

Temp fix

Long Term Fix

Recommend Projects

Recommend Topics

Recommend Org

Jobs