meilisearch / integration-guides Goto Github PK
View Code? Open in Web Editor NEWCentral reference for Meilisearch integrations.
Home Page: https://meilisearch.com
License: MIT License
Central reference for Meilisearch integrations.
Home Page: https://meilisearch.com
License: MIT License
Hello!
We have more and more usage of our SDKs and integrations. In order to gauge feature usage in these different libraries, MeiliSearch could use some anonymously collected telemetry data.
Using this information, we would be able to measure:
We shall use a different platform for telemetry and bug report. For bug report, Sentry seems to be the obvious choice. We should check if it's possible to configure Sentry to work for our libraries and not for the whole binaries they are used in.
For telemetry, there are several options that I will begin to benchmark: Posthog, Amplitude, Segment, or any other tools you might suggest!
The telemetry should be disableable as easily as in MeiliSearch (using an ENV variable).
What do you think? Is there anything you think we should track in order to better understand the users?
Related to this issue meilisearch/meilisearch#824, the settings are going to change in the v0.13.0.
For the SDKs, it means:
acceptNewFields
routesTODO:
When merging bump at the start of the week bors
raises the following error:
{{:badmatch,
{:error, :push, 422,
"{\"message\":\"At least 1 approving review is required by reviewers with write access.\",\"documentation_url\":\"https://docs.github.com/articles/about-protected-branches\"}"}},
Is this something that we want? Something that can be changed?
I guess it's the equivalent of what we have done on DO: https://cloud.google.com/marketplace
Add a reference to meilisearch-dotnet repo in the WIP section
This repository and the integration guidelines should be more visible in the SDKs README so that the contributors can see them if they want to create their own one.
Later:
To copy/past and to put at the end of the repositories.
<hr>
**MeiliSearch** provides and maintains many **SDKs and Integration tools** like this one. We want to provide everyone with an **amazing search experience for any kind of project**. If you want to contribute, make suggestions, or just know what's going on right now, visit us in the [integration-guides](https://github.com/meilisearch/integration-guides) repository.
When MeiliSearch introduces a new change in their API design we should be involved and aware.
Those changes could potentially imply changes or new features that should be added to the clients.
To ensure the flow of information a process should be made and applied. We want to avoid unnecessary complexity.
This is a draft of what the process could be:
http-changes
(?)integration-guides
) in which we discuss how it will be implemented in the SDK'sbump-meilisearch-v0.*.*
. The version is the one from MeiliSearch in which that change/feature will be addedintegration-guide
xxx-SDK
+ for example MeiliSearch bump
?MeiliSearch Changes: Search Method is now POST and not GET
. Or should it instead have a specific label meilisearch-changes
or something like thatMeiliSearch bump
label addedNeed an opinion from @Kerollmops @MarinPostma @curquiza @eskombro @tpayet
(edit by @curquiza)
meilisearch-http
by the core-engine teammeilisearch-http
. The core-team (and the SDK team if needed) discusses the final decision in this issue so that everyone is kept informed.integration-guides
) in which we discuss how it will be implemented in the SDKs. The issue should be tagged as MeiliSearch bump
.MeiliSearch Bump
label (this step should be automatized!!)meilisearch-bump-vX.X.X
(X.X.X = next MeiliSearch version) branch exists in this SDK repository, we have to create one.bump-meilisearch-vX.X.X
branch and should be reviewed before merging. Be aware about breaking changes: a breaking-change
label should be added to the PR if it is!meilisearch-bump-vX.X.X
, a final review is done by an SDK team member before being merged into master
.To further ensure that the stringify
action of our filter
works correctly I would suggest adding the following tests in all the clients
In this case only the one movie with the genre sci fi
will be found.
dataset:
const dataset = [
{
id: 123,
title: 'Pride and Prejudice',
comment: 'A great book',
genre: 'romance',
},
{
id: 456,
title: 'Le Petit Prince',
comment: 'A french book about a prince that walks on little cute planets',
genre: 'adventure',
},
{
id: 2,
title: 'Le Rouge et le Noir',
comment: 'Another french book',
genre: 'romance',
},
{
id: 1,
title: 'Alice In Wonderland',
comment: 'A weird book',
genre: 'adventure',
},
{
id: 1344,
title: 'The Hobbit',
comment: 'An awesome book',
genre: 'sci fi',
},
{
id: 4,
title: 'Harry Potter and the Half-Blood Prince',
comment: 'The best book',
genre: 'fantasy',
},
{ id: 42, title: "The Hitchhiker's Guide to the Galaxy", genre: 'fantasy' },
]
the sci fi
genre has been added to the Hobbit.
Then a test should be made on a query that normally find more than one result. By adding a filter
on genre:sci fi
it should be reduced to only one hit.
test(`${permission} key: Search with filter with spaces`, async () => {
await client
.getIndex(index.uid)
.search('h', {
filter: ['genre = "sci fi"'],
})
.then((response: Types.SearchResponse) => {
expect(response).toHaveProperty('hits', expect.any(Array))
expect(response.hits.length).toEqual(1)
})
})
Set il all SDK and integration as user agent for help to know the sdk usage. Keep the default user agent and add our own.
For example: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15 MeiliSearchReact/0.16.0
.
Composed: MeiliSearch+SDK/Version.
According to this change in MS (meilisearch/meilisearch#808), all the SDK should remove the GET sys-info
route access.
v0.13.0
, not the v0.12.0
TODO:
Is your feature request related to a problem? Please describe.
For most of the projects i need the feature set MeiliSearch provides i have almost always a regular database as my main data store β and this is almost always prostgreSQL. And in those cases data is often duplicated or at least linked with ids. Say i have a movie database with a long description for each movie i want to do a text search in. Those two datasets needs to be synced between the two data stores and it is often the case that i need to do multiple network/database round trips across that two data stores.
Say i have MeiliSearch to search for "Wizzard, Magik" and it returns some ids (123, 432, 666) that i use to query data in postgres that returns the full movie entry (Harry Potter, Persey Jackson, The Sorcererβs Apprentice) so i can query the costomers that have watched that movies the last month on my streaming platform. You get the idea.
There is some effort behind the synchronization process and in handling the data in code and doing round trips and in modeling the data itself.
Describe the solution you'd like
I would like to have a PostgreSQL extension that is able to handle most of those complications. To make things short and don't waste time with an unsatisfying explanation from my side the idea is to have something similar to ZomboDB. ZomboDB is "linking" Elasticsearch and PostgreSQL in such a way that i only have to interact with PostgreSQL alone as a proxy to Elasticsearch. As Elasticsearch is way to excessive for almost all my use cases (i don't have petabytes of data to search through) and MeiliSearch covers all of previous usecases and is way easier to setup, maintain, configure and use this would ease up the process to "just quickly" add a text search to your project by just.
CREATE TABLE movies (
id BIGINT NOT NULL PRIMARY KEY,
name text NOT NULL,
keywords varchar(64)[],
short_summary text,
long_description meili.fulltext,
price bigint,
inventory_count integer,
discontinued boolean default false,
availability_date date
);
CREATE INDEX idx_movies
ON movies
USING meili ((movies.*))
WITH (url='localhost:7700/');
SELECT *
FROM movies
WHERE movies ==> 'botman+robin&limit=2';
According to this PR, MeiliSearch is going to have a POST /indexes/:uid/search
route. The GET
is still there of course
How should we handle it in our SDKs?
I checked and Algolia only uses the POST
one in their SDKs for the search
method:
Two options:
Do the same as Algolia. The GET
method for search should only be used in specific cases so MS needs to keep it but is way slower. See this discussion. Even Algolia warns us to us it.
As the GET
method is used in really specific cases and might lead to performance issues, considering not to use it until a user asks for it could be an option.
Rename the search
method into search_with_get
and the new search
method would use POST
. However, it means duplicating the code and most of all, all the tests. It could be a little bit painful for nothing...
I have a preference for option 1: it involves less work until our users ask for it π
What do you think @tpayet, @eskombro, @bidoubiwa?
NB: of course, the prototype for the search
method will not change because of the POST
usage, and will be as described in #28.
I notice each SDK has its own version of handling the health
method.
I suggest we find the best way to handle it.
Currently, the GET /health
route returns 204
without any body if MeilISearch is running. If MeiliSearch is not running, the SDK would catch another error/exception (ConnectionRefused
for example) but not an error coming from the MeiliSerach API.
health()
:
true
when MeiliSearch is healthy.false
when MeiliSearch is unhealthy (an error is caught)health()
:
true
when MeiliSearch is healthy.health()
:
true
when MeiliSearch is healthy.is_healthy()
:
true
when MeiliSearch is healthy.false
when MeiliSearch is unhealthy (an error is caught)π’ Let's the vote begin with the right emoji linked to each option! Of course, feel free to argue your point of view and to suggest another option!
TODO:
dependency lock files are not designed to be used by packages that are themselves dependencies of other packages.
Committing package-lock.json to the source code version control means that the project maintainers and CI systems will use a specific version of dependencies that may or may not match those defined in package.json. Because package-lock.json cannot be added to NPM registry (by design; see NPM shrinkwrap), projects that depend on a project that uses package-lock.json will themselves use package.json to resolve project's dependencies, i.e. what works for project maintainers/ CI systems might not work when the project is used as a dependency.
By design yarn.lock and package-lock.json are not added to NPM (see npm shrinkwrap)
Thus keeping the yarn.lock file is useful for two reasons:
About the second reason, this is what the article says:
I would support a variation of package-lock.json if it could somehow only apply to devDependencies. I can see some (albeit small) benefit to wanting your development environment not break if there is a broken release. I would personally prefer my environment to break and become aware that a dependency in my toolkit requires attention (and depending on the nature of the issue either offer help, subscribe to an issue or replace the dependency). After all, you can easily patch your dependency tree if you need to lock down a specific version for development purposes.
However, there is no such option and using lock files at the moment will create the risks described in this article β namely that the dependencies that you use do not match those that your users will depend on. Responsible development requires that your script works with the latest versions of dependencies (and yes that includes transitive dependencies) satisfied by semver.
I also think that keeping yarn.lock file only for this use case is a lot of work. Updating minors using dependabot or asking contributors to manually rebase because of conflicts in yarn.lock is a pain.
On the other hand, having a failing development environment is not a pleasant experience for contributors. I would like your opinion on the matter.
This applies also to https://github.com/meilisearch/instant-meilisearch, https://github.com/meilisearch/docs-searchbar.js, https://github.com/meilisearch/vuepress-plugin-meilisearch
Keep yarn.lock π
Remove yarn.lock π
The goal of this issue is to make the CONTRIBUTING.md less "heavy" for the contributors. Indeed, the more the file is big, the less they are going to read it.
Here is what I suggest:
Release Process
section was in the Development Workflow
: I suggest to move it to a separated section and to specify this section concerns only the Administrators, I mean the Meili team. If the contributors are curious, they can read. If they don't care, they know they can skip this section.Automated Changelogs
section smaller: as for the Bors section, the purpose is to reduce the CONTRIBUTING.md file and to redirect to the release-drafter guide. See #49.I created a PR so that you can approve on it! #50
I know a lot other things could be improved, this is just the first steps of a general change π
As of today, the documentation of the SDK's only exists in their respective readme.
Following stripe's way, the SDKs do not have independent documentation except for the front-end javascript sdk.
Potentially, we want to add more complex functions to all the SDK's and at that point we might consider independent documentation for the SDK's.
Meanwhile! My suggestion is to use the commentary-documentation of each language
python example:
In each popular language there (should) exist libraries that let you build a documentation based on those comments (pdoc3 for python). Maybe we could consider creating language.meilisearch
.com with the generated documentation.
For example it would be python.meilisearch.com
for the doc generated py pdoc3.
Those commentary-documentation are also useful for IDE as they use those commentaries to generate tooltips.
Example with writeFileSync
Generated tooltip when using this function in another project:
(note: it is already done in python SDK minus potentially review, and halfway done in javascript SDK)
Three suggestion of which I would like to discuss, know your opinion, and get your possible approval
Following this issue: meilisearch/meilisearch#729 the placeholder
search is soon to arrive in MeiliSearch. The placeholder search is meant to have search results even on request without any search query.
The displayed documents are sorted by the custom criteria specified in the settings when the query parameter is empty or not present at all (i.e /search?q= or /search).
This implies that the query
query parameter will not be mandatory anymore.
I think it is confirmed by what I observe in the source code:
fn new_search(&self, query: Option<String>) -> SearchBuilder;
Where Option
makes it optional.
Should we reconsider the usage of the search method in most of our clients
from:
search(query: string, parameters: object)
to:
search(parameters: object)
@curquiza Could you please rename this repository as integrations-guides
? It seems that I dont have the right to do it by myself π¬
While I was working in the Python SDK I realized that you can send a primary key as a aparameter while creating or updating documents. If a primaryKey is already set, the default behavior seems to be the same in MeiliSearch and in every SDK: it ignores your 'new' primary key, and makes the addition/modification you requested. No exception or error, no notification, nothing, it is just ignored.
In my opinion, this leads the user to think that the 'new' primary key was set, and lead to confusion and introducing bugs in different parts of his code (as it "seems" to work) without understanding what happened exactly and it might be hard to debug afterwards.
I propose throwing an error/exception/panic to tell the user he's actually requesting something that can't be done!
Python current index.update_documents() method:
def update_documents(self, documents, primary_key=None):
"""Update documents in the index
Parameters
----------
documents: list
List of dics containing each a document, or json string
primary_key: string
The primary-key used in MeiliSearch index. Ignored if already set up.
Returns
----------
update: `dict`
Dictionnary containing an update id to track the action:
https://docs.meilisearch.com/references/updates.html#get-an-update-status
"""
if primary_key is None:
url = '{}/{}/{}'.format(
self.config.paths.index,
self.uid,
self.config.paths.document
)
else:
url = '{}/{}/{}?{}'.format(
self.config.paths.index,
self.uid,
self.config.paths.document,
urllib.parse.urlencode({'primaryKey': primary_key})
)
return self.http.put(url, documents)
I would appreciate your opinion on this!
I guess it's the equivalent of what we have done on DO: https://github.com/netlify/addons
Not totally sure about this one, if it make sense for us, but Netlify is really popular so it's worth a try :)
I guess it's the equivalent of what we have done on DO: https://elements.heroku.com/buildpacks
Integrations-guides
seems like an English mistake to me ;)
(also, asked the American at my side, she agrees)
In the next release (v0.16.0), MeiliSearch would be able to create the index if it does not exist during:
The goal of this issue is to integrate the implicit index creation in the integration-side process by improving the user experience.
index('indexUid')
Currently, here are the lines to add documents:
index = client.create_index('movies') # If your index does not exist
index = client.get_index('movies') # If you already created your index
index.add_documents([...])
The goal is to avoid this index creation step and to provide a function that defines the "scope" where the users want to work: index(...)
. It will return an Index
object/struct but without doing any HTTP call.
In all situations, the new way to add documents will be:
index = client.index('movies')
index.add_documents([...])
Prototype:
index(indexUid: String) -> Index object/struct
Limitation: because the index()
method returns an Index
object/struct but does not do any HTTP call, this object/struct contains a primary_key
attribute that would be set to an undefined
value which would not necessarely the reality in MeiliSearch. Indeed, we have to differentiate the Index
object in RAM from the index present in the MeiliSearch instance.
Because the users have the possibility to skip the index creation, we should let them add the primary key during the document addition/update.
The prototypes should be:
add_documents(documents: Array, options: optional dict) -> JSON with updateId or a Update object
update_documents(documents: Array, options: optional dict) -> JSON with updateId or a Update object
The options
parameter will be used as follow: { primaryKey: String }
.
Ex: client.index('movies').add_documents([...], { primaryKey: 'name' })
Limitation: when writing client.index('movies').add_documents([...], { primaryKey: 'name' })
it could imply the primary key is only for this specific batch of documents. Which is not. The primary-key would be only take into account in the first documents addition. This ambiguous behavior is also present when using cURL
but we should have this in mind to answer to our users if needed.
Here are the modification to the getting started:
import meilisearch
client = meilisearch.Client('http://127.0.0.1:7700', 'masterKey')
# An index is where the documents are stored.
index = client.index('movies')
documents = [β¦]
# If the index 'movies' does not exist, MeiliSearch creates it when you first add the documents.
index.add_documents(documents)
....
The unmentionned parts of the getting started stay unchanged.
getIndex
, createIndex
and getOrCreateIndex
?createIndex
and getOrCreateIndex
stay unchanged =>
getIndex
will now do a HTTP call before returning the Index
object. In some SDKs, this method already does an HTTP call: in this case, nothing changes for this method.
getIndex
is used. Now .index()
should be called instead, otherwise, it would do an HTTP call.
The code samples have to be updated. Because the get_index
method is almost called everywhere, almost all the code-samples are concerned.
Edit after the python implementation:
We did the implementation of the implicit index creation in the Python SDK as a first try. Here what we improved/changed compared to the description above.
primary_key
attribute should stay as accurate as possible, it means it should be set/updated during index manipulation (get/creation/update)get_or_create_index
method now does at least one (sometimes 2) HTTP call to return a Index
object containing the correct primary_key
value. Plus, the name get
in get_or_create_index
would imply an HTTP call. See the related discussion: meilisearch/meilisearch-python#175 (comment)Currently, all the SDKs have an Examples
section.
Maybe I'm in the future because it's not done yet for all the SDK but anyway π¬... once an SDK has completed all the examples for the main docs (I mean docs.meilisearch.com) including examples in guides (like the JS does), not only in the API references, I think we should remove the Example section
and redirect the user to the API references.
What do you think?
(I don't forget that some custom methods for SDKs like wait_for_pending_update
are not going to be documented in the main docs, but the current Example
section is not related to them and I suggest we discuss that in another discussion π)
If you agree with this removal, how could we redirect clearly to the API references?
There is a rust-based plugin for postgresql to use elasticsearch as index for full text-search
I have been using this plugin and meilisearch for a while, and I think it will be very interesting if they can be combined, maybe into a meilisearch-postgresql plugin.
As of today, only the getting started guide has code samples of most of the SDK's.
Our objective is to get every example in the documentation to have different tabs with the different SDK's like this:
Now to achieve this results those are the steps:
Because it is more easy to write code in a file that has a good extension (.js
for javascript) I made a little CLI that generated the files based on a list of samples:
( for this example I only took the list of examples for the API references of the indexes route)
For this section to be completed:
In each generated file I have put the link to the place in the documentation where the sample will be. This is to help find a context to each example.
(link must be improved, they are bad for example purpose)
Once every sample has been completed, it would be a hassle for the documentation to require each sample one by one.
I made this little tool that compiles every sample into a json
file. This is the file that will be gathered by the documentation:
(i have put the same code in each sample file just for the example)
As you can see this example generated a php-sample.json
. This sample contains the following:
[
{
"id": "create_index_examples_1",
"content": "<?php\n $index = $client->createIndex('books');\n?>"
},
{
"id": "delete_index_examples_1",
"content": "<?php\n $index->deleteIndex();\n?>"
},
...
]
This file will be fetched on build in the documentation. Because it is in JSON file, no parsing is required at all by the documentation to read them as JSON objects.
In the documentation, we will use the following component to generate the tabs with all the fetched content:
<Codesample id="create_index_examples_1" />
This will add at that position a tab element with all the fetched SDK's.
To make it more easy for a user of a specific SDK to see all the examples in a given language this is what stripe suggest in the documentation :
https://stripe.com/docs/api?lang=go
go
By clicking on one of this logo it will change all the default samples to the one chosen.
MOVED TO : #3
I guess it's the equivalent of what we have done on DO: https://aws.amazon.com/marketplace
We want to provide a method in the MeiliSearch SDKs to wait for the asynchronous update.
Really useful in the testing part instead of using sleep
methods everwhere.
We chose the name WaitForPendingUpdate
for this method.
Except for the JS: getUpdateSync
because of JS convention.
The parameters should be named as updateId
, interval
and timeout
.
This method must have a break condition.
This method is a loop that checks the status of the update every interval
until a timeout
.
Both parameters must be customizable by the user.
If it's possible with the language, both parameters should be optional and the method should set default values if they are not defined by the user.
If it's not, like in Golang, another function (like a "wrapper") could be provided with default parameters.
If the method times out, it should return a TimeOut Exception
.
public function waitForPendingUpdate($update_id, $timeout_in_ms = 2000, $interval_in_ms = 10)
{
$timeout_temp = 0;
while ($timeout_in_ms > $timeout_temp) {
$res = $this->getUpdateStatus($update_id);
if ('enqueued' != $res['status']) {
return $res;
}
$timeout_temp += $interval_in_ms;
usleep(1000 * $interval_in_ms);
}
throw new TimeOutException();
}
At least, we should provide the following tests:
interval
and timeout
timeout
.Do it in:
Wrong repo
https://github.com/searchkit/searchkit by @searchkit is a ready to use React UI for Elastic search. I'm not sure if Meilisearch endpoints and responses are compatible with the searchkit. If not, may refer similar or other UI kits too in the integration guides.
We are thinking about creating a getOrCreateIndex
method.
This method returns an Index
object. It checks if the index already exists and if it's not, it creates the index.
By trying to create the index, and catching the exception. In every situation, only one HTTP call is done.
An example in ruby:
require 'meilisearch'
URL = 'http://127.0.0.1:7700'
def get_or_create_index(index_uid, primary_key = nil)
client = MeiliSearch::Client.new(URL)
begin
index = client.create_index(uid: index_uid, primaryKey: primary_key)
rescue MeiliSearch::HTTPError => e
raise e unless e.message.include? 'index already exists'
index = client.index(index_uid)
end
return index
end
index = get_or_create_index('indexUID')
puts index.show
(In ruby the index
method is the equivalent of getIndex
in other SDKs, and this method does not do any HTTP call)
This example is out of the MeiliSearch library. Of course, this method must be included in the client class.
This method must be tested:
Do it in:
Removed from the to-do list:
Don't hesitate to suggest other ideas to implement the method, or if I forgot any tests π
We want to offer the ability to export any SQL table to MeiliSearch.
The most used databases offer a SQL interface, so we should be able to read any table and stores its rows into MeiliSearch.
I started to write a Ruby script that could export rows from an sqlite table, but is it the right tool to do this?
We might not need performance here, but a tool that most people could use without having to install lots of dependencies.
Some databases such as PostgreSQL offer the ability to write extension, this could be the cleanest way to work with automatic export but this kind of work might tedious if we have to do it with every table.
The MVP could be a Ruby (or any other scripting language) script, that can connect to SQLite, PostgreSQL or MySQL that we could run on Heroku as a scheduled task and that could take any table from any database and make a full export to a given index in MeiliSearch.
I would love to have any input from the team to see if can't make something more simpler than this?
I create this issue as a continuation of this one that is an investigation specifically about naming for error handling π
The goal of this issue is to detail the basic requirements of the SDKs error handlers.
I would like to keep this discussion as clear as possible, it means if you think I forgot basic points you can continue the discussion here with really clear explanations. Or you can contact me directly (if you can) so that I can update this post π
The goal of an error handler in an SDK is to say to the user "this package (so MeiliSearch) you are using, among the multitude of packages, raises an error".
That's why our SDK should not return an Axios
(JS client library) or Httparty
(equivalent in ruby) error. Otherwise, it becomes complicated for the user to understand where exactly the error comes from.
The name of the error is really important, and should give 2 pieces of information:
MeiliSearch
Following the previous issue about error handler naming, here are the 3 minimum errors the SDKs should raise:
MeiliSearchApiError
: MeiliSearch (HTTP) sends back a 4xx
or a 5xx
. There are most of the SDKs errors.MeiliSearchCommunicationError
: problem with communication, for example, the MeiliSearch instance is unreachable.
the suggestion in the previous issue is
CommunicationError
but I really think it's important that the name of MeiliSearch appears in the error, otherwise it becomes complicated for the user to understand where the error comes from.
MeiliSearchError
: for any other errors in the SDK (wrong parameter if the SDK checks the parameters...etc)Some language allows namespacing. So the raised error could be MeiliSearch::ApiError
instead of MeiliSearchApiError
. It depends on the language. We should adopt the best practice of each language π
For example, in the Ruby and PHP SDKs, we used the namespace MeiliSearch
.
In the JS one, we use the prefix MeiliSearch
instead, directly in the name of the error.
Also, depending on the language, the Error
could be changed as Exception
for example (ex: PHP).
Even if we define a generic MeiliSearchApiError
we can also define more specific errors concerning the API. I particularly think about an error for This index already exists
.
But I suggest we talk about that in another issue once all the SDKs will have their own error handler according to this issue π
Just telling this is possible π
The name of the raised error is great, but not enough. We should display at least an error message.
In the case of the generic MeilISearchApiError
, the expection message has to be the message of the MeiliSearch (HTTP) response.
Again for the generic MeiliSearchApiError
, if it's possible, we should also display the HTTP code error. I say "if it's possible" because all the language interpreter does not use the same way to display exception raising. We should at least manage to display the error message π
Depending on the good practices of each language, we should also provide methods in our custom errors. The goal of these methods is to help the user during his/her code implementation and for debugging.
Generally, a to_sring
or message
method has to be implemented.
For example, with the PHP example, the __toString
method is the method called by the interpreter to display the raising exception.
For the generic MeiliSearchApiError
, a method/attribute should be provided for the HTTP code (ex: httpCode
) and for the HTTP message response (ex: httpMessage
or message
).
The goal of this issue is to stay basic, that's why I did not detail more what to do, especially because it depends a lot on each language.
If some methods or other kind of error implementation seems really essential to you, it will deserve its own issue on this repository π We should stay simple for the beginning with these 3 kinds of error.
Tell me if some parts are not clear to you, deserve more explanation, or if I forgot some problematic points in my logic.
The changes related to the new MeiliSearch bump, see the related milestones.
Here are the "emergency" changes to code before releasing MeiliSearch.
PUT /health routes
For SDKs providing a method for this route, we should remove it:
These changes might impact some test suites but not the code-base itself.
displayedAttributes
fieldsThese changes might impact some test suites but not the code-base itself.
The following section is not a emergency change, but could be done before releasing MeiliSearch if you find some times to do it.
Please refer to this more detailed issue #48.
We could provide a deleteAllIndexes
method.
This function is currently used in most of our SDKs but in the tests section. It's not a method provided for the users yet.
Delete all the indexes.
If there is no index, it does nothing.
In the Client
class:
public function deleteAllIndexes()
{
foreach ($this->getAllIndexes() as $index) {
$this->deleteIndex($index['uid']);
}
}
Then, do it for:
I just start to group all the remarks I got thanks to contributors and after our meetings:
CONTRIBUTING.md
-> related to this discussion.Done in #33
Having a TOC in a README is really useful but is a pain to maintain. The best way is to generate it automatically.
We could enforce the user to use (so to install) tool like doctoc and to run it before submitting it.
-> A script could check that the TOC is up-to-date for each PR.
-> in the README, the TOC could be surrounded by comments Do not change manually, use doctoc instead
(for example).
We should automatize PR on README to avoid opening manually a PR for each SDK for each change.
Related to my answer in the Release Discussion.
Since we have contributors doing great and useful work, I want to add a new step to our Release Process.
This process is described in each README in the Workflow Development
-> Release
section.
So far we have (for the Ruby one, the others are almost the same except for PHP and Laravel):
MeiliSearch tools follow the Semantic Versioning Convention.
You must do a PR modifying the file lib/meilisearch/version.rb with the right version.
VERSION = 'X.X.X'
Once the changes are merged on master, in your terminal, you must be on the master branch and push a new tag with the right version:
$ git checkout master
$ git pull origin master
$ git tag vX.X.X
$ git push --tag origin master
A GitHub Action will be triggered and push the new gem on RubyGems.
I want to add this sentence right after this paragraph and in the same section:
Then, you must create a release for this tag via the [GitHub interface](https://github.com/meilisearch/meilisearch-ruby/releases) (by editing the tag).<br>
The release description should detail:
- the changes merged since the last release with the link to the related PRs.
- the names of external contributors who participated in this release.
For the Laravel and PHP SDKs, I will only add the second part of this paragraph, because the release process already asks for creating a release via the GitHub interface.
Steps:
Release
process in the Go README yetWorkflow Development
at all yetSome actions in MeiliSearch are asynchronous (documents manipulation, settings...). The associated methods in the SDKs return information about the update
with an enqueued
status and we don't know when the action is processed.
We already provide, for most of the SDKs, the waitForPendingUpdate
method (related #1). But it means this kind of usage (in ruby):
update = index.add_documents(...)
index.wait_for_pending_update(update['updateId'])
We could provide a "wait" version of each method for better user experience, for example:
index.add_documents_wait(...)
First of all, how can we name these methods?
A simple way would be to add a suffix to each method. I suggested wait
without thinking about it a lot. So, I would like to read your ideas π
Instead of re-coding all the methods, we could find a way to generate the version of each concerned method, because there are just the same but with a wait_for_pending_update
method.
The goal is to avoid code duplication.
I haven't investigated this part yet.
index.add_documents(...).wait_for_pending_update()
Not sure if this solution is more user friendly than the other one. Investigation needed π
This solution would imply changing the return of all the current methods (big breaking). They should return an Update
object instead of a JSON object containing information about the update. This Update
object should contain the wait_for_pending_update
method.
def add_documents(documents, primary_key = nil, wait_for_update = false)
This solution might be the easiest one to code, but problem with the Go!
A solution could be not to provide these methods for the Go language and wait for the demand/help of the community. Do not forget the WaitForPendingUpdate
method is already available in the SDK Go.
Even if you would rather have other suggestions than suggestion 1, I really like having your suggestions of naming for the suggestion 1, so that we can really compare them in term of user experience.
Suggestion 1 doesn't imply any breaking but we have to find a way to generate the code in every language. And a lot of tests would be added, at least one for each new method.
Suggestion 2 involves big breaking but seems easy to code: just an Update
object to add.
Suggestion 3 is the easiest one to code, and does not imply any breaking, but problem with the Go language! A lot of tests would be needed too.
=> The goal is to find what is the more user-friendly for our users, not only what we would like to code because it's easier π
Add some automatic tags to newly created Issues or PRs would be very useful to organize SDK workflow. Use automation tools, filtering, etc...
There is also Auto-labeler
MeiliSearch v0.15.0 adds dumps
functionality, implemented in:
Dumps were previously named
backups
In order to provide access to this functionality from SDKs, they need to implement 2 new routes, and update the code samples:
dump
using the /dumps
POST route.To start the creation of a dump, a request must be sent as follows:
curl -X POST "http://localhost:7700/dumps"
If the creation of a dump is accepted, the HTTP status code 202
will be received and a uid
in the json body will inform the user of the unique identifier of the dump that is being created.
HTTP/1.1 202 Accepted
content-encoding: gzip
content-type: application/json
date: Tue, 06 Oct 2020 05:32:43 GMT
transfer-encoding: chunked
{
"status": "processing",
"uid": "20201006-053243949"
}
HTTP/1.1 409 Conflict
content-encoding: gzip
content-type: application/json
date: Tue, 06 Oct 2020 05:42:36 GMT
transfer-encoding: chunked
{
"errorCode": "dump_already_in_progress",
"errorLink": "https://docs.meilisearch.com/errors#dump_already_in_progress",
"errorType": "invalid_request_error",
"message": "Another dump is already in progress"
}
dump
creation status, using the /dumps/<dump-uid>/status
route.If the user needs to know the status of it's dump creation, a request must be sent as follows:
curl -X GET "http://localhost:7700/dumps/<dump-uid>/status"
HTTP/1.1 200 OK
content-encoding: gzip
content-type: application/json
date: Tue, 06 Oct 2020 05:48:00 GMT
transfer-encoding: chunked
{
"status": "done",
"uid": "20201006-054235910"
}
All code-samples
should be updated and added the following examples in their respective language:
post_dump_1: |-
$ curl -X POST 'http://localhost:7700/dumps'
get_dump_status_1: |-
$ curl -X GET 'http://localhost:7700/dumps/20201101-110357260/status'
SDKs:
MeiliSearch is an API with which you communicate through HTTP requests.
Because of that, it uses the HTTP error standards to inform you of the state of each request.
A wrapper, on the other hand, creates a sweet sugar coat around the API to make the communication easier in a given language. Although the wrapper uses HTTP requests, the user does not.
This is why I questioned to which extends do we want to communicate to the user the same way the API communicates with us: through HTTP error codes.
Imagine a case where you have badly formatted a search parameter the following way:
index.search(null)
If javascript validates data before sending it to MeiliSearch, it will raise an error like this (the text could be different):
MeiliSearchError: invalid query parameter
If javascript DOES NOT validate data before sending it to MeiliSearch, MeiliSearch API will send this back
HTTP/1.1 400 Bad Request
{"message":"invalid query parameter"}%
And could be formatted this way by javascript error handler:
HttpError 400: Bad request: invalid query parameter
This create an inconsistency in the error types.
How does the other API does it:
StripeError
StripeCardError
StripeInvalidRequestError
StripeAPIError
StripeAuthenticationError
StripePermissionError
StripeRateLimitError
StripeConnectionError
StripeSignatureVerificationError
StripeIdempotencyError
StripeInvalidGrantError
This error will natively output like this:
StripeAPIError: Invalid JSON received from the Stripe API
ApiError
Used in those cases :
Which means that the error output will natively look like this:
ApiError: User not found
So no mention of Algolia (but you will be able to see that it comes from algolia in the stack that appears below the error), and the status can be fetch this way: e.status
My favorite approach is the one from Stripe. Nevertheless, in a short term I would suggest this one:
Methods implemented by the SDKs are not documented anywhere at this moment. The only way a user can learn about these methods (waitForPendingUpdate
, getOrCreateIndex
, ...) is going through the source code, reading the methods or tests.
Those methods that are implemented (or supposed to be implemented) in every SDK should be documented somewhere, not only so the user can understand how they work, but even to let him discover them. There are two main possibilities:
In the docs, we have a dedicated page for SDKs. We can document those methods there once, and add samples for each language.
In every readme, we can explain them and give an exmaple
We document them in the official docs, and in every readme we write a sentence saying that those methods exist, and redirect the user to the official docs.
I would like to have your opinion on this! Thanks!
For the moment, MeiliSearch is not stable and uses a version tag like v0.X.Y
.
Currently, the SDKs are not stable either and they use the same X
than MeiliSearch for the versioning.
Problem: the MeiliSearch breakings are not the same as the SDKs ones.
Solution: after discussion, we decided to stop following the MeiliSearch versioning. Each SDK README contains a "Compatibility with MeiliSearch" section, so we should improve this section to explain the compatibilities to the user.
Here is the sentence I suggest, for the SDK python for example, once the v0.11
of MeiliSearch is out (the python package can stay in v0.10.1
):
This package is compatible with these versions of MeiliSearch:
- `v0.11.X`
- `v0.10.X`
(The package might be compatible with the v0.9, but we are not going to mention the version of MeiliSearch before the v0.10
in this section since it was before we decided to change the versioning)
I suggest being simply because the SDKs are not stable yet. I mean do not mention the other SDK versions than the current one.
Ex: the SDK is in v0.11
, the README will only mention the v0.11
(which is the current SDK version) and the MeiliSearch compatibilities. We are not going to talk about previous SDK versions like the v0.10
or the v0.9
in the README of the v0.11
.
It's been a long time I'm thinking about adding a CONTRIBUTING.md file, especially when I saw @erlend-sh was starting the PR for the MeiliSearch repository.
The goal of this file is to:
Of course, I took most of my inspiration in the Erlend's PR but I tried to take into account the Kero remark: not to provide a too long file that could discourage some contributors. The Development Worklow
needs to be detailed and accurate of course since it's mandatory to work. But I tried to make the Git Guidelines
simple (and that's why I put it to the end of the file). As we are beginners in the OS community, I would rather a contributor who contributes but we need to redirect him/her to the CONTRIBUTING.md, that no contribution at all.
The goal here is to make it simple with no complexe rules, but to give a base and a reference to our contributors. Thus, and since the rules are really easy to follow, if you notice a contributor that does not follow the guidelines, don't hesitate to redirect him to the CONTRIBUTING.md file (nicely of course π).
As it's not convenient at all to discuss together about a CONTRIBUTING.md template here, I'm going to create a PR that everyone is welcome to review π the PR #34 is merged now.
Here is the final contributing template.
In every repository READMEs, we need to:
Development Workflow
section in the README and replace it by this section #33 (comment)Todo:
Feel free to check the boxes if the changes do not apply to the suggested SDK.
Link: meilisearch/meilisearch#944
Routes impacted:
facetsDistribution
in a search query /indexes/test/search?facetsDistribution=%5B%22color%22%5D
This apply only to SDKs that made specific tests on facetDistributions should test if they are not breaking with the new MeiliSearch version.
This PR has been reverted
Link: Sort fields without consistency in lexicographical order meilisearch/meilisearch#943
Routes impacted:
displayedAttributes
in GET /index/:uid:/settings
and GET /index/:uid:/settings/displayed-attributes
routesfacetsDistribution
in a search query /indexes/test/search?facetsDistribution=%5B%22color%22%5D
This impact only SDKs that still have tests that sorts responses from MeiliSearch
This issue is not ready for this version
Link: Refactor /health route meilisearch/meilisearch#940
The update health route should be removed from every SDK
This issue is not ready for this version
link: Dump Meilisearch DataBase triggering an HTTP route meilisearch/meilisearch#840 & dump meilisearch/meilisearch#887
cf #44
// Will be discussed in another issue
This issue is not ready for this version
link: Considere an empty query search as a placeholder search meilisearch/meilisearch#916
For SDKs that tested an empty string as a query to verify that MeiliSearch returns an empty string, tests must be changed.
SDKs:
Over at @denoland, we want to start using Meili to power the search results for the module registry at https://deno.land/x . The backend is mostly built on AWS, and we would like to keep using AWS to host a Meili server.
My first thought was to use the official Meili Docker image and use aws Fargate with an EFS mount, though we also thought about provisioning an EC2 instance with an EBS volume.
Do you have any material or suggestions on the most efficient and cost effective way to run Meilisearch on AWS?
CC: @lucacasonato
Currently, there is no SDK documentation except the READMEs and the Getting Started in the docs.
But, in this part of the docs, the code is hard-written: the code snippets are not automatically generated. If the SDK methods change, the documentation needs to be changed too.
We should find a way to link the docs repository and each of SDKs.
In one PR, the SDK maintainers should be able to change the code of the SDK and the docs at the same time.
With this kind of solution, it would be able to add more code snippets in API References.
A "simple" solution would be to add a template file in each SDK that the documentation would be able to read and then generate code snippets according to this file.
Example of a template file:
###
# guides/introduction/quick_start_guide.html#create-your-index
$ gem install meilisearch
---
require 'meilisearch'
client = MeiliSearch::Client.new('http://127.0.0.1:7700')
index = client.create_index('movies')
###
###
# guides/introduction/quick_start_guide.html#add-documents
require 'json'
movies_json = File.read('movies.json')
movies = JSON.parse(movies_json)
index.add_documents(movies)
###
###
# guides/introduction/quick_start_guide.html#search
index.search('botman')
###
Bonus:
Bonus ++:
Codecov is already used in MeiliSearch, it helps us to know if a PR add non-tested code. If it's the case, we don't accept the PR. It's easy to install, easy to configure. Don't hesitate to ping me if you need help.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.