Support full text search of commands about atuin

[ellie]

AMAZING PROJECT!

Thanks! 😊

I commented on the PR too, but just to elaborate

I'd like to introduce the concept of different "search modes". Currently the default is just a prefix search, and then what you propose is more like FTS search (although I'd like to add SQLite FTS at some point too!)

As part of my response to this, I mentioned the possibility for a raw SQL mode too, so we're already at the point of potentially having "prefix", "full", and "sql"

Also see: #58

[gennaios]

Hello,

Just installed. Very nice.

Concerning FTS, I am used to, and I think many of us are, searching by wildcard and specifying strings in any order, e.g. one need not use exact phrase with web or store search engines; when we use the web or other searches, we need not perform multiple searches with the same terms in permutations of plural, word order, and so forth. Thus searching for example build go could match entries that contain all terms that begin with each specified. I think that's the closest to what we are used to. It's been a while since I've written SQLite FTS queries, but i believe it'd be MATCH "build* AND go*". Wouldn't it be nice? :)

https://www.sqlite.org/fts5.html#full_text_query_syntax

[ellie]

Hey @gennaios

Thanks for your comment! Glad you like it :)

Please see this part of the documentation for different search modes: https://github.com/ellie/atuin/blob/main/docs/config.md#search_mode. It might not quite match what you want, but will be closer to it

I'm definitely considering changing the defaults for the next release

And also wrt SQLite FTS, as you can see in my earlier comment it's definitely something we are aware of, and will consider for a future release. The other things to consider here are that adding FTS wil increase the index time, which might not be desirable for imports

[gennaios]

Hello @ellie,

I have set the preferences to fulltext. Though it doesn't behave as I am used to, I haven't looked at your code, but I imagine it is not behaving as wild card search, e.g., searching "ca to" doesn't find "cat torrc", e.g., using ca* AND to* as the MATCH string.

As for performance, I have a 4+ million row 15 GB db in SQLite and FTS is still quite fast on a fairly old machine. Performance is quite good as I've been using FTS for a while now. If you are willing to do such, I can look at my old code and see how it is. I parse the search search by separating each term by spaces, and then append asterisk to each term to create the MATCH string. Any words with a hyphen I also replace such with a plus, as that's how FTS works. Trying pluses or asterisks in the search for atuin also seem to not work. Maybe I have something misconfigured though I do have fulltext set in preferences.

[ellie]

Sure @gennaios, it's definitely something I'll consider in the future. In the past I've used SQLite FTS for >20 million rows and it's performed well, however there was a non negligible index cost. I highly doubt that'll be a problem here, but it is still something to be aware of.

You should find that putting asterisks in works fine though - otherwise % will work, as that's what SQLite used for it's wildcard character

Before committing to SQLite as a FTS solution I'd also like to look into other Rust-based databases. Super low latency "as you type" searching is very important for Atuin

[gennaios]

No prob. Someday would be nice.

Understood low latency is an issue. I think it should be fine considering any potential size of history. I believe there'd be only one field in the FTS index so it shouldn't be too bad. Index creation as you know is the same if you are already using SQLite FTS, though maybe you're not?

I have tried the rust SQLite package. It does perform better than the go library I use. Compared to my main use in Swift, unsure. If there are alternative databases that would perform better, that's possible.

There is also this though I haven't looked into it too much:

Native support for pluggable backends in SQLite
https://sqlite.org/forum/forumpost/69feb1a3af64a52b47dfd4b5c0dd89187f45de63e48f2543670b800814d5fe95

There are also various possible PRAGMAs, e.g. as I use an Apple Silicon Mac, I recently changed my DB to use a page size of 16k.

[gennaios]

btw, noticed in the new changelog you're using sqlx? I do not have a lot of Rust experience, yet in my small project, I use rusqlite. It is supposed to be faster. As you have more experience, perhaps you've already looked at it. But if not, fyi.

[ellie]

156893d

So we did used to use rusqlite, but I wanted to be able to use async. Especially on the server side of things, as we do see a fairly high request rate. I migrated the server to sqlX before I did the client, but then we switched them over so that we did not have to build in two different libraries at the same time. Rusqlite is defintiely easier to get started with though!