zsh-histdb import about atuin

[noyez]

I've created a branch ( https://github.com/noyez/atuin/tree/noyez_zsh_histdb_import ). It compiles, and the tests run, but i haven't got it to run in practice b/c of async code. Specifically, the import traits are synchronous, but sqlx-rs is asynchronous, but i can't figure out how to call async code in sync function -- which itself is call from async. So the call stack is:

• async fn
• sync fn
• block_on ( async {} )
• ... and BOOM, this is not allowed.

The resulting error: Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.'

I do struggle w/ rust's async code, so if you have any ideas, i'm all ears. I'll revisit this soon, but i need to move on for now.

[noyez]

@norpol I'm just looking at the code, and i realize that i had protections against invalid strings only in the command text, but NOT in the directory -- which is the dir error that you're seeing -- name NOR in the hostname. I'll submit a patch to fix that. Would you be able to test code from my fork if i push an update?

[noyez]

As a user of zsh-histdb, i'd love to see this (esp as i've found histdb to be a little slow w/ a large history). As a rust developer, i could probably submit a PR for this. Do you have any vision on how you'd like to see done? If not, i can just dive in.

[conradludgate]

If you know of a good way of detecting a regular ZSH install vs a ZSH + histdb setup, then we can have a histdb.rs file that manages the extraction

Although thinking about it, the current import code (using their Readers and Iterators is a bit obtuse and fugly) would be hard to interopt with sqlite... I might rewrite it to be considerably simpler. I first rewrote this code when I started getting confident with Rust so I felt like it was super smart... 😂

[noyez]

You're right that sqlx traits don't seem to lend itself well to the Read + Seek + Iterator implementations, at least as far as i can tell, i've never used sqlx before. Either way, i'll get something working, i hope.

[conradludgate]

If needed, I can refactor our import code to be less sophisticated. Simple is better, I wish I realised that a year ago 🤦

[norpol]

There are already two PRs merged that relate to this ticket: #393 and #395.

I managed to successfully import, though there was an error message with some error dir with id 22 doesn't contain UTF-8.

Interestingly checking the sqlite-file, ID 22 was a valid string.

I managed to find the faulty directory field by running this code in a Python console.

import sqlite3
con = sqlite3.connect("zsh-history.db")
cur = con.cursor()
for row in cur.execute("SELECT * FROM places ORDER BY id"):
   print(row)

Which would raise an error, so I fixed that field value and afterwards import worked.

---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
Input In [1], in <cell line: 4>()
      2 con = sqlite3.connect("zsh-history.db")
      3 cur = con.cursor()
----> 4 for row in cur.execute("SELECT * FROM places ORDER BY id"):
      5    print(row)

OperationalError: Could not decode to UTF-8 column 'dir' with text '/mnt/waid/pl/static/cd_rips/2012091712033700�'

I'm really a database noob, otherwise I'd provide some more help than that.

[noyez]

@norpol Do you have a sample sqlite db file with that non utf-8 directory that i could use as a test?

[norpol]

@noyez I'm trying to replicate the faulty database string, though if I recreate a database the error is slightly different.

        Atuin
======================
          🌍
       🐘🐘🐘🐘
          🐢
======================
Importing history...
Importing history from zsh_histdb
Error: error occurred while decoding column "dir": mismatched types; Rust type `alloc::string::String` (as SQL type `TEXT`) is not compatible with SQL type `BLOB`

Caused by:
    mismatched types; Rust type `alloc::string::String` (as SQL type `TEXT`) is not compatible with SQL type `BLOB`

Location:
    atuin-client/src/import/zsh_histdb.rs:92:40

is the new error, the previous one is

        Atuin
======================
          🌍
       🐘🐘🐘🐘
          🐢
======================
Importing history...
Importing history from zsh_histdb
Error: error occurred while decoding column "dir": incomplete utf-8 byte sequence from index 44

Caused by:
    incomplete utf-8 byte sequence from index 44

Location:
    atuin-client/src/import/zsh_histdb.rs:92:40

though index 44 is a regular string (could that be some sort of async problem?).

Attached is my recreated database, the main one is really big (1.5 Million records) and is a bit sensitive.

I have no idea how zsh-histdb ended up with that invalid entry.

zsh-history-new.db.gz

[noyez]

@norpol If you are able, try the following branch : https://github.com/noyez/atuin/tree/noyez_fix_dir_hostname_utf8

I update the dir and hostnames to be more utf-8 friendly.

[norpol]

Awesome, works great! Thank you so much.

atuin search -c /mnt/waid/pl/static/cd_rips/2012091712033700�
atuin search -c /mnt/waid/pl/static/cd_rips/2012091712033700
# 2018-12-02 14:45:22	less extra_metadata.txt	0s

[noyez]

Should be addressed by PR #393 and PR #430

Can this be closed?

[ellie]

Yup it totally can! Thanks again for the work 🚀