TabNine suggestions block Sublime for multiple seconds (on popup and selection) about tabnine-sublime

[boaz-codota]

Hey @kakra

Thank you for the awesome breakdown of the issue. We indeed not friendly to copy-on-write situations, and we will look into improving that.

Do you have any suggestions as to how to improve this? Or must we sacrifice consistency and batch write the state?

[kakra]

I think SQlite has a solution for this by using write-ahead logging (that's the WAL files), here's some starting point on the topic: https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs

Maybe the database files of TabNine should use something similar (I think you already use SQlite).

I've already done that for other SQlite databases and it can be done without knowledge for that application using the steps described in the link, section "How to enable SQLite WAL" (bottom of the page).

[kakra]

If possible, try to avoid fsync patterns for synchronization - fsync is quite horrible on btrfs performance-wise. Btrfs doesn't need it being a cow file system. But I'm not sure how to handle that, maybe through some env var like TABNINE_NO_FSYNC=1 or something that advanced users could set manually in their profile.

[kakra]

Also, avoid in-place updates of blocks in files and prefer appending data to some sort of log file which you could merge at process start or regular intervals back into the data file (similar to WAL).

[kakra]

I'm not sure if SQlite has file access modes which use fsync() - if they do, I'd research if that is really needed with WAL, and if it isn't, you may want to explicitly disable that access mode. Btrfs really doesn't like fsync because it flushes internal trees and transactions.