[Perf] Laggy when scrolling quickly through a list about hunter

[rabite0]

Oh, you're on macOS! In that case the config file is in

$HOME/Library/Preferences/hunter/config

It's what the Apple guidelines say, but for a terminal app it's probably not the best choice. I should change this, but without macOS it's hard to test :).

Interesting. I guess that means iTerm is slow :).

But I have to admit that the drawing in hunter is not very efficient either. There is a lot of overdraw, especially when it clears a big area, even though there were just a few lines. I guess that could and should be optimized to lower the load on the terminal.

I'm also thinking about this dropping of keys in the input buffer. But the problem is that it can't process/draw/whatever and react to your input fast enough. It will still feel laggy, so it's more of a workaround, than a fix. Although that's' probably better than being uncontrollable for a time.

[rabite0]

Done. Not sure if it's an improvement though.

[rabite0]

Ok, so as I thought, keeping the number of visible files around gives a huge boost to performance. It's almost funny. I can now browse a 100k directory without any trouble. A problem that I'm now noticing is that CPU usage goes up more and more the further I scroll down. At the start it's at 20-30%, but at the bottom it goes to around 80%. Still, pretty nice improvement for such a small change.

The flame graph looks very clean by now, which is nice, because I can easily remove most those huge bars without too much trouble.

[rabite0]

Making this smooth was one of my top priorities while writing hunter, so this is serious. :)

What's your CPU?

Did you compile with "cargo --release"? The --release part is really important, since the default debug build doesn't optimize very much and adds a lot of overhead due to overflow checks and such.

The debug build easily eats up a full core and them some on my system, but with --release it drops down to 30-40% in the worst case when scrolling with various types of previews being shown and animations on. I've been experimenting with enabling LTO, which reduces this further to around 20% and makes the binary smaller. CPU: i3 5005 2x2Ghz

If hunter can't process whatever the key makes it do fast enough, then those build up. I guess it might be possible to flush the input buffer after reading a key, but then it might drop something you actually want..

[therealklanni]

I didn't compile via cargo build, I used cargo +nightly install hunter. Does the --release flag still apply for this type of build?

---

Processor Name: Intel Core i5
Processor Speed: 2.4 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 3 MB
Memory: 16 GB

[rabite0]

No, I think cargo install builds with --release by default, so that should be fine.

What about CPU usage? Does it use up all of your CPU during scrolling? Can your terminal keep up? Does it still happen when you zoom in on the main column ("c")? That also disables preview processing completely.

If that helps you could try disabling just the animations. (Put "animations=off" in "~/.config/hunter/config". If that fixes it, it means your terminal has problems drawing fast enough. To be fair, hunter's animations are a little bit abusive ;).

[therealklanni]

I'm seeing a moderate CPU spike in iTerm when this happens. Main-column zoom in ("c") doesn't appear to have any positive impact.

I tried it in Terminal (default MacOS terminal app, as opposed to iTerm) and I saw a CPU spike here as well, though about 30% less, and I wasn't seeing the "inertial over-scroll" happen.

I think it's not a "bug" in the sense that anything unexpected is happening (it seems). Likely more to do with perf issues with my terminal combined with the high load of hunter on the CPU. Likely hunter spikes CPU just doing expected threaded operations, maybe partly due to animations, but iTerm has a hard time just keeping up with the redraws (this appears to be the larger issue).

I'll let you decide if that means hunter needs more perf improvements ;)

---

(The "animations=off" config didn't seem to do anything, still saw animations)

[therealklanni]

iTerm certainly isn't the most performant, but it has some nice features :)

Thanks for the details, I'll try putting the config there and see what happens. (Update: still doesn't seem to disable animations)

---

$ cat ~/Library/Preferences/hunter/config
animations=off

$ cat ~/.config/hunter/config
animations=off

[rabite0]

I got reports that it's loading the file, but since I don't have access to macOS, I can't really test or debug it myself. I guess I'll try again in a VM some time. I'll probably also add some startup options/flags to disable this soon, that could be used as a workaround in the mean time.

[rabite0]

Ok, so I implemented stdin flushing.

Surprisingly, I actually noticed a slight improvement myself. When I checked how big the buffer is I saw that the there sometimes were 1-2 characters buffered up during scrolling, which caused it to slightly overshoot during scrolling.

I'm still wondering why that is, maybe the termial doesn't consistently read the keys as fast as the key repeats when it's pressed down.

Anyway, that should more or less fix this issue.

[therealklanni]

Awesome, thanks for the update. Let me know if there's anything I can do to help with the config thing.

[rabite0]

I pushed a new branch "osx_config". If you could test that it would be great.

It should print two messages to stderr when it builds the path to the config file, so it might look a bit messy unless you start it with "2>dbg.txt". It should print the correct path there.

If that works, but still doesn't affect what it does, you might see something in the log (press g).

Be aware that the config file is loaded asynchronously, so if it's not ready immediately you will see the default config until it's ready, but that's not likely to happen anyway.

[therealklanni]

Will have a look soon after the weekend. Cheers :)

[therealklanni]

OK, sorry for the wait.

I took a look at the source, and I found that I had the config wrong. I had animations=off and I saw in the code it should be animation=off.

So, with animation off (yes the config_path code is working on macOS), it does speed up significantly. However, the "inertia effect" is still (just) noticeable. It's very minimal, overall much more performant. Any latency with animations off is probably more to do with the ITerm lag.

[rabite0]

Great, then I will merge the osx-config stuff into master soon. Thanks for testing this, it bugged me for a long time.

Regarding the inertia, I could try flushing stdin before reading, right now it's flushed after reading a key. But I noticed that this breaks pasting into the minibuffer. I should probably limit the flushing to the browsing only.

[rabite0]

Need to stop flushing in the minibuffer. Different problem though.

[magnetophon]

I have similar issues: slow scrolling when I hold down-arrow; though no inertia.
It only happens in a huge dir, and I get masses of this error:

Stop it!! Key(Char('B')) does nothing!

This actually blocks hunter: when I set my keyrepeat slow enough not to get the errors, I end up with much faster scrolling, though still slow.

It's a 73G dir with almost 74.000 files and subdirs; /nix/store/, part of NixOS.

In this dir, xset r rate 180 10 gives medium speed scrolling and no errors.
xset r rate 180 20 gives occasional errors.

My normal rate: xset r rate 180 100 slows it down to a crawl.
lf and nnn scroll at full speed, and even ranger has only occasional hiccups.
In normal sized directories, hunter manages fine and scrolls seemingly as fast as the others.

Some other comparisons, all done in this same dir:

load time:
ranger & lf: +/- 25 sec
hunter: 13s
nnn: instant

memory usage:
ranger: 12%
hunter & lf: 1,7%
nnn: 0.1%

cpu usage while scrolling:
hunter: 100% in one process, 0% in the others
ranger:30-55%
lf: 15-35%
nnn: 4.6%

nnn isn't really a fair comparison here, since it doesn't do previews.

This is in alacritty, though kitty behaves the same.
In case it matters: I compiled using rustPackages 1.39.0 cause I couldn't get Nightly to work on NixOS.

I have this config:

animation=off
show_hidden=off
select_cmd=fd -t f | fzf -m
cd_cmd=fd -t d | fzf
icons=off
media_mute=off
media_previewer=hunter-media

[rabite0]

There are some issues with very large directories that I missed initially. The largest directory I tested on had ~3000 files in them so I never noticed.

It used to be the case that hunter completely hung up on fsnotify events in huge directories because the update algorithm was inefficient and ran on the main thread, too. Same when selecting 10k files at once and running "rm $s". Complete lockup, due to ineffiecnt string handling in my string library. That one has been fixed completely, but fs updates is still very suboptimal and needs some work. At least it's off the main thread for now.

As far as scrolling goes there is a low hanging fruit I'm going to tackle soon. I just need to add a field to the Files struct that stores the amount of files that are shown (depends on hidden file setting, filters, etc.) because right now this is calculated by iterating over ALL files and filtering out what shouldn't be shown. In a flame graph I've seen this takes a substantial amount of time since it's done many times per key press, so that alone should give a nice boost.

Loading times can probably be improved a lot, too. Loading a directory happens in stages already (first only names, then metadata (size, ctime, etc), then calculate file count in directories) and at least for previews it only fetches metadata for stuff that's on screen. You might have noticed it in the preview pane. Since rendering updates is now off the main-thread this can probably also be done for the main view without causing stutter.

Just to be clear, by loading time you mean the time until the content shows up, not the time hunter takes to load and accept input?

EDIT: I just tested nnn again and it's definitely quiet a bit faster to load 100k files. On the other hand it hangs until the directory is loaded, whereas hunter still lets you move out of the directory, even as it is loaded.

I think the main advantage of nnn is that it only has to load one directory, whereas hunter has to load 3 at once. If hunter prioritized the main view, so that the other views waited until the main view is done loading it could probably get pretty close.

Tested on a HDD atm, so YMMV.

[rabite0]

8fc070b

[rabite0]

With the latest update hunter handles directories with 1M (!) files just fine during browsing. CPU usage with 1M files is under 10% at the top and still under 50% at the bottom on my system. That is WITH previews and animations. When I press (c)olumn to zoom in on the main view it's around 4% in a "small" directory of 6K files. Otherwise it tops out at 10% with all the fancy stuff.

Quite an amazing improvement and I didn't even change the fundamental algorithms. All this just by caching the number of displayed files in the directory and the currently selected file.

Two problems remain: Loading 1M files is quite slow and hunter will easily use up a GB of memory. I'll tackle that next.

[magnetophon]

Thank you!
Scrolling is as fast as the others now.

You mentioned that the UI shouldn't block when loading a dir.
It does block her: I can't go forward or back while loading.
Note: this blockage only happens when going into a subdir of a huge dir.
When going into the huge dir, I can go out during loading just fine.

[rabite0]

That is terrible and shouldn't ever happen when just moving around. One of the main reasons I started writing hunter in the first place was because I hate such lock ups, so to me that's a critical bug. :)

Nice catch, will look into it.

[magnetophon]

Note, I naively did ```cargo run .`` so I got a dev build. (I'm new to rust, only saw this after I had already posted the above report)

cargo build --release gave me:

 Compiling image v0.21.3
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/nix/store/32b72z72pi90zk487crx01d7pif58iz1-rust-1.42.0-nightly-2020-01-23-41f41b235/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/bart/source/hunter/target/release/deps/hunter_media-68c5c7dbddf310c8.hunter_media.5p98ffnk-cgu.0.rcgu.o" "-o" "/home/bart/source/hunter/target/release/deps/hunter_media-68c5c7dbddf310c8" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/bart/source/hunter/target/release/deps" "-L" "/home/bart/source/hunter/target/release/build/backtrace-sys-980a30403224bcf6/out" "-L" "/home/bart/source/hunter/target/release/build/sixel-sys-66a5e13ab6d85c68/out/lib" "-L" "/nix/store/32b72z72pi90zk487crx01d7pif58iz1-rust-1.42.0-nightly-2020-01-23-41f41b235/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/tmp/rustcECuFlT/libbacktrace_sys-dfedd0d9717fe308.rlib" "-Wl,--start-group" "/tmp/rustcECuFlT/libbacktrace_sys-eb090ce8dab9193f.rlib" "-Wl,--end-group" "/nix/store/wwz3k8ddpnhi78vh1j4k5b80mqnj3356-rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-19546023911d06ec.rlib" "-Wl,-Bdynamic" "-lsixel" "-lgstvideo-1.0" "-lgstapp-1.0" "-lgstbase-1.0" "-lgstreamer-1.0" "-lgobject-2.0" "-lgobject-2.0" "-lglib-2.0" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /nix/store/x5m45fcnky99r0k41kmdwmjb7zw5k4z4-binutils-2.31.1/bin/ld: cannot find -lgstvideo-1.0
          /nix/store/x5m45fcnky99r0k41kmdwmjb7zw5k4z4-binutils-2.31.1/bin/ld: cannot find -lgstapp-1.0
          /nix/store/x5m45fcnky99r0k41kmdwmjb7zw5k4z4-binutils-2.31.1/bin/ld: cannot find -lgstbase-1.0
          /nix/store/x5m45fcnky99r0k41kmdwmjb7zw5k4z4-binutils-2.31.1/bin/ld: cannot find -lgstreamer-1.0
          /nix/store/x5m45fcnky99r0k41kmdwmjb7zw5k4z4-binutils-2.31.1/bin/ld: cannot find -lgobject-2.0
          /nix/store/x5m45fcnky99r0k41kmdwmjb7zw5k4z4-binutils-2.31.1/bin/ld: cannot find -lgobject-2.0
          /nix/store/x5m45fcnky99r0k41kmdwmjb7zw5k4z4-binutils-2.31.1/bin/ld: cannot find -lglib-2.0
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: could not compile `hunter`.

[rabite0]

Looks like you're missing gstreamer, or at least haven't set it up so that it can be linked in. Unfortunately I'm not familiar with nix, so I can't help you there. But you can try to disable video support by copiling with
cargo build --release --no-default-features --features img
Or even without img, (Not recommended ;) )

You mentioned that the UI shouldn't block when loading a dir.
It does block her: I can't go forward or back while loading.
Note: this blockage only happens when going into a subdir of a huge dir.
When going into the huge dir, I can go out during loading just fine.

Seems to be working as intended now.

[magnetophon]

Cool, I can build a release version and the blockage is gone, great!

One other perf issue:
The others are only slow when loading the dir the first time, and are instant from there on.
Hunter has the same duration every time...

PS: There was a typo in my comparison: hunter takes 13s to load, not 3.
Edited the post.

[rabite0]

The main reason is that hunter precalculates all the formatted lines (with name, color, padding, size, etc.) in a directory before it draws anything, which is quite suboptimal and also the reason it's using so much memory. This was supposed to be an optimization, but with large directories it's clearly counterproductive.

Nevertheless, I worked out some additional major speedups in the last two days and on my machine a directory with 500k files shows up in about ~4 seconds, which I think is quite reasonable already, although I can see that it would be annoying to wait that long every time.

I never thought about testing such gigantic directories before and thus hunter hasn't been optimized for them yet. Although with the huge progress in the last few days and no end in sight I'm starting to think that it might become faster than anything else at some point, or at least very competitive.

[magnetophon]

Thank you!!
Loading of the huge dir is almost instant now, and it scrolls it at full speed. 🥇
Nice work!!!

[rabite0]

Nah, I'm not quite done yet, I think I can shave off another 80% or so with a tiny bit of tweaking. ;)
After that improvements are going to become a bit more tricky, but using threads seems like a good way to make it even faster still.

Glad to hear it's working great for you now.

[rabite0]

Alright, I pushed another couple of huge speedups. It's about as fast as it can get now. The only way to make it faster is to thoroughly redesign some internals, asynchronous prefetching and maybe caching more stuff. It's seriously FAST now. The bottleneck is the overhaed from the readdir syscall, which is pretty slow (has to pointer chase a linked list, which takes some time with 500k files).

Still, it loads 500k files in under 2 seconds now. I'm going to make it even faster eventually, but for now this is as good as it can get.

If anyone still has problems feel free to reopen.

[magnetophon]

As lightning! ⚡