Termshark causing high CPU utilization and frequently crashing about termshark

[pocc]

Thanks so much @thebyrdman-git ! ❤️

[gcla]

Hi @thebyrdman-git - thanks for the bug report. This is worrying! Can you tell us more about how you're using termshark? Are you reading extremely large pcaps or from an interface? Does termshark jump up to that level of CPU usage quickly, or gradually over time? If I send instructions for profiling termshark, would you give it a try and report back? (Oops, just realized you said a small pcap in the other issue you raised. I'll send profiling instructions.)

[thebyrdman-git]

Yes, the pcap files I've attempted to read are very small, i.e., a few MBs. The CPU spikes quickly and doesn't reside for several seconds.

[gcla]

Hi @thebyrdman-git - I tried running termshark on a raspberry pi and I am seeing similar high CPU especially when using the mouse + the scroll-wheel on the packet list section of the UI. Investigating...

[gcla]

Hi @thebyrdman-git - I pushed a change to gowid, the TUI widget library underneath termshark, to optimize handling of mouse events. The idea is to skip sequences of mouse movement events unless they contribute something actionable like a button click. It's not perfect because other gowid apps might want all mouse events, but it's ok for now. Later I can try to detect when there's a backlog and start dropping mouse movement events, as an improvement. I just updated termshark to use the latest version of gowid. If you compile termshark from source + HEAD, you should get the latest. Would you mind giving it a try? Something like this:

export GO111MODULE=on
git clone https://github.com/gcla/termshark
cd termshark
go install ./...

then termshark will be in ~/go/bin/.

Let me know!

[gcla]

Hi @thebyrdman-git - a small addendum to the above - I'm about to push some more commits, so if you want to isolate yourself to include just the performance fix, try this instead

export GO111MODULE=on
git clone https://github.com/gcla/termshark
cd termshark
git checkout 41dcb7e
go install ./...

[gcla]

You can also use this binary produced by the CI/CD build:

https://storage.googleapis.com/termshark/0cc19e423b58322f2ba0de9645a2001de8727443/termshark_linux_amd64/termshark

[thebyrdman-git]

Sorry, I was out of town over the weekend. I have implemented the performance fix now. I have observed some improvement, but the program still hangs my system when scrolling through larger pcaps.

For example,

20K test pcap - brief max 39% CPU utilization for termshark
385M test pcap - continual high CPU utilization which peaked around ~120% CPU utilization

[gcla]

Hi @thebyrdman-git - I have a couple more performance improvements lined up now, but also a suspicion as to the cause. First of all, a couple of things to rule out.

• termshark only loads the PDML for a pcap 1000 packets at a time (cheating really). If you page down through the packet list, when you get close to a multiple of 1000, termshark will kick off another tshark process to get the PDML for the next 1000. It does cache these sets of PDML, but they are loaded lazily first time. I think you are seeing termshark itself peg the CPU high, but I want to be sure this isn't a tshark process also causing your system to feel temporarily laggy?

• if you run termshark like this, you can see when Golang starts garbage collecting. Unfortunately it writes to /dev/tty, so it'll mess up termshark's output and might be hard to track

GODEBUG=gctrace=1 termshark -r foo.pcap

It's normal to see some collection activity. If it seems to go nuts, then I'll have more questions...

My theory - do you see termshark lag when you pass through packets with very large payloads? For example, if you cursor-down the packet list across a packet with length > 2000 - does it start to feel laggy? You can hit '|' to rearrange the views to get a better look at the packet as hex bytes.

[thebyrdman-git]

"I think you are seeing termshark itself peg the CPU high, but I want to be sure this isn't a tshark process also causing your system to feel temporarily laggy?"

I am not observing the tshark process itself pegging the CPU.

I ran some additional tests in the following format, "GODEBUG=gctrace=1 termshark -r foo.pcap," on the 385MB test pcap. Then, after the program opened, I hit Page Down a few times and observed values as high as ~600% and ~700% CPU (see screenshot)

.
"My theory - do you see termshark lag when you pass through packets with very large payloads? For example, if you cursor-down the packet list across a packet with length > 2000 - does it start to feel laggy?"

Yes, performance does degrade as the payload increases.

[thebyrdman-git]

Doing some further testing, I scrolled through and paged down inside of a 20KB pcap with a total of 493 packets and observed no lagging and observed that the CPU usage only reached ~53%.

[thebyrdman-git]

I am also testing on other OS types. I ran the same tests on Red Hat Enterprise Linux 8 VM running the latest kernel, but the impact is not as severe. The CPU is generally higher as the payload increases, i.e., spiking as high as 180% with the larger pcap. The smaller pcap was read very easily and with no lagging.

uname -r
4.18.0-80.11.2.el8_0.x86_64

[gcla]

Thanks for all the time you've spent on this so far @thebyrdman-git, I really appreciate it.

Would you mind trying this new binary? https://storage.googleapis.com/termshark/7a551a45258db4714e67ca001e05c2c1cb18ee13/termshark_linux_amd64/termshark

I swapped out the widget that displays the hex/bytes view. Please consider it experimental and incomplete - for example, the new widget doesn't yet support copy-mode. But I think it's otherwise functional. Let me know!

[thebyrdman-git]

I think we're headed in the right direction. I tested using the latest experimental version in Fedora 29, and I've experienced much better performance reading the test 385MB pcap. The scrolling and paging down functions are much smoother, and I've haven't observed the same type of lag. I've also observed that when I'm scrolling down that the CPU utilization briefly increases (which is expected, of course), and then it immediately decreases when I stop either scrolling or paging up/down. The overall CPU utilization is much lower than before, the highest briefly peaking around ~100%. Thank you for your continued work to develop this tool. I am excited for it to be a reliable tool for a large number of Linux engineers. :)

[gcla]

Hello! One more binary:

https://storage.googleapis.com/termshark/0cf134e8d8b1b6d1c04dc9fe9829e81838293384/termshark_linux_amd64/termshark

The improvements are probably less noticeable this time, but let me know if anything misbehaves. It passes the tests, such as they are, and some rudimentary manual testing by me. These changes are intended to reduce the pressure to garbage collect so often.

[thebyrdman-git]

I'm having an issue with that URL:

$ git clone https://storage.googleapis.com/termshark/0cf134e8d8b1b6d1c04dc9fe9829e81838293384/termshark_linux_amd64/termshark
Cloning into 'termshark'...
fatal: unable to access 'https://storage.googleapis.com/termshark/0cf134e8d8b1b6d1c04dc9fe9829e81838293384/termshark_linux_amd64/termshark/': The requested URL returned error: 403

[gcla]

Sorry, I should've said it's just a single binary at that url - just wget and run :-)

[thebyrdman-git]

I'm getting lagging again after adding this binary on both Fedora and RHEL 8. It's better if I wait until the file loads completely before paging down or scrolling down, but it's easy to spike the CPU for me. What is your experience with implementing the binary on your raspberry pi?

[gcla]

Bah, sorry :-( Do you think it's worse than the first one you tested? Does the lag occur reliably when scrolling past particular packets?

On my pi, the performance is ok. Not amazing, but these recent changes do improve matters somewhat, especially when scrolling past packets with large (>15K) payloads.

At risk of using up your goodwill, I made another binary:

https://storage.googleapis.com/termshark/ba32296e01e78d4e295e19e2a33b30962f375fdc/termshark_linux_amd64/termshark

It won't be faster, but it has a new switch, --debug e.g.

termshark --debug -r foo.pcap

If you run that way, you can generate a Go performance profile by sending SIGUSR1 e.g.

pkill -SIGUSR1 termshark

It will start immediately and profile for 20 seconds. If you could scroll around and make the lag occur during that 20 second period, I can take a look at the profile and see what it reveals. It'll be saved as ~/.cache/termshark/cpu-*.prof.

More info here: https://github.com/gcla/termshark/blob/v2docs/docs/UserGuide.md (search for "Problems")

[thebyrdman-git]

On the contrary, the latest binary seems to be working very well for me. I have not been able to reproduce the same lagginess as before. I can hold down on the Page Up/Down keys without any sort of lag and I watch the CPU decrease to 0% in htop (tree mode filtering on "shark") almost immediately after I stop scrolling. Let me know if you still think generating the profile is needed.

[gcla]

Thanks @thebyrdman-git , I'll close this issue.