[Potential Bug] High CPU useage due to stuck fetch cycle about flexo

[chrishoage]

Unfortunately it looks like the logs rotated out and I'm having difficulty with coredump (running an ubuntu host at the moment)

This happens with some frequency so next time it does I will immediately gather logs and investigate how to generate a core dump when I have more time to dedicate to debugging this

In the mean time leaving this open may help gather information if anyone else has similar behavior (or not - which would indicate this is something broken with my setup)

Thank you for your help!

[nroi]

Can you please provide the flexo version you're using? (e.g. docker image ls nroi/flexo).

Also, please provide more details about how the clients are interacting with flexo, for example:

• Have you noticed any errors or something unexpected when you run pacman?
• Do you have any scripts, cronjobs or whatever that are running pacman, or otherwise interacting with flexo? Or are you certain that all requests sent to flexo are done because an actual human is running the pacman command?

I'm asking the second question because I've noticed some regularity in those log statements:

[2023-08-04T06:22:11.124Z INFO  flexo] Received request for path "core/os/x86_64/core.db".

The core.db file is the first file requested by pacman when you run pacman -Syu or pacman -Sy or similar. If you grep for those requests in the log statements, you get this:

[2023-08-04T06:22:11.124Z INFO  flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:27:15.828Z INFO  flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:32:20.478Z INFO  flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:37:25.131Z INFO  flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:42:29.479Z INFO  flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:47:33.904Z INFO  flexo] Received request for path "core/os/x86_64/core.db". Range start: 0

If you compare the timestamps, they are all 5 minutes and about 4.5 seconds apart. Even though I cannot rule out that this is caused by a flexo bug, I just want to ask if you have a cron job or something before I start chasing ghosts 😉

[chrishoage]

Can you please provide the flexo version you're using? (e.g. docker image ls nroi/flexo).

❯ docker image ls nroi/flexo
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
nroi/flexo   latest    6609a4785656   5 days ago      112MB
nroi/flexo   <none>    601652172d6e   10 months ago   112MB

Have you noticed any errors or something unexpected when you run pacman?
Do you have any scripts, cronjobs or whatever that are running pacman, or otherwise interacting with flexo? Or are you certain that all requests sent to flexo are done because an actual human is running the pacman command?

I have a polybar script which runs checkupdates every 5 minutes to check to see if there are any pending updates.

The logs may be a red herring in this case. The eating a lot of CPU until it restarts happens maybe once a month or so?

It doesn't "go away" and even happens when the system that runs checkupdates is asleep and not hitting the flexo mirror for updates. In my screenshot above for example in my screenshot above no computer on my network was asking for updates every 5 minutes.

So perhaps the logs are not showing what is eating the CPU.

I didn't have the presence of mind to inspect the container to see what process was eating CPU (I restarted the container from bed with my phone since I could hear my server was working hard for no reason 😅) - next time this happens I will do so to provide additional information.

[nroi]

I didn't have the presence of mind to inspect the container to see what process was eating CPU

Alright. If it's not too much effort, and you have no concerns regarding privacy, please also provide a coredump (I don't expect any privacy-relevant data in the flexo coredump, but that's up to you). You can send it to me via e-mail: [email protected]

Here's how to create a coredump on flexo running via docker:

• docker exec -it <container-id> /bin/bash
• then, inside the docker container, kill the process via: kill -11 $(pidof flexo).
• the coredump is then on the host system inside the /var/lib/systemd/coredump directory (if you're running ArchLinux).

[nroi]

Also, if you have any logs available from the time when the CPU load was so high, that would also be helpful.