[Dota2] fossilize eats all RAM and makes the system unresponsive as a result. about fossilize

[HansKristian-Work]

Someone reported on Discord that the issue is resolved now, so closing.

[Zren]

If it only used up my physicaly RAM, I wouldn't mind, but it begins filling up my swap (I've seen it fill all 10Gb). So it's maxed CPU/RAM/Swap, and probably also maxed the SSD I/O which is what causes major slowness in Linux.

I don't mind how long it takes, I just want some resources available to web browse when the game is updating/fossilizing.

OS: Manjaro

[Samega7Cattac]

@fmartins-andre It only takes a long time the first time, the next times are fast and only on some updates will be needed.

@simvux That's not a bug, it's normal and expected to use all threads it can finish faster

@MajorGonzo read the 2 above ^

[Plagman]

Thanks for the reports. The case where any fossilize_replay processes are running while a game is running is not intended. Is anyone able to reliably reproduce such a scenario?

[kakra]

@alexeysvrv With the screenshot of @jakogut the problem is easily visible: The system has 32 cores and the current implementation duplicates the database metadata on every core. An update will follow soon which stops the memory duplication by using shared memory between all cores. The thread count itself should not matter too much then because the update will also use optimized scheduling. Amount of write-back can still be a demanding issue with that many cores.

The architecture has thus already been changed and optimized. Remaining issues with write-back due to shader caching happening inside drivers is a problem that cannot be properly fixed in fossilize. This needs work from the driver makers. But let's first see how it works when the updated code-base landed in Steam.

[simvux]

I'm having the same issue but for seemingly completely different reasons. Ram-wise I'm fine, but the processing step maxes out all my 12 threads, not leaving one for other tasks.

This feels like a terrible idea for something that can kick in in the background automatically after game updates...

OS: Void Linux
Kernel: 5.7.13_1
CPU: Ryzen 5 2600X
RAM: 16GB

Maybe this should be a separate issue?

[simvux]

@Samega7Cattac it's not a bug, just a really bad overlooked flaw. It should most definitely not render the computer unusable just to get a 5% faster compilation time from using 12 instead of 11 threads. I tend to want to use my computer during this processing step.

[theforbiddenpool]

Having the same problem too with Path of Exile. I started to skip the processing of shaders before the game launched because it was taking me 2-3 hours to finish (I wanted to play), and I think after that the game started to crash at least once per game session. Now I realize it may have been from lack of RAM, since today I looked at it I had my 16GB full + half of my swap. Needless to say, my computer froze. I don't remember that happening before I started to skip the shaders processing, but I don't have the time to test it today.
My computer specs:

Arch Linux
Kernel version: 5.7.12-arch1-1
CPU: i5-7500
RAM: 16GB
GPU: Nvidia GTX 1050 Ti

It seems most people in this thread are using mesa, but the problem seems to be the same with Nvidia OpenGL. The version I'm currently running:

OpenGL core profile version string: 4.6.0 NVIDIA 450.57

Fossilize processes don't terminate even after I close Steam and Path of Exile unless I do it manually. If PoE was crashing due lack of memory, well.. that's rough.

[xggrnx]

Disable vulcan in dota2.
`
cd /steamapps/shadercache

find . -type f -print0 | xargs -0 rm
`

https://www.reddit.com/r/DotA2/comments/c199kv/dota_2_crashes_on_launch_on_linux_with_shader/erc9r8t?utm_source=share&utm_medium=web2x&context=3

helps me.

[RawPorkSushi]

Slightly tangentially, in my case opening Steam itself doesn't cause the Resource-devouring-Fossilize processes. In my case, it is only when I launch Borderlands 3, and it only happens roughly 60% of the time, and about half of those times, it takes over my system such that it's nigh impossible to escape/cancel/skip/shut-down.

In the attached screenshot, you can see that Fossilize processes haven't entirely taken over my system, this time, but close. I can also confirm that hitting the skip button forces the entire procedure to run in the background, thus rendering the game unplayable for ... well sometimes it seems permanent until an entire system reboot.

[kakra]

@jakogut Latest commits have a memory optimization which should mostly eliminate it. If you know which game has shaders that cause 64 GB of memory usage, you might want to compile the current version, copy the shader database from Steam to a new location, and run fossilize-replay on this test set with --progress on cmdline (to force it into the same mode that Steam uses). I still need to test this but the changes are already merged so I think they'll work.

It'll still cause a lot of write-back pressure in the page cache which may trigger the kernel to swap data out. But that's a whole different kind of beast that cannot be properly solved currently without dirty hacks. If your IO subsystem is fast enough, it should be no problem. If the updates landed in the Steam distribution, you'll probably see that fossilize-replay child processes will show up as "exe" instead of "fossilize-replay" in top/htop (may depend on settings) - then you'll know that the updates landed.

The new updates will also change scheduling behavior and readahead behavior of the process which should further improve things.

[kakra]

@jakogut Oh I see you're using zswap... You may want to disable that. At least for me it tends to fill the swap with trash and probably end up with useless compressed data in RAM, in the end thrashing the page cache. Or at least use a much smaller memory pool: The default of 20% is just too high for modern systems. I'd go with something like 5%. Also, oomkiller should not be needed, modern kernels are quite good and should not touch interactive processes. You can verify that by adding oomscore column to htop (oh I saw you did that). Using uksmd may be more beneficial if you're using a lot of processes with potentially duplicated memory.

[HansKristian-Work]

Fossilize should already be limiting its memory usage fairly aggressively as this is a known problem. Not sure why this happens in Steam.

[DarkDefender]

Are you able to reproduce it on your end? (With the steam beta)

[HansKristian-Work]

Haven't triaged it, but I'll look at it.

[HansKristian-Work]

Ok, I tried replaying locally and I can reproduce rather large memory consumption when replaying on the 5700xt. This does not happen when I run with a dummy device, so the memory consumption must happen in the driver. I suspect it's retaining a lot of memory in the disk caches, and multiply that over lots of processes, it becomes a problem. I've seen this issue before, but to my knowledge it was fixed in Mesa a while back.

Seems like I might need to poke around to see where it's hogging so much memory ...

[DarkDefender]

If it is any help, I am currently using and building mesa from the git master branch.
So I am seeing this with the absolutely bleeding edge drivers.

[rLy07]

Hey, I didn't know that this has been tracked here before I opened my report at steam for linux. As you can see this effects Shadow of the Tomb Raider as well. Also I'm not sure if the following should be treated as a separate issue (or if is an issue at all) but background processing happens every time I start steam even when there are no updates to the games. On top of that today SotTR run twice(few other games run between the 2) without restarting steam.

[Samega7Cattac]

Noticed from time to time it starts to release some memory (intended, crash or terminated?) but the consumed memory is by far higher than the released so it will still happen.

Arch Linux
kernel version: 5.7.2-arch1-1
mesa version: 20.1.1-1
Graphic card: AMD R7 m265

How many processes and threads should it be using in normal conditions?

[lewisdiamond]

I'm also on Arch Linux:

Operating System Version:
    "Arch Linux" (64 bit)
    Kernel Name:  Linux
    Kernel Version:  5.6.15-arch1-1
Video Card:
    Driver:  X.Org Radeon RX 580 Series (POLARIS10, DRM 3.36.0, 5.6.15-arch1-1, LLVM 10.0.0)
    Driver Version:  4.6 (Compatibility Profile) Mesa 20.1.1
    OpenGL Version: 4.6
    Desktop Color Depth: 24 bits per pixel
Number of Monitors:  2
    Number of Logical Video Cards:  1
    Primary Display Resolution:  1920 x 1200
    Desktop Resolution: 4480 x 1440
    Primary Display Size: 20.39" x 12.76" (24.02" diag)
                                            51.8cm x 32.4cm (61.0cm diag)
    Primary VRAM: 8192 MB

Memory:
    RAM:  16007 Mb

I have installed earlyoom to kill anything that is not steam/dota2 and almost all other processes get killed and processing vulkan shaders takes the full 16GB, barely anything left for the system to run.

[arcsaber]

I'm having the same issue on KDE neon with an RX570 (Mesa 19.2.8). Doesn't even help to launch the game with -gl, already got two abandons because I didn't manage to restart Dota within five minutes after a crash, even though I had killed all steam processes in htop.
How to fix this mess?

[rLy07]

I'm having the same issue on KDE neon with an RX570 (Mesa 19.2.8). Doesn't even help to launch the game with -gl, already got two abandons because I didn't manage to restart Dota within five minutes after a crash, even though I had killed all steam processes in htop.
How to fix this mess?

You can probably turn off background shader processing for the time being.

[arcsaber]

You can probably turn off background shader processing for the time being.

I never had it turned on in the first place. I updated to latest oibaf drivers, let the pregame shader processing run through and am going to test this in low priority now...

[rLy07]

You can probably turn off background shader processing for the time being.

I never had it turned on in the first place. I updated to latest oibaf drivers, let the pregame shader processing run through and am going to test this in low priority now...

I don't have Dota so I don't know it then. Does it have a separate thing for shader processing from the steam wide one that was introduced recently? Or does the shader processing supposed to run at the start of a game? I haven't played SotTR for a while so I don't know if it would run at the start and I haven't really noticed it with other games but those usually only take a few seconds to compile.

[arcsaber]

Does it have a separate thing for shader processing from the steam wide one that was introduced recently?

No I have never seen this shader processing until recently. I made the following observation, there are two types of launches for me:

• I start Dota and the shader processing basically skips itself after a few seconds, I can't see any fossilize process in htop
• I start Dota and fossilize tries to do its thing, but after around 40 minutes it's at around 15% and has already filled up my 16gig ram and further is 1gig into swap. I cancelled it.

So I have to suspect that when I skip shader processing, rarely fossilize starts itself in the background and wrecks my system. I'll occasionally check htop while playing and edit this message if I'll witness anything.
Edit: yep, after a fresh restart I open steam, start Dota, shader processing closes itself after about five seconds, but about five minutes later i found that fossilize is already at 4gig ram and I killed it.

[Samega7Cattac]

@arcsaber there's an option on dota settings to disable "compute shaders", try that

[arcsaber]

@arcsaber there's an option on dota settings to disable "compute shaders", try that

Thanks, I tried but after launching Dota fossilize still starts.
On a side note, once I close Dota (e.g. if there's an update), I can't launch it anymore, neither can I stop steam, have to kill the process and restart steam. But that's less of a nuisance.

[Plagman]

When you observe this, do you see the memory usage after Dota has started and when it's shown its window? Are there any contents in the window or just black?

When that happens do you see the memory being consumed by fossilize-replay processes, or Dota itself?

[HansKristian-Work]

I narrowed down a few things:

• When Fossilize does not use its own pipeline cache, Mesa maintains an internal pipeline cache in memory which becomes massive over time.
• RADV_DEBUG=nomemorycache is a workaround.

#85 should fix the memory explosion problem. --log-memory for me is now massively improved and the memory usage seems to be bounded.

[DarkDefender]

@HansKristian-Work Any ETA for when it will be in the steam beta?

[HansKristian-Work]

The reporter said it was already in Steam, I don't know anything beyond that.

[DarkDefender]

Hmm, I still have the issue then. I just tried 2 hours ago. (Steam says that I'm up to date, and I'm opted into the beta client program)
It seems like it is still slowly eating all my available ram when building the shaders for dota2.

[Saroumane]

Today, 10:40 AM : PC started (cold boot) : Ubuntu 20.04, Radeon RX5700 XT. Steam autostarted.
... : browse web
11h14 : all RAM and all swap are used by "fossilize_replay", has to quit steam.

1st time I see this.

[DarkDefender]

I just wanted to chime in and say it seems to be solved for me now.
It is both a lot faster and only takes around 1 GB of ram or so it seems.

I'll leave this open so others can confirm as well.

[arcsaber]

The hero selection screen didn't load for me, more specifically only the top row of empty rectangles loaded and I had sound (you may now select your hero), screen was black otherwise. I checked processes and found fossilize_replay. I got an abandon because I couldn't reconnect quickly enough and have to wait an hour. =( Seems like I have to check for fossilize every single time.

Edit: I also got low priority, so because of this mess I have/had to play at least five matches in low priority, that's at least around 2,5 hours of my life wasted in toxic environments for bugs which are out of my control. Come on guys...

[Samega7Cattac]

I'm also using the last version (Jul 6)
this time it managed to finish processing shaders, but using ~11Gb (at least not all 16Gb)

[alexeysvrv]

@arcsaber I have same problem with selection screen. And system same give me low priority

[Heryhelder]

I am facing this problem today playing dota 2.
Using Fedora 32 KDE Plasma
Ryzen 5 2400G
RAM 6gb

[rLy07]

For me it seems to be solved when compiling Shadow of the Tomb Raider shaders. After yesterday's update there were a couple of shader downloads and processing and while memory usage was a bit higher for SotTR than for other games, it stayed around 6-7GB(total usage at the time, before processing was around 4GB).

[arcsaber]

Yesterday and the day before I had some fossilize and loading issues with Aghanims Labyrinth, but everything seems fixed now and things are running smoothly as they should. Didn't try vanilla Dota yet, but I guess there shouldn't be a difference.

Thanks, devs!

[fmartins-andre]

Same issue here. Just saying

[koreanfan]

I use opengl renderer for dota 2, i even dont have vulkan dlc installed, i have checked only enable shader pre-caching. When i launch dota shows vulkan shader cach window ( why this window shows up if i use opengl and i dont have installed vulkan dlc), so i skip this, after that i start play dota. In anytime fossilize process hang my pc, so i got afk in game and i got penalty for afk, but this is not the end, after this match ended, other 4 teammtes report me for disruptive gameplay, and i got ban for 1 week, but this not the end, i got same ban for one week for my 3 acc (rank immortal 52, rank immortal 80 , rank immortal 95). So i create new acc and start to play 100 hours for enable ranked match, but this not the end, when i start first game on this new acc, your perfect process fossilize again destroy my gameplay and i got reports. Keep your job, and later when my 3 ban ended, with 100% warranty i can said that i will get reports again and i got ban for 6 month cuz we have your perfect fossilize, keep doing your great job, thumbs up.

[koreanfan]

AHAHH, are you serly. I comment this cuz of your fossilize hang my pc and i was afk, and got penalty and 4 reports and got ban for a week on my 3 acc (immortals) and you marked as disruptive content this post. AHHAHAHAHAH . Your fossilize is disruptive must be marked as disruptive code.

[fmartins-andre]

man, my PC is taking 1-2 hours to finish the fossilize routine! I only play Dota, once or two times a week, and the need to wait this amount of time to play one or two matches is unviable! This is preventing me to game!

[arcsaber]

Edit: disregard this, hero selection didn't load again and I got an abandon. I've reverted to oibaf drivers, I didn't have any issues with those and Aghanims Labyrinth loading time seems 10-20 seconds shorter with oibaf too.

Hi guys, long story short: I installed kislak-mesa drivers and now everything seems to work as intended, even smoother than with oibaf drivers. I don't know if anything has been updated in the steam code in the meantime, but anyone still having trouble with fossizile should get kislak-mesa drivers (I installed it because it's recommended for steam proton, completely unrelated issue). Cheers

[pfwb]

Just another 'me too'.

OS: openSUSE Tumbleweed 20200807
GPU: AMD 270X (RADV)
RAM: 16GB
Steam Build: Jul 31 2020

It barely finishes without bringing down the entire system, consuming almost all available RAM.

[xggrnx]

Same problem.

CPU: Ryzen 5 3600 RAM: 16GB OS: Calculate Linux 20.6 Kernel: 5.4.45

Got LP T.T

[MajorGonzo]

Same, and I don't have DOTA. I have Warframe, and it had a large update, so it's probably processing for that. Worse than using all my ram, and locking all 12 of my cores, it was cooking my CPU, which was running at 90C.
Forced a shutdown and then rebooted. Now just using 3-4 cores, and as I'm typing this, it finished, after about 15 minutes.

Ryzen 5 3600, 16GB Ram, Ubuntu 18.04 kernel 5.4.0-42.

Until this is fixed, I'm gonna opt out of the beta, which is where this comes from, I believe.

[marianomacchi]

I've had the same issue with Path Of Exile, that was being processed by fossilize while I was playing another game (Quake Live).

OS: Fedora 32
KERNEL: 5.7.14-200.fc32.x86_64
CPU: AMD Ryzen 3 3200G with Radeon Vega Graphics
GPU: Radeon RX 580
GPU DRIVER: 4.6 Mesa 20.1.5
RAM: 16 GB

I haven't noticed if it eats RAM (will keep an eye on it) but when I boot up there's usually 3 fossilize_replay processes that are hammering the CPU and they keep hammering for hours until I get annoyed from the CPU fan getting loud and kill them.

OS: OpenSUSE Tumbleweed (kernel 5.8.1)
CPU: AMD Ryzen 5 3600
GPU: Nvidia GTX1060 6GB (driver 450.57)
RAM: 16GB (no swap)

Edit:
Looked at top, memory usage seems stable, all 3 processes seems to be using about 0.9GiB (~5 in the %MEM column) each, all 3 constantly have 100 on %CPU, they also are from the same command so based on the game ID I can see in the command all of them are for Path of Exile.
Now my computer doesn't freeze, lag or run out of memory but those 3 processes have been running for 54+ minutes. This doesn't seem normal, maybe there's some incompatibility with Path of Exile?
If I kill those processes about 5 new ones spawn (few at a time), run for less than 10 seconds and done, my system can chill again.

[arcsaber]

I just got a joinbug (black hero selection) with oibaf drivers, I'm officially out of ideas.

As a workaround: after killing dota2 it takes a ton of time to validate the game (I have it on a hdd too). Is there any way to prevent this? I think it'd make me able to rejoin the game in time. I'll also start making a botgame first before starting ranked, maybe it will help.

[amitavmohanty01]

I see 4 fossilize_replay processes and 1 one of them is always taking up ~100% CPU. So, this processing is taking up a lot of CPU resources.

%Cpu(s):  2.1 us, 11.4 sy, 22.7 ni,  1.9 id, 60.9 wa,  0.3 hi,  0.6 si,  0.0 st
MiB Mem :   7830.8 total,    485.0 free,   3743.9 used,   3601.8 buff/cache
MiB Swap:   5120.0 total,   5104.0 free,     16.0 used.   1296.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                              
   5515 dknight   39  19   54076  50632   3724 R 100.0   0.6  14:18.93 fossilize_repla                                                                                                                                                      
   5552 dknight   39  19 1145084 265332  11632 S  18.3   3.3   1:41.50 fossilize_repla                                                                                                                                                      
   5554 dknight   39  19 1145092 268600  11736 S  12.3   3.3   1:45.07 fossilize_repla 

I don't think rendering the system frozen is in anyway helpful.

[RawPorkSushi]

I've intermittently had this problem launching Borderlands 3 on Steam via Proton. (I can give more system detail if wanted).

The most recent time was on Aug 22nd, and may have been associated with a recent update. There were about a dozen fossilize processes, one or two of which taking up >100% of my CPU (I assume that means multiple cores). After it completed (which took roughly 30 minutes), the game was unplayable and eventually crashed.

The game has worked fine at least once since this event, without any issue.

[CRCinAU]

This isn't just DOTA - I've seen it with X-Plane, GTA V etc etc.

I just ran out of memory and even my browsers crashed on a system with 32Gb of RAM and zero swap.

Processing in the background helps - but not when you actually want to play something....

[amitavmohanty01]

The processes remain busy even after Steam is dead. That is just plain obnoxious.

[alvaro893]

This is driving me mad. Every time I open steam I face this issue, I don't need to start any game. In 5 minutes has got most of my RAM.

Operating System: Manjaro Linux (Archlinux)
KDE Plasma Version: 5.19.4
Kernel Version: 5.8.1-3-MANJARO
Processor: AMD Ryzen 5 3600X 6-Core Processor
Memory: 15.6 GiB of RAM
Graphics Processor: Radeon RX 570 Series ( AMDGPU, Mesa 20.1.5 )

[CountClavicula]

I have just made a Github account to report the same issue with X-Plane. My computer becomes unusable (freezing) after starting X-Plane. When I enable Background Processing, I hoped that I would be least be able to use my PC for internet, but fossilize still occupies all my memory and swap space (16 Gb) and the PC gets stuck.

My system specs:
`OS: Manjaro Linux x86_64
Kernel: 5.4.60-2-MANJARO
CPU: AMD Ryzen 7 3700X (16) @ 3.600GHz
CPU: AMD ATI Radeon RX 5700 XT
Memory: 7563MiB / 16022MiB '

Is there a way that I can for example manually specify how much memory fossilize is allowed to take for the processes? I prefer longer waiting times to a freezing PC.

[John-Gee]

@CountClavicula I don't know if fossilize has an option for that, but I got curious and found this thread with options that could allow you to override that yourself: https://unix.stackexchange.com/questions/44985/limit-memory-usage-for-a-single-linux-process.

[amitavmohanty01]

@Plagman Yes, I have seen that happening while playing Dota 2 with shader pre-caching enabled. Please tell me what details you would want from me.

[alexeysvrv]

@Plagman When I play Dota 2, even if I clicked "Skip" updating the Vulcan shaders when starting the game, sometimes this process (or rather 4 pieces) starts and after a short time the computer starts to slow down, so I often get LP in Dota 2. I have a hard drive, because of these processes, the game takes a very long time to load. By the way, this process takes a lot of time on my computer (if you don't skip it when starting the game). About 1,5-2 hours. Maybe it depends on the fact that I have a fairly old 4-core processor and 4 GB of RAM.

[kisak-valve]

Hello @xggrnx, that workaround is outright hazardous, not easy to understand to a new user, and given without sufficient warnings, which is why I've hidden it.

You could have just as easily suggested disabling shader pre-caching or removing <steamapps folder>/shadercache/570/ which is much clearer.

[simvux]

@Plagman Can confirm that pressing the "skip" button instead causes the processing to be done while the game is running (and/or starting).

Since it eats all resources the game usually remains frozen in the main menu until the processing is complete.

I haven't however had it randomly occur while in a game or anything like that.

[shadowman79]

Fossilize remains after DOTA 2 process killed in terminal and closing of steam in UI. - it has 25% CPU load (i3 CPU 550 @ 3.20GHz). It prevent system to open browser and have in-time response from system UI somehow. That's why to have system working in this scenario I had to reboot system in additional after killing of DOTA & closing steam.

[simvux]

This appears to have been solved in Dota for me. The fossilize pre-processing step is both a lot faster and only puts all my cores to 80-90% thus allowing my system to continue operating like normal. I also now have a fancy progress bar during that step.

Probably was fixed in some update? If not I did also reinstall Dota recently so who knows...

[RawPorkSushi]

Just launching Borderlands 3 today, came across the Fossilize instances devouring all my resources once again. It took nearly exactly 11 for most of the Fossilize instances to finish, which then allowed the game to load properly after that. One instance continued to run in the background for about 2 minutes after the game loaded.

Here's a screen cap showing the Borderlands 3 executable running (but the game hanging on the black screen) plus the Fossilize instances using up most of the system:

All in all, this wasn't the absolutely worst example, but it was definitely hella annoying to have to wait that long.

[Samega7Cattac]

Needed to make a large one this time.
Used 50% of my CPU and RAM was more than ok.
Still I would prefer that ppl could change those options.
For me I prefer to have him use all threads and go grab some snacks.
And if increasing the RAM limit would make it faster I would do it too since during process I have 12Gb of RAM laying around.

[Samega7Cattac]

any probability this being related to this dota2 issue

[Samega7Cattac]

fossilize running just by starting steam
Maybe we should create another issue for this?

[kisak-valve]

Hello @Samega7Cattac, Steam having Fossilize use 25% of the overall cpu cores when the background processing of vulkan shaders setting is enabled is the normal intended behavior. That is not a bug.

[Samega7Cattac]

I tho I had it disabled for testing but it's indeed enabled mb

[shadowman79]

It seems that hang in DOTA2/forssilize issue with has been resolved with some of update, about 2 weeks it's working fine.

[pfwb]

It seems that hang in DOTA2/forssilize issue with has been resolved with some of update, about 2 weeks it's working fine.

Yep, it seems so. I just tried it with Vulkan and no more high memory usage. No more flushing the entire disk page cache!

[dchrzanowski]

Thanks for the reports. The case where any fossilize_replay processes are running while a game is running is not intended. Is anyone able to reliably reproduce such a scenario?

The problem is the inconsistency of this bug. When I launch dota I see the fossilize_repla processes spawn, a dialog window pops up with the shader processing that I always skip, because it never fully completes. After pressing skip the fossilize_repla processes are killed and the game starts. Now, these fossilize_repla processes sometimes are spawned again. These respawns happened the most after playing for a while, a few times during gameplay and sometimes after the game is closed and/or steam as well.

To recover the system lag I do a killall fossilize_repla and the stability is regained.

I could be imagining things but I'm half thinking that this issue started since I started using Discord. But I fail to see how these could be connected.

[demoth]

the problem with vulkan shader compilation is still there. for example for Path of Exile it just never finishes, 100% use of all computing power and after couple of hours the progress is going back and forth. System info https://pastebin.com/j6wWFuvd

[Skiski]

I also have this problem since I installed Horizon Zero Dawn. It takes arounf 1h to compile and it makes my system totally unresponsive (like, the mouse cursor is not even moving) for tens of minutes sometime. When Steam download the shaders, it's almost 4Go, so it looks like it's compiling everything each time.

I've got a Ryzen 3600 and an Nvidia GTX 960, with proprietary drivers (xorg-x11-drv-nvidia-455.28-1.fc32.x86_64).

Here is an example of usage:
`
top - 11:28:24 up 1:02, 1 user, load average: 4,35, 4,76, 4,70
Tasks: 336 total, 2 running, 333 sleeping, 0 stopped, 1 zombie
%Cpu(s): 8,8 us, 0,4 sy, 24,8 ni, 65,7 id, 0,0 wa, 0,2 hi, 0,1 si, 0,0 st
MiB Mem : 16015,3 total, 195,4 free, 7109,4 used, 8710,4 buff/cache
MiB Swap: 20000,0 total, 20000,0 free, 0,0 used. 8145,9 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                

3864 fedoski 20 0 1578640 936796 85400 S 100,3 5,7 51:27.91 steam
7985 fedoski 39 19 362592 229688 73176 S 100,3 1,4 1:08.57 fossilize_repla
7983 fedoski 39 19 362592 230780 73244 S 100,0 1,4 1:08.75 fossilize_repla
7984 fedoski 39 19 362592 231932 74412 S 99,7 1,4 1:08.92 fossilize_repla
`

[J4nsen]

I've the same problem with HZD on Arch Linux/Nvidia(455.28-1). Compiling the shaders locks up my system with >40GByte free ram.

[kakra]

Fossilize should already be limiting its memory usage fairly aggressively as this is a known problem. Not sure why this happens in Steam.

My findings show that it's dominating the page cache which will inadvertently hurt IO latency of unrelated processes a lot and cause swapping of anonymous memory used by other applications just to blow up the cache for data that's probably only read and written once. As a background service, fossilize_replay needs to manage its page cache usage and amount of dirty writeback data. I've proposed #101 trying to fix this. It also adjusts scheduling settings because I found it dominating the CPU shares, too, despite being nice=19. This is due to group scheduling that the kernel uses for some years now: Nice only has an effect among the processes belonging to the same group (#99).

I suspect it's retaining a lot of memory in the disk caches, and multiply that over lots of processes, it becomes a problem. I've seen this issue before, but to my knowledge it was fixed in Mesa a while back.

@HansKristian-Work Yes, that's exactly what happens. Does mesa write its own files, like the nvidia driver does? As pointed out in #101 the cache thrashing of writes is a bit more difficult to tackle. It would probably need fsync() and/or cache hinting be integrated right into those libraries.

cgroups memory accounting is a good tool to see that a vast amount of memory used can be accounted to caching. It may be a good tool to limit the memory usage to a sane value without modifying third party libs. However, limiting the upper memory bound of a cgroup may cause its memory to be swapped out and make the cache too small, resulting in it thrashing the disks instead of the cache. Usually it's better to give other cgroups a lower memory bound, so their cache isn't taken away by other processes. But we cannot control the others, just ourselves. Maybe fossilize could rate limit its children that cause writes? It could watch the load or PSI to see if write latency increases and then throttle itself. But I don't see a sane way of auto-tuning itself by watching the load. Users would probably have to provide a loadavg target manually.

It could work when using PSI, tho: A thread could write to /proc/pressure/io (or the proper cgroup item) to register an event until stalling surpasses a defined latency, then pause its children. But not sure which baseline to choose because HDDs show very different results of SSDs.

[kakra]

@HansKristian-Work It looks like the children spawner currently divides the cache by worker threads, then feeds each slice to a worker which in turn may run multi-threaded, giving a fixed size of workers per cache with variable sized slices. This creates a lot of pressure on the IO subsystem if all threads start reading and writing in parallel.

Maybe it would be better to divide the cache into a fixed size of variable amount of slices, then queue them to a spawner that dynamically forks children based on a loadavg estimator that targets a specific load, with an upper limit of 75% of cores. The current load can be easily calculated by extrapolating it from the current load values which would have a too coarse time resolution for this to properly work. This should auto-tune quite well to what the IO subsystem is capable of.

For this, the slices should be small enough to still have a quick reaction to load changes, so each slice should be finished within a few seconds.

Is there some chart how spawning of workers and threads works?

[kakra]

@HansKristian-Work Why is it using 1 GB of memory per process? Imagine a high-end processor with 32 cores or more: Would it be taking 32 GB of memory then? Or is this depending on the database size which is 9 GB on my test data with 8 cores?

Whatever it is, we also need to consider this scenario: A high-end processor with 32 cores would create a huge amount of write-back pressure for the file system. So I'm currently thinking of throttling the write-back somehow. But if each process would still account for 1 GB of memory, this is kinda futile... So we'd probably need to estimate the amount of children as a compromise between memory usage and cores. My cache hinting patches already show an improvement as it no longer increases swap usage on my system (which went to 2+ GB without the patches) but it still slowly increases because we don't throttle write-back yet.

When running the multithreaded version of fossilize-replay, only uses slightly more memory than a single process of the multiprocess version. So maybe it should allocate that memory before forking? I'm not sure if that's possible but there should be a way to make all children share their memory? Or simply give up on the multiprocess version and even when running from Steam it should use the multithreaded version? What's the rationale of multithreaded CLI vs. multiprocess Steam client children?

Update: Looking at the code, it looks like each slave process does a lot of identical memory initialization, something that adds up to 1 GB per process. I think this should be done before forking the children, so they share the same allocated memory. This should be save because writing to shared memory of forked children will fault on write and duplicate the memory page. OTOH, why is --progress working with a master/slave model anyways? Couldn't it just run a single multithreaded child?

[kakra]

Tracing the processes shows that the vast amount of writes come from the NVIDIA driver caching the shaders. fossilize-replay itself doesn't seem to write data at all (except to stdout). @HansKristian-Work Is that correct?

[Mayurifag]

Is there any workaround to shut up the fossilize_replay processes?

[kakra]

@Mayurifag What do you mean? For tracing?

[Nisk]

This IS NOT fixed by a long-shot. Happening to me on kernel 5.9.1with nvidia 455.28 driver on 1080Ti.
I have Ryzen 2700X and 16GB of RAM, as soon as this kick-off my RAM usage jumps to 100% and system becomes unresponsive.

[folknor]

Using the steam client beta, I just noticed this process today. For me it's not using more than 200mb of memory combined between 2 processes, but it's using 100% CPU while running.

Total runtime was roughly 3-4 minutes, more than enough time for me to notice something was going on and noticing the process, Googling it, and coming here to comment.

EDIT: Judging by the command line, it was processing 578650, which is "The Outer Worlds", a game I've just installed 2 days ago.

[kakra]

Usually, you would not notice it. But there's a caveat with how modern kernels and desktop environments work which make it more perceivable, especially since the introduction of pressure-vessel. And RAM usage depends on the game it's processing, and there are a few games that will cause it using gigabytes of memory.

The CPU usage should be fixed when the next update gets pushed with Steam: It'll still use 100% CPU but it will give up this share more easily so you should not notice it any longer. The memory issue is more complicated because it's a combination of duplicated memory and cache domination (with 32+ GB RAM, you probably only observe the cache domination issue). This latter issue can look like high CPU usage in the best case but stall the complete system for multiple minutes in the worst case.

For solely the CPU usage, you could try disabling process autogrouping in the kernel until the updates have landed in the Steam client:

sudo sysctl -w kernel.sched_autogroup_enabled=0

or maybe better disable it at boot time by adding a kernel cmdline parameter to your bootloader:

linux ... noautogroup

How to add this to the bootloader or automate sysctl at boot time in your distribution is beyond the scope of this comment.

Hint: Disabling this setting can make distribution of CPU shares more unfair in desktop environments when running apps that spawn a lot of processes, i.e. building software from source.

[MajorGonzo]

I've been monitoring this for a while now (created a cron to capture cpu usage and processes every minute), and it runs two different ways. Sometimes, it just runs on a schedule, and in this case, only uses 3 of my 12 cores, all running at 100%, and takes about 10 minutes. However, if I start Warframe, and it decides it needs to run first, then it uses all 12 cores at near 100%, and consumes about 4-8 GB of my 16GB memory, and takes about 2-3 minutes. Because this was cooking my CPU at 90C, I bought a Hyper 212 Black cpu cooler, and now my Ryzen 5 3600 stays at about between 72 and 75C during the process.

[kakra]

Yeah, I've seen it consume 21 GB of memory (includes cache usage when measured with cgroup memory accounting, PSS itself was around 10 GB) with 100% CPU across 8 cores for multiple hours when it processes the PoE shaders. This behavior manages to push 3 GB of RAM into swap, resulting in laggy desktop usage the next day (and while it is running). So it manages to even slow down a 32 GB system.

Thus I started to analyze this and submitted some patches, the IO and CPU priority patches have been merged. RAM and cache usage is more complicated and also comes from external components (GPU driver) that fossilize cannot control itself. Fixes and work-around are being worked on but it probably takes some time.

[Saroumane]

There is something I don't understand : If we want to avoid this shader precompilation, shouldn't we benefit from shaders downloading ? Or we can only download OpenGL shaders, whereas Vulkan shaders must be processed locally ?

[kakra]

@Saroumane Shaders will be compiled per driver and GPU individually, what gets downloaded is just a capture of the pipelines. Fossilize will replay those on your driver and GPU so the GPU driver will cache them in its local shader cache. That way, the shaders will no longer be compiled on demand during gameplay which reduces stutters, also games may load faster because they don't have that much work to do when doing their initial shader caching at startup. Being an intermediate layer, DXVK does something similar: It records all pipeline states it came across and replays them in advance at next start.

So actually, it's not shaders that are downloaded but a pipeline state cache that was recorded while the game was running - at least that's my understanding. That also means, it can't be avoided unless you accept gameplay stalls when shaders must be compiled by your driver on demand mid-game.

[jakogut]

I had the same problem yesterday, fossilize_replay chewed through 64 GB of RAM, and the system was unresponsive for ~5 minutes before the OOM killer took over. If the OOM killer reaps the wrong process, this has likely potential to cause data loss.

Could fossilize_replay be run under a separate user with a cgroup that has a memory limit, so this doesn't happen anymore?

[kakra]

@jakogut On my test set with PoE shaders, the processes use 8x 1.2 GB of RAM with additional 200 MB shared. That's around 9-10 GB PSS. When running inside a cgroup with memory accounting, you can notice that it accounts for 21 GB of memory, and that's due to disk caching. That's over 11 GB of disk cache, with a good part of it being write-back (aka dirty) cache. That amount of write-back has potential of dominating your cache and still slow the system down. If your system is swapless, that means it will discard your precious disk cache from other applications because it has no way of swapping out idle/inactive anonymous memory, with swap it mostly depends on vm.swappiness if it prefers discarding disk cache or swap out anonymous memory.

[jakogut]

@kakra Make that ~80 GB immediately after opening Steam. 🤣

Is it chewing through so much memory because of the number of threads? Maybe a thread limit would be beneficial?

I'll try out your previous suggestion and see if it helps, thanks.

[alexeysvrv]

@kakra Please explain in simple terms why this process eats up all the RAM, why is this happening? What does this process do with shaders? Kind of like the video driver compiles the shaders?
Do you need major architectural changes (rewrite all code) to completely get rid of the problem?

[kakra]

@jakogut Please don't use the VIRT column in htop to measure the memory: That's just allocated memory which may actually be not used at all. RES is resident memory which actually is memory present in RAM. Memory swapped out MAY account for "VIRT" (but VIRT can also be memory that was never used). "SHR" is the amount of resident memory that is shared with other processes, so it counts only once. Thus, PSS (proportional set size) is a much better measure to take. htop cannot show that, but some desktop task monitors do, and atop does (which I use to constantly monitor my system).

What htop shows as "memory used" is cached + buffers + resident memory. Most of the time, most of it is "free" memory because the kernel can easily discard it for resident memory allocations.

@jakogut Your system looks strange... Could you sort htop by RES? Having 12 GB swap with 74 GB used RAM doesn't look right given the process list below. But looks like Death Stranding makes even bigger shader state caches as PoE... Would be a nice test set if I owned that game.

[alexeysvrv]

@kakra And tell me, can this be the reason that the game (Dota 2) using Vulkan shows an unstable frame rate? Maybe it's offtopic, but when rendering an opengl, the fps is stable on my system 40-50. On the Vulkan render, the frame rate is usually 25-30 and changes abruptly up and down. Or are these already questions to the source 2 engine itself?

[jakogut]

@kakra

@jakogut Please don't use the VIRT column in htop to measure the memory: That's just allocated memory which may actually be not used at all. RES is resident memory which actually is memory present in RAM. Memory swapped out MAY account for "VIRT" (but VIRT can also be memory that was never used). "SHR" is the amount of resident memory that is shared with other processes, so it counts only once. Thus, PSS (proportional set size) is a much better measure to take. htop cannot show that, but some desktop task monitors do, and atop does (which I use to constantly monitor my system).

I'm familiar with RES, VIRT, and SHR. Not so much PSS, so thanks for that. Htop does actually support it, it just wasn't enabled in my config, so I've gone ahead and added it.

@jakogut Oh I see you're using zswap... You may want to disable that. At least for me it tends to fill the swap with trash and probably end up with useless compressed data in RAM, in the end thrashing the page cache. Or at least use a much smaller memory pool: The default of 20% is just too high for modern systems. I'd go with something like 5%.

I've just disabled it.

Also, oomkiller should not be needed, modern kernels are quite good and should not touch interactive processes. You can verify that by adding oomscore column to htop (oh I saw you did that). Using uksmd may be more beneficial if you're using a lot of processes with potentially duplicated memory.

Hmm. Well, I had earlyoom enabled because fossilize_replay was consistently exhausting my system's memory regardless of how much I have. I'd like to try uksmd.

[kakra]

On the Vulkan render, the frame rate is usually 25-30 and changes abruptly up and down.

I'm not playing Dota and don't know what may be wrong here. Since Vulkan apps can render on multiple cores in parallel much more efficiently, you may very well see fluctuating fps if another background service is running at the same time. This should be mostly fixed by the batch-scheduler patches by me that went into the code. Also, autogroup-scheduling kernels are supported better now. But I don't know when Valve will deploy the updated fossilize-replayer. OTOH, 25-30 fps with Vulkan while 40-50 fps with OpenGL looks like either a problematic driver or the game is just not optimized very well for Vulkan. So you may actually see a very different problem and fossilize is not at fault for that particularly. You should wait for the updated fossilize and see if it improves things.

[kakra]

Not so much PSS, so thanks for that. Htop does actually support it

Oh really? Didn't know. Will enable! :-)

PSS uses the RES size of the process, then adds the SHR size roughly divided by the amount of processes that share the same memory. This is an estimation as it is difficult to find which process shares memory with another. But it gives an educated guess of how much memory that process is really using in proportion to other processes in the system, IOW, if you kill that process, you may get roughly that much memory back.

I've just disabled it.

Let me know if it improves things. It did for me but it has been a while since I disabled zswap - it was only causing problems for me (to the point where it managed to confuse btrfs). Also, latency behavior of the system was a lot worse under load. uksmd is a lot cheaper, and usually deduplication can achieve much higher compression rates. But uksmd is depending a lot on bit-to-bit identical copies of memory while zswap just removes redundant bits by compression. So it's mostly useful for VMs or when running a lot of processes that load the same data (fossilize doesn't, it loads similar but not identical data into its memory). But compared to zswap, uksmd shared pages are still resident in memory and can be used by processes without swapping or page-faults.

[kakra]

Not so much PSS, so thanks for that. Htop does actually support it

Oh really? Didn't know. Will enable! :-)

Hmm, I cannot find that. Maybe I'm using an older version of htop? Or do you confuse that with PERCENT_MEM? That is purely based on RES.

Update: Nevermind... Upgraded to htop 3.0, and now it has PSS. Nice. Hmm, it looks different to what I expect, tho. atop -1RAp shows different values. That's probably due to how both sum and divide, htop show one line per process, atop can group processes.

[kakra]

I'd like to try uksmd.

May need a kernel patch: https://lwn.net/Articles/788633/

I should add that to my custom kernel. If it proves useful to fossilize, it may be worth enabling ksm memory advise in fossilize so uksmd (and the patch) would not be needed.