Steam keeps crashing when fossilize process "Deus Ex: Human Revolution - Director'cut" shaders about fossilize

[kakra]

I wonder if there was something broken in the shader db files that made fossilize crash. Uninstalling the game probably removed the shader db, too. And so it started back from a clean state. May have been enough to just clear the shader cache directory instead. I've seen similar problems before when games still used one shared cache, I needed to clear ~/.nv to prevent DXVK from crashing with an out of memory error. This was before fossilize was introduced.

[kakra]

You should at least run with a small swap file, not without any swap at all. Otherwise you'll give the kernel a hard time to find memory under allocation pressure. Also, please check if your system enabled zswap, and disable that.

[Saroumane]

Sorry If I was not clear, I edit : no swap was used (but I have a swap file of 8 GB of available)

[kakra]

Could you check if zswap is enabled and its pool size:

# grep '^' /sys/module/zswap/parameters/*

[Saroumane]

I have never heard about it before :

grep '^' /sys/module/zswap/parameters/*
/sys/module/zswap/parameters/accept_threshold_percent:90
/sys/module/zswap/parameters/compressor:lzo
/sys/module/zswap/parameters/enabled:N
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/same_filled_pages_enabled:Y
/sys/module/zswap/parameters/zpool:zbud

What do you think ? It seems not enabled.

[kakra]

@Saroumane Well, that's actually memory compression. Instead of sending pages directly to swap, "zbud" will compress two separate pages down into one page, achieving a compression of exactly 2:1 for compressible pages (many are). These "buddy pages" are then stored in the swap cache pool. No disk IO was caused until now. It will keep a dynamic pool of up to 20% of RAM by default, after which compressed pages may be swapped to disk. While the transparent handling of swap pages is a stable kernel feature (i.e. data centers can use remote swap that way), the zbud compression feature isn't. As such, there are a lot of side effects that may not play well under memory exhaustion, leading to situations that you found: high cache usage but memory allocation fails. This is mainly a feature to reduce write wear with swap on SSD. Some distributions enable it by default (or make it very easy to enable). IMO, the default pool size is too high for a desktop system (I'm using 5% in a 32 GB system), or at least it's very hard to find a good default value because this totally depends on what workload you're going to apply to the system.

But zswap is turned off for you. So you should be fine.

Additional note: Above description is very simplified, zbud is actually the pool allocator (not the compressor) which is a buddy allocator that holds space for two compressed pages to fit within one page and put it in transient memory (memory which is there but not directly accessible by the CPU), that transient part is a stable kernel feature and can also be used to migrate RAM of running VMs from one host to another (live migration), or to borrow swap from another machine in a server farm over high-throughput network. There's also zbud3 putting exactly three pages into one, achieving compression of 3:1 that way. Getting a page back from transient memory means it will need to get two (or three for zbud3) free pages from RAM and uncompress the buddy page again. This introduces some fragile implications under high memory contention.

Another problem you may be facing is memory fragmentation with transparent huge pages enabled. Please look at:

# grep '^' /sys/kernel/mm/transparent_hugepage/**

[Saroumane]

Thanks for explanation. I guess it's not a feature for desktop end-user like me, or maybe as a desperate action if I lack physical RAM (but with big shortcomings)

Anyway :

$ grep '^' /sys/kernel/mm/transparent_hugepage/**
/sys/kernel/mm/transparent_hugepage/defrag:always defer defer+madvise [madvise] never
/sys/kernel/mm/transparent_hugepage/enabled:always [madvise] never
/sys/kernel/mm/transparent_hugepage/hpage_pmd_size:2097152
grep: /sys/kernel/mm/transparent_hugepage/khugepaged: Is a directory
/sys/kernel/mm/transparent_hugepage/shmem_enabled:always within_size advise [never] deny force
/sys/kernel/mm/transparent_hugepage/use_zero_page:1

[kakra]

enabled = madvise looks fine, there should be no problem with memory fragmentation.

Did you use any "tuning" tricks like changing the swappiness parameter? Which kernel version are you using?

[Saroumane]

After I uninstalled the game and reboot (and re-installed the game), I have no more any crash problems... And the game plays normally.
I checked the shader cache :
SteamLibrary/steamapps/shadercache/238010
contains 40 982 items (1.2 GB)
So I guess the problem was temporary and I close the issue.

For the record, I don't think I have real tuning tricks for RAM or CPU
for the Ryzen CPU, my governors are "ondemand"
for the swappiness of my (unused) swap, I think it's still the default Ubuntu value :

$ cat /proc/sys/vm/swappiness
60

I have this kernel : 5.8.16-050816-generic