TES IV: Oblivion and Fallout: NV terrible performance about wine-nine-standalone

[axeldavy]

I played Fallout NV completly on gallium nine two years ago, and there was no perf issue. Not sure what could cause this potential regression. Could you post a screenshot of your config panel to replicate exactly ? any mod ?

[cyro666]

I had everything set to max, 1920x1080, no mods. I will send you the config panel once I am home.

[cyro666]

I presume this is what you wanted? As I said, no mods, vanilla with all the DLC. Also, not sure why it says R9 290, but I definitely have an RX 590. I also tried turning off vsync, same performance.

[axeldavy]

Yes this is what I wanted. If I cannot reproduce, I'll ask you to give me a screenshot in game with the GALLIUM_HUD set to handpicked statistics.

[axeldavy]

I cannot reproduce.
At first I had indeed 30 fps at some outdoor scene, going down to 20 fps with csmt_force=0 but then I realized I had a debug patch on my build. With that fixed, for the same scene I get 60 fps.
I also tried archlinux's mesa package (combined with archlinux's wine package and wine-nine-standalone), and I got 60 fps as well. 45 fps with wine.

That sounds like a cpu overhead issue which I don't have (Intel vs AMD) ? Recompiling 32bits mesa with -march=native may help (not easy with meson and 32bits builds), I can give example.

[cyro666]

I actually used the version installed with protontricks. I will try using the Arch package today and later compile it with march=native or in some other ways. It is weird though, because I never saw any of the cores' utilization more than 40%. Maybe it's cache misses or something. And even weirder, wined3d works better.

[axeldavy]

I suggest -march=native for the mesa package. I used the wine archlinux package, that may affect the result.

[axeldavy]

To compile with -march=native in 32bits build, you need to replace the cross-file.
--cross-file fast.native

with fast.native:
[binaries]
c = '/usr/bin/gcc'
cpp = '/usr/bin/g++'
ar = '/usr/bin/gcc-ar'
strip = '/usr/bin/strip'
pkgconfig = '/usr/bin/pkg-config-32'
llvm-config = '/usr/bin/llvm-config32'

[properties]
c_args = ['-m32' , '-march=native', '-fstack-protector']
c_link_args = ['-m32', '-march=native', '-fstack-protector']
cpp_args = ['-m32', '-march=native', '-fstack-protector']
cpp_link_args = ['-m32', '-march=native', '-fstack-protector']

[host_machine]
system = 'linux'
cpu_family = 'x86'
cpu = 'i686'
endian = 'little'

[cyro666]

Alright, thanks, will do once I have the time.

[neVERberleRfellerER]

I also have strange performance issues in (modded) Fallout NV which, until now, I attributed to crappy optimization of the engine. It might not be CPU overhead, because it SEEMS that CPU utilization goes down when GPU load goes down which happens when FPS go down (e.g. just looking in different directions make it clearer in HUD... but it could be just some strange random effect, I don't know;visible on second screenshot).
Screenshot with Gallium HUD


Screenshots are compressed, because without compression they are 10.4 MiB.

[cyro666]

Yeah, I think I saw pretty much the same thing (but can't confirm right now), CPU load going down and also GPU load going down. What is your setup (hardware)?

[axeldavy]

my understanding is some recent amd cpus don't share well cached data, and if you share between a thread and another that are on different core groups you suffer bad perf. Is there any way you could force all the app threads to be for example on the 4 first cores ?

[cyro666]

I personally have the older FX-6300, which is a weird thing anyways. The FX architecture has 2 cores sharing the floating point unit. It was supposed to be a 6-core CPU in my case, but is 6 core only for some loads, for floating point loads it's 3 core. I don't know how the cache is distributed, but as I said, I will try march=native and maybe also try to limit the thing to just 2 cores, or something idk...

[neVERberleRfellerER]

I have Ryzen 7 1700 and RX 580. With nine, I can play Mass Effects, S.T.A.L.K.E.R. Anomaly in 4k at 60 FPS, but for some reason FO NV runs poorly.

@axeldavy
taskset -p 0,1,2,3 $fonv_pid
output: mask ffff, affinity 123
If this is correct, it does not change anything.

[axeldavy]

I just notice the trend that similar reports have Amd as well. Any idea ? Le jeu. 18 avr. 2019 à 09:27, neVERberleRfellerER <[email protected]> a écrit :

…

I have Ryzen 7 1700 and RX 580. With nine, I can play Mass Effects, S.T.A.L.K.E.R. Anomaly in 4k at 60 FPS, but for some reason FO NV runs poorly. @axeldavy <https://github.com/axeldavy> taskset -p 0,1,2,3 $fonv_pid output: mask ffff, affinity 123 If this is correct, it does not change anything. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATNXMLFAGRKYRCYX6HVRF3PRAPFNANCNFSM4HGJ6LCQ> .

[dhewg]

I don't have an AMD cpu, but I do get similar effects with vsync on other games.
Using thread_submit=true tearfree_discard=true improves that again though. Does that help on your end too?

[neVERberleRfellerER]

thread_submit=true helps in S.T.A.L.K.E.R. and Mass Effect 1 (otherwise FPS go to 30 and GPU load goes to like 10%), but does not do anything in FO NV since FreeSync became supported. It helped before FreeSync, but FPS in Strip were same as they are now.
Disabling vsync does not change FPS in FO NV since FreeSync became supported.
nine still helps a lot: without nine I get 17 FPS in scene from screenshots (with 12 % GPU load and 18 % CPU utilization). Notice that 18 % CPU utilization is the same as when game runs at 60 FPS with nine.
Now I have enabled antialiasing (8 samples) and it does not change FPS statistic at all (well, FPS are consistently 3 FPS higher with AA, but that's just weird and probably just random). Setting all settings to minimum improves performance by 2 FPS (which is even lower thatn gain from enabled AA and probably just random fluctuation).

[cyro666]

There was not difference after building with -march=native. I will detail my build process just in case I did anything wrong, but this should've produced correct code:

I built the lib32-mesa package with the Arch Build System. I set CFLAGS="-march=native -O2 -pipe -fno-plt -fstack-protector" and CXXFLAGS to the same in /etc/makepkg.conf. Then I built the package with asp checkout lib32-mesa and makepkg -s. While the compilation was running, I checked with ps aux and before each gcc process there was a whole bunch of flags which weren't there before, when I tried compiling with -mtune=generic. One of those flags was -march=bdver2, which is the correct native flag for my CPU. After that, I installed the built lib32-mesa package. I also tried -O3, but the results were the same. I am fairly confident this worked.

Should I have built it in a different way? This was essentially the easiest way, I think, but I can try your way.

I also tried it with my custom compiled wine + wine-nine from the Arch packages, same result.

I just don't get it. Other games work fine, but the games using this engine are just terrible.

[axeldavy]

I don't know whether what you did was good or not for that purpose.
meson considers that 32 bits is 'cross build' and most flags don't affect it. Thus I am surprised you get a difference with setting CFLAGS and CXXFLAGS.

Could you take a screenshot in game with the hud:
export GALLIUM_HUD="fps,cpu0+cpu1+cpu2+cpu3,GPU-load,primitives-generated+draw-calls,ps-invocations+vs-invocations,buffer-wait-time"

have you tried also if thread_submit=true and tearfree_discard=true changed fps at all.

[axeldavy]

One way to take such a screenshot is to use a tool like spectacle with which you can set a counter (for example 1 minutes)

[neVERberleRfellerER]

@axeldavy Gallium HUD with your parameters shows, that it's indeed CPU bottlenecked. That's unfortunate.

(this was taken by looking in direction on screenshot, then in opposite direction and then back).

[axeldavy]

That is ... surprising.
Usually you don't get a thread at 100%, it is more the behaviour with wine. With nine it usually is spread among several cores which are at about 60%.
Maybe somehow in your config the worker thread is super heavy compared to what it should be for reasons I don't know.

[axeldavy]

If sure the game is using nine, I would suggest to try the native d3dx9_* libs (install with winetricks).

[cyro666]

I had a very little time in this morning, but still did this: I get a completely different story, more in line with what you said, @axeldavy. Except the fps is not where it should be. Didn't have time to test much else right now. This was on Proton 4.2-3, 100% sure Gallium is loading, saw the "Native Direct3D 9" message in steam output.
I did test this with a performance CPU governor and without. It was way worse without:

After setting the CPU governor to performance, it was smooth when looking at some sceneries, but not when looking at something else.

It's not even looking at a wall or something. Look at this screenshoot bellow. When I was looking this way, I had poor performance, but just a few seconds ago, I was looking 10° to the left and it was almost smooth 60fps. There isn't even that much of a difference between these two sceneries in terms of rendering (at least I think):

Very late, I noticed your variables didn't include all 6 cores (for my case), so I added them:

The biggest hit is felt when moving the mouse around, but the poor performance is evident also when stationary looking at some specific angles and at some angles it's pure 60fps.

I will need a little clarification about thread_submit=true and tearfree_discard=true. Are these environment variables, or do I set them somewhere else? Later, if I have time, I will try compiling the library your way.

EDIT: Just tried de-activating Gallium 9 with this HUD active. I can confirm it's way different now. I don't get a constant 60 fps, but it also never goes bellow 45. Unfortunately, I don't have a FreeSync monitor, so this is not acceptable. And the cores are all over the place now, way more active, up to 90%. Did this just to make sure Gallium Nine really was active. Should I also try setting those d3dx9_* libs to native?

[axeldavy]

thanks for the testing.
These are indeed environment variables.

In my testing, I found the overhead of gallium nine is very low in this game (you get max 2% if having all draw calls do nothing, or removing the global mutex - which gave a few artifacts -).

Maybe somehow the clock of your cpu remains rather low for some reasons ? You can try assign the game to 2, 3 or 4 cores see if the increased use of these cores help, or anything else you have in mind.

You can also try with the d3dx9_ libs

[neVERberleRfellerER]

@axeldavy Game loads just d3dx9_38 and using native one does not change anything. thread_submit=true and tearfree_discard=true (and false) also don't seem to change anything.

@cyro666

The biggest hit is felt when moving the mouse around, but the poor performance is evident also when stationary looking at some specific angles and at some angles it's pure 60fps.

Yes, even very small mouse movements cause FPS drops for me. For example looking at center of Prospector Saloon at Goodsprings from the road and then looking halfway towards one of its sides sometimes cause even 20 FPS drop.

[axeldavy]

If the issue is with the mouse, I'd suggest to try force software cursor.

It's easier to rebuild the wine-nine-standalone.
In
https://github.com/iXit/wine-nine-standalone/blob/master/d3d9-nine/present.c#L1160
https://github.com/iXit/wine-nine-standalone/blob/master/d3d9-nine/present.c#L1184
add
return D3DERR_INVALIDCALL;
at the very start of the functions.

The mesa lib will revert to software cursor.

edit: as the game may expect some events due to hardware cursor, this is not guaranteed to work.

[dhewg]

Yes, even very small mouse movements cause FPS drops for me. For example looking at center of Prospector Saloon at Goodsprings from the road and then looking halfway towards one of its sides sometimes cause even 20 FPS drop.

That sounds familiar. I've seen a WINE patch about that recently, something about high dpi mouse. Can you lower the resolution of your mouse (my zowie has a button for that at the bottom)?

[neVERberleRfellerER]

@dhewg That didn't help.

Update: okay, below is what @cyro666 already said in

but the poor performance is evident also when stationary looking at some specific angles and at some angles it's pure 60fps

But I have something interesting. So far I though that looking at "nothing" always results in 60 FPS. Well, it's actually not the case. There is particular spot on ground in Goodspring that causes massive FPS drop and just by looking a little in any direction and basically still seeing the same scene (e.g. moving mark on ground by 1 cm) returns FPS back to 60 (from 35). Also, looking at sky in Goodsprings causes FPS to vary between 40 and 60, but it's stable 60 for sky in The Strip.

[cyro666]

Yes indeed, the problem is also there when doing nothing. And yes, indeed, looking at the same scene from a just a tiny different degree/angle sometimes gives 60 fps.

I will check my CPU clock, but I think I remember doing that once to test the performance CPU governor in Rise of the Tomb Raider and all the cores were constantly at 3.5GHz. Now that I think of it, there is a distinct possibility, but rather small that the load here is too small and the CPU just isn't boosting. But it should be in my experience when using the performance CPU governor. Will check this when I have time.

[cyro666]

Breakthrough! Setting thread_submit=true improves the performance greatly in my case. It now works at a constant 60fps in almost all cases, but when looking at a scene with a lot of things to render (think very large distance), the performance still drops to 50-ish fps, although that is playable. Same in Oblivion, except it's a bit more prominent, the performance dropping to 30fps.

So that helped a lot, but it's not completely perfect. I tried limiting the cores, but that made it worse in the case of only 2 cores (one started hitting 100%) and about 2-3 fps worse when setting only 4 cores.

EDIT: I also monitored the clocks of the cores and they were all definitely boosting. In no case ever was there CPU utilization bigger than 50%

EDIT 2: To make it clear, in NV I had to actually put an effort to find a camera position where the fps dropped, it was at 60fps almost all the time. In Oblivion, the drops were a lot more noticable, though. It is definitely better than before, but I wouldn't call it completely playable for modern standards.

[axeldavy]

Yes, even very small mouse movements cause FPS drops for me.

To analyze what is going on with high dps mouse, it would help if you report on the impact of using software cursor (see my message about it).

[neVERberleRfellerER]

@axeldavy

To analyze what is going on with high dps mouse, it would help if you report on the impact of using software cursor (see my message about it).

It makes no difference. I guess that FPS drops are caused by changes in viewing angle (as said earlier - even looking at ground and changing the angle slightly sometimes makes over 20 FPS differences and these remain until angle is changed again).

[cyro666]

It could be the case that the problems @neVERberleRfellerER is facing are different than mine. We established he is getting bottlenecked on one core for some reason, but the load in my case is evenly spread, never even going over 60% on any of the cores (far from that it seems). I will try experimenting with software cursor anyways.

[cyro666]

Also, I experimented with one more thing yesterday. I found I still had an old drive with Windows 10 for this machine. I ran NV and Oblivion to see the difference and it was an absolutely stable 60fps 100% of the time, sooo... Yeah.

[neVERberleRfellerER]

My problem is probably not caused by nine. D9VK shows similar performance characteristics (overally, it gives me 5 more FPS, but game crashes during firefights). It also suffers from FPS drops when looking at specific angles so CPU is being exhausted by something else. This goes in line with observation, that lowering reoslution, details, disabling shadows and so on does not improve performance.
Possibly related: Gothic 2 seems to be affected by something similar (this is DDraw based, so no nine or d9vk) and there is 100 % single core usage. Lowering details or resolution does not improve performance (only when I set visibility to something that feels like 1 m, FPS go up). Other game, Sacred, works better with LIBGL_ALWAYS_SOFTWARE=1 than without it, because without it, it is bottlenecked by CPU more and software rendering utilize more cores. It does not make much sense to me. Is it kernel driver bug? CPU boost works correctly and utilized cores work at maximum frequency.

[cyro666]

I also tried D9VK, had slightly worse performance. The CPU cores were utilized a bit more, but not as much as with wined3d

To add to this, I cannot even confirm this is a regression. Could've been this way forever. In all honesty I don't remember playing NV or Oblivion on Linux ever before. The only reason I tried was because I bought this new graphics card and thought this machine should be easily capable of this.

Also tried compiling a wine version with pba patches to maybe see a better wined3d performance. Unfortunately, pba needs wine 3.19 and earlier. And they can't run NV for me, I get a crash with a stack overflow on wined3d.

This could take some time since compiling wine and trying out different versions takes considerable time, but I don't intend on giving up soon.

[axeldavy]

There are performance tools (perf, operf, etc) which can estimate where app spents time. I would suggest look at these tools to investigate.

[neVERberleRfellerER]

Given capture done wine

perf record -e cpu-clock

then

perf report --stdio --sort comm,dso

for nine says

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 685K of event 'cpu-clock:u'
# Event count (approx.): 171321750000
#
# Overhead  Command          Shared Object               
# ........  ...............  ............................
#
    77.63%  FalloutNV.exe    perf-28419.map              
     7.23%  FalloutNV.exe    d3dadapter9.so.1.0.0        
     5.18%  FalloutNV.exe    dsound.dll.so               
     1.81%  FalloutNV.exe    libpthread-2.29.so          
     1.35%  FalloutNV.exe    ntdll.dll.so                
     1.26%  FalloutNV.exe    libc-2.29.so                
     0.97%  FalloutNV.exe    kernel32.dll.so             
     0.54%  FalloutNV.:cs0   d3dadapter9.so.1.0.0

Does this mean, that Ryzen 1700 is not enough for this game?

for d9vk says

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 459K of event 'cpu-clock:u'
# Event count (approx.): 114837250000
#
# Overhead  Command          Shared Object               
# ........  ...............  ............................
#
    64.33%  FalloutNV.exe    perf-29591.map              
    13.88%  FalloutNV.exe    ntdll.dll.so                
     5.88%  FalloutNV.exe    dsound.dll.so               
     3.49%  FalloutNV.exe    libc-2.29.so                
     3.07%  FalloutNV.exe    kernel32.dll.so             
     1.41%  FalloutNV.exe    [vdso]                      
     1.29%  wineserver       wineserver                  
     1.02%  FalloutNV.exe    libvulkan_radeon.so         
     0.99%  FalloutNV.exe    libpthread-2.29.so          
     0.86%  wineserver       libc-2.29.so                
     0.68%  FalloutNV.exe    libpulsecommon-12.2.so      
     0.58%  FalloutNV.exe    msvcrt.dll.so     

for wined3d it says

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 821K of event 'cpu-clock:u'
# Event count (approx.): 205286250000
#
# Overhead  Command          Shared Object               
# ........  ...............  ............................
#
    45.17%  FalloutNV.exe    perf-30537.map              
    29.23%  FalloutNV.exe    wined3d.dll.so              
     6.84%  FalloutNV.exe    radeonsi_dri.so             
     5.29%  FalloutN:gdrv0   radeonsi_dri.so             
     3.00%  FalloutNV.exe    dsound.dll.so               
     1.89%  FalloutNV.exe    libc-2.29.so                
     1.55%  FalloutNV.exe    ntdll.dll.so                
     0.89%  FalloutNV.exe    libpthread-2.29.so          
     0.53%  FalloutNV.exe    kernel32.dll.so             
     0.43%  FalloutNV.exe    d3d9.dll.so                 
     0.40%  FalloutNV.:cs0   radeonsi_dri.so             
     0.38%  FalloutN:gdrv0   libc-2.29.so                
     0.28%  FalloutN:gdrv0   libpthread-2.29.so          
     0.28%  wineserver       wineserver        

Performance with nine and wined3d during profiling was same as without profiling, but with d9vk FPS dropped to 0-1.

Out of curiosity I ran perf on Gothic 2 too and it's quite interesting:

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 511K of event 'cpu-clock:u'
# Event count (approx.): 127794500000
#
# Overhead  Command          Shared Object           
# ........  ...............  ........................
#
    45.20%  Gothic2.exe      wined3d.dll.so          
    24.46%  Gothic2.exe      radeonsi_dri.so         
    10.92%  Gothic2.exe      Gothic2.exe             
     6.92%  Gothic2.exe      ntdll.dll.so            
     3.77%  Gothic2.exe      perf-31114.map          
     3.18%  Gothic2.exe      dsound.dll.so           

The amount of time spent in radeonsi_dri seems to little too much to me. But profile is very different from Fallout New Vegas, so it's probably unrelated and wined3d just sucks for Gothic 2.

[axeldavy]

I get the following with nine (forcing vsync off with vbank_mode=0, the game didn't seem to care I had vsync off in its config menu).

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 583K of event 'cpu-clock'
# Event count (approx.): 145918000000
#
# Overhead  Command          Shared Object                    
# ........  ...............  .................................
#
    55.45%  swapper          [kernel.vmlinux]                 
    16.79%  Main thread      perf-21904.map                   
     5.29%  CSMT-Worker      d3dadapter9.so.1.0.0             
     2.75%  Main thread      d3dadapter9.so.1.0.0             
     2.26%  FalloutNV.exe    dsound.dll.so                    
     2.05%  Main thread      [kernel.vmlinux]                 
     1.89%  wineserver       [kernel.vmlinux]                 
     1.09%  Main thread      libpthread-2.28.so               
     0.75%  Steam.exe        [kernel.vmlinux]                 
     0.58%  CSMT-Worker      libc-2.28.so                     
     0.58%  Main thread      libc-2.28.so 

No idea what swapper is. Probably is part of the main exe as well as the perf-21904.map. I had about 100fps on average.

[neVERberleRfellerER]

Actual game exe is in perfs report right column too, but with very low percent. Does this mean, that in my case, perf took 77.63 % of CPU time and since it does not have any impact on performance, it means that game actually is not CPU bottlenecked and that something else causes poor performance?
This seems to be in line with behavior of Sacred - that old game always uses 100 % of CPU (e.g. it turns even Windows laptops into heaters) and it consumes a lot of CPU in perf report (15.86 %) and "perf.pid.map" took only 0.78 % (compared to 77.63 % for FONV).

And now the absolute mystery. When I combine taskset 0x3 (limit to 2 CPU cores) with perf, like this:

taskset 0x3 perf record -e cpu-clock wine ...

I get this:

Notice that CPU usage does not change with FPS anymore. When I run this without perf, usage of one core goes to 100 as seen in previous posts.
Perf says:

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 714K of event 'cpu-clock:u'
# Event count (approx.): 178681000000
#
# Overhead  Command          Shared Object               
# ........  ...............  ............................
#
    72.93%  FalloutNV.exe    perf-7752.map               
    13.41%  FalloutNV.exe    d3dadapter9.so.1.0.0        
     3.22%  FalloutNV.exe    libpthread-2.29.so          
     3.03%  FalloutNV.exe    dsound.dll.so               
     1.72%  FalloutNV.exe    libc-2.29.so                
     1.22%  FalloutNV.exe    ntdll.dll.so                
     0.96%  FalloutNV.exe    kernel32.dll.so             
     0.40%  wineserver       wineserver                  
     0.34%  FalloutNV.:cs0   d3dadapter9.so.1.0.0        
     0.32%  FalloutNV.exe    [vdso]                      
     0.31%  FalloutNV.exe    libpulsecommon-12.2.so      
     0.28%  winedbg          dbghelp.dll.so              
     0.22%  wineserver       libc-2.29.so                
     0.18%  FalloutNV.exe    d3d9-nine.dll.so            
     0.16%  mpegaudioparse1  libmpg123.so.0.44.8  

Does that make any sense to any of you? Because it seems really weird to me.

UPDATE
The 30->60 FPS jump is just random and not caused by vsync. I can create event 40 and32 FPS by looking at different things.

[axeldavy]

Just in case... Did you try vblank_mode=0 ?

[neVERberleRfellerER]

Yes. I can get 67 FPS if I go to corner in opposite direction of what is visible in previous screenshot. It makes no difference in any other way.

[axeldavy]

Did you also use native quartz dll ? I had to set it to have the game run, but maybe you use the integrated one and somehow it hits perf.

[neVERberleRfellerER]

Yes.

From dsound, I have native
devenum ,dmband, dmcompos, dmime, dmloader, dmscript, dmstyle, dmsynth, dmusic, dmusic32, dswave, quartz

From dx I have native
d3dcompiler_43, d3dcompiler_47, d3dx9_31, d3dx9_38, d3dx9_43

[axeldavy]

Ok. Currently I was running with only quartz native for FalloutNV, but I guess the other dlls cannot hurt.

I don't know what happens that makes it slow for your cpu... if you have any other ideas of things to test, say it.

[neVERberleRfellerER]

After setting everything to builtin and using only native quartz I see no difference, so extra dlls indeed didn't hurt.
Game must spend the time somewhere. It is not CPU bottlenecked, it is not GPU bottlenecked. It runs from SSD so it's not IO bottlenecked. Perf consumers more CPU than the game itself and it does not have any effect on FPS. What is the game doing? Would we see high CPU usage if it was waiting for some locks? If not, can I check somehow?

Update
I ran the game with WINEDEBUG+relay and out of 6 GB log, 5 GB are these

0009:Call KERNEL32.InterlockedIncrement(140005b4) ret=00e87136
0009:Ret  KERNEL32.InterlockedIncrement() retval=00000003 ret=00e87136
0009:Call KERNEL32.InterlockedDecrement(140005b4) ret=00e8710c
0009:Ret  KERNEL32.InterlockedDecrement() retval=00000002 ret=00e8710c
0009:Call KERNEL32.InterlockedIncrement(14000464) ret=00e87136
0009:Ret  KERNEL32.InterlockedIncrement() retval=00000003 ret=00e87136
0009:Call KERNEL32.InterlockedDecrement(14000464) ret=00e8710c
0009:Ret  KERNEL32.InterlockedDecrement() retval=00000002 ret=00e8710c

From the rest, 200 MB are heap operations. 100 MB are various

0040:Ret  ntdll.RtlReleaseResource() retval=00000000 ret=7cfbc926
0040:Call ntdll.RtlAcquireResourceShared(0ed215ac,00000001) ret=7cfbc80e
0040:Ret  ntdll.RtlAcquireResourceShared() retval=00000001 ret=7cfbc80e
0040:Call ntdll.RtlReleaseResource(0ed215ac) ret=7cfbc926

and friends. 150 MB are GetLastError. 200 MB of memsets. 120 MB of VirtualAlloc. 40 MB of sleep. 30 MB of semaphore work. Thats all interesting.

[cyro666]

On my side, I decided to see just how much difference there is between Windows and Linux, so I plugged that disk with a Win installation in again. I turned off v-sync in both Linux and Windows. There was no difference in Linux in the problematic regions.

However, on Windows, I was surprised. The game ran barely over 60fps and dipped down to 55 sometimes. Only inside it went to 80fps, but even that was rare. After that, I decided to do an Oblivion benchmark as well and... It had the same problems as on Linux, dips down to 40 fps in problematic areas. For the last benchmark, I went to the big boat scene in Dark Messiah of Might and Magic, because it was problematic in Linux as well. Lo and behold... 45 fps.

At this point I feel like I must apologize for making you all wonder what's wrong. Windows does have a bit better performance, but not by much, it seems.

I monitored the CPU utilization in Windows and 3 cores were usually around 50% while the other 3 were more like 30%. It seems like there's a problem with this engine's CPU utilization on the FX line of AMD processors. I'm not a computer scientist, but I am a soon to be electronics engineer. And to me this looks like there are massive cache misses with the way this engine runs on my processor. Unfortunately, I can't really overclock my RAM, the computer doesn't boot when setting the RAM clock any higher.

Again, I apologize, I just thought this computer should've easily been capable of this since I played the most modern titles at 60fps without a problem. It seems older games were really bad at utilizing more cores.

As for you problem, @neVERberleRfellerER, I hope you find a solution. The Ryzen line should be capable of this. My CPU is way shittier than yours. Maybe try overclocking your RAM? I heard the Ryzen processors are sensitive to RAM speeds. My brother also had better system speed and stability with a Ryzen after recently updating the BIOS/EFI/whatever.

[cyro666]

Also, to anyone wondering how I fixed the performance issues:

1. Set the environment variable thread_submit=true
2. Set a performance governor for your CPU
3. Under the view distance options, set everything to about half, except Actor Fade (you want to be able to spot NPCs from long distances, after all)

This gives me a constant 60fps. I have not tried the Strip area though, because I don't really have a save with that area. Same thing goes for Oblivion, the lower view distances helped the most. Everything else can be set to maximum detail.

[neVERberleRfellerER]

In my case no matter the distance, or quality or anything, I still get the same performance. This could be because my FONV is modded and mods could place more things or something and it is enough to "bottleneck the engine". Increasing memory clock from 2400 MHz to 2933 MHz didn't seem to help in noticeable way.
Anyway, this is very probably problem of the engine not being ready for 7 years distant future. My guess is that full memory barries placed by InterlockedIncrement and decrement cripple the performance on Ryzen CPUs due to their high memory latency (at least on 1xxx series, the 2xxx seem to be just fine on various recording on the Internet). I still wonder if it shoudl be recorded as high CPU usage (it's the CPU waiting after all).
Since d9vk and wined3d show the same performance characteristics and what was said in previous paragraph, this is not problem in nine but in the engine.
I guess that this can be closed now. We can reopen if some actual evidence leads back to nine.
Thank you all for help and assistance.

[axeldavy]

@neVERberleRfellerER: Did you try Proton's esync ? I think it's supposed to reduce the cost of these barries.

[neVERberleRfellerER]

@axeldavy Esync does not help here, because InterlockedIncrement/Decrement are just 5 instructions and these are same with and without esync.
https://github.com/wine-mirror/wine/blob/master/dlls/kernel32/sync.c#L2569
https://github.com/zfigura/wine/blob/esync/dlls/kernel32/sync.c#L2553
It should be fast, but maybe high portion of resources is shared between threads (my guess: reference counting garbage collector) and if bus locks are applied, it effectively reduces the game to single core.That would also explain, why performance does not chnage after game is pinned to 2 cores (I guess one is used by game, other by nine), while games such as Mass Effect suffer huge performance losses (from 60 FPS to 10-15 FPS) from being pinned to even 4 cores and not even loading in acceptable time when pinned to 2 cores.
Esync helps in slightly different scenarios, e.g. all WaitForXObject (and everything that goes to NtWaitForMultipleObjects).

[axeldavy]

Why would InterlockedIncrement and InterlockedDecrement be slow ? We use similar atomic instruction in gallium for object refcounting all the time. D3d9 is a C++ api which uses a refcount system for everything.

[neVERberleRfellerER]

@axeldavy I see. Then I am wrong and game is slow for different reasons, but I don't have any idea what these reasons might be.

[neVERberleRfellerER]

The game is just weird. Look at these two pictures (captured with d9vk because I was just testing it with prof when I noticed the draw calls). Notice the number of draw calls.


Moving the mouse in any direction drops the number of draw calls to around 300 again. Same performance drop happens with nine at this and also other places, it also happens when looking at random distant points and so on. This makes it feel like FPS drop during mouse movement.

UPDATE 1
It's not the location, it's really the angle. When I position the camera in a way that gives one of the massive FPS drops and then leave my mouse, I can walk away using strafe and forward/back keys to very different location and as long as the viewing angle remains the same, number of draw calls and low FPS remain. There are multiple of problematic angles and these angles are different in different locations (after loading screen they are different, but they are always same in same location even after going out and back again - compared using in-game compass).

[neVERberleRfellerER]

Okay, probably the last thing. I have these two pictures with nine taken under the map with help from noclip. There is nothing to render, yet rotating the camera to precise angle causes huge increase in vs invocations and draw calls. Unless there is some other simple explanation, I'd just call it a game bug and be done with this. It have already sucked too much time and I don't have Windows to test this.

[cyro666]

Yeah, I guess we can close this if nobody has any more ideas. My issues are "fixed" anyways.

[axeldavy]

For those with 30 fps problem, could you try:
thread_submit=true throttle_value=3
(it needs both)

[dhewg]

Yeah, I guess we can close this if nobody has any more ideas. My issues are "fixed" anyways.