Mouse movement or speed is erratic in games (especially Wine) / Support relative mouse movement without uinput about selkies-gstreamer

[ehfd]

@joyflix99 Check #55 as well for a similar attempt

[ehfd]

+1
Mouse speed is quite fast in Selkies by default, and it should definitely be slower.

Normally this needs to be done from the application, not the system or container.
Some games or graphics application have these mouse speed scaling/mouse sensitivity options, but not all of them.
Use https://github.com/selkies-project/selkies-vdi/tree/master/images/uinput-device-plugin for now.
However, would be better mouse speed scaling/sensitivity was possible just with JavaScript and Xorg. It should be possible after how Selkies achieved without uinput.

[ehfd]

Relevant code: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/gstwebrtc_app.py and https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/webrtc_input.py

[ehfd]

I think that there should be a way for mouse behavior to work correctly even with the current configuration.

if pointer is locked and remote and local resolution is significantly different, e.g., 4k vs. 1920x1080, pointer motion is very slow, acceleration should be adjusted.

I suspect that this is because the mouse cursor movements should be compensated for this resolution difference. This explains why pointer mode is slower or faster when the remote vs local resolution is different.

Communication between the screen size of the web interface and the screen size of the server to calculate the precise mouse movement intensity and sending to the X server with Xlib should be possible, and this may solve this issue without relying on uinput.

if pointer is not locked, mainly games cannot be controlled as mouse jumps randomly over screen. Could be related to resolution change? It does not happen e.g., with tuxracer. But it does happen in witcher 2.

This is definitely related to relative mouse movement being sent as absolute mouse movements instead. I think going to pointer locked mode for 3D games such as The Witcher 2 is definitely adequate.

If cursors of these applications or games catch the relative, also called the raw mouse motion, Xlib doesn't work correctly. There should also be a way to compensate this behavior in the web interface for the Python application to scale the remote screen size and the local screen size to properly send movements with Xlib.

[ehfd]

The issues I see while testing various games or applications that my colleagues lent me for testing (continuously updated, VKD3D indicates DirectX 12 applications run on Wine, DXVK indicates DirectX 9-11 applications run on Wine):

Cyberpunk 2077 (VKD3D): Cursor disappears at start screen, tried to enter settings to see if it can be fixed but can't find the cursor. Unplayable.
Euro Truck Simulator 2: Works on Native but not on Wine with DXVK for D3D11. Cursor disappears at start screen. Unplayable on Wine. As this is the same behavior as Cyberpunk 2077 with only Wine and not native, this may indicate that how Wine handles cursors makes them disappear in Selkies.
Mount & Blade II: Bannerlord (DXVK): 3D mode (battle, duel) is unplayable. Cursor always points right-down. Partially playable, some unplayable.
The Witcher 3: Wild Hunt (DXVK, D3D11): Cursor movement to the right is much faster than left, fixed by turning on Hardware Cursors in settings. Playable but movement is slightly erratic.
Minecraft (Native): Mouse movement is also erratic, fixed by turning off Raw Cursors in settings. Playable.
Unreal Engine Games (OpenGL or Vulkan): Requires the -norelativemousemode to work, only works above 4.26.
Most SDL-based games: Cursor works on its own in relative mouse mode, which means SDL_SetRelativeMouseMode must be changed libsdl-org/SDL#1836.

Did anyone solve this? I am having same issue when running GTA5 using wine. Mouse works fine for GTA Vice city but it is a little laggy but with GTA5 mouse movement is unpredictable even in main menu. Please can anyone help me how I can solve this. I think this is a very interesting project

[ehfd]

@joyflix99 Could you make the remote resolution same as the local resolution then use full screen mode on the web browser (reduce resolution locally as needed)?

Also, try enabling fullscreen mode instead of borderless windowed mode on the Wine game. This configuration varies per game.

@ehfd Thank you for such quick response :) I just tried that but it didn't help. I think I will try tracing mouse input in wine, I think it is somewhere in xinput11.c file. And rebuild it to read mouse input from unix pipe and then write mouse data to this unix pipe. There is one drawback to this approach, you will not be able to interact with anything outside wine. I will let you know how it goes

[ehfd]

selkies-gstreamer/src/selkies_gstreamer/webrtc_input.py

Line 271 in 52631b9

self.mouse.move(x, y)

This issue mainly attributes to inputting the mouse movement without any scaling with pynput.mouse.Controller().

Coordinating this with the window size of the web client can fix the issue.

[ehfd]

https://gist.github.com/loochek/f0d875fb8c2adcb92431e13868cbd079

This is a very interesting take and currently works for most Wine games.

[vamman]

Mouse is completely unusable running UE4 game through Wine. Is there any solution?

[ehfd]

Mouse is completely unusable running UE4 game through Wine. Is there any solution?
@vamman

We are looking at it. Next in line for priority. @loochek 's take doesn't work?

[vamman]

Mouse is completely unusable running UE4 game through Wine. Is there any solution?
@vamman

We are looking at it. Next in line for priority. @loochek 's take doesn't work?

I looked at @loochek 's implementation and it looks to be for vpads? Or does his work actually also address the mouse issues?

[vamman]

Mouse is completely unusable running UE4 game through Wine. Is there any solution?
@vamman

We are looking at it. Next in line for priority. @loochek 's take doesn't work?

I looked at @loochek 's implementation and it looks to be for vpads? Or does his work actually also address the mouse issues?

Alright I patched @loochek gist in and it does not improve my mouse issues.

[ehfd]

@vamman Ah, for mouse...

I have that issue too, but it's a lot of work to do to fix it.

One hint that I found is that it would work if there is a "game cursor mode" where the mouse cursor is fixed to the center of the screen and only relative movements, not absolute coordinates, are passed to X.org.

[vamman]

Good, now I know my issue isn't just me ;)

I have some potential ideas to solve this issue.

Do you have a method that allows me to keep the browser cursor visible until I "click into" the application? For example in your EGL example the KDE cursor is present as soon as I mouse over the window. I want the native browser cursor visible until I click in. I also want the ability to hit the ESC key and the native browser cursor pop out.

This is normal behavior for most "canvas" apps. They are using the Pointer Lock API.

[ehfd]

@vamman Note that Ctrl+Shift+Click already is the Pointer Lock API (it is in the docs). But even that is a bit finicky. All native games are okay and most Wine games work, though.

Refer: #28 (comment)

[ehfd]

@vamman Try Ctrl+Shift+Click, and also try virtual desktop for Wine (Lutris has a setting for that, it restricts the cursors to the space). I'm not sure virtual desktop would work, but it could.

[vamman]

Have you seen how cloud morph and Epic are handling this? Epic is actually grabbing the mouse position from the browser and sending it directly to their application via a web socket. Cloud morph does the same thing essentially but instead send mouse position to a generic exe that translates the position relative.

I'll give the virtual desktop a try.

[ehfd]

Have you seen how cloud morph and Epic are handling this? Epic is actually grabbing the mouse position from the browser and sending it directly to their application via a web socket. Cloud morph does the same thing essentially but instead send mouse position to a generic exe that translates the position relative.

Epic, as in pixel streaming? I know @adamrehn .

Cloud morph does the same thing essentially but instead send mouse position to a generic exe that translates the position relative.

We also do this, but it's simply a bit unrefined. We just have to fine tune things and make sure it works out.

My hypothesis is that in relative cursor mode, the absolute mouse cursor should be locked to the center and only make relative movements.

Alternatively, if you aren't using containers, you could use uinput which already does things well.

selkies-gstreamer/src/selkies_gstreamer/webrtc_input.py

Line 256 in 7c4d8b6

def send_mouse(self, action, data):

This is our code. Take a look at the section. I would appreciate if you could make suggestions.

[vamman]

Absolutely, happy to take a look at this. I am using containers.

[ehfd]

https://github.com/selkies-project/selkies-gstreamer/tree/master/addons/gst-web/src

Our web interface as well.

[ehfd]

@vamman There is an option on PlayOnLinux or Lutris called Mouse Warp Override. Setting it to forced could solve things, combined with the Wine Virtual Desktop. At least for what I experienced so far.

[ehfd]

Relevant? https://www.reddit.com/r/wine_gaming/comments/v1sml1/joystick_input_works_correctly_in_wine_control/

[ehfd]

Update: Perhaps, try using https://github.com/ValveSoftware/gamescope as an alternative.

https://github.com/akdor1154/gamescope-pkg

[ehfd]

Relevant: https://web.archive.org/web/20211003152946/https://parsec.app/blog/game-streaming-tech-in-the-browser-with-parsec-5b70d0f359bc

[ehfd]

Please test #97 in Wine, if you follow.

[ehfd]

#28 (comment)
I confirm that most issues in Wine and SDL shown above have been solved.
Relative mouse movement in Cyberpunk 2077 and Mount & Blade 2: Bannerlord with Wine works well.
Relative mouse behavior in Wine now is solved.

One issue persists where the mouse movement speed is slower when the remote resolution is smaller than the local resolution, and vice versa.
Another issue is to be raised.