Scrolling is too fast on touchpads about selkies-gstreamer

[ehfd]

In the above sentence are you referring to the scrolling issue or screen resolution?
The scrolling issue.

For the resolution issue... A larger DPI should be used when launching the X server for now. Hard to change the DPI on the fly. Also, I believe Xfce is not that good at HiDPI.

[ehfd]

Is there a difference between the resolution of your client and the resolution of the remote screen?

[PMohanJ]

I'm sorry @ehfd , I didn't get it. What do you mean by client vs remote screen resolution?
Are you talking about screen resolution?

[ehfd]

Could you post a screenshot of the menu (small crescent at the right) so that the two things underlined are visible?

[PMohanJ]

Sure, now I get it. Here is the screenshot.

[ehfd]

Okay. The screen resolution is the same. It's another bug.

@danisla

[ehfd]

Potentially solvable in tandem to #98

[PMohanJ]

From what I observed:

• In the JS code, input.js, the 'mousewheel' event it getting triggered almost 3x time on touchpad swipe compared to physical mouse wheel events.
• Also in webrtc_input.py, the self.mouse.scroll(dx, 1) method just scrolls one notch up or down (-1 and 1).

So I'm thinking as touchpad scroll event are thrice than mouse scroll, thus making the scrolling fast. Please correct me if I'm wrong.

[ehfd]

You're right. We will fix that.

However, we are still unsure why the Mac's touchpad is hypersensitive in whatever browser you are using. We would need more information on your environment.

[ehfd]

Related to #76.

[PMohanJ]

Here's my environment information:

• Brave browser version: [Version 1.57.53 Chromium: 116.0.5845.114 (Official Build) (x86_64)]
• MacBook Pro 2020, Os version: macOS Ventura 13.4.1
• I've also tried in google chrome as well: Version 117.0.5938.132 (Official Build) (x86_64).

I remember trying to access the remote desktop few weeks back via windows laptop as well using brave browser and had encountered same scrolling sensitive issue with touchpad. Please let me know if more information is needed. Thank you!

[PMohanJ]

@ehfd I'm not an expert in python or js but I'd like to work on this issue. Could you guide me please.

[ehfd]

I also reproduce in Windows.

[PMohanJ]

Hello @ehfd, any update on this issue?
I've actually tried a workaround for this, in which I tried to reduce the mouse scroll events that are being sent to remote client by the following logic. (basically setting a timeframe of 150ms before sending the next scroll event to remote client)

_isTouchpad(deltaY) {
    return deltaY && Number.isInteger(deltaY) ? true : false;
}
_mouseWheelWrapper(event) {
    if (this._isTouchpad(event.deltaY)) {
        if (this.allowTrackpadScrolling) {
            // stop trackpad scroll event until next 150ms
            this.allowTrackpadScrolling = false;
            this._mouseWheel(event);

            setTimeout(() => {this.allowTrackpadScrolling = true}, 150)
        }
    } else {
        this._mouseWheel(event);
    }
}  

I can see a difference upon scrolling the system applications like scrolling in terminal, but when it comes to browsers in remote client, it's still same as before (scrolling is still hypersensitive).

Also one small doubt, may I know why the icons are too small when the remote and client resolution is same.

[ehfd]

Thanks for your investigation. Greatly appreciate it.

Looks like this needs a solution from the JavaScript side. We will investigate the issue and come back to you.

[PMohanJ]

Thanks to you for replying.

Looks like this needs a solution from the JavaScript side.

In the above sentence are you referring to the scrolling issue or screen resolution?

[ehfd]

I have moved the DPI issue to #110.

[PMohanJ]

Hi @ehfd. I actually got some working solution to this hypersensitiveness of scroll for touchpad. Let me know what you think of this

So the problem here as stated before is, the number of events that get triggered for scroll on touchpad is much higher in number compared to typical mouse wheel. This is due to the effect of continuous scroll or accelerated scroll as mentioned here: https://developer.mozilla.org/en-US/docs/Web/API/Element/mousewheel_event#chrome

If the device supports continuous scroll (e.g., trackpad of MacBook or mouse wheel which can be turned smoothly), the value is computed from accelerated scroll amount.

To mitigate this I tried setting a threshold to limit the amount of the scroll events being sent to the server as said long before #99 (comment). But this does't provide the smoother scroll effect.

When looked into neko, I saw the same technique of limiting number of events, but instead of just sending an event with button_mask just to scroll one notch up/down, they're sending a magnitude of value.

_mouseWheel(event) {
        var mtype = (document.pointerLockElement ? "m2" : "m");
        var button = 3;
        if (event.deltaY < 0) {
            button = 4;
        }
        // truncate the floating point decimal numbers and get an absolute value
        var deltaY = Math.abs(Math.trunc(event.deltaY));
        // this._scrollMagnitue=10
        var magnitude = Math.min(deltaY, this._scrollMagnitude); 

        var mask = 1 << button;
        var toks;
        // Simulate button press and release.
        for (var i = 0; i < 2; i++) {
            if (i === 0)
                this.buttonMask |= mask;
            else
                this.buttonMask &= ~mask;
            toks = [
                mtype,
                this.x,
                this.y,
                this.buttonMask,
                magnitude
            ];
            this.send(toks.join(","));
        }
        event.preventDefault();
    }

And on the server side I'd modified the part to support sending of multiple scroll events to X server based on the received magnitude. It's working quite reasonably compared to before.

But this fails to provide smooth scrolling in devices other than Mac. This is because the delta values differ from device to device. In mac the least delta value is 1 and in other devices it could be 4 or 5, producing different levels of granularity based on the hardware of the device and OS. This kind of makes the scrolling at high pace than user intended pace on non-mac devices.

So to mitigate this I tried to normalise the values by taking the least delta as a scale factor and scale the subsequent values with this factor. This resolved the issue of touchpad on other devices.

But, fails for mouse wheel events of non-mac devices by producing much slower scroll effect. This is due to the delta values being constant in most of the devices (except for Mac's connected mice) and exists in the range of 100 to 130. By applying the normalising technique, the delta values would always end up at a constant value 1 in mouse wheel generated events. Thus producing much slower scroll effect due to constant magnitude of 1 and threshold of mouse wheel events.

To resolve this we've two options:

• make the mouse wheel events as continuous scroll just like mac does, so that we'd have inertia scrolling that would generate more number of events.
• or remove the threshold on mouse wheel events

I've chosen the second approach (as it's easier to manipulate things at hand than to generate non existing ones) to remove the threshold on mouse wheel events. This solved the problem for touchpads and mice both in mac and non-mac devices.

Kindly let me know your thoughts on this approach. I've the changes ready and can make the PR as soon as possible.

[ehfd]

@PMohanJ Lovely!
I like the approaches.

[PMohanJ]

I've pushed the image to docker hub. You can test it by spinning up a container:

docker run -d --name selkies-dev -p 8080:8080 -p 3478:3478 pmohanj/selkies-dev-scroll:latest

Note: If you shift from touchpad to mouse, then the transition won't be smooth for 3-4 scroll notches. This is due to the fact that the scale factor needs to be adjusted.

Also the touchpad scrolling is still bit sensitive (it wouldn't be as same as our physical devices). This is primarily due to the fact that browsers still abstract the actual input event that occur that OS level, so we don't have raw inputs to manipulate them. And on the servers side the package pynput is basically calling the XTEST extension module of Xserver that was originally intended to test Xserver functions. So all it does is button press and release, which is same as XTestFakeButtonEvent that neko is using

int XTestFakeButtonEvent(display, button, is_press, delay);
Display *display;
unsigned int button;
Bool is_press;
unsigned long delay;

So on the server side we don't really have much control.

[ehfd]

@PMohanJ Do you have the PR?

[ehfd]

And perhaps investigating #102 in conjunction would help with a lot of hassle. Would it be possible?

[PMohanJ]

I'll open a PR in few minutes.
And yeah, I'll try that #102 issue as well.