Python app crashes when pressing any key in a non-English layout about selkies-gstreamer

[ehfd]

This error itself should be fixed in the next release. Using a fork of Python-Xlib which fixes this.

AZERTY Keyboard is handled in #9.

[ehfd]

This is going to take some time. Do restart Selkies to reconnect. In the container, you can terminate selkies-gstreamer using htop and it will automatically reload.

[ehfd]

I believe this is a pynput problem.

[ehfd]

Should be solved with #9.

[ehfd]

Revamping of the Javascript libraries are necessary, especially vuetify and guacamole-keyboard.

[ehfd]

Priority Number 3 IMO (with #9 and #22) @danisla

[maxpain]

Any updates?

[ehfd]

Currently discussed in #85

[ehfd]

In short, need to implement xorg.c into ctypes (courtesy of @cruizba).

I think I found a possible way to fix all inconsistencies and errors related to the keyboard, potentially paving the way for the removal of pyinput and, possibly in the future, python-xlib. You can find my work in this branch of my fork: GitHub - cruizba/selkies-gstreamer.

Let me share the backstory of how I arrived to this.

Upon further investigation, I realized that certain keysyms which reach this section of webrtc_input.py, particularly those representing Spanish characters on my keyboard (like ñ), lacked corresponding keycodes. I've noticed that X server loads some generic keycodes, but it doesn’t cover all of them. This might relate to the default keyboard setting, which is typically a Generic 105-key PC, but I am not sure.

In a remote desktop environment, the server is unaware of the client's keyboard. Therefore, modifying the server's keyboard configuration seemed impractical as a solution to me...

To explore how other projects handle keyboard input remotely, I looked into Neko. Neko, along with Guacamole keyboard lib in the browser, manages keyboard events from the browser. When the keysym reaches the neko server, they process input data through a set of C functions in xorg.c, which they integrate with Go.

The key function of xorg.c is XKey(). It appears that XKey dynamically assigns KeySyms to Keycodes when there is no existing association. If a keysym is sent without a corresponding keycode, it registers this new mapping in the XOrg keyboard configuration (As far as I can understand from the code). This approach is also utilized in TigerVNC's source code for keyboard keys mapping!

Consequently, I've created xorg.py to integrate the build of xorg.c and changed the keyboard part of webrtc_input.py to utilize the new xorg class. This solution works exceptionally well! It detects all keyboard keys correctly without altering any remote keyboard settings and while using my own keyboard configuration.

The challenge was creating a new addon and compiling the xorg.so file to implement this functionality. The project now requires additional headers and libraries. However, the good news is that this addon could potentially replace python-xlib for mouse interactions and other X interactions.

If you're interested in testing it, my branch is fully functional. Simply run vscode with the devcontainer as usual and execute the task [run] re-build, re-install and run selkies-gstreamer. It's configured to compile and install the library.

No, files present at https://github.com/cruizba/selkies-gstreamer/tree/fix-inputs/addons/xorg-iface are just a simple module using x11 libs, it is not part of the X library.

[cruizba]

I am so happy to see this :).

[ehfd]

@cruizba You deserve an acknowledgment. We're also trying to solve the keycode (Spanish) issue with a ctypes version of your .so file.

I hope everything comes together to the next release.