Konsole screen doesn't update properly about xpra

[palves]

Here's an example of bad output. I was hammering on the keyboard, and that was the result. Note how the screenshot shows three cursors, which is non-sensical. The second line of the command, where the first cursor is shown on the right at the end of the line, should be filled with text in a similar pattern all the way to the right hand side, I did not type a bunch of spaces. The third line of the command is correct, I stopped typing there. But the last time, the one with an empty command prompt, shouldn't be there. I still haven't pressed enter for the command that starts with "asdfjklf". That's output from a previous command that shouldn't be there because I had pressed Ctrl-C since.

[palves]

There's the exact same window, after xpra manages to refresh it properly. Sometimes it takes a few seconds to auto-refresh properly, sometimes it takes more than a few, I can't spot a pattern.

[totaam]

Is this a standard installation with all the codecs available?
Does it only affect konsole? Or also xterm or gnome-terminal / lxterminal?
Does this happen with --opengl=no / --opengl=force?
Can you include xpra info | grep windows for this session?
Can you run your server with -d compress so we can see what compression it is using?
My guess is that konsole is using transparency (usually unnecessarily) and perhaps this isn't handled properly when compressing / decompressing the pixels, or when painting (with / without opengl).
Looks fine running on Fedora.

[palves]

Is this a standard installation with all the codecs available?

It's the deb packages from https://xpra.org:

 $ apt -qq list xpra | grep installed
 xpra/jammy,now 5.0.8-r0-1 amd64 [installed]

Does it only affect konsole? Or also xterm or gnome-terminal / lxterminal?

Does not affect xterm, nor gnome-terminal Have not tried lxterminal.

Does this happen with --opengl=no / --opengl=force?

Tried both (and saw from the stdout output on the client that they had the effect of enabling/disabling opengl), and it made no difference.

Can you include xpra info | grep windows for this session?

$ xpra info | grep windows
child.command=('/usr/bin/python3', '/usr/bin/xpra', '--windows=no', '_audio_record', '-', '-', 'pulsesrc', 'device=Xpra-Speaker.monitor', 'opus', '', '1.0')
client.windows=True
commands.start-menu.Multimedia.Entries.VLC media player.MimeTypes=('application/ogg', 'application/x-ogg', 'audio/ogg', 'audio/vorbis', 'audio/x-vorbis', 'audio/x-vorbis+ogg', 'video/ogg', 'video/x-ogm', 'video/x-ogm+ogg', 'video/x-theora+ogg', 'video/x-theora', 'audio/x-speex', 'audio/opus', 'application/x-flac', 'audio/flac', 'audio/x-flac', 'audio/x-ms-asf', 'audio/x-ms-asx', 'audio/x-ms-wax', 'audio/x-ms-wma', 'video/x-ms-asf', 'video/x-ms-asf-plugin', 'video/x-ms-asx', 'video/x-ms-wm', 'video/x-ms-wmv', 'video/x-ms-wmx', 'video/x-ms-wvx', 'video/x-msvideo', 'audio/x-pn-windows-acm', 'video/divx', 'video/msvideo', 'video/vnd.divx', 'video/avi', 'video/x-avi', 'application/vnd.rn-realmedia', 'application/vnd.rn-realmedia-vbr', 'audio/vnd.rn-realaudio', 'audio/x-pn-realaudio', 'audio/x-pn-realaudio-plugin', 'audio/x-real-audio', 'audio/x-realaudio', 'video/vnd.rn-realvideo', 'audio/mpeg', 'audio/mpg', 'audio/mp1', 'audio/mp2', 'audio/mp3', 'audio/x-mp1', 'audio/x-mp2', 'audio/x-mp3', 'audio/x-mpeg', 'audio/x-mpg', 'video/mp2t', 'video/mpeg', 'video/mpeg-system', 'video/x-mpeg', 'video/x-mpeg2', 'video/x-mpeg-system', 'application/mpeg4-iod', 'application/mpeg4-muxcodetable', 'application/x-extension-m4a', 'application/x-extension-mp4', 'audio/aac', 'audio/m4a', 'audio/mp4', 'audio/x-m4a', 'audio/x-aac', 'video/mp4', 'video/mp4v-es', 'video/x-m4v', 'application/x-quicktime-media-link', 'application/x-quicktimeplayer', 'video/quicktime', 'application/x-matroska', 'audio/x-matroska', 'video/x-matroska', 'video/webm', 'audio/webm', 'audio/3gpp', 'audio/3gpp2', 'audio/AMR', 'audio/AMR-WB', 'video/3gp', 'video/3gpp', 'video/3gpp2', 'x-scheme-handler/mms', 'x-scheme-handler/mmsh', 'x-scheme-handler/rtsp', 'x-scheme-handler/rtp', 'x-scheme-handler/rtmp', 'x-scheme-handler/icy', 'x-scheme-handler/icyx', 'application/x-cd-image', 'x-content/video-vcd', 'x-content/video-svcd', 'x-content/video-dvd', 'x-content/audio-cdda', 'x-content/audio-player', 'application/ram', 'application/xspf+xml', 'audio/mpegurl', 'audio/x-mpegurl', 'audio/scpls', 'audio/x-scpls', 'text/google-video-pointer', 'text/x-google-video-pointer', 'video/vnd.mpegurl', 'application/vnd.apple.mpegurl', 'application/vnd.ms-asf', 'application/vnd.ms-wpl', 'application/sdp', 'audio/dv', 'video/dv', 'audio/x-aiff', 'audio/x-pn-aiff', 'video/x-anim', 'video/x-nsv', 'video/fli', 'video/flv', 'video/x-flc', 'video/x-fli', 'video/x-flv', 'audio/wav', 'audio/x-pn-au', 'audio/x-pn-wav', 'audio/x-wav', 'audio/x-adpcm', 'audio/ac3', 'audio/eac3', 'audio/vnd.dts', 'audio/vnd.dts.hd', 'audio/vnd.dolby.heaac.1', 'audio/vnd.dolby.heaac.2', 'audio/vnd.dolby.mlp', 'audio/basic', 'audio/midi', 'audio/x-ape', 'audio/x-gsm', 'audio/x-musepack', 'audio/x-tta', 'audio/x-wavpack', 'audio/x-shorten', 'application/x-shockwave-flash', 'application/x-flash-video', 'misc/ultravox', 'image/vnd.rn-realpix', 'audio/x-it', 'audio/x-mod', 'audio/x-s3m', 'audio/x-xm', 'application/mxf')
exit-with-windows=
state.windows=

Can you run your server with -d compress so we can see what compression it is using?

I've attached the resulting server.log.

My guess is that konsole is using transparency (usually unnecessarily) and perhaps this isn't handled properly when compressing / decompressing the pixels, or when painting (with / without opengl).
Looks fine running on Fedora.

Could well be. I see some warnings about transparency on stdout on the client side:

 2024-04-05 13:05:53,168 Warning: cannot handle window transparency
 2024-04-05 13:05:53,168  screen is not composited
 ...
 2024-04-05 13:06:00,312 Warning: window 0x1 changed its transparency attribute
 2024-04-05 13:06:00,312  from False to True, behaviour is undefined
 ...

and the server.log also shows similar warnings. I have background transparency set to 0% on Konsole, I really do not want transparency, but maybe Konsole always does something with it unnecessarily, as you say.

The server.log also shows errors like:

 2024-04-05 13:06:00,365 Error: failed to encode h264 frame
 2024-04-05 13:06:00,365  using x264 video encoder:
 2024-04-05 13:06:00,365  expected BGRX but got BGRA
 2024-04-05 13:06:00,365  source: XShmImageWrapper(BGRA: 0, 0, 1353, 595)
 2024-04-05 13:06:00,365  options:

...

server.log

[totaam]

Warning: cannot handle window transparency
screen is not composited

That's the problem.
You're using a non-compositing window manager - as per https://github.com/Xpra-org/xpra/wiki/Reporting-Bugs, this is a major point to report.

[palves]

I'll try compositing.

I was just trying something else though -- thinking this had to do with the h264 errors, I changed Picture/Encoding at runtime from the tray icon on the client, from "automatic", to PNG (24/32bpp), and the issue seems to have disappeared. Then I changed to "video stream", and still no problem. Then I changed back to "automatic", and the problems reappeared. More, I was doing a tail -f /run/user/1000/xpra/9999/server.log on the server as I was banging at the Konsole window to trigger the problem, and I can clearly see that when the problem triggers, and the Konsole window doesn't update properly, I see errors like these:

2024-04-05 13:29:18,294 Error: failed to encode h264 frame
2024-04-05 13:29:18,294  using x264 video encoder:
2024-04-05 13:29:18,294  expected BGRX but got BGRA
2024-04-05 13:29:18,294  source: XShmImageWrapper(BGRA: 4, 87, 1335, 12)
2024-04-05 13:29:18,294  options:

and it looks like it's one error per key that I press (i.e., probably one error per screen update). If I wait sufficiently for the problem to "fix itself", the errors stop appearing with the key presses.

How can compositing on the client side affect encoding on the server side?

[totaam]

How can compositing on the client side affect encoding on the server side?

The source image is BGRA (A meaning with alpha) but the client is going to discard the alpha channel anyway, so the server can process it as if it was BGRX (X meaning ignoring alpha).

[palves]

OK, enabling compositing fixes it.

I also tried:

1. • starting xpra client with compositing enabled, and then,
2. • disabling kwin compositing (while the client is running and I have a Konsole window up) with either
     2.1) - "$ qdbus org.kde.KWin /Compositor suspend" (can be reenabled with "resume"),
     2.2) - or by going to kwin settings, disabling composing and restarting kwin with "kwin --replace &"
3. • and, then banging at konsole

and I don't see the problem. Presumably because the server keeps sending the alpha channel to the client?

If I start both client and server with compositing off, and then enable compositing, and then restart the client (keeping the same server session), I then see these on the server.log while I'm banging at the keyboard in Konsole:

2024-04-05 14:52:59,305 Error: failed to setup a video pipeline for auto encoding with source format BGRA
2024-04-05 14:52:59,305  all encoders: 
2024-04-05 14:52:59,306  supported CSC modes: 
2024-04-05 14:52:59,306  supported encoders: 
2024-04-05 14:52:59,306  encoders CSC modes: 

However, the Konsole terminal window still updates properly.

The source image is BGRA (A meaning with alpha) but the client is going to discard the alpha channel anyway, so the server can process it as if it was BGRX (X meaning ignoring alpha).

So IIUC, when compositing is off, the stream is set up as BGRX? And then when the source turns out as BGRA, then we get an encoding error, and frames are dropped. Did I get that right?

It sounds to me like a bug then. I'm obviously not familiar with Xpra internals, but can't the encoding pipeline be setup to drop the alpha channel instead of failing the conversion? Or, if this can happen, can't the stream/client be set up as BGRA in the first place even if the A channel is going to be dropped on the client? Even if that results in more bytes over the wire, it seems better to me than dropping frames with encoding errors. Correctness should trump speed, IMHO.

BTW, how can we see what encoding was selected by "auto"? I see in the server.log:

 2024-04-05 15:04:52,342  automatic picture encoding enabled, also available:
 2024-04-05 15:04:52,342   h264, vp9, vp8, png, png/P, png/L, webp, avif, rgb24, rgb32, jpeg, jpega, scroll

but it doesn't say which one of those available was selected, nor the detail of BGRX vs BGRA.

[totaam]

Another problem was that konsole was not mapped to a text application - which is why it ended up using a video encoder, it is now correctly mapped: 332fdc0.

As for BGRA vs BGRX, 5059bd9 should avoid the Error: failed to encode h264 frame if you somehow find another application with transparency that isn't mapped to text yet.

---

And then when the source turns out as BGRA, then we get an encoding error, and frames are dropped. Did I get that right?

No, the server will use another non-video encoding as fallback when that happens, without any noticeable delay - just an ugly log message.

---

BTW, how can we see what encoding was selected by "auto"?

As per my previous answer:

Can you run your server with -d compress so we can see what compression it is using?

---

but it doesn't say which one of those available was selected, nor the detail of BGRX vs BGRA.

It can't. Each screen update may end up using a different encoding.

[palves]

if you somehow find another application with transparency that isn't mapped to text yet.

I guess I could try a tree with the BGRA vs BGRX fix, and without the konsole=text fix to confirm. (Though I'm not currently setup to build Xpra.) What would be the expected behavior? Still visual glitches, or something else?

And then when the source turns out as BGRA, then we get an encoding error, and frames are dropped. Did I get that
right?

No, the server will use another non-video encoding as fallback when that happens, without any noticeable delay - just
an ugly log message.

But then I shouldn't have noticed any visual artifacts, no? Or is the falling back actually not triggering properly? (aka another bug.)

With "-d compress", I see server log lines like:

2024-04-05 16:59:20,752 compress:   1.2ms for 1243x27   pixels at    4,118  for wid=1     using      webp with ratio   3.8%  (  132KB to     6KB), sequence    73, client_options={'rgb_format': 'BGRA', 'quality': 100, 'encoder': 'webp', 'flush': 0}, options={'optimize': False, 'auto_refresh': True, 'quality': 100, 'speed': 50, 'rgb_formats': ('YUV420P', 'YUV422P', 'YUV444P', 'GBRP', 'BGRX', 'RGBX', 'RGB', 'BGR', 'NV12'), 'lz4': True, 'alpha': False, 'window-size': (1277, 658)}

whenever the screen updates correctly, but when the h264 conversion error triggers, I see the

  2024-04-05 16:59:38,739 Error: failed to encode h264 frame

errors, but no compress: output, as in, no frame update seems to have been sent at all, aka, no fallback?

As per my previous answer:

Can you run your server with -d compress so we can see what compression it is using?

Ah! I didn't make the connection between "compress" and the encoding, I naivelly assumed that was about a separate compression pass. Sorry.

but it doesn't say which one of those available was selected, nor the detail of BGRX vs BGRA.

It can't. Each screen update may end up using a different encoding.

Oh, that's quite interesting!

Thanks a lot for the fixes.

Now I'm curious on the difference between "text" and "non-text" apps and why that is important. :-) Sorry, can't help myself, I find Xpra quite interesting, and I've been trying to use it for a couple years, but there was always something in the way. Looks like I'll be able to use it from now on. Massive thanks for writing this.

[totaam]

and without the konsole=text fix to confirm

Or you can disable the feature:

xpra/xpra/server/window/content_guesser.py

Line 20 in ca3f0d5

GUESS_CONTENT = envbool("XPRA_GUESS_CONTENT", True)

XPRA_GUESS_CONTENT=0 xpra start ...
You could also override the default on-disk definitions using your own overrides:

xpra/xpra/server/window/content_guesser.py

Line 22 in ca3f0d5

CONTENT_TYPE_DEFS = os.environ.get("XPRA_CONTENT_TYPE_DEFS", "")

XPRA_CONTENT_TYPE_DEFS="class-instance:konsole=video,class-instance:foo=picture" xpra start ..
(untested - but should work)

But then I shouldn't have noticed any visual artifacts, no? Or is the falling back actually not triggering properly? (aka another bug.)

Correct, fixed: dbc4cbf

Now I'm curious on the difference between "text" and "non-text" apps and why that is important.

text applications should never ever be sent using lossy picture compression that could make them blurry on the client.
Whereas browser or video type of applications can / should use lossy compression and downscaling to ensure we can provide a good framerate without consuming Gbps of bandwidth. They also often benefit more from using video encoders.

---

A single 6x13 character in an xterm can be sent as:

• a few bytes using plain BGRX compressed with lz4
• it would take hundreds of bytes to send it as png or jpeg (header overheads, etc) - and cost much more CPU time at both ends
• even more if you tried to use a video encoder (which would have to scan the whole window)
  So knowing what type of content it is dealing with helps xpra make the right decision about how to compress things without wasting resources.

---

Though I'm not currently setup to build Xpra

Once the 5.0.8 aarch64 / riscv64 builds are complete (these can take many days), I will be making new beta 6.0 builds with these fixes.

[totaam]

Works-for-me(tm).