Wrong multi monitor capturing on Ubuntu 20.04 / Debian Buster 10.9 about firefly_luciferin

[sblantipodi]

this is the best description of an issue that I have seen on GitHub since ever. wow.
I will look into it, hope to give you an answer soon and possibly a fix in the same time.

Thank you Patrick.

[sblantipodi]

@patrick-blom
thank you for the best bug report I have seen and for helping me in fixing the problem.

This fix will be merged on the master branch in some days, once merged the issue will be closed automatically.

I hope to continue to see you around Patrick!

Thanks,
Davide

[sblantipodi]

@patrick-blom can you uninstall Firefly Luciferin and try this ALPHA version please?
In this URL, you can find and download the ALPHA artifacts.
https://github.com/sblantipodi/firefly_luciferin/actions/runs/839056957

Does it fix "the duplication problem"?

[patrick-blom]

@sblantipodi yes, in the ALPHA that looks good.

Now the behavior is as follows:
The only monitor which leds are active, is atm Mon2. If I drag a window to the right side of Mon 1, the leds on the right side of Mon 2 are active, If I drag a window to the left side of Mon 3, the leds on the left side of Mon 2 are on. Same with up and down.

It seems, that now the capturing effects all monitors but only use Mon 2 (the center) as output, but I could not reproduce any duplication. No matter which direction, the output is not duplicated and fixed on Mon 2.

So yes, the duplication problem seems to be fixed. 🤓 Nice work 👍

Additional Info: The instance for Mon 3, does not show a Producing or Consuming FPS count. The instance for Mon 2 does

[sblantipodi]

@patrick-blom on Discord you wrote me this: "btw: I disabled the lid for testing purposes, that worked like a charm 😄"

can you elaborate this please? what happened when you disabled the lid using the current stable Firefly Luciferin?

[patrick-blom]

@sblantipodi After a detailed check I think I have to revert that post 🙈 😕The stable version also has the duplication bug when the lid is closed.

The Alpha behaves nearly the same when the lid is closed but with a different output. If the lid is closed Mon 3 becomes Mon 1 and Mon 2 stays the same.

If that happens, Mon 1 will be used as Output and the leds of the right side will turn on if I drag a Windows to the right side of Mon 2

[sblantipodi]

I'm not able to reproduce it so I need to ask you some more questions...
what happen if you drag a windows on the side of Mon2 using the ALPHA version?

Does the led behind the mon2 turn on correctly?

if you open a console and write this command:
gst-launch-1.0 ximagesrc display-name=:3 ! videoscale ! videoconvert ! autovideosink

a window should open and capture what is shown on screen 3, can you confirm it? if no, what do you see?

can you do the same with the same command changing the display name?
gst-launch-1.0 ximagesrc display-name=:2 ! videoscale ! videoconvert ! autovideosink
and
gst-launch-1.0 ximagesrc display-name=:1 ! videoscale ! videoconvert ! autovideosink

thank you!!!!

[patrick-blom]

No problem! Ask as many questions as needed.

"
what happen if you drag a windows on the side of Mon2 using the ALPHA version?
Does the led behind the mon2 turn on correctly?
"
No, it does not work correctly. The leds on the side of Mon 2 (left & right) only light up if I drag a window to the sides of Mon 1 and Mon 3.

The output of the commands are as follows (I translated it using google translate because my OS runs on the german language):

~/Downloads » gst-launch-1.0 ximagesrc display-name=:2 ! videoscale ! videoconvert ! autovideosink  
Line is set to PAUSED ...
ERROR: Line does not want to be paused.
ERROR: From element / GstPipeline: pipeline0 / GstXImageSrc: ximagesrc0: Could not open X display for reading
Additional error diagnostic information:
gstximagesrc.c (160): gst_ximage_src_open_display (): / GstPipeline: pipeline0 / GstXImageSrc: ximagesrc0:
NULL returned from getting xcontext
Line is set to ZERO ...
Line is emptied ...

~/Downloads » gst-launch-1.0 ximagesrc display-name=:3 ! videoscale ! videoconvert ! autovideosink     
Line is set to PAUSED ...
ERROR: Line does not want to be paused.
ERROR: From element / GstPipeline: pipeline0 / GstXImageSrc: ximagesrc0: Could not open X display for reading
Additional error diagnostic information:
gstximagesrc.c (160): gst_ximage_src_open_display (): / GstPipeline: pipeline0 / GstXImageSrc: ximagesrc0:
NULL returned from getting xcontext
Line is set to ZERO ...
Line is emptied ...

If I run the command with display-name=:1 I will get a new window with this content:

Console output:

gst-launch-1.0 ximagesrc display-name =: 1! videoscale! videoconvert! autovideosink
Line is set to PAUSED ...
Line is active and does not require a FORWARD ...
Line is set to PLAY ...
New clock: GstSystemClock
WARNING: From element / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage: Many buffers are discarded.
Additional error diagnostic information:
gstbasesink.c (2902): gst_base_sink_is_too_late (): / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage:
There may be a time stamping problem, or this computer is too slow.
WARNING: From element / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage: Many buffers are discarded.
Additional error diagnostic information:
gstbasesink.c (2902): gst_base_sink_is_too_late (): / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage:
There may be a time stamping problem, or this computer is too slow.
WARNING: From element / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage: Many buffers are discarded.
Additional error diagnostic information:
gstbasesink.c (2902): gst_base_sink_is_too_late (): / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage:
There may be a time stamping problem, or this computer is too slow.
WARNING: From element / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage: Many buffers are discarded.
Additional error diagnostic information:
gstbasesink.c (2902): gst_base_sink_is_too_late (): / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage:
There may be a time stamping problem, or this computer is too slow.
WARNING: From element / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage: Many buffers are discarded.
Additional error diagnostic information:
gstbasesink.c (2902): gst_base_sink_is_too_late (): / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage:
There may be a time stamping problem, or this computer is too slow.
WARNING: From element / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage: Many buffers are discarded.
Additional error diagnostic information:
gstbasesink.c (2902): gst_base_sink_is_too_late (): / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage:
There may be a time stamping problem, or this computer is too slow.
ERROR: From Element / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage: Output window was closed
Additional error diagnostic information:
xvimagesink.c (555): gst_xv_image_sink_handle_xevents (): / GstPipeline: pipeline0 / GstAutoVideoSink: autovideosink0 / GstXvImageSink: autovideosink0-actual-sink-xvimage
Execution ended after 0: 00: 16.857567016
Line is set to PAUSED ...
Line is set to READY ...
Line is set to ZERO ...
Line is emptied ...

Looks like, the problem is not Luciferin, I think its more my OS / that ximagesrc component that inteprets all 3 displays as 1 big display 🤔

[patrick-blom]

@sblantipodi
I played around with that command you wrote me and I was able to capture all my diplays separately by providing the start / end positions.

In my setup ponit 0.0 is the upper left corner of Mon 3 so I could capture each display using the following commands:

• Mon1: gst-launch-1.0 ximagesrc display-name=:1.0 startx=3840 ! videoscale ! videoconvert ! autovideosink
• Mon2: gst-launch-1.0 ximagesrc display-name=:1.0 startx=1920 endx=3840 ! videoscale ! videoconvert ! autovideosink
• Mon3: gst-launch-1.0 ximagesrc display-name=:1.0 endx=1920 ! videoscale ! videoconvert ! autovideosink

All displays are on 1920x1080

Just looked at your code and had an idea, of cause I'm not a Java dev but maybe it's somekind of helpful.

If I get it right, it looks like gstreamer behaves different on linux than on windows. I seems to me, that on windows the displays are separated but on linux they will be combined to one big screen.

So maybe you could solve the problem by doing some math and provide the startx / endx positions at this point here:

https://github.com/sblantipodi/firefly_luciferin/blob/master/src/main/java/org/dpsoftware/FireflyLuciferin.java#L272

The display resolution can be fetched from the settings, I guess.

Cheers Patrick

[sblantipodi]

@patrick-blom
you are not a java dev but from your answers I can understand that you are a dev 😄

can you tell me if this command shows only the contents on display 1 please?
Mon1: gst-launch-1.0 ximagesrc display-name=:1.0 startx=3840 ! videoscale ! videoconvert ! autovideosink

this shows only contents for monitor 2?
Mon2: gst-launch-1.0 ximagesrc display-name=:1.0 startx=1920 endx=3840 ! videoscale ! videoconvert ! autovideosink

and this monitor 3 only?
Mon3: gst-launch-1.0 ximagesrc display-name=:1.0 startx=1920 endx=3840 ! videoscale ! videoconvert ! autovideosink

[patrick-blom]

@sblantipodi 😄 Thx

Regarding your questions:

can you tell me if this command shows only the contents on display 1 please?
Mon1: gst-launch-1.0 ximagesrc display-name=:1.0 startx=3840 ! videoscale ! videoconvert ! autovideosink

Yes it does

this shows only contents for monitor 2?
Mon2: gst-launch-1.0 ximagesrc display-name=:1.0 startx=1920 endx=3840 ! videoscale ! videoconvert ! autovideosink

Yes it does

and this monitor 3 only?
Mon3: gst-launch-1.0 ximagesrc display-name=:1.0 startx=1920 endx=3840 ! videoscale ! videoconvert ! autovideosink

No it doesn't because its the same command as for monitor 2 ^^

This command will only show the contents of monitor 3
Mon3: gst-launch-1.0 ximagesrc display-name=:1.0 startx=0 endx=1920 ! videoscale ! videoconvert ! autovideosink

This will work aswell:
Mon3: gst-launch-1.0 ximagesrc display-name=:1.0 endx=1920 ! videoscale ! videoconvert ! autovideosink

[sblantipodi]

@patrick-blom
if you want to try another apha, here it is:
https://github.com/sblantipodi/firefly_luciferin/actions/runs/843386229

thank you again!

[sblantipodi]

please note that on Linux Luciferin capture the screen only if something is moving on the screen.
just in case you see 0FPS on a screen that have nothing moving on on it.

[patrick-blom]

@sblantipodi Funny thing regarding the fps, even if there is no movement on the monitors, its even producing 1.2-1.7 fps ^^

But back top topic. First off all this alpha works much better than the version before, thx a lot 🤓👍

The verison, in detail, behaves likes this:

• All movements of Mon 3 are fine. There is no unexpected behavior.
• The leds of Mon 2 are now more dim than the ones of Mon 3. You can reproduce that if you drag a windows slowly along the top of both mon's.( All instances at 100% brightness )
• The leds displays of mon 2 and mon 1 are still combinded, so the right leds of mon 2 will be on if I move a window to the right side of mon 1. Same for top and down movements.
• If I place a window between mon 1 and mon 2 (half of both) no leds will lite up. I think that happens, because of ximagesrc thinks I'm in the middle of a screen. I would expect, that only the leds of the right side of mon 2 will be on.
• If I place a window between mon 2 and mon 3 only the leds of mon 3 (right side) will turn on not the leds of mon 2 (left side).

Regarding the last point (mon2 & mon3), I dont know the behavior of the windows version, but I think that the current behavior is totally ok. Because I would expect that the whole background light has the same intensity over all leds / monitors. If both sides where on mon 3 (right side) and mon 2 (left side), it might be to bright because of double amount of the leds.

Cheers Patrick

[sblantipodi]

@patrick-blom
1-2FPS is normal, the important thing is that the FPS increase if something is moving on the screen.

I'm sorry for the time you are wasting behind those trials and errors Patrick.
This should really fix it.
https://github.com/sblantipodi/firefly_luciferin/actions/runs/844831472

[patrick-blom]

@sblantipodi That's absolutely no problem. I know how hard debugging is, and every minute that I can help you with testing the alpha is a minute that makes Luciferin a better application. So even if this problem is fixed, if you need someone to test a new feature under linux, besides your own tests, feel free to ping me 🤓

Back to the alpha, sry but nope 🙈:
The behavior is nearly the same to the last version, with slight differences.

• The dim problem of mon 2 is still present also that it seems mon 1 and mon 2 are combined and seen as one big display.
• Mon 3 has recreated the duplication bug. But only for the left side. So if I drag a window to the left side the leds of mon 3 (left) and mon 2 (left) will turn on.
• The combination of mon 1 and mon 2. It seems that the virtual edge of the right side of mon 1 has moved into the left direction. In the last version I had to drag the window to the very outer edge of mon 1 to light up the right leds of mon 2. Now the lights will turn on earlier, I think a few hundred pixels.

last version

current version

[sblantipodi]

@patrick-blom
I finally attached another monitor to my Linux box, and here it is.
https://github.com/sblantipodi/firefly_luciferin/actions/runs/847579949

[patrick-blom]

@sblantipodi
I just installed the latest alpha, and what should I say, it works perfectly 😍!

• no doublication
• the binding of monitors works as expeteced
• all leds are at the same level of brightness

Thank you for all your effort and patience to fix this bug! Great Job 👍