wmww / wayland-debug

A command line tool to help debug Wayland clients and servers

Wayland debug

A CLI for viewing, filtering, and setting breakpoints on Wayland protocol messages.

Quickstart

$ sudo snap install wayland-debug
$ wayland-debug -r gedit
Switching to new client connection A
 0.0000 → [email protected]_registry(registry=new wl_registry@2a)
 0.0000 → [email protected](callback=new wl_callback@3a)
 0.0001 [email protected]_id(id=3) -- [email protected] after 0.0001s ↲
…

Run mode

Enabled with -r/--run, followed by the program you want to debug and it's command line arguments. This will run the program with WAYLAND_DEBUG=1 and parse the resulting protocol messages. If you want to supply a filter or other arguments to wayland-debug you need to do that before the -r.

GDB mode

Enabled with -g/--gdb. All subsequent command line arguments are sent directly to a new GDB instance with wayland-debug running as a plugin. GDB mode supports setting breakpoints on Wayland messages.

GDB mode requires a libwayland that is built with debug symbols and no inlining (ie a debug build). The wayland-debug snap comes with such a libwayland, however if you're not using the snap or on an older/non-libc system, you may need to build libwayland yourself to use GDB mode. See libwayland_debug_symbols.md for details.

Pipe/file modes

libwayland has native support for dumping a simplified version of protocol messages. This is enabled by running a Wayland application with the WAYLAND_DEBUG environment variable set to 1 (or client or server). wayland-debug can parse these messages (either by loading them from a file, or receiving them via stdin). Note that if a program opens multiple Wayland connections the information becomes ambiguous and wayland-debug can't process it (see #5). The run mode works around this by adding a patched libwayland to the LD_LIBRARY_PATH.

Further info

For a list of command line arguments, run:

$ wayland-debug -h

Message matchers are used to filter messages and set breakpoints. For matcher syntax documentation, run:

$ wayland-debug --matcher-help

For a list of GDB mode commands, start GDB mode and run:

(gdb) wlh

To run in GDB mode without using the snap, or if the libwayland from the snap doesn't work for some reason, you need to build libwayland from source.

Examples

In these examples program can be any native Wayland app or server, such as gedit, weston-terminal or sway. wayland-debug can be replaced with ./main.py if you're not using the snap.

Filtering messages

Only show pointer, surface.commit and surface.destroy messages on the first connection (connection A).

wayland-debug -f 'A: wl_pointer, wl_surface.[commit, destroy]' -r program

GDB with arguments

Run the program with arguments immediately, and quit when done (everything after -g is passed to GDB, refer to GDB's docs for details)

wayland-debug -g --ex r --ex q --args program arg1 arg2

GDB breakpoint

Spin up an instance of GDB, and run the program inside it. Show all messages, but break when an XDG thing is configured or when object ID 12 is used.

wayland-debug -b 'xdg_*.configure, 12' -g program
(gdb) run

Piping messages from stdin

This parses libwayland's default debugging output.

WAYLAND_DEBUG=1 program 2>&1 | wayland-debug -p

Loading from a file

Similar to the last example, but loads libwayland output from a file.

WAYLAND_DEBUG=1 program 2>path/to/file.log
wayland-debug -l path/to/file.log

Filtering piped input

Run with piped input. Show all pointer events except .motion and .frame

WAYLAND_DEBUG=1 program 2>&1 1>/dev/null | wayland-debug -p -f 'wl_pointer.[! motion, frame]'

Negative filter

Load a file showing everything but callbacks and frame messages.

wayland-debug -l dir/file.log -f '! wl_callback, .frame'

Running the tests

Run the python3 version of pytest (pytest-3 on Ubuntu) in the project's root directory. The integration tests will attempt to build a Wayland C program, so you'll need the Wayland development libraries as well as meson and ninja.

To install all test dependencies on Ubuntu, run sudo apt install python3-pytest libwayland-dev wayland-protocols gdb meson ninja-build. You'll also need your debug libwayland built (./resources/get-libwayland.sh).

wayland-debug's Issues

Receiving events is *very* slow in GDB mode

I noticed this running in Kate. It can only receive a few dozen pointer events per second. Pretty sure it's wayland-debug's fault and not GDBs (or maybe how wayland-debug is using GDB?). Sending events is much quicker.

Matcher syntax is overcomplicated and annoying

The domain specific matching language was a cool idea, but it currently has some issues. Notably, it's possible to write very specific and short matchers but the syntax is unintuitive. Even I have to look it up some times and I wrote the damn thing. For a quick-and-dirty debugging tool, this is unacceptable. I'd like to rewrite the matcher system using a more generic token system. Basically, the Wayland messages displayed would be a series of tokens (0.1557 → wl_display @ 1 . 0 sync ( callback = new wl_callback @ 18 . 0 )). The matcher would also be a series of tokens, with wildcards and whatnot. This would allow for a more generic matcher implementation, and a more obvious syntax (look at how messages are displayed). A message should be a valid matcher for identical messages.

Lots of assertions errors when trying to debug in GDB mode

Hi, this tool works great when used in run mode but when I try and debug the same program in gdb mode I get a bunch of assertion errors of the form:

Traceback (most recent call last):
  File "/usr/lib/wayland_debug/backends/gdb_plugin/plugin.py", line 70, in stop
    connection_id, message = self.message_extractor()
                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/wayland_debug/backends/gdb_plugin/extract.py", line 153, in sent_message
    object = wl.UnresolvedObject(object_id, None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/wayland_debug/core/wl/object.py", line 85, in __init__
    super().__init__(obj_id)
  File "/usr/lib/wayland_debug/core/wl/object.py", line 12, in __init__
    assert obj_id > 0
           ^^^^^^^^^^
AssertionError

When I open GDB via the flag, I also see:

Function "wl_connection_destroy" not defined.
Function "wl_closure_invoke" not defined.
Function "wl_closure_dispatch" not defined.
Function "serialize_closure" not defined.

Any ideas as to what might be going on?

No version

The project doesn't really have a version, -v is the verbose flag and --version isn't a thing. Having a version is kinda expected for all software and it being accessible via -v/--version is generally expected for command line tools.

Add the project to Freedesktop's Wayland debugging utility list

Freedesktop keeps track of debugging utilities at https://wayland.freedesktop.org/extras.html and it'd be nice to have this tool listed there.
It looks like that a PR to https://gitlab.freedesktop.org/wayland/wayland.freedesktop.org could be made to make that happen.

Need integration tests

Unit test coverage is coming right along, but we still need integrations tests

KWin debug?

Hi, thanks for the tool!
I wonder if KWin itself could also be debugged with it?

Can not use load-from-file mode in pipe

It stops at the end to wait for input, should auto-detect it's not being run interactively and not do that.

Can not handle server-created objects in loaded file/WAYLAND_DEUBG mode

It seems #27 may not have been solved for this case, failing test on the way.

Can not deal with multiple connections when parsing libwayland's `WAYLAND_DEBUG=1` output

Currently there is no possible solution to this problem (workarounds below). I'm creating this issue as an explanation.

Background

Any Wayland client or server built with libwayland produces Wayland messages on stderr when run with the environment variable WAYLAND_DEBUG=1 set. The produced output is similar to wayland-debug's unfiltered output, but uglier and with less context. There is one critical missing piece: there is no info about which connection a message is on.

wayland-debug fails when parsing WAYLAND_DEBUG=1 output with multiple connections. I will not put effort into making it continue to syntax highlight in this case, as most of it's other features would still be impossible. Instead, you can try...

Workarounds

GDB mode

If you can run as a GDB extension (./main.py -g some_wayland_compositor), this problem goes away. GDB mode is great, and has extra features (breakpoints on Wayland messages!), but you need libwayland debug symbols installed and it can be slow in some cases.

Run wayland-debug on the client you care about

Clients generally only make a single Wayland connection, so using the WAYLAND_DEBUG=1 output from a client instead of a compositor might fix the problem.

Only spew client or server messages

Instead of WAYLAND_DEBUG=1, you can also run a libwayland app with WAYLAND_DEBUG=client or WAYLAND_DEBUG=server. This will only show messages from that type of connection. This usually won't work, as the most common problem is when a compositor has multiple client connections. Even if you are only running one client, wallpaper and panels may use Wayland internally.

Support for weston-debug proto debug scope as read end of a pipe

Hi,

Recently weston has got weston-debug application which can piggy back on the protocol as well. It can be invoked using weston-debug proto. It supports displaying multiple clients and has a slight different format WAYLAND_DEBUG=1, but I think would be nice to get pretty formatting and filtering support.

I guess would be nice to do weston-debug proto | ./main.py [options]

Example of weston-debug proto dump with multiple clients.
[2018-12-27 18:55:57.228][proto] client 0x555ed56d3970 rq [email protected]()
[2018-12-27 18:55:57.228][proto] client 0x555ed56d3970 ev [email protected]()
[2018-12-27 18:55:57.228][proto] client 0x555ed56d3970 rq [email protected]_opaque_region(nil)
[2018-12-27 18:55:57.243][proto] client 0x555ed56c8620 ev [email protected](31418518)
[2018-12-27 18:55:57.243][proto] client 0x555ed56c8620 ev [email protected]_id(16)
[2018-12-27 18:55:57.243][proto] client 0x555ed56d3970 ev [email protected](31418518)
[2018-12-27 18:55:57.243][proto] client 0x555ed56d3970 ev [email protected]_id(16)
[2018-12-27 18:55:57.244][proto] client 0x555ed56d3970 rq [email protected](new id wl_callback@16)
[2018-12-27 18:55:57.244][proto] client 0x555ed56d3970 rq [email protected](wl_buffer@19, 0, 0)
[2018-12-27 18:55:57.244][proto] client 0x555ed56d3970 rq [email protected]_buffer(61, 62, 127, 127)
[2018-12-27 18:55:57.244][proto] client 0x555ed56d3970 rq [email protected]()
[2018-12-27 18:55:57.244][proto] client 0x555ed56d3970 ev [email protected]()
[2018-12-27 18:55:57.244][proto] client 0x555ed56c8620 rq [email protected](new id wl_callback@16)
[2018-12-27 18:55:57.244][proto] client 0x555ed56c8620 rq [email protected](wl_buffer@19, 0, 0)
[2018-12-27 18:55:57.244][proto] client 0x555ed56c8620 rq [email protected]_buffer(61, 62, 127, 127)
[2018-12-27 18:55:57.244][proto] client 0x555ed56c8620 rq [email protected]()
[2018-12-27 18:55:57.244][proto] client 0x555ed56c8620 ev [email protected]()
[2018-12-27 18:55:57.244][proto] client 0x555ed56c8620 rq [email protected]_opaque_region(nil)
[2018-12-27 18:55:57.244][proto] client 0x555ed56d3970 rq [email protected]_opaque_region(nil)
[2018-12-27 18:55:57.260][proto] client 0x555ed56c8620 ev [email protected](31418535)
[2018-12-27 18:55:57.260][proto] client 0x555ed56c8620 ev [email protected]_id(16)
[2018-12-27 18:55:57.260][proto] client 0x555ed56d3970 ev [email protected](31418535)
[2018-12-27 18:55:57.260][proto] client 0x555ed56d3970 ev [email protected]_id(16)
[2018-12-27 18:55:57.261][proto] client 0x555ed56d3970 rq [email protected](new id wl_callback@16)
[2018-12-27 18:55:57.261][proto] client 0x555ed56d3970 rq [email protected](wl_buffer@20, 0, 0)
[2018-12-27 18:55:57.261][proto] client 0x555ed56d3970 rq [email protected]_buffer(61, 62, 127, 127)
[2018-12-27 18:55:57.261][proto] client 0x555ed56d3970 rq [email protected]()
[2018-12-27 18:55:57.261][proto] client 0x555ed56d3970 ev [email protected]()
[2018-12-27 18:55:57.261][proto] client 0x555ed56c8620 rq [email protected](new id wl_callback@16)
[2018-12-27 18:55:57.261][proto] client 0x555ed56c8620 rq [email protected](wl_buffer@20, 0, 0)
[2018-12-27 18:55:57.261][proto] client 0x555ed56c8620 rq [email protected]_buffer(61, 62, 127, 127)
[2018-12-27 18:55:57.261][proto] client 0x555ed56c8620 rq [email protected]()
[2018-12-27 18:55:57.261][proto] client 0x555ed56c8620 ev [email protected]()
[2018-12-27 18:55:57.261][proto] client 0x555ed56d3970 rq [email protected]_opaque_region(nil)
[2018-12-27 18:55:57.261][proto] client 0x555ed56c8620 rq [email protected]_opaque_region(nil)
[2018-12-27 18:55:57.276][proto] client 0x555ed56c8620 ev [email protected](31418552)
[2018-12-27 18:55:57.277][proto] client 0x555ed56c8620 ev [email protected]_id(16)
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 ev [email protected](31418552)
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 ev [email protected]_id(16)
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 rq [email protected](new id wl_callback@16)
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 rq [email protected](wl_buffer@18, 0, 0)
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 rq [email protected]_buffer(61, 62, 127, 127)
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 rq [email protected]()
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 ev [email protected]()
[2018-12-27 18:55:57.277][proto] client 0x555ed56d3970 rq [email protected]_opaque_region(nil)
[2018-12-27 18:55:57.277][proto] client 0x555ed56c8620 rq [email protected](new id wl_callback@16)
[2018-12-27 18:55:57.277][proto] client 0x555ed56c8620 rq [email protected](wl_buffer@18, 0, 0)
[2018-12-27 18:55:57.277][proto] client 0x555ed56c8620 rq [email protected]_buffer(61, 62, 127, 127)
[2018-12-27 18:55:57.277][proto] client 0x555ed56c8620 rq [email protected]()
[2018-12-27 18:55:57.277][proto] client 0x555ed56c8620 ev [email protected]()
[2018-12-27 18:55:57.278][proto] client 0x555ed56c8620 rq [email protected]_opaque_region(nil)

GDB mode lacks message timestamps

When running in GDB, all messages are timestamped 0.0. This is set here, and should instead capture the current time. Setting it to a baseless timestamp should "just work", as I believe the timestamp base is set to the first message timestamp automatically.

main.py is messy

main.py does too much, should be split up into other files

Pipe/file mode regex broken with current libwayland main

Not sure when libwayland introduced the change. Just an added space at the front of timestamp. Should be trivial to fix.

Performence tests

Would be nice to be able to measure performance, especially of the GDB backend

When loading messages from file, can not enter commands after break

Run ./main.py -l sample_logs/short.log -b [global]
Expected: stops at the first [email protected] global()
Actual: Says it stopped, but keeps running to the end

The upcoming integration tests will include a failing test for this

libwayland debug symbol detection broken on Ubuntu

On Ubuntu (tested on 19.04 development release, as of 2/21/19), I get the error Error: Installed libwayland lacks debug symbols, GDB mode will not function. If I then run the program, it works fine.

Uses non-std logging

Should be using the logging module from the standard library

Unresolved object when server generated IDs are reused

This needs tests and a fix. I noticed it in gedit when copying/pasting. What's happening is server-generated IDs (those above 4278190080 don't seem to get a delete_id event, so when they're reused wayland-debug gets confused. The problem is only visible when the type changes, as apparently recreating the same ID of the same type is allowed (this should probably also be addressed)

Mir crashes when XWayland client connects

This happens when running in GDB with wayland-debug, but NOT when just running in GDB

Matcher can't match connection

As noted in #5, matchers can't match on a particular connection. The list command can filter by connection by prepending the connection name and a colon to the matcher (ex B: .foo will match the .foo messages only in the B connection), but this is implemented in Controller.list_command instead of matcher.py. We should change this so this syntax can be used for all matchers including filters and breakpoints.

Doesn't work well for FF in GDB mode

When I run Firefox with it in GDB mode there are lots of thread warnings and I've also seen crashes related to gtk_primary_selection_device.

Fixed point accuracy is low

Expected-fail integration test test_extracts_fixed_point_numbers_with_high_accuracy tests this. Not sure of the cause. Maybe libwayland is loosing the accuracy?? Not a priority.

pipe and file mode broken in the snap

$ WAYLAND_DEBUG=1 gedit 2>&1 | wayland-debug -p
cannot change profile for the next exec call: No such file or directory

Works fine not in the snap. Snap is classic (unconfined). Strange.

Some object IDs too big

We seem to be getting object IDs in the billions, and they are not consistently recognized as the same number (perhaps because depending on how they come in, they are stored as different int types and overflow differently). The types I've observed this on are wl_data_offer and gtk_primary_selection_offer. We should now just leave those objects unresolved and process everything else normally.