xserver/test/pyxtest

pyxtest - pytest-based X server test suite

This is a pytest-based test suite that launches X servers and sends crafted protocol requests to verify that security vulnerabilities and other bugs are properly handled.

It can be run against Xvfb, Xwayland, or Xorg but the latter is potentially flaky and requires some setup outside the test suite. The test suite uses both AddressSanitizer (ASAN) and valgrind for detecting memory errors such as out-of-bounds reads/writes and use-after-free.

Running tests

Via meson

The test suite (via Xvfb) is integrated into the meson tests and can be run with normal meson commands.

# run the python test suite
meson test --suite pyxtest
# run a set of tests
meson test pyxtest-test_randr.py

Consult the meson documentation for further details.

Directly with pytest

For running against a custom path, point the test suite at the server binary to test using environment variables or CLI options:

# Using environment variable
XVFB_PATH=build/hw/vfb/Xvfb pytest test/pyxtest/ -v

# Using --server-path
pytest test/pyxtest/ -v --server-path=build/hw/vfb/Xvfb

# Using the system Xvfb (fallback if no path is set)
pytest test/pyxtest/ -v

The normal pytest options work as expected (-k for test selection, etc.)

Running with AddressSanitizer (ASAN)

ASAN is a compile-time instrumentation that detects memory errors such as heap buffer overflows and use-after-free. To use ASAN, build the server with sanitizer support:

meson setup build-asan -Db_sanitize=address -Db_lundef=false
meson compile -C build-asan

Then run the tests against the ASAN-built binary:

XSERVER_ASAN=1 XVFB_PATH=build-asan/hw/vfb/Xvfb pytest test/pyxtest/ -v

When using meson test, XSERVER_ASAN is set automatically if the build was configured with -Db_sanitize=address.

Tests marked with @pytest.mark.asan are skipped unless XSERVER_ASAN=1 is set. When ASAN detects an error, the server process is killed and the ASAN error report is included in the test failure message.

Note: ASAN and valgrind are mutually incompatible. When XSERVER_ASAN=1 is set, valgrind wrapping is automatically disabled even if --valgrind is passed.

Running with valgrind

The --valgrind flag runs all servers under valgrind:

pytest test/pyxtest/ -v --valgrind

Tests marked with @pytest.mark.valgrind automatically run their server under valgrind even without the --valgrind flag. This is useful for bugs that are only detectable via valgrind (e.g. use of uninitialised values).

Testing multiple server types

By default only Xvfb is tested. Use --server-type to test additional servers. Tests using the xserver fixture are automatically run once per server type:

pytest test/pyxtest/ -v --server-type=xvfb --server-type=xwayland

CLI options

Option	Description
--valgrind	Run all X servers under valgrind memcheck
--valgrind-suppressions=PATH	Path to a valgrind suppressions file
--server-type=TYPE	Server type to test (xvfb, xwayland, xorg). Repeatable. Default: xvfb
--server-path=PATH	Explicit path to the X server binary

Environment variables

The server binary is located by checking, in order:

1. --server-path CLI option
2. XVFB_PATH / XWAYLAND_PATH / XORG_PATH environment variable
3. XSERVER_BUILDDIR environment variable (looks for hw/vfb/Xvfb etc.)
4. build/ directory relative to the source root
5. System PATH (prints a warning)

VALGRIND_SUPPRESSIONS can point to a suppressions file.

XSERVER_ASAN set to 1 indicates the server binary was built with AddressSanitizer. This is set automatically by meson when -Db_sanitize=address is used. It can also be set manually.

Test markers

Marker	Effect
@pytest.mark.asan	Test requires ASAN (XSERVER_ASAN=1 must be set
@pytest.mark.valgrind	Test requires valgrind (skipped if XSERVER_ASAN is set)
@pytest.mark.xwayland_only	Test is skipped unless --server-type=xwayland
@pytest.mark.xorg_only	Test is skipped unless --server-type=xorg
@pytest.mark.swapped_client	Test uses a byte-swapped (big-endian) client connection

Writing a new test

1. Create or edit a test_*.py file.

2. Use the xserver and xclient or xclient_swapped fixtures to get a running server and connection:

   def test_something(self, xserver, xclient):
       # xclient is a RawX11Connection to xserver
       ...
       assert xserver.is_alive, "Server crashed"
   

3. If the test needs an extension, create a fixture that handles negotiation:

   @pytest.fixture
   def render_client(xclient):
       ext = xclient.query_extension(Extension.RENDER)
       if not ext:
           pytest.skip("RENDER not available")
       # ... send version negotiation ...
       return xclient
   

4. Build protocol requests using dataclasses from proto/:

   from proto import xi
   
   req = xi.XIChangeHierarchyRequest(
       opcode=opcode,
       num_changes=1,
       changes_data=change_data,
   )
   xclient.send_request(req.to_bytes())
   

5. If a new extension module is needed, create proto/myext.py with constants and @dataclass request builders following the existing pattern.

6. If the bug is only detectable via a memory sanitizer (OOB reads, use-after-free), mark the test with @pytest.mark.asan. Use @pytest.mark.valgrind only for bugs that specifically require valgrind (e.g. use of uninitialised values that ASAN does not detect).