meson: group all tests under suite names

Three suite names to allow for filtering tests: 'valgrind', 'root',
'hardware'. The latter two require root/hardware to succeed, the former labels
tests that should be run under valgrind.

Usage is documented in the docs now, but basically:

$ meson test --setup=valgrind --suite=valgrind
$ meson test --no-suite=root

This is documented a bit now and because we now rely on meson test, let's
replace all ninja test invocations with meson test instead for consistency.

Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>

doc/user/test-suite.rst

@@ -4,18 +4,37 @@
 libinput test suite
 ==============================================================================
 
-libinput ships with a number of tests all run automatically on ``ninja test``.
-The primary test suite is the ``libinput-test-suite``. When testing,
-the ``libinput-test-suite`` should always be invoked to check for
-behavior changes. The test suite relies on the kernel and udev to function
-correctly. It is not suitable for running inside containers.
+libinput's primary test suite can be invoked with
 
-.. note:: ``ninja test`` runs more than just the test suite, you **must**
-	run all tests for full coverage.
+::
 
-The test suite runner uses
-`Check <http://check.sourceforge.net/doc/check_html/>`_ underneath the hood
-but most of the functionality is abstracted into *litest* wrappers.
+	$ sudo ./builddir/libinput-test-suite
+
+When developing libinput, the ``libinput-test-suite`` should always be
+run to check for behavior changes and/or regressions. For quick iteration,
+the number of tests to run can be filtered, see :ref:`test-filtering`.
+This allows for developers to verify a subset of tests (e.g.
+touchpad tap-to-click) while hacking on that specific feature and only run
+the full suite when development is done finished.
+
+.. note:: The test suite relies on udev and the kernel, specifically uinput.
+	It creates virtual input devices and replays the events. This may
+	interfere with your running session. The test suite is not suitable
+	for running inside containers.
+
+In addition, libinput ships with a set of (primarily janitorial) tests that
+must pass for any merge request. These tests are invoked by calling
+``meson test -C builddir`` (or ``ninja test``). The ``libinput-test-suite`` is
+part of that test set by default.
+
+The upstream CI runs all these tests but not the ``libinput-test-suite``.
+This CI is run for every merge request.
+
+.. _test-job-control:
+
+------------------------------------------------------------------------------
+Job control in the test suite
+------------------------------------------------------------------------------
 
 The test suite runner has a make-like job control enabled by the ``-j`` or
 ``--jobs`` flag and will fork off as many parallel processes as given by this
@@ -47,8 +66,8 @@ Permissions required to run tests
 Most tests require the creation of uinput devices and access to the
 resulting ``/dev/input/eventX`` nodes. Some tests require temporary udev rules.
 **This usually requires the tests to be run as root**. If not run as
-root, the test suite runner will exit with status 77, interpreted as
-"skipped" by ninja.
+root, the test suite runner will exit with status 77, an exit status
+interpreted as "skipped".
 
 .. _test-filtering:
 
@@ -161,7 +180,7 @@ environment variable, if set, also enables verbose mode.
 ::
 
      $ ./builddir/libinput-test-suite --verbose
-     $ LITEST_VERBOSE=1 ninja test
+     $ LITEST_VERBOSE=1 meson test -C builddir
 
 .. _test-installed:
 
@@ -190,3 +209,35 @@ To configure libinput to install the tests, use the ``-Dinstall-tests=true``
 meson option::
 
   $ meson builddir -Dtests=true -Dinstall-tests=true <other options>
+
+.. _test-meson-suites:
+
+------------------------------------------------------------------------------
+Meson test suites
+------------------------------------------------------------------------------
+
+This section is primarily of interest to distributors that want to run test
+or developers working on libinput's CI.
+
+Tests invoked by ``meson test`` are grouped into test suites, the test suite
+names identify when the respective test can be run:
+
+- ``valgrind``: tests that can be run under valgrind (in addition to a
+  normal run)
+- ``root``: tests that must be run as root
+- ``hardware``: tests that require a VM or physical machine
+- ``all``: all tests, only needed because of
+  `meson bug 5340 <https://github.com/mesonbuild/meson/issues/5340>`_
+
+The suite names can be provided as filters to ``meson test
+--suite=<suitename>`` or ``meson test --no-suite=<suitename>``.
+For example, if running a container-based CI, you may specify the test
+suites as:
+
+::
+
+   $ meson test --no-suite=machine  # only run container-friendly tests
+   $ meson test --suite=valgrind --setup=valgrind  # run all valgrind-compatible tests
+   $ meson test --no-suite=root  # run all tests not requiring root
+
+These suites are subject to change at any time.

meson.build

@@ -275,7 +275,9 @@ quirks_data = [
 
 test('quirks-in-meson.build',
      find_program('quirks/test-quirks-in-meson.build.sh'),
-     args : [meson.source_root()])
+     args : [meson.source_root()],
+     suite : ['all']
+     )
 
 config_h.set_quoted('LIBINPUT_QUIRKS_FILES', ':'.join(quirks_data))
 config_h.set_quoted('LIBINPUT_QUIRKS_SRCDIR', dir_src_quirks)
@@ -438,7 +440,8 @@ libinput_quirks = executable('libinput-quirks',
 			    )
 test('validate-quirks',
      libinput_quirks,
-     args: ['validate', '--data-dir=@0@'.format(dir_src_quirks)]
+     args: ['validate', '--data-dir=@0@'.format(dir_src_quirks)],
+     suite : ['all']
      )
 
 configure_file(input : 'tools/libinput-quirks.man',
@@ -469,7 +472,9 @@ libinput_list_devices = executable('libinput-list-devices',
 				   install_dir : libinput_tool_path,
 				   install : true,
 				  )
-test('list-devices', libinput_list_devices)
+test('list-devices',
+     libinput_list_devices,
+     suite : ['all', 'root', 'hardware'])
 
 configure_file(input : 'tools/libinput-list-devices.man',
 	       output : 'libinput-list-devices.1',
@@ -610,6 +615,7 @@ if get_option('buildtype') == 'debug' or get_option('buildtype') == 'debugoptimi
 	test('tool-option-parsing',
 	     find_program('tools/test-tool-option-parsing.py'),
 	     args : ['--tool-path', libinput_tool.full_path()],
+	     suite : ['all', 'root'],
 	     timeout : 240)
 endif
 
@@ -623,18 +629,21 @@ test_builddir_lookup = executable('test-builddir-lookup',
 				  install : false)
 test('tools-builddir-lookup',
      test_builddir_lookup,
-     args : ['--builddir-is-set'])
+     args : ['--builddir-is-set'],
+     suite : ['all'])
 test('tools-builddir-lookup-installed',
      find_program('test/helper-copy-and-exec-from-tmp.sh'),
      args : [test_builddir_lookup.full_path(), '--builddir-is-null'],
      env : ['LD_LIBRARY_PATH=@0@'.format(meson.build_root())],
+     suite : ['all'],
      workdir : '/tmp')
 
 ############ tests ############
 
 test('symbols-leak-test',
      find_program('test/symbols-leak-test'),
-     args : [ join_paths(dir_src, 'libinput.sym'), dir_src])
+     args : [ join_paths(dir_src, 'libinput.sym'), dir_src],
+     suite : ['all'])
 
 # build-test only
 executable('test-build-pedantic',
@@ -671,7 +680,10 @@ if get_option('tests')
 	dep_check = dependency('check', version : '>= 0.9.10')
 
 	leftover_rules = find_program('test/check-leftover-udev-rules.sh')
-	test('leftover-rules', leftover_rules, is_parallel : false)
+	test('leftover-rules',
+	     leftover_rules,
+	     is_parallel : false,
+	     suite : ['all'])
 
 	gstack = find_program('gstack', required : false)
 	config_h.set10('HAVE_GSTACK', gstack.found())
@@ -802,14 +814,18 @@ if get_option('tests')
 					  dependencies : deps_litest,
 					  c_args : defs_litest_selftest,
 					  install : false)
-	test('test-litest-selftest', test_litest_selftest)
+	test('test-litest-selftest',
+	     test_litest_selftest,
+	     suite : ['all'])
 
 	def_LT_VERSION = '-DLIBINPUT_LT_VERSION="@0@:@1@:@2@"'.format(libinput_lt_c, libinput_lt_r, libinput_lt_a)
 	test_library_version = executable('test-library-version',
 					  ['test/test-library-version.c'],
 					  c_args : [ def_LT_VERSION ],
 					  install : false)
-	test('test-library-version', test_library_version)
+	test('test-library-version',
+	     test_library_version,
+	     suite : ['all'])
 
 	test_utils_sources = [
 		'src/libinput-util.h',
@@ -821,7 +837,9 @@ if get_option('tests')
 				include_directories : [includes_src, includes_include],
 				dependencies : deps_litest,
 				install: false)
-	test('test-utils', test_utils)
+	test('test-utils',
+	     test_utils,
+	     suite : ['all'])
 
 	libinput_test_runner_sources = litest_sources + [
 		'src/libinput-util.h',
@@ -860,10 +878,12 @@ if get_option('tests')
 
 	test('libinput-test-suite-runner',
 	     libinput_test_runner,
+	     suite : ['all', 'valgrind', 'root', 'hardware'],
 	     timeout : 1200)
 
 	test('libinput-test-deviceless',
 	     libinput_test_runner,
+	     suite : ['all', 'valgrind'],
 	     args: ['--filter-deviceless'])
 
 	valgrind = find_program('valgrind', required : false)