randr: Check for overflow in RRChangeProviderProperty()

A client might send a request causing an integer overflow when computing
the total size to allocate in RRChangeProviderProperty().

To avoid the issue, check that total length in bytes won't exceed the
maximum integer value.

CVE-2025-49180

This issue was discovered by Nils Emmerich <nemmerich@ernw.de> and
reported by Julian Suleder via ERNW Vulnerability Disclosure.

Signed-off-by: Olivier Fourdan <ofourdan@redhat.com>
Reviewed-by: Peter Hutterer <peter.hutterer@who-t.net>
Part-of: <https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/2024>
(cherry picked from commit 3c3a4b767b)
This commit is contained in:
Olivier Fourdan 2025-05-20 15:18:19 +02:00 committed by Alan Coopersmith
parent d308edc0e5
commit 07203788d3

View file

@ -179,7 +179,8 @@ RRChangeProviderProperty(RRProviderPtr provider, Atom property, Atom type,
if (mode == PropModeReplace || len > 0) {
void *new_data = NULL, *old_data = NULL;
if (total_len > MAXINT / size_in_bytes)
return BadValue;
total_size = total_len * size_in_bytes;
new_value.data = (void *) malloc(total_size);
if (!new_value.data && total_size) {