randr: check rotated virtual size limits correctly

Commit d1107918d4 introduced checks to
the RandR path that cause RRSetScreenConfig requests to fail if the
size is too large.  Unfortunately, when RandR 1.1 rotation is enabled
it compares the rotated screen dimensions to the unrotated limits,
which causes 90- and 270-degree rotation to fail unless your screen
happens to be square:

  X Error of failed request:  BadValue (integer parameter out of range for operation)
    Major opcode of failed request:  153 (RANDR)
    Minor opcode of failed request:  2 (RRSetScreenConfig)
    Value in failed request:  0x780
    Serial number of failed request:  14
    Current serial number in output stream:  14

Fix this by moving the check above the code that swaps the dimensions
based on the rotation.

Signed-off-by: Aaron Plattner <aplattner@nvidia.com>
Reviewed-by: Jeremy Huddleston <jeremyhu@apple.com>
Tested-by: Robert Hooker <robert.hooker@canonical.com>
Tested-by: Kent Baxley <kent.baxley@canonical.com>
Signed-off-by: Keith Packard <keithp@keithp.com>
(cherry picked from commit b6c7b9b2f3)
This commit is contained in:
Aaron Plattner 2011-05-24 16:02:42 -07:00 committed by Jeremy Huddleston
parent 613e0e9ef7
commit 50b9d3142f

View file

@ -910,12 +910,6 @@ ProcRRSetScreenConfig (ClientPtr client)
*/
width = mode->mode.width;
height = mode->mode.height;
if (rotation & (RR_Rotate_90|RR_Rotate_270))
{
width = mode->mode.height;
height = mode->mode.width;
}
if (width < pScrPriv->minWidth || pScrPriv->maxWidth < width) {
client->errorValue = width;
free(pData);
@ -927,6 +921,12 @@ ProcRRSetScreenConfig (ClientPtr client)
return BadValue;
}
if (rotation & (RR_Rotate_90|RR_Rotate_270))
{
width = mode->mode.height;
height = mode->mode.width;
}
if (width != pScreen->width || height != pScreen->height)
{
int c;