nir: Fixup 10/12 bit SW decoder YCbCr formats
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run

The highest possible values that can be represented with
16/12/10 bits are 65535/4095/1023, not 65536/4096/1024.
In order to ensure 1023 maps to 65535 in the Sx10 case
we thus need to multiply by 65535 / 1023 ~= 64.06158
instead of 64.

Fixes: a166d7609f ("gles: Add support for 10/12/16 bit SW decoder YCbCr formats")
Suggested-by: Benjamin Otte <otte@redhat.com>
Signed-off-by: Robert Mader <robert.mader@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Eric R. Smith <eric.smith@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37077>
This commit is contained in:
Robert Mader 2025-07-24 15:52:32 +02:00 committed by Marge Bot
parent 4a42ea6785
commit 1772380307

View file

@ -443,13 +443,13 @@ convert_yuv_to_rgb(nir_builder *b, nir_tex_instr *tex,
nir_def *m2 = nir_f2fN(b, nir_build_imm(b, 4, 32, m->v[2]), bit_size);
if (options->lower_sx10_external & (1u << texture_index)) {
m0 = nir_fmul_imm(b, m0, 64.0f);
m1 = nir_fmul_imm(b, m1, 64.0f);
m2 = nir_fmul_imm(b, m2, 64.0f);
m0 = nir_fmul_imm(b, m0, 65535.0f / 1023.0f);
m1 = nir_fmul_imm(b, m1, 65535.0f / 1023.0f);
m2 = nir_fmul_imm(b, m2, 65535.0f / 1023.0f);
} else if (options->lower_sx12_external & (1u << texture_index)) {
m0 = nir_fmul_imm(b, m0, 16.0f);
m1 = nir_fmul_imm(b, m1, 16.0f);
m2 = nir_fmul_imm(b, m2, 16.0f);
m0 = nir_fmul_imm(b, m0, 65535.0f / 4095.0f);
m1 = nir_fmul_imm(b, m1, 65535.0f / 4095.0f);
m2 = nir_fmul_imm(b, m2, 65535.0f / 4095.0f);
}
nir_def *result =