rather than try to enumerate everything a driver might want with an unmanageable
collection of booleans, just do a filter callback + data. this ends up simpler
overall, and will allow Intel to use this pass for just 64-bit images without
needing to add even more booleans.
while we're churning the pass signature, also do a quick port to
nir_shader_intrinsics_pass
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com> [NIR and V3D]
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32907>
Declaring a variable-length array (VLA) based on a variable that can be
0 is declared dangerous.
In this case, the variable can't take value 0, so adding an assertion
fixes the issue.
This was detected by static analyzer.
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32819>
A negative hole size means the loads overlap. This will be used by drivers
to handle overlapping loads in the callback easily.
Reviewed-by: Mel Henning <drawoc@darkrefraction.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32699>
The case for converting a 32-bit integer to 16-bit float is not
correctly implemented.
Fixes: 214121e9b0 ("broadcom/compiler: handle fp16 conversion ops")
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32593>
As the lowering is applied on a load uniform intrinsic, there must be an
offset source number.
This fixes CID#1604734 ("Negative array index read") detected by
Coverity Scan.
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32593>
This is a prerequisite for enabling nir_opt_varyings for all gallium
drivers.
nir_lower_io_passes (called by the GLSL linker) only uses NIR options
to lower indirect IO access before lowering IO and calling
nir_opt_varyings.
Most drivers report full support for indirect IO and lower it themselves,
which prevents compaction of lowered indirectly accessed varyings because
nir_opt_varyings doesn't touch indirect varyings.
Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io> (Rb for asahi)
Reviewed-by: Pavel Ondračka <pavel.ondracka@gmail.com> (for r300)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32423>
This will need to be set to true when the GLSL linker lowers IO, which
can later be unlowered by st/mesa, and then drivers can lower it again
without load_interpolated_input. Therefore, it can't be a global
immutable option.
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32229>
The Early-Z optimization is disabled when there is a discard
instruction in the shader used in the draw call.
But if discard is the only reason to disable Early-Z, and at
draw call time the updates in the draw call are disabled we
can enable Early-Z using a shader variant.
If there are occlussion queries active we also need to disable
Early-z optimization.
So this patch enables Early-Z in this scenario.
The performance improvement is significant when running gfxbench
benchmark showing an average improvement of 11.15%
fps_avg helped: gl_gfxbench_aztec_high.trace: 3.13 -> 3.73 (19.13%)
fps_avg helped: gl_gfxbench_aztec.trace: 4.82 -> 5.68 (17.88%)
fps_avg helped: gl_gfxbench_manhattan31.trace: 5.10 -> 6.00 (17.59%)
fps_avg helped: gl_gfxbench_manhattan.trace: 7.24 -> 8.36 (15.52%)
fps_avg helped: gl_gfxbench_trex.trace: 19.25 -> 20.17 ( 4.81%)
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32028>
If you allow an unsupported component count in the callback for loads,
nir_opt_load_store_vectorize will align num_components to the next supported
vector size, essentially overfetching.
This changes all callbacks to reject it. AMD will enable it in a later commit.
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29398>
It will be used to allow merging loads with a hole between them.
Reviewed-by: Qiang Yu <yuq825@gmail.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29398>
Sprinkle around a few traces that were useful in locating submit and
fence waits.
Signed-off-by: Christian Gmeiner <cgmeiner@igalia.com>
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31575>
There are some restrictions when pairing a new TMU lookup with
a previous LDTMU and we had code to handle this but we were not
limiting the restriction only to TMU lookups.
total instructions in shared programs: 10856992 -> 10823967 (-0.30%)
instructions in affected programs: 1823670 -> 1790645 (-1.81%)
helped: 10212
HURT: 110
Instructions are helped.
total max-temps in shared programs: 2234069 -> 2233153 (-0.04%)
max-temps in affected programs: 15100 -> 14184 (-6.07%)
helped: 660
HURT: 3
Max-temps are helped.
total sfu-stalls in shared programs: 15935 -> 15967 (0.20%)
sfu-stalls in affected programs: 317 -> 349 (10.09%)
helped: 31
HURT: 57
Inconclusive result (%-change mean confidence interval includes 0).
total inst-and-stalls in shared programs: 10872927 -> 10839934 (-0.30%)
inst-and-stalls in affected programs: 1824656 -> 1791663 (-1.81%)
helped: 10199
HURT: 111
Inst-and-stalls are helped.
total nops in shared programs: 185612 -> 185767 (0.08%)
nops in affected programs: 4865 -> 5020 (3.19%)
helped: 164
HURT: 256
Nops are HURT.
Reviewed-by: Jose Maria Casanova Crespo <jmcasanova@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31574>
Models C0 and D0 support these opcodes too.
total instructions in shared programs: 10869461 -> 10856992 (-0.11%)
instructions in affected programs: 1467666 -> 1455197 (-0.85%)
helped: 6012
HURT: 1413
Instructions are helped.
total threads in shared programs: 431014 -> 431010 (<.01%)
threads in affected programs: 8 -> 4 (-50.00%)
helped: 0
HURT: 2
total uniforms in shared programs: 5432771 -> 5430909 (-0.03%)
uniforms in affected programs: 183047 -> 181185 (-1.02%)
helped: 976
HURT: 128
Uniforms are helped.
total max-temps in shared programs: 2235272 -> 2234069 (-0.05%)
max-temps in affected programs: 38163 -> 36960 (-3.15%)
helped: 1262
HURT: 168
Max-temps are helped.
total spills in shared programs: 4331 -> 4363 (0.74%)
spills in affected programs: 964 -> 996 (3.32%)
helped: 6
HURT: 47
total fills in shared programs: 6527 -> 6622 (1.46%)
fills in affected programs: 2047 -> 2142 (4.64%)
helped: 6
HURT: 47
total sfu-stalls in shared programs: 15807 -> 15935 (0.81%)
sfu-stalls in affected programs: 787 -> 915 (16.26%)
helped: 71
HURT: 172
Sfu-stalls are HURT.
total inst-and-stalls in shared programs: 10885268 -> 10872927 (-0.11%)
inst-and-stalls in affected programs: 1469423 -> 1457082 (-0.84%)
helped: 5998
HURT: 1417
Inst-and-stalls are helped.
total nops in shared programs: 184280 -> 185612 (0.72%)
nops in affected programs: 10000 -> 11332 (13.32%)
helped: 311
HURT: 1193
Nops are HURT.
The results show a reduction in register pressure, but an increase in
spills, which looks contradictory. This is because for some reason, this
optimization makes the NIR scheduler produce code for some shaders in Godot
that cause additional spilling, but the problem seems to be exclusive to
Godot shaders and not really related to the optimization itself but to
how the NIR scheduler works. Excluding Godot shaders we actually see a
decrease in spills and a slightly larger improvement in instruction
counts:
total instructions in shared programs: 10720106 -> 10707621 (-0.12%)
instructions in affected programs: 1375316 -> 1362831 (-0.91%)
helped: 5948
HURT: 1364
Instructions are helped.
total threads in shared programs: 428248 -> 428244 (<.01%)
threads in affected programs: 8 -> 4 (-50.00%)
helped: 0
HURT: 2
total spills in shared programs: 3729 -> 3712 (-0.46%)
spills in affected programs: 451 -> 434 (-3.77%)
helped: 6
HURT: 0
total fills in shared programs: 4738 -> 4714 (-0.51%)
fills in affected programs: 564 -> 540 (-4.26%)
helped: 6
HURT: 0
Comparing only shaders from Godot:
total instructions in shared programs: 149355 -> 149371 (0.01%)
instructions in affected programs: 92350 -> 92366 (0.02%)
helped: 64
HURT: 49
Inconclusive result (value mean confidence interval includes 0).
total max-temps in shared programs: 16477 -> 16472 (-0.03%)
max-temps in affected programs: 180 -> 175 (-2.78%)
helped: 5
HURT: 0
Max-temps are helped.
total spills in shared programs: 602 -> 651 (8.14%)
spills in affected programs: 513 -> 562 (9.55%)
helped: 0
HURT: 47
total fills in shared programs: 1789 -> 1908 (6.65%)
fills in affected programs: 1483 -> 1602 (8.02%)
helped: 0
HURT: 47
Reviewed-by: Jose Maria Casanova Crespo <jmcasanova@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31480>
Shader-db shows this is beneficial, even if it comes with a small
increase in register pressure.
total instructions in shared programs: 10889197 -> 10869857 (-0.18%)
instructions in affected programs: 3625014 -> 3605674 (-0.53%)
helped: 14911
HURT: 8324
Instructions are helped.
total threads in shared programs: 431034 -> 431014 (<.01%)
threads in affected programs: 40 -> 20 (-50.00%)
helped: 0
HURT: 10
Threads are HURT.
total uniforms in shared programs: 5308006 -> 5432767 (2.35%)
uniforms in affected programs: 2204951 -> 2329712 (5.66%)
helped: 9
HURT: 30766
Uniforms are HURT.
total max-temps in shared programs: 2226471 -> 2235269 (0.40%)
max-temps in affected programs: 272670 -> 281468 (3.23%)
helped: 2372
HURT: 8479
Max-temps are HURT.
total spills in shared programs: 4318 -> 4331 (0.30%)
spills in affected programs: 39 -> 52 (33.33%)
helped: 2
HURT: 7
total fills in shared programs: 6514 -> 6527 (0.20%)
fills in affected programs: 42 -> 55 (30.95%)
helped: 2
HURT: 7
total sfu-stalls in shared programs: 15166 -> 15808 (4.23%)
sfu-stalls in affected programs: 2389 -> 3031 (26.87%)
helped: 513
HURT: 944
Inconclusive result (%-change mean confidence interval includes 0).
total inst-and-stalls in shared programs: 10904363 -> 10885665 (-0.17%)
inst-and-stalls in affected programs: 3660930 -> 3642232 (-0.51%)
helped: 14878
HURT: 8450
Inst-and-stalls are helped.
total nops in shared programs: 183672 -> 184256 (0.32%)
nops in affected programs: 12532 -> 13116 (4.66%)
helped: 1841
HURT: 2251
Nops are HURT.
Reviewed-by: Jose Maria Casanova Crespo <jmcasanova@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31355>
ldvary instructions have implicit writes to rf0 (r5 in Pi4) that are
read in follow-up instructions to complete the interpolation calculations
so we rather not allocate ldunif(a)'s dst to rf0/r5 during these sequence
too to facilitate pairing.
This gives us -0.25% of instructions for fragment shaders in shader-db for
Pi5 and -0.64% on Pi4.
Shader-db Pi5:
total instructions in shared programs: 10890641 -> 10889197 (-0.01%)
instructions in affected programs: 575506 -> 574062 (-0.25%)
helped: 2506
HURT: 1378
Instructions are helped.
total max-temps in shared programs: 2226555 -> 2226471 (<.01%)
max-temps in affected programs: 5061 -> 4977 (-1.66%)
helped: 139
HURT: 78
Max-temps are helped.
total sfu-stalls in shared programs: 15143 -> 15166 (0.15%)
sfu-stalls in affected programs: 310 -> 333 (7.42%)
helped: 134
HURT: 195
Inconclusive result (value mean confidence interval includes 0).
total inst-and-stalls in shared programs: 10905784 -> 10904363 (-0.01%)
inst-and-stalls in affected programs: 577053 -> 575632 (-0.25%)
helped: 2497
HURT: 1415
Inst-and-stalls are helped.
total nops in shared programs: 183945 -> 183672 (-0.15%)
nops in affected programs: 3862 -> 3589 (-7.07%)
helped: 478
HURT: 234
Nops are helped.
Shader-db Pi4:
total instructions in shared programs: 12842116 -> 12835720 (-0.05%)
instructions in affected programs: 996970 -> 990574 (-0.64%)
helped: 6027
HURT: 367
Instructions are helped.
total max-temps in shared programs: 2251877 -> 2251707 (<.01%)
max-temps in affected programs: 2670 -> 2500 (-6.37%)
helped: 167
HURT: 9
Max-temps are helped.
total sfu-stalls in shared programs: 21132 -> 21093 (-0.18%)
sfu-stalls in affected programs: 114 -> 75 (-34.21%)
helped: 92
HURT: 55
Sfu-stalls are helped.
total inst-and-stalls in shared programs: 12863248 -> 12856813 (-0.05%)
inst-and-stalls in affected programs: 1008237 -> 1001802 (-0.64%)
helped: 6070
HURT: 359
Inst-and-stalls are helped.
total nops in shared programs: 281645 -> 281200 (-0.16%)
nops in affected programs: 2241 -> 1796 (-19.86%)
helped: 501
HURT: 88
Nops are helped.
Reviewed-by: Jose Maria Casanova Crespo <jmcasanova@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31355>
We flag nodes used to ldunif dst so we can try and favor allocating
rf0 to them, so be more explicit about its purpose.
Reviewed-by: Jose Maria Casanova Crespo <jmcasanova@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31355>
Adds a new instruction type that stores metadata that might be useful
for debugging purposes. Passes must ignore these instructions when
making decisions.
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18903>
This enables some algebraic optimizations.
No changes in shader-db, but it does cause some CTS tests to
produce less instructions.
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30614>
ir3's lowering of variables to scratch memory has to treat 8-bit values as
16-bit ones when comparing such value's size against the given threshold
since those values are handled through 16-bit half-registers. But those
values can still use natural 8-bit size and alignment for storing inside
scratch memory.
nir_lower_vars_to_scratch now accepts two size-and-alignment functions,
one used for calculating the variable size and the other for calculating
the size and alignment needed for storing inside scratch memory. Non-ir3
uses of this pass can just duplicate the currently-used function. ir3
provides a separate variable-size function that special-cases 8-bit types.
Signed-off-by: Zan Dobersek <zdobersek@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29875>
We usually emit flags right before consuming them but this is
suboptimal from the point of view of register pressure: if an
instruction is only used to generate flags then waiting to emit
it right before reading the flags extends the liveness of the
sources used to generate the flags for no gain. This pass will
check for such instructions and try to move them as early as
possible.
Shader-db results below show this is effective to reduce register
pressure, allowing a few shaders to increase thread counts and/or
reduce spilling:
total instructions in shared programs: 11057173 -> 11057076 (<.01%)
instructions in affected programs: 1955543 -> 1955446 (<.01%)
helped: 4214
HURT: 3905
Inconclusive result (value mean confidence interval includes 0).
total threads in shared programs: 425096 -> 425170 (0.02%)
threads in affected programs: 74 -> 148 (100.00%)
helped: 37
HURT: 0
Threads are helped.
total uniforms in shared programs: 3846275 -> 3845674 (-0.02%)
uniforms in affected programs: 23574 -> 22973 (-2.55%)
helped: 217
HURT: 30
Uniforms are helped.
total max-temps in shared programs: 2222910 -> 2220488 (-0.11%)
max-temps in affected programs: 61904 -> 59482 (-3.91%)
helped: 2145
HURT: 113
Max-temps are helped.
total spills in shared programs: 4294 -> 4280 (-0.33%)
spills in affected programs: 148 -> 134 (-9.46%)
helped: 8
HURT: 0
total fills in shared programs: 6497 -> 6468 (-0.45%)
fills in affected programs: 291 -> 262 (-9.97%)
helped: 8
HURT: 0
total sfu-stalls in shared programs: 14344 -> 14611 (1.86%)
sfu-stalls in affected programs: 1308 -> 1575 (20.41%)
helped: 217
HURT: 335
Inconclusive result (%-change mean confidence interval includes 0).
total inst-and-stalls in shared programs: 11071517 -> 11071687 (<.01%)
inst-and-stalls in affected programs: 1946767 -> 1946937 (<.01%)
helped: 4191
HURT: 3909
Inconclusive result (value mean confidence interval includes 0).
total nops in shared programs: 270628 -> 269829 (-0.30%)
nops in affected programs: 22032 -> 21233 (-3.63%)
helped: 1213
HURT: 571
Inconclusive result (%-change mean confidence interval includes 0).
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30511>
Instead of having a hardcoded list of endian-independent format aliases
in the header, generate them from the format definitions.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29649>
Since .sat, .nsat and .max0 are only supported with FMOV we can't copy
propagate an FMOV with any of these unpack modifiers into a different
opcode.
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30086>
We usually name the functions that depend on hardware version as
v3d<version>_foo.
Keep the same convention in QPU and lower_image_load_store, so it makes
easier when searching for versioned functions.
Acked-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Jose Maria Casanova Crespo <jmcasanova@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30000>
Ensure unsigned integers are used instead of signed ones when performing
left bit shifts.
This has been detected by the Undefined Behaviour Sanitizer (UBSan).
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29911>
We always emit multop+umul24 to implement integer multiply and
this is the only scenario in which we use multop, so if we decide
to DCE umul24 we should also DCE the previous multop.
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29909>
This is not safe when we have conditional spills since we could be
spilling disabled lanes with undefined values that could overwrite
valid data for those lanes from a previous spill of the same temp
that was unconditional (or that condionally enabled those same
lanes).
Fixes some Piglit OpenCL tests as well as the following OpenCL tests:
integer_divideAssign
integer_moduloAssign
integer_mad_sat
integer_ops integer_divideAssign
integer_ops integer_mad_sat
integer_ops integer_moduloAssign
integer_ops quick_char_math
integer_ops quick_short_math
math_brute_force half_powr
math_brute_force pow
math_brute_force pown
math_brute_force powr
math_brute_force rootn
Fixes: 597560e27c ('broadcom/compiler: always enable per-quad on spill operations')
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29909>
The multop instruction implicitly writes rtop which is not preserved
acrosss thread switches. We can spill the sources of the multop
(since these would happen before multop) and the destination of
umul24 (since that would happen after umul24).
Fixes some OpenCL tests when V3D_DEBUG=opt_compile_time is used to
choose a different compile configuration.
cc: mesa-stable
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29909>
