For mesh/task shaders, the thread payload provides a local invocation
index, but it's always linear so it doesn't give the correct value when
quad derivatives are in use.
The lowering pass where all of this is done correctly for compute
shaders assumes load_local_invocation_index will be lowered in the
backend for mesh/task, calculates the values for the quads correctly but
then avoid replacing the original intrinsic and we remain with the wrong
results.
Add an intel specific intrinsic and always lower the generic one to that
(or whatever else was calculated) to avoid ambiguities and fix the value
for quad derivatives.
Fixes future CTS tests using mesh/task shaders under:
dEQP-VK.spirv_assembly.instruction.compute.compute_shader_derivatives.*
Fixes: d89bfb1ff7 ("intel/brw: Reorganize lowering of LocalID/Index to handle Mesh/Task")
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39276>
This avoids generating some useless math that would need to be cleaned
up later, without complicating things too much.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
This helps cut down URB messages on tessellation and mesh shaders
significantly. fossil-db results on Battlemage:
Instrs: 505172392 -> 505207187 (+0.01%); split: -0.00%, +0.01%
Send messages: 23678197 -> 23656126 (-0.09%); split: -0.09%, +0.00%
Cycle count: 63150470088 -> 63147482640 (-0.00%); split: -0.01%, +0.00%
Spill count: 576554 -> 576616 (+0.01%)
Fill count: 545304 -> 545413 (+0.02%)
Max live registers: 141099192 -> 141150675 (+0.04%); split: -0.00%, +0.04%
Max dispatch width: 39856192 -> 39856208 (+0.00%)
Totals from 4231 (0.27% of 1583648) affected shaders:
Instrs: 1620161 -> 1654956 (+2.15%); split: -0.25%, +2.40%
Send messages: 128652 -> 106581 (-17.16%); split: -17.18%, +0.03%
Cycle count: 24650700 -> 21663252 (-12.12%); split: -12.82%, +0.70%
Spill count: 378 -> 440 (+16.40%)
Fill count: 1308 -> 1417 (+8.33%)
Max live registers: 364676 -> 416159 (+14.12%); split: -0.24%, +14.36%
Max dispatch width: 67952 -> 67968 (+0.02%)
There are several reasons we didn't go with nir_opt_vectorize_io:
1. nir_opt_vectorize_io appears to work on the slot location level.
We want to be able to vectorize based on the URB offsets, especially
for cases like point size, layer, and viewport which have different
VARYING_SLOT_* values but live in the same vec4 in a URB entry.
2. We want vec8 stores, and nir_opt_vectorize_io only seems to vectorize
within a single 32-bit vec4. It does handle 8 components, but that's
only for packing 16-bit values into a 32-bit vec4.
Improves performance of Sascha Willems' tessellation demo by around 4%
on Meteorlake.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
Both the URB Global Offset and Per-Slot Offsets are specified to be
unsigned numbers. The URB Global Offset is only 11 bits, and so is
limited to be between [0, 2047]. While the per-slot offsets are
given as U32 values, it would appear that adding the two offsets
does not handle 32-bit overflow/unsigned wrap correctly.
This pops up in Piglit's TCS variable-indexing tests, which ends up
performing loads from offset (x - 16) and a base of 18, and at an offset
(x) with a base of 2. These should be equivalent, but when x <= 15, the
per-slot offset calculated in the shader is negative (0xfffffff[0-f])
and adding the base of 18 is not wrapping around correctly to [2, 17].
To work around this, avoid using the global offset when the per-slot
offset is present, and just add the two in the shader where unsigned
wrap works correctly.
Tigerlake and later don't seem to have this issue.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
We were checking for 0xf which is fine for vec4, but vec8 gets 0xff.
Either way, nothing is writemasked, so we can skip sending the mask.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
This makes it easier for NIR passes to distinguish between inputs and
outputs without having to reason about which URB handle source was
passed to the intrinsic. It probably also makes it a bit easier for
humans to read the NIR too.
v2: Don't add memory mode to store intrinsics. It's always output.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
Minor adjustments to formatting of the copyright line, but keep
dates and holders. "Authors" entries that could be
obtained via Git logs were also removed.
The license in brw_disasm.c and elk_disasm.c don't match directly
any SPDX pattern I could find, so kept as is.
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39503>
The previous Gfx12+ implementation using bit masking is failing for FP8
types, so replacing with explicit lookup tables.
For float types, the encoding now aligns with brw_data_type_float, ensuring
correct behavior for DPAS and other 3-source instructions.
Fixes: d1d4e3d530 ("brw: Add EU assembler support for float8")
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39448>
Code kept track of blocks both in a linked list and
in an array. Change the client code of the list to
just use the array so we just maintain one.
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39246>
The code currently don't remove blocks, when a block is about to become
empty, the code will replace the last instruction with a NOP.
If we want to have actual block removals again, there are other
strategies than removing them as we iterate (e.g. allow empty blocks
and then collect them in a pass or right after iteration).
So remove those macros.
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39246>
Change the few other cases to an inline function that
does the same job. This macro will change in ways that
are not compatible with the non-assembler usages.
Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39363>
This optimization doesn't work when the ray query index isn't uniform across
the subgroup, which is something the spec allows. While there are some smart
ways to fix this and still avoid unnecessary spilling, its not worth investing
the time until we find a realtime raytracing workload that actually needs to
use multiple live ray queries for something.
Fixes: 1f1de7eb ("anv,brw: Allow multiple ray queries without spilling to a shadow stack")
Acked-by: Sagar Ghuge <sagar.ghuge@intel.com>
Acked-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39445>
This commit change the BVH layout a little so that we can load the BVH
offset as constant rather than reading from memory.
We have to force the instance leaves pointer at the end which gets used
in copy.comp shader.
Totals:
Instrs: 54798 -> 54728 (-0.13%)
Send messages: 3854 -> 3847 (-0.18%)
Cycle count: 1915106 -> 1913954 (-0.06%); split: -0.07%, +0.01%
Non SSA regs after NIR: 18594 -> 18575 (-0.10%)
Totals from 7 (7.37% of 95) affected shaders:
Instrs: 5532 -> 5462 (-1.27%)
Send messages: 367 -> 360 (-1.91%)
Cycle count: 132592 -> 131440 (-0.87%); split: -1.01%, +0.14%
Non SSA regs after NIR: 1989 -> 1970 (-0.96%)
PERCENTAGE DELTAS Shaders Instrs Send messages Cycle count Non SSA regs after NIR
q2rtx-rt-pipeline 95 -0.13% -0.18% -0.06% -0.10%
--------------------------------------------------------------------------------------
All affected 7 -1.27% -1.91% -0.87% -0.96%
--------------------------------------------------------------------------------------
Total 95 -0.13% -0.18% -0.06% -0.10%
Signed-off-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39106>
In software scoreboard (Gfx12+) use information from previous
instructions to trim out-of-order dependencies. For example, in
send g1, g2 ($1)
mov g3, g1 ($1.dst) // Depends on g1 (destination of $1)
mov g4, g2 ($1.src) // Depends on g2 (source of $1)
mov g5, g1 ($1.dst) // Depends on g1 (destination of $1)
only the first `mov` needs to be annotated, because the execution will
stall until that dependency is fulfilled, which in this case means the
`send` is done and `g1` was already written.
Note that while `$x.dst` implies `$x.src`, the reverse is not true, so
if the first `mov` did not exist, both second and third `mov` in the
example would have to keep their annotations.
This patch add resolution of implicit out-of-order dependencies that are
visible inside a block.
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3526>
There's agreement now these are helpful and widely supported. We can
always fallback to a custom vector class later if necessary.
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3526>
So that we can put the coarse_pixel_dispatch value available to NIR
lowering.
LNL internal fossildb changes:
Totals from 40 (0.01% of 490838) affected shaders:
Instrs: 33321 -> 33311 (-0.03%); split: -0.04%, +0.01%
Cycle count: 780136 -> 779936 (-0.03%); split: -0.03%, +0.00%
Max live registers: 5292 -> 5298 (+0.11%)
Non SSA regs after NIR: 26638 -> 26464 (-0.65%)
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38996>
Makes a bunch of copy propagation and other passes work much better.
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39382>
Not only is it questionable for code quality to not call nir_opt_algebraic_late
after nir_opt_algebraic, it also breaks correctness for late lowerings.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39180>
Allows a shader to have multiple ray queries without spilling them to a shadow
stack. Instead, the driver provides the shader with an array of multiple
RTDispatchGlobals structs to give each query its own dedicated stack.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38778>
