Follow the blob and optimize subgroup operation using brcst.active and
getlast when supported.
The transformation consists of two parts. First, a NIR transform
replaces subgroup operations with a sequence of new brcst_active_ir3
intrinsics followed by a new [type]_clusters_ir3 intrinsic (where type
can be reduce, inclusive_scan, or exclusive_scan).
The brcst_active_ir3 intrinsic is lowered directly to a brcst.active
instruction. The other intrinsics get lowered to a new macro
(OPC_SCAN_CLUSTERS_MACRO) which later gets emitted as a loop (using
getlast/getone) that iterates all clusters and produces the requested
scan result.
OPC_SCAN_CLUSTERS_MACRO has a number of optional arguments. First, since
the exclusive scan result is not a natural by-product of the loop but
has to be calculated explicitly, its destination is optional. This is
necessary since adding it unconditionally will produce unused
instructions that won't be DCE'd anymore at this point. Second, when
performing 32b MUL_U reductions (that expand to multiple instructions),
an extra scratch register is necessary.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6387
Signed-off-by: Job Noorman <jnoorman@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26950>
../src/compiler/nir/nir_builder.h: In function ‘nir_build_deref_follower’:
../src/compiler/nir/nir_builder.h:1607:1: error: control reaches end of non-void function [-Werror=return-type]
1607 | }
Fixes: 4a4e175738
nir: Support deref instructions in lower_var_copies
Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27345>
../src/compiler/nir/nir_lower_int64.c: In function ‘lower_int64_intrinsic’:
../src/compiler/nir/nir_lower_int64.c:1347:1: error: control reaches end of non-void function [-Werror=return-type]
1347 | }
Fixes: bf7a114246
nir/lower_int64: Add lowering for some 64-bit subgroup ops
Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27345>
Otherwise, opt_if_simplification() can attempt to insert an inot after a
jump.
Fixes RADV compilation of a Cyberpunk 2077 pipeline with
PIPELINE_CREATE_DISABLE_OPTIMIZATION_BIT.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27193>
Fixes RADV compilation of a Doom Eternal pipeline with
PIPELINE_CREATE_DISABLE_OPTIMIZATION_BIT, because
nir_opt_non_uniform_access was skipped and later passes don't expect
non-uniform access.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: b1619109ca ("nir/lower_non_uniform: remove non_uniform flags after lowering")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27192>
No driver supports urol/uror on all bit sizes. Intel gen11+ only for 16
and 32 bit, Nvidia GV100+ only for 32 bit. Etnaviv can support it on 8,
16 and 32 bit.
Also turn the `lower` into a `has` option as only two drivers actually
support `uror` and `urol` at this momemt.
Fixes crashes with CL integer_rotate on iris and nouveau since we emit
urol for `rotate`.
v2: always lower 64 bit
Fixes: fe0965afa6 ("spirv: Don't use libclc for rotate")
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by (Intel and nir): Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: David Heidelberg <david.heidelberg@collabora.com>
Acked-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27090>
freeing the nir shader should free the xfb info too. found with valgrind
leakcheck.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Reviewed-by: Antonino Maniscalco <antonino.maniscalco@collabora.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27093>
It is broken and clang uses fp32 for float constants if the fp64 extension
isn't enabled anyway. SPIRVs can't use fp64 constants with printf unless
they enable the Float64 cap, which also requires cl_khr_fp64 to be
supported.
So just remove it and rely on clang handling -cl-single-precision-constant
correctly, which at the moment doesn't seem to be the case, but we can
think about that once we plan to support cl_khr_fp64.
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26541>
When we use source from ALU instruction directly, the default swizzle array
should be populated with the same amount of components as the src has.
Otherwise, if we use nir_ssa_alu_instr_src_components, it can return
the destination components count that is lower than component index
actually used in that source. This can lead to false equality
between 0 (uninitialized) and 0 (.x) in swizzle comparison below.
Fixes: c6ee46a7 ("nir: Add nir_alu_srcs_negative_equal")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8704
Signed-off-by: Sviatoslav Peleshko <sviatoslav.peleshko@globallogic.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22655>
This pass was a mess. Rewrite it as modern NIR, fixing the metadata issues in
the process.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26976>
We were wrongly telling `nir_const_value_as_uint()` that `iter` had
`bit_size` bits, but in one case it is explicitly i64. This works on
little endian platforms, but caused the nir_loop_unroll_test.fadd{,_rev}
tests to fail on big endian platforms.
Bug: https://bugs.gentoo.org/921297
Fixes: 268ad47c11 ("nir/loop_analyze: Handle bit sizes correctly in calculate_iterations")
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26964>
Robust against separable shaders, and still makes sense for lowered I/O drivers,
whereas just counting FS variables and expecting them to match with the VS is...
questionable.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Signed-off-by: antonino <antonino.maniscalco@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26888>
we can replace varyings with point sprites, we just need to fix up .zw
appropriately. do that with some bcsels, ALU is cheap.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26963>
When a file is too large, -Wmisleading-indentantion will give the warning
below, that we can't prevent from a #pragma:
```
src/compiler/nir/nir_opt_algebraic.c: In function ‘nir_opt_algebraic’:
src/compiler/nir/nir_opt_algebraic.c:1469069: note: ‘-Wmisleading-indentation’ is disabled from this point onwards, since column-tracking was disabled due to the size of the code/headers
1469069 | nir_foreach_function_impl(impl, shader) {
|
src/compiler/nir/nir_opt_algebraic.c:1469069: note: adding ‘-flarge-source-files’ will allow for more column-tracking support, at the expense of compilation time and memory
```
See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89549 for details.
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25315>
nir_intrinsic_ballot_* are no subgroup operations.
nir_intrinsic_rotate was missing.
nir_intrinsic_mbcnt_amd is not a subgroup operation.
nir_intrinsic_writelane_amd only affects a single invocation.
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18249>
These are now covered by nir_opt_loop():
- opt_if_loop_last_continue()
- opt_merge_breaks()
- opt_if_loop_terminator()
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24940>
This new pass aims to simplify loop control-flow by reducing the number
of break and continue statements. It also supersedes nir_opt_trivial_continues().
For this purpose, it implements 3 optimizations:
- opt_loop_terminator(), as previously
- opt_loop_merge_break_continue(), similar to opt_merge_breaks() incl. continues
- opt_loop_last_block(), a generalization of opt_if_loop_last_continue()
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24940>
