Commit graph

1032 commits

Author SHA1 Message Date
Konstantin Seurer
2a5d8d4cf4 aco: Unify demote and demote_if selection
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24906>
2023-09-01 07:23:33 +00:00
Konstantin Seurer
9af91edda9 aco: Use bytes() instead of size() in emit_wqm
This should get most of the cases that would fail validation.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24906>
2023-09-01 07:23:33 +00:00
Samuel Pitoiset
e104718c9f aco: allow separate compilation of NGG shaders
Also prevent to emit a long-jump for VS as NGG without a GS.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24907>
2023-09-01 06:52:40 +00:00
Samuel Pitoiset
bfb39031f1 aco: flag blocks with long-jump as export_end for separate compilation
This will allow us to adjust fix_exports().

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24907>
2023-09-01 06:52:40 +00:00
Qiang Yu
a2ba50aee6 aco: add vs prolog instruction selection for radeonsi
Port from llvm si_llvm_build_vs_prolog().

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24712>
2023-08-31 02:39:15 +00:00
Qiang Yu
3f87413811 aco: prepare fix_ls_vgpr_init_bug to be used by gl vs prolog
Prolog does not have nir shader.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24712>
2023-08-31 02:39:15 +00:00
Qiang Yu
eb4f871034 aco: pass sw_stage when setup_isel_context
We are going to add more shader parts.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24712>
2023-08-31 02:39:15 +00:00
Samuel Pitoiset
f30d47c518 aco: fix emitting TCS epilogs end on GFX9+
With merged shaders, the long-jump should be emitted inside the
divergent if (ie. only for TCS invocations) and other non TCS
invocations should just end the program.

This fixes a bunch of failures with CTS by forcing TCS epilogs on
RDNA2.

Not sure how RadeonSI will handle that but maybe doing the merged
wave info thing in epilogs would help.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24832>
2023-08-29 09:30:07 +00:00
Samuel Pitoiset
8ebb29245c aco: add support for compiling {VS,TES}+GS separately on GFX9+
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24862>
2023-08-25 10:22:41 +00:00
Samuel Pitoiset
80177e0296 aco: add support for compiling VS+TCS separately on GFX9+
The VS will just jump to the TCS.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24697>
2023-08-25 07:22:04 +00:00
Samuel Pitoiset
aba16211a8 aco: disable shared VGPRs for non-monolithic shaders on GFX9+
For unmerged shaders on GFX9+, we would need to jump to the second
shader part which means shared VGPRs can't be enabled.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24697>
2023-08-25 07:22:04 +00:00
Georg Lehmann
6d949e18fd aco: fix u2f16 with 32bit input
The vulkan spec says all conversions are correctly rounded, so if the input
is larger than the largest fp16 value, we need to return MAX_FLOAT/inf
instead of cutting off the msbs.

Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24826>
2023-08-23 12:25:56 +00:00
Rhys Perry
9169fbf83c aco: clarify bpermute pseudo opcode names
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24693>
2023-08-23 12:36:46 +01:00
Konstantin Seurer
bdec044c88 aco: Do not fixup registers if there are no shader calls
Frees up some registers when using monolithic compilation.

Quake II RTX and Control (with monolithic compilation):

Totals from 10 (29.41% of 34) affected shaders:
MaxWaves: 77 -> 98 (+27.27%)
Instrs: 49047 -> 48984 (-0.13%); split: -0.16%, +0.03%
CodeSize: 260420 -> 259880 (-0.21%); split: -0.25%, +0.04%
VGPRs: 1328 -> 1104 (-16.87%)
Latency: 477134 -> 479377 (+0.47%); split: -0.05%, +0.52%
InvThroughput: 137763 -> 114108 (-17.17%)
VClause: 1318 -> 1286 (-2.43%); split: -2.66%, +0.23%
SClause: 1295 -> 1293 (-0.15%); split: -0.54%, +0.39%
Copies: 7838 -> 7782 (-0.71%); split: -0.82%, +0.10%
Branches: 2592 -> 2589 (-0.12%)
PreSGPRs: 874 -> 796 (-8.92%)
PreVGPRs: 1283 -> 1013 (-21.04%)

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24809>
2023-08-22 15:46:29 +00:00
Samuel Pitoiset
a29e2c6fbc aco: implement create_tcs_jump_to_epilog()
This implements jumping from the main TCS to the epilog.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24643>
2023-08-22 06:10:32 +00:00
Samuel Pitoiset
fc9283938f aco: adjust TCS epilogs for RADV
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24643>
2023-08-22 06:10:32 +00:00
Samuel Pitoiset
0c2adc7ada aco: fix jumping from main TCS to epilog on GFX9+
On GFX9+, VS is merged with TCS which means this function is called
twice and the epilog was emitted in both shader parts.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24643>
2023-08-22 06:10:32 +00:00
Qiang Yu
51a8479a51 aco: use semantic location as io temp index
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
facbd13df1 aco: skip scratch init when no scratch arg provide
epilog does not use scratch so has no scratch arg.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
d3333609e6 aco: don't emit s_endpgm for tcs with epilog
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
b41d3e42f7 aco: add tcs epilog generation for radeonsi
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
5cf6f4f9a7 aco: allow tcs with epilog to keep nir store output instruction
Used to same tess factor outputs.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
5d05ae5df0 aco: add tcs end regs for epilog usage
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
7c7062f8f9 aco: move jump to epilog out of ic_merged_wave_info
TCS may be wrapped with if/else when merged shader. Jump
to epilog or end with regs should not be inside it.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Faith Ekstrand
65b6ac8aa4 nir: Rename nir_instr_type_ssa_undef to nir_instr_type_undef
We already renamed the type, we just need to rename the enum and the
casting helper functions.

Generated with sed:

    sed -i -e 's/nir_instr_type_ssa_undef/nir_instr_type_undef/g' src/**/*.h src/**/*.c src/**/*.cpp
    sed -i -e 's/nir_instr_as_ssa_undef/nir_instr_as_undef/g' src/**/*.h src/**/*.c src/**/*.cpp

and two tiny whitespace fixups in lima.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24703>
2023-08-15 17:44:27 +00:00
Konstantin Seurer
c291f612cb radv/rt: Rename shader_pc and next_shader
The names basically had the same meaning. Changing them to
uniform_shader_addr and shader_addr makes it clear, that
uniform_shader_addr is the jump target and shader_addr is the next
shader executed by this invocation.

The next_ prefix is dropped since both are input and output variables.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23565>
2023-08-15 11:11:16 +02:00
Faith Ekstrand
4695bebc79 nir: Drop nir_dest
Instead, we replace every use of it with nir_def.  Most of this commit
was generated by sed:

   sed -i -e 's/dest.ssa/def/g' src/**/*.h src/**/*.c src/**/*.cpp

A few manual fixups were required in lima and the nir_legacy code.

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24674>
2023-08-14 21:22:53 +00:00
Faith Ekstrand
6c1d32581a nir: Drop nir_alu_dest
Instead, we replace it directly with nir_def.  We could replace it with
nir_dest but the next commit gets rid of that so this avoids unnecessary
churn.  Most of this commit was generated by sed:

   sed -i -e 's/dest.dest.ssa/def/g' src/**/*.h src/**/*.c src/**/*.cpp

There were a few manual fixups required in the nir_legacy.c and
nir_from_ssa.c as nir_legacy_reg and nir_parallel_copy_entry both have a
similar pattern.

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24674>
2023-08-14 21:22:53 +00:00
Faith Ekstrand
977999d836 nir: Get rid of nir_dest_is_divergent()
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24674>
2023-08-14 21:22:53 +00:00
Alyssa Rosenzweig
09d31922de nir: Drop "SSA" from NIR language
Everything is SSA now.

   sed -e 's/nir_ssa_def/nir_def/g' \
       -e 's/nir_ssa_undef/nir_undef/g' \
       -e 's/nir_ssa_scalar/nir_scalar/g' \
       -e 's/nir_src_rewrite_ssa/nir_src_rewrite/g' \
       -e 's/nir_gather_ssa_types/nir_gather_types/g' \
       -i $(git grep -l nir | grep -v relnotes)

   git mv src/compiler/nir/nir_gather_ssa_types.c \
          src/compiler/nir/nir_gather_types.c

   ninja -C build/ clang-format
   cd src/compiler/nir && find *.c *.h -type f -exec clang-format -i \{} \;

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Acked-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24585>
2023-08-12 16:44:41 -04:00
Georg Lehmann
c4f356faf4 aco: always use rtne for fquantize2f16
The SPIR-V spec says:

If Value is positive with a magnitude too large to represent as a
16-bit floating-point value, the result is positive infinity.
If Value is negative with a magnitude too large to represent as a
16-bit floating-point value, the result is negative infinity.

This is only the case for rtne v_cvt_f16_f32

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24617>
2023-08-11 12:37:23 +00:00
Alyssa Rosenzweig
cf8ceb6e0a aco: Remove is_ssa check
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24432>
2023-08-03 22:40:29 +00:00
Samuel Pitoiset
f4ec2e7bb3 radv,aco: move has_epilog to radv_shader_info
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24404>
2023-08-02 16:59:18 +00:00
Samuel Pitoiset
f433d39935 aco: add infra for compiling TCS epilogs
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24417>
2023-08-02 07:29:50 +00:00
Qiang Yu
572625ea6c aco: extract aco_compile_shader_part from aco_compile_ps_epilog
Will be shared with radeonsi tcs epilog and other shader parts build.

Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24417>
2023-08-02 07:29:50 +00:00
Alyssa Rosenzweig
51db19f7a2 nir: Rename scoped_barrier -> barrier
sed + ninja clang-format + fix up spacing for common code.

If you are unhappy that I did not manually change the whitespace of your driver,
you need to enable clang-format for it so the formatting would happen
automatically.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24428>
2023-08-01 23:18:29 +00:00
Georg Lehmann
82920c99a5 aco: fix non constant 16bit bitnz/bitz
Fixes: 573e98f34a ("aco: implement nir_op_bitz/bitnz")
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24420>
2023-08-01 15:54:31 +00:00
Georg Lehmann
1e8d445bcf aco/gfx11: use v_cmp_class_f16 with opsel for bitnz/bitz
Foz-DB GFX11:
Totals from 1607 (1.21% of 132657) affected shaders:
MaxWaves: 36511 -> 36513 (+0.01%)
Instrs: 2412601 -> 2408770 (-0.16%); split: -0.16%, +0.00%
CodeSize: 12434716 -> 12414536 (-0.16%); split: -0.17%, +0.00%
VGPRs: 108872 -> 108860 (-0.01%)
Latency: 27555570 -> 27541897 (-0.05%); split: -0.05%, +0.00%
InvThroughput: 3144382 -> 3140929 (-0.11%); split: -0.11%, +0.00%
SClause: 65762 -> 65770 (+0.01%)
Copies: 127148 -> 127170 (+0.02%); split: -0.06%, +0.08%
PreVGPRs: 96737 -> 96447 (-0.30%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24380>
2023-08-01 07:52:42 +00:00
Timur Kristóf
be11fee2a7 aco: Refactor select_program to smaller functions.
This prepares for allowing to compile 1 shader at a time
for merged shader stages.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23874>
2023-07-28 14:27:29 +00:00
Qiang Yu
599b50b448 aco,radv: replace tess_input_vertices shader info param
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24204>
2023-07-24 01:49:20 +00:00
Bas Nieuwenhuizen
e536d31a46 aco: Fix some constant patterns in 16-bit vec4 construction with s_pack.
Fixes: 04e3d7ad93 ("aco: improve nir_op_vec with constant operands")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24286>
2023-07-22 22:11:15 +00:00
Bas Nieuwenhuizen
2fcf7c7014 aco: fix nir_op_vec8/16 with 16-bit elements.
Fixes: 5718347c2b ("aco: implement vec2/3/4 with subdword operands")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24286>
2023-07-22 22:11:15 +00:00
Timur Kristóf
9fb9e54d69 aco: Remove subgroup_id and num_subgroups intrinsics.
These are lowered in NIR now.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24005>
2023-07-14 12:17:24 +00:00
Timur Kristóf
dc3bbd351a aco: Fix subgroup_id intrinsic on GFX10.3+.
Change this to match how it works in the LLVM backend.

Cc: mesa-stable
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24005>
2023-07-14 12:17:24 +00:00
Rhys Perry
ae48fae658 aco: remove 64-bit integer conversion opcodes
These are handled by nir_lower_int64 now.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23926>
2023-07-03 10:38:28 +00:00
Rhys Perry
25c49e491f aco,ac/llvm,ac/nir,vtn: unify cube opcodes
fossil-db (navi21):
Totals from 17068 (12.79% of 133461) affected shaders:
Instrs: 24743703 -> 24743572 (-0.00%); split: -0.00%, +0.00%
CodeSize: 132579952 -> 132580620 (+0.00%); split: -0.00%, +0.00%
VGPRs: 1227840 -> 1227984 (+0.01%)
Latency: 403180114 -> 403251188 (+0.02%); split: -0.00%, +0.02%
InvThroughput: 75311302 -> 75320892 (+0.01%); split: -0.00%, +0.01%
VClause: 415400 -> 415402 (+0.00%); split: -0.00%, +0.00%
Copies: 1715404 -> 1715258 (-0.01%); split: -0.01%, +0.01%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Gert Wollny <gert.wollny@collabora.com> (r600)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23930>
2023-06-30 15:35:03 +00:00
Georg Lehmann
573e98f34a aco: implement nir_op_bitz/bitnz
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23298>
2023-06-29 13:39:30 +00:00
Rhys Perry
58f8e0e2a0 nir,aco: add INCLUDE_HELPERS index to reduce intrinsic
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23621>
2023-06-27 18:53:50 +00:00
Rhys Perry
c5fc89019b aco: include helpers in emit_uniform_{reduce,scan}
Found by inspection.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23621>
2023-06-27 18:53:49 +00:00
Vitaliy Triang3l Kuzmin
dda425df09 aco: Implement fragment shader interlock intrinsics
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Vitaliy Triang3l Kuzmin <triang3l@yandex.ru>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22250>
2023-06-26 15:58:04 +00:00