Commit graph

2660 commits

Author SHA1 Message Date
Samuel Pitoiset
a29e2c6fbc aco: implement create_tcs_jump_to_epilog()
This implements jumping from the main TCS to the epilog.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24643>
2023-08-22 06:10:32 +00:00
Samuel Pitoiset
e03c09dfb2 aco: allow SGPRs operands with p_jump_to_epilog
For TCS epilogs, we will have to pass SGPRs.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24643>
2023-08-22 06:10:32 +00:00
Samuel Pitoiset
fc9283938f aco: adjust TCS epilogs for RADV
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24643>
2023-08-22 06:10:32 +00:00
Samuel Pitoiset
0c2adc7ada aco: fix jumping from main TCS to epilog on GFX9+
On GFX9+, VS is merged with TCS which means this function is called
twice and the epilog was emitted in both shader parts.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24643>
2023-08-22 06:10:32 +00:00
Qiang Yu
51a8479a51 aco: use semantic location as io temp index
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
4756388038 aco,radeonsi: save const addr to symbol
For radeonsi to relocation const data when combine multiple
shader parts to a single one. So the final shader binary will
begin with exec code of all parts then const data.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
facbd13df1 aco: skip scratch init when no scratch arg provide
epilog does not use scratch so has no scratch arg.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
d3333609e6 aco: don't emit s_endpgm for tcs with epilog
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
b41d3e42f7 aco: add tcs epilog generation for radeonsi
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
a2484b20f9 aco: add pending_lds_access option for insert waitcnt
For tcs epilog to add p_barrier at the beginning to sync
main shader part tess factor LDS write.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
5cf6f4f9a7 aco: allow tcs with epilog to keep nir store output instruction
Used to same tess factor outputs.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
5d05ae5df0 aco: add tcs end regs for epilog usage
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
7c7062f8f9 aco: move jump to epilog out of ic_merged_wave_info
TCS may be wrapped with if/else when merged shader. Jump
to epilog or end with regs should not be inside it.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Qiang Yu
85d9646288 aco: add p_end_with_regs pseudo instruction
Used by radeonsi shader parts to pass args from one part to another.
It has variable number of operands to reserve fixed registers with
wanted value.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24442>
2023-08-16 02:27:45 +00:00
Faith Ekstrand
65b6ac8aa4 nir: Rename nir_instr_type_ssa_undef to nir_instr_type_undef
We already renamed the type, we just need to rename the enum and the
casting helper functions.

Generated with sed:

    sed -i -e 's/nir_instr_type_ssa_undef/nir_instr_type_undef/g' src/**/*.h src/**/*.c src/**/*.cpp
    sed -i -e 's/nir_instr_as_ssa_undef/nir_instr_as_undef/g' src/**/*.h src/**/*.c src/**/*.cpp

and two tiny whitespace fixups in lima.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24703>
2023-08-15 17:44:27 +00:00
Rhys Perry
c9b177db0e aco: don't create sendmsg(dealloc_vgprs) if scratch is used
LLVM does something similar: https://reviews.llvm.org/D153295

fossil-db (gfx1100):
Totals from 21 (0.02% of 133461) affected shaders:
Instrs: 147428 -> 147396 (-0.02%)
CodeSize: 797188 -> 797060 (-0.02%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 2930317cea ("aco/gfx11: deallocate VGPRs at the end of the shader")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24669>
2023-08-15 11:00:31 +00:00
Konstantin Seurer
c291f612cb radv/rt: Rename shader_pc and next_shader
The names basically had the same meaning. Changing them to
uniform_shader_addr and shader_addr makes it clear, that
uniform_shader_addr is the jump target and shader_addr is the next
shader executed by this invocation.

The next_ prefix is dropped since both are input and output variables.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23565>
2023-08-15 11:11:16 +02:00
Faith Ekstrand
4695bebc79 nir: Drop nir_dest
Instead, we replace every use of it with nir_def.  Most of this commit
was generated by sed:

   sed -i -e 's/dest.ssa/def/g' src/**/*.h src/**/*.c src/**/*.cpp

A few manual fixups were required in lima and the nir_legacy code.

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24674>
2023-08-14 21:22:53 +00:00
Faith Ekstrand
6c1d32581a nir: Drop nir_alu_dest
Instead, we replace it directly with nir_def.  We could replace it with
nir_dest but the next commit gets rid of that so this avoids unnecessary
churn.  Most of this commit was generated by sed:

   sed -i -e 's/dest.dest.ssa/def/g' src/**/*.h src/**/*.c src/**/*.cpp

There were a few manual fixups required in the nir_legacy.c and
nir_from_ssa.c as nir_legacy_reg and nir_parallel_copy_entry both have a
similar pattern.

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24674>
2023-08-14 21:22:53 +00:00
Faith Ekstrand
977999d836 nir: Get rid of nir_dest_is_divergent()
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24674>
2023-08-14 21:22:53 +00:00
Rhys Perry
efb80c1715 aco: summarize register demand after handling branches
Fixes various dEQP-VK.ray_query.builtin.rayqueryterminate.* crashes.

fossil-db (gfx1100):
Totals from 196 (0.15% of 133461) affected shaders:
PreSGPRs: 8342 -> 8558 (+2.59%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 5a536eca9c ("aco: calculate correct register demand for branch instructions")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24579>
2023-08-14 15:08:37 +00:00
Konstantin Seurer
e2a02f3910 clang-format: Disable formatting by default
This should make `git clang-format` usable for patches that modify
clang formatted and manually formatted code.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9492
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24645>
2023-08-13 16:48:49 +02:00
Alyssa Rosenzweig
09d31922de nir: Drop "SSA" from NIR language
Everything is SSA now.

   sed -e 's/nir_ssa_def/nir_def/g' \
       -e 's/nir_ssa_undef/nir_undef/g' \
       -e 's/nir_ssa_scalar/nir_scalar/g' \
       -e 's/nir_src_rewrite_ssa/nir_src_rewrite/g' \
       -e 's/nir_gather_ssa_types/nir_gather_types/g' \
       -i $(git grep -l nir | grep -v relnotes)

   git mv src/compiler/nir/nir_gather_ssa_types.c \
          src/compiler/nir/nir_gather_types.c

   ninja -C build/ clang-format
   cd src/compiler/nir && find *.c *.h -type f -exec clang-format -i \{} \;

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Acked-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24585>
2023-08-12 16:44:41 -04:00
Georg Lehmann
c4f356faf4 aco: always use rtne for fquantize2f16
The SPIR-V spec says:

If Value is positive with a magnitude too large to represent as a
16-bit floating-point value, the result is positive infinity.
If Value is negative with a magnitude too large to represent as a
16-bit floating-point value, the result is negative infinity.

This is only the case for rtne v_cvt_f16_f32

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24617>
2023-08-11 12:37:23 +00:00
Georg Lehmann
7a3e5dd2ec aco: use s_bitreplicate_b64_b32 to set exec to 0xffff0000ffff0000
Foz-DB Navi21:
Totals from 29 (0.02% of 132657) affected shaders:
Instrs: 19342 -> 19301 (-0.21%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24561>
2023-08-09 20:29:01 +00:00
Faith Ekstrand
04cc8c6f8d clang-format: Set ColumnLimit to 78
This matches the tree-wide .editorconfig.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24429>
2023-08-04 17:46:42 +00:00
Alyssa Rosenzweig
cf8ceb6e0a aco: Remove is_ssa check
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24432>
2023-08-03 22:40:29 +00:00
Yonggang Luo
cd3ea02da0 ac/radv: decouple radv vulkan driver and compiler from gallium
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24464>
2023-08-03 09:45:42 +00:00
Samuel Pitoiset
f4ec2e7bb3 radv,aco: move has_epilog to radv_shader_info
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24404>
2023-08-02 16:59:18 +00:00
Samuel Pitoiset
f433d39935 aco: add infra for compiling TCS epilogs
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24417>
2023-08-02 07:29:50 +00:00
Qiang Yu
572625ea6c aco: extract aco_compile_shader_part from aco_compile_ps_epilog
Will be shared with radeonsi tcs epilog and other shader parts build.

Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24417>
2023-08-02 07:29:50 +00:00
Samuel Pitoiset
ac99fbe591 aco: add aco_shader_info::tcs::has_epilog
This will be used by both RADV and RadeonSI to jump from the main TCS
to the epilog.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24417>
2023-08-02 07:29:50 +00:00
Alyssa Rosenzweig
51db19f7a2 nir: Rename scoped_barrier -> barrier
sed + ninja clang-format + fix up spacing for common code.

If you are unhappy that I did not manually change the whitespace of your driver,
you need to enable clang-format for it so the formatting would happen
automatically.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24428>
2023-08-01 23:18:29 +00:00
Mike Blumenkrantz
4d8c53b070 radv: bump max xfb output to 128
this is the number of components supported for streamout

Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24285>
2023-08-01 20:47:34 +00:00
Georg Lehmann
82920c99a5 aco: fix non constant 16bit bitnz/bitz
Fixes: 573e98f34a ("aco: implement nir_op_bitz/bitnz")
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24420>
2023-08-01 15:54:31 +00:00
Georg Lehmann
1e8d445bcf aco/gfx11: use v_cmp_class_f16 with opsel for bitnz/bitz
Foz-DB GFX11:
Totals from 1607 (1.21% of 132657) affected shaders:
MaxWaves: 36511 -> 36513 (+0.01%)
Instrs: 2412601 -> 2408770 (-0.16%); split: -0.16%, +0.00%
CodeSize: 12434716 -> 12414536 (-0.16%); split: -0.17%, +0.00%
VGPRs: 108872 -> 108860 (-0.01%)
Latency: 27555570 -> 27541897 (-0.05%); split: -0.05%, +0.00%
InvThroughput: 3144382 -> 3140929 (-0.11%); split: -0.11%, +0.00%
SClause: 65762 -> 65770 (+0.01%)
Copies: 127148 -> 127170 (+0.02%); split: -0.06%, +0.08%
PreVGPRs: 96737 -> 96447 (-0.30%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24380>
2023-08-01 07:52:42 +00:00
Georg Lehmann
1659d982c3 aco: combine a & ~b to bfi(b, 0, a)
Foz-DB Navi21:
Totals from 905 (0.68% of 132657) affected shaders:
Instrs: 1223583 -> 1221016 (-0.21%); split: -0.22%, +0.01%
CodeSize: 6567272 -> 6567064 (-0.00%); split: -0.04%, +0.03%
SpillSGPRs: 1231 -> 1223 (-0.65%)
SpillVGPRs: 829 -> 823 (-0.72%); split: -1.45%, +0.72%
Latency: 40952209 -> 40946230 (-0.01%); split: -0.02%, +0.01%
InvThroughput: 9411929 -> 9397932 (-0.15%); split: -0.17%, +0.02%
VClause: 29108 -> 29112 (+0.01%); split: -0.04%, +0.05%
Copies: 105272 -> 105221 (-0.05%); split: -0.28%, +0.23%
Branches: 29330 -> 29329 (-0.00%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24383>
2023-08-01 07:15:28 +00:00
Timur Kristóf
be11fee2a7 aco: Refactor select_program to smaller functions.
This prepares for allowing to compile 1 shader at a time
for merged shader stages.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23874>
2023-07-28 14:27:29 +00:00
Rhys Perry
37fbfa655a aco: insert s_nop before VGPR deallocation
A hazard requires this, apparently.

https://reviews.llvm.org/D155681

fossil-db (gfx1100):
Totals from 23175 (17.36% of 133461) affected shaders:
Instrs: 41240100 -> 41263275 (+0.06%)
CodeSize: 211858524 -> 211951224 (+0.04%)
Latency: 469738570 -> 469738576 (+0.00%)
InvThroughput: 58013998 -> 58013999 (+0.00%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24320>
2023-07-26 13:58:22 +00:00
Georg Lehmann
92900d8bf4 aco: improve get_gfx11_true16_mask description
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24294>
2023-07-24 14:12:19 +00:00
Georg Lehmann
8fbebb6a2a aco/gfx11: fix get_gfx11_true16_mask with v_cmp_class_f16
The second operand is 16bit, so the we need to use VOP3 to address v128-v255.

Closes: #9413
Fixes: 6872f8d861 ("aco/gfx11: allow true 16-bit instructions to access v128+")
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24294>
2023-07-24 14:12:19 +00:00
Qiang Yu
599b50b448 aco,radv: replace tess_input_vertices shader info param
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24204>
2023-07-24 01:49:20 +00:00
Bas Nieuwenhuizen
e536d31a46 aco: Fix some constant patterns in 16-bit vec4 construction with s_pack.
Fixes: 04e3d7ad93 ("aco: improve nir_op_vec with constant operands")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24286>
2023-07-22 22:11:15 +00:00
Bas Nieuwenhuizen
2fcf7c7014 aco: fix nir_op_vec8/16 with 16-bit elements.
Fixes: 5718347c2b ("aco: implement vec2/3/4 with subdword operands")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24286>
2023-07-22 22:11:15 +00:00
Timur Kristóf
9fb9e54d69 aco: Remove subgroup_id and num_subgroups intrinsics.
These are lowered in NIR now.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24005>
2023-07-14 12:17:24 +00:00
Timur Kristóf
dc3bbd351a aco: Fix subgroup_id intrinsic on GFX10.3+.
Change this to match how it works in the LLVM backend.

Cc: mesa-stable
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24005>
2023-07-14 12:17:24 +00:00
Timur Kristóf
127999989a aco: Add MESA_SHADER_KERNEL to instruction selection setup.
Treat it the same way as MESA_SHADER_COMPUTE.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24112>
2023-07-12 11:24:44 +00:00
Daniel Schürmann
7e4870e8e5 amd: Do shader binary alignment for prefetch at memory allocation time.
This makes it consistent between drivers and compilers.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23799>
2023-07-11 12:01:45 +00:00
Daniel Schürmann
437bf4fccb amd: move end-of-code marker padding to ACO.
This makes it consistent between drivers and compilers.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23799>
2023-07-11 12:01:45 +00:00
Daniel Schürmann
c778803d67 aco/assembler: change prefetch mode on GFX10.3+ during loops if beneficial
Totals from 8864 (6.68% of 132726) affected shaders: GFX11

CodeSize: 90776128 -> 90923760 (+0.16%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23748>
2023-07-11 07:15:43 +00:00