Commit graph

238 commits

Author SHA1 Message Date
Qiang Yu
ed419f46aa aco: remove early_rast wait insert
It's done in nir position export.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20691>
2023-02-03 12:27:44 +00:00
Rhys Perry
6872f8d861 aco/gfx11: allow true 16-bit instructions to access v128+
It looks like the LLVM assembler promotes true 16-bit instructions to VOP3
in this case.

No fossil-db changes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20251>
2023-01-10 16:01:38 +00:00
Rhys Perry
c9846158cd aco/gfx11: reduce scratch allocation alignment
fossil-db (gfx1100):
Totals from 112 (0.08% of 134574) affected shaders:
Scratch: 1513472 -> 1455360 (-3.84%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20534>
2023-01-09 21:46:13 +00:00
Rhys Perry
192486b7aa aco/gfx11: export mrtz in discard early exit for non-color shaders
If a shader doesn't export any color targets and instead only exports
mrtz, the discard early exit block should match.

Fixes artifacts on Lara in Rise of the Tomb Raider benchmark and hair in
The Witcher 3 (classic).

https://reviews.llvm.org/D128185

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Fixes: bc8da20dda ("aco: export MRT0 instead of NULL on GFX11")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20345>
2022-12-16 15:35:28 +00:00
Bas Nieuwenhuizen
cd3bf56ace aco: Add helper to get cycle info for an instruction.
For use in s_delay_alu tracking

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19743>
2022-12-07 22:05:25 +00:00
Bas Nieuwenhuizen
352e492c7b aco: Add isTrans helper.
For the s_delay_alu tracking.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19743>
2022-12-07 22:05:25 +00:00
Rhys Perry
3a96977542 radv,aco: remove old GS copy shader code
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18898>
2022-11-29 14:28:11 +00:00
Samuel Pitoiset
bb90d29660 aco: add p_dual_src_export_gfx11 for dual source blending on GFX11
Dual source blending must be in strict WQM mode.

Cc: 22.3 mesa-stable
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19643>
2022-11-16 18:35:10 +00:00
Daniel Schürmann
efc0835787 aco: move statistics enum to aco_shader_info.h
to make it accessible from the driver.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19721>
2022-11-15 16:34:07 +00:00
Rhys Perry
50073d6135 aco/gfx11: increase gfx1100/gfx1101 physical vgprs
https://reviews.llvm.org/D134522

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18825>
2022-11-02 17:09:32 +00:00
Rhys Perry
ea8ddf5c26 aco: add storage_gds
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19345>
2022-10-28 21:50:05 +00:00
Daniel Schürmann
a36e27e507 aco: change thread_local memory resource to pointer
Apparently the TLS constructor doesn't work well if RADV
is instantiated multiple times and/or used by a program with
already existing threads.

Fixes: a128d444cb ('aco: use monotonic_buffer_resource for instructions')
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19219>
2022-10-25 09:08:08 +00:00
Timur Kristóf
a17e801a9c aco: Add ACO_DEBUG=novalidateir option.
This disables IR validation in debug/debugoptimized builds.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18103>
2022-10-24 20:14:16 +00:00
Timur Kristóf
0cceab788e aco: Move is_dead to aco_ir.h to allow it to get inlined.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18103>
2022-10-24 20:14:16 +00:00
Rhys Perry
36703a60bf aco: add ACO_DEBUG=force-waitdeps
GFX11 has a lot of complicated data dependency hazards.

For debugging GFX10+ data dependency hazards. This creates an excessive
amount of s_waitcnt_depctr.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18273>
2022-10-19 02:46:03 +00:00
Timur Kristóf
d8639b7a80 aco: Allow explicitly removing jumps on GFX10+ when beneficial.
"Removing jumps" in ACO means skipping the jump instruction
at the beginning of a divergent branch (but still modify exec).

ACO already supports implicitly removing jumps when it decides
that executing a branch with empty exec mask is more beneficial
than a jump.

This commit adds the possibility to use this explicitly
through nir_selection_control. ACO will respect this
setting and remove the branch instructions when this is specified,
unless it decides that this would cause bugs (eg. exp instruction).

There are two cases that benefit from the new change:

1. When the application requests to "flatten" a branch (ie.
remove control flow), we now respect that.
2. When the compiler stack determines that a divergent branch
is always taken.

v2 by Georg Lehmann: fixed applying sel_ctrl to else blocks

Fossil DB stats on Navi 21:

Totals from 13 (0.01% of 134906) affected shaders:
CodeSize: 136616 -> 136496 (-0.09%)
Instrs: 26196 -> 26166 (-0.11%)
Latency: 417928 -> 417889 (-0.01%)
Branches: 1241 -> 1211 (-2.42%)

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17921>
2022-10-11 15:42:54 +00:00
Rhys Perry
2930317cea aco/gfx11: deallocate VGPRs at the end of the shader
fossil-db (gfx1100):
Totals from 65987 (40.81% of 161689) affected shaders:
Instrs: 57123207 -> 57199947 (+0.13%)
CodeSize: 308402500 -> 308709460 (+0.10%)
Latency: 680527139 -> 680527160 (+0.00%)
InvThroughput: 131620026 -> 131620045 (+0.00%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17710>
2022-09-30 20:57:02 +00:00
Rhys Perry
7cecc81683 aco/gfx11: fix s_waitcnt printing
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17710>
2022-09-30 20:57:02 +00:00
Daniel Schürmann
a128d444cb aco: use monotonic_buffer_resource for instructions
As monotonic_buffer_resource is not thread-safe,
we use a thread_local instance which gets allocated once.

This change reduces the compile time spent in ACO by
approximately 10%.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18112>
2022-09-28 09:25:20 +00:00
Daniel Schürmann
0b76e22a96 aco: simplify operands_offset calculation in create_instruction()
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18112>
2022-09-28 09:25:20 +00:00
Rhys Perry
aadb7aef01 aco: add VINTERP instruction format
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17333>
2022-09-26 14:49:56 +00:00
Rhys Perry
55cd74d468 aco: add LDSDIR instruction format
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17333>
2022-09-26 14:49:56 +00:00
Rhys Perry
4e55b5b851 aco: update assembler for GFX11
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17333>
2022-09-26 14:49:56 +00:00
Rhys Perry
d8d99c3c4f aco: add GFX11 opcode numbers
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17333>
2022-09-26 14:49:56 +00:00
Rhys Perry
7a1b522148 aco: rename Interp_instruction to VINTRP_instruction
These is clearer since GFX11 adds another interpolation format.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17333>
2022-09-26 14:49:56 +00:00
Rhys Perry
fb13ed6ff0 aco: fix long-jump version of discard early exit
It isn't safe to modify the exec mask before the discard block, and the
definition interferes with GFX11 NOP insertion.

Just use s[0:1] instead, since we won't be using it.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18125>
2022-08-25 16:10:53 +00:00
Rhys Perry
dd105f7c1e aco: fix assembly of vopc_sdwa writing exec
We would assemble an instruction writing vcc instead.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: 5ffc73896f ("aco/assembler: Fix v_cmpx with SDWA.")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
2022-08-16 17:31:33 +00:00
Rhys Perry
f60cb8d0af aco: rename is_cmp to is_fp_cmp
The old name is no longer accurate.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
2022-08-16 17:31:33 +00:00
Georg Lehmann
578d0a1934 aco/ir: Add vcmpx opcode to get_cmp_info.
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17763>
2022-07-27 14:52:29 +00:00
Samuel Pitoiset
270cc39648 aco: add support for compiling PS epilogs
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17485>
2022-07-18 18:40:02 +00:00
Rhys Perry
d2d94b62f2 aco: initialize scratch base registers on GFX9-GFX10.3
fossil-db (navi21):
Totals from 1142 (0.70% of 162293) affected shaders:
Instrs: 271636 -> 271974 (+0.12%)
CodeSize: 1532020 -> 1533792 (+0.12%)
Latency: 7484066 -> 7485698 (+0.02%)
InvThroughput: 4048824 -> 4049579 (+0.02%)
SClause: 4171 -> 4212 (+0.98%)
PreSGPRs: 11203 -> 12276 (+9.58%)

fossil-db (vega10):
Totals from 3327 (2.06% of 161355) affected shaders:
Instrs: 257413 -> 257601 (+0.07%)
CodeSize: 1424244 -> 1425372 (+0.08%)
Latency: 8598402 -> 8600466 (+0.02%)
InvThroughput: 7906335 -> 7908234 (+0.02%)
SClause: 4932 -> 4973 (+0.83%)
PreSGPRs: 22010 -> 25405 (+15.42%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>
2022-07-08 14:49:03 +00:00
Rhys Perry
cbeb25ce91 aco: make FLAT_instruction::offset signed
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>
2022-07-08 14:49:03 +00:00
Georg Lehmann
efdb323ad2 aco/ir: Pad SOP2 and SOPC to the same size as SOPK.
Being able to directly cast instructions simplifies optimizations.

Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15999>
2022-07-06 09:54:54 +00:00
Rhys Perry
84b404d34d aco: don't use 32-bit fp inline constants for fp16 vop3p literals
If we're applying the literal 0x3f800000 to a fp16 vop3p instruction, we
shouldn't use the 1.0 inline constant, because the hardware will use the
16-bit 1.0: 0x00003c00.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16296>
2022-07-05 16:39:56 +00:00
Rhys Perry
f937c5be7c aco: add and use constantValue16()
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16296>
2022-07-05 16:39:56 +00:00
Dave Airlie
a179e1aede aco/radv: drop radv_nir_compiler_options from aco.
Add a new aco input and options structs, then convert from radv
pieces on submit.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16521>
2022-05-17 06:15:25 +00:00
Marek Olšák
39800f0fa3 amd: change chip_class naming to "enum amd_gfx_level gfx_level"
This aligns the naming with PAL.

Acked-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Acked-by: Pierre-Eric Pellou-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16469>
2022-05-13 14:56:22 -04:00
Dave Airlie
c44d5d61ce aco: remove radv vs prolog key from aco internals.
This creates an aco specific key, and converts radv to it.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16342>
2022-05-11 19:07:11 +00:00
Dave Airlie
04c07a2413 aco/radv: convert to aco shader info at the radv level.
This removes the radv shader info type from aco completely.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16342>
2022-05-11 19:07:11 +00:00
Dave Airlie
87df607ff5 aco: move to a minimal aco shader info struct.
This should be kept to only things aco uses, and expanded when
radeonsi support is added. Things should be removed if lowered in NIR.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16342>
2022-05-11 19:07:11 +00:00
Dave Airlie
a2701bfdb8 aco: move info pointer to a copy.
This is just setup to move this to a different struct later.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16342>
2022-05-11 19:07:11 +00:00
Daniel Schürmann
8d8c59b4cd aco: split num_waves adjustment into separate function
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16039>
2022-04-29 15:39:10 +00:00
Daniel Schürmann
6220046ad1 aco: remove 'max_waves' and use 'num_waves' to adjust for LDS and workgroup size
Totals from 21 (0.02% of 134913) affected shaders: (GFX10.3)
VGPRs: 1024 -> 1176 (+14.84%)
CodeSize: 127824 -> 127664 (-0.13%); split: -0.17%, +0.04%
MaxWaves: 416 -> 378 (-9.13%)
Instrs: 22521 -> 22502 (-0.08%); split: -0.17%, +0.09%
Latency: 146386 -> 143154 (-2.21%); split: -2.21%, +0.00%
InvThroughput: 28379 -> 28944 (+1.99%); split: -0.23%, +2.22%
VClause: 575 -> 579 (+0.70%); split: -0.87%, +1.57%
SClause: 692 -> 645 (-6.79%)
Copies: 780 -> 747 (-4.23%); split: -4.74%, +0.51%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16039>
2022-04-29 15:39:10 +00:00
Rhys Perry
2135c88d9c aco: fix signedness of DS_instruction::offset0/1
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13778>
2022-04-13 23:08:07 +00:00
Daniel Schürmann
d703a0e808 aco: remove register hints entirely
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15408>
2022-04-13 21:52:43 +00:00
Daniel Schürmann
007cb02db9 aco: use branch definition as scratch register for SSA lowering
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15505>
2022-03-28 07:36:46 +00:00
Daniel Schürmann
2d1e6b756e aco: remove 'high' parameter from can_use_opsel()
No fossil-db changes.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15551>
2022-03-25 22:02:50 +00:00
Rhys Perry
e12bee3cb7 aco: improve support for v_fma_mix
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14769>
2022-03-17 19:04:17 +00:00
Timur Kristóf
cd0dd5d6b7 aco: Add storage class for Task Shader payload.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15161>
2022-02-25 13:20:08 +01:00
Daniel Schürmann
13c3137960 aco: merge block_kind_uses_[demote|discard_if]
These serve the same purpose. The new name is
block_kind_uses_discard.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14805>
2022-02-08 16:16:07 +00:00