Commit graph

162 commits

Author SHA1 Message Date
Daniel Schürmann
d659ce0d6c aco/ra: prevent underflow register for p_create_vector operands
It could happen that we tested negative out-of-range
registers for p_create_vector operands resulting in a crash.

Fixes: 8962510e38 ('aco/ra: Conservatively refactor get_reg_specified to use PhysRegInterval')
Closes: #4697
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10799>
2021-05-14 17:26:41 +00:00
Rhys Perry
776ba40115 aco: add and use Program::progress
This is used when printing the program and to avoid updating register
demand during post-RA liveness analysis.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10315>
2021-04-21 11:09:33 +00:00
Rhys Perry
5b8a4516e6 aco/ra: remove live-in temporary from live_out_per_block when moving it
Otherwise, handle_loop_phis() might pass it to handle_live_in() and then
we could have two phis for this variable.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 7c64623e94 ("aco/ra: refactor SSA repairing during register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10236>
2021-04-14 19:04:08 +00:00
Rhys Perry
11fde1247c aco/ra: use original names when renaming loop carried phi operands
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 7c64623e94 ("aco/ra: refactor SSA repairing during register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10236>
2021-04-14 19:04:08 +00:00
Daniel Schürmann
8e43abcd2c aco/ra: remove exec handling for phis
These are not temporaries anymore.

Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9763>
2021-03-25 15:12:19 +00:00
Daniel Schürmann
3284f755a3 aco/ra: allow m0 in get_reg_specified()
Totals from 1 (0.00% of 136546) affected shaders (Navi10):
CodeSize: 12788 -> 12776 (-0.09%)
Instrs: 2441 -> 2438 (-0.12%)
Latency: 29713 -> 29731 (+0.06%)
InvThroughput: 14857 -> 14866 (+0.06%)
Copies: 354 -> 353 (-0.28%)
Branches: 66 -> 65 (-1.52%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9763>
2021-03-25 15:12:19 +00:00
Daniel Schürmann
4bfbd4de84 aco/ra: iterate backwards when coalescing phis
Aligning the phi definition with the operand from
the else- block can reduce the number of branches
if the else- block is otherwise empty.

Totals from 16 (0.01% of 136546) affected shaders (Navi10:
CodeSize: 707848 -> 707312 (-0.08%); split: -0.09%, +0.01%
Instrs: 126534 -> 126400 (-0.11%); split: -0.13%, +0.02%
Latency: 6399306 -> 6395082 (-0.07%)
InvThroughput: 6134374 -> 6132119 (-0.04%); split: -0.04%, +0.00%
SClause: 1879 -> 1871 (-0.43%)
Copies: 36316 -> 36219 (-0.27%); split: -0.37%, +0.10%
Branches: 4154 -> 4127 (-0.65%); split: -0.67%, +0.02%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9763>
2021-03-25 15:12:19 +00:00
Daniel Schürmann
7c64623e94 aco/ra: refactor SSA repairing during register allocation
The previous approach attempted to construct phi nodes
on-demand and on-the-fly. Due to several bugs, it became
necessary to always create incomplete phis for all live-in
variables on loop headers, which is highly inefficient.

The new approach assumes that live-in variables on loop-
headers don't get renamed, and afterwards does one renaming
pass per loop nest. This greatly simplifies the code and
reduces the memory footprint.

Totals from 37 (0.03% of 136546) affected shaders (Navi10):
CodeSize: 588148 -> 588020 (-0.02%); split: -0.03%, +0.01%
Instrs: 111793 -> 111761 (-0.03%); split: -0.04%, +0.01%
Latency: 4546013 -> 4545611 (-0.01%); split: -0.02%, +0.01%
InvThroughput: 2806217 -> 2805730 (-0.02%); split: -0.03%, +0.01%
VClause: 2044 -> 2046 (+0.10%)
SClause: 3889 -> 3884 (-0.13%)
Copies: 17730 -> 17700 (-0.17%); split: -0.23%, +0.06%
Branches: 3282 -> 3280 (-0.06%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Acked-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9763>
2021-03-25 15:12:19 +00:00
Daniel Schürmann
3ea2c05b32 aco/ra: split register_file initialization into separate function
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9763>
2021-03-25 15:12:19 +00:00
Daniel Schürmann
e4902d4574 aco/ra: split affinity creation into separate function
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9763>
2021-03-25 15:12:19 +00:00
Tony Wasserka
893aab57a8 aco/ra: Avoid unnecessary copying of std::vectors
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9801>
2021-03-24 22:20:25 +00:00
Daniel Schürmann
3413c48375 aco/ra: allow VCC on SMEM sbase operand on GFX10+
Totals from 26 (0.02% of 146267) affected shaders (Navi10):
CodeSize: 324764 -> 324560 (-0.06%)
Instrs: 61090 -> 61039 (-0.08%)
Cycles: 2681952 -> 2668872 (-0.49%)
VMEM: 5773 -> 5769 (-0.07%)
SMEM: 1621 -> 1611 (-0.62%)
SClause: 2546 -> 2543 (-0.12%)
Copies: 4671 -> 4620 (-1.09%)

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9531>
2021-03-18 17:15:00 +00:00
Rhys Perry
3d4c13f3b8 aco: add DeviceInfo
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8761>
2021-02-15 13:44:22 +00:00
Daniel Schürmann
8b793f9567 aco: remove dead code for the handling of exec temporaries
Totals from 26026 (18.67% of 139391) affected shaders (Navi10):
PreSGPRs: 370993 -> 326177 (-12.08%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8870>
2021-02-12 22:41:31 +00:00
Daniel Schürmann
171fbe3ae1 aco: check get_reg_specified() on register hints
This ensures that max_used_sgpr is adjusted accordingly.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8921>
2021-02-12 19:00:18 +00:00
Daniel Schürmann
dd16e21e97 aco: also consider VCC in get_reg_specified()
This allows split_vector and others to keep their VCC position.

Totals from 4573 (3.28% of 139391) affected shaders (Navi10):
CodeSize: 54292268 -> 54289324 (-0.01%); split: -0.03%, +0.03%
Instrs: 10327645 -> 10326941 (-0.01%); split: -0.04%, +0.04%
Cycles: 744410748 -> 744034732 (-0.05%); split: -0.07%, +0.02%
VMEM: 749093 -> 749092 (-0.00%); split: +0.00%, -0.00%
SMEM: 269306 -> 269322 (+0.01%)
SClause: 358746 -> 358744 (-0.00%)
Copies: 826051 -> 823910 (-0.26%); split: -0.55%, +0.29%
Branches: 355074 -> 356493 (+0.40%); split: -0.01%, +0.41%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8921>
2021-02-12 19:00:18 +00:00
Daniel Schürmann
b98a4d4dd7 aco: refactor GPR limit calculation
This patch delays the calculation of GPR limits in order to
precisely incorporate extra registers (VCC etc.) and shared VGPRs.

Additionally, the allocation granularity is used to set the config.
This has some effect on the reported SGPR stats.

Totals (Navi10):
SGPRs: 6971787 -> 17753642 (+154.65%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8921>
2021-02-12 19:00:18 +00:00
Rhys Perry
cc8613c8d5 aco: add fallback algorithm in get_reg()
The generated code is often terrible, but the situations where this is
needed are rare.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8713>
2021-02-05 14:41:24 +00:00
Tony Wasserka
5df057a3e6 aco/ra: Add helper to get a PhysRegInterval for the register demand
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8737>
2021-02-03 08:17:49 +00:00
Tony Wasserka
5fdc298e4b aco/ra: Remove preprocessor guards for print_regs
This debug helper isn't used anywhere by default, but compiling it anyway
ensures future refactors don't break it.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8719>
2021-01-27 12:08:47 +01:00
Tony Wasserka
3675aefa84 aco/ra: Fix build with print_regs enabled
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8719>
2021-01-27 12:08:47 +01:00
Tony Wasserka
b241a050da aco/ra: Fix print_regs using the wrong constant to check for blocked slots
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8719>
2021-01-27 12:08:47 +01:00
Rhys Perry
e115b01948 aco: return references in instruction cast methods
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8595>
2021-01-22 14:12:33 +00:00
Rhys Perry
1d245cd18b aco: use format-check methods
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8595>
2021-01-22 14:12:32 +00:00
Rhys Perry
70dbcfa1c9 aco: use instruction cast methods
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8595>
2021-01-22 14:12:32 +00:00
Rhys Perry
441ead5fb3 aco: remove Format::{VOP3A,VOP3B}
These are really the same as Format::VOP3.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8595>
2021-01-22 14:12:32 +00:00
Rhys Perry
824eba2148 aco: don't consider a phi trivial if same's register doesn't match the def
For example:
 s2: %688:s[32-33] = p_linear_phi %3:s[10-11], %688:s[32-33]
would have been considered trivial.

This might happen due to parallelcopies when assigning phi registers.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 69b6069dd2 ("aco: refactor try_remove_trivial_phi() in RA")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8645>
2021-01-22 12:42:47 +00:00
Rhys Perry
af9977a3d5 aco: add affinity for non-sequential MIMG operands
fossil-db (GFX10.3):
Totals from 42008 (30.14% of 139391) affected shaders:
VGPRs: 2139116 -> 2147696 (+0.40%); split: -0.06%, +0.46%
CodeSize: 199109120 -> 198637852 (-0.24%); split: -0.24%, +0.01%
Instrs: 37713901 -> 37714574 (+0.00%); split: -0.02%, +0.03%
Cycles: 1621911328 -> 1621634168 (-0.02%); split: -0.02%, +0.01%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8523>
2021-01-20 16:46:54 +00:00
Rhys Perry
faf3e9a27f aco: move VADDR to the end of the operand list
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8523>
2021-01-20 16:46:54 +00:00
Rhys Perry
fcda9b6737 aco: fix convert_to_SDWA() check in add_subdword_definition()
v_or_b32 with a v2b definition should use SDWA if is_partial=true.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 56345b8c61 ("aco: allow reading/writing upper halves/bytes when possible")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8577>
2021-01-19 15:38:51 +00:00
Tony Wasserka
b603875482 aco/ra: Use PhysRegInterval for count_zero
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
c30e83cc51 aco/ra: Use PhysRegInterval for collect_vars parameters
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
0959b7c435 aco/ra: Use PhysReg when indexing into RegisterFile's containers
This gets rid of a lot of implicit/explicit conversions from PhysReg to
unsigned.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
c3660f4781 aco/ra: Use PhysReg for member functions of PhysRegInterval
This replaces the various PhysReg{lb} casts that had been all over the place.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
d2d0096c0c aco/ra: Remove unused function parameter
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
d9e1375e27 aco/ra: Use std::all_of to simplify a loop
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
f7e6b61379 aco/ra: Add helpers to test for intersection/containment of reg intervals
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
88f21ad87a aco/ra: Move commonly repeated code to a helper function
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
8962510e38 aco/ra: Conservatively refactor get_reg_specified to use PhysRegInterval
All expressions have been replaced by their closest equivalent. No major
simplification efforts have been made to minimize risk of regressions.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
46c9d76134 aco/ra: Use std::all_of to simplify a loop
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
2b3b2f7ff5 aco/ra: Use std::find_if(_not) to clean up get_reg_simple
This makes for a more self-describing iteration behavior, and it gets rid
of the need for the duplicated "final check" at the bottom.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
ebdb362937 aco/ra: Add iterator interface for PhysRegInterval
This enables various loops to use range-based for.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:06 +00:00
Tony Wasserka
689ce1f39d aco/ra: Remove always-false conditions
All code paths that set "found" to true either break or return before the
loop header is reached again, so the checks are unnecessary.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:05 +00:00
Tony Wasserka
46eee40abc aco/ra: Conservatively refactor existing code to use PhysRegInterval
All expressions have been replaced by their closest equivalent. No major
simplification efforts have been made to minimize risk of regressions.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:05 +00:00
Tony Wasserka
9bbd6162a9 aco/ra: Introduce PhysRegInterval helper class
This mainly clarifies the semantics of register bounds (inclusive vs
exclusive), and further groups related varaibles together to clarify
sliding-window-style loops.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:05 +00:00
Tony Wasserka
67c1f32228 aco/ra: Update register use bounds before recursing in get_regs_for_copies
Delaying the call to adjust_max_used_regs until after get_regs_for_copies
returns puts the RA context into a state where registers past max_used_gpr
may be blocked. This isn't an issue on its own, but it adds a surprising
corner case to get_reg_simple that is easily avoided now.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7799>
2021-01-13 18:21:05 +00:00
Daniel Schürmann
6ecbccfb23 aco: optimize v_pk_fma_f16 -> v_pk_fmac_f16 on GFX10
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6680>
2021-01-13 17:46:56 +00:00
Daniel Schürmann
4a75a28698 aco/RA: fix subdword operands on VOP3P instructions
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6680>
2021-01-13 17:46:55 +00:00
Daniel Schürmann
00cf077c15 aco/ra: fix infinite recursion in get_reg_simple() with subdword registers
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>

Fixes: f8c7661eca ('aco: try to better align 8+ dword SGPR vectors')
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8425>
2021-01-12 16:14:00 +00:00
Rhys Perry
f8c7661eca aco: try to better align 8+ dword SGPR vectors
This doesn't have much of an effect, but it helps avoid a
pathological case for Assassin's Creed Valhalla and a RDR2 shader with a
future change.

fossil-db (Sienna):
Totals from 55074 (39.51% of 139391) affected shaders:
SGPRs: 3515076 -> 3567744 (+1.50%); split: -0.01%, +1.51%
CodeSize: 206942120 -> 206941868 (-0.00%); split: -0.00%, +0.00%
Instrs: 39625900 -> 39625837 (-0.00%); split: -0.00%, +0.00%
Cycles: 1640088780 -> 1640088828 (+0.00%); split: -0.00%, +0.00%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4070
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8416>
2021-01-11 16:35:19 +00:00