fdo-mirrors/mesa
1
0
Fork 0
mirror of https://gitlab.freedesktop.org/mesa/mesa.git synced 2026-05-17 18:18:06 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
217232 commits 66 branches 935 tags 932 MiB
Commit graph

136 commits

Author SHA1 Message Date
Rhys Perry
bf56f1efd3 aco: move update(fixed_reg_demand) into update_vgpr_sgpr_demand 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
 2026-01-19 09:17:50 +00:00
Natalie Vock
897c95c37e aco: Include arbitrarily fixed registers in max_reg_demand 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
 2026-01-12 21:46:50 +00:00
Rhys Perry
6a5982cfe5 aco/scheduler: fix register demand check 
Fixes deathloop/01f8d58bf245663b with gfx1201.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 668259ef0b ("aco/scheduler: move clauses through RAR dependencies")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38402>
 2025-11-13 09:07:12 +00:00
Daniel Schürmann
668259ef0b aco/scheduler: move clauses through RAR dependencies
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
Details
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Details 
For simplicity, we limit this feature to only one RAR-dependency per clause.
This allows to quickly correct the register demand changes that occur by
switching the kill flags.

Totals from 5861 (7.34% of 79839) affected shaders: (Navi48)
Instrs: 4891340 -> 4883789 (-0.15%); split: -0.21%, +0.06%
CodeSize: 25556612 -> 25527244 (-0.11%); split: -0.16%, +0.05%
VGPRs: 347044 -> 347140 (+0.03%); split: -0.13%, +0.16%
Latency: 32697095 -> 32642428 (-0.17%); split: -0.25%, +0.08%
InvThroughput: 4975909 -> 4975086 (-0.02%); split: -0.06%, +0.05%
VClause: 102152 -> 93852 (-8.13%); split: -8.22%, +0.10%
SClause: 101232 -> 101205 (-0.03%); split: -0.03%, +0.00%
Copies: 305189 -> 305651 (+0.15%); split: -0.56%, +0.71%
Branches: 87032 -> 87045 (+0.01%); split: -0.00%, +0.02%
VALU: 2776634 -> 2777097 (+0.02%); split: -0.06%, +0.08%
SALU: 662066 -> 660379 (-0.25%); split: -0.26%, +0.01%
VOPD: 4801 -> 4800 (-0.02%); split: +1.21%, -1.23%

Totals from 5680 (7.12% of 79825) affected shaders: (Vangogh)
MaxWaves: 111282 -> 111290 (+0.01%)
Instrs: 4955907 -> 4950709 (-0.10%); split: -0.15%, +0.04%
CodeSize: 26026264 -> 26014272 (-0.05%); split: -0.10%, +0.05%
VGPRs: 320784 -> 320776 (-0.00%); split: -0.03%, +0.03%
Latency: 35645457 -> 35584438 (-0.17%); split: -0.32%, +0.15%
InvThroughput: 8233912 -> 8236524 (+0.03%); split: -0.10%, +0.13%
VClause: 107017 -> 96804 (-9.54%); split: -9.69%, +0.15%
SClause: 98633 -> 98592 (-0.04%); split: -0.05%, +0.01%
Copies: 394041 -> 393584 (-0.12%); split: -0.52%, +0.40%
Branches: 120235 -> 120231 (-0.00%); split: -0.02%, +0.01%
VALU: 3183571 -> 3183114 (-0.01%); split: -0.06%, +0.05%
SALU: 735546 -> 734143 (-0.19%); split: -0.20%, +0.01%

Totals from 2507 (3.96% of 63370) affected shaders: (Vega10)

MaxWaves: 13643 -> 13637 (-0.04%)
Instrs: 1496453 -> 1496135 (-0.02%); split: -0.11%, +0.09%
CodeSize: 7777880 -> 7776608 (-0.02%); split: -0.09%, +0.07%
VGPRs: 134164 -> 134104 (-0.04%); split: -0.11%, +0.07%
Latency: 17465181 -> 17483075 (+0.10%); split: -0.36%, +0.47%
InvThroughput: 8830470 -> 8851751 (+0.24%); split: -0.09%, +0.33%
VClause: 42012 -> 38825 (-7.59%); split: -8.00%, +0.42%
SClause: 34586 -> 34549 (-0.11%); split: -0.12%, +0.01%
Copies: 137896 -> 137668 (-0.17%); split: -0.86%, +0.69%
VALU: 1092468 -> 1092240 (-0.02%); split: -0.11%, +0.09%
SALU: 132956 -> 132569 (-0.29%); split: -0.34%, +0.05%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38135>
 2025-11-11 11:31:52 +00:00
Daniel Schürmann
65ba8a0e8b aco/scheduler: refactor downwards dependency check 
We can also ignore killed operands when checking for RAR dependencies
as these cannot appear later anymore.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38135>
 2025-11-11 11:31:52 +00:00
Daniel Schürmann
ce3cc03153 aco/scheduler: use hashmap for RAR_dependencies 
Store information about the (relative) position of the RAR dependency.
This will allow to correct for register-demand changes when scheduling across.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38135>
 2025-11-11 11:31:52 +00:00
Daniel Schürmann
6c0dd8164f aco/scheduler: remove MoveState::RAR_dependencies_clause 
Since moving clauses as batch, this can easily be derived from RAR_dependencies.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38135>
 2025-11-11 11:31:52 +00:00
Daniel Schürmann
5ef47ba231 aco/scheduler: assert that the register demand stays within pre-determined bounds 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38135>
 2025-11-11 11:31:52 +00:00
Daniel Schürmann
82ba730994 aco/scheduler: remove unused include 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38135>
 2025-11-11 11:31:51 +00:00
Natalie Vock
f0c613765c aco: Add preload_preserved pseudo instruction 
These are helper instructions for the spill_preserved pass to insert
reloads for registers that are preserved by the ABI, yet
clobbered by the callee shader.

There is one p_reload_preserved instruction at the end of each block.
This allows us to insert reloads early, to alleviate the high latency of
scratch reloads.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37381>
 2025-11-06 12:09:39 +00:00
Natalie Vock
f0d3d0ad21 aco/scheduler: Bail early on unreorderable instructions
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
Details
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Details 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37212>
 2025-09-22 11:13:50 +00:00
Natalie Vock
28dc185966 aco/sched: Handle calls 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34531>
 2025-09-15 17:16:20 +00:00
Rhys Perry
0ee1c137f9 aco: don't move release barriers after interlock end 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36491>
 2025-09-09 12:34:40 +00:00
Rhys Perry
7c056dd473 aco: add is_atomic_or_control_instr helper 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36491>
 2025-09-09 12:34:40 +00:00
Rhys Perry
1105f7b98f aco: fix signed integer overflow 
Fix UBSan error:
runtime error: signed integer overflow: 2147483647 + 32 cannot be represented in type 'int'

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37055>
 2025-09-03 11:47:00 +00:00
Daniel Schürmann
0546ecfadb aco/scheduler: small refactor of schedule_VMEM() 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:12 +00:00
Daniel Schürmann
0c590eb903 aco/scheduler: schedule VMEM store clauses during the regular forward pass 
Totals from 1456 (1.82% of 79839) affected shaders: (Navi48)

MaxWaves: 37780 -> 37128 (-1.73%); split: +0.15%, -1.87%
Instrs: 3788175 -> 3788435 (+0.01%); split: -0.04%, +0.04%
CodeSize: 20468648 -> 20467432 (-0.01%); split: -0.04%, +0.03%
VGPRs: 86820 -> 91440 (+5.32%); split: -0.10%, +5.42%
Latency: 26866232 -> 26858867 (-0.03%); split: -0.04%, +0.01%
InvThroughput: 3491741 -> 3828339 (+9.64%); split: -0.02%, +9.66%
VClause: 90413 -> 89426 (-1.09%); split: -1.27%, +0.18%
SClause: 130532 -> 130530 (-0.00%); split: -0.00%, +0.00%
Copies: 347397 -> 347806 (+0.12%); split: -0.11%, +0.23%
Branches: 117476 -> 117496 (+0.02%)
VALU: 1897427 -> 1897830 (+0.02%); split: -0.02%, +0.04%
SALU: 602365 -> 602379 (+0.00%)
VOPD: 1259 -> 1251 (-0.64%); split: +0.24%, -0.87%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:12 +00:00
Daniel Schürmann
f601eb8555 aco/scheduler: move clauses as batch 
Totals from 391 (0.49% of 79839) affected shaders:

Instrs: 612478 -> 612515 (+0.01%); split: -0.06%, +0.06%
CodeSize: 3342896 -> 3343228 (+0.01%); split: -0.04%, +0.05%
Latency: 6909794 -> 6909938 (+0.00%); split: -0.03%, +0.03%
VClause: 10752 -> 10167 (-5.44%); split: -5.46%, +0.02%
Copies: 26623 -> 26627 (+0.02%); split: -0.00%, +0.02%
VALU: 377494 -> 377499 (+0.00%); split: -0.00%, +0.00%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:12 +00:00
Daniel Schürmann
70f0c065e8 aco/scheduler: ignore potential SMEM stalls when forming clauses 
Totals from 4190 (5.25% of 79839) affected shaders: (Navi48)

MaxWaves: 117020 -> 117014 (-0.01%)
Instrs: 4801892 -> 4801547 (-0.01%); split: -0.06%, +0.05%
CodeSize: 25327632 -> 25325500 (-0.01%); split: -0.05%, +0.04%
VGPRs: 236452 -> 236488 (+0.02%)
Latency: 30569070 -> 30539464 (-0.10%); split: -0.13%, +0.04%
InvThroughput: 4891650 -> 4891062 (-0.01%); split: -0.03%, +0.01%
VClause: 119615 -> 118763 (-0.71%); split: -1.02%, +0.31%
SClause: 100482 -> 100297 (-0.18%); split: -0.44%, +0.26%
Copies: 326644 -> 326756 (+0.03%); split: -0.19%, +0.22%
Branches: 98982 -> 98980 (-0.00%)
VALU: 2712397 -> 2712534 (+0.01%); split: -0.02%, +0.03%
SALU: 591836 -> 591817 (-0.00%); split: -0.00%, +0.00%
VOPD: 993 -> 987 (-0.60%); split: +0.20%, -0.81%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:11 +00:00
Daniel Schürmann
d3a0f268b9 aco/scheduler: short-cut downwards_move_clause() when no movement is done 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:11 +00:00
Daniel Schürmann
8543b6cf2e aco/scheduler: remove DownwardsCursor::clause_demand 
As we stop scheduling after forming clauses, this value
is not needed anymore.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:10 +00:00
Daniel Schürmann
5ae30deffb aco/scheduler: remove DownwardsCursor::insert_demand_clause 
This partially reverts 93872270f0 ('aco/scheduler: keep track of RegisterDemand at DownwardsCursor::insert_idx{_clause}').

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:10 +00:00
Daniel Schürmann
e95d728a98 aco/scheduler: split downwards_move_clause() from downwards_move() 
We will do batched moves for clauses with the next commit.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:09 +00:00
Daniel Schürmann
37299a8d1a aco/scheduler: Stop downwards scheduling after encountering the first clause 
Totals from 9899 (12.40% of 79839) affected shaders: (Navi48)

MaxWaves: 276355 -> 276317 (-0.01%); split: +0.01%, -0.02%
Instrs: 8781768 -> 8766504 (-0.17%); split: -0.25%, +0.07%
CodeSize: 46297556 -> 46236104 (-0.13%); split: -0.19%, +0.06%
VGPRs: 574680 -> 574800 (+0.02%); split: -0.00%, +0.03%
Latency: 54261324 -> 54357916 (+0.18%); split: -0.14%, +0.32%
InvThroughput: 9122700 -> 9121115 (-0.02%); split: -0.07%, +0.05%
VClause: 222062 -> 218499 (-1.60%); split: -2.33%, +0.73%
SClause: 167138 -> 163233 (-2.34%); split: -2.43%, +0.09%
Copies: 602395 -> 598560 (-0.64%); split: -1.21%, +0.57%
Branches: 161939 -> 161932 (-0.00%); split: -0.01%, +0.00%
VALU: 5063999 -> 5060199 (-0.08%); split: -0.14%, +0.07%
SALU: 988254 -> 988285 (+0.00%); split: -0.02%, +0.02%
VOPD: 2478 -> 2443 (-1.41%); split: +0.40%, -1.82%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:09 +00:00
Daniel Schürmann
fb6b95517e aco/scheduler: check dependencies of entire clause upfront 
and bail if any instruction of the clause can't be moved.

Totals from 4310 (5.40% of 79839) affected shaders:

MaxWaves: 115826 -> 115834 (+0.01%)
Instrs: 6256436 -> 6257599 (+0.02%); split: -0.05%, +0.07%
CodeSize: 32816488 -> 32820768 (+0.01%); split: -0.04%, +0.05%
VGPRs: 260184 -> 260172 (-0.00%)
Latency: 41207213 -> 41052150 (-0.38%); split: -0.45%, +0.07%
InvThroughput: 6822608 -> 6815208 (-0.11%); split: -0.14%, +0.03%
VClause: 148412 -> 147133 (-0.86%); split: -1.03%, +0.17%
SClause: 120854 -> 120856 (+0.00%); split: -0.01%, +0.01%
Copies: 425910 -> 427276 (+0.32%); split: -0.25%, +0.57%
VALU: 3572293 -> 3573647 (+0.04%); split: -0.03%, +0.07%
VOPD: 2803 -> 2816 (+0.46%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36599>
 2025-08-19 16:59:08 +00:00
Daniel Schürmann
d3743dd7ba aco/scheduler: improve scheduling heuristic
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
Details
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Details 
The heuristic we are currently using still stems from the GCN era
with the only adjustments being made for RDNA was to double (or triple)
the wave count.

This rewrite aims to detangle some concepts and provide more consistent results.

 - wave_factor: The purpose of this value is to reflect that RDNA SIMDs can
                accomodate twice as many waves as GCN SIMDs.
 - reg_file_multiple: This value accounts for the larger register file of wave32
                      and some RDNA3 families.
 - wave_minimum: Below this value, we don't sacrifice any waves. It corresponds
                 to a register demand of 64 VGPRs in wave64.
 - occupancy_factor: Depending on target_waves and wave_factor, this controls
                     the scheduling window sizes and number of moves.

The main differences from the previous heuristic is a lower wave minimum and
a slightly less aggressive reduction of waves.
It also increases SMEM_MAX_MOVES in order to mitigate some of the changes
from targeting less waves.

Totals from 62777 (78.63% of 79839) affected shaders: (Navi48)
MaxWaves: 1880983 -> 1848028 (-1.75%); split: +0.01%, -1.76%
Instrs: 40904711 -> 40800797 (-0.25%); split: -0.39%, +0.14%
CodeSize: 217132208 -> 216748832 (-0.18%); split: -0.29%, +0.12%
VGPRs: 3019304 -> 3099596 (+2.66%); split: -0.11%, +2.77%
Latency: 268857129 -> 265951122 (-1.08%); split: -1.33%, +0.25%
InvThroughput: 40960938 -> 41044533 (+0.20%); split: -0.18%, +0.39%
VClause: 794000 -> 782913 (-1.40%); split: -2.24%, +0.84%
SClause: 1192476 -> 1150831 (-3.49%); split: -3.94%, +0.45%
Copies: 2720470 -> 2700148 (-0.75%); split: -1.84%, +1.09%
Branches: 785926 -> 785951 (+0.00%); split: -0.01%, +0.01%
VALU: 22918411 -> 22890189 (-0.12%); split: -0.19%, +0.06%
SALU: 5281201 -> 5289486 (+0.16%); split: -0.21%, +0.36%
VOPD: 8790 -> 8685 (-1.19%); split: +1.08%, -2.28%

Totals from 62081 (77.77% of 79825) affected shaders: (Navi31)
MaxWaves: 1848555 -> 1812347 (-1.96%); split: +0.01%, -1.97%
Instrs: 39794460 -> 39704180 (-0.23%); split: -0.39%, +0.16%
CodeSize: 208987052 -> 208621524 (-0.17%); split: -0.31%, +0.13%
VGPRs: 3046284 -> 3135156 (+2.92%); split: -0.11%, +3.03%
Latency: 268863465 -> 265218186 (-1.36%); split: -1.59%, +0.23%
InvThroughput: 41101515 -> 41167075 (+0.16%); split: -0.22%, +0.38%
VClause: 795316 -> 774899 (-2.57%); split: -3.17%, +0.61%
SClause: 1177294 -> 1135451 (-3.55%); split: -4.06%, +0.51%
Copies: 2743254 -> 2725127 (-0.66%); split: -1.90%, +1.24%
Branches: 801395 -> 801428 (+0.00%); split: -0.01%, +0.02%
VALU: 23898938 -> 23871294 (-0.12%); split: -0.20%, +0.08%
SALU: 3908807 -> 3919130 (+0.26%); split: -0.23%, +0.50%
VOPD: 8529 -> 8500 (-0.34%); split: +1.29%, -1.63%

Totals from 44996 (71.01% of 63370) affected shaders: (Vega10)

MaxWaves: 307074 -> 304808 (-0.74%); split: +0.63%, -1.37%
Instrs: 22743534 -> 22716240 (-0.12%); split: -0.22%, +0.10%
CodeSize: 117284856 -> 117173212 (-0.10%); split: -0.19%, +0.09%
SGPRs: 3249008 -> 3330480 (+2.51%); split: -0.36%, +2.87%
VGPRs: 1901400 -> 1943880 (+2.23%); split: -0.60%, +2.83%
Latency: 224839126 -> 222878477 (-0.87%); split: -1.19%, +0.31%
InvThroughput: 114389570 -> 114316559 (-0.06%); split: -0.17%, +0.11%
VClause: 482012 -> 473304 (-1.81%); split: -2.86%, +1.05%
SClause: 757799 -> 717092 (-5.37%); split: -5.64%, +0.27%
Copies: 2182735 -> 2183598 (+0.04%); split: -1.17%, +1.21%
Branches: 396026 -> 395996 (-0.01%); split: -0.03%, +0.02%
VALU: 16740283 -> 16728098 (-0.07%); split: -0.14%, +0.07%
SALU: 2133575 -> 2145863 (+0.58%); split: -0.29%, +0.86%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30720>
 2025-08-06 09:16:33 +00:00
Daniel Schürmann
df2697c9ab aco/scheduler: remove unused include of unordered_set 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33644>
 2025-02-21 13:49:41 +00:00
Daniel Schürmann
93872270f0 aco/scheduler: keep track of RegisterDemand at DownwardsCursor::insert_idx{_clause} 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33644>
 2025-02-21 13:49:41 +00:00
Daniel Schürmann
f58654e98f aco/scheduler: keep track of RegisterDemand at UpwardsCursor::insert_idx 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33644>
 2025-02-21 13:49:41 +00:00
Daniel Schürmann
52253da783 aco: unify get_addr_sgpr_from_waves() and get_addr_vgpr_from_waves() into one function 
which returns the limit as RegisterDemand.

Also remove the unused get_extra_sgprs() from aco_ir.h.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33644>
 2025-02-21 13:49:41 +00:00
Daniel Schürmann
6ea9443726 aco/scheduler: stop rounding down the target number of waves on GFX10+ 
This way, it can make use of uneven wave numbers.

Totals from 4078 (5.14% of 79395) affected shaders: (Navi21)
MaxWaves: 58715 -> 65460 (+11.49%); split: +11.49%, -0.01%
Instrs: 5033684 -> 5048244 (+0.29%); split: -0.09%, +0.38%
CodeSize: 26833884 -> 26898780 (+0.24%); split: -0.07%, +0.32%
VGPRs: 302360 -> 265312 (-12.25%); split: -12.26%, +0.01%
Latency: 34636448 -> 36044242 (+4.06%); split: -0.08%, +4.14%
InvThroughput: 7999403 -> 7662697 (-4.21%); split: -4.55%, +0.34%
VClause: 105403 -> 111996 (+6.26%); split: -0.40%, +6.66%
SClause: 132996 -> 133460 (+0.35%); split: -0.81%, +1.16%
Copies: 297036 -> 308122 (+3.73%); split: -0.64%, +4.37%
Branches: 89376 -> 89390 (+0.02%); split: -0.00%, +0.02%
VALU: 3477621 -> 3488510 (+0.31%); split: -0.05%, +0.36%
SALU: 484211 -> 484191 (-0.00%); split: -0.08%, +0.08%

Totals from 1840 (2.32% of 79395) affected shaders: (Navi31)

MaxWaves: 30714 -> 34182 (+11.29%)
Instrs: 3102955 -> 3131001 (+0.90%); split: -0.05%, +0.95%
CodeSize: 16160564 -> 16273100 (+0.70%); split: -0.04%, +0.74%
VGPRs: 174540 -> 150600 (-13.72%)
Latency: 23521914 -> 24515055 (+4.22%); split: -0.07%, +4.29%
InvThroughput: 4373397 -> 4202912 (-3.90%); split: -4.40%, +0.50%
VClause: 59087 -> 64091 (+8.47%); split: -0.24%, +8.71%
SClause: 74844 -> 75366 (+0.70%); split: -0.53%, +1.22%
Copies: 184396 -> 197747 (+7.24%); split: -0.25%, +7.49%
Branches: 46015 -> 46028 (+0.03%); split: -0.00%, +0.03%
VALU: 1929286 -> 1942709 (+0.70%); split: -0.02%, +0.71%
SALU: 216126 -> 215983 (-0.07%); split: -0.18%, +0.12%
VOPD: 1216 -> 1217 (+0.08%); split: +1.40%, -1.32%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33644>
 2025-02-21 13:49:41 +00:00
Daniel Schürmann
676b39d31f aco/scheduler: always respect min_waves on GFX10+ 
It could theoretically happen that for large workgroups,
the scheduler used more registers than allowed.

No fossil changes.

Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33644>
 2025-02-21 13:49:40 +00:00
Timur Kristóf
e9069eec8a aco: Move NGG pos export scheduling determination to drivers. 
And don't schedule them on GFX11+ at all.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33218>
 2025-01-30 15:26:45 +00:00
Georg Lehmann
c279e63a79 aco: rename p_early_exit_if to if_not 
It exits the shaders if the condition is false, not true.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32731>
 2024-12-30 13:05:23 +00:00
Samuel Pitoiset
107f29c39a aco: do not reorder s_trap instructions 
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32055>
 2024-11-11 15:46:36 +00:00
Daniel Schürmann
5a39cbdef6 aco: change signature of get_live_changes() and get_temp_registers() 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30182>
 2024-08-14 08:11:48 +00:00
Daniel Schürmann
e5d920e0b9 aco/scheduler: enable live variables validation when ACO_DEBUG=validate-livevars is set 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30182>
 2024-08-14 08:11:48 +00:00
Daniel Schürmann
6729e81d15 aco/live_var_analysis: inline block->register_demand updates 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962>
 2024-07-10 12:31:02 +00:00
Daniel Schürmann
6c6f382d68 aco: add RegisterDemand member to Instruction 
Since we never need both at the same time, we can use
a union with pass_flags.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962>
 2024-07-10 12:31:02 +00:00
Daniel Schürmann
26c58ca9de aco/scheduler: fix register_demand validation debug code 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29804>
 2024-06-26 09:42:33 +00:00
Georg Lehmann
046414e061 aco: add more anonymous namespaces 
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29740>
 2024-06-18 17:53:07 +00:00
Daniel Schürmann
a497d105e3 aco: move live var information into struct Program 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29713>
 2024-06-14 14:32:35 +00:00
Daniel Schürmann
2322ab427e aco/scheduler: remove unused register_demand parameter 
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29713>
 2024-06-14 14:32:35 +00:00
Rhys Perry
8e475bba61 aco: implement nir_intrinsic_nop_amd and nir_intrinsic_sleep_amd 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Acked-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29466>
 2024-06-06 14:26:52 +00:00
Rhys Perry
2a4424425a aco/gfx12: fix s_wait_event immediate 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Acked-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29466>
 2024-06-06 14:26:52 +00:00
Rhys Perry
fae2a85d57 aco/gfx12: implement subgroup shader clock 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29330>
 2024-05-28 10:52:11 +00:00
Rhys Perry
cb81ec7a61 aco: don't count certain pseudo towards VMEM_STORE_CLAUSE_MAX_GRAB_DIST 
fossil-db (navi31):
Totals from 1023 (1.29% of 79395) affected shaders:
MaxWaves: 29258 -> 29240 (-0.06%)
Instrs: 1134024 -> 1133163 (-0.08%); split: -0.10%, +0.02%
CodeSize: 5682108 -> 5678696 (-0.06%); split: -0.08%, +0.02%
VGPRs: 60248 -> 60272 (+0.04%); split: -0.08%, +0.12%
Latency: 3797510 -> 3792797 (-0.12%); split: -0.18%, +0.05%
InvThroughput: 781270 -> 781239 (-0.00%); split: -0.03%, +0.03%
VClause: 24701 -> 23976 (-2.94%); split: -3.55%, +0.61%
Copies: 75177 -> 75169 (-0.01%); split: -0.20%, +0.19%
VALU: 659939 -> 659962 (+0.00%); split: -0.02%, +0.02%
VOPD: 2040 -> 2009 (-1.52%); split: +0.29%, -1.81%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28948>
 2024-05-14 12:30:54 +00:00
Rhys Perry
88e03feb27 aco: schedule LDS instructions 
fossil-db (navi31):
Totals from 1823 (2.30% of 79395) affected shaders:
MaxWaves: 53845 -> 53827 (-0.03%); split: +0.02%, -0.05%
Instrs: 1736317 -> 1731200 (-0.29%); split: -0.38%, +0.09%
CodeSize: 8876760 -> 8857908 (-0.21%); split: -0.29%, +0.08%
VGPRs: 91688 -> 92276 (+0.64%); split: -0.03%, +0.67%
Latency: 11743095 -> 11698872 (-0.38%); split: -0.42%, +0.04%
InvThroughput: 2070526 -> 2067440 (-0.15%); split: -0.17%, +0.02%
VClause: 39048 -> 39058 (+0.03%); split: -0.01%, +0.03%
SClause: 35371 -> 35406 (+0.10%); split: -0.02%, +0.12%
Copies: 104335 -> 104384 (+0.05%); split: -0.21%, +0.26%
Branches: 29769 -> 29794 (+0.08%); split: -0.00%, +0.09%
VALU: 970925 -> 970974 (+0.01%); split: -0.01%, +0.02%
SALU: 146222 -> 146345 (+0.08%); split: -0.01%, +0.09%
VOPD: 1119 -> 1162 (+3.84%); split: +4.29%, -0.45%

fossil-db (navi21):
Totals from 37078 (46.70% of 79395) affected shaders:
MaxWaves: 990093 -> 990025 (-0.01%)
Instrs: 21130662 -> 21182543 (+0.25%); split: -0.01%, +0.26%
CodeSize: 110205364 -> 110415032 (+0.19%); split: -0.01%, +0.20%
VGPRs: 1407168 -> 1410768 (+0.26%)
Latency: 90024839 -> 89929196 (-0.11%); split: -0.11%, +0.01%
InvThroughput: 17170356 -> 17167412 (-0.02%); split: -0.02%, +0.00%
VClause: 392830 -> 392825 (-0.00%); split: -0.01%, +0.01%
SClause: 463150 -> 463188 (+0.01%); split: -0.00%, +0.01%
Copies: 1768433 -> 1768483 (+0.00%); split: -0.02%, +0.02%
Branches: 605989 -> 606011 (+0.00%); split: -0.00%, +0.00%
VALU: 11614810 -> 11614912 (+0.00%); split: -0.00%, +0.00%
SALU: 3794531 -> 3794655 (+0.00%); split: -0.00%, +0.00%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28763>
 2024-04-23 12:31:59 +00:00
Rhys Perry
0ee4fa33bc aco: schedule LDSDIR instructions 
fossil-db (navi31):
Totals from 33850 (42.63% of 79395) affected shaders:
MaxWaves: 1011236 -> 1011204 (-0.00%)
Instrs: 23589117 -> 23559185 (-0.13%); split: -0.21%, +0.08%
CodeSize: 126099716 -> 125968376 (-0.10%); split: -0.17%, +0.07%
VGPRs: 1348632 -> 1356012 (+0.55%); split: -0.09%, +0.63%
Latency: 183233795 -> 180997751 (-1.22%); split: -1.33%, +0.11%
InvThroughput: 27081576 -> 27056383 (-0.09%); split: -0.15%, +0.06%
VClause: 386453 -> 386551 (+0.03%); split: -0.11%, +0.13%
SClause: 811941 -> 813023 (+0.13%); split: -0.38%, +0.52%
Copies: 1279706 -> 1280051 (+0.03%); split: -0.46%, +0.49%
Branches: 416940 -> 416938 (-0.00%); split: -0.02%, +0.02%
VALU: 13566410 -> 13567367 (+0.01%); split: -0.04%, +0.04%
SALU: 1835804 -> 1835652 (-0.01%); split: -0.02%, +0.01%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11013
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28763>
 2024-04-23 12:31:59 +00:00
Rhys Perry
0bc8a9be67 aco: make store clauses more aggressively 
Apparently this significantly improves performance of a radeonsi resolve
shader.

fossil-db (navi31):
Totals from 2372 (2.99% of 79395) affected shaders:
MaxWaves: 59903 -> 59863 (-0.07%)
Instrs: 3508838 -> 3506178 (-0.08%); split: -0.10%, +0.02%
CodeSize: 18516272 -> 18505956 (-0.06%); split: -0.07%, +0.02%
VGPRs: 152708 -> 154604 (+1.24%)
Latency: 27881253 -> 27861445 (-0.07%); split: -0.07%, +0.00%
InvThroughput: 4076649 -> 4076220 (-0.01%); split: -0.03%, +0.02%
VClause: 92696 -> 89409 (-3.55%); split: -3.55%, +0.01%
Copies: 310787 -> 311697 (+0.29%); split: -0.03%, +0.32%
VALU: 1891048 -> 1891933 (+0.05%); split: -0.01%, +0.05%
VOPD: 2534 -> 2559 (+0.99%); split: +1.07%, -0.08%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11014
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28763>
 2024-04-23 12:31:59 +00:00