fdo-mirrors/mesa
1
0
Fork 0
mirror of https://gitlab.freedesktop.org/mesa/mesa.git synced 2025-12-21 15:50:11 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
142567 commits 64 branches 914 tags 1 GiB
Commit graph

56 commits

Author SHA1 Message Date
Daniel Schürmann
1e2639026f aco: Format. 
Manually adjusted some comments for more intuitive line breaks.

Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11258>
 2021-07-12 21:27:31 +00:00
Daniel Schürmann
0eea0e55ad aco: add 'common/' and 'llvm/' prefix to #includes 
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11271>
 2021-07-12 12:09:31 +00:00
Daniel Schürmann
59fdaa1985 aco: reorder and cleanup #includes 
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11271>
 2021-07-12 12:09:31 +00:00
Rhys Perry
d64f5a3f9d aco: move VMEM instructions below descriptor loads 
This is to prevent sequences like:
   a = descriptor_load()
   vmem(a)
   b = descriptor_load()
   vmem(b)
and instead create:
   a = descriptor_load()
   b = descriptor_load()
   vmem(a)
   vmem(b)

fossil-db (GFX10.3):
Totals from 114521 (78.30% of 146267) affected shaders:
VGPRs: 4540352 -> 4540216 (-0.00%); split: -0.03%, +0.02%
CodeSize: 289864228 -> 289114652 (-0.26%); split: -0.29%, +0.03%
MaxWaves: 2940234 -> 2940338 (+0.00%); split: +0.00%, -0.00%
Instrs: 55112418 -> 54919910 (-0.35%); split: -0.38%, +0.03%
Latency: 956528393 -> 954682011 (-0.19%); split: -0.24%, +0.05%
InvThroughput: 229280830 -> 229238107 (-0.02%); split: -0.04%, +0.02%
VClause: 1141832 -> 1139002 (-0.25%); split: -0.63%, +0.38%
SClause: 2357840 -> 2225008 (-5.63%); split: -6.01%, +0.38%
Copies: 3316040 -> 3331519 (+0.47%); split: -0.31%, +0.77%
Branches: 1187212 -> 1186919 (-0.02%); split: -0.03%, +0.01%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6489>
 2021-06-14 15:47:37 +00:00
Rhys Perry
bc71222cd9 aco: don't move descriptor loads below buffer loads 
fossil-db (GFX10.3):
Totals from 52870 (36.15% of 146267) affected shaders:
VGPRs: 2109936 -> 2110056 (+0.01%); split: -0.01%, +0.01%
CodeSize: 134898056 -> 134812748 (-0.06%); split: -0.08%, +0.02%
MaxWaves: 1347354 -> 1347346 (-0.00%)
Instrs: 25598063 -> 25575415 (-0.09%); split: -0.11%, +0.02%
Latency: 432491613 -> 432047723 (-0.10%); split: -0.12%, +0.02%
InvThroughput: 90940977 -> 90927545 (-0.01%); split: -0.03%, +0.01%
VClause: 570039 -> 570019 (-0.00%); split: -0.05%, +0.04%
SClause: 1145076 -> 1139040 (-0.53%); split: -0.60%, +0.07%
Copies: 1513949 -> 1513102 (-0.06%); split: -0.32%, +0.26%
Branches: 524279 -> 524275 (-0.00%); split: -0.03%, +0.03%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6489>
 2021-06-14 15:47:37 +00:00
Tony Wasserka
3c390e2eb6 aco/scheduler: Move cursor handling state to dedicated interfaces 
This clarifies the semantics of the index variables compared to the previous
version, which used the same variables in a slightly different way depending
on whether they were used for downwards moves or upwards ones.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10885>
 2021-06-07 12:09:39 +02:00
Tony Wasserka
81761a311e aco/scheduler: Clean up register demand tracking 
Refactoring total_demand and total_demand_clause to cover non-overlapping
instruction intervals makes the code easier to follow and allows the register
demand to be updated more efficiently in some cases.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10885>
 2021-06-07 12:09:39 +02:00
Tony Wasserka
80ee9d3947 aco/scheduler: Verify register demand invariants in debug mode 
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10644>
 2021-05-13 15:27:57 +00:00
Tony Wasserka
50ba919d37 aco/scheduler: Fix register demand computation for upwards moves 
The initial value needs to be taken from the instruction that is being
moved over, not the one to be moved.

Additionally the parameter of this function was removed because it was
misleading. Setting it to any value other than source_idx would cause
register_demand to be initialized incorrectly. (Instead, the maximum
demand among the covered instructions would need to be determined.)

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10644>
 2021-05-13 15:27:57 +00:00
Tony Wasserka
c528af1076 aco/scheduler: Fix register demand computation for downwards moves 
Previously, changes in total_demand_clause were not always propagated to
total_demand. For instance, clause moves do not change the local register
demand at the end of a clause, yet they may still affect the total maximum.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 8235bc6411 ("aco: try to group together VMEM loads of the same resource")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4533
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10644>
 2021-05-13 15:27:57 +00:00
Rhys Perry
dfa38fa0c7 aco: group loads from the same vertex binding into the same clause 
In the future, we might have vertex attribute loads from the same binding
but with different descriptors. Since they will be loading from the same
buffer, we should continue grouping them into clauses.

No fossil-db changes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7871>
 2021-05-10 12:09:14 +00:00
Rhys Perry
3d4c13f3b8 aco: add DeviceInfo 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8761>
 2021-02-15 13:44:22 +00:00
Daniel Schürmann
947bf0bd67 aco: don't decrease the vgpr_limit when encountering bpermute 
Instead we recalculate vgpr_limit on demand, depending on
the number of needed shared VGPRs.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8921>
 2021-02-12 19:00:18 +00:00
Rhys Perry
e115b01948 aco: return references in instruction cast methods 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8595>
 2021-01-22 14:12:33 +00:00
Rhys Perry
1d245cd18b aco: use format-check methods 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8595>
 2021-01-22 14:12:32 +00:00
Rhys Perry
70dbcfa1c9 aco: use instruction cast methods 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8595>
 2021-01-22 14:12:32 +00:00
Rhys Perry
489aa8c7cb aco: fix num_waves on GFX10+ 
There are half the SIMDs per CU and physical_vgprs should be 512 instead
of 256.

fossil-db (GFX10.3):
Totals from 3622 (2.60% of 139391) affected shaders:
VGPRs: 298192 -> 289732 (-2.84%); split: -3.43%, +0.59%
CodeSize: 29443432 -> 29458388 (+0.05%); split: -0.00%, +0.06%
MaxWaves: 21703 -> 23395 (+7.80%); split: +7.84%, -0.05%
Instrs: 5677920 -> 5681438 (+0.06%); split: -0.01%, +0.07%
Cycles: 280715524 -> 280895676 (+0.06%); split: -0.00%, +0.07%
VMEM: 981142 -> 981894 (+0.08%); split: +0.18%, -0.10%
SMEM: 243315 -> 243454 (+0.06%); split: +0.07%, -0.02%
VClause: 88991 -> 89767 (+0.87%); split: -0.02%, +0.89%
SClause: 200660 -> 200659 (-0.00%); split: -0.00%, +0.00%
Copies: 430729 -> 434160 (+0.80%); split: -0.07%, +0.86%
Branches: 158004 -> 158021 (+0.01%); split: -0.01%, +0.02%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8523>
 2021-01-20 16:46:54 +00:00
Rhys Perry
4ea0ce2f55 aco: remove can_reorder semantic in get_sync_info_with_hack 
No fossil-db changes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8416>
 2021-01-11 16:35:19 +00:00
Daniel Schürmann
b1e12747b9 aco: create VMEM clauses slightly more aggressive 
Totals from 3325 (2.39% of 139391) affected shaders (NAVI10):
SGPRs: 331528 -> 331056 (-0.14%); split: -0.14%, +0.00%
VGPRs: 306164 -> 337764 (+10.32%); split: -0.02%, +10.34%
CodeSize: 38843180 -> 38865388 (+0.06%); split: -0.04%, +0.10%
MaxWaves: 18908 -> 17028 (-9.94%); split: +0.01%, -9.95%
Instrs: 7423908 -> 7427934 (+0.05%); split: -0.06%, +0.12%
Cycles: 527411756 -> 526388408 (-0.19%); split: -0.21%, +0.02%
VMEM: 1148421 -> 992660 (-13.56%); split: +0.10%, -13.67%
SMEM: 227337 -> 232380 (+2.22%); split: +2.26%, -0.04%
VClause: 146416 -> 111171 (-24.07%); split: -24.10%, +0.03%
SClause: 243674 -> 243689 (+0.01%); split: -0.00%, +0.01%
Copies: 663496 -> 660333 (-0.48%); split: -0.85%, +0.37%
Branches: 223725 -> 223721 (-0.00%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7903>
 2020-12-22 15:08:40 +01:00
Daniel Schürmann
ac40301dbb aco: schedule position exports in the same pass as memory operations 
No fossil-db changes.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7903>
 2020-12-22 15:08:40 +01:00
Daniel Schürmann
0287ebeef3 aco: fix def-use distance calculation when scheduling. 
This change also increases the VMEM_MAX_MOVES
to mitigate some of the scheduling changes.

Totals from 34301 (24.61% of 139391) affected shaders:
SGPRs: 2515440 -> 2552304 (+1.47%); split: -1.25%, +2.71%
VGPRs: 1786676 -> 1794724 (+0.45%); split: -0.31%, +0.76%
CodeSize: 151079856 -> 151209828 (+0.09%); split: -0.06%, +0.15%
MaxWaves: 392454 -> 388966 (-0.89%); split: +0.39%, -1.28%
Instrs: 28870746 -> 28895907 (+0.09%); split: -0.09%, +0.17%
Cycles: 960450680 -> 961315796 (+0.09%); split: -0.09%, +0.18%
VMEM: 19027987 -> 19796223 (+4.04%); split: +7.49%, -3.45%
SMEM: 2434691 -> 2394829 (-1.64%); split: +2.80%, -4.43%
VClause: 551776 -> 543051 (-1.58%); split: -1.73%, +0.15%
SClause: 1230147 -> 1227637 (-0.20%); split: -1.40%, +1.20%
Copies: 1957640 -> 1963617 (+0.31%); split: -1.11%, +1.41%
Branches: 611747 -> 612504 (+0.12%); split: -0.11%, +0.23%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7903>
 2020-12-22 15:08:40 +01:00
Daniel Schürmann
3f14140f48 aco: allow to schedule SALU/SMEM through exec changes 
Totals from 16794 (12.05% of 139391) affected shaders (NAVI10):
SGPRs: 757760 -> 762048 (+0.57%); split: -0.39%, +0.95%
VGPRs: 402844 -> 402744 (-0.02%); split: -0.04%, +0.02%
CodeSize: 22290900 -> 22285068 (-0.03%); split: -0.06%, +0.04%
MaxWaves: 294163 -> 294222 (+0.02%); split: +0.03%, -0.01%
Instrs: 4190074 -> 4188513 (-0.04%); split: -0.08%, +0.04%
Cycles: 40685028 -> 40678640 (-0.02%); split: -0.03%, +0.02%
VMEM: 7711867 -> 7704315 (-0.10%); split: +0.28%, -0.38%
SMEM: 942472 -> 1007052 (+6.85%); split: +7.15%, -0.30%
VClause: 92990 -> 92974 (-0.02%); split: -0.03%, +0.01%
SClause: 263700 -> 263810 (+0.04%); split: -0.38%, +0.42%
Copies: 277467 -> 276988 (-0.17%); split: -0.37%, +0.20%
Branches: 45899 -> 45896 (-0.01%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7903>
 2020-12-22 15:08:40 +01:00
Timur Kristóf
731f8fc9dd aco: Use program->num_waves as maximum in scheduler. 
The scheduler doesn't take SGPR use into account, which can be
a limiting factor on older GPUs. This patch fixes a CTS test crash
on GFX6.

CC: mesa-stable
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8040>
 2020-12-11 13:32:26 +00:00
Tony Wasserka
34bc9477de aco: Clean up symbol names and comments related to NGG 
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Acked-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7094>
 2020-10-21 09:49:38 +00:00
Tony Wasserka
86c227c10c aco: Use strong typing to model SW<->HW stage mappings 
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Acked-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7094>
 2020-10-21 09:49:38 +00:00
Samuel Pitoiset
97afb2a0a9 aco: remove unused radv_shader.h includes 
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7061>
 2020-10-14 15:09:34 +00:00
Samuel Pitoiset
408195ec53 aco: remove useless occurences of radv_nir_compiler_options 
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7061>
 2020-10-14 15:09:34 +00:00
Rhys Perry
7b4c24eb67 aco: don't move memory accesses to before control barriers 
Fixes random failures of dEQP-VK.image.qualifiers.volatile.cube_array.r32i
and similar tests on Vega.

fossil-db (Navi):
Totals from 6 (0.00% of 135946) affected shaders:
VMEM: 1218 -> 1110 (-8.87%); split: +2.46%, -11.33%
SMEM: 174 -> 189 (+8.62%)
Copies: 84 -> 87 (+3.57%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: cd392a10d0 ('radv/aco,aco: use scoped barriers')
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6174>
 2020-08-11 14:16:00 +01:00
Rhys Perry
07250a92da aco: implement subgroup shader_clock on GFX10.3 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5546>
 2020-08-04 20:39:33 +01:00
Rhys Perry
37988b5b8e aco: fix max_waves_per_simd on Polaris, VegaM and GFX10.3 
fossil-db (Polaris):
Totals from 20263 (14.75% of 137414) affected shaders:
SGPRs: 871407 -> 871679 (+0.03%); split: -0.00%, +0.03%
VGPRs: 513828 -> 550028 (+7.05%); split: -1.68%, +8.72%
CodeSize: 18869680 -> 18828148 (-0.22%); split: -0.23%, +0.01%
MaxWaves: 162012 -> 162030 (+0.01%); split: +0.01%, -0.00%
Instrs: 3629172 -> 3618817 (-0.29%); split: -0.30%, +0.02%
Cycles: 15682244 -> 15638244 (-0.28%); split: -0.30%, +0.02%
VMEM: 10675942 -> 10673344 (-0.02%); split: +0.18%, -0.21%
SMEM: 1209717 -> 1206088 (-0.30%); split: +0.03%, -0.33%
VClause: 81780 -> 81227 (-0.68%); split: -0.73%, +0.06%
SClause: 231724 -> 231561 (-0.07%); split: -0.07%, +0.00%
Copies: 187126 -> 180831 (-3.36%); split: -3.62%, +0.26%
Branches: 26841 -> 26837 (-0.01%); split: -0.03%, +0.01%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5546>
 2020-08-04 20:39:33 +01:00
Rhys Perry
cd392a10d0 radv/aco,aco: use scoped barriers 
fossil-db (Navi):
Totals from 109 (0.08% of 132058) affected shaders:
SGPRs: 5416 -> 5424 (+0.15%)
CodeSize: 460500 -> 460508 (+0.00%); split: -0.07%, +0.07%
Instrs: 87278 -> 87272 (-0.01%); split: -0.09%, +0.09%
Cycles: 2241996 -> 2241852 (-0.01%); split: -0.04%, +0.04%
VMEM: 33868 -> 35539 (+4.93%); split: +5.14%, -0.20%
SMEM: 7183 -> 7184 (+0.01%); split: +0.36%, -0.35%
VClause: 1857 -> 1882 (+1.35%)
SClause: 2052 -> 2055 (+0.15%); split: -0.05%, +0.19%
Copies: 6377 -> 6380 (+0.05%); split: -0.02%, +0.06%
PreSGPRs: 3391 -> 3392 (+0.03%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4905>
 2020-07-28 16:56:34 +00:00
Rhys Perry
d1f992f3c2 aco: rework barriers and replace can_reorder 
fossil-db (Navi):
Totals from 273 (0.21% of 132058) affected shaders:
CodeSize: 937472 -> 936556 (-0.10%)
Instrs: 158874 -> 158648 (-0.14%)
Cycles: 13563516 -> 13562612 (-0.01%)
VMEM: 85246 -> 85244 (-0.00%)
SMEM: 21407 -> 21310 (-0.45%); split: +0.05%, -0.50%
VClause: 9321 -> 9317 (-0.04%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4905>
 2020-07-28 16:56:34 +00:00
Rhys Perry
1bbb64f300 aco: add missing add_to_hazard_query 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4905>
 2020-07-28 16:56:34 +00:00
Rhys Perry
1f2fd9c62e aco: don't reorder barriers in the scheduler 
Unless we're reordering it around a barrier of the same type

No shader-db changes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4880>
 2020-05-28 10:34:03 +00:00
Timur Kristóf
0c0691d43e aco: Const correctness for get_barrier_interaction. 
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4536>
 2020-04-24 17:58:57 +00:00
Timur Kristóf
e4da482d9e aco/ngg: Schedule position exports of NGG VS/TES. 
Similarly to the HW VS stage, the HW NGG GS stage also
benefits from executing these exports as early as possible.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>
 2020-04-07 11:29:35 +00:00
Timur Kristóf
35e58314d8 aco: Treat s_setprio as a scheduling barrier. 
We want to execute instructions after s_setprio in the given
priority, so we must prevent the scheduler from scheduling beyond
s_setprio, otherwise some instructions could be executed in a
different priority.

Rename hazard_fail_memtime to hazard_fail_unreorderable and include
s_setprio in the list of unreorderable opcodes.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>
 2020-04-07 11:29:35 +00:00
Rhys Perry
c51348bd9b aco: move some register demand helpers into aco_live_var_analysis.cpp 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3914>
 2020-03-16 16:09:02 +00:00
Rhys Perry
625d8705f0 aco: don't stop scheduling at exports 
This allows us to move v_cvt_pkrtz_f16_f32 instructions upwards, improving
schedules and (somewhat unintentionally) moving the exports slightly
closer together.

Totals from affected shaders:
SGPRS: 1030224 -> 1030248 (0.00 %)
VGPRS: 794080 -> 794392 (0.04 %)
Spilled SGPRs: 127117 -> 127117 (0.00 %)
Spilled VGPRs: 0 -> 0 (0.00 %)
Scratch size: 0 -> 0 (0.00 %) dwords per thread
Code Size: 89028152 -> 89032312 (0.00 %) bytes
LDS: 0 -> 0 (0.00 %) blocks
Max Waves: 65252 -> 65219 (-0.05 %)
SMEM score: 843808.00 -> 843918.00 (0.01 %)
VMEM score: 5331687.00 -> 5397802.00 (1.24 %)
SMEM clauses: 567659 -> 567655 (-0.00 %)
VMEM clauses: 290715 -> 290716 (0.00 %)
Instructions: 17143219 -> 17144259 (0.01 %)
Cycles: 1098442808 -> 1098446968 (0.00 %)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3776>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3776>
 2020-03-13 14:04:50 +00:00
Rhys Perry
6b4c31f814 aco: allow barriers to be skipped during scheduling 
Much better scheduling apparently in 160 shaders

Totals from affected shaders:
SGPRS: 6272 -> 6344 (1.15 %)
VGPRS: 4832 -> 4844 (0.25 %)
Spilled SGPRs: 0 -> 0 (0.00 %)
Spilled VGPRs: 0 -> 0 (0.00 %)
Scratch size: 0 -> 0 (0.00 %) dwords per thread
Code Size: 467192 -> 467428 (0.05 %) bytes
LDS: 459 -> 459 (0.00 %) blocks
Max Waves: 1407 -> 1409 (0.14 %)
SMEM score: 9309.00 -> 11216.00 (20.49 %)
VMEM score: 26679.00 -> 33652.00 (26.14 %)
SMEM clauses: 1817 -> 1776 (-2.26 %)
VMEM clauses: 2286 -> 2288 (0.09 %)
Instructions: 86537 -> 86596 (0.07 %)
Cycles: 676260 -> 676568 (0.05 %)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3776>
 2020-03-13 14:04:50 +00:00
Rhys Perry
928ac97875 aco: add helpers for ensuring correct ordering while scheduling 
Pipeline-db changes in 721 shaders.

Totals from affected shaders:
SGPRS: 42336 -> 42656 (0.76 %)
VGPRS: 38368 -> 38636 (0.70 %)
Spilled SGPRs: 11967 -> 11967 (0.00 %)
Spilled VGPRs: 0 -> 0 (0.00 %)
Scratch size: 0 -> 0 (0.00 %) dwords per thread
Code Size: 5268088 -> 5269840 (0.03 %) bytes
LDS: 1069 -> 1069 (0.00 %) blocks
Max Waves: 4473 -> 4447 (-0.58 %)
SMEM score: 41155.00 -> 41826.00 (1.63 %)
VMEM score: 146339.00 -> 147471.00 (0.77 %)
SMEM clauses: 24434 -> 24535 (0.41 %)
VMEM clauses: 16637 -> 16592 (-0.27 %)
Instructions: 996037 -> 996388 (0.04 %)
Cycles: 76476112 -> 75281416 (-1.56 %)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3776>
 2020-03-13 14:04:50 +00:00
Rhys Perry
2cd760847a aco: add helpers for moving instructions for scheduling 
No pipeline-db changes

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3776>
 2020-03-13 14:04:50 +00:00
Daniel Schürmann
71440ba0f5 aco: reorder VMEM operands in ACO IR 
For all VMEM instructions, the resource constant is now
in operands[0]. For MIMG instructions, the sampler shares
operands[1] with write data in case this instruction writes memory.
Moving the VADDR to be the last operand for MIMG is the first step to
support Navi NSA encoding.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3602>
 2020-01-29 18:45:23 +00:00
Rhys Perry
40bb81c9dd radv/aco,aco: implement GS on GFX9+ 
v2: implement GFX10
v3: rebase
v7: rebase after shader args MR
v8: fix gs_vtx_offset usage on GFX9/GFX10
v8: use unreachable() instead of printing intrinsic
v8: rename output_state to ge_output_state
v8: fix formatting around nir_foreach_variable()
v8: rename some helpers in the scheduler
v8: rename p_memory_barrier_all to p_memory_barrier_common
v8: fix assertion comparing ctx.stage against vertex_geometry_gs

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2421>
 2020-01-24 13:35:07 +00:00
Rhys Perry
b5c9688516 aco: limit register usage for large work groups 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
 2020-01-10 12:10:37 +00:00
Rhys Perry
afe1a8ff5b aco: fix vgpr alloc granule with wave32 
We still need to increase the number of physical vgprs

Totals from affected shaders:
SGPRS: 671976 -> 675288 (0.49 %)
VGPRS: 550112 -> 562596 (2.27 %)
Spilled SGPRs: 0 -> 0 (0.00 %)
Spilled VGPRs: 0 -> 0 (0.00 %)
Code Size: 27621660 -> 27606532 (-0.05 %) bytes
Max Waves: 81083 -> 87833 (8.32 %)
Instructions: 5391560 -> 5389031 (-0.05 %)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
 2019-12-21 12:38:42 +01:00
Rhys Perry
389ee819c0 aco: improve FLAT/GLOBAL scheduling 
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
 2019-11-29 17:46:02 +00:00
Daniel Schürmann
efe737fc4f aco: fix accidential reordering of instructions when scheduling 
Fixes: 8678699918 "aco: implement VGPR spilling"

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
 2019-11-04 20:14:14 +01:00
Rhys Perry
8235bc6411 aco: try to group together VMEM loads of the same resource 
v2: remove accidental shaderInt16 change
v2: simplify can_move_down initialization
v2: simplify VMEM_CLAUSE_MAX_GRAB_DIST

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
 2019-10-30 17:23:49 +01:00
Daniel Schürmann
8b5aee78cc aco: don't schedule instructions through depending VMEM instructions 
Previously, the scheduler tried to move up instructions from below depending
VMEM instructions only to move them down again when scheduling the VMEM
instruction.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
 2019-10-30 16:12:10 +00:00