Commit graph

2571 commits

Author SHA1 Message Date
Samuel Pitoiset
3cd482920a radv,aco: stop using vs_common_out.export_clip_dists
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13028>
2021-09-24 16:27:23 +00:00
Rhys Perry
8cf37fc8a8 aco/tests: add idep_amdgfxregs_h
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: 9bf30c4a5c ("aco/tests: add tests for form_hard_clauses()")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12017>
2021-09-24 11:53:23 +01:00
Christian Gmeiner
3d65cea6ee util/bitset: s/BITSET_SET_RANGE/BITSET_SET_RANGE_INSIDE_WORD
Prep work for the next commit.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11321>
2021-09-21 20:25:31 +00:00
Bas Nieuwenhuizen
8ca54b4d38 radv: Support nir_intrinsic_load_global_constant.
SPIR-V parsing can result in some direct constant usage for shader
records. Lower this early to a global based intrinsic so that it
doesn't interfere with the later 32-bit offset based constants
for scratch usage.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12592>
2021-09-21 01:53:39 +00:00
Bas Nieuwenhuizen
c299968988 aco: Add support for ray launch size.
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12592>
2021-09-21 01:53:39 +00:00
Bas Nieuwenhuizen
817553c052 aco: Implement call scope.
Since we do no repacking yet, just use invocation.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12592>
2021-09-21 01:53:39 +00:00
Bas Nieuwenhuizen
b6be96a2bd radv: Modify load_sbt_amd intrinsic to get the descriptor.
That way we can get the address to the entry, which is needed for
some nir builtins because extra data in the entry can be used as
shader input.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12592>
2021-09-21 01:53:39 +00:00
Timur Kristóf
966cff9cfa aco/isel: Fix emit_vop2_instruction to apply 16/24-bit flags properly.
Previously it used a builder function but didn't use the return
value from that function, so the flags were not applied.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12786>
2021-09-20 12:39:03 +02:00
Timur Kristóf
f2e41eda9e aco: Add ability to optimize v_lshl + v_sub into v_mad_i32_i24.
Also change combine_add_lshl to use check_vop3_operands instead
of its own checks of the operands.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12786>
2021-09-20 12:39:03 +02:00
Filip Gawin
6b5e9352ef aco: cleanup assignment of unique_ptrs
Reviewed-by: Joshua Ashton <joshua@froggi.es>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12903>
2021-09-18 11:09:24 +00:00
Rhys Perry
f24f62f4ad aco/nops: fix handle_raw_hazard_internal when visiting the current block
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12720>
2021-09-17 16:43:00 +00:00
Rhys Perry
a8cc911aaf aco/nops: add State
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12720>
2021-09-17 16:43:00 +00:00
Rhys Perry
bdf7eed045 aco/nops: create handle_raw_hazard_instr helper
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12720>
2021-09-17 16:43:00 +00:00
Rhys Perry
bd07118b56 aco/nops: use up-to-date mask_size
fossil-db (Pitcairn):
Totals from 6 (0.00% of 129702) affected shaders:
CodeSize: 8760 -> 8736 (-0.27%)
Instrs: 1714 -> 1708 (-0.35%)
Latency: 12325 -> 12302 (-0.19%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12720>
2021-09-17 16:43:00 +00:00
Rhys Perry
6ed18749de aco: allow live-range splits of linear vgprs in top-level blocks
Fixes dEQP-VK.ssbo.phys.layout.random.8bit.all_per_block_buffers.46 on
GFX8.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
8d50385bbd aco: implement linear vgpr copies
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
b1e4794f0f aco/tests: add regalloc.scratch_sgpr.create_vector
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
f41200d289 aco/tests: fix finish_ra_test()
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
207b2cd434 aco: find scratch reg for sub-dword psuedo instructions which read sgprs
If there's a sgpr operand before a sub-dword operand, a scratch register
will not be found on GFX6/7.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
f2a48f4eae aco: find a scratch register for sub-dword copies on GFX7 if scc is empty
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
da74e0f012 aco: rewrite print_reg_class()
Make it work for any regclass, and print linear VGPRs differently from
logical ones.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
e2b77e6cd4 aco: add and use RegClass::resize helper
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Rhys Perry
783609a849 aco: add RegClass::is_linear_vgpr helper
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12172>
2021-09-17 14:36:03 +00:00
Daniel Schürmann
eb8ec12b23 aco/ra: Fix potential out-of-bounds array accesses.
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12748>
2021-09-10 19:39:18 +00:00
Timur Kristóf
536580b139 aco: Add some useful info to the README for debugging.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12748>
2021-09-10 19:39:18 +00:00
Rhys Perry
c1e668d5d1 aco/ra: don't use ds_write_b8_d16_hi/ds_write_b16_d16_hi on GFX8
GFX8 doesn't support these opcodes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: c75138ed64 ("aco/ra: refactor subdword definition info")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12721>
2021-09-06 15:10:26 +00:00
Timur Kristóf
268158a758 aco/optimize_postRA: Use iterators instead of operator[] of std::array.
Also add a few more assertions to make sure the registers are
within the bounds of the array.

Cc: mesa-stable
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Reviewed-by: Joshua Ashton <joshua@froggi.es>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12682>
2021-09-03 15:00:55 +00:00
Timur Kristóf
bb956464cb aco: Skip code paths to emit copies when there are no copies.
Found while running with libstdc++ debug mode.
Fixes the following:

Error: attempt to advance a dereferenceable (start-of-sequence)
iterator -1 steps, which falls outside its valid range.

Cc: mesa-stable
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12682>
2021-09-03 15:00:55 +00:00
Timur Kristóf
728ed892df aco: Use Builder reference in emit_copies_block.
Found while running with libstdc++ debug mode.
Fixes the following:

Error: attempt to copy-construct an iterator from a singular iterator.

Cc: mesa-stable
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12682>
2021-09-03 15:00:55 +00:00
Rhys Perry
8037b21573 aco/ra: allow v1b operands with 16-bit instructions
Instruction selection can create these.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: ec1bbfa608 ("aco/ra: refactor subdword operand stride")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12617>
2021-09-03 13:21:28 +00:00
Rhys Perry
2a7fa132be aco: implement udot_4x8/sdot_4x8/udot_2x16/sdot_2x16 opcodes
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12617>
2021-09-03 13:21:28 +00:00
Rhys Perry
e0d232c2fc aco: implement nir_op_pack_32_4x8
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12617>
2021-09-03 13:21:28 +00:00
Rhys Perry
4dd420f76d radv,aco: implement iadd_sat
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12617>
2021-09-03 13:21:28 +00:00
Rhys Perry
54f83d718a aco/spill: add temporary operands of exec phis to next_use_distances_end
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: dfb10e4f4b ("aco/spill: don't count phis as variable access")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12702>
2021-09-03 14:01:27 +01:00
Rhys Perry
f241bd3749 aco: don't coalesce constant copies into non-power-of-two sizes
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12702>
2021-09-03 14:01:27 +01:00
Daniel Schürmann
8bd7e2392b aco: preserve subdword RC when lowering p_insert/p_extract
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12640>
2021-09-02 20:39:17 +02:00
Daniel Schürmann
73481338fe aco/print_ir: always print SDWA dst & src selections
This way, it becomes more apparent how SDWA behaves.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12640>
2021-09-02 20:39:17 +02:00
Daniel Schürmann
0988f7b9ba aco: remove explicit dst_preserve flag
Instead, we can rely on the fact that subdword definitions
must preserve the unused bits while dword definitions either
pad or sign-extend.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12640>
2021-09-02 20:39:17 +02:00
Daniel Schürmann
9e3ff06c38 aco: rewrite SDWA selector
This commit introduces a new struct SubdwordSel
in order to ease and clean up the usage of SDWA
selections. This includes removing the distinction
between register-allocated and fixed SDWA selections.
Instead, SDWA selections can now also access the high
bits of subdword variables. Alignment and sizes are
validated accordingly. Size, offset and sign_extend
can be evaluated via helper methods.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12640>
2021-09-02 20:39:17 +02:00
Daniel Schürmann
cc4682ed47 aco: fix p_insert lowering with 16bit sources
The previous lowering only wrote a single byte.

Fixes: 2f94353735 ('aco: add p_extract/p_insert')
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12640>
2021-09-02 20:39:17 +02:00
Daniel Schürmann
ef7d840a70 aco/lower_phis: optimize loop exit phis
This optimization works by ensuring that disabled lanes
are zero'd before any merge sequence.

Totals from 6075 (4.05% of 150170) affected shaders: (GFX10.3)
CodeSize: 57913908 -> 57913212 (-0.00%)
Instrs: 11055852 -> 11055678 (-0.00%)
Latency: 438705219 -> 438534652 (-0.04%); split: -0.04%, +0.00%
InvThroughput: 125284101 -> 125251397 (-0.03%); split: -0.03%, +0.00%
Copies: 807388 -> 821035 (+1.69%); split: -0.00%, +1.69%
Branches: 391827 -> 391782 (-0.01%)
PreSGPRs: 574841 -> 574838 (-0.00%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11659>
2021-09-02 16:41:52 +02:00
Daniel Schürmann
207249d2b2 aco/lower_phis: propagate constants before emitting merge code
This generalizes a previous optimization.

Totals from 521 (0.35% of 150170) affected shaders: (GFX10.3)
CodeSize: 1680348 -> 1678884 (-0.09%)
Instrs: 307994 -> 307628 (-0.12%)
Latency: 5799845 -> 5792655 (-0.12%)
InvThroughput: 994859 -> 994030 (-0.08%)
Copies: 18992 -> 18767 (-1.18%)
Branches: 10143 -> 10037 (-1.05%)
PreSGPRs: 21904 -> 21853 (-0.23%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11659>
2021-09-02 16:41:52 +02:00
Daniel Schürmann
aed7b7d185 aco/lower_bool_phis: avoid creating trivial phis
For this purpose, get_ssa() is also refactored slightly.

Totals from 4 (0.00% of 150170) affected shaders: (GFX10.3)
CodeSize: 15504 -> 15376 (-0.83%)
Instrs: 2942 -> 2910 (-1.09%)
Latency: 292444 -> 291642 (-0.27%)
InvThroughput: 30842 -> 30770 (-0.23%)
Copies: 164 -> 150 (-8.54%)
Branches: 96 -> 82 (-14.58%)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11659>
2021-09-02 16:41:52 +02:00
Daniel Schürmann
6d8ea54654 aco: refactor lower_phis()
No fossil-db changes.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11659>
2021-09-02 16:41:52 +02:00
Daniel Schürmann
e4c5062fb7 aco: fix init_any_pred_defined() for loop header phis
This includes setting the correct end point of the propagation and
not propagating the incoming values after the loop header.
This patch also changes the propagation to a single iteration for
loop exit phis.

No fossil-db changes.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>

aco: don't propagate incoming value in init_any_pred_defined()

No fossil-db changes.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11659>
2021-09-02 16:41:52 +02:00
Timur Kristóf
9d20cf2732 aco: Fix invalid usage of std::fill with std::array.
In this case std::array doesn't behave like a regular array, therefore
it is NOT okay to index it outside the array, even though std::fill
needs us to do so.

Change the syntax to do the same thing slightly differently,
and add an assertion to make sure the registers are always within
the array's bounds.

Closes: #5289
Fixes: 0e4747d3fb
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12664>
2021-09-01 09:33:28 +00:00
Rhys Perry
33ddbd220f aco: remove DPP when applying constants/literals/sgprs
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12601>
2021-08-31 16:58:20 +00:00
Rhys Perry
7d95f7510f aco/tests: test copy propagation with DPP instructions
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12601>
2021-08-31 16:58:20 +00:00
Rhys Perry
e27946ca11 aco: don't constant propagate to DPP instructions
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12601>
2021-08-31 16:58:20 +00:00
Rhys Perry
9df9fe7dfa aco: include utility in isel
For std::exchange().

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: c1d11bb92c ("aco: Add loop creation helpers.")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/5301
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12614>
2021-08-30 14:28:00 +00:00