Commit graph

6122 commits

Author SHA1 Message Date
Illia Abernikhin
aa4ac5ff8b utils: Merge util/debug.* into util/u_debug.* and remove util/debug.*
Rename env_var_as_unsigned() -> debug_get_num_option(), because duplicate
Rename env_var_as_bool() -> debug_get_bool_option(), because duplicate

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7177

Signed-off-by: Illia Abernikhin <illia.abernikhin@globallogic.com>
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19336>
2022-11-02 07:25:39 +00:00
Kenneth Graunke
fde99747e9 nir: Drop infer_non_readable option for nir_opt_access()
Everybody sets it to true now, and the only reason for the option to
exist was to work around a bug that's now been fixed.

Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19162>
2022-11-02 03:42:04 +00:00
Samuel Pitoiset
53a8dd1d42 radv: move GDS counters after reserved GDS offsets for streamout
Otherwise, queries might return invalid data because they used
the same offsets as NGG streamout.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19442>
2022-11-01 19:47:07 +00:00
Samuel Pitoiset
0bccf8f773 radv: use defines instead of magic values for GDS counters offset
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19442>
2022-11-01 19:47:07 +00:00
Samuel Pitoiset
d2f3e01f61 radv/amdgpu: add a kernel GDS management workaround for -ENOMEM
The kernel keeps returning -ENOMEM if multiple processes allocate GDS,
this always happen while running VKCTS. This solution is loosely based
on RadeonSI, except that it includes a timeout of 1s to exit the loop.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19440>
2022-11-01 18:23:27 +00:00
Samuel Pitoiset
7e3be0b3ce radv: assign IO var locations for VS/TES with NGG streamout
NGG streamout lowering requires nir_shader::num_outputs to be set to
the total number of outputs in order to compute the pervertex LDS size
correctly. This is wasting LDS memory but it's currently the only viable
solution.

This fixes a bunch of dEQP-VK.transform_feedback.* failures.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19436>
2022-11-01 16:47:13 +00:00
Samuel Pitoiset
c2ff7c3eb2 radv: enable VK_EXT_transform_feedback with NGG streamout on GFX11
Untested but it mostly work on GFX10.3.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19435>
2022-11-01 11:28:45 +01:00
Samuel Pitoiset
75faaac895 radv: allow to enable NGG streamout with RADV_PERFTEST=ngg_streamout
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19435>
2022-11-01 11:28:43 +01:00
Samuel Pitoiset
4621c4936b radv: enable all possible features with NGG streamout
Tested by forcing NGG streamout on GFX10.3.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19435>
2022-11-01 11:28:41 +01:00
Samuel Pitoiset
d467ff7b0e radv: set the correct buffer size for NGG streamout
If the size is passed through CmdBindTransformFeedback() uses that.

This partially fixes dEQP-VK.transform_feedback.simple.multiquery_1
by reporting the correct number of primitives written (the computation
is based on the buffer size). There is still a bug around GDS offsets
that will be fixed later.

Tested on GFX10.3 by forcing NGG streamout.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19433>
2022-11-01 10:41:36 +01:00
Samuel Pitoiset
76555a4777 radv: fix primitives generated query with NGG only
According to the AMD registers database, SAMPLE_STREAMOUTSTATS no
longer exists on GFX11. This fixes primitives generated query if only
the NGG path is used. Tested on GFX10.3 by forcing NGG everywhere.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19410>
2022-11-01 07:26:31 +00:00
Samuel Pitoiset
8dafda4a07 radv: suspend/resume primitives generated query separately for legacy/NGG
With NGG only, only the GDS query counter will be incremented.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19410>
2022-11-01 07:26:31 +00:00
Samuel Pitoiset
e83ec595df radv: use 32-bit GDS counters for primitives generated query with NGG
Do not need to use 64-bit operations.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19410>
2022-11-01 07:26:31 +00:00
Samuel Pitoiset
1e925ce565 radv: reword an incorrect comment about primitives generated query
This is not the XFB counter.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19410>
2022-11-01 07:26:31 +00:00
Konstantin Seurer
1c7d208302 radv/rt: Set vars.arg in the traversal shader
Fixes: f4270b7 ("radv/rt: create traversal shader independent from main shader")
Closes: #7591
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19402>
2022-10-31 17:25:55 +00:00
Rhys Perry
b7ea47ede6 radv,aco: don't use lower_to_fragment_fetch_amd on GFX11+
FMask doesn't exist on GFX11. Have txf_ms take the fragment_fetch_amd
path.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19375>
2022-10-31 16:26:30 +00:00
Rhys Perry
b9a3d8b0ab radv/gfx11: don't create texop_samples_identical
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19375>
2022-10-31 16:26:30 +00:00
Rhys Perry
d31e5c8efc radv/gfx11: don't create fmask copy/expand pipelines
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19375>
2022-10-31 16:26:30 +00:00
Rhys Perry
140cefe95a ac/nir: lower gfx11 vertex parameter exports
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19228>
2022-10-31 14:33:43 +00:00
Rhys Perry
e6d26cb288 nir,ac/nir,aco,radv: replace has_input_*_amd with more general intrinsics
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19228>
2022-10-31 14:33:43 +00:00
Samuel Pitoiset
9f692f5aa6 radv: do not unconditionally disable NGG streamout lowering in NIR
This is still always disabled because use_ngg_streamout is FALSE
but it will be turned on at some point.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19317>
2022-10-31 13:48:39 +00:00
Samuel Pitoiset
d653ca4c52 radv: lower NIR intrinsics for NGG streamout
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19317>
2022-10-31 13:48:39 +00:00
Samuel Pitoiset
3bafe6e0bc radv: call nir_io_add_intrinsic_xfb_info() after IO lowering
This is needed for NGG streamout which gets the XFB info directly
from intrinsics.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19317>
2022-10-31 13:48:39 +00:00
Samuel Pitoiset
ed865440db radv/llvm: prevent emitting streamout outputs for NGG
They are directly emitted from NIR.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19317>
2022-10-31 13:48:39 +00:00
Bas Nieuwenhuizen
5d04064642 radv: Handle attribute ring intrinsic correctly with LLVM.
Again, if we don't set progress to false we get fun stuff.

Fixes: 8bf1aa1b76 ("radv: add lowering for nir_intrinsic_load_ring_attr_{offset}_amd")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19404>
2022-10-31 13:17:04 +00:00
Bas Nieuwenhuizen
45ff58cfd1 radv: Handle GSVS ring intrinsic correctly with LLVM.
If we don't set progress to false we get a mess as a replacement is
still attempted.

Fixes: 382831c986 ("radv,nir: add intrinsics for streamout and GS copy shaders")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19404>
2022-10-31 13:17:04 +00:00
Bas Nieuwenhuizen
ec9d71498e radv: Use correct types for loading the rings with LLVM.
Ring descriptors are v4i32, not i8.

Fixes: cb117cdc96 ("radv/llvm: use ac_build_gep0_type to get args types")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19404>
2022-10-31 13:17:04 +00:00
Samuel Pitoiset
d4ec3f21cf Revert "radv: add a pointer to radv_shader_binary in radv_shader"
This is actually not necessary because we compile and upload binaries
directly from libraries with GPL. This introduced random double free
crashes because binaries were potentially freed by concurrent threads.

Root cause found by Ishi.

This reverts commit f8d887527a.

Reviewed-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19383>
2022-10-31 12:16:38 +00:00
Samuel Pitoiset
25e311e9d3 radv: implement transform feedback queries with NGG streamout
The control bit is written to the upper bits because GDS counters
are 32-bits only, this allows to re-use the existing query shader.

Tested on GFX10.3.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19325>
2022-10-31 08:22:29 +00:00
Bas Nieuwenhuizen
78519987b9 radv: Speculatively tune RT pipelines for GFX11.
With ACO not supporting VOPD and the high number of SALU instructions,
we're likely better off using wave64 until we can actually benchmark
this and fix these issues.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19288>
2022-10-31 02:39:34 +00:00
Bas Nieuwenhuizen
9369b40725 radv: Use PLOC for BVH building
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Bas Nieuwenhuizen
271865373e radv: Add PLOC shader
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Friedrich Vock
0c0f179037 radv: Add global sync utilities
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Friedrich Vock
608fa1bd25 radv/rt: Track number of inactive leaf nodes
To avoid emitting nodes with only invalid children in PLOC.

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Friedrich Vock
f502b3aab3 radv/rt: Dispatch internal converter indirectly
Preparation for using the converter with PLOC.

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Friedrich Vock
49c0995918 radv/rt: Fix internal converter synchronization
Fixes: e83e4faf ("radv: Only emit parents from parents that actually end up in the tree.")
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Friedrich Vock
fa578f280e radv: Add radv_indirect_unaligned_dispatch
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Friedrich Vock
030a1f6843 radv: Use a struct for AABBs
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Bas Nieuwenhuizen
ccf0a69e05 radv: Make the number of internal nodes be written on the GPU.
Opens the door of algorithms with a variable number of nodes.

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Bas Nieuwenhuizen
0e23df959e radv: Add BVH IR header.
To include GPU state passed between stages but not in a node.

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Friedrich Vock
37525c11d1 radv: Rename emulated float helpers
Use only conversion functions now.

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19292>
2022-10-30 19:48:46 +00:00
Samuel Pitoiset
eae2867122 radv: move nir_opt_idiv_const/nir_lower_idiv after NGG lowering
NGG streamout lowering creates some idiv instructions that need to be
lowered.

No fossil-db results because it's currently broken.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19364>
2022-10-28 18:19:57 +00:00
Samuel Pitoiset
e2fcbd4a37 radv/llvm: fix dual source blending on GFX11
Untested but this should be similar to RadeonSI.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19367>
2022-10-28 17:03:37 +00:00
Samuel Pitoiset
d172fc1fca radv: fix VRS limit when attachmentFragmentShadingRate is disabled
Can be reproduced on GFX10.3 with RADV_DEBUG=nohiz.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19374>
2022-10-28 16:30:29 +00:00
Samuel Pitoiset
c41997f29f radv: fix suspending/resuming pipeline statistics queries with GDS
This probably doesn't fix anything in practice because GDS is only
used for the number of generated primitives by GS and meta operations
don't use GS.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19348>
2022-10-28 08:12:30 +00:00
Alyssa Rosenzweig
941c37c085 nir/lower_idiv: Remove imprecise_32bit_lowering
NIR has two implementations of lower_idiv, keyed on the
imprecise_32bit_lowering flag. This flag is misleading: the results when
setting this flag "imprecise", they're completely wrong for some values.
If a backend has a native implementation of umul_high, the correct path
isn't that much more expensive. If it doesn't, it's substantially slower
for highp integer divison... but in practice, non-constant highp integer
division is pretty rare.

After a painful migration of the tree, this code path has no more users.
Remove it so nobody else gets the bright idea of using it again.

Closes: #6555
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19303>
2022-10-27 19:37:14 +00:00
Rhys Perry
93fb84237f ac/nir: add ac_nir_lower_ngg_options
These signatures were getting ridiculous.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19340>
2022-10-27 13:31:40 +00:00
Daniel Schürmann
c80137fcba radv/rt: overwrite hit args with undef in case of a miss
This helps some variable coalescing.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19188>
2022-10-27 09:45:39 +00:00
Daniel Schürmann
f4270b7659 radv/rt: create traversal shader independent from main shader
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19188>
2022-10-27 09:45:39 +00:00
Iago Toral Quiroga
9deef4cde6 vulkan/runtime: include robustness info when hashing a shader stage
Suggested by Jason Ekstrand.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18883>
2022-10-27 08:17:11 +00:00