This helps cut down URB messages on tessellation and mesh shaders
significantly. fossil-db results on Battlemage:
Instrs: 505172392 -> 505207187 (+0.01%); split: -0.00%, +0.01%
Send messages: 23678197 -> 23656126 (-0.09%); split: -0.09%, +0.00%
Cycle count: 63150470088 -> 63147482640 (-0.00%); split: -0.01%, +0.00%
Spill count: 576554 -> 576616 (+0.01%)
Fill count: 545304 -> 545413 (+0.02%)
Max live registers: 141099192 -> 141150675 (+0.04%); split: -0.00%, +0.04%
Max dispatch width: 39856192 -> 39856208 (+0.00%)
Totals from 4231 (0.27% of 1583648) affected shaders:
Instrs: 1620161 -> 1654956 (+2.15%); split: -0.25%, +2.40%
Send messages: 128652 -> 106581 (-17.16%); split: -17.18%, +0.03%
Cycle count: 24650700 -> 21663252 (-12.12%); split: -12.82%, +0.70%
Spill count: 378 -> 440 (+16.40%)
Fill count: 1308 -> 1417 (+8.33%)
Max live registers: 364676 -> 416159 (+14.12%); split: -0.24%, +14.36%
Max dispatch width: 67952 -> 67968 (+0.02%)
There are several reasons we didn't go with nir_opt_vectorize_io:
1. nir_opt_vectorize_io appears to work on the slot location level.
We want to be able to vectorize based on the URB offsets, especially
for cases like point size, layer, and viewport which have different
VARYING_SLOT_* values but live in the same vec4 in a URB entry.
2. We want vec8 stores, and nir_opt_vectorize_io only seems to vectorize
within a single 32-bit vec4. It does handle 8 components, but that's
only for packing 16-bit values into a 32-bit vec4.
Improves performance of Sascha Willems' tessellation demo by around 4%
on Meteorlake.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
Both the URB Global Offset and Per-Slot Offsets are specified to be
unsigned numbers. The URB Global Offset is only 11 bits, and so is
limited to be between [0, 2047]. While the per-slot offsets are
given as U32 values, it would appear that adding the two offsets
does not handle 32-bit overflow/unsigned wrap correctly.
This pops up in Piglit's TCS variable-indexing tests, which ends up
performing loads from offset (x - 16) and a base of 18, and at an offset
(x) with a base of 2. These should be equivalent, but when x <= 15, the
per-slot offset calculated in the shader is negative (0xfffffff[0-f])
and adding the base of 18 is not wrapping around correctly to [2, 17].
To work around this, avoid using the global offset when the per-slot
offset is present, and just add the two in the shader where unsigned
wrap works correctly.
Tigerlake and later don't seem to have this issue.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
We were checking for 0xf which is fine for vec4, but vec8 gets 0xff.
Either way, nothing is writemasked, so we can skip sending the mask.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
By default, load/store vectorization uses nir_round_up_components()
to round up loads and possibly writemasked stores to the next valid
NIR vector width. However, some backends may not support load/stores
at all sizes. For example, older Intel supports only power-of-two
vector widths. Newer Intel also supports vec2 and vec3, but not
vec5/6/7. By providing a callback, backends can request promotion
to their next supported memory load/store vector width.
The existing "should we vectorize?" callback should continue to return
false for unsupported vector widths (i.e. beyond the maximum supported).
With this new callback, they do not need to say "no" to vectorization
that would normally produce an unsupported count (e.g. vec5/6/7) but
instead request that the component count be rounded up appropriately.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
This adds a new option, round_up_store_components, which rounds up the
number of components for stores that support writemasking to the next
valid vector size. For example, vec4+vec2 stores would round up from
6 components (which wouldn't be supported) to a full supportable vec8
store, relying on writemasking to ensure the correct pieces are written.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
URB intrinsics are simply memory load/stores to a special memory region,
so it's pretty reasonable to handle these in the memory vectorizer. We
treat emit_vertex_* intrinsics as a barrier for shader outputs.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Acked-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
This makes it easier for NIR passes to distinguish between inputs and
outputs without having to reason about which URB handle source was
passed to the intrinsic. It probably also makes it a bit easier for
humans to read the NIR too.
v2: Don't add memory mode to store intrinsics. It's always output.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39250>
Tested on PTL, fixes various copy_and_blit tests that utilize compute
after ab9d3528dc that exposed this to them.
Fixes: ab9d3528dc ("anv: fix queue check in anv_blorp_execute_on_companion on xe3")
Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39548>
MHW has a long-running shader compile step on first
launch that is significantly sped up by disabling
Link Time Optimization in the ANV driver.
Shader compile times with LTO disabled are 50% of
baseline measurements and the benchmark shows no
stastically significant change to performance
(tested on LNL-M OOB)
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39544>
HiZ must only be cleared when the full HiZ workaround is enabled. This
means that the previous slow clear draw would disable HiZ because it
hits the conditions (ie. depth/stencil enable and depth writes enabled).
So, the draw and the dispatch can run in parallel by moving the barrier
earlier.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39433>
Needs to consider the base offset, otherwise it's resolving to the
first 3D slice.
Fixes very recent VKCTS coverage dEQP-VK.pipeline.*.multisample.m10_resolve.*.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39393>
In this case, do the wrapping logic on our end and normalize right away
to host time domain.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38770>
Deal with VRR vs FRR as well.
Loosely based on earlier work by Keith Packard and Emma Anholt
(MR 38472 for reference).
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38770>
Looks like the "new" page flip handler was in 2.4.78 in 2017. Mesa
requires at least 2.4.109, so we can retire this.
Reviewed-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38770>
This is mostly provided for convenience, but it's not implementable by
applications when we're using blit queues for PRIME, so it's quite useful
to have.
This is reworked from previous GOOGLE_display_timing
MRs by Keith Packard and Emma Anholt.
See MR 38472 for reference.
Rather than exposing PRESENT_STAGE_LOCAL, we expose all timestamps in
one unified domain to simplify the implementation.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38770>
Only weakness right now is that we cannot implement VRR vs FRR query reliably.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Reviewed-by: Derek Foreman <derek.foreman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38770>
Since this does most of the work to determine the right aux usage for
a depth texture, turn it into a helper that returns that aux usage in
order to avoid duplication of logic between it and its callers.
Suggested-by: Nanley Chery <nanley.g.chery@intel.com>
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
v2: Add additional AUX state transition test-cases for HIZ_CCS (Nanley).
v3: Assume partial resolve is equivalent to full resolve on legacy HiZ
surfaces during isl_aux_state_transition_aux_op() instead of
asserting (Nanley).
v4: Move some tests into different group, add more MCS tests (Nanley).
Acked-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
This appears to be needed to guarantee that a resolved depth surface
has no remaining fast-cleared blocks on DG2 as well as MTL. After
this series this should no longer be hit in practice since we'll be
doing partial resolves in most cases, but it seems sensible to keep
and correct the workaround for our peace of mind to make sure that
full resolves are truly resolving the main surface.
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
This appears to be needed to guarantee that a resolved depth surface
has no remaining fast-cleared blocks on DG2 as well as MTL. After
this series this should no longer be hit in practice since we'll be
doing partial resolves in most cases, but it seems sensible to keep
and correct the workaround for our peace of mind to make sure that
full resolves are truly resolving the main surface.
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
Issue a partial resolve instead of a full resolve from
transition_depth_buffer() when the final usage requires the
CCS-compressed surface to provide a complete representation of the
image.
This significantly improves performance of applications that
frequently interleave depth rendering and sampling on non-WT surfaces
(e.g. MSAA surfaces). Nba2K23-trace-dx11-2160p-ultra improves
performance by about 260% with this on MTL, DG2 shows a similar
benefit.
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
This updates the isl_aux_state transition helpers to consider partial
resolves for HiZ-CCS surfaces, and as a side effect of the update to
isl_aux_prepare_access() partial resolves should be implicitly enabled
in iris now for platforms that support it.
v2: HiZ partial resolves aren't enough to remove cleared blocks unlike
color partial resolves (Nanley).
v3: Treat ISL_AUX_STATE_CLEAR similar to
ISL_AUX_STATE_COMPRESSED_HIER_DEPTH so we can continue using it
after depth buffer fast clears. Drop flagging partial_resolve ==
true for HiZ usages so we don't do the wrong thing while preparing
access of a surface in ISL_AUX_STATE_CLEAR state.
v4: Assume partial resolve is equivalent to full resolve on legacy HiZ
surfaces during isl_aux_state_transition_aux_op() instead of
asserting (Nanley).
Acked-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
v2: Define additional enum BLORP_OP_HIZ_PARTIAL_RESOLVE to track
partial resolves (Nanley).
v3: Add comment regarding fall back to full resolve on Gfx12.0 (Nanley).
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
As long as the surface is in a state with valid AUX state with
identity contents of the HiZ surface (E.g. in
ISL_AUX_STATE_COMPRESSED_CLEAR, ISL_AUX_STATE_COMPRESSED_NO_CLEAR,
ISL_AUX_STATE_RESOLVED or ISL_AUX_STATE_PASS_THROUGH states) we can
keep compression enabled, which works around hardware bugs on MTL and
DG2, and will be helpful to switch to partial resolves in a future
commit.
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
Update anv_layout_to_aux_state() to return the
ISL_AUX_STATE_COMPRESSED_HIER_DEPTH state in cases where we may be
rendering into a HiZ surface in non-WT aux mode, instead of
ISL_AUX_STATE_COMPRESSED_CLEAR.
v2: No need to handle ISL_AUX_STATE_COMPRESSED_HIER_DEPTH in
anv_layout_to_fast_clear_type() since it should never be reached
(Nanley).
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
For transitions to a state that requires the image to be fully defined
by the primary+CCS surface without necessarily requiring a valid
primary we have to perform a resolve if the initial state was
ISL_AUX_STATE_COMPRESSED_HIER_DEPTH, which isn't fully defined by its
primary+CCS surface. This full resolve will be replaced with a more
efficient partial resolve in a future commit, but we have to do this
up front in order to avoid breaking bisectability.
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
We can end up in this situation in cases where the application uses a
layout that allows both rendering and sampling from a depth surface,
since in such cases we will attempt to render with HIZ CCS WT usage as
a side effect of using ISL_AUX_USAGE_HIZ_CCS_WT for all layouts that
allow the image to be sampled from.
Disabling fast clears for that case isn't expected to cause
performance regression since before this series for HiZ CCS non-WT
images transitioning to such a layout we would have issued a full
resolve and used ISL_AUX_USAGE_NONE, which also doesn't support fast
clears.
Multisample depth images should still get fast clears after this
commit in cases where the rendering and sampling is split into
separate render pasess with a layout transition between them that
transitions the image from a W/O layout into a R/W one -- Such
transitions will be handled with a relatively cheap partial resolve in
a subsequent commit.
v2: Add details of additional findings about these hardware issues in
comment.
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
v3: Pass aspect bit consistent with layout to
anv_layout_to_aux_usage() instead of defaulting to
VK_IMAGE_ASPECT_DEPTH_BIT.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
Currently anv_fast_clear_depth_stencil() doesn't know the correct
layout of the depth and stencil images, instead it uses
ANV_IMAGE_LAYOUT_EXPLICIT_AUX to force the base AUX usage of each
plane, which can be inconsistent with the VkImageLayout currently in
use. Plumb the correct depth and stencil layouts.
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
The clear color state has to be allocated since we will be sampling
from non-WT HiZ CCS depth surfaces without disabling compression.
v2: Use isl_aux_usage_has_ccs() instead of open coding (Nanley).
v3: Use stricter condition on Gfx12.0 to avoid allocating buffer
unnecessarily (Nanley).
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31139>
