Commit graph

6733 commits

Author SHA1 Message Date
Rhys Perry
d3476650f1 nir/load_store_vectorize: set is_store for shared append/consume
So that check_for_aliasing() doesn't ignore them.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Backport-to: 25.1
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36442>
2025-07-31 08:39:10 +00:00
Rhys Perry
eeddf6b9e2 nir/load_store_vectorize: check for interfering shared2 before vectorizing
Only affected shaders in radv_fossils are f1_23.

fossil-db (navi21):
Totals from 3 (0.00% of 79825) affected shaders:
Instrs: 2700 -> 2730 (+1.11%)
CodeSize: 17096 -> 17228 (+0.77%)
Latency: 8424 -> 8726 (+3.58%)
InvThroughput: 3768 -> 3778 (+0.27%); split: -0.05%, +0.32%
Copies: 224 -> 234 (+4.46%)
PreVGPRs: 291 -> 287 (-1.37%)
VALU: 1989 -> 1996 (+0.35%); split: -0.05%, +0.40%

fossil-db (gfx1201):
Totals from 3 (0.00% of 79839) affected shaders:
Instrs: 2862 -> 2908 (+1.61%)
CodeSize: 17868 -> 18064 (+1.10%)
Latency: 7567 -> 7854 (+3.79%)
InvThroughput: 2810 -> 2802 (-0.28%)
Copies: 122 -> 120 (-1.64%)
PreVGPRs: 291 -> 287 (-1.37%)
VALU: 1890 -> 1885 (-0.26%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/13616
Backport-to: 25.1
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36442>
2025-07-31 08:39:09 +00:00
Alyssa Rosenzweig
4f1bafa6d5 nir: drop load_sample_id_no_per_sample
unused now.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36429>
2025-07-30 22:13:23 +00:00
Alyssa Rosenzweig
04be5a8910 nir/lower_system_values: simplify load_helper_invocation lowering
The two backends (ir3, dxil) using the lowering have
info->fs.uses_sample_shading matching set when sample shading is used -- the VK
drivers pass the rasterization state flag into the compiler, while
freedreno and d3d12 have caps->force_persample_interp so the frontend
creates a shader variant with info->fs.uses_sample_shading set.

This means that we can drop the sample-id SHL/AND in the pixel-rate
shading case, which in turn means that drivers don't need to have a
load_sample_id() that doesn't trigger sample-rate shading (which Adreno
doesn't have).

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36429>
2025-07-30 22:13:23 +00:00
Georg Lehmann
e43ef6533b nir/opt_algebraic: remove 8bit roundtrip when vectorizing i2i16(unpack_4x8(a).zw)
Explicit 16bit instructions are nicer to vectorize.

Helps FSR4 on GFX11 marginally.

Foz-DB Navi31:
Totals from 10 out of 14 FSR4 shaders:
Instrs: 59781 -> 58518 (-2.11%)
CodeSize: 413428 -> 404156 (-2.24%)
Latency: 193770 -> 190768 (-1.55%)
InvThroughput: 226274 -> 221628 (-2.05%)
VClause: 796 -> 793 (-0.38%); split: -1.01%, +0.63%
Copies: 3342 -> 3008 (-9.99%); split: -11.01%, +1.02%
PreSGPRs: 312 -> 305 (-2.24%)
VALU: 51448 -> 50213 (-2.40%)
SALU: 1074 -> 1048 (-2.42%)
VOPD: 1783 -> 1718 (-3.65%); split: +0.95%, -4.60%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36117>
2025-07-30 07:25:51 +00:00
Georg Lehmann
92cf02ee23 nir/search: support swizzles on expressions in replacement patterns
Before this wasn't a compile time error, but it also didn't do anything.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36117>
2025-07-30 07:25:51 +00:00
Marek Olšák
d61edf079b nir: add nir_move_only_convergent/divergent
This will be needed by nir_opt_move_reorder_loads, which will use
the move flags.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:53 -04:00
Marek Olšák
35bbc8405b nir: add more nir_move_options
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:51 -04:00
Marek Olšák
44d78c4451 nir: handle load_input_vertex in nir_can_move_instr
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:49 -04:00
Marek Olšák
8d3e76c250 nir: split nir_move_load_frag_coord from nir_move_load_input
It's a pure system value on AMD, not an input.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:48 -04:00
Marek Olšák
5083769fcb nir: renumber nir_move_options
for future commits

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:46 -04:00
Marek Olšák
8d584586f5 nir: handle can_reorder robustly in nir_can_move_instr
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:44 -04:00
Marek Olšák
c229c93540 nir: change how can_mov_out_of_loop is set for intrinsics in nir_can_move_instr
Set to false first, then set to true when needed.

More intrinsics will set false.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:42 -04:00
Marek Olšák
ad1cfcc841 nir: mark inverse_ballot & is_subgroup_invocation_lt_amd as CAN_REORDER
nir_can_move_instr already ignores that this flag isn't set.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36357>
2025-07-29 16:20:39 -04:00
Marek Olšák
2eea9b968d nir/group_loads: rename to nir_opt_group_loads
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
It's meant to be an optimization pass.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:59 +00:00
Marek Olšák
3bceb7b622 nir/group_loads: don't use pass_flags to store the indirection level
no change in behavior

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:59 +00:00
Marek Olšák
7b9acabe89 nir/group_loads: store our custom instr->index in an array
we'll put more stuff in the new structure

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:58 +00:00
Marek Olšák
821dc611c5 nir/group_loads: use nir_instr_next/prev
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:58 +00:00
Marek Olšák
318a57c57c nir/group_loads: make is_grouped_load use get_load_resource
and unify get_intrinsic_resource & get_uniform_inst_resource into the new
helper get_load_resource.

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:57 +00:00
Marek Olšák
b9f7dc5d26 nir/group_loads: remove mostly duplicated function is_memory_load
is_grouped_load does the same thing

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:57 +00:00
Marek Olšák
ef0a5e1a65 nir/group_loads: invert the return value of can_move to reflect its true meaning
The previous commit handles non-reorderable instructions better, so this is
no longer needed.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:56 +00:00
Marek Olšák
ad8b5325d4 nir/group_loads: group any reorderable intrinsics regardless of barriers
Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:56 +00:00
Marek Olšák
e0d8979dea nir/group_loads: split is_barrier into is_barrier + is_terminate
and handle demote

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:56 +00:00
Marek Olšák
7d9aca2235 nir/group_loads: allow moving loads across instructions without defs
It wouldn't group these:
    load (reordeable)
    store
    load (reordeable)

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:55 +00:00
Marek Olšák
aa732f6f30 nir/group_loads: handle more loads
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36100>
2025-07-29 19:28:55 +00:00
Antonio Ospite
8825bf9b48 nir: fix returning _Bool instead of pointer
When building for C23 the compiler warns about returning a boolean when
a different type is expected instead.

Change the code to return NULL instead of false, fixing the following
error:

-----------------------------------------------------------------------
../src/compiler/nir/nir_lower_io_indirect_loads.c: In function ‘get_load_once_variable’:
../src/compiler/nir/nir_lower_io_indirect_loads.c:87:17: error: incompatible types when returning type ‘_Bool’ but ‘nir_variable **’ {aka ‘struct nir_variable **’} was expected
   87 |          return false;
      |                 ^~~~~
-----------------------------------------------------------------------

Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36323>
2025-07-29 14:07:06 +00:00
Rhys Perry
928c9c618d nir/opt_access: support RT/callable shaders
I don't know if this affects any real application.

No fossil-db changes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35938>
2025-07-28 09:19:01 +00:00
Rhys Perry
a9a1da0264 nir/uub: fix 8/16-bit overflow
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Backport-to: 25.1
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/13552
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/13553
Tested-by: @LingMan
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36372>
2025-07-28 08:46:51 +00:00
Georg Lehmann
845961ab77 nir: remove NIR_PASS_V
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/10409
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36381>
2025-07-25 23:09:56 +00:00
Pierre-Eric Pelloux-Prayer
718fd47dde nir/lower_io: make sure range is not 0
For samplers the type_size() callback can return 0, which triggers
a NIR validation error.
In this case set range to ~0 which means the range is unknown.

Acked-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36263>
2025-07-25 15:47:20 +02:00
Marek Olšák
d9cb26fdf9 nir: handle store_buffer_amd in nir_intrinsic_writes_external_memory
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36099>
2025-07-24 18:41:38 +00:00
Marek Olšák
a15c1e1896 nir: silence a warning in nir_opt_shrink_vectors
gcc assumes the array is a string, so it treats it as having 15 elements
instead of 16.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36099>
2025-07-24 18:41:38 +00:00
Marek Olšák
b35e8a38e7 nir/opt_move_to_top: check can_reorder & can_speculate
No effect now, but may be needed in the future.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36099>
2025-07-24 18:41:38 +00:00
Marek Olšák
6a85761c4c nir/divergence_analysis: simplify nir_vertex_divergence_analysis
by reusing nir_divergence_analysis_impl.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36099>
2025-07-24 18:41:38 +00:00
Marek Olšák
09e607c385 nir: add access to load_smem_amd (for ACCESS_CAN_SPECULATE)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36099>
2025-07-24 18:41:38 +00:00
Marek Olšák
688a639117 nir: add nir_tex_instr::can_speculate
Set to true everywhere except:
- spirv_to_nir used by Vulkan
- bindless handles in GLSL
- some internal shaders and driver-specific code

Acked-by: Job Noorman <job@noorman.info>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36099>
2025-07-24 18:41:38 +00:00
Marek Olšák
b4afe848a1 nir: add nir_instr_can_speculate helper (for LICM)
Acked-by: Job Noorman <job@noorman.info>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36099>
2025-07-24 18:41:37 +00:00
Alyssa Rosenzweig
8a1a410389 treewide: use SWAP macro
Via Coccinelle patch + manual clean up:

    @@
    identifier temporary, a, b;
    type T;
    @@

    -T temporary = a;
    -a = b;
    -b = temporary;
    +SWAP(a, b);

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36297>
2025-07-23 19:49:47 +00:00
Alyssa Rosenzweig
3f795a2b8d nir/divergence_analysis: handle more AGX
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Reviewed-by: Marek Olšák <maraeo@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36265>
2025-07-23 14:15:57 +00:00
Alyssa Rosenzweig
ebc18de6f5 nir/opt_vectorize_io: allow i/o semantics w/o component
load_uvs_index_agx always implicitly uses component 0.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Reviewed-by: Marek Olšák <maraeo@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36265>
2025-07-23 14:15:57 +00:00
Alyssa Rosenzweig
75f25b35f8 nir: handle frag_coord_z/w intrinsics
so we can gather_info later.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Reviewed-by: Marek Olšák <maraeo@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36265>
2025-07-23 14:15:57 +00:00
Alyssa Rosenzweig
8716012b21 glsl,nir: factor out nir_opt_varyings_bulk
Correctly/optimally using nir_opt_varyings directly is pretty tricky. For GL, we
have all the right logic in the GLSL linker. for VK, we don't want to duplicate
this dance in every driver. Wrap it all up in a nir_opt_varyings_bulk helper
that operates on an entire pipeline of nir_shader's, following the GLSL linker's
logic. This is suitable for Vulkan drivers.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Reviewed-by: Marek Olšák <maraeo@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36265>
2025-07-23 14:15:57 +00:00
Georg Lehmann
fcc9203550 nir/opt_remove_phis: skip unreachable phis
block->imm_dom is NULL for unreachable phis, so the dominance checks would crash.
These blocks should be removed by nir_opt_dead_cf, so don't bother optimizing
them here.

Fixes: 60776f87c3 ("nir/opt_remove_phis: rematerialize constants")
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35935>
2025-07-23 12:24:31 +00:00
Rhys Perry
7ed9fdf85b nir/search: check variable requirements even if it's already seen
Even if it's already seen, the variable might have some unchecked
requirements.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/12320
Backport-to: 25.1
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32837>
2025-07-23 09:43:33 +00:00
Rhys Perry
f45026751f nir/cf: have nir_remove_after_cf_node remove phis at the start too
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Backport-to: 25.1
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35975>
2025-07-23 09:06:36 +00:00
Georg Lehmann
e1be7213e4 nir/schedule: return progress and fix metadata
Tracking if we changed the shader is somewhat complicated, so assume
we always do.

Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36291>
2025-07-23 06:47:58 +00:00
Alyssa Rosenzweig
e0b0f7e73c nir: add ALU reassocation pass
See the comment at the top file :-)

The ideas in this pass are based on LLVM. The implementation itself is from
scratch because have you /tried/ to read that thing?

Because LLVM and therefore prop drivers do these optimizations, this should help
narrow Mesa's performance gap with the blobs.

This probably needs some tuning for best results on other ISAs, but the stats
for AGX speak for themselves, see next commits.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Mel Henning <mhenning@darkrefraction.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36147>
2025-07-22 23:17:01 +00:00
Alyssa Rosenzweig
22b37c16c8 nir: add nir_alu_src_rewrite_scalar helper
this is a little annoying.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mel Henning <mhenning@darkrefraction.com>
Acked-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36147>
2025-07-22 23:17:01 +00:00
Ian Romanick
477c0cbbf9 nir/print: Don't segfault checking has_debug_info
For example, divergence analysis can call nir_print_instr with an
instruction that doesn't have a block set. When that happens,
print_state::shader will be NULL.

I stumbled on this while testing !36147.

v2: Use nir_instr::has_debug_info instead. Suggested by Konstantin.

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Fixes: ce0f30b230 ("nir: Add variable debug info to instructions")
Fixes: 3aeab4ce40 ("nir/print: Do not print debug information when gathering it")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36267>
2025-07-22 17:41:38 +00:00
Alyssa Rosenzweig
5403a52e4d nir: add vbo_stride_agx
for static VS input + dynamic strides.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36271>
2025-07-22 11:21:50 +00:00