Commit graph

21 commits

Author SHA1 Message Date
Faith Ekstrand
26e32417b9 nir: Add an option to make lower_phis_to_regs_block() less clever
Right now it tries to place reg_write instructions as far up the
predecessor chain as possible.  This is useful for a bunch of the passes
that call it since it ensures they don't get placed in dead blocks or in
single successors and things like that.  But it screws up NAK's control
flow lowering so we need the option to turn it off and make the pass
place the reg_write instructions in the most obvious place possible.

Fixes: b013d54e4f ("nak/lower_cf: Flag phis as convergent when possible")
Reviewed-by: Mel Henning <mhenning@darkrefraction.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36914>
2025-08-29 01:24:56 +00:00
Marek Olšák
3aadae22ad nir: make nir_block::predecessors & dom_frontier sets non-malloc'd
We can just place the set structures inside nir_block.

This reduces the number of ralloc calls by 6.7% when compiling Heaven
shaders with radeonsi+ACO using a release build (i.e. not including
nir_validate set allocations, which are also removed).

Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36728>
2025-08-21 06:13:48 +00:00
Alyssa Rosenzweig
bcf1a1c20b treewide: use nir_def_block
Via Coccinelle patch:

    @@
    expression definition;
    @@

    -definition->parent_instr->block
    +nir_def_block(definition)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Marek Olšák <maraeo@gmail.com>
Acked-by: Karol Herbst <kherbst@redhat.com>
Acked-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36489>
2025-08-01 15:34:24 +00:00
Alyssa Rosenzweig
cc6e3b84cb treewide: use nir_def_as_*
Via Coccinelle patch:

    @@
    expression definition;
    @@

    -nir_instr_as_alu(definition->parent_instr)
    +nir_def_as_alu(definition)

    @@
    expression definition;
    @@

    -nir_instr_as_intrinsic(definition->parent_instr)
    +nir_def_as_intrinsic(definition)

    @@
    expression definition;
    @@

    -nir_instr_as_phi(definition->parent_instr)
    +nir_def_as_phi(definition)

    @@
    expression definition;
    @@

    -nir_instr_as_load_const(definition->parent_instr)
    +nir_def_as_load_const(definition)

    @@
    expression definition;
    @@

    -nir_instr_as_deref(definition->parent_instr)
    +nir_def_as_deref(definition)

    @@
    expression definition;
    @@

    -nir_instr_as_tex(definition->parent_instr)
    +nir_def_as_tex(definition)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Marek Olšák <maraeo@gmail.com>
Acked-by: Karol Herbst <kherbst@redhat.com>
Acked-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36489>
2025-08-01 15:34:24 +00:00
Antonio Ospite
ddf2aa3a4d build: avoid redefining unreachable() which is standard in C23
In the C23 standard unreachable() is now a predefined function-like
macro in <stddef.h>

See https://android.googlesource.com/platform/bionic/+/HEAD/docs/c23.md#is-now-a-predefined-function_like-macro-in

And this causes build errors when building for C23:

-----------------------------------------------------------------------
In file included from ../src/util/log.h:30,
                 from ../src/util/log.c:30:
../src/util/macros.h:123:9: warning: "unreachable" redefined
  123 | #define unreachable(str)    \
      |         ^~~~~~~~~~~
In file included from ../src/util/macros.h:31:
/usr/lib/gcc/x86_64-linux-gnu/14/include/stddef.h:456:9: note: this is the location of the previous definition
  456 | #define unreachable() (__builtin_unreachable ())
      |         ^~~~~~~~~~~
-----------------------------------------------------------------------

So don't redefine it with the same name, but use the name UNREACHABLE()
to also signify it's a macro.

Using a different name also makes sense because the behavior of the
macro was extending the one of __builtin_unreachable() anyway, and it
also had a different signature, accepting one argument, compared to the
standard unreachable() with no arguments.

This change improves the chances of building mesa with the C23 standard,
which for instance is the default in recent AOSP versions.

All the instances of the macro, including the definition, were updated
with the following command line:

  git grep -l '[^_]unreachable(' -- "src/**" | sort | uniq | \
  while read file; \
  do \
    sed -e 's/\([^_]\)unreachable(/\1UNREACHABLE(/g' -i "$file"; \
  done && \
  sed -e 's/#undef unreachable/#undef UNREACHABLE/g' -i src/intel/isl/isl_aux_info.c

Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36437>
2025-07-31 17:49:42 +00:00
Alyssa Rosenzweig
9a58a8257e treewide: Switch to nir_progress
Via the Coccinelle patch at the end of the commit message, followed by

sed -ie 's/progress = progress | /progress |=/g' $(git grep -l 'progress = prog')
ninja -C ~/mesa/build clang-format
cd ~/mesa/src/compiler/nir && clang-format -i *.c
agxfmt

    @@
    identifier prog;
    expression impl, metadata;
    @@

    -if (prog) {
    -nir_metadata_preserve(impl, metadata);
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    -return prog;
    +return nir_progress(prog, impl, metadata);

    @@
    expression prog_expr, impl, metadata;
    @@

    -if (prog_expr) {
    -nir_metadata_preserve(impl, metadata);
    -return true;
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -return false;
    -}
    +bool progress = prog_expr;
    +return nir_progress(progress, impl, metadata);

    @@
    identifier prog;
    expression impl, metadata;
    @@

    -nir_metadata_preserve(impl, prog ? (metadata) : nir_metadata_all);
    -return prog;
    +return nir_progress(prog, impl, metadata);

    @@
    identifier prog;
    expression impl, metadata;
    @@

    -nir_metadata_preserve(impl, prog ? (metadata) : nir_metadata_all);
    +nir_progress(prog, impl, metadata);

    @@
    expression impl, metadata;
    @@

    -nir_metadata_preserve(impl, metadata);
    -return true;
    +return nir_progress(true, impl, metadata);

    @@
    expression impl;
    @@

    -nir_metadata_preserve(impl, nir_metadata_all);
    -return false;
    +return nir_no_progress(impl);

    @@
    identifier other_prog, prog;
    expression impl, metadata;
    @@

    -if (prog) {
    -nir_metadata_preserve(impl, metadata);
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    -other_prog |= prog;
    +other_prog = other_prog | nir_progress(prog, impl, metadata);

    @@
    identifier prog;
    expression impl, metadata;
    @@

    -if (prog) {
    -nir_metadata_preserve(impl, metadata);
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    +nir_progress(prog, impl, metadata);

    @@
    identifier other_prog, prog;
    expression impl, metadata;
    @@

    -if (prog) {
    -nir_metadata_preserve(impl, metadata);
    -other_prog = true;
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    +other_prog = other_prog | nir_progress(prog, impl, metadata);

    @@
    expression prog_expr, impl, metadata;
    identifier prog;
    @@

    -if (prog_expr) {
    -nir_metadata_preserve(impl, metadata);
    -prog = true;
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    +bool impl_progress = prog_expr;
    +prog = prog | nir_progress(impl_progress, impl, metadata);

    @@
    identifier other_prog, prog;
    expression impl, metadata;
    @@

    -if (prog) {
    -other_prog = true;
    -nir_metadata_preserve(impl, metadata);
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    +other_prog = other_prog | nir_progress(prog, impl, metadata);

    @@
    expression prog_expr, impl, metadata;
    identifier prog;
    @@

    -if (prog_expr) {
    -prog = true;
    -nir_metadata_preserve(impl, metadata);
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    +bool impl_progress = prog_expr;
    +prog = prog | nir_progress(impl_progress, impl, metadata);

    @@
    expression prog_expr, impl, metadata;
    @@

    -if (prog_expr) {
    -nir_metadata_preserve(impl, metadata);
    -} else {
    -nir_metadata_preserve(impl, nir_metadata_all);
    -}
    +bool impl_progress = prog_expr;
    +nir_progress(impl_progress, impl, metadata);

    @@
    identifier prog;
    expression impl, metadata;
    @@

    -nir_metadata_preserve(impl, metadata);
    -prog = true;
    +prog = nir_progress(true, impl, metadata);

    @@
    identifier prog;
    expression impl, metadata;
    @@

    -if (prog) {
    -nir_metadata_preserve(impl, metadata);
    -}
    -return prog;
    +return nir_progress(prog, impl, metadata);

    @@
    identifier prog;
    expression impl, metadata;
    @@

    -if (prog) {
    -nir_metadata_preserve(impl, metadata);
    -}
    +nir_progress(prog, impl, metadata);

    @@
    expression impl;
    @@

    -nir_metadata_preserve(impl, nir_metadata_all);
    +nir_no_progress(impl);

    @@
    expression impl, metadata;
    @@

    -nir_metadata_preserve(impl, metadata);
    +nir_progress(true, impl, metadata);

squashme! sed -ie 's/progress = progress | /progress |=/g' $(git grep -l 'progress = prog')

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33722>
2025-02-26 15:19:53 +00:00
Alyssa Rosenzweig
91872c9c51 nir: clang-format
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33722>
2025-02-26 15:19:53 +00:00
Rhys Perry
0484044b1a nir/opt_loop: rematerialize header block derefs in their use blocks
Otherwise, we could end up with phis of derefs.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Fixes: 6b4b044739 ("nir/opt_loop: add loop peeling optimization")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31324>
2024-10-01 12:24:22 +00:00
Georg Lehmann
a9f8089240 nir: replace nir_opt_remove_phis_block with a single source version
This is what callers actually want, and it simplifies nir_opt_remove_phis
because we can assume dominance meta data is valid.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31031>
2024-09-27 05:19:16 +00:00
Rhys Perry
64ac601049 nir/opt_loop: skip peeling if the loop ends with any kind of jump
Any kind of jump prevents us from moving it to the top of the loop, not
just breaks.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 6b4b044739 ("nir/opt_loop: add loop peeling optimization")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31002>
2024-09-12 23:36:58 +00:00
Rhys Perry
af3b099e0a nir/opt_loop: skip peeling if the break is non-trivial
If this nir_if contains continues or other breaks, we can't move it
outside the loop.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 6b4b044739 ("nir/opt_loop: add loop peeling optimization")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31002>
2024-09-12 23:36:57 +00:00
Rhys Perry
4f44a944bb nir/opt_if: fix fighting between split_alu_of_phi and peel_initial_break
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 6b4b044739 ("nir/opt_loop: add loop peeling optimization")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11822
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31002>
2024-09-12 23:36:57 +00:00
Konstantin Seurer
0fc3c52e43 nir/opt_loop: Fix handling else-breaks in merge_terminators
If both breaks are in the else branch, we have to use iand.

Fixes: 9995f33 ("nir: add merge loop terminators optimisation")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11726
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30850>
2024-08-29 05:49:35 +00:00
Rob Clark
563ec4754a nir/opt_loop: Don't peel initial break if loop ends in break
A loop that looks like:

   loop {
      do_work_1();
      if (cond) {
         break;
      } else {
      }
      do_work_2();
      break;
   }

We can't pull that break ahead of do_work_1() after hoisting the initial
do_work_1() out of the loop.  So bail in this case.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11711
Fixes: 6b4b044739 ("nir/opt_loop: add loop peeling optimization")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30702>
2024-08-17 14:27:02 +00:00
Timothy Arceri
b26ef8f153 nir: correctly track current loop in nir_opt_loop()
We were not restoring an outer loop as the current loop after we had
finished processing a nested loop.

Fixes: 9995f336e6 ("nir: add merge loop terminators optimisation")

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29686>
2024-06-13 15:00:35 +00:00
Rhys Perry
92af96e0b3 nir/opt_loop: fix formatting
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29686>
2024-06-13 15:00:35 +00:00
Rhys Perry
cb51a93c1e nir/opt_loop: rematerialize derefs instead of creating phis
Fixes NIR validation of hogwarts_legacy/00ac08423ad6e422.

Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Fixes: 9995f336e6 ("nir: add merge loop terminators optimisation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29686>
2024-06-13 15:00:35 +00:00
Timothy Arceri
9995f336e6 nir: add merge loop terminators optimisation
Merge two consecutive basic terminators.

Acked-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28998>
2024-06-11 01:42:23 +00:00
Daniel Schürmann
6b4b044739 nir/opt_loop: add loop peeling optimization
This optimization turns:

     loop {
        do_work_1();
        if (cond) {
           break;
        } else {
        }
        do_work_2();
     }

 into:

     do_work_1();
     if (cond) {
     } else {
        loop {
           do_work_2();
           do_work_1();
           if (cond) {
              break;
           } else {
           }
        }
     }

RADV GFX11:
Totals from 925 (1.17% of 79395) affected shaders:
MaxWaves: 20583 -> 20455 (-0.62%)
Instrs: 5260489 -> 5361418 (+1.92%); split: -0.63%, +2.55%
CodeSize: 26965388 -> 27501104 (+1.99%); split: -0.48%, +2.47%
VGPRs: 70304 -> 70712 (+0.58%)
SpillSGPRs: 2163 -> 2159 (-0.18%)
Scratch: 51200 -> 69632 (+36.00%)
Latency: 36404844 -> 34542213 (-5.12%); split: -5.51%, +0.39%
InvThroughput: 6628474 -> 6384249 (-3.68%); split: -4.19%, +0.50%
VClause: 124997 -> 127008 (+1.61%); split: -0.43%, +2.04%
SClause: 121774 -> 120799 (-0.80%); split: -3.21%, +2.40%
Copies: 357048 -> 360850 (+1.06%); split: -0.62%, +1.68%
Branches: 171985 -> 168082 (-2.27%); split: -3.61%, +1.34%
PreSGPRs: 59812 -> 60088 (+0.46%); split: -0.20%, +0.66%
PreVGPRs: 60325 -> 60586 (+0.43%); split: -0.29%, +0.72%
VALU: 2882263 -> 2951373 (+2.40%); split: -0.37%, +2.77%
SALU: 636373 -> 640091 (+0.58%); split: -0.87%, +1.46%
VMEM: 200059 -> 204612 (+2.28%); split: -0.09%, +2.36%
SMEM: 173328 -> 174343 (+0.59%); split: -2.34%, +2.92%
VOPD: 1064 -> 898 (-15.60%); split: +0.09%, -15.70%

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28150>
2024-05-03 13:01:29 +00:00
Daniel Schürmann
bf43af984a nir/opt_loop_cf: generalize removal of "trivial" continues
So that is also handles break statements and works in
arbitrarily nested control flow.

Totals from 905 (1.18% of 76636) affected shaders: (RADV, GFX11)
Instrs: 605164 -> 605548 (+0.06%); split: -0.01%, +0.08%
CodeSize: 3162036 -> 3163472 (+0.05%); split: -0.01%, +0.06%
Latency: 2045559 -> 1387622 (-32.16%)
InvThroughput: 352344 -> 231676 (-34.25%)
SClause: 16092 -> 16088 (-0.02%); split: -0.04%, +0.02%
Copies: 41286 -> 41297 (+0.03%); split: -0.02%, +0.05%
Branches: 19949 -> 19929 (-0.10%)
PreSGPRs: 33413 -> 33385 (-0.08%)
PreVGPRs: 19177 -> 19135 (-0.22%)

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24940>
2024-01-03 20:48:05 +00:00
Daniel Schürmann
9808ef0349 nir/opt_loop: move loop control-flow optimizations into separate pass
This new pass aims to simplify loop control-flow by reducing the number
of break and continue statements. It also supersedes nir_opt_trivial_continues().

For this purpose, it implements 3 optimizations:
- opt_loop_terminator(), as previously
- opt_loop_merge_break_continue(), similar to opt_merge_breaks() incl. continues
- opt_loop_last_block(), a generalization of opt_if_loop_last_continue()

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24940>
2024-01-03 20:48:04 +00:00