In the C23 standard unreachable() is now a predefined function-like
macro in <stddef.h>
See https://android.googlesource.com/platform/bionic/+/HEAD/docs/c23.md#is-now-a-predefined-function_like-macro-in
And this causes build errors when building for C23:
-----------------------------------------------------------------------
In file included from ../src/util/log.h:30,
from ../src/util/log.c:30:
../src/util/macros.h:123:9: warning: "unreachable" redefined
123 | #define unreachable(str) \
| ^~~~~~~~~~~
In file included from ../src/util/macros.h:31:
/usr/lib/gcc/x86_64-linux-gnu/14/include/stddef.h:456:9: note: this is the location of the previous definition
456 | #define unreachable() (__builtin_unreachable ())
| ^~~~~~~~~~~
-----------------------------------------------------------------------
So don't redefine it with the same name, but use the name UNREACHABLE()
to also signify it's a macro.
Using a different name also makes sense because the behavior of the
macro was extending the one of __builtin_unreachable() anyway, and it
also had a different signature, accepting one argument, compared to the
standard unreachable() with no arguments.
This change improves the chances of building mesa with the C23 standard,
which for instance is the default in recent AOSP versions.
All the instances of the macro, including the definition, were updated
with the following command line:
git grep -l '[^_]unreachable(' -- "src/**" | sort | uniq | \
while read file; \
do \
sed -e 's/\([^_]\)unreachable(/\1UNREACHABLE(/g' -i "$file"; \
done && \
sed -e 's/#undef unreachable/#undef UNREACHABLE/g' -i src/intel/isl/isl_aux_info.c
Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36437>
this is more explicit than vec2's and hence has fewer footguns. in particular
it's easier to handle with preambles in a sane way.
modelled on what ir3 does.
there's probably room for more clean up but for now this unblocks what I want to
do.
stats don't seem concerning.
Totals from 692 (1.29% of 53701) affected shaders:
MaxWaves: 441920 -> 442112 (+0.04%)
Instrs: 1588748 -> 1589304 (+0.03%); split: -0.05%, +0.08%
CodeSize: 11487976 -> 11491620 (+0.03%); split: -0.04%, +0.07%
ALU: 1234867 -> 1235407 (+0.04%); split: -0.06%, +0.10%
FSCIB: 1234707 -> 1235249 (+0.04%); split: -0.06%, +0.10%
IC: 380514 -> 380518 (+0.00%)
GPRs: 117292 -> 117332 (+0.03%); split: -0.08%, +0.11%
Preamble instrs: 314064 -> 313948 (-0.04%); split: -0.05%, +0.01%
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35949>
Class represents an indexed "ideal" register class, where non-general classes
only allow defs that choose that class in the def_size callback.
nir_opt_preamble will try to assign specialized classes where possible, falling
back to the general class once the special-purpose classes are exhausted.
AGX will use this mechanism to promote bindless texture handles to bound texture
registers where possible, falling back to pushing the handle as a uniform where
not possible. Supporting multiple classes in nir_opt_preamble allows this
multi-level hoisting to work in a single nir_opt_preamble call with proper
global behaviour.
Add this concept to nir_opt_preamble so we can use it in AGX later in this MR.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Job Noorman <job@noorman.info>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35949>
this lets the backend implement bounded loads (i.e. robust SSBOs) in a way
that's more clever than a full branch. similar idea to
load_global_constant_bound which should eventually be merged into this.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Job Noorman <job@noorman.info>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35720>
This is needed for cl_khr_mipmap_image, specifically the OpenCL C
function get_image_num_mip_levels.
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30834>
This intrinsic helps to read the W coordinate stored in the QPU register
when initializing the input data for the fragment shaders.
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28072>
In general, it is unsafe to speculatively hoist conditionally executed loads
into the preamble. For example, if the shader does:
if (ptr is valid) {
foo(*ptr)
}
we cannot dereference ptr in the preamble without knowing that the pointer is
valid (which may not be determinable, since it might not be uniform).
nir_opt_preamble needs to stop speculating in this case, or otherwise using
preambles can cause faults on legal shaders.
However, some platforms may be able to speculate loads safely. For example,
Apple hardware is able to suppress MMU faults, making speculation safe. This is
controlled global register to control this behaviour, set at boot-time by the
kernel. (macOS suppresses these faults unconditionally, this feature may be
used in their implementation of sparse textures. Currently Linux does not
suppress any faults but this may change later.)
Since nir_opt_preamble should work soundly and optimally on a variety of
platforms, we need to respect the ACCESS flag.
Thanks to the if-else hoisting implemented earlier in the series, this isn't too
terrible of a band-aid on Asahi:
total instructions in shared programs: 1499674 -> 1507699 (0.54%)
instructions in affected programs: 78865 -> 86890 (10.18%)
helped: 0
HURT: 337
Instructions are HURT.
total bytes in shared programs: 10238284 -> 10279308 (0.40%)
bytes in affected programs: 554504 -> 595528 (7.40%)
helped: 3
HURT: 334
Bytes are HURT.
total halfregs in shared programs: 452049 -> 454015 (0.43%)
halfregs in affected programs: 7569 -> 9535 (25.97%)
helped: 7
HURT: 150
Halfregs are HURT.
There are no shader-db changes on ir3 as expected, since ir3 can safely
speculate all instructions in my shader-db.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Connor Abbott <cwabbott0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24011>
Add infrastructure to reconstruct if's. Later in the series, this will let us
hoist loads from inside uniform if's without speculating. For now, it lets us
handle phi's in nir_opt_preamble in a straightforward way.
Results on AGX are good:
total instructions in shared programs: 1504730 -> 1499674 (-0.34%)
instructions in affected programs: 153673 -> 148617 (-3.29%)
helped: 496
HURT: 0
Instructions are helped.
total bytes in shared programs: 10287768 -> 10238284 (-0.48%)
bytes in affected programs: 1113724 -> 1064240 (-4.44%)
helped: 496
HURT: 0
Bytes are helped.
total halfregs in shared programs: 452669 -> 452049 (-0.14%)
halfregs in affected programs: 14825 -> 14205 (-4.18%)
helped: 152
HURT: 99
Halfregs are helped.
total threads in shared programs: 16469504 -> 16470784 (<.01%)
threads in affected programs: 8960 -> 10240 (14.29%)
helped: 10
HURT: 0
Threads are helped.
Results on ir3 is a bit more of a wash but still should be a win overall: The
regression in moves seems scary, but the cost model already accounts for them as
evidenced by instruction count coming out ahead.
total instructions in shared programs: 3108750 -> 3105993 (-0.09%)
instructions in affected programs: 317367 -> 314610 (-0.87%)
helped: 675
HURT: 242
Instructions are helped.
total nops in shared programs: 673152 -> 675048 (0.28%)
nops in affected programs: 74551 -> 76447 (2.54%)
helped: 353
HURT: 347
Inconclusive result (%-change mean confidence interval includes 0).
total non-nops in shared programs: 2435598 -> 2430945 (-0.19%)
non-nops in affected programs: 232664 -> 228011 (-2.00%)
helped: 816
HURT: 38
Non-nops are helped.
total mov in shared programs: 78201 -> 84011 (7.43%)
mov in affected programs: 10726 -> 16536 (54.17%)
helped: 60
HURT: 781
Mov are HURT.
total cov in shared programs: 74964 -> 74906 (-0.08%)
cov in affected programs: 273 -> 215 (-21.25%)
helped: 17
HURT: 0
Cov are helped.
total dwords in shared programs: 6716814 -> 6748726 (0.48%)
dwords in affected programs: 879778 -> 911690 (3.63%)
helped: 12
HURT: 948
Dwords are HURT.
total full in shared programs: 193210 -> 193212 (<.01%)
full in affected programs: 278 -> 280 (0.72%)
helped: 12
HURT: 22
Inconclusive result (value mean confidence interval includes 0).
total constlen in shared programs: 493632 -> 494816 (0.24%)
constlen in affected programs: 19904 -> 21088 (5.95%)
helped: 9
HURT: 306
Constlen are HURT.
total cat0 in shared programs: 742476 -> 745046 (0.35%)
cat0 in affected programs: 84455 -> 87025 (3.04%)
helped: 277
HURT: 489
Cat0 are HURT.
total cat1 in shared programs: 153303 -> 159059 (3.75%)
cat1 in affected programs: 17810 -> 23566 (32.32%)
helped: 69
HURT: 780
Cat1 are HURT.
total cat2 in shared programs: 1144508 -> 1140731 (-0.33%)
cat2 in affected programs: 121284 -> 117507 (-3.11%)
helped: 841
HURT: 0
Cat2 are helped.
total cat3 in shared programs: 942098 -> 934804 (-0.77%)
cat3 in affected programs: 87140 -> 79846 (-8.37%)
helped: 855
HURT: 1
Cat3 are helped.
total cat4 in shared programs: 65261 -> 65249 (-0.02%)
cat4 in affected programs: 42 -> 30 (-28.57%)
helped: 12
HURT: 0
Cat4 are helped.
total sstall in shared programs: 237311 -> 241281 (1.67%)
sstall in affected programs: 33755 -> 37725 (11.76%)
helped: 179
HURT: 493
Sstall are HURT.
total (ss) in shared programs: 58166 -> 58795 (1.08%)
(ss) in affected programs: 4535 -> 5164 (13.87%)
helped: 35
HURT: 664
(ss) are HURT.
total systall in shared programs: 503784 -> 503805 (<.01%)
systall in affected programs: 3170 -> 3191 (0.66%)
helped: 16
HURT: 13
Inconclusive result (value mean confidence interval includes 0).
total (sy) in shared programs: 27261 -> 27259 (<.01%)
(sy) in affected programs: 76 -> 74 (-2.63%)
helped: 8
HURT: 5
Inconclusive result (value mean confidence interval includes 0).
total waves in shared programs: 439848 -> 439872 (<.01%)
waves in affected programs: 160 -> 184 (15.00%)
helped: 12
HURT: 0
Waves are helped.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Connor Abbott <cwabbott0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24011>
The way backends walk NIR when translating. This will make it easy to filter
can_move based on the parent control flow.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Connor Abbott <cwabbott0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24011>
First, we need to give the parent_instr field a unique name to be able to
replace with a helper. We have parent_instr fields for both nir_src and
nir_def, so let's rename nir_src::parent_instr in preparation for rework.
This was done with a combination of sed and manual fix-ups.
Then we use semantic patches plus manual fixups:
@@
expression s;
@@
-s->renamed_parent_instr
+nir_src_parent_instr(s)
@@
expression s;
@@
-s.renamed_parent_instr
+nir_src_parent_instr(&s)
@@
expression s;
@@
-s->parent_if
+nir_src_parent_if(s)
@@
expression s;
@@
-s.renamed_parent_if
+nir_src_parent_if(&s)
@@
expression s;
@@
-s->is_if
+nir_src_is_if(s)
@@
expression s;
@@
-s.is_if
+nir_src_is_if(&s)
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24671>
In Vulkan, UBOs are lowered by nir_lower_explicit_io, and the ubo_base_agx
sysval is unused (since it doesn't handle descriptor sets). That makes the UBO
lowering GL-only and hence belongs with the GL driver rather than the compiler.
This lets us delete the ubo_base_agx sysval.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24847>
Doing a descriptor crawl with binding tables requires a real binding table in
the shader, which won't work for VK or merged shader stages in GL. Instead,
let's lower anything that needs a crawl to bindless in the driver, so the
compiler code doesn't need to know anything about descriptor binding models.
That gets rid of the texture_base_agx sysval, which is problematic when there
are multiple descriptor sets worth of textures.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24847>
We already renamed the type, we just need to rename the enum and the
casting helper functions.
Generated with sed:
sed -i -e 's/nir_instr_type_ssa_undef/nir_instr_type_undef/g' src/**/*.h src/**/*.c src/**/*.cpp
sed -i -e 's/nir_instr_as_ssa_undef/nir_instr_as_undef/g' src/**/*.h src/**/*.c src/**/*.cpp
and two tiny whitespace fixups in lima.
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24703>
It's kosher to get load_preamble intrinsics ahead of time if the driver is
pushing sysvals. Handle them like load_uniform.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by-(with-sparkles): Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20562>
Some backends may wish to reserve early uniforms for internal system values, and
use the remaining space for preamble storage. In this case, it's convenient to
teach nir_opt_preamble about a reserved offset. It's logical to treat the output
*size instead of an in/out variable that nir_opt_preamble adds to. This requires
a slight change to the consumers to zero the input.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by-(with-sparkles): Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20562>
We'll use formatted loads and some system values to lower UBOs and VBOs to
global memory in NIR, using the AGX-specific format support and addressing
arithmetic to optimize the emitted code.
Add the intrinsics and teach nir_opt_preamble how to move them so we don't
regress UBO pushing.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19996>
For used by different counter.
Vulkan:
1. VK_QUERY_PIPELINE_STATISTIC_GEOMETRY_SHADER_PRIMITIVES_BIT,
sum generated primitives of all 4 streams when GS.
2. VK_QUERY_TYPE_PRIMITIVES_GENERATED_EXT, count generated
primitives for all 4 streams when VS/TES/GS.
3. VK_QUERY_TYPE_TRANSFORM_FEEDBACK_STREAM_EXT, count generated
and streamout primitives for all 4 streams when VS/TES/GS.
OpenGL:
1. GL_GEOMETRY_SHADER_PRIMITIVES_EMITTED_ARB, sum generated
primitives for all 4 streams when GS.
2. GL_PRIMITIVES_GENERATED, count generated primitives for all 4
streams when VS/TES/GS.
3. GL_TRANSFORM_FEEDBACK_PRIMITIVES_WRITTEN, count streamout
primitives for all 4 streams when VS/TES/GS.
pipeline_stat_query_enabled_amd is for Vulkan 1 and OpenGL 1.
xfb_query_enabled_amd is for Vulkan 2/3 and OpenGL 2/3.
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19015>
nir_opt_preamble will be crucial to optimize out the lowering for array
textures on AGX, which involves this AGX-specific sysval.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Connor Abbott <cwabbott0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18813>
This changes the trace rays logic to always use
VkTraceRaysIndirectCommand2KHR and implements
vkCmdTraceRaysIndirect2KHR. I renamed the
load_sbt_amd to sbt_base_amd and moved the SBT
load lowering from ACO to NIR.
Note that we can not just upload one pointer to
all the trace parameters because that would
be incompatible with traceRaysIndirect.
Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16430>
Found this while working on traceRaysIndirect2.
I don't think this is relevant for now at least
since we don't use the pass in RADV.
Fixes: 938c9d9 ("nir: Add a ray launch size addr intrinsic")
Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16430>
This pass tries to move computations that are uniform for the entire
draw to the preamble. There's also an API for backends to insert their
own instructions into the preamble, for porting existing UBO pushing
passes.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13148>
