There are a lot of restrictions for bfloat16. The one that prevents this
very useful optimization from being possible is, "Broadcast of bfloat16
scalar is not supported."
Part of the reason this MR exists is to build up to implementing BF
support, and there are a couple more commits that implement
this. However, it fails on both real hardware and simulation:
Instruction is: mad (8|M0) r6.0<1>:f 0xBF80:bf r2.0<8;1>:f r64.0<0>:f
In bfloat/float mixed mode, bfloat src must be packed.
Alas.
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
These can't mix with F values, but if the non-constant sources are
already HF, this is allowed in src0.
No shader-db changes on any Intel platform.
fossil-db:
Ice Lake
Totals:
Instrs: 236027458 -> 236027442 (-0.00%)
Cycle count: 24515944704 -> 24515945379 (+0.00%)
Totals from 8 (0.00% of 798454) affected shaders:
Instrs: 10226 -> 10210 (-0.16%)
Cycle count: 58567 -> 59242 (+1.15%)
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
Nothing can generate this currently, but a future commit will.
The Bspec and experimentation support the following limitations:
- Gfx11: Either src0 or src2 can be W or UW.
- Gfx12: Either src0 or src2 can be W or UW.
- Gfx12.5: Both src0 and src2 can be W or UW.
- Gfx20: Both src0 and src2 can be W or UW.
v2: Add missing break statement.
v3: Leave the MAD handling in the case with the other 3 source
instructions. Suggested by Caio.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
This is very unlikely for floating point MAD. At some point I intend
to add internal integer MAD uses, and this could occur there.
No shader-db or fossil-db changes on any Intel platform.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
v2: Move the full constant folding part to
brw_constant_fold_instruction. Suggested by Caio. I did this by
extracting the core part of the folding to a helper function.
v3: Delete stale comment. Noticed by Caio.
shader-db:
All Intel platforms had similar results. (Lunar Lake shown)
total instructions in shared programs: 18090847 -> 18090843 (<.01%)
instructions in affected programs: 150 -> 146 (-2.67%)
helped: 1 / HURT: 0
total cycles in shared programs: 919664648 -> 919663210 (<.01%)
cycles in affected programs: 3426 -> 1988 (-41.97%)
helped: 1 / HURT: 0
LOST: 1
GAINED: 0
fossil-db:
All Intel platforms had similar results. (Lunar Lake shown)
Totals:
Instrs: 220496486 -> 220496403 (-0.00%)
Cycle count: 31610880908 -> 31610879044 (-0.00%); split: -0.00%, +0.00%
Totals from 70 (0.01% of 702439) affected shaders:
Instrs: 47018 -> 46935 (-0.18%)
Cycle count: 6335504 -> 6333640 (-0.03%); split: -0.11%, +0.09%
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
This enables constant combining to do its job.
v2: Restore accidentally deleted line from a comment. Noticed by Caio.
shader-db:
All Intel platforms had similar results. (Lunar Lake shown)
total cycles in shared programs: 919668392 -> 919669310 (<.01%)
cycles in affected programs: 10125264 -> 10126182 (<.01%)
helped: 348 / HURT: 194
fossil-db:
All Intel platforms had similar results. (Lunar Lake shown)
Totals:
Cycle count: 31610720660 -> 31610692748 (-0.00%); split: -0.00%, +0.00%
Totals from 9066 (1.29% of 702433) affected shaders:
Cycle count: 810411934 -> 810384022 (-0.00%); split: -0.01%, +0.00%
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
No shader-db changes on any Intel platform.
fossil-db:
Meteor Lake, DG2, Tiger Lake, and Ice Lake had similar results. (Meteor Lake shown)
Totals:
Cycle count: 25096109024 -> 25096108722 (-0.00%); split: -0.00%, +0.00%
Totals from 4106 (0.51% of 797610) affected shaders:
Cycle count: 63266176 -> 63265874 (-0.00%); split: -0.01%, +0.01%
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
Always propagate into any source. Let commute_immedates and constant
combining sort out the mess. It's literally their job.
No shader-db changes on any Intel platform. The fossil-db changes just
appear to be subtle changes in register allocation if the immediate
source changes from src0 to src2.
v2: Update the comment in commute_immediates. Suggested by Caio.
fossil-db:
Lunar Lake, Meteor Lake, and DG2 had similar results. (Lunar Lake shown)
Totals:
Cycle count: 31610720510 -> 31610720660 (+0.00%); split: -0.00%, +0.00%
Totals from 8 (0.00% of 702433) affected shaders:
Cycle count: 5522382 -> 5522532 (+0.00%); split: -0.00%, +0.00%
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
No shader-db or fossil-db changes on any Intel platform. This commit
just prevents issues with a later commit, "brw/copy: Don't try to be
clever about ADD3 constant propagation."
v2: Use 'can_promote = true; break;' instead of 'return
true;'. Suggested by Caio.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
Fold the cases where one of the sources is zero or two of the sources
are constants. Both case will result in a regular ADD.
No shader-db or fossil-db changes on any Intel platform. This commit
just prevents issues with a later commit, "brw/copy: Don't try to be
clever about ADD3 constant propagation."
v2: Move the full constant folding part to
brw_constant_fold_instruction. Suggested by Caio.
v3: Eliminate the impossible src.file == BAD_FILE case in
brw_fs_opt_algebraic. Suggested by Caio.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
The current assertion fails as soon as a MAD with src0 and src2 being
immediate is detected.
The assertion was supposted to catch, "If it's ADD3, only one of src0
and src2 can be immediate." The detect this, the opcode test should have
been !=.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Fixes: c1c09e3c4a ("brw/emit: Add correct 3-source instruction assertions for each platform")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
Some callers of brw_constant_fold_instruction depend on the result being
a MOV of immediate when progress is made. Previously `MUL dst:D src0:D
1:D` would be converted to `MOV dst:D src0:D`. There was also no
handling for `MUL dst:D imm0:D imm1:D`.
This could cause problems if one of the immedate values was -1. The
existing code would convert this to a `MOV dst:D imm0:D` and set the
negate flag on src0. That is not correct.
v2: Fix the is_negative_one case handling of the non-negative-one
source. Add a comment explaining the assertion. Both suggested by Caio.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Fixes: 2cc1575a31 ("brw/algebraic: Refactor constant folding out of brw_fs_opt_algebraic")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
Some callers of brw_constant_fold_instruction depend on the result being
a MOV of immediate when progress is made. Previously `ADD dst:D src0:D
0:D` would be converted to `MOV dst:D src0:D`. There was also no
handling for `ADD dst:D imm0:D imm1:D`.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Fixes: 2cc1575a31 ("brw/algebraic: Refactor constant folding out of brw_fs_opt_algebraic")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32436>
Almost all cases now handled with default arguments. The only real
extra work that was being done was pushed to the client code in
debug_optimizer().
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32596>
so we can use designated initializers and other fun features.
all affected shaders are in gfxbench:
total instructions in shared programs: 2750549 -> 2750497 (<.01%)
instructions in affected programs: 10832 -> 10780 (-0.48%)
helped: 4
HURT: 2
Inconclusive result (value mean confidence interval includes 0).
total alu in shared programs: 2278478 -> 2278760 (0.01%)
alu in affected programs: 7040 -> 7322 (4.01%)
helped: 2
HURT: 4
Alu are HURT.
total fscib in shared programs: 2276985 -> 2277267 (0.01%)
fscib in affected programs: 7040 -> 7322 (4.01%)
helped: 2
HURT: 4
Fscib are HURT.
total bytes in shared programs: 19922466 -> 19922734 (<.01%)
bytes in affected programs: 71412 -> 71680 (0.38%)
helped: 4
HURT: 2
Inconclusive result (value mean confidence interval includes 0).
total regs in shared programs: 865070 -> 865086 (<.01%)
regs in affected programs: 142 -> 158 (11.27%)
helped: 0
HURT: 2
total uniforms in shared programs: 2120930 -> 2121034 (<.01%)
uniforms in affected programs: 244 -> 348 (42.62%)
helped: 0
HURT: 2
total scratch in shared programs: 11576 -> 11600 (0.21%)
scratch in affected programs: 2744 -> 2768 (0.87%)
helped: 0
HURT: 2
total spills in shared programs: 958 -> 868 (-9.39%)
spills in affected programs: 958 -> 868 (-9.39%)
helped: 6
HURT: 0
total fills in shared programs: 732 -> 626 (-14.48%)
fills in affected programs: 732 -> 626 (-14.48%)
helped: 4
HURT: 2
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
this alignment matches what nir_lower_scratch_to_var wants. this is not
correctness bearing but it mitigates stats regressions.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
important for reproducability. wondering if we can do this in common code but
not sure how yet.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
hk already does. this quiesches warnings with single argument static_assert
which we want for CL parity.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
add enough preprocessor guards that we can include this from CL and get basic
implementations of things. FIXED packs are missing due to llroundf (probably
fixable).
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
this is helpful to indirect draw munging code, which applies to at least 3
stacks using driver CL stuff (current Intel, shortterm Asahi, mediumterm
Panfrost)
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
In an attempt to make OpenCL shaders more "batteries included", start building
up a standard library. Based on libagx.h.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mary Guillemard <mary.guillemard@collabora.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
although rusticl isn't lighting it up yet, it's helpful to get
sub_group_ballot for driver CL, which is all standard Vulkan-compatible spirv.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32529>
In this case, num_sources is bigger than this->sources, so if we loop
up to num_sources (instead of this->sources) we'll end up reading past
the end of old_src[]. Only copy up to what we originally had.
This was found by code inspection, I'm not aware of any applications
failing due to the lack of this patch.
Fixes: d9e737212d ("intel/brw: Add a src array for the common case in fs_inst")
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Signed-off-by: Paulo Zanoni <paulo.r.zanoni@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32600>
As of writing, PanVK on v10 HW is in pretty good shape. It's not yet
conformant, but we were passing over 99.9% of the CTS last time I
checked. That's probably good enough to drop the opt-in here.
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32561>
We already have and use vk_warn_non_conformant_implementation(), so
we're already being clear that PanVK is not yet conformant. Let's not
repeat that information here, and instead focus on it not being
well-tested.
This brings the wording more or less in-line with NVK.
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32561>
This is beneficial to applications that rely on
the implicit primitive ID from VS.
- We don't have to disable provoking vertex reuse,
which results in more efficient vertex processing.
- There is no LDS access needed to export the primitive ID,
because it is already available to GS threads.
- As a consequence of not needing LDS, we can use this
together with NGG passthrough mode.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32270>
We want to make the implicit VS primitive ID a per-primitive
output attribute, which means that this has to be last.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32270>
This configuration will be enabled in RADV in a subsequent commit.
On GFX10.3:
Do this together with the primitive export, to avoid adding extra
CF, and to ensure optimal access of the export space.
On GFX11:
It's not an export but a memory store instruction, so always do
it earlier and ensure the optimal attribute ring access pattern.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32270>
It doesn't do anything useful for other stages.
In VS, we use this when the implicit primitive ID is needed,
so that we can export that as a per-vertex attribute of the
provoking vertex.
In TES, the patch ID (which is used as the primitive ID) is
already a per-vertex input VGPR, so it doesn't make sense to
configure this.
In GS, the primitive ID is explicitly written by the shader,
so it makes no sense to disable provoking vertex reuse in the
input.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32270>
Match AMDVLK and radeonsi.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32577>
Match the limit of radeonsi and RADV.
No fossil-db changes.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32577>
