We have this small calculations repeated in each Intel driver, so move
them to a single place to be reused. Also includes "right_mask" since
is always used in the same context and depends on the dispatch info
values.
Reviewed-by: Marcin Ślusarz <marcin.slusarz@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10504>
There's a recently discovered HW bug affecting hardware at least as far
back as Skylake where, if the LOD is out-of-bounds for any SIMD lane,
then garbage may be returned in all SIMD lanes. The easy solution is to
set lower_txs_lod so that we always have a constant LOD of 0 which we
know a priori is always in-bounds. Fortunately, not many shaders
actually use textureSize() with LOD.
Shader-db results on Ice Lake:
total instructions in shared programs: 19948537 -> 19948564 (<.01%)
instructions in affected programs: 3859 -> 3886 (0.70%)
helped: 0
HURT: 7
One of the shaders is in Civilization: Beyond Earth, and the rest are
all in Civilization VI.
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
Cc: mesa-stable@lists.freedesktop.org
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10538>
The source_depth_to_render_target flag can get set on old gen4-5 HW in a
few cases which are independent of the app writing gl_FragDepth. It
should be safe to just use fetch_payload_reg in that case instead of
depending in interpolation setup. This fixes a bug with certain very
simple shaders where we might end up not including the depth when we
should have.
While we're here, rework the logic around setting src_depth and add a
comment so it's more clear what's going on.
Fixes: 6d4070f3dd "intel/compiler: add support for fragment coordinate..."
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10596>
There were two bugs which crep in here as part of 64551610d1:
forgetting that exec sizes in HW are in log2 space and having the
exec_size condition for the subtype backwards.
Fixes: 64551610d1 "intel/compiler: rework message descriptors..."
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10588>
Those helpers exist primarily to sort out some of the weirdness around
Gen4-6 dataport access. On Gen5 and earlier, everything was called
"dataport" and, instead of the SFID we have today there was a "target
cache" parameter in the descriptor. There are also some bits that moved
around on various gens depending on read vs. write. Starting with Gen6,
most things which target one of the data cache SFIDs should use
brw_dp_desc() instead.
v2: Drop backward comment (Ken)
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7455>
It's a Gen6 XFB thing. It's never used for anything else so there's no
point in having a target cache switch.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7455>
v2: Drop new internal opcodes (Jason)
Simplify code (Jason)
v3: Add Z computation for coarse pixels
v4: Document things a little
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7455>
v2: Use the new inst->ex_desc field (Jason)
v3: Drop CPS LoD compensation from sampler messages (Lionel)
v4: Drop useless uses_rate_shading (Ken)
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7455>
Render target message descriptors are slightly different from the
dataport ones. In particular the msg_type field is on bits 14:17 for
RT while bits 14:18 for DP.
v2: Drop unused send_commit_msg field in brw_fb_write_desc() (Ken)
v3: Rebase on top renaming (Lionel)
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Suggested-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7455>
With the missing else, this prints the compacted hex followed by hex
for an uncompacted version of the compacted instruction. It also
doesn't print hex for instructions that are not compacted.
Fixes: bc4a127d6e ("intel/disasm: Label support in shader disassembly for UIP/JIP")
Fixes: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4245
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10535>
Normally, we never see NULL in a source. However, starting with
eab1c55590, we can with a SHADER_OPCODE_SEND if it only has the first
payload. We were inserting barriers which adds unnecessary scheduling
dependencies and takes a lot of compile time because inserting a single
barrier is an O(n) operation.
All the extra O(n) can have a surprisingly large effect. This cuts the
runtime of dEQP-VK.binding_model.buffer_device_address.set3.depth3.
basessbo.convertcheckuv2.store.single.std140.frag by a factor of 20x for
a debug build.
Shader-db results on ICL:
total instructions in shared programs: 19918983 -> 19921610 (0.01%)
instructions in affected programs: 884074 -> 886701 (0.30%)
helped: 1688
HURT: 817
helped stats (abs) min: 1 max: 163 x̄: 4.23 x̃: 1
helped stats (rel) min: 0.02% max: 12.50% x̄: 1.08% x̃: 0.61%
HURT stats (abs) min: 1 max: 2674 x̄: 11.95 x̃: 2
HURT stats (rel) min: 0.11% max: 70.22% x̄: 1.71% x̃: 1.03%
95% mean confidence interval for instructions value: -1.97 4.06
95% mean confidence interval for instructions %-change: -0.28% -0.06%
Inconclusive result (value mean confidence interval includes 0).
total cycles in shared programs: 976503324 -> 975884809 (-0.06%)
cycles in affected programs: 82581703 -> 81963188 (-0.75%)
helped: 4144
HURT: 5010
helped stats (abs) min: 1 max: 79294 x̄: 311.31 x̃: 8
helped stats (rel) min: <.01% max: 53.69% x̄: 2.00% x̃: 0.51%
HURT stats (abs) min: 1 max: 92266 x̄: 134.04 x̃: 8
HURT stats (rel) min: <.01% max: 218.09% x̄: 3.25% x̃: 0.53%
95% mean confidence interval for cycles value: -119.85 -15.29
95% mean confidence interval for cycles %-change: 0.68% 1.07%
Inconclusive result (value mean confidence interval and %-change mean confidence interval disagree).
total spills in shared programs: 10659 -> 12014 (12.71%)
spills in affected programs: 441 -> 1796 (307.26%)
helped: 7
HURT: 12
total fills in shared programs: 11551 -> 14429 (24.92%)
fills in affected programs: 993 -> 3871 (289.83%)
helped: 8
HURT: 11
total sends in shared programs: 1025832 -> 1025353 (-0.05%)
sends in affected programs: 2241 -> 1762 (-21.37%)
helped: 105
HURT: 1
helped stats (abs) min: 1 max: 87 x̄: 4.57 x̃: 2
helped stats (rel) min: 5.56% max: 54.72% x̄: 11.37% x̃: 10.00%
HURT stats (abs) min: 1 max: 1 x̄: 1.00 x̃: 1
HURT stats (rel) min: 100.00% max: 100.00% x̄: 100.00% x̃: 100.00%
95% mean confidence interval for sends value: -7.39 -1.65
95% mean confidence interval for sends %-change: -12.95% -7.70%
Sends are helped.
LOST: 93
GAINED: 109
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4648
Fixes: eab1c55590 "intel/fs: Support SENDS in SHADER_OPCODE_SEND"
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10412>
This patch includes a number of reworks and fixes squashed in by
Nanley Chery, Sagar Ghuge, Jordan Justen and Francisco Jerez.
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
The PIXEL_X/Y opcodes used by the current implementation are broken on
XeHP due to the new regioning restrictions of the floating-point pipe.
We could have the regioning lowering pass fix it in theory by lowering
the conversions into separate MOV instructions, but that would be more
costly than this implementation that only needs a pair of pipelined
ADDs and a pair of pipelined MOVs.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
Unfortunately the funky Align1 regions used by the code generator in
order to implement derivatives efficiently aren't available to the
floating-point pipeline on XeHP. We need to lower them into a number
of pipelined integer shuffle instructions followed by the
floating-point difference computation.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
[ Francisco Jerez: Add TODO comment explaining why this is helpful and
how we could better fix it. ]
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
It has been removed from the hardware.
[jordan.l.justen@intel.com: Move to brw_postprocess_nir]
v2: Switch to nir_lower_idiv_precise (Rhys).
v3: Fix for interface changes of nir_lower_idiv.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
From the BSpec:
"When multiplying DW X DW, resulting dst can only be QW precision. If
DW precision is required at output than MUL/MACH macro must be used."
So for now simply lower it. We might want to revisit it later.
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
The floating-point and double-precision FPU pipelines of XeHP
platforms don't support arbitrary regioning modes, corresponding
channels of sources and destination are required to be aligned to the
same sub-register offset, similar to the restriction FP64 instructions
had on CHV/BXT platforms.
Most violations of this restriction can be fixed easily by teaching
has_dst_aligned_region_restriction() about the change so the regioning
lowering pass gets rid of any unsupported regioning. For cases where
this is not sufficient (e.g. because a virtual instruction internally
uses some regioning mode not supported by the floating-point pipeline)
the regioning lowering pass is extended with an additional
lower_exec_type() codepath that bit-casts sources and destination to
an integer type whenever the execution type is not supported by the
instruction.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
The regioning mode used here is no longer supported by the
floating-point pipeline. We could run the regioning lowering pass in
order to fix it with some extra copies, but it's more efficient to
change the instruction to use integer types.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
According to the hardware spec "Vx1 and VxH indirect addressing for
Float, Half-Float, Double-Float and Quad-Word data must not be used."
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
In combination with the previous changes we can just check whether an
instruction has any potentially unsatisfied dependencies on more than
one pipeline, and if so use TGL_PIPE_ALL synchronization with an
appropriate RegDist counter, otherwise synchronize with the single
pipeline it has a dependency on, if any.
Only minor difficulty is caused by the fact that the hardware doesn't
have any way to encode pipeline information when a RegDist and an SBID
dependency need to be provided simultaneously, in which case the
synchronization pipeline is inferred by the hardware. We need to
verify that the hardware's inference will give the correct result
(which may not be the case if e.g. some data was bit-cast from a
different type), and if not emit separate SYNC instructions instead of
baking the RegDist dependency into the instruction (Note that SET SBID
dependencies must always be baked into the corresponding out-of-order
instruction).
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
This extends the current ordered_address instruction counter to a
vector with one component per asynchronous ALU pipeline, allowing us
to track the last instruction that accessed a register separately for
each ALU pipeline of the XeHP EU, making it straightforward to
infer the right cross-pipeline synchronization annotations.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
v2: Make unit tests happy (with ubsan as run by GitLab automation).
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
This was a placeholder for the XeHP cross-pipeline synchronization
code, bring it back.
This reverts commit a80e44902f.
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
Define two helper functions local to the software scoreboard lowering
pass describing the behavior of the hardware and code generator:
inferred_sync_pipe() calculates the ALU pipeline the hardware will
implicitly synchronize with when a RegDist SWSB annotation is used
without providing explicit pipeline synchronization information,
inferred_exec_pipe() infers the ALU pipeline that will execute the
instruction.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
The execution units of XeHP platforms have multiple asynchronous ALU
pipelines instead of (as far as software is concerned) the single
in-order pipeline that handled most ALU instructions except for
extended math in the original Xe. It's now the compiler's
responsibility to identify cross-pipeline dependencies and insert
synchronization annotations whenever necessary, which are encoded as
some additional bits of the SWSB instruction field.
This commit represents the cross-pipeline synchronization annotations
as part of the existing tgl_swsb structure used for codegen. The
existing tgl_swsb_*() helpers used by hand-crafted assembly are
extended to default to TGL_PIPE_ALL big-hammer synchronization in
order to ensure backwards compatibility with the existing assembly.
The following commits will extend the software scoreboard lowering
pass in order to keep track of cross-pipeline dependencies across IR
instructions, and insert more specific pipeline annotations in the
SWSB field.
The disassembler is also extended here to print out any existing
pipeline sync annotations.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10000>
clang generates a warning if there's no explicit break or fall-through
annotation. The latter would be kind of silly in this case, and not
robust against any future changes turning the fall-through invalid.
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10220>
One exception is src/amd/addrlib/, for which -Wimplicit-fallthrough is
explicitly disabled.
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10220>
