Size is in bytes, not bits.
Fixes plenty of crashes in CI, like
dEQP-VK.synchronization.op.single_queue.event.write_image_fragment_read_image_tess_eval.image_128_r32_uint.
Fixes: 46f6e2ddbb ("aco: Implement storage image A16.")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18266>
It isn't safe to modify the exec mask before the discard block, and the
definition interferes with GFX11 NOP insertion.
Just use s[0:1] instead, since we won't be using it.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18125>
When SCC is clobbered between s_cmp and its operand's writer,
the current optimization that eliminates s_cmp won't kick in.
However, when s_cmp is the only user of its operand temporary,
it is possible to "pull down" the instruction that wrote the operand.
Fossil DB stats on Navi 21:
Totals from 63302 (46.92% of 134906) affected shaders:
CodeSize: 176689272 -> 176418332 (-0.15%)
Instrs: 33552237 -> 33484502 (-0.20%)
Latency: 205847485 -> 205816205 (-0.02%); split: -0.02%, +0.00%
InvThroughput: 34321285 -> 34319908 (-0.00%); split: -0.00%, +0.00%
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16266>
This was simply left out by accident when I wrote this.
Fossil DB stats on Navi 21:
Totals from 70165 (52.01% of 134906) affected shaders:
CodeSize: 246375656 -> 245814396 (-0.23%)
Instrs: 46519773 -> 46379458 (-0.30%)
Latency: 385159303 -> 385089261 (-0.02%); split: -0.02%, +0.00%
InvThroughput: 66490172 -> 66487867 (-0.00%); split: -0.00%, +0.00%
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16266>
Also delete them when they are already dead in process_instruction().
No Fossil DB changes.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16266>
get_ssa_temp's and NIR's bit size can differ for scalar sources.
This causes broken packing of the MIMG operands with A16/G16.
Fixes: f5f73db846 ("aco: Support 16bit sources for texture ops.")
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18008>
A situation where it doesn't match is probably not possible, so this
probably doesn't fix anything.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
If this is true, then the only instruction the loops visit is
p_logical_end and the loops are no-ops.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
We shouldn't remove a p_cbranch_nz branch in this situation.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: b731be2e96 ("aco: Remove branch instruction when exec is constant non-zero.")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
The isConstant() check isn't useful. If it's a constant, then the
physReg() check will fail.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: baab6f18c9 ("aco: Optimize branching sequence during SSA elimination.")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
We would assemble an instruction writing vcc instead.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: 5ffc73896f ("aco/assembler: Fix v_cmpx with SDWA.")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
The old name is no longer accurate.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18077>
This optimization was broken for two reasons:
- s_and_saveexec has two operands, the copy value and exec
- s_and_saveexec has an exec read, so exec_write_used will always be true
before we find branch_exec_val_idx
Foz-DB Navi21:
Totals from 31453 (23.31% of 134913) affected shaders:
CodeSize: 204831260 -> 204831156 (-0.00%)
Instrs: 38157117 -> 38157091 (-0.00%)
Latency: 533708882 -> 531211721 (-0.47%); split: -0.47%, +0.00%
InvThroughput: 107088408 -> 106719188 (-0.34%); split: -0.35%, +0.00%
Copies: 2326179 -> 2502490 (+7.58%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18049>
We need to use the implicit destination.
Fixes: baab6f18c9 ("aco: Optimize branching sequence during SSA elimination.")
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18049>
Inserting in the instructions vector may invalidate the exec_val reference,
so do that last.
Fixes: baab6f18c9 ("aco: Optimize branching sequence during SSA elimination.")
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18049>
This makes it more likely to hit the fast path for count == 1
in the split_store_data function.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17923>
According to LLVM, we only need to care about VOPC which writes exec.
No fossil-db changes.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17697>
This prevents issues where we insert a s_waitcnt_vscnt(0) at the start of
a block or very end of the shader because we're joining two blocks (for
example, one with has_VMEM=true and the other with
has_branch_after_DS=true).
fossil-db (navi10):
Totals from 2441 (1.51% of 161220) affected shaders:
Instrs: 1383964 -> 1384094 (+0.01%); split: -0.07%, +0.08%
CodeSize: 7438212 -> 7438760 (+0.01%); split: -0.05%, +0.06%
Latency: 13780665 -> 13679664 (-0.73%); split: -1.53%, +0.80%
InvThroughput: 2950835 -> 2921511 (-0.99%); split: -1.06%, +0.07%
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17697>
For example, "DS -> branch -> VMEM -> branch -> DS".
fossil-db (navi10):
Totals from 639 (0.40% of 161220) affected shaders:
Instrs: 629090 -> 628254 (-0.13%); split: -0.19%, +0.06%
CodeSize: 3410164 -> 3406748 (-0.10%); split: -0.14%, +0.04%
Latency: 7834755 -> 7821011 (-0.18%); split: -0.70%, +0.52%
InvThroughput: 1369698 -> 1374495 (+0.35%); split: -0.12%, +0.47%
A lot of the fossil-db changes are noise.
threekingdoms.8db138826c386a62.1.foz/0b222ed175eebad0 is an example of a
shader that actually has this issue.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Fixes: c037ba1bb7 ("aco/gfx10: Mitigate LdsBranchVmemWARHazard.")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17697>
