Timothy Arceri
34c52d8cb9
nir: fix typo in lower_double options handling
...
Seems the intention was to check that both flags were not enabled
instead we were checking that the floor flag was both set and not
set so the result would always be false.
Fixes: 3749a6ecd2https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19642 >
2022-11-11 14:36:00 +00:00
Gert Wollny
917d992b32
nir/algeraic_opt: use double options too for lowering ftrunc@64
...
ftrunc@64 also might need lowering on fp64 only, especially now
that it might be introduced by nir_lower_int64.
Fixes: 29da985682https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19657 >
2022-11-11 09:29:31 +00:00
Alyssa Rosenzweig
45a111c21c
nir/opt_algebraic: Fuse c - a * b to FMA
...
Algebraically it is clear that
-(a * b) + c = (-a) * b + c = fma(-a, b, c)
But this is not clear from the NIR
('fadd', ('fneg', ('fmul', a, b)), c)
Add rules to handle this case specially. Note we don't necessarily want
to solve this by pushing fneg into fmul, because the rule opt_algebraic
(not the late part where FMA fusing happens) specifically pulls fneg out
of fmul to push fneg up multiplication chains.
Noticed in the big glmark2 "terrain" shader, which has a cycle count
reduced by 22% on Mali-G57 thanks to having this pattern a ton and being
FMA bound.
BEFORE: 1249 inst, 16.015625 cycles, 16.015625 fma, ... 632 quadwords
AFTER: 997 inst, 12.437500 cycles, .... 504 quadwords
Results on the same shader on AGX are also quite dramatic:
BEFORE: 1294 inst, 8600 bytes, 50 halfregs, ...
AFTER: 1154 inst, 8040 bytes, 50 halfregs, ...
Similar rules apply for fabs.
v2: Use a loop over the bit sizes (suggested by Emma).
shader-db on Valhall (open + small subset of closed), results on Bifrost
are similar:
total instructions in shared programs: 167975 -> 164970 (-1.79%)
instructions in affected programs: 92642 -> 89637 (-3.24%)
helped: 492
HURT: 25
helped stats (abs) min: 1.0 max: 252.0 x̄: 6.25 x̃: 3
helped stats (rel) min: 0.30% max: 20.18% x̄: 3.21% x̃: 2.91%
HURT stats (abs) min: 1.0 max: 5.0 x̄: 2.80 x̃: 3
HURT stats (rel) min: 0.46% max: 9.09% x̄: 3.89% x̃: 3.37%
95% mean confidence interval for instructions value: -6.95 -4.68
95% mean confidence interval for instructions %-change: -3.08% -2.65%
Instructions are helped.
total cycles in shared programs: 10556.89 -> 10538.98 (-0.17%)
cycles in affected programs: 265.56 -> 247.66 (-6.74%)
helped: 88
HURT: 2
helped stats (abs) min: 0.015625 max: 3.578125 x̄: 0.20 x̃: 0
helped stats (rel) min: 0.65% max: 22.34% x̄: 5.65% x̃: 4.25%
HURT stats (abs) min: 0.0625 max: 0.0625 x̄: 0.06 x̃: 0
HURT stats (rel) min: 8.33% max: 12.50% x̄: 10.42% x̃: 10.42%
95% mean confidence interval for cycles value: -0.28 -0.12
95% mean confidence interval for cycles %-change: -6.30% -4.30%
Cycles are helped.
total fma in shared programs: 1582.42 -> 1535.06 (-2.99%)
fma in affected programs: 871.58 -> 824.22 (-5.43%)
helped: 502
HURT: 9
helped stats (abs) min: 0.015625 max: 3.578125 x̄: 0.09 x̃: 0
helped stats (rel) min: 0.60% max: 25.00% x̄: 5.46% x̃: 4.82%
HURT stats (abs) min: 0.015625 max: 0.0625 x̄: 0.03 x̃: 0
HURT stats (rel) min: 4.35% max: 12.50% x̄: 6.22% x̃: 4.35%
95% mean confidence interval for fma value: -0.11 -0.08
95% mean confidence interval for fma %-change: -5.58% -4.93%
Fma are helped.
total cvt in shared programs: 665.55 -> 665.95 (0.06%)
cvt in affected programs: 61.72 -> 62.12 (0.66%)
helped: 33
HURT: 43
helped stats (abs) min: 0.015625 max: 0.359375 x̄: 0.04 x̃: 0
helped stats (rel) min: 1.01% max: 25.00% x̄: 6.68% x̃: 4.35%
HURT stats (abs) min: 0.015625 max: 0.109375 x̄: 0.04 x̃: 0
HURT stats (rel) min: 0.78% max: 38.46% x̄: 10.85% x̃: 6.90%
95% mean confidence interval for cvt value: -0.01 0.02
95% mean confidence interval for cvt %-change: 0.23% 6.24%
Inconclusive result (value mean confidence interval includes 0).
total quadwords in shared programs: 93376 -> 91736 (-1.76%)
quadwords in affected programs: 25376 -> 23736 (-6.46%)
helped: 169
HURT: 1
helped stats (abs) min: 8.0 max: 128.0 x̄: 9.75 x̃: 8
helped stats (rel) min: 1.52% max: 33.33% x̄: 8.35% x̃: 8.00%
HURT stats (abs) min: 8.0 max: 8.0 x̄: 8.00 x̃: 8
HURT stats (rel) min: 25.00% max: 25.00% x̄: 25.00% x̃: 25.00%
95% mean confidence interval for quadwords value: -11.18 -8.11
95% mean confidence interval for quadwords %-change: -8.95% -7.36%
Quadwords are helped.
total threads in shared programs: 4697 -> 4701 (0.09%)
threads in affected programs: 4 -> 8 (100.00%)
helped: 4
HURT: 0
helped stats (abs) min: 1.0 max: 1.0 x̄: 1.00 x̃: 1
helped stats (rel) min: 100.00% max: 100.00% x̄: 100.00% x̃: 100.00%
95% mean confidence interval for threads value: 1.00 1.00
95% mean confidence interval for threads %-change: 100.00% 100.00%
Threads are helped.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Marek Ol<C5><A1><C3><A1>k <marek.olsak@amd.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com> [v1]
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19312 >
2022-11-01 22:39:45 -04:00
Karol Herbst
e58c004870
nir/algebraic: add vec8/16 cmp lowering
...
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19150 >
2022-10-29 10:31:39 +00:00
Karol Herbst
5efbef833a
nir/algebraic: generalize vector_cmp lowering
...
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19150 >
2022-10-29 10:31:39 +00:00
Karol Herbst
1d6014f267
nir/algebraic: add 8 and 64 bit urol and uror lowering
...
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19150 >
2022-10-29 10:31:39 +00:00
Georg Lehmann
125741dbae
nir/opt_algebraic: Optimize various find_msb_rev patterns.
...
From dxvk, dxil-spirv, fxc, dxc and others.
Totals from 177 (0.13% of 134913) affected shaders:
CodeSize: 1079504 -> 1059872 (-1.82%)
Instrs: 195381 -> 192269 (-1.59%)
Latency: 3664137 -> 3631951 (-0.88%)
InvThroughput: 599479 -> 585675 (-2.30%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18951 >
2022-10-22 11:57:33 +02:00
Georg Lehmann
7505be3497
nir/opt_algebraic: Add an option to lower uclz.
...
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18951 >
2022-10-22 11:57:10 +02:00
Georg Lehmann
1e552b9c95
nir/opt_algebraic: Mirror optimizations for find_msb_rev.
...
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18951 >
2022-10-22 11:56:44 +02:00
Rhys Perry
1ae73bc076
nir/algebraic: optimize b<<a + c<<a
...
fossil-db (navi21):
Totals from 248 (0.18% of 135636) affected shaders:
Instrs: 85836 -> 85611 (-0.26%); split: -0.27%, +0.00%
CodeSize: 481304 -> 480332 (-0.20%); split: -0.21%, +0.00%
Latency: 9596559 -> 9596152 (-0.00%); split: -0.00%, +0.00%
InvThroughput: 1423707 -> 1423670 (-0.00%)
SClause: 3872 -> 3874 (+0.05%)
PreSGPRs: 5034 -> 5038 (+0.08%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19137 >
2022-10-20 18:57:23 +00:00
Alyssa Rosenzweig
ac2964dfbd
nir: Be smarter fusing ffma
...
If there is a single use of fmul, and that single use is fadd, it makes
sense to fuse ffma, as we already do. However, if there are multiple
uses, fusing may impede code gen. Consider the source fragment:
a = fmul(x, y)
b = fadd(a, z)
c = fmin(a, t)
d = fmax(b, c)
The fmul has two uses. The current ffma fusing is greedy and will
produce the following "optimized" code.
a = fmul(x, y)
b = ffma(x, y, z)
c = fmin(a, t)
d = fmax(b, c)
Actually, this code is worse! Instead of 1 fmul + 1 fadd, we now have 1
fmul + 1 ffma. In effect, two multiplies (and a fused add) instead of
one multiply and an add. Depending on the ISA, that could impede
scheduling or increase code size. It can also increase register
pressure, extending the live range.
It's tempting to gate on is_used_once, but that would hurt in cases
where we really do fuse everything, e.g.:
a = fmul(x, y)
b = fadd(a, z)
c = fadd(a, t)
For ISAs that fuse ffma, we expect that 2 ffma is faster than 1 fmul + 2
fadd. So what we really want is to fuse ffma iff the fmul will get
deleted. That occurs iff all uses of the fmul are fadd and will
themselves get fused to ffma, leaving fmul to get dead code eliminated.
That's easy to implement with a new NIR search helper, checking that all
uses are fadd.
shader-db results on Mali-G57 [open shader-db + subset of closed]:
total instructions in shared programs: 179491 -> 178991 (-0.28%)
instructions in affected programs: 36862 -> 36362 (-1.36%)
helped: 190
HURT: 27
total cycles in shared programs: 10573.20 -> 10571.75 (-0.01%)
cycles in affected programs: 72.02 -> 70.56 (-2.02%)
helped: 28
HURT: 1
total fma in shared programs: 1590.47 -> 1582.61 (-0.49%)
fma in affected programs: 319.95 -> 312.09 (-2.46%)
helped: 194
HURT: 1
total cvt in shared programs: 812.98 -> 813.03 (<.01%)
cvt in affected programs: 118.53 -> 118.58 (0.04%)
helped: 65
HURT: 81
total quadwords in shared programs: 98968 -> 98840 (-0.13%)
quadwords in affected programs: 2960 -> 2832 (-4.32%)
helped: 20
HURT: 4
total threads in shared programs: 4693 -> 4697 (0.09%)
threads in affected programs: 4 -> 8 (100.00%)
helped: 4
HURT: 0
v2: Update trace checksums for virgl due to numerical differences.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18814 >
2022-10-15 17:47:31 +00:00
Gert Wollny
2e50bf19cd
nir: move fusing csel and comparisons to opt_late_algebraic
...
With that simple comparisons are cleaned up properly.
This helps with some tesselation shaders on r600.
Shader-db stats R600/Cayman:
--------------------------------------------------------------
total dw in shared programs: 1621806 -> 1620884 (-0.06%)
dw in affected programs: 41650 -> 40728 (-2.21%)
helped: 211
HURT: 4
helped stats (abs) min: 2 max: 26 x̄: 4.46 x̃: 4
helped stats (rel) min: 0.30% max: 9.68% x̄: 2.87% x̃: 2.52%
HURT stats (abs) min: 2 max: 8 x̄: 5.00 x̃: 5
HURT stats (rel) min: 0.23% max: 1.67% x̄: 1.02% x̃: 1.09%
95% mean confidence interval for dw value: -4.81 -3.77
95% mean confidence interval for dw %-change: -3.03% -2.57%
Dw are helped.
total gprs in shared programs: 41192 -> 41182 (-0.02%)
gprs in affected programs: 731 -> 721 (-1.37%)
helped: 53
HURT: 45
helped stats (abs) min: 1 max: 3 x̄: 1.23 x̃: 1
helped stats (rel) min: 5.88% max: 40.00% x̄: 16.56% x̃: 14.29%
HURT stats (abs) min: 1 max: 2 x̄: 1.22 x̃: 1
HURT stats (rel) min: 7.69% max: 40.00% x̄: 19.42% x̃: 20.00%
95% mean confidence interval for gprs value: -0.37 0.16
95% mean confidence interval for gprs %-change: -3.92% 3.85%
Inconclusive result (value mean confidence interval includes 0).
total alu_groups in shared programs: 203677 -> 203632 (-0.02%)
alu_groups in affected programs: 2876 -> 2831 (-1.56%)
helped: 68
HURT: 30
helped stats (abs) min: 1 max: 4 x̄: 1.46 x̃: 1
helped stats (rel) min: 0.84% max: 25.00% x̄: 7.48% x̃: 5.41%
HURT stats (abs) min: 1 max: 6 x̄: 1.80 x̃: 1
HURT stats (rel) min: 1.98% max: 33.33% x̄: 10.09% x̃: 5.61%
95% mean confidence interval for alu_groups value: -0.81 -0.11
95% mean confidence interval for alu_groups %-change: -4.20% <.01%
Alu_groups are helped.
total loops in shared programs: 72 -> 72 (0.00%)
loops in affected programs: 0 -> 0
helped: 0
HURT: 0
total cf in shared programs: 88230 -> 88233 (<.01%)
cf in affected programs: 71 -> 74 (4.23%)
helped: 1
HURT: 4
helped stats (abs) min: 1 max: 1 x̄: 1.00 x̃: 1
helped stats (rel) min: 33.33% max: 33.33% x̄: 33.33% x̃: 33.33%
HURT stats (abs) min: 1 max: 1 x̄: 1.00 x̃: 1
HURT stats (rel) min: 1.89% max: 33.33% x̄: 17.14% x̃: 16.67%
95% mean confidence interval for cf value: -0.51 1.71
95% mean confidence interval for cf %-change: -24.20% 38.29%
Inconclusive result (value mean confidence interval includes 0).
total stack in shared programs: 3827 -> 3827 (0.00%)
stack in affected programs: 0 -> 0
helped: 0
HURT: 0
LOST: 0
GAINED: 0
Total CPU time (seconds): 45.32 -> 41.69 (-8.01%)
--------------------------------------------------------------
v2: Simplify replacement pattern (Rhys Perry)
v3: fix ws (Alexander Orzechowski)
v4: move the original lowering to opt_late_algebraic and
drop cleanup code (Alyssa)
v5: Add shader-sb stats (Alyssa)
Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18970 >
2022-10-14 13:08:15 +00:00
Georg Lehmann
bfb12a3b6a
nir/opt_algebraic: Optimize more (a cmp b ? a : b) to min/max.
...
Foz-DB Navi21:
Totals from 112 (0.08% of 134913) affected shaders:
CodeSize: 1618384 -> 1618172 (-0.01%); split: -0.06%, +0.04%
Instrs: 307695 -> 307535 (-0.05%); split: -0.05%, +0.00%
Latency: 3590228 -> 3589658 (-0.02%); split: -0.02%, +0.00%
InvThroughput: 563692 -> 563447 (-0.04%); split: -0.05%, +0.01%
Copies: 24541 -> 24519 (-0.09%); split: -0.10%, +0.01%
Branches: 13480 -> 13468 (-0.09%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18548 >
2022-09-30 11:10:52 +00:00
Rhys Perry
b301c33f65
nir/algebraic: optimize fabs(bcsel(b, fneg(a), a))
...
fossil-db (Sienna Cichlid):
Totals from 207 (0.15% of 134913) affected shaders:
VGPRs: 7152 -> 6928 (-3.13%)
CodeSize: 762404 -> 752888 (-1.25%)
MaxWaves: 6138 -> 6146 (+0.13%)
Instrs: 144031 -> 142184 (-1.28%)
Latency: 817783 -> 807286 (-1.28%)
InvThroughput: 151031 -> 147497 (-2.34%)
VClause: 1490 -> 1453 (-2.48%)
SClause: 3357 -> 3331 (-0.77%); split: -0.92%, +0.15%
Copies: 9632 -> 9555 (-0.80%); split: -0.81%, +0.01%
Branches: 4306 -> 4270 (-0.84%)
PreSGPRs: 11232 -> 11218 (-0.12%); split: -0.15%, +0.03%
PreVGPRs: 6307 -> 6121 (-2.95%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14772 >
2022-09-14 12:16:07 +00:00
Rhys Perry
c23411a970
nir/algebraic: optimize bits=umin(bits, 32-(offset&0x1f))
...
Optimizes patterns which are created by recent versions of vkd3d-proton,
when constant folding doesn't eliminate it entirely:
- ubitfield_extract(value, offset, umin(bits, 32-(offset&0x1f)))
- ibitfield_extract(value, offset, umin(bits, 32-(offset&0x1f)))
- bitfield_insert(base, insert, offset, umin(bits, 32-(offset&0x1f)))
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13225 >
2022-09-13 20:36:06 +00:00
Georg Lehmann
4d7fe94f3a
nir/opt_algebraic: Optimize unpacking of upcasts to 64bit integers.
...
Foz-DB Navi21:
Totals from 7 (0.01% of 134913) affected shaders:
CodeSize: 213364 -> 213028 (-0.16%)
Instrs: 38347 -> 38319 (-0.07%)
Latency: 780148 -> 779776 (-0.05%)
InvThroughput: 520098 -> 519851 (-0.05%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18435 >
2022-09-08 14:37:56 +00:00
Georg Lehmann
6eb4dfca23
nir/opt_algebraic: Optimize d3d9 pow with fmulz.
...
Foz-DB Navi21:
Totals from 69 (0.05% of 134913) affected shaders:
CodeSize: 255684 -> 253788 (-0.74%); split: -0.74%, +0.00%
Instrs: 46307 -> 46052 (-0.55%); split: -0.55%, +0.00%
Latency: 533255 -> 530742 (-0.47%); split: -0.48%, +0.01%
InvThroughput: 110001 -> 109156 (-0.77%)
VClause: 839 -> 844 (+0.60%); split: -1.19%, +1.79%
SClause: 1411 -> 1395 (-1.13%)
Copies: 1828 -> 1816 (-0.66%); split: -1.09%, +0.44%
PreSGPRs: 2243 -> 2232 (-0.49%)
PreVGPRs: 2213 -> 2192 (-0.95%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18145 >
2022-08-31 17:07:24 +00:00
Georg Lehmann
9c2c47884d
nir/opt_algebraic: Optimize check for single bit.
...
Foz-DB Navi21:
Totals from 3239 (2.40% of 134913) affected shaders:
SpillSGPRs: 110 -> 102 (-7.27%)
CodeSize: 17426512 -> 17344808 (-0.47%); split: -0.48%, +0.01%
Instrs: 3194264 -> 3179366 (-0.47%)
Latency: 20498012 -> 20481419 (-0.08%); split: -0.08%, +0.00%
InvThroughput: 3311738 -> 3311282 (-0.01%); split: -0.02%, +0.00%
SClause: 145810 -> 145690 (-0.08%)
Copies: 171748 -> 169009 (-1.59%); split: -1.63%, +0.03%
Branches: 86610 -> 86370 (-0.28%)
PreSGPRs: 138036 -> 137104 (-0.68%)
PreVGPRs: 138540 -> 138545 (+0.00%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17429 >
2022-08-31 18:36:33 +02:00
Daniel Schürmann
9b843f8e4a
nir/opt_algebraic: a & ~a -> 0
...
Also re-ordered some optimizations for better readability.
Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18250 >
2022-08-30 14:10:22 +00:00
Emma Anholt
f6c5b1d6c6
nir: Split usub_sat lowering flag from uadd_sat.
...
Intel vec4 would like to do uadd_sat, but use lowering for usub_sat.
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17637 >
2022-07-22 17:54:28 +00:00
Georg Lehmann
aac8ddae2f
nir/opt_algebraic: Optimize [ui](add|sub)_sat with 0.
...
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17468 >
2022-07-13 07:34:09 +00:00
Rhys Perry
bc1ea2fda9
nir/algebraic: optimize bcsel(c, fsin/cos_amd(a), fsin/cos_amd(b))
...
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10587 >
2022-07-07 22:18:08 +00:00
Ian Romanick
a2a2fbc510
nir/algebraic: Fix NaN-unsafe fcsel patterns
...
For example, the proof for this pattern
(('bcsel', ('flt', 'a@32', 0), 'b@32', 'c@32'), ('fcsel_ge', a, c, b)),
would be
bcsel(a < 0, b, c)
bcsel(!(a < 0), c, b)
bcsel(a >= 0, c, b)
fcsel_ge(a, c, b)
However, !(a < 0) => (a >= 0) is well known to produce different
results if `a` is NaN.
Instead of that replacement, use this replacement:
bcsel(a < 0, b, c)
bcsel(-0 < -a, b, c)
bcsel(0 < -a, b, c)
fcsel_gt(-a, b, c)
This is NaN-safe and exact.
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Fixes: 0f5b3c37c5https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17048 >
2022-06-22 19:26:59 +00:00
Georg Lehmann
bfc25d6ec9
nir: Add optional lowering for mul_32x16.
...
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13895 >
2022-06-01 17:09:25 +00:00
Jason Ekstrand
836ff4b586
nir/algebraic: Add two more pack/unpack rules
...
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16591 >
2022-05-23 14:10:54 +00:00
Gert Wollny
3749a6ecd2
nir: honor lower_double options for ffloor and ffract
...
v2: Don't lower ffloor@64 to ffract@64 when both ops are
to be lowered. Settle on ffloor in opt_algebraic because
in can be lowered to other ops in lower_double_ops.
Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>(v1)
Jason Ekstrand <jason.ekstrand@collabora.com> (v1)
Reviewed-by: Emma Anholt <emma@anholt.net> (v1)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16431 >
2022-05-16 15:03:05 +00:00
Georg Lehmann
bc5c68fc08
nir/opt_algebraic: Optimize Doom Eternal's word extract by LSB.
...
Foz-db GFX10_3:
Totals from 419 (0.31% of 134913) affected shaders:
CodeSize: 4126032 -> 4121756 (-0.10%)
Instrs: 783608 -> 782541 (-0.14%)
Latency: 7889664 -> 7888521 (-0.01%); split: -0.02%, +0.00%
InvThroughput: 1315690 -> 1314863 (-0.06%); split: -0.06%, +0.00%
VClause: 11826 -> 11830 (+0.03%)
SClause: 27736 -> 27734 (-0.01%)
Copies: 50493 -> 50428 (-0.13%); split: -0.13%, +0.01%
PreSGPRs: 23264 -> 23265 (+0.00%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16436 >
2022-05-12 17:10:41 +00:00
Jason Ekstrand
df1876f615
nir: Mark negative re-distribution on fadd as imprecise
...
Otherwise, it would mutate `fneg(fadd(-0, 0))` into `fadd(0, -0)` which
isn't correct since -0 + (+0) = +0 + (-0) = +0.
This fixes the OpenCL contraction tests on Iris.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16041 >
2022-05-12 00:05:10 +00:00
Georg Lehmann
60c9a45562
nir/opt_algebraic: Simple xor/ishr optimizations.
...
The first pattern here removes the xor-swap pattern.
Foz-DB GFX10_3:
Totals from 305 (0.23% of 134913) affected shaders:
CodeSize: 1589040 -> 1585164 (-0.24%)
Instrs: 284344 -> 283375 (-0.34%)
Latency: 4205148 -> 4198472 (-0.16%); split: -0.16%, +0.00%
InvThroughput: 708745 -> 708739 (-0.00%)
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16411 >
2022-05-10 19:29:31 +00:00
Georg Lehmann
66e917fff6
nir/opt_algebraic: Fix mask in shift by constant combining.
...
The comment above is correct, but the code to calculate the mask was broken.
No Foz-db changes outside of noise.
Fixes: 0e6581b87dhttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15990 >
2022-05-10 18:47:21 +00:00
Karol Herbst
a2c9e1cb50
nir: add 16 and 64 bit fisnormal lowering
...
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16206 >
2022-04-28 18:36:52 +00:00
Gert Wollny
47d3f7c69f
nir: Don't optimize to 64 bit fsub if the driver doesn't support it
...
Fixes: a4840e15abhttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16130 >
2022-04-27 00:01:20 +00:00
Jason Ekstrand
1755730362
nir: Lower all bit sizes of usub_borrow
...
It's not clear why this is restricted to 32-bit besides that being the
only bit size where GLSL has an intrinsic for this. All drivers that
set this probably want it lowered for all bit sizes as far as I can
tell.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6353
Fixes: 8a3e344180https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16146 >
2022-04-25 21:27:09 +00:00
Emma Anholt
e4aa5f7889
nir: Skip fround_even on already-integral values.
...
Just like the other make-the-float-an-integer opcodes. Noticed in a
gallium nine shader run through TGSI-to-NIR, where the array index had
been floored by the user, but got implicitly rounded by DX9 array
indexing.
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15870 >
2022-04-16 13:07:09 -07:00
Emma Anholt
6947016b46
nir: Add lowering for fround_even on r300.
...
When we put NIR in the compiler stack for r300, indirect addressing broke
for gallium nine. DX's array indirects round the float value, so the DX
shader gets mapped to a TGSI "ARR ADDR[0] src.x" instruction. Translating
that to NIR maps to r0[f2i32(fround(src.x))]. While we might hope that in
translation back using nir-to-tgsi after optimization we would recognize
the construct and emit ARR again, that's going to be error prone (think
"what if src.x is in a NIR register?") so we need a fallback plan. r300
will be able to handle this lowering, so get it in place first to fix the
regression.
Fixes : #6297
Fixes: 7d2ea9b0edhttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15870 >
2022-04-16 13:07:09 -07:00
Georg Lehmann
16be909936
nir: Add an option to lower 64bit iadd_sat.
...
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15421 >
2022-03-28 20:02:52 +00:00
Georg Lehmann
922916bf64
nir: Move lower_usub_sat64 to nir_lower_int64_options.
...
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15421 >
2022-03-28 20:02:52 +00:00
Ian Romanick
06eb9fb125
nir/algebraic: Optimize some cases of (sXX(a, b) != 0.0)
...
I noticed the SGE case while looking at the output of
shaders/closed/steam/trine-2/fp-3.shader_test on i915g. These are
especially bad on i915 that needs two instructions to implement SNE.
An alternative would be to duplicate the sne(sXX(a, b), 0.0) rules in an
algebraic pass that occurs after bool_to_float. Doing the work earlier
seems preferable.
i915
total instructions in shared programs: 788274 -> 788223 (<.01%)
instructions in affected programs: 666 -> 615 (-7.66%)
helped: 5
HURT: 0
helped stats (abs) min: 9 max: 12 x̄: 10.20 x̃: 9
helped stats (rel) min: 5.00% max: 11.11% x̄: 8.12% x̃: 8.16%
95% mean confidence interval for instructions value: -12.24 -8.16
95% mean confidence interval for instructions %-change: -10.81% -5.43%
Instructions are helped.
LOST: 0
GAINED: 2
The two gained shaders are assembly fragment programs in Euro Truck
Simulator 2.
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15210 >
2022-03-03 00:07:58 +00:00
Bas Nieuwenhuizen
d1530a3f3b
Revert "nir/algebraic: distribute fmul(fadd(a, b), c) when b and c are constants"
...
This reverts commit a1af902531https://gitlab.freedesktop.org/mesa/mesa/-/issues/5423
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14532 >
2022-01-21 16:58:11 +00:00
Rhys Perry
495debebad
nir/algebraic: optimize expressions using fmulz/ffmaz
...
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13436 >
2022-01-20 22:54:42 +00:00
Rhys Perry
f2fbba7920
nir/algebraic: optimize open-coded fmulz/ffmaz
...
This pattern will be found in future versions of D3D9 DXVK.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13436 >
2022-01-20 22:54:42 +00:00
Rhys Perry
312a284980
nir/algebraic: add ignore_exact() wrapper
...
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13436 >
2022-01-20 22:54:42 +00:00
Rhys Perry
7f05ea3793
nir: add nir_op_fmulz and nir_op_ffmaz
...
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13436 >
2022-01-20 22:54:42 +00:00
Connor Abbott
9c9e8c3349
nir: Reorder ffma and fsub combining
...
It's relatively common to do something like "a * b - c", which on most
GPUs can be implemented in a single instruction. Before
opt_algebraic_late this will be something like
"fadd(fmul(a, b), fneg(c))", and we want to turn it info
"ffma(a, b, fneg(c))". But because the fsub pattern was first we
instead turned it into "fsub(fmul(a, b), c)". Fix this by reordering
them.
Selected shader-db results on freedreno:
total instructions in shared programs: 1561330 -> 1551619 (-0.62%)
instructions in affected programs: 780272 -> 770561 (-1.24%)
helped: 1941
HURT: 491
helped stats (abs) min: 1 max: 147 x̄: 7.98 x̃: 4
helped stats (rel) min: 0.07% max: 30.77% x̄: 4.36% x̃: 3.17%
HURT stats (abs) min: 1 max: 307 x̄: 11.76 x̃: 5
HURT stats (rel) min: 0.09% max: 18.71% x̄: 2.26% x̃: 1.38%
95% mean confidence interval for instructions value: -4.57 -3.41
95% mean confidence interval for instructions %-change: -3.21% -2.84%
Instructions are helped.
total nops in shared programs: 358926 -> 356263 (-0.74%)
nops in affected programs: 167116 -> 164453 (-1.59%)
helped: 1395
HURT: 859
helped stats (abs) min: 1 max: 108 x̄: 6.80 x̃: 3
helped stats (rel) min: 0.17% max: 100.00% x̄: 19.15% x̃: 10.57%
HURT stats (abs) min: 1 max: 307 x̄: 7.95 x̃: 3
HURT stats (rel) min: 0.00% max: 381.82% x̄: 20.04% x̃: 10.00%
95% mean confidence interval for nops value: -1.77 -0.59
95% mean confidence interval for nops %-change: -5.55% -2.87%
Nops are helped.
total non-nops in shared programs: 1202404 -> 1195356 (-0.59%)
non-nops in affected programs: 496682 -> 489634 (-1.42%)
helped: 1951
HURT: 265
helped stats (abs) min: 1 max: 39 x̄: 4.02 x̃: 3
helped stats (rel) min: 0.07% max: 15.38% x̄: 2.97% x̃: 1.96%
HURT stats (abs) min: 1 max: 22 x̄: 2.97 x̃: 2
HURT stats (rel) min: 0.05% max: 10.00% x̄: 1.14% x̃: 0.75%
95% mean confidence interval for non-nops value: -3.38 -2.99
95% mean confidence interval for non-nops %-change: -2.60% -2.36%
Non-nops are helped.
total systall in shared programs: 288317 -> 292975 (1.62%)
systall in affected programs: 87876 -> 92534 (5.30%)
helped: 388
HURT: 431
helped stats (abs) min: 1 max: 214 x̄: 14.39 x̃: 8
helped stats (rel) min: 0.25% max: 100.00% x̄: 22.12% x̃: 11.96%
HURT stats (abs) min: 1 max: 232 x̄: 23.77 x̃: 7
HURT stats (rel) min: 0.00% max: 1300.00% x̄: 51.71% x̃: 17.30%
95% mean confidence interval for systall value: 3.07 8.30
95% mean confidence interval for systall %-change: 9.49% 23.97%
Systall are HURT.
(The systall hurt is probably just due to having having fewer
instructions to hide latency with.)
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14554 >
2022-01-18 17:44:50 +00:00
Danylo Piliaiev
b8d486f298
nir/algebraic: Separate has_dot_4x8 into has_sdot_4x8 and has_udot_4x8
...
Adreno GPUs has native instruction for unsigned and mixed dot_4x8 but
not signed dot product.
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13986 >
2022-01-10 13:20:39 +02:00
Daniel Schürmann
17ecd0b31a
nir/opt_algebraic: lower fneg_hi/lo to fmul
...
This pattern, found in the FSR upscaling shader,
helps the vectorization efforts by keeping the
chain of vectorized instructions intact.
Radeon can optimize it to per-component fneg modifiers.
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13688 >
2021-12-21 13:23:37 +01:00
Rhys Perry
403ae3b48e
nir/algebraic: optimize more 64-bit imul with constant source
...
Two 64-bit shifts and an addition are usually faster than the several
multiplications nir_lower_int64 creates.
No fossil-db changes.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14227 >
2021-12-17 18:51:24 +00:00
Rhys Perry
a2d8c5b26d
nir/algebraic: optimize a*#b & -4
...
fossil-db (Sienna Cichlid):
Totals from 611 (0.47% of 128647) affected shaders:
CodeSize: 3096680 -> 3090976 (-0.18%)
Instrs: 570494 -> 569249 (-0.22%)
Latency: 5765865 -> 5759619 (-0.11%)
InvThroughput: 969840 -> 967608 (-0.23%)
VClause: 9690 -> 9688 (-0.02%)
Copies: 42884 -> 42894 (+0.02%); split: -0.01%, +0.03%
PreVGPRs: 28290 -> 28288 (-0.01%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13752 >
2021-12-03 13:41:07 +00:00
Rhys Perry
12294026d5
nir/algebraic: optimize Cyberpunk 2077's open-coded bitfieldReverse()
...
fossil-db (Sienna Cichlid):
Totals from 9 (0.01% of 128647) affected shaders:
CodeSize: 29900 -> 28640 (-4.21%)
Instrs: 5677 -> 5443 (-4.12%)
Latency: 96561 -> 95025 (-1.59%)
Copies: 571 -> 544 (-4.73%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13673 >
2021-11-05 09:31:04 +00:00
Jason Ekstrand
b71bdc3404
nir/algebraic: Add some opts for comparisons of comparisons
...
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13167 >
2021-10-07 18:21:11 +00:00
