Commit graph

4202 commits

Author SHA1 Message Date
Georg Lehmann
6b9d28ab9b aco/insert_fp_mode: insert fp mode in reverse
This allows us to skip the mode set by changing
the initial mode in the command stream.

Foz-DB Navi48:
Totals from 14 (0.02% of 82405) affected shaders:
Latency: 79417 -> 79438 (+0.03%)

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:22 +00:00
Georg Lehmann
7212a75c5e aco/insert_fp_mode: exclude some instructions that will never round
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:22 +00:00
Georg Lehmann
d6356191b9 aco: add fma_mix opcodes with rtz fp16 rounding
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:22 +00:00
Rhys Perry
f0f53e624c aco/tests: remove vcc definitions from p_call
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
The version of instruction selection that got merged doesn't have vcc
definitions, so this shouldn't either.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
adf5c7cba4 aco: remove dead p_call code in live_var_analysis
The version of instruction selection that got merged doesn't have vcc
definitions.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
ba798120c6 aco/ra: split blocking vectors if needed when handling fixed operands
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
5ebefceb42 aco/ra: move split_blocking_vectors higher
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
24fe4a5b58 aco/ra: copy precolor affinities to p_create_vector/p_split_vector
fossil-db (navi31):
Totals from 7 (0.01% of 84369) affected shaders:
Instrs: 2742 -> 2704 (-1.39%); split: -1.82%, +0.44%
CodeSize: 15300 -> 15052 (-1.62%); split: -1.93%, +0.31%
VGPRs: 516 -> 504 (-2.33%)
Latency: 12478 -> 12504 (+0.21%); split: -0.24%, +0.45%
InvThroughput: 2350 -> 2300 (-2.13%)
Copies: 350 -> 272 (-22.29%)
VALU: 1626 -> 1592 (-2.09%)
VOPD: 280 -> 236 (-15.71%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39387>
2026-01-20 10:53:18 +00:00
Rhys Perry
893e9d7595 aco: don't spill no-op copies of input parameters in preserved registers
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
8a3cc7200e aco/sched: don't use previously unused preserved registers
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
da1dca28a7 aco/ra: consider already-used preserved registers to be free
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
103160ff2f aco/ra: prefer clobbered registers in get_reg_specified()
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
2b20d568e0 aco/ra: prefer clobbered registers in callees
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
21b1118a08 aco/ra: refactor get_reg_impl and get_regs_for_copies using tuples
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:52 +00:00
Rhys Perry
61919b93fa aco/ra: always abort loop in get_regs_for_copies() if candidate is worse
fossil-db (navi31):
Totals from 2 (0.00% of 79825) affected shaders:
Instrs: 58162 -> 58178 (+0.03%)
CodeSize: 299132 -> 299212 (+0.03%)
Latency: 72864 -> 72861 (-0.00%)
VClause: 977 -> 984 (+0.72%)
Copies: 7433 -> 7428 (-0.07%)
Branches: 1602 -> 1603 (+0.06%)
VALU: 37100 -> 37095 (-0.01%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:51 +00:00
Rhys Perry
137f71895d aco/ra: always prefer earlier regs in get_reg_impl() if costs are the same
Before, this would have updated best_win if it encounters an assignment
with the same num_moves and num_vars but is aligned, even if the previous
best assignment was also aligned.

fossil-db (navi31):
Totals from 7693 (9.64% of 79825) affected shaders:
Instrs: 9767559 -> 9766370 (-0.01%); split: -0.11%, +0.09%
CodeSize: 51008244 -> 51002396 (-0.01%); split: -0.09%, +0.08%
VGPRs: 497224 -> 497212 (-0.00%); split: -0.00%, +0.00%
Latency: 80435654 -> 80373798 (-0.08%); split: -0.20%, +0.12%
InvThroughput: 12426170 -> 12414490 (-0.09%); split: -0.15%, +0.06%
VClause: 217725 -> 217438 (-0.13%); split: -0.59%, +0.46%
SClause: 230555 -> 230551 (-0.00%); split: -0.00%, +0.00%
Copies: 758396 -> 757988 (-0.05%); split: -0.81%, +0.75%
Branches: 200669 -> 200662 (-0.00%); split: -0.01%, +0.01%
VALU: 6040415 -> 6039979 (-0.01%); split: -0.10%, +0.09%
SALU: 960327 -> 960671 (+0.04%); split: -0.03%, +0.07%
VOPD: 426 -> 458 (+7.51%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:50 +00:00
Rhys Perry
0a744ba735 aco: increase max_reg_demand to help avoid preserved VGPRs
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:50 +00:00
Rhys Perry
bf56f1efd3 aco: move update(fixed_reg_demand) into update_vgpr_sgpr_demand
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:50 +00:00
Rhys Perry
a932bc40d3 aco/ra: don't use update_vgpr_sgpr_demand in increase_register_file
In a future commit, update_vgpr_sgpr_demand() should only be given the
register demand calculated from the number of live temporaries, not the
maximum register used.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:49 +00:00
Rhys Perry
6e618ac327 aco: don't try to preserve SCC in callees
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:48 +00:00
Rhys Perry
88ee78c6b7 aco: track number of post-RA spilled vgprs/sgprs
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:47 +00:00
Rhys Perry
d2a9122cfa aco/tests: add function call regalloc tests
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
aa09335485 aco/ra: emit p_split_vector after p_parallelcopy
This fixes situations where the variable being split is a parallelcopy
definition.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
7caa0ab897 aco/ra: fix split_blocking_vectors with some subdword vectors
For example, v6b at byte=2 should be split into v2b,v1 instead of v1,v2b.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
667771950d aco/ra: update register file when recreating blocking vectors
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
5cf991ecfb aco/ra: use original name for blocking vectors rename
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
829e3a6a7d aco/ra: fix operands when recreating blocking vectors
It is likely that these have been moved.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
6e6e40ab5c aco/ra: discard tmp_file after get_regs_for_copies fails
get_regs_for_copies() might change tmp_file, so we need to create a new
one.

Recursively handle_call() with an modified register_file instead of
repeating get_regs_for_copies().

This also updates register_file for the caller.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
159154a741 aco/ra: remove dead code in split_blocking_vectors
collect_vars_from_bitset() from the caller should have already cleared
these.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
89d06f7db5 aco: add RA validation for p_call
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
d0801c9eb7 aco: always run RA validation during tests
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
bf30a57440 aco/ra: omit renaming when necessary when moving copy definitions
This should resolve the FIXME.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:41 +00:00
Rhys Perry
8dcfce3587 aco/ra: refactor update_renames slightly
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:39 +00:00
Emma Anholt
ed8676dc28 nir: Rename the unit_test_*_amd intrinics to be un-vendored.
We'll reuse these from the nir_opt_algebraic_pattern_test.

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39076>
2026-01-15 19:09:37 +00:00
Natalie Vock
cc81c7de23 nir,aco: Clean up useless lowering of sbt_base_amd
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:07 +00:00
Natalie Vock
0a1911b220 radv,aco: Use function call structure for RT programs
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:07 +00:00
Natalie Vock
c5d796c902 radv/rt: Use function call structure in NIR lowering
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:06 +00:00
Natalie Vock
06c2e90e35 aco: Note if a parameter needs to be explicitly preserved
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Natalie Vock
2bf403ccec aco: Remove bypass_reg_preservation
Not needed anymore

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Natalie Vock
8525993d02 aco/ra: Move register preservation logic in last block to p_return
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Natalie Vock
eb86ae3b8c aco/spill_preserved: Fix spilled VGPR overflow handling
We modified the spill offset inside the function, which made subsequent
calls to emit_vgpr_spills_reloads reload from the wrong offset.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Rhys Perry
7a09e4a740 aco: use correct addition opcodes in gfx6-8 RT prolog
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Fixes: 60dd9d797e ("aco: Swizzle ray launch IDs in the RT prolog")
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39232>
2026-01-14 11:23:42 +00:00
Rhys Perry
da728d5a1a aco: micro-optimize ray launch ID swizzling
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39232>
2026-01-14 11:23:42 +00:00
Natalie Vock
473cf6046a aco/spill_preserved: Preserve linear VGPRs even if they aren't p_spill operands
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
2026-01-12 21:46:50 +00:00
Natalie Vock
1ef2691221 aco/spill: Fix preserved reload operand update
p_logical_end is actually after p_reload_preserved, so this didn't do
anything.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
2026-01-12 21:46:50 +00:00
Natalie Vock
548062f10e aco/insert_waitcnt: Don't determine linearity by reg number
VGPRs can be linear too, and RT function calls will add VMEM
instructions acting on linear VGPRs. Using the linear VGPR in a block
with only linear preds will cause the pass to incorrectly skip inserting
s_waitcnt.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
2026-01-12 21:46:50 +00:00
Natalie Vock
7c12603933 aco/lower_to_hw_instr: Preserve linearity of lowered linear VGPRs
So subsequent passes like waitcnt insertion can know these are linear.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
2026-01-12 21:46:50 +00:00
Natalie Vock
0d93e8ce54 aco: Don't insert p_reload_preserved in loops
This can't work.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
2026-01-12 21:46:50 +00:00
Natalie Vock
c816f699b2 aco/spill_preserved: Only reload linear VGPRs at end
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
2026-01-12 21:46:50 +00:00
Natalie Vock
897c95c37e aco: Include arbitrarily fixed registers in max_reg_demand
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39157>
2026-01-12 21:46:50 +00:00