Commit graph

4210 commits

Author SHA1 Message Date
Natalie Vock
e8f1dc687c aco: Use parameter assignment hints for any-hit shaders
Query the signature of the traversal function stored in the any-hit
shader and make the parameter locations between the two match up, to
remove unnecessary movs inside the traversal loop.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39314>
2026-01-20 21:49:55 +00:00
Natalie Vock
a32709674a aco: Add parameter assignment hints
Parameter assignment hints allow to influence parameter register
assignment logic with user-specified affinities. If there is an affinity
declared for a parameter, the assignment logic will try to match the
registers a parameter and its affinity are assigned.

It also allows to hint that certain registers are not suitable for
assigning parameters to and should be avoided.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39314>
2026-01-20 21:49:55 +00:00
Natalie Vock
2d6ecf303a aco: Add and use nir_abi_to_aco helper
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39314>
2026-01-20 21:49:55 +00:00
Natalie Vock
a03e9287c3 radv/rt: Compile ahit/isec shaders to asm
We can express any-hit/intersection shaders as functions, too.
Any-hit/Intersection shaders need the usual parameters like launch
IDs/descriptor data/ray properties, origin, direction/etc., but also
some special parameters related to traversal state. Any-hit/intersection
shaders need to return whether the hit was accepted and/or traversal
should be terminated, as well as the intersection T value (for
intersection shaders). Both any-hit and intersection shaders also need
to be passed hit attributes via parameters. Closest-Hit shaders need
those too, but we pass them out-of-band via LDS. LDS is used for the
traversal stack when any-hit/intersection shaders, so we need to pass
them via parameters.

Hit attributes are similar to ray payloads in the sense that they're
dynamically sized depending on how much space the application uses.
However, unlike ray payloads, hit attribute sizes have a strict upper
bound of 8 dwords. To make managing parameters easier, we put all hit
attributes in a single vector parameter with 0-8 components. This
prevents having a function with two sets of arbitrary numbers of
parameters.

This commit sets up ahit/isec function signatures and implements
lowering for ahit/isec-specific intrinsics in the context of these
functions. Subsequent commits will merely have to call into these
functions to execute a separate-compiled any-hit/intersection shader.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39314>
2026-01-20 21:49:55 +00:00
Natalie Vock
b52adac42c aco: Tweak ABI register param limits
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39314>
2026-01-20 21:49:54 +00:00
Natalie Vock
7a2f050daa aco: Put boolean parameters inside SGPRs
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39314>
2026-01-20 21:49:54 +00:00
Georg Lehmann
939b4a6476 aco/optimizer: apply v_cvt_pkrtz_f16_f32 as fma_mix to operands
Foz-DB Navi21:
Totals from 2085 (2.14% of 97591) affected shaders:
Instrs: 4880879 -> 4882355 (+0.03%); split: -0.04%, +0.07%
CodeSize: 26869332 -> 26881744 (+0.05%); split: -0.02%, +0.06%
VGPRs: 93944 -> 94160 (+0.23%); split: -0.06%, +0.29%
Latency: 40035558 -> 40035595 (+0.00%); split: -0.02%, +0.02%
InvThroughput: 10333800 -> 10329093 (-0.05%); split: -0.06%, +0.01%
VClause: 139147 -> 139148 (+0.00%)
Copies: 454527 -> 454656 (+0.03%); split: -0.00%, +0.03%
VALU: 3214838 -> 3211105 (-0.12%)

Foz-DB Navi48:
Totals from 2349 (2.41% of 97637) affected shaders:
Instrs: 6471998 -> 6471817 (-0.00%); split: -0.05%, +0.05%
CodeSize: 34793372 -> 34808748 (+0.04%); split: -0.02%, +0.06%
VGPRs: 141804 -> 142560 (+0.53%)
Latency: 45225910 -> 45226000 (+0.00%); split: -0.01%, +0.01%
InvThroughput: 9152634 -> 9149850 (-0.03%); split: -0.04%, +0.01%
VClause: 148536 -> 148537 (+0.00%)
Copies: 527206 -> 527336 (+0.02%); split: -0.01%, +0.03%
VALU: 3491701 -> 3487347 (-0.12%); split: -0.12%, +0.00%
VOPD: 669 -> 683 (+2.09%); split: +2.69%, -0.60%

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:23 +00:00
Georg Lehmann
c6b74705dd aco/optimizer: support fma_mix with rtz
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:23 +00:00
Georg Lehmann
6b9d28ab9b aco/insert_fp_mode: insert fp mode in reverse
This allows us to skip the mode set by changing
the initial mode in the command stream.

Foz-DB Navi48:
Totals from 14 (0.02% of 82405) affected shaders:
Latency: 79417 -> 79438 (+0.03%)

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:22 +00:00
Georg Lehmann
7212a75c5e aco/insert_fp_mode: exclude some instructions that will never round
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:22 +00:00
Georg Lehmann
d6356191b9 aco: add fma_mix opcodes with rtz fp16 rounding
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38815>
2026-01-20 14:48:22 +00:00
Rhys Perry
f0f53e624c aco/tests: remove vcc definitions from p_call
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
The version of instruction selection that got merged doesn't have vcc
definitions, so this shouldn't either.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
adf5c7cba4 aco: remove dead p_call code in live_var_analysis
The version of instruction selection that got merged doesn't have vcc
definitions.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
ba798120c6 aco/ra: split blocking vectors if needed when handling fixed operands
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
5ebefceb42 aco/ra: move split_blocking_vectors higher
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39390>
2026-01-20 13:33:16 +00:00
Rhys Perry
24fe4a5b58 aco/ra: copy precolor affinities to p_create_vector/p_split_vector
fossil-db (navi31):
Totals from 7 (0.01% of 84369) affected shaders:
Instrs: 2742 -> 2704 (-1.39%); split: -1.82%, +0.44%
CodeSize: 15300 -> 15052 (-1.62%); split: -1.93%, +0.31%
VGPRs: 516 -> 504 (-2.33%)
Latency: 12478 -> 12504 (+0.21%); split: -0.24%, +0.45%
InvThroughput: 2350 -> 2300 (-2.13%)
Copies: 350 -> 272 (-22.29%)
VALU: 1626 -> 1592 (-2.09%)
VOPD: 280 -> 236 (-15.71%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39387>
2026-01-20 10:53:18 +00:00
Rhys Perry
893e9d7595 aco: don't spill no-op copies of input parameters in preserved registers
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
8a3cc7200e aco/sched: don't use previously unused preserved registers
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
da1dca28a7 aco/ra: consider already-used preserved registers to be free
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
103160ff2f aco/ra: prefer clobbered registers in get_reg_specified()
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
2b20d568e0 aco/ra: prefer clobbered registers in callees
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:53 +00:00
Rhys Perry
21b1118a08 aco/ra: refactor get_reg_impl and get_regs_for_copies using tuples
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:52 +00:00
Rhys Perry
61919b93fa aco/ra: always abort loop in get_regs_for_copies() if candidate is worse
fossil-db (navi31):
Totals from 2 (0.00% of 79825) affected shaders:
Instrs: 58162 -> 58178 (+0.03%)
CodeSize: 299132 -> 299212 (+0.03%)
Latency: 72864 -> 72861 (-0.00%)
VClause: 977 -> 984 (+0.72%)
Copies: 7433 -> 7428 (-0.07%)
Branches: 1602 -> 1603 (+0.06%)
VALU: 37100 -> 37095 (-0.01%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:51 +00:00
Rhys Perry
137f71895d aco/ra: always prefer earlier regs in get_reg_impl() if costs are the same
Before, this would have updated best_win if it encounters an assignment
with the same num_moves and num_vars but is aligned, even if the previous
best assignment was also aligned.

fossil-db (navi31):
Totals from 7693 (9.64% of 79825) affected shaders:
Instrs: 9767559 -> 9766370 (-0.01%); split: -0.11%, +0.09%
CodeSize: 51008244 -> 51002396 (-0.01%); split: -0.09%, +0.08%
VGPRs: 497224 -> 497212 (-0.00%); split: -0.00%, +0.00%
Latency: 80435654 -> 80373798 (-0.08%); split: -0.20%, +0.12%
InvThroughput: 12426170 -> 12414490 (-0.09%); split: -0.15%, +0.06%
VClause: 217725 -> 217438 (-0.13%); split: -0.59%, +0.46%
SClause: 230555 -> 230551 (-0.00%); split: -0.00%, +0.00%
Copies: 758396 -> 757988 (-0.05%); split: -0.81%, +0.75%
Branches: 200669 -> 200662 (-0.00%); split: -0.01%, +0.01%
VALU: 6040415 -> 6039979 (-0.01%); split: -0.10%, +0.09%
SALU: 960327 -> 960671 (+0.04%); split: -0.03%, +0.07%
VOPD: 426 -> 458 (+7.51%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:50 +00:00
Rhys Perry
0a744ba735 aco: increase max_reg_demand to help avoid preserved VGPRs
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:50 +00:00
Rhys Perry
bf56f1efd3 aco: move update(fixed_reg_demand) into update_vgpr_sgpr_demand
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:50 +00:00
Rhys Perry
a932bc40d3 aco/ra: don't use update_vgpr_sgpr_demand in increase_register_file
In a future commit, update_vgpr_sgpr_demand() should only be given the
register demand calculated from the number of live temporaries, not the
maximum register used.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:49 +00:00
Rhys Perry
6e618ac327 aco: don't try to preserve SCC in callees
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:48 +00:00
Rhys Perry
88ee78c6b7 aco: track number of post-RA spilled vgprs/sgprs
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38679>
2026-01-19 09:17:47 +00:00
Rhys Perry
d2a9122cfa aco/tests: add function call regalloc tests
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
aa09335485 aco/ra: emit p_split_vector after p_parallelcopy
This fixes situations where the variable being split is a parallelcopy
definition.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
7caa0ab897 aco/ra: fix split_blocking_vectors with some subdword vectors
For example, v6b at byte=2 should be split into v2b,v1 instead of v1,v2b.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
667771950d aco/ra: update register file when recreating blocking vectors
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
5cf991ecfb aco/ra: use original name for blocking vectors rename
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
829e3a6a7d aco/ra: fix operands when recreating blocking vectors
It is likely that these have been moved.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
6e6e40ab5c aco/ra: discard tmp_file after get_regs_for_copies fails
get_regs_for_copies() might change tmp_file, so we need to create a new
one.

Recursively handle_call() with an modified register_file instead of
repeating get_regs_for_copies().

This also updates register_file for the caller.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
159154a741 aco/ra: remove dead code in split_blocking_vectors
collect_vars_from_bitset() from the caller should have already cleared
these.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
89d06f7db5 aco: add RA validation for p_call
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
d0801c9eb7 aco: always run RA validation during tests
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:43 +00:00
Rhys Perry
bf30a57440 aco/ra: omit renaming when necessary when moving copy definitions
This should resolve the FIXME.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:41 +00:00
Rhys Perry
8dcfce3587 aco/ra: refactor update_renames slightly
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38811>
2026-01-16 10:05:39 +00:00
Emma Anholt
ed8676dc28 nir: Rename the unit_test_*_amd intrinics to be un-vendored.
We'll reuse these from the nir_opt_algebraic_pattern_test.

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39076>
2026-01-15 19:09:37 +00:00
Natalie Vock
cc81c7de23 nir,aco: Clean up useless lowering of sbt_base_amd
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:07 +00:00
Natalie Vock
0a1911b220 radv,aco: Use function call structure for RT programs
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:07 +00:00
Natalie Vock
c5d796c902 radv/rt: Use function call structure in NIR lowering
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:06 +00:00
Natalie Vock
06c2e90e35 aco: Note if a parameter needs to be explicitly preserved
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Natalie Vock
2bf403ccec aco: Remove bypass_reg_preservation
Not needed anymore

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Natalie Vock
8525993d02 aco/ra: Move register preservation logic in last block to p_return
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Natalie Vock
eb86ae3b8c aco/spill_preserved: Fix spilled VGPR overflow handling
We modified the spill offset inside the function, which made subsequent
calls to emit_vgpr_spills_reloads reload from the wrong offset.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29580>
2026-01-14 14:19:05 +00:00
Rhys Perry
7a09e4a740 aco: use correct addition opcodes in gfx6-8 RT prolog
Some checks are pending
macOS-CI / macOS-CI (dri) (push) Waiting to run
macOS-CI / macOS-CI (xlib) (push) Waiting to run
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Fixes: 60dd9d797e ("aco: Swizzle ray launch IDs in the RT prolog")
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39232>
2026-01-14 11:23:42 +00:00