This changes allow us to support memory pool of bos with
ANV_BO_ALLOC_AUX_CCS set.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33558>
This is implementing the functions in anv_slab_bo and actually
enabling memory pool.
This is heavily based on Iris memory pool implementation, the main
difference is that the concept of heaps only exist in anv_slab_bo, we
have function that takes the anv_bo_alloc_flags and decides what heap
to place that bo.
Some anv_bo_alloc_flags blocks memory pool, we can relax and remove
some flags from this denied list later.
This feature can be disabled in runtime by setting
ANV_DISABLE_SLAB=true, this can help us to easily check if bugs
are due to this feature or not.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33558>
Allocating larger buffers allows KMD/HW to enable optimizations
that makes access to memory faster, also because of minimum alignment
required in some cases we allocate 4k or 64k long buffers for
usages that only needs a few bytes, wasting a lot of memory.
Memory pool takes care of both of those things and here I'm
adding the base infrastruture to implement this feature.
The next patch will implement the functions in anv_slab_bo.c, spliting
it in two to make review easier.
The idea here is take the same approach as Iris and use pb_slab.h.
In 99% of the places it will be transparent that anv_bo is actually
a slab of a larger and real anv_bo, the remaning 1% of the places are
handled here.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33558>
That will make easy to implement memory pool in the next patches as we
need to calculate the VMA aligment without the KMD alignment requirement
for memory pool.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33558>
This function will be needs in two places in the next patches.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33558>
The intention of this block was to set one of the flags that is used
to select a PAT index but this was doing more than that.
It was promoting WB+0 way coherency BOs to WC+1 way coherency possibly
causing regression in platforms without LLC.
anv_device_get_pat_entry() return WC/writecombining if no flags is
set so we don't need this block after all.
Reported-by: Sushma Venkatesh Reddy <sushma.venkatesh.reddy@intel.com>
Fixes: a65e982b44 ("anv: Split ANV_BO_ALLOC_HOST_CACHED_COHERENT into two actual flags")
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29769>
Introduce a macro so that drivers don't need to rely on the isl_surf
struct to determine the size of the CCS buffer on gfx12.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28942>
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 806281f61f ("anv: add a new reserved pool for capture/release")
Reviewed-by: Rohan Garg <rohan.garg@intel.com>
Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29057>
All these vfuncs funnel down to either stubs or the xe_vm_bind_op()
function. By returning int we're shifting VkResult generation to the
callers, which are simply not doing the correct job. If they get
VkResult they can simply throw the errors up the stack without having
to erroneously try to figure out what really happened.
Today the callers are returning either VK_ERROR_UNKNOWN or
VK_ERROR_OUT_OF_DEVICE_MEMORY, but after the patch we're returning
either VK_ERROR_OUT_OF_HOST_MEMORY or VK_ERROR_DEVICE_LOST.
Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Signed-off-by: Paulo Zanoni <paulo.r.zanoni@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27926>
On Gfx12.0, CCS allocations have to be allocated per image because the
format of the image goes into the AUX-TT PTEs. The effect on memory
allocations is limited since the main surface granularity in the
AUX-TT PTE is 64KB.
On Gfx12.5, the granularity of the AUX-TT PTE is 1MB. This creates a
lot of waste in the application memory allocations. Fortunately the HW
doesn't care about the format put into the PTEs anymore. So it becomes
possible to have 2 images share the same PTE.
To implement this we bring back an earlier version of AUX-TT mappings
where we used to allocate additional CCS space at the end of the
VkDeviceMemory objects. On Gfx12.5, if the BO has additional CCS
space, we will now map the main surface to that space.
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jianxun Zhang <jianxun.zhang@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26822>
It is possible to free memory backing images before images are
destroyed :
VkFreeMemory:
"Memory can be freed whilst still bound to resources, but those
resources must not be used afterwards."
The spec leaves us the option to keep a reference on the associated
memory and free it only when all the bound resources have been
destroyed. Here we choose to free memory immediately.
One particular test in the CTS
(dEQP-VK.synchronization.internally_synchronized_objects.pipeline_cache_graphics)
does the following :
imgA = vkCreateImage()
imgB = vkCreateImage()
memA = vkAllocateMemory()
vkBindImageMemory(imgA, memA) # Aux mapping with ref count = 1
vkFreeMemory(memA) # Aux mapping removed, ref count = 0
memB = vkAllocateMemory() # Same address as memA
vkBindImageMemory(imgB, memB)
vkDestroyImage(imgA) # Removes the mapping of imgB-memB
vkQueueSubmit() # hang with pagefault in AUX-TT
The solution implemented in this change is to not do anything AUX-TT
related in vkFreeMemory(). This soluation has some consequences,
because a virtual memory address range freed and reallocated cannot be
rebound in the AUX-TT until all the associated resources have released
their AUX-TT mapping (to bring back the AUX-TT refcount of the range
to 0). This should still be better than keeping the memory allocated
through refcounting of the anv_bo.
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 7b87e1afbc ("anv: track & unbind image aux-tt binding")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/10528
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27566>
Buffers that are not dedicated can also be used for CCS mapped images,
so they need to be aligned to the AUX-TT requirements.
GTK+ is running into such case where it creates an image with a CCS
modifier. When requesting the alignment through
vkGetImageMemoryRequirements() the 64KB/1MB alignment is returned, but
the binding fails with an assert because the VkDeviceMemory has not
been aligned to the AUX-TT requirement and we cannot disable CCS since
the modifier requires it.
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 4cdd3178fb ("anv: Meet CCS alignment reqs with dedicated allocs")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/10433
Reviewed-by: Jianxun Zhang <jianxun.zhang@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27258>
Launch with :
$ MESA_VK_TRACE=rmv MESA_VK_TRACE_TRIGGER=/tmp/trig ./my_app
In another terminal, trigger a capture :
$ touch /tmp/trig
The application with create a snapshot and print out :
RMV capture saved to '/tmp/my_app_2024.01.19_10.56.33.rmv'
Then just open it with RMV :
./RadeonMemoryVisualizer /tmp/my_app_2024.01.19_10.56.33.rmv
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26843>
In platforms with LLC we can promote all bos to cached+coherent
without any performance penality.
On the up side CPU reads can hit cache this way.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26457>
As suggested by Lionel, here adding ANV_BO_ALLOC_HOST_COHERENT
and with that ANV_BO_ALLOC_HOST_CACHED_COHERENT is now defined by
(ANV_BO_ALLOC_HOST_COHERENT | ANV_BO_ALLOC_HOST_CACHED).
In some callers of anv_device_alloc_bo() was necessary to add
ANV_BO_ALLOC_HOST_COHERENT as no other flag was set and that
was the default behavior up to now.
A change that could look not related is the removal of the
intel_flush_range() in anv_device_init_trivial_batch(), that was done
because trivial_batch_bo is HOST_COHERENT so no flush is necessary.
And it did not made sense to make it ANV_BO_ALLOC_HOST_CACHED_COHERENT
as it was never read in CPU.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26457>
This changes allow us to support HOST_COHERENT, HOST_CACHED and
HOST_COHERENT + HOST_CACHED memory types for platforms that has
the PAT uAPI.
Be aware that Xe KMD will not be able to support cached only memory
types, anv_xe_physical_device_init_memory_types() will reflect that
but internal usage should not allocate
VK_MEMORY_PROPERTY_HOST_CACHED_BIT only memory, hence the assert
added.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25462>
v2: Add assert on VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT in vkMapMemory
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8064>
Now that alloc_flags is stored in anv_bo we can get rid of is_external,
has_fixed_address and has_client_visible_address flags that can
be inferred from alloc_flags.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26099>
When anv_device_map_bo() is called from anv_device_alloc_bo() it gets
VkMemoryPropertyFlags set to 0 so it ends up with a write-combine
caching for integrated platforms with LLC, see 'if (!(property_flags &
VK_MEMORY_PROPERTY_HOST_CACHED_BIT)))'.
Current approach also has issues when mapping with anv_MapMemory2KHR()
as it would not have information to know that BO is a scanout.
It was also not properly calculating mmap mode for platforms with PAT
uAPI before "anv: Change default PAT entry to WC".
So here storing alloc_flags to anv_bo so there is no mismatches
between different code paths then using it to properly
calculate the mmap mode.
alloc_flags in anv_bo will also be used to calculate PAT index in
future patches.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26099>
This flag is required to properly calculate the PAT index of the
imported BO.
Cc: mesa-stable
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26099>
Stop allocating CCS at the end of some BOs. Anv no longer uses that
memory range.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jianxun Zhang <jianxun.zhang@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25003>
At image bind time, we require BOs to meet aux-map alignment
requirements in order to enable CCS on images. This is a heuristic
controlled by anv_bo_allows_aux_map().
To improve the chances of getting a properly aligned BO, we make use of
the dedicated allocation extension. Firstly, we report to applications a
preference for dedicated memory if an image would like to use the aux
map. Secondly, we align the VMA for dedicated allocations to meet
aux-map requirements.
To make enabling modifiers much easier on integrated gfx12, report
dedicated allocations as a requirement for modifiers which specify CCS.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com> (v1)
Reviewed-by: Jianxun Zhang <jianxun.zhang@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25003>
At image bind time, if an image's addresses can be placed into the
aux-map without causing conflicts with a pre-existing mapping, do so.
The code aux management code in the binding function operates on a
per-plane basis. So, use the per-plane CCS memory range from the image
rather than the CCS memory region for the entire BO.
Another way to avoid aux-map conflicts is to rely solely on having a
dedicated allocation for an image. Unfortunately, not all workloads
change their behavior when drivers report a preference for dedicated
allocations. In particular, 3DMark Wild Life Extreme does not make more
dedicated allocations and such a solution was measured to perform ~16%
worse than this solution. With this solution, I did not measure a loss
of CCS on that benchmark.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6304
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com> (v1)
Reviewed-by: Jianxun Zhang <jianxun.zhang@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25003>
Make intel_aux_map_add_mapping return false if a mapping is attempted
that would conflict with an existing one. If this function doesn't
return false, it will either fail to return or return true.
The Vulkan driver will make use of this feature to opportunistically
enable CCS if a BO's VMA range has not been already mapped.
Reviewed-by: Jianxun Zhang <jianxun.zhang@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25003>
Imported bos already imported need special handling in i915.
That handling was moved to
anv_i915_gem_import_bo_alloc_flags_to_bo_flags() as the number of
imported bos is low.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25044>
The bo_flags are i915 specific and should not be handled in common
code, so here adding it to backend as it is in the hot-path.
There still i915 bo_flags handling in anv_device_import_bo() that
will be handled in the next patch.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25044>
The only driver that has a vm_bind ioctl is xe.ko, and its vm_bind
ioctl is not called GEM vm_bind, it's just DRM_IOCTL_XE_VM_BIND
(without GEM anywhere). Back when i915.ko was going to have a vm_bind
ioctl it had GEM on its name, so I guess that's how "gem" appeared in
the naming here, but now nothing does, so let's get rid of it.
Also, these vfuncs we have are specifically made to bind and unbind
whole BOs, so rename them to vm_bind_bo() and vm_unbind_bo() in order
to try to clarify what they mean. The goal is to add a more generic
vm_bind() later that can do anything.
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Signed-off-by: Paulo Zanoni <paulo.r.zanoni@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24681>
Xe KMD only requires userptr to be bound to VM, so here reusing
workaround_bo->gem_handle id to all userptr bos in Xe version of
gem_create_userptr(). The Xe version of gem_close() will make sure
that workaround_bo->gem_handle is not closed when userptr bos
are closed.
With the same gem_handle for all userptr bos, it was also necessary
skip the anv_device_lookup_bo() and manually allocate memory to store
anv_bo in host heap memory, what lead to some small changes in
anv_device_release_bo() as well.
The remaining changes are the support to VM bind userptr bos and the
gem_vm_bind() call in anv_device_import_bo_from_host_ptr().
Fixes: dEQP-VK.memory.external_memory_host*
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23787>
struct anv_bo will be needed in the next patch to properly handle
closure of userptr bos.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23787>
Xe support of userptr will be implemented in the next patch,
this is just moving the i915 and stub functions to KMD backend.
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23787>
