Commit graph

110 commits

Author SHA1 Message Date
Iago Toral Quiroga
22d6326449 v3dv: check the render area against the clip window
And flag dirty scissor state if the render area is constraining the
current clip window, so that we emit a new clip window with the next
draw call.

Also, remove the early emission of a clip window for the render area
if we didn't have any scissor state. TLB clears ignore the clip
window, so this was doing nothing for us.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:29 +00:00
Iago Toral Quiroga
e5642a36a0 v3dv: store the clip window in the command buffer state
We will need this so we can match a render area for a new render pass
against the current clip rect and decide if we need to make adjustments.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:29 +00:00
Iago Toral Quiroga
7a39e5e902 v3dv: implement partial color attachment clears
This is achieved by rendering a quad in the clear color for each layer
of each attachment being cleared. Right now we emit each clear in a
separate job with a single attachment framebuffer, but in the future
we may be able to extend the solution to using multiple render targets
and clear multiple attachments with a single job.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:29 +00:00
Alejandro Piñeiro
6e39565e59 v3dv/cmd_buffer: allow return in the middle of variant update if needed
Right now shader variant update on the cmd_buffer is based on populate
a new key using the descriptor bounds, assuming that we would get one
final descriptor for any usage on the shader. But if the descriptors
are being bound with more that one call to CmdBindDescriptorSet, that
would not be true, as the first calls would not bind all the
descriptors. Right now this was raising an assert.

Now we allow that as possible, and for the case of checking variants,
we just stop it, and we don't clean up the SHADER_VARIANT flag.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:29 +00:00
Alejandro Piñeiro
07addb4183 v3dv/cmd_buffer: update shader variants at CmdBindDescriptorSets/CmdBindPipeline
Specially after CmdBindDescriptorSets, it is likely that we would need
a new shader variant, like for example if sampler descriptor sets are
bound.

At that moment a new v3d key is populated, using as base the one used
at pipeline creation, so only cmd_buffer depending values are changed.

Then a new variant is requested. Note that internally it is handled
with a cache, so no new compilation will be done if not needed.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:29 +00:00
Alejandro Piñeiro
f76dad8f89 v3dv/pipeline: add support for shader variants
So far, we were doing the compilation to qpu when the pipeline was
created (as part of vkCreateGraphicsPipeline).

But this would not be correct when some specific descriptors are
involved, like textures. For that case some nir lowerings depend on
the texture format, and that info is not available until the specific
descriptors are bound to the command buffer. In the same way, the same
command buffer with a given pipeline could get their descriptor bound
again.

So it would be needed to support compilation variants of the same
shader. So finally, the v3d_key would work as keys, as the variants
would be tracked with a hash table.

This commit introduces the new structures for that. What we were
building as the final qpu shader would become the initial default
variant for the pipeline. We are also saving the keys used at that
point, to avoid needing to fully regenerate them when a new variant is
created. Not just for performance, but also to avoid needing to track
the graphics pipeline create info structure.

The code to handle updating the current variant would be done on
following commits.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:29 +00:00
Alejandro Piñeiro
f2d16529d1 v3dv/cmd_buffer: push constants not using descriptor anymore
v3dv_descriptor is going to be expanded with more data, so it doesn't
make sense anymore to handle a fake descriptor for the push
constants. Introducing a new struct, that is just a pair
bo/offset. Initially named v3dv_resource, as it could be the base to
reuse bos for different resources (like assembly bo)

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
45e86d8a07 v3dv: don't emit the subpass RCL for jobs that have emitted their own
This fixes multi-layer vkCmdClearAttachments CTS tests. The underlying
problem here is that even though this command runs inside a render pass,
it is implemented as a separate job that emits its own RCL to program
render target color clears, so we should not emit the subpass RCL for it.

Fixes 250+ CTS tests (all but a1r5g5b5) in:
dEQP-VK.api.image_clearing.core.clear_color_attachment.cube_layers.*
dEQP-VK.api.image_clearing.core.clear_color_attachment.multiple_layers.*
dEQP-VK.api.image_clearing.core.clear_depth_stencil_attachment.multiple_layers.*

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
e86f381ea1 v3dv: drop incorrect assertion on number of clear values at render pass begin
There can be more clear values than attachments, we should just ignore them
in that case.

Fixes some tests in:
dEQP-VK.fragment_operations.*

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
69ce1f4fd4 v3dv: only emit config bits and varyings packets if needed
These should be emitted only if we have a new pipeline.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Alejandro Piñeiro
e903075335 v3dv/cmd_bufffer: rename and split emit_graphics_pipeline
The name suggests that this method emits the full graphics pipeline,
but that is not the case (ie: scissor is emitted at a different
point).

Right now that method is mostly emitting the gl_shader state plus some
other packets. So we just renamed it to emit_gl_shader_state, and move
the other packet emission to new emission methods.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
6bd4e69e9c v3dv: stencil state fixes
First this makes it so we only clear dirty stencil state if we actually
emit the stencil packets. Second, now we check if we need to emit stencil
whenever a new pipeline is bound, since a new pipeline may not change the
dynamic stencil state but might still be changing other aspects of stencil,
which means that even if the dynamic stencil state is not dirty, we might
still need to emit new stencil packets.

This fixes a regression in VkRunner test depth-buffer.vk_shader_test after
we dropped the redundant emission of stencil state, since that redundant
emission was happening unconditionally whenever we had a new pipeline.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
c5784fbef4 v3dv: drop redundant emission of stencil state
If needed we are already calling this at draw time since stencil
can be dynamic state.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
7c1910f453 v3dv: rewrite dirty state handling
The current implementation assumed that we would clear all dirty state
after we have emitted a pipeline, but that is not always true. In
particular, we don't emit blend constants unless we need them, so we
can't clear its dirty bit until we have bound a pipeline that actually
requires them.

The change implemented here has individual emit functions clear pipeline
states they hadle as they emit the corresponding state and we clear
the dirty pipeline bit at the end.

This fixes some CTS pipeline blend tests where we have multiple draws
with blending and only some of them require blend constants. In this case,
the original behavior would clear the blend constants dirty bit on draw
calls that don't actually emit blend constants (because they don't need
them), making the later draw calls that do need them fail because they
don't emit them either (since the previous draw calls cleared the dirty
bit).

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
c525849d95 v3dv: only emit blend state if the pipeline is dirty
Except for blend constants, which can be dynamic state.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
1e54b62a61 v3dv: implement dynamic state for blend constants
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
4c15131b1d v3dv: always flush draw calls if we are doing sRGB blending
Not quite sure why this is required though. Conversion from/to
sRGB happens on tile loads and stores, with the tile buffer
being always linear, so there should be no difference.

Fixes all test failures in:
dEQP-VK.pipeline.blend.format.r8g8b8a8_srgb.*

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
761b0c02f8 v3dv: add an 'always flush' mode
In this mode, which can be activated with V3D_DEBUG=always_flush like
in the GL driver, we flush every draw call separately. For now this
is useful for debugging, but we can also set the flag internally on
specific jobs when we identify scenarios where we need the same behavior.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
d73bb59113 v3dv: implement color blending
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
8ed2e53e0d v3dv: support submits without a command buffer
It is valid to submit with an empty list ofcommand buffers, however,
we still need to wait on the pWaitSemaphores provided and only signal
the pSignalSemaphores and fence once we have finished waiting on them
to honor the semantics of the submission.

Because waiting and signaling happens in the kernel, the easiest way
to do this is to submit a trivial no-op job to the GPU. To do this,
we need to refactor some of our code so that code that might have been
operating on a command buffer starts operating on a job instead, so we
can resuse most of our infrastructure to create the no-op job.

Additionally, because no-op jobs are created internally by the driver,
we are responsible for destroying them too. For this, we bind a fence
to each no-op job we submit and we test for completion of in-flight
no-op jobs (and destory them if completed) every time vkQueueSubmit
is called.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
5cfc3b5cc9 v3dv: handle the case where we fail to allocate a new job gracefully
There are a handful of tests that simulate 'out of memory' situations
during swapchain image creation, and these can lead to failed job
allocations when the driver is running on the prime blit path, as that
involves creating a command buffer. The tests expect us to handle this
scenario gracefully and return an appropriate OOM error as a result.
This make sure we don't try to dereference a job if we failed to allocate
it so we don't crash and can return the OOM error gracefully in the
process.

Fixes:
dEQP-VK.wsi.xlib.swapchain.simulate_oom.*

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
e5adc33e80 v3dv: don't leak job allocations
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
a1337c027c v3dv: work around viewport Z scale hardware bug
It looks like when the Z scale is small enough the hardware clipper
won't work properly.

Fixes:
dEQP-VK.draw.inverted_depth_ranges.nodepthclamp_deltazero

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
db629c0f4b v3dv: fix viewport Z
Vulkan has Z NDC range in [0, 1], we where using OpenGL's [-1, 1].

Fixes:
dEQP-VK.draw.inverted_depth_ranges.nodepthclamp_deltasmall
dEQP-VK.draw.inverted_depth_ranges.nodepthclamp_deltaone

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
57f5fa08cb v3dv: fix scissor outside viewport
In this scenario we can end up generating a clip window where
the max coordinates are smaller than the min coordinates and the simulator
asserts.

Fixes:
dEQP-VK.draw.scissor.dynamic_scissor_outside_viewport
dEQP-VK.draw.scissor.static_scissor_outside_viewport

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
d734dba8a4 v3dv: implement interpolation qualifiers
Also, Vulkan uses the same provoking vertex rules are Direct3D, which
changes from OpenGL's default. Make sure we honor that.

Fixes:
dEQP-VK.spirv_assembly.instruction.graphics.cross_stage.*

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
518714ae1d v3dv: drop incorrect assertion
This was intended to check that we only got
VK_COMMAND_BUFFER_USAGE_RENDER_PASS_CONTINUE_BIT
on secondary command buffers, but the spec states that this flag
should be ignored for primary command buffers.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
26bfbf29c2 v3dv: don't reset loader data on command buffers
We were clearing memory to 0 on create and reset, including the
loader data, which is not correct on reset since it would cleat
the loader dispatch table for the command buffer. We should only
clear driver data.

Also, don't use vk_zalloc for the command buffer allocation, since
we are already clearing on reset and we always reset when we begin
recording.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
313809487c v3dv: more frame tiling refactors
This puts all the information required to setup frame tiling into
v3dv_frame_tiling so we no longer need a framebuffer to start a
frame. This makes the code simpler, since frame tiling calculations
happen automatically when we start a new frame and simplifies
the implementation of copy and clear operations that used to
requiere that we setup a fake framebuffer with no actual attachments,
which was a bit of a kludge.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
2a3fa914d4 v3dv: rewrite frame tiling setup
So far we have been getting away with computing frame tiling information for
the framebuffer object, but that is not correct, since different subpasses
may access different subsets of the framebuffer, with each requiring a
different configuration because the number of render targets and the maximum
bpp can change for each subpass.

This adds a v3dv_frame_tiling struct to keep the frame tiling information and
rewrites the code to compute this for every new job we start.

Fixes a bunch of tests in dEQP-VK.pipeline.render_to_image.*

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
44cda4dbfe v3dv: fix subpass tracking in the command buffer state
When we create a new job for a new subpass, we might have to finish
the current job for the previous subpass, so it is important that we
we don't get ahead of ourselves and increment the current subpass index
in the command buffer state until we are really done finishing the
previous job.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
6a909387c2 v3dv: take memory format from appropriate miplevel for image load/store
We were storing the format of the base miplevel in the image view and
we were typically using that instead of the taking the format from
the appropriate image slice. This was a problem when loading or storing
a miplevel other than the base which happened to have a different format.

This also removed the tiling field from the image view to avoid repeating
the same mistake in the future.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Alejandro Piñeiro
5d1089b88d v3dv/descriptor: support for dynamic ubo/ssbo
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:28 +00:00
Iago Toral Quiroga
404220cc71 v3dv: fix supertile coverage when render are size is 0.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
aa9ad02e75 v3dv: implement vkResetCommandPool
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
79569fb796 v3dv: implement vkResetCommandBuffer
Also, implement implicit reset via vkCmdBeginCommandBuffer.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
83554840a5 v3dv: add support for primitive restarts on indexed draw calls
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
914a23a8a0 v3dv: implement indirect draws
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Alejandro Piñeiro
e9085e56ed v3dv/cmd_buffer: support for push constants
By default they are trivially lowered to load_uniform.

We still need to allocate an UBO for push constants, used for those
that are accessed using a non-const index. This is automatically
handled by the compiler, as it cames back as asking a
QUNIFORM_UBO_ADDR. This is what already does for gallium.

Note that if needing the UBO, we are uploading the full push constant
data. An improvement would be to try to upload only the data that
needs to rely on the UBO (so non-const accesses to uniforms).

Also, the code is not handling getting out of space from the UBO
bo. This would be tackled at a different commit.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
b3f9a3beaf v3dv: implement vkCmdUpdateBuffer
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Alejandro Piñeiro
83e83257f8 v3dv: no need to manually add assembly bo to the job
The _cl utility methods are already doing that implicitly, as a way to
ensure that any address emitted gets its bo included on the job.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
3dff3f3a45 v3dv: take the number of layers from the framebuffer
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
0dc4416827 v3dv: ignore image view aspects for depth/stencil attachments
From the Vulkan spec:

   "When an image view of a depth/stencil image is used as a
    depth/stencil framebuffer attachment, the aspectMask is ignored
    and both depth and stencil image subresources are used."

So in that scenario, we ignore the aspect mask on the view and go
check the actual format of the underlying image to decide if we
have depth or depth+stencil aspects.

This gets VkRunner's depth-buffer.shader_test to pass.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
1c6938484e v3dv: implement indexed draws
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
f48788896e v3dv: implement vkCmdClearAttachments
For now this only implements a fast path using the tile buffer, so it
can only be used when clearing full images, but this is good enough
for VkRunner.

The implementation is a bit tricky because this command executes
inside a render pass, and yet, since we are using the tile buffer to
clear, this needs to go in its own job. This means that with this, we
need to be able to split a subpass into multiple jobs which creates
some issues.

For example, certain operations, such as the subpass load operation
(particularly if it is a clear) should only happen on the first job of
the subpass and subsequent jobs in the same subpass should always
load.

Similarly, we should not discard the last store on an attachment
unless we know it is the last job for the last subpass that uses the
attachment.

To handle these cases we add two new flags to the job, one to know if
the job is not the first in a subpass (is_subpass_continue) and
another one to know if a job is the last in a subpass
(is_subpass_finish).

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
255d4245fd v3dv: add a helper to get the Z/S buffer from an aspect mask
We will be using this when we need to implement other clearing commands.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
68692297ba v3dv: add a helper to compute the hardware clear color
We will be using this when we implement other clearing commands
(currently this is only used for the atatchment load clear operation).

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Alejandro Piñeiro
34cf475102 v3dv/descriptor_set: support for array of ubo/ssbo
For that we include the array_index when asking for a ubo/ssbo index
from the descriptor_map.

Until now, array_index was not included, but the descriptor_map took
into account the array_size. This had the advantage that you only need
a entry on the descriptor map, and the index was properly return.

But this make it complex to get back the set, binding and array_index
back from the ubo/ssbo binding. So it was more easy to just add
array_index. Somehow now the "key" on the descriptor map is the
combination of (set, binding, array_index).

Note that this also make sense as the vulkan api identifies each array
index as a descriptor, so for example, from spec,
VkDescriptorSetLayoutBinding:descriptorCount

 "descriptorCount is the number of descriptors contained in the
  binding, accessed in a shader as an array"

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Alejandro Piñeiro
9afd24f89a v3dv: initial descriptor set support
Focused on getting the basic UBO and SSBO cases implemented. So no
dynamic offset, push contanst, samplers, and so on.

This include a initial implementation for CreatedescriptorPool,
CreateDescriptorSetLayout, AllocateDescriptorSets,
UpdateDescriptorSets, CreatePipelineLayout, and CmdBindDescriptorSets.

Also introduces lowering vulkan intrinsics. For now just
vulkan_resource_index.

We also introduce a descriptor_map, in this case for the ubos and
ssbos, used to assign a index for each set/binding combination, that
would be used when filling back the details of the ubo or ssbo on
other places (like QUNIFORM_UBO_ADDR or QUNIFORM_SSBO_OFFSET).

Note that at this point we don't need a bo for the descriptor pool, so
descriptor sets are not getting a piece of it. That would likely
change as we start to support more descriptor set types.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00
Iago Toral Quiroga
29a7d99c4a v3dv: fix the mess with dynamic state handling
The general idea is that we always emit from our dynamic state, and when
a particular piece of state is not dynamic, we just set our dynamic state
from the pipeline state, however, the implementation was quite confusing:
the mask of dynamic states flagged states that were not dynamic and some
places woud mix dirty flags and dynamic state flags. We also were not
updating the dynamic state mask in the command buffer, etc.

This patch, hopefully, simplifies all this and makes it less confusing,
starting by making the dynamic state mask flag dynamic states, fixing
the places where we would confuse dirty state flags with dynamic state
flags, making sure that our command buffer state is setup correctly
and that we only emit state when it is actually dirty.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6766>
2020-10-13 21:21:27 +00:00