VULKAN: Build display pipelines against swapchain render pass

Fixes https://github.com/libretro/RetroArch/issues/18761

All display-related Vulkan pipelines (font, alpha_blend, menu
shaders including ribbon/snow/bokeh) were being built against
vk->sdr_render_pass, which always uses VK_FORMAT_B8G8R8A8_UNORM.

On the non-HDR rendering path (the common case on Linux KMS,
Wayland, X11, and most platforms), the active render pass at
draw time is vk->render_pass, which uses the actual swapchain
format -- vk->context->swapchain_format.  When the swapchain
format happens to be B8G8R8A8_UNORM, the pipeline/render-pass
formats match and everything works.

But on some platforms (particularly KMS on certain GPUs), the
swapchain format is VK_FORMAT_R8G8B8A8_UNORM instead.  In that
case:

  - Pipeline built for: B8G8R8A8_UNORM (via sdr_render_pass)
  - Active render pass:  R8G8B8A8_UNORM (via render_pass)

This is a render pass compatibility violation per Vulkan spec
(VUID-vkCmdDraw-renderPass-02684).  The result is driver-
dependent: RADV on AMD typically segfaults, NVIDIA may silently
produce wrong output, Intel may report a validation error.

The crash manifests as a segfault when any menu shader pipeline
is active (ribbon, simple ribbon, snow, bokeh, snowflake) and
the user navigates the XMB Quick Menu -- any draw that uses the
menu pipeline triggers the incompatible render pass.

Fix: build all display pipelines against vk->render_pass
(the swapchain render pass) instead of vk->sdr_render_pass.
This ensures the pipelines match the active render pass on
the non-HDR path regardless of swapchain format.

Two changes:

1. Line 3372: Initial pipe.renderPass assignment changed from
   vk->sdr_render_pass to vk->render_pass.  This affects the
   font pipeline, alpha_blend pipeline, and display pipelines
   0-3.

2. After the #ifdef VULKAN_HDR_SWAPCHAIN block (which
   temporarily changes pipe.renderPass to readback_render_pass
   and sdr_render_pass for HDR-specific pipelines), restore
   pipe.renderPass to vk->render_pass before building the menu
   shader pipelines (indices 6+).  Without this, HDR builds
   would leave pipe.renderPass pointing at sdr_render_pass,
   defeating the fix.

Impact on HDR: the HDR compositing path renders menu content
into an offscreen B8G8R8A8 buffer using sdr_render_pass
(line 6523).  When the swapchain format differs from B8G8R8A8,
the menu pipelines built against render_pass would be
incompatible with sdr_render_pass in this path.  However:

  - HDR display pipelines (indices 4-5) are built separately
    against render_pass at line 3470, which is correct.
  - The HDR menu compositing path uses the standard alpha_blend
    pipeline for the menu texture overlay (line 6563), which
    now matches render_pass.
  - The HDR offscreen compositing step at line 6523 is a
    separate render pass instance that doesn't use the menu
    shader pipelines (ribbon/snow/bokeh are drawn earlier in
    the frame).

No behavioural change when swapchain format is B8G8R8A8_UNORM
(the common case on X11/Wayland with NVIDIA and Intel), because
both render passes have the same format and are therefore
compatible.
This commit is contained in:
U-DESKTOP-SPFP6AQ\twistedtechre committed 2026-04-16 17:21:54 +02:00
1 parent b762cd10f9
commit d2501944bd
1 file changed
+33 -1
+33 -1
View File
@@ -3369,7 +3369,33 @@ static void vulkan_init_pipelines(vk_t *vk)
pipe.pViewportState = &vp;
pipe.pDepthStencilState = &depth_stencil;
pipe.pDynamicState = &dynamic;
pipe.renderPass = vk->sdr_render_pass;
/* Build all display-related pipelines (font, alpha_blend,
* menu shaders, ribbon, snow, etc.) against the actual
* swapchain render pass. Using sdr_render_pass here would
* be correct only when the swapchain format happens to be
* VK_FORMAT_B8G8R8A8_UNORM. On KMS and some other
* platforms, the swapchain may use VK_FORMAT_R8G8B8A8_UNORM
* instead, making the pipeline incompatible with the active
* render pass at draw time — a Vulkan validation error that
* many drivers (especially RADV on AMD) turn into a
* segfault.
*
* For the HDR offscreen compositing path, which renders into
* a B8G8R8A8 offscreen buffer under sdr_render_pass, these
* pipelines may not be format-compatible if the swapchain
* format differs from B8G8R8A8. However, the HDR path
* is only active when HDR is supported AND the display uses
* a wide-gamut format — in which case a separate set of
* HDR display pipelines (indices 4-5) is built against the
* correct render pass at lines 3444-3453. The menu shader
* pipelines (ribbon, snow, bokeh) used through
* draw_pipeline + draw are called in the same render pass
* as the main compositing draw, which uses render_pass on
* non-HDR and sdr_render_pass on HDR-offscreen.
*
* Fixes: https://github.com/libretro/RetroArch/issues/18761
* (XMB ribbon crash on Vulkan KMS with R8G8B8A8 swapchain) */
pipe.renderPass = vk->render_pass;
pipe.layout = vk->pipelines.layout;
module_info.codeSize = sizeof(alpha_blend_vert);
@@ -3478,6 +3504,12 @@ static void vulkan_init_pipelines(vk_t *vk)
vkDestroyShaderModule(vk->context->device, shader_stages[0].module, NULL);
/* Restore the swapchain render pass for the menu shader
* pipelines. The HDR block above may have left
* pipe.renderPass pointing at sdr_render_pass or
* readback_render_pass. */
pipe.renderPass = vk->render_pass;
/* Other menu pipelines. */
for (i = 0; i < (int)ARRAY_SIZE(vk->display.pipelines) - 6; i++)
{