third_party_vulkan-loader/docs/LoaderDebugging.md

Debugging The Vulkan Desktop Loader

Table of Contents

• Debugging Issues
• Loader Logging
• Debugging Possible Layer Issues
  • Enable Layer Logging
  • Disable Layers
  • Selectively Re-enable Layers
• Allow specific layers to be ignored by VK_LOADER_LAYERS_DISABLE
• Debugging Possible Driver Issues
  • Enable Driver Logging
  • Selectively Enable Specific Drivers

Debugging Issues

If your application is crashing or behaving weirdly, the loader provides several mechanisms for you to debug the issues.

NOTE: This functionality is all specific to the desktop Vulkan loader and does not work for the Android loader.

Loader Logging

The Vulkan desktop loader has added logging functionality that can be enabled by using the VK_LOADER_DEBUG environment variable. The results will be output to the standard output, but will also be passed to any VK_EXT_debug_utils messengers present as well. The variable can be set to a comma-delimited list of debug level options which include:

• error      Report any errors encountered by the loader
• warn      Report any warnings encountered by the loader
• info        Report info-level messages generated by the loader
• debug    Report debug-level messages generated by the loader
• layer      Report all layer-specific messages generated by the loader
• driver     Report all driver-specific messages generated by the loader
• all          Report all messages generated by the loader (includes all of the above)

If you're not sure where the issue comes from, at least set it to output all messages through the "info" level:

set VK_LOADER_DEBUG=error,warn,info

Then, you can search the list for any errors or warnings that might provide a hint at why you're seeing issues.

For more info on enabling loader logging, refer to the Enable Loader Debug Layer Output and the Table of Debug Environment Variables below.

Debugging Possible Layer Issues

Enable Layer Logging

If you suspect a layer issue, set the loader logging to specifically output layer messages in addition to warnings and errors:

set VK_LOADER_DEBUG=error,warn,layer

Most important layer messages should go out with error or warning levels set, but this will provide more layer-specific info as well such as:

• What layers are found
• Where they were found
• If they are implicit, what environment variables can be used to disable them
• If there is any incompatibility with a given layer, this could include:
  • The layer library file (.so/.dll) wasn't found
  • The layer library is the wrong bit-depth for the executing application (i.e. 32-bit vs 64-bit)
  • The layer itself doesn't support the application desired version of Vulkan
• If any environment variables are disabling any layers

For example, the output of the loader looking for implicit layers may look like the following:

LAYER: Searching for layer manifest files
LAYER:  In following locations:
LAYER:   /home/${USER}/.config/vulkan/implicit_layer.d
LAYER:   /etc/xdg/vulkan/implicit_layer.d
LAYER:   /usr/local/etc/vulkan/implicit_layer.d
LAYER:   /etc/vulkan/implicit_layer.d
LAYER:   /home/${USER}/.local/share/vulkan/implicit_layer.d
LAYER:   /home/${USER}/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:   /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:   /usr/local/share/vulkan/implicit_layer.d
LAYER:   /usr/share/vulkan/implicit_layer.d
LAYER:  Found the following files:
LAYER:   /home/${USER}/.local/share/vulkan/implicit_layer.d/renderdoc_capture.json
LAYER:   /home/${USER}/.local/share/vulkan/implicit_layer.d/steamfossilize_i386.json
LAYER:   /home/${USER}/.local/share/vulkan/implicit_layer.d/steamfossilize_x86_64.json
LAYER:   /home/${USER}/.local/share/vulkan/implicit_layer.d/steamoverlay_i386.json
LAYER:   /home/${USER}/.local/share/vulkan/implicit_layer.d/steamoverlay_x86_64.json
LAYER:   /usr/share/vulkan/implicit_layer.d/nvidia_layers.json
LAYER:   /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json

Then, the loading of layer libraries is reported similar to this:

LAYER | DEBUG: Loading layer library libVkLayer_khronos_validation.so
LAYER | INFO: Insert instance layer VK_LAYER_KHRONOS_validation (libVkLayer_khronos_validation.so)
LAYER | DEBUG: Loading layer library libVkLayer_MESA_device_select.so
LAYER | INFO: Insert instance layer VK_LAYER_MESA_device_select (libVkLayer_MESA_device_select.so)

Finally, when the Vulkan instance is created, you can see the full instance call-chain from a functional standpoint with output like this:

LAYER: vkCreateInstance layer callstack setup to:
LAYER:  <Application>
LAYER:    ||
LAYER:  <Loader>
LAYER:    ||
LAYER:  VK_LAYER_MESA_device_select
LAYER:      Type: Implicit
LAYER:         Disable Env Var:  NODEVICE_SELECT
LAYER:      Manifest: /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:      Library:  libVkLayer_MESA_device_select.so
LAYER:    ||
LAYER:  VK_LAYER_KHRONOS_validation
LAYER:      Type: Explicit
LAYER:      Manifest: /usr/share/vulkan/explicit_layer.d/VkLayer_khronos_validation.json
LAYER:      Library:  libVkLayer_khronos_validation.so
LAYER:    ||
LAYER:  <Drivers>

In this scenario, two layers were used (the same two that were loaded earlier):

• VK_LAYER_MESA_device_select
• VK_LAYER_KHRONOS_validation

This information now shows us that the VK_LAYER_MESA_device_select is loaded first, followed by VK_LAYER_KHRONOS_validation which will then continue into any available drivers. It also shows that VK_LAYER_MESA_device_select is an implicit layer which implies that it wasn't directly enabled by the application. On the other hand, VK_LAYER_KHRONOS_validation is shown as an explicit layer which indicates that it was likely enabled by the application.

Disable Layers

NOTE: This functionality is only available with Loaders built with version 1.3.234 of the Vulkan headers and later.

Sometimes, implicit layers can cause issues with an application. Because of this, the next step is to try to disable one or more of the listed implicit layers. You can use the filtering environment variables (VK_LOADER_LAYERS_ENABLE and VK_LOADER_LAYERS_DISABLE) to selectively enable or disable various layers. If you're not sure what to do, try disabling all implicit layers manually by setting VK_LOADER_LAYERS_DISABLE to ~implicit~.

  set VK_LOADER_LAYERS_DISABLE=~implicit~

This will disable all implicit layers and the loader will report any disabled layers to the logging output when layer logging is enabled in the following way:

WARNING | LAYER:  Implicit layer "VK_LAYER_MESA_device_select" forced disabled because name matches filter of env var 'VK_LOADER_LAYERS_DISABLE'.
WARNING | LAYER:  Implicit layer "VK_LAYER_AMD_switchable_graphics_64" forced disabled because name matches filter of env var 'VK_LOADER_LAYERS_DISABLE'.
WARNING | LAYER:  Implicit layer "VK_LAYER_Twitch_Overlay" forced disabled because name matches filter of env var 'VK_LOADER_LAYERS_DISABLE'.

Selectively Re-enable Layers

NOTE: This functionality is only available with Loaders built with version 1.3.234 of the Vulkan headers and later.

When trying to diagnose problems caused by layers, it is useful to first disable all layers and re-enable each layer individually. If the problem reappears, then it is immediately clear which layer is the source of the issue.

For example, from the above given list of disabled layers, let’s selectively re-enable one:

set VK_LOADER_LAYERS_DISABLE=~implicit~
set VK_LOADER_LAYERS_ENABLE=*AMD*

This would keep both the "VK_LAYER_MESA_device_select" and "VK_LAYER_Twitch_Overlay" layers disabled, while enabling the "VK_LAYER_AMD_switchable_graphics_64" layer. If everything continues to work, then the evidence seems to suggest the issue is likely not related to the AMD layer. This would lead to enabling one other layer and trying again:

set VK_LOADER_LAYERS_DISABLE=~implicit~
set VK_LOADER_LAYERS_ENABLE=*AMD*,*twitch*

And so forth.

For more info on how to use the filtering environment variables, refer to the Layer Filtering section of the LoaderLayerInterface document.

Allow specific layers to be ignored by VK_LOADER_LAYERS_DISABLE

NOTE: VK_LOADER_LAYERS_DISABLE is only available with Loaders built with version 1.3.262 of the Vulkan headers and later.

When using VK_LOADER_LAYERS_DISABLE to disable implicit layers, it is possible to allow specific layers to be enabled using VK_LOADER_LAYERS_ENABLE. However, this has the effect of forcing layers to be enabled, which is not always desired. Implicit layers have the ability to only be enabled when a layer specified environment variable is set, allow for context dependent enablement. VK_LOADER_LAYERS_ENABLE ignores that context.

Thus, a different environment variable is needed: VK_LOADER_LAYERS_ALLOW

The behavior of VK_LOADER_LAYERS_ALLOW is similar to VK_LOADER_LAYERS_ENABLE except that it does not force a layer to be enabled. The way to think about this environment variable is that every layer matching VK_LOADER_LAYERS_ALLOW is excluded from being forcibly disabled by VK_LOADER_LAYERS_DISABLE. this allows for implicit layers that are context dependent to be enabled depending on the relevant context instead of force enabling them.

Example: Disable all implicit layers except for any layers that have steam or mesa in their name.

set VK_LOADER_LAYERS_DISABLE=~implicit~
set VK_LOADER_LAYERS_ALLOW=*steam*,*Mesa*

Debugging Possible Driver Issues

Enable Driver Logging

NOTE: This functionality is only available with Loaders built with version 1.3.234 of the Vulkan headers and later.

If you suspect a driver issue, set the loader logging to specifically output driver messages:

set VK_LOADER_DEBUG=error,warn,driver

Most important driver messages should go out with error or warning levels set, but this will provide more driver-specific info as well such as:

• What drivers are found
• Where they were found
• If there is any incompatibility with a given driver
• If any environment variables are disabling any of the drivers

For example, the output of the loader looking for drivers on a Linux system may look like the following (NOTE: additional spaces have been removed from the output for easier reading):

DRIVER: Searching for driver manifest files
DRIVER:    In following folders:
DRIVER:       /home/$(USER)/.config/vulkan/icd.d
DRIVER:       /etc/xdg/vulkan/icd.d
DRIVER:       /etc/vulkan/icd.d
DRIVER:       /home/$(USER)/.local/share/vulkan/icd.d
DRIVER:       /home/$(USER)/.local/share/flatpak/exports/share/vulkan/icd.d
DRIVER:       /var/lib/flatpak/exports/share/vulkan/icd.d
DRIVER:       /usr/local/share/vulkan/icd.d
DRIVER:       /usr/share/vulkan/icd.d
DRIVER:    Found the following files:
DRIVER:       /usr/share/vulkan/icd.d/intel_icd.x86_64.json
DRIVER:       /usr/share/vulkan/icd.d/lvp_icd.x86_64.json
DRIVER:       /usr/share/vulkan/icd.d/radeon_icd.x86_64.json
DRIVER:       /usr/share/vulkan/icd.d/lvp_icd.i686.json
DRIVER:       /usr/share/vulkan/icd.d/radeon_icd.i686.json
DRIVER:       /usr/share/vulkan/icd.d/intel_icd.i686.json
DRIVER:       /usr/share/vulkan/icd.d/nvidia_icd.json
DRIVER: Found ICD manifest file /usr/share/vulkan/icd.d/intel_icd.x86_64.json, version "1.0.0"
DRIVER: Found ICD manifest file /usr/share/vulkan/icd.d/lvp_icd.x86_64.json, version "1.0.0"
DRIVER: Found ICD manifest file /usr/share/vulkan/icd.d/radeon_icd.x86_64.json, version "1.0.0"
DRIVER: Found ICD manifest file /usr/share/vulkan/icd.d/lvp_icd.i686.json, version "1.0.0"
DRIVER: Requested driver /usr/lib/libvulkan_lvp.so was wrong bit-type. Ignoring this JSON
DRIVER: Found ICD manifest file /usr/share/vulkan/icd.d/radeon_icd.i686.json, version "1.0.0"
DRIVER: Requested driver /usr/lib/libvulkan_radeon.so was wrong bit-type. Ignoring this JSON
DRIVER: Found ICD manifest file /usr/share/vulkan/icd.d/intel_icd.i686.json, version "1.0.0"
DRIVER: Requested driver /usr/lib/libvulkan_intel.so was wrong bit-type. Ignoring this JSON
DRIVER: Found ICD manifest file /usr/share/vulkan/icd.d/nvidia_icd.json, version "1.0.0"

Then when the application selects the device to use, you will see the Vulkan device call chain reported in the following way (NOTE: additional spaces have been removed from the output for easier reading):

DRIVER: vkCreateDevice layer callstack setup to:
DRIVER:    <Application>
DRIVER:      ||
DRIVER:    <Loader>
DRIVER:      ||
DRIVER:    <Device>
DRIVER:        Using "Intel(R) UHD Graphics 630 (CFL GT2)" with driver: "/usr/lib64/libvulkan_intel.so"

Selectively Enable Specific Drivers

NOTE: This functionality is only available with Loaders built with version 1.3.234 of the Vulkan headers and later.

You can now use the filtering environment variables (VK_LOADER_DRIVERS_SELECT and VK_LOADER_DRIVERS_DISABLE) to control what drivers the loader will attempt to load. For drivers, the string globs passed into the above environment variables will be compared against the driver JSON file name since there is no driver name known to the loader until much later in the Vulkan initialization process.

For example, to disable all drivers except Nvidia you could do the following:

set VK_LOADER_DRIVERS_DISABLE=*
set VK_LOADER_DRIVERS_SELECT=*nvidia*

The loader outputs messages like the following when the environment variables are used:

WARNING | DRIVER: Driver "intel_icd.x86_64.json" ignored because not selected by env var 'VK_LOADER_DRIVERS_SELECT'
WARNING | DRIVER: Driver "radeon_icd.x86_64.json" ignored because it was disabled by env var 'VK_LOADER_DRIVERS_DISABLE'

For more info on how to use the filtering environment variables, refer to the Driver Filtering section of the LoaderDriverInterface document.