Re-arrange/update loader docs
Update the loader documentation in the following ways: 1. Move documentation to new "docs" folder" 2. Split LoaderAndLayerInterface.md into multiple docs to focus on target audience: - Top-Level LoaderInterfaceArchitecture.md - Applications LoaderApplicationInterface.md - Layers LoaderLayerInterface.md - ICDs LoaderImplementationInterface.md 3. Upload newer images and their corresponding original Inkscape files. 4. Cleanup and update sections on Linux directory search 5. Add new sections to detail items missing - VkConfig - Handling undef Fuchsia 6. Language cleanup Thanks to @charles-lunarg and @smcv for feedback and various section language corrections.
@ -20,7 +20,7 @@ There is also a separate loader, maintained by Google, which is used on Android.
|
||||
The following components are available in this repository:
|
||||
|
||||
- [ICD Loader](loader/)
|
||||
- [Loader Documentation](loader/LoaderAndLayerInterface.md)
|
||||
- [Loader Documentation](docs/LoaderInterfaceArchitecture.md)
|
||||
- [Tests](tests/)
|
||||
|
||||
## Contact Information
|
||||
@ -40,7 +40,7 @@ management details.
|
||||
Includes directions for building all components.
|
||||
|
||||
Architecture and interface information for the loader is in
|
||||
[loader/LoaderAndLayerInterface.md](loader/LoaderAndLayerInterface.md).
|
||||
[docs/LoaderInterfaceArchitecture.md](docs/LoaderInterfaceArchitecture.md).
|
||||
|
||||
## Version Tagging Scheme
|
||||
|
||||
|
703
docs/LoaderApplicationInterface.md
Normal file
@ -0,0 +1,703 @@
|
||||
<!-- markdownlint-disable MD041 -->
|
||||
[![Khronos Vulkan][1]][2]
|
||||
|
||||
[1]: https://vulkan.lunarg.com/img/Vulkan_100px_Dec16.png "https://www.khronos.org/vulkan/"
|
||||
[2]: https://www.khronos.org/vulkan/
|
||||
|
||||
# Application Interface to Loader
|
||||
[![Creative Commons][3]][4]
|
||||
|
||||
<!-- Copyright © 2015-2021 LunarG, Inc. -->
|
||||
|
||||
[3]: https://i.creativecommons.org/l/by-nd/4.0/88x31.png "Creative Commons License"
|
||||
[4]: https://creativecommons.org/licenses/by-nd/4.0/
|
||||
|
||||
## Table of Contents
|
||||
|
||||
* [Overview](#overview)
|
||||
* [Interfacing with Vulkan Functions](#interfacing-with-vulkan-functions)
|
||||
* [Vulkan Direct Exports](#vulkan-direct-exports)
|
||||
* [Directly Linking to the Loader](#directly-linking-to-the-loader)
|
||||
* [Dynamic Linking](#dynamic-linking)
|
||||
* [Static Linking](#static-linking)
|
||||
* [Indirectly Linking to the Loader](#indirectly-linking-to-the-loader)
|
||||
* [Best Application Performance Setup](#best-application-performance-setup)
|
||||
* [ABI Versioning](#abi-versioning)
|
||||
* [Windows Dynamic Library Usage](#windows-dynamic-library-usage)
|
||||
* [Linux Dynamic Library Usage](#linux-dynamic-library-usage)
|
||||
* [MacOs Dynamic Library Usage](#macos-dynamic-library-usage)
|
||||
* [Application Layer Usage](#application-layer-usage)
|
||||
* [Meta-Layers](#meta-layers)
|
||||
* [Implicit vs Explicit Layers](#implicit-vs-explicit-layers)
|
||||
* [Override Layer](#override-layer)
|
||||
* [Forcing Layer Source Folders](#forcing-layer-source-folders)
|
||||
* [Exception for Elevated Privileges](#exception-for-elevated-privileges)
|
||||
* [Forcing Layers to be Enabled](#forcing-layers-to-be-enabled)
|
||||
* [Overall Layer Ordering](#overall-layer-ordering)
|
||||
* [Application Usage of Extensions](#application-usage-of-extensions)
|
||||
* [Instance and Device Extensions](#instance-and-device-extensions)
|
||||
* [WSI Extensions](#wsi-extensions)
|
||||
* [Unknown Extensions](#unknown-extensions)
|
||||
|
||||
## Overview
|
||||
|
||||
This is the Application-centric view of working with the Vulkan loader.
|
||||
For the complete overview of all sections of the loader, please refer
|
||||
to the [LoaderInterfaceArchitecture.md](LoaderInterfaceArchitecture.md) file.
|
||||
|
||||
## Interfacing with Vulkan Functions
|
||||
|
||||
There are several ways Vulkan functions may be interfaced through the loader:
|
||||
|
||||
|
||||
### Vulkan Direct Exports
|
||||
|
||||
The loader library on Windows, Linux, Android, and macOS will export all core
|
||||
Vulkan entry-points and all appropriate Window System Interface (WSI)
|
||||
entry-points.
|
||||
This is done to make it simpler to get started with Vulkan development.
|
||||
When an application links directly to the loader library in this way, the
|
||||
Vulkan calls are simple *trampoline* functions that jump to the appropriate
|
||||
dispatch table entry for the object they are given.
|
||||
|
||||
|
||||
### Directly Linking to the Loader
|
||||
|
||||
#### Dynamic Linking
|
||||
|
||||
The loader is distributed as a dynamic library (.dll on Windows or .so on Linux
|
||||
or .dylib on macOS) which gets installed to the system path for dynamic
|
||||
libraries.
|
||||
Furthermore, the dynamic library is generally installed to Windows
|
||||
systems as part of driver installation and is generally provided on Linux
|
||||
through the system package manager.
|
||||
This means that applications can usually expect a copy of the loader to be
|
||||
present on a system.
|
||||
If applications want to be completely sure that a loader is present, they can
|
||||
include a loader or runtime installer with their application.
|
||||
|
||||
#### Static Linking
|
||||
|
||||
In previous versions of the loader, it was possible to statically link the
|
||||
loader.
|
||||
**This was removed and is no longer possible.**
|
||||
The decision to remove static linking was because of changes to the driver
|
||||
which made older applications that statically linked unable to find newer
|
||||
drivers.
|
||||
|
||||
Additionally, static linking posed several problems:
|
||||
- The loader can never be updated without re-linking the application
|
||||
- The possibility that two included libraries could contain different versions
|
||||
of the loader
|
||||
- Could cause conflicts between the different loader versions
|
||||
|
||||
The only exception to this is for macOS, but is not supported or tested.
|
||||
|
||||
### Indirectly Linking to the Loader
|
||||
|
||||
Applications are not required to link directly to the loader library, instead
|
||||
they can use the appropriate platform-specific dynamic symbol lookup on the
|
||||
loader library to initialize the application's own dispatch table.
|
||||
This allows an application to fail gracefully if the loader cannot be found.
|
||||
It also provides the fastest mechanism for the application to call Vulkan
|
||||
functions.
|
||||
An application only needs to query (via system calls such as `dlsym`) the
|
||||
address of `vkGetInstanceProcAddr` from the loader library.
|
||||
The application then uses `vkGetInstanceProcAddr` to load all functions
|
||||
available, such as `vkCreateInstance`, `vkEnumerateInstanceExtensionProperties`
|
||||
and `vkEnumerateInstanceLayerProperties` in a platform-independent way.
|
||||
|
||||
### Best Application Performance Setup
|
||||
|
||||
To get the best possible performance in a Vulkan application, the application
|
||||
should set up its own dispatch table for every Vulkan API entry-point.
|
||||
For every instance-level Vulkan command in the dispatch table, the function pointer
|
||||
should be queried and filled in by using the results of `vkGetInstanceProcAddr`.
|
||||
Additionally, for every device-level Vulkan command, the function pointer
|
||||
should be queried and filled in using the resulsts of `vkGetDeviceProcAddr`.
|
||||
|
||||
*Why do this?*
|
||||
|
||||
The answer comes in how the call chain of instance functions are implemented
|
||||
versus the call chain of a device functions.
|
||||
Remember, a [Vulkan instance is a high-level construct used to provide Vulkan
|
||||
system-level information](LoaderInterfaceArchitecture.md#instance-specific).
|
||||
Because of this, instance functions need to be broadcast to every available
|
||||
implementation on the system.
|
||||
The following diagram shows an approximate view of an instance call chain with
|
||||
three enabled layers:
|
||||
|
||||
![Instance Call Chain](./images/loader_instance_chain.png)
|
||||
|
||||
This is also how a Vulkan device function call chain looks if queried
|
||||
using `vkGetInstanceProcAddr`.
|
||||
On the other hand, a device function doesn't need to worry about the broadcast
|
||||
because it knows specifically which associated implementation and which
|
||||
associated physical device the call should terminate at.
|
||||
Because of this, the loader doesn't need to get involved between any enabled
|
||||
layers and the implementation.
|
||||
Thus, using a loader-exported Vulkan device function, the call chain
|
||||
in the same scenario as above would look like:
|
||||
|
||||
![Loader Device Call Chain](./images/loader_device_chain_loader.png)
|
||||
|
||||
An even better solution would be for an application to perform a
|
||||
`vkGetDeviceProcAddr` call on all device functions.
|
||||
This further optimizes the call chain by removing the loader all-together under
|
||||
most scenarios:
|
||||
|
||||
![Application Device Call Chain](./images/loader_device_chain_app.png)
|
||||
|
||||
Also, notice if no layers are enabled, the application function pointers point
|
||||
**directly to the implementation**.
|
||||
With many function calls, the lack of indirection in each adds up to non-trivial
|
||||
performance savings.
|
||||
|
||||
**NOTE:** There are some device functions which still require the loader to
|
||||
intercept them with a *trampoline* and *terminator*.
|
||||
There are very few of these, but they are typically functions which the loader
|
||||
wraps with its own data.
|
||||
In those cases, even the device call chain will continue to look like the
|
||||
instance call chain.
|
||||
One example of a device function requiring a *terminator* is
|
||||
`vkCreateSwapchainKHR`.
|
||||
For that function, the loader needs to potentially convert the KHR_surface
|
||||
object into an implementation-specific KHR_surface object prior to passing down
|
||||
the rest of the function's information to the implementation.
|
||||
|
||||
Remember:
|
||||
* `vkGetInstanceProcAddr` is used to query instance and physical device
|
||||
functions, but can query all functions.
|
||||
* `vkGetDeviceProcAddr` is only used to query device functions.
|
||||
|
||||
|
||||
### ABI Versioning
|
||||
|
||||
The Vulkan loader library will be distributed in various ways including Vulkan
|
||||
SDKs, OS package distributions and Independent Hardware Vendor (IHV) driver
|
||||
packages.
|
||||
These details are beyond the scope of this document.
|
||||
However, the name and versioning of the Vulkan loader library is specified so
|
||||
an app can link to the correct Vulkan ABI library version.
|
||||
ABI backwards compatibility is guaranteed for all versions with the same major
|
||||
number (e.g. 1.0 and 1.1).
|
||||
|
||||
#### Windows Dynamic Library Usage
|
||||
|
||||
On Windows, the loader library encodes the ABI version in its name such that
|
||||
multiple ABI incompatible versions of the loader can peacefully coexist on a
|
||||
given system.
|
||||
The Vulkan loader library file name is `vulkan-<ABI version>.dll`.
|
||||
For example, for Vulkan version 1.X on Windows the library filename is
|
||||
`vulkan-1.dll`.
|
||||
This library file can typically be found in the `windows\system32`
|
||||
directory (on 64-bit Windows installs, the 32-bit version of the loader with
|
||||
the same name can be found in the `windows\sysWOW64` directory).
|
||||
|
||||
#### Linux Dynamic Library Usage
|
||||
|
||||
For Linux, shared libraries are versioned based on a suffix.
|
||||
Thus, the ABI number is not encoded in the base of the library filename as on
|
||||
Windows.
|
||||
|
||||
On Linux, applications that have a hard dependency on Vulkan should request
|
||||
linking to the unversioned name `libvulkan.so` in their build system.
|
||||
For example by importing the CMake target `Vulkan::Vulkan` or by using the
|
||||
output of `pkg-config --cflags --libs vulkan` as compiler flags.
|
||||
As usual for Linux libraries, the compiler and linker will resolve this to
|
||||
a dependency on the correct versioned SONAME, currently `libvulkan.so.1`.
|
||||
Linux applications that load Vulkan-Loader dynamically at runtime do not
|
||||
benefit from this mechanism, and should instead make sure to pass the
|
||||
versioned name such as `libvulkan.so.1` to `dlopen()`, to ensure that they
|
||||
load a compatible version.
|
||||
|
||||
#### MacOs Dynamic Library Usage
|
||||
|
||||
MacOs linking is similar to Linux, with the exception being that the standard
|
||||
dynamic library is named `libvulkan.dylib` and the ABI versioned library is
|
||||
currently named `libvulkan.1.dylib`.
|
||||
|
||||
|
||||
## Application Layer Usage
|
||||
|
||||
Applications desiring Vulkan functionality beyond what Vulkan implementations
|
||||
on their system already expose, may use various layers to augment the API.
|
||||
A layer cannot add new Vulkan core API entry-points that are not exposed in
|
||||
Vulkan.h.
|
||||
However, layers may offer implementations of extensions that introduce
|
||||
additional entry-points beyond what is available without those layers.
|
||||
These additional extension entry-points can be queried through the Vulkan
|
||||
extension interface.
|
||||
|
||||
A common use of layers is for API validation which can be enabled during
|
||||
application development and left out when releasing the application.
|
||||
This allows easy control of the overhead resulting from enabling validation of
|
||||
the application's usage of the API, which wasn't always possible in previous
|
||||
graphics APIs.
|
||||
|
||||
To find out what layers are available to an application, use
|
||||
`vkEnumerateInstanceLayerProperties`.
|
||||
This will report all layers that have been discovered by the loader.
|
||||
The loader looks in various locations to find layers on the system.
|
||||
For more information see the
|
||||
[Layer discovery](LoaderLayerInterface.md#layer-discovery)
|
||||
section in the
|
||||
[LoaderLayerInterface.md document](LoaderLayerInterface.md) document.
|
||||
|
||||
To enable specific layers, simply pass the names of the layers to
|
||||
enable in the `ppEnabledLayerNames` field of the `VkInstanceCreateInfo` during
|
||||
a call to `vkCreateInstance`.
|
||||
Once done, the layers that have been enabled will be active for all Vulkan functions
|
||||
using the created `VkInstance`, and any of its child objects.
|
||||
|
||||
**NOTE:** Layer ordering is important in several cases since some layers
|
||||
interact with each other.
|
||||
Be careful when enabling layers as this may be the case.
|
||||
See the [Overall Layer Ordering](#overall-layer-ordering) section for more
|
||||
information.
|
||||
|
||||
The following code section shows how to go about enabling the
|
||||
`VK_LAYER_KHRONOS_validation` layer.
|
||||
|
||||
```
|
||||
char *instance_layers[] = {
|
||||
"VK_LAYER_KHRONOS_validation"
|
||||
};
|
||||
const VkApplicationInfo app = {
|
||||
.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO,
|
||||
.pNext = NULL,
|
||||
.pApplicationName = "TEST_APP",
|
||||
.applicationVersion = 0,
|
||||
.pEngineName = "TEST_ENGINE",
|
||||
.engineVersion = 0,
|
||||
.apiVersion = VK_API_VERSION_1_0,
|
||||
};
|
||||
VkInstanceCreateInfo inst_info = {
|
||||
.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO,
|
||||
.pNext = NULL,
|
||||
.pApplicationInfo = &app,
|
||||
.enabledLayerCount = 1,
|
||||
.ppEnabledLayerNames = (const char *const *)instance_layers,
|
||||
.enabledExtensionCount = 0,
|
||||
.ppEnabledExtensionNames = NULL,
|
||||
};
|
||||
err = vkCreateInstance(&inst_info, NULL, &demo->inst);
|
||||
if (VK_ERROR_LAYER_NOT_PRESENT == err) {
|
||||
// Couldn't find the validation layer
|
||||
}
|
||||
```
|
||||
|
||||
At `vkCreateInstance` and `vkCreateDevice`, the loader constructs call chains
|
||||
that include the application specified (enabled) layers.
|
||||
Order is important in the `ppEnabledLayerNames` array; array element 0 is the
|
||||
topmost (closest to the application) layer inserted in the chain and the last
|
||||
array element is closest to the driver.
|
||||
See the [Overall Layer Ordering](#overall-layer-ordering) section for more
|
||||
information on layer ordering.
|
||||
|
||||
**NOTE:** *Device Layers Are Now Deprecated*
|
||||
> `vkCreateDevice` originally was able to select layers in a similar manner to
|
||||
`vkCreateInstance`.
|
||||
> This led to the concept of "instance layers" and "device layers".
|
||||
> It was decided by Khronos to deprecate the "device layer" functionality and
|
||||
> only consider "instance layers".
|
||||
> Therefore, `vkCreateDevice` will use the layers specified at
|
||||
`vkCreateInstance`.
|
||||
> Because of this, the following items have been deprecated:
|
||||
> * `VkDeviceCreateInfo` fields:
|
||||
> * `ppEnabledLayerNames`
|
||||
> * `enabledLayerCount`
|
||||
> * The `vkEnumerateDeviceLayerProperties` function
|
||||
|
||||
|
||||
### Meta-Layers
|
||||
|
||||
Meta-layers are layers which contain an ordered list of other layers to enable.
|
||||
This is to allow grouping layers together in a specified order so that they can
|
||||
interact properly.
|
||||
Originally, this was used to group together the individual Vulkan Validation
|
||||
layers in the proper order to avoid conflicts.
|
||||
It was necessary because instead of a single Validation layer, validation was
|
||||
split into multiple component layers.
|
||||
The new `VK_LAYER_KHRONOS_validation` layer pulled everything into a single
|
||||
layer, dropping the need for meta layers.
|
||||
While not necessary for validation anymore, VkConfig does use meta layers to
|
||||
group layers together based on user's preferences.
|
||||
More can be found out about this functionality through both the
|
||||
[VkConfig documentation](https://github.com/LunarG/VulkanTools/blob/master/vkconfig/README.md)
|
||||
and the section later on the [Override Layer](#override-layer).
|
||||
|
||||
Meta-layers are detailed more in the
|
||||
[Meta-Layers](LoaderLayerInterface.md#meta-layers) section of the
|
||||
[LoaderLayerInterface.md](LoaderLayerInterface.md) file in this folder.
|
||||
|
||||
|
||||
### Implicit vs Explicit Layers
|
||||
|
||||
![Different Types of Layers](./images/loader_layer_order.png)
|
||||
|
||||
Explicit layers are layers which are enabled by an application (e.g. with the
|
||||
vkCreateInstance function as mentioned previously).
|
||||
|
||||
Implicit layers are enabled automatically by their very existence, unless
|
||||
requiring an additional manual enable step, unlike explicit layers that must be
|
||||
enabled explicitly.
|
||||
For example, certain application environments (e.g. Steam or an automotive
|
||||
infotainment system) may have layers which they always want enabled for all
|
||||
applications that they start.
|
||||
Other implicit layers may be for all applications started on a given system
|
||||
(e.g. layers that overlay frames-per-second).
|
||||
|
||||
Implicit layers have an additional requirement over explicit layers in that
|
||||
they require being able to be disabled by an environmental variable.
|
||||
This is due to the fact that they are not visible to the application and could
|
||||
cause issues.
|
||||
A good principle to keep in mind would be to define both an enable and disable
|
||||
environment variable so the users can deterministically enable the
|
||||
functionality.
|
||||
On Desktop platforms (Windows, Linux, and macOS), these enable/disable settings
|
||||
are defined in the layer's JSON file.
|
||||
|
||||
Discovery of system-installed implicit and explicit layers is described later
|
||||
in the [Layer discovery](LoaderLayerInterface#layer-discovery)
|
||||
section in the
|
||||
[LoaderLayerInterface.md](LoaderLayerInterface.md) document.
|
||||
|
||||
Implicit and explicit layers may be found in different locations based on the
|
||||
underlying operating system.
|
||||
The table below details more information:
|
||||
|
||||
<table style="width:100%">
|
||||
<tr>
|
||||
<th>Operating System</th>
|
||||
<th>Implicit Layer Identification</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Windows</td>
|
||||
<td>Implicit layers are located in a different Windows registry location
|
||||
than explicit layers.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Linux</td>
|
||||
<td>Implicit layers are located in a different directory location than
|
||||
explicit layers.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Android</td>
|
||||
<td>There is **No Support For Implicit Layers** on Android.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>macOS</td>
|
||||
<td>Implicit layers are located in a different directory location than
|
||||
explicit layers.</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
|
||||
#### Override Layer
|
||||
|
||||
The "Override Layer" is a special implicit meta-layer created by the
|
||||
[VkConfig](https://github.com/LunarG/VulkanTools/blob/master/vkconfig/README.md)
|
||||
tool and available by default when the tool is running.
|
||||
Once VkConfig exits, the override layer is removed, and the system should
|
||||
return to standard Vulkan behavior.
|
||||
Whenever the override layer is present in the layer search path, the loader will
|
||||
pull it into the layer call stack with the standard implicit layers along with
|
||||
all layers contained in the list of layers to load.
|
||||
This allows an end-user or developer to easily force on any number of layers
|
||||
and settings via VkConfig.
|
||||
|
||||
The override layer is discussed more in the
|
||||
[Override Meta-Layer](LoaderLayerInterface.md#override-meta-layer) section of the
|
||||
[LoaderLayerInterface.md](LoaderLayerInterface.md) file in this folder.
|
||||
|
||||
|
||||
### Forcing Layer Source Folders
|
||||
|
||||
Developers may need to use special, pre-production layers, without modifying
|
||||
the system-installed layers.
|
||||
|
||||
This can be accomplished in one of two ways:
|
||||
|
||||
1. Selecting specific layer paths using the
|
||||
[VkConfig](https://github.com/LunarG/VulkanTools/blob/master/vkconfig/README.md)
|
||||
tool shipped with the Vulkan SDK.
|
||||
2. Directing the loader to look for layers in specific folders by using the
|
||||
`VK_LAYER_PATH` environment variable.
|
||||
|
||||
The `VK_LAYER_PATH` environment variable can contain multiple paths separated by
|
||||
the operating-system specific path separator.
|
||||
On Windows, this is a semicolon (`;`), while on Linux and macOS it is a colon
|
||||
(`:`).
|
||||
|
||||
If `VK_LAYER_PATH` exists, the folders listed in it will be scanned for explicit
|
||||
layer manifest files.
|
||||
Implicit layer discovery is unaffected by this environment variable.
|
||||
Each directory listed should be the full pathname of a folder containing layer
|
||||
manifest files.
|
||||
|
||||
See the
|
||||
[Table of Debug Environment Variables](LoaderInterfaceArchitecture.md#table-of-debug-environment-variables)
|
||||
in the [LoaderInterfaceArchitecture.md document](LoaderInterfaceArchitecture.md)
|
||||
for more details.
|
||||
|
||||
|
||||
#### Exception for Elevated Privileges
|
||||
|
||||
For security reasons, `VK_LAYER_PATH` is ignored if running with elevated
|
||||
privileges.
|
||||
Because of this, `VK_LAYER_PATH` can only be used for applications that do not
|
||||
use elevated privileges.
|
||||
|
||||
For more information see
|
||||
[Elevated Privilege Caveats](LoaderInterfaceArchitecture.md#elevated-privilege-caveats)
|
||||
in the top-level
|
||||
[LoaderInterfaceArchitecture.md][LoaderInterfaceArchitecture.md] document.
|
||||
|
||||
|
||||
### Forcing Layers to be Enabled on Windows, Linux and macOS
|
||||
|
||||
Developers may want to enable layers that are not enabled by the given
|
||||
application they are using.
|
||||
|
||||
This can be also be accomplished in one of two ways:
|
||||
|
||||
1. Selecting specific layers using the
|
||||
[VkConfig](https://github.com/LunarG/VulkanTools/blob/master/vkconfig/README.md)
|
||||
tool shipped with the Vulkan SDK.
|
||||
2. Directing the loader to look for additional layers by name using the
|
||||
`VK_INSTANCE_LAYERS` environment variable.
|
||||
|
||||
Both can be used to enable additional layers which are not specified (enabled)
|
||||
by the application at `vkCreateInstance`.
|
||||
|
||||
The `VK_INSTANCE_LAYERS` environment variable is a list of layer names to enable
|
||||
separated by the operating-system specific path separator.
|
||||
On Windows, this is a semicolon (`;`), while on Linux and macOS it is a colon
|
||||
(`:`).
|
||||
The order of the names is relevant with the first layer name in the list being
|
||||
the top-most layer (closest to the application) and the last layer name in the
|
||||
list being the bottom-most layer (closest to the driver).
|
||||
See the [Overall Layer Ordering](#overall-layer-ordering) section for more
|
||||
information.
|
||||
|
||||
Application specified layers and user specified layers (via environment
|
||||
variables) are aggregated and duplicates removed by the loader when enabling
|
||||
layers.
|
||||
Layers specified via environment variable are top-most (closest to the
|
||||
application) while layers specified by the application are bottom-most.
|
||||
|
||||
An example of using these environment variables to activate the validation
|
||||
layer `VK_LAYER_KHRONOS_validation` on Linux or macOS is as follows:
|
||||
|
||||
```
|
||||
> $ export VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation
|
||||
```
|
||||
|
||||
See the
|
||||
[Table of Debug Environment Variables](LoaderInterfaceArchitecture.md#table-of-debug-environment-variables)
|
||||
in the [LoaderInterfaceArchitecture.md document](LoaderInterfaceArchitecture.md)
|
||||
for more details.
|
||||
|
||||
|
||||
### Overall Layer Ordering
|
||||
|
||||
The overall ordering of all layers by the loader based on the above looks
|
||||
as follows:
|
||||
|
||||
![Loader Layer Ordering](./images/loader_layer_order_calls.png)
|
||||
|
||||
Ordering may also be important internally to the list of explicit layers.
|
||||
Some layers may be dependent on other behavior being implemented before
|
||||
or after the loader calls it.
|
||||
For example: An overlay layer may want to use `VK_LAYER_KHRONOS_validation`
|
||||
to verify that the overlay layer is behaving appropriately.
|
||||
This requires putting the overlay layer closer to the application so that the
|
||||
validation layer can intercept any Vulkan API calls the overlay layer needs to
|
||||
make to function.
|
||||
|
||||
## Application Usage of Extensions
|
||||
|
||||
Extensions are optional functionality provided by a layer, the loader, or an
|
||||
implementation.
|
||||
Extensions can modify the behavior of the Vulkan API and need to be specified
|
||||
and registered with Khronos.
|
||||
These extensions can be implemented by a Vulkan implementation, the loader, or a
|
||||
layer to expose functionality not available in the core API.
|
||||
Information about various extensions can be found in the Vulkan Spec, and
|
||||
vulkan.h header file.
|
||||
|
||||
|
||||
### Instance and Device Extensions
|
||||
|
||||
As hinted at in the
|
||||
[Instance Versus Device](LoaderInterfaceArchitecture.md#instance-versus-device)
|
||||
section of the main
|
||||
[LoaderInterfaceArchitecture.md](LoaderInterfaceArchitecture.md) document,
|
||||
there are two types of extensions:
|
||||
* Instance Extensions
|
||||
* Device Extensions
|
||||
|
||||
An instance extension modifies existing behavior or implements new behavior on
|
||||
instance-level objects, such as `VkInstance` and `VkPhysicalDevice`.
|
||||
A device extension does the same for device-level objects, such as `VkDevice`,
|
||||
`VkQueue`, and `VkCommandBuffer` as well as any children of those objects.
|
||||
|
||||
It is **very** important to know what the type of an extension is because
|
||||
instance extensions must be enabled with `vkCreateInstance` while device
|
||||
extensions are enabled with `vkCreateDevice`.
|
||||
|
||||
When calling `vkEnumerateInstanceExtensionProperties` and
|
||||
`vkEnumerateDeviceExtensionProperties`, the loader discovers and aggregates all
|
||||
extensions of their respective type from layers (both explicit and implicit),
|
||||
implementations, and the loader before reporting them to the application.
|
||||
|
||||
Looking at `vulkan.h`, both functions are very similar,
|
||||
for example, the `vkEnumerateInstanceExtensionProperties` prototype looks as
|
||||
follows:
|
||||
|
||||
```
|
||||
VkResult
|
||||
vkEnumerateInstanceExtensionProperties(
|
||||
const char *pLayerName,
|
||||
uint32_t *pPropertyCount,
|
||||
VkExtensionProperties *pProperties);
|
||||
```
|
||||
|
||||
While the `vkEnumerateDeviceExtensionProperties` prototype looks like:
|
||||
|
||||
```
|
||||
VkResult
|
||||
vkEnumerateDeviceExtensionProperties(
|
||||
VkPhysicalDevice physicalDevice,
|
||||
const char *pLayerName,
|
||||
uint32_t *pPropertyCount,
|
||||
VkExtensionProperties *pProperties);
|
||||
```
|
||||
|
||||
The "pLayerName" parameter in these functions is used to select either a single
|
||||
layer or the Vulkan platform implementation.
|
||||
If "pLayerName" is NULL, extensions from Vulkan implementation components
|
||||
(including loader, implicit layers, and implementations) are enumerated.
|
||||
If "pLayerName" is equal to a discovered layer module name then only extensions
|
||||
from that layer (which may be implicit or explicit) are enumerated.
|
||||
|
||||
**Note:** While device layers are deprecated, the instance enabled layers are
|
||||
still present in the device call-chain.
|
||||
|
||||
Duplicate extensions (e.g. an implicit layer and implementation might report
|
||||
support for the same extension) are eliminated by the loader.
|
||||
For duplicates, the implementation version is reported and the layer version is
|
||||
culled.
|
||||
|
||||
Also, extensions **must be enabled** (in `vkCreateInstance` or `vkCreateDevice`)
|
||||
before the functions associated with the extensions can be used.
|
||||
If an extension function is queried using either `vkGetInstanceProcAddr` or
|
||||
`vkGetDeviceProcAddr`, but the extension has not been enabled, undefined behavior
|
||||
could result.
|
||||
The Validation layers will catch this invalid API usage.
|
||||
|
||||
|
||||
### WSI Extensions
|
||||
|
||||
Khronos-approved WSI extensions are available and provide Windows System
|
||||
Integration support for various execution environments.
|
||||
It is important to understand that some WSI extensions are valid for all
|
||||
targets, but others are particular to a given execution environment (and
|
||||
loader).
|
||||
This desktop loader (currently targeting Windows, Linux, and macOS) only
|
||||
enables and directly exports those WSI extensions that are appropriate to the
|
||||
current environment.
|
||||
For the most part, the selection is done in the loader using compile-time
|
||||
preprocessor flags.
|
||||
All versions of the desktop loader currently expose at least the following WSI
|
||||
extension support:
|
||||
- VK_KHR_surface
|
||||
- VK_KHR_swapchain
|
||||
- VK_KHR_display
|
||||
|
||||
In addition, each of the following OS targets for the loader support target-
|
||||
specific extensions:
|
||||
|
||||
| Windowing System | Extensions available |
|
||||
|----------------|--------------------|
|
||||
| Windows | VK_KHR_win32_surface |
|
||||
| Linux (Wayland) | VK_KHR_wayland_surface |
|
||||
| Linux (X11) | VK_KHR_xcb_surface and VK_KHR_xlib_surface |
|
||||
| macOS (MoltenVK) | VK_MVK_macos_surface |
|
||||
| QNX (Screen) | VK_QNX_screen_surface |
|
||||
|
||||
It is important to understand that while the loader may support the various
|
||||
entry-points for these extensions, there is a handshake required to actually
|
||||
use them:
|
||||
* At least one physical device must support the extension(s)
|
||||
* The application must use such a physical device when creating a logical
|
||||
device
|
||||
* The application must request the extension(s) be enabled while creating the
|
||||
instance or logical device (this depends on whether or not the given extension
|
||||
works with an instance or a device)
|
||||
|
||||
Only then can the WSI extension be properly used in a Vulkan program.
|
||||
|
||||
|
||||
### Unknown Extensions
|
||||
|
||||
With the ability to expand Vulkan so easily, extensions will be created that
|
||||
the loader knows nothing about.
|
||||
If the extension is a device extension, the loader will pass the unknown
|
||||
entry-point down the device call chain ending with the appropriate
|
||||
implementation entry-points.
|
||||
The same thing will happen if the extension is an instance extension which
|
||||
takes a physical device parameter as its first component.
|
||||
However, for all other instance extensions the loader will fail to load it.
|
||||
|
||||
*But why doesn't the loader support unknown instance extensions?*
|
||||
<br/>
|
||||
Let's look again at the instance call chain:
|
||||
|
||||
![Instance call chain](./images/loader_instance_chain.png)
|
||||
|
||||
Notice that for a normal instance function call, the loader has to handle
|
||||
passing along the function call to the available implementations.
|
||||
If the loader has no idea of the parameters or return value of the instance
|
||||
call, it can't properly pass information along to the implementations.
|
||||
There may be ways to do this, which will be explored in the future.
|
||||
However, for now, the loader does not support instance extensions which don't
|
||||
expose entry points that take a physical device as their first parameter.
|
||||
|
||||
Because the device call-chain does not normally pass through the loader
|
||||
*terminator*, this is not a problem for device extensions.
|
||||
Additionally, since a physical device is associated with one implementation, we
|
||||
can use a generic *terminator* pointing to one implementation.
|
||||
This is because both of these extensions terminate directly in the
|
||||
implementation they are associated with.
|
||||
|
||||
*Is this a big problem?*
|
||||
<br/>
|
||||
No!
|
||||
Most extension functionality only affects either a physical or logical device
|
||||
and not an instance.
|
||||
Thus, the overwhelming majority of extensions should be supported with direct
|
||||
loader support.
|
||||
|
||||
### Filtering Out Unknown Instance Extension Names
|
||||
|
||||
In some cases, an implementation may support instance extensions that are not
|
||||
supported by the loader.
|
||||
For the above reasons, the loader will filter out the names of these unknown
|
||||
instance extensions when an application calls
|
||||
`vkEnumerateInstanceExtensionProperties`.
|
||||
Additionally, this behavior will cause the loader to emit an error during
|
||||
`vkCreateInstance` if the application still attempts to use one of these
|
||||
extensions.
|
||||
The intent is to protect applications so that they don't inadvertently use
|
||||
functionality which could lead to a crash.
|
||||
|
||||
On the other hand, if the extension must be forced on, the filtering may be
|
||||
disabled by defining the `VK_LOADER_DISABLE_INST_EXT_FILTER` environment
|
||||
variable to a non-zero number.
|
||||
This will effectively disable the loader's filtering of instance extension
|
||||
names.
|
||||
|
||||
|
||||
[Return to the top-level LoaderInterfaceArchitecture.md file.](LoaderInterfaceArchitecture.md)
|
1169
docs/LoaderImplementationInterface.md
Normal file
786
docs/LoaderInterfaceArchitecture.md
Normal file
@ -0,0 +1,786 @@
|
||||
<!-- markdownlint-disable MD041 -->
|
||||
[![Khronos Vulkan][1]][2]
|
||||
|
||||
[1]: https://vulkan.lunarg.com/img/Vulkan_100px_Dec16.png "https://www.khronos.org/vulkan/"
|
||||
[2]: https://www.khronos.org/vulkan/
|
||||
|
||||
# Architecture of the Vulkan Loader Interfaces
|
||||
[![Creative Commons][3]][4]
|
||||
|
||||
<!-- Copyright © 2015-2021 LunarG, Inc. -->
|
||||
|
||||
[3]: https://i.creativecommons.org/l/by-nd/4.0/88x31.png "Creative Commons License"
|
||||
[4]: https://creativecommons.org/licenses/by-nd/4.0/
|
||||
## Table of Contents
|
||||
* [Overview](#overview)
|
||||
* [Who Should Read This Document](#who-should-read-this-document)
|
||||
* [The Loader](#the-loader)
|
||||
* [Layers](#layers)
|
||||
* [Implementations](#implementations)
|
||||
* [VkConfig](#vkconfig)
|
||||
|
||||
* [Important Vulkan Concepts](#important-vulkan-concepts)
|
||||
* [Instance Versus Device](#instance-versus-device)
|
||||
* [Dispatch Tables and Call Chains](#dispatch-tables-and-call-chains)
|
||||
|
||||
* [Elevated Privilege Caveats](#elevated-privilege-caveats)
|
||||
|
||||
* [Application Interface to the Loader](#application-interface-to-the-loader)
|
||||
* [Layer Interface with the Loader](#layer-interface-with-the-loader)
|
||||
* [Implementation Interface with the Loader](#implementation-interface-with-the-loader)
|
||||
|
||||
* [Table of Debug Environment Variables](#table-of-debug-environment-variables)
|
||||
* [Glossary of Terms](#glossary-of-terms)
|
||||
|
||||
## Overview
|
||||
|
||||
Vulkan is a layered architecture, made up of the following elements:
|
||||
* The Vulkan Application
|
||||
* [The Vulkan Loader](#the-loader)
|
||||
* [Vulkan Layers](#layers)
|
||||
* [Implementations](#implementations)
|
||||
* [VkConfig](#vkconfig)
|
||||
|
||||
![High Level View of Loader](./images/high_level_loader.png)
|
||||
|
||||
The general concepts in this document are applicable to the loaders available
|
||||
for Windows, Linux, Android, and macOS systems.
|
||||
|
||||
|
||||
### Who Should Read This Document
|
||||
|
||||
While this document is primarily targeted at developers of Vulkan applications,
|
||||
drivers and layers, the information contained in it could be useful to anyone
|
||||
wanting a better understanding of the Vulkan runtime.
|
||||
|
||||
|
||||
### The Loader
|
||||
|
||||
The application sits at the top and interfaces directly with the Vulkan
|
||||
loader.
|
||||
At the bottom of the stack sits the implementations.
|
||||
An implementation can control one or more physical devices capable of rendering
|
||||
Vulkan, implement a conversion from Vulkan into a native graphics API (like
|
||||
[MoltenVk](https://github.com/KhronosGroup/MoltenVK], or implement a fully
|
||||
software path that can be executed on a CPU to simulate a Vulkan device (like
|
||||
[SwiftShader](https://github.com/google/swiftshader) or LavaPipe).
|
||||
Remember, Vulkan-capable hardware may be graphics-based, compute-based, or
|
||||
both.
|
||||
Between the application and the implementations the loader can inject any
|
||||
number of optional [layers](#layers) that provide special functionality.
|
||||
The loader is critical to managing the proper dispatching of Vulkan
|
||||
functions to the appropriate set of layers and implementations.
|
||||
The Vulkan object model allows the loader to insert layers into a call-chain
|
||||
so that the layers can process Vulkan functions prior to the implementation
|
||||
being called.
|
||||
|
||||
This document is intended to provide an overview of the necessary interfaces
|
||||
between each of these.
|
||||
|
||||
|
||||
#### Goals of the Loader
|
||||
|
||||
The loader was designed with the following goals in mind:
|
||||
1. Support one or more Vulkan-capable implementations on a user's system
|
||||
without them interfering with one another.
|
||||
2. Support Vulkan Layers which are optional modules that can be enabled by an
|
||||
application, developer, or standard system settings.
|
||||
3. Keep the overall overhead of the loader to the minimum possible.
|
||||
|
||||
|
||||
### Layers
|
||||
|
||||
Layers are optional components that augment the Vulkan development environment.
|
||||
They can intercept, evaluate, and modify existing Vulkan functions on their
|
||||
way from the application down to the implementations and back up.
|
||||
Layers are implemented as libraries that can be enabled in different ways
|
||||
and are loaded during CreateInstance.
|
||||
Each layer can choose to hook, or intercept, Vulkan functions which in
|
||||
turn can be ignored, inspected, or augmented.
|
||||
Any function a layer does not hook is simply skipped for that layer and the
|
||||
control flow will simply continue on to the next supporting layer or
|
||||
implementation.
|
||||
Because of this, a layer can choose whether to intercept all known Vulkan
|
||||
functions or only a subset it is interested in.
|
||||
|
||||
Some examples of features that layers may expose include:
|
||||
* Validating API usage
|
||||
* Tracing API calls
|
||||
* Debugging aids
|
||||
* Profiling
|
||||
* Overlay
|
||||
|
||||
Because layers are optional and dynamically loaded, they can be enabled
|
||||
and disabled as desired.
|
||||
For example, while developing and debugging an application, enabling
|
||||
certain layers can assist in making sure it properly uses the Vulkan API.
|
||||
But when releasing the application, those layers are unnecessary
|
||||
and thus won't be enabled, increasing the speed of the application.
|
||||
|
||||
|
||||
### Implementations
|
||||
|
||||
The library that implements Vulkan, either through supporting a physical
|
||||
hardware device directly, converting Vulkan commands into native graphics
|
||||
commands, or simulating Vulkan through software, is considered
|
||||
"an implementation".
|
||||
The most common type of implementation is still the Installable Client Driver
|
||||
(or ICD).
|
||||
The loader is responsible for discovering available Vulkan implementations on
|
||||
the system.
|
||||
Given a list of available implementations, the loader can enumerate all the
|
||||
available physical devices and provide this information for an application.
|
||||
|
||||
|
||||
#### Installable Client Drivers
|
||||
|
||||
Vulkan allows multiple ICDs each supporting one or more devices.
|
||||
Each of these devices is represented by a Vulkan `VkPhysicalDevice` object.
|
||||
The loader is responsible for discovering available Vulkan ICDs via the standard
|
||||
implementation search on the system.
|
||||
|
||||
|
||||
### VkConfig
|
||||
|
||||
VkConfig is a tool LunarG has developed to assist with modifying the Vulkan
|
||||
environment on the local system.
|
||||
It can be used to find layers, enable them, change layer settings, and other
|
||||
useful features.
|
||||
VkConfig can be found by either installing the
|
||||
[Vulkan SDK](https://vulkan.lunarg.com/) or by building the source out of the
|
||||
[LunarG VulkanTools GitHub Repo](https://github.com/LunarG/VulkanTools).
|
||||
|
||||
VkConfig generates three outputs, two of which work with the Vulkan loader and
|
||||
layers.
|
||||
These outputs are:
|
||||
* The Vulkan Override Layer
|
||||
* The Vulkan Layer Settings File
|
||||
* VkConfig Configuration Settings
|
||||
|
||||
These files are found in different locations based on your platform:
|
||||
|
||||
<table style="width:100%">
|
||||
<tr>
|
||||
<th>Platform</th>
|
||||
<th>Output</th>
|
||||
<th>Location</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th rowspan="3">Linux</th>
|
||||
<td>Vulkan Override Layer</td>
|
||||
<td>$USER/.local/share/vulkan/implicit_layer.d/VkLayer_override.json</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Vulkan Layer Settings</td>
|
||||
<td>$USER/.local/share/vulkan/settings.d/vk_layer_settings.txt</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>VkConfig Configuration Settings</td>
|
||||
<td>$USER/.local/share/vulkan/settings.d/vk_layer_settings.txt</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th rowspan="3">Windows</th>
|
||||
<td>Vulkan Override Layer</td>
|
||||
<td>%HOME%\AppData\Local\LunarG\vkconfig\override\VkLayerOverride.json</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Vulkan Layer Settings</td>
|
||||
<td>(registry) HKEY_CURRENT_USER\Software\Khronos\Vulkan\Settings</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>VkConfig Configuration Settings</td>
|
||||
<td>(registry) HKEY_CURRENT_USER\Software\LunarG\vkconfig </td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
The [Override Meta-Layer](./LoaderLayerInterface.md#override-meta-layer) is
|
||||
an important part of how VkConfig works.
|
||||
This layer, when found by the loader, forces the loading of the desired layers
|
||||
that were enabled inside of VkConfig as well as disables those layers that
|
||||
were intentionally disabled (including implicit layers).
|
||||
|
||||
The Vulkan Layer Settings file can be used to specify certain behaviors and
|
||||
actions each enabled layer is expected to perform.
|
||||
These settings can also be controlled by VkConfig, or they can be manually
|
||||
enabled.
|
||||
For details on what settings can be used, refer to the individual layers.
|
||||
|
||||
In the future, VkConfig may have additional interactions with the Vulkan
|
||||
loader.
|
||||
|
||||
More details on VkConfig can be found in its
|
||||
[GitHub documentation](https://github.com/LunarG/VulkanTools/blob/master/vkconfig/README.md).
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
|
||||
## Important Vulkan Concepts
|
||||
|
||||
Vulkan has a few concepts that provide a fundamental basis for its organization.
|
||||
These concepts should be understood by any one attempting to use Vulkan or
|
||||
develop any of its components.
|
||||
|
||||
|
||||
### Instance Versus Device
|
||||
|
||||
An important concept to understand, which is brought up repeatedly throughout this
|
||||
document, is how the Vulkan API is organized.
|
||||
Many objects, functions, extensions, and other behavior in Vulkan can be
|
||||
separated into two groups:
|
||||
* [Instance-specific](#instance-specific)
|
||||
* [Device-specific](#device-specific)
|
||||
|
||||
|
||||
#### Instance-Specific
|
||||
|
||||
A "Vulkan instance" (`VkInstance`) is a high-level construct used to provide
|
||||
Vulkan system-level information and functionality.
|
||||
|
||||
##### Instance Objects
|
||||
|
||||
A few Vulkan objects associated directly with an instance are:
|
||||
* `VkInstance`
|
||||
* `VkPhysicalDevice`
|
||||
* `VkPhysicalDeviceGroup`
|
||||
|
||||
##### Instance Functions
|
||||
|
||||
An "instance function" is any Vulkan function where the first parameter is an
|
||||
[instance object](#instance-objects) or no object at all.
|
||||
|
||||
Some Vulkan instance functions are:
|
||||
* `vkEnumerateInstanceExtensionProperties`
|
||||
* `vkEnumeratePhysicalDevices`
|
||||
* `vkCreateInstance`
|
||||
* `vkDestroyInstance`
|
||||
|
||||
An application can link directly to all core instance functions through the
|
||||
Vulkan loader's headers.
|
||||
Alternatively, an application can query function pointers using
|
||||
`vkGetInstanceProcAddr`.
|
||||
`vkGetInstanceProcAddr` can be used to query any instance or device entry-points
|
||||
in addition to all core entry-points.
|
||||
|
||||
If `vkGetInstanceProcAddr` is called using a `VkInstance`, then any function
|
||||
pointer returned is specific to that `VkInstance` and any additional objects
|
||||
that are created from it.
|
||||
|
||||
##### Instance Extensions
|
||||
|
||||
Extensions to Vulkan are similarly associated based on what type of
|
||||
functions they provide.
|
||||
Because of this, extensions are broken up into instance or device extensions
|
||||
where most, if not all of the functions, in the extension are of the
|
||||
corresponding type.
|
||||
For example, an "instance extension" is composed primarily of "instance
|
||||
functions" which primarily take instance objects.
|
||||
These will be discussed in more detail later.
|
||||
|
||||
|
||||
#### Device-Specific
|
||||
|
||||
A Vulkan device (`VkDevice`), on the other-hand, is a logical identifier used
|
||||
to associate functions with a particular Vulkan physical device
|
||||
(`VkPhysicalDevice`) through a particular implementation on a user's system.
|
||||
|
||||
##### Device Objects
|
||||
|
||||
A few of the Vulkan constructs associated directly with a device include:
|
||||
* `VkDevice`
|
||||
* `VkQueue`
|
||||
* `VkCommandBuffer`
|
||||
|
||||
##### Device Functions
|
||||
|
||||
A "device function" is any Vulkan function which takes any device object as its
|
||||
first parameter or a child object of the device.
|
||||
The vast majority of Vulkan functions are device functions.
|
||||
Some Vulkan device functions are:
|
||||
* `vkQueueSubmit`
|
||||
* `vkBeginCommandBuffer`
|
||||
* `vkCreateEvent`
|
||||
|
||||
Vulkan devices functions may be queried using either `vkGetInstanceProcAddr` or
|
||||
`vkGetDeviceProcAddr`.
|
||||
If an application chooses to use `vkGetInstanceProcAddr`, each call will have
|
||||
additional function calls built into the call chain, which will reduce
|
||||
performance slightly.
|
||||
If, instead, the application uses `vkGetDeviceProcAddr`, the call chain will be
|
||||
more optimized to the specific device, but the returned function pointers will
|
||||
**only** work for the device used when querying them.
|
||||
Unlike `vkGetInstanceProcAddr`, `vkGetDeviceProcAddr` can only be used on
|
||||
Vulkan device functions.
|
||||
|
||||
The best solution is to query instance extension functions using
|
||||
`vkGetInstanceProcAddr`, and to query device extension functions using
|
||||
`vkGetDeviceProcAddr`.
|
||||
See
|
||||
[Best Application Performance Setup](LoaderApplicationInterface.md#best-application-performance-setup)
|
||||
section in the
|
||||
[LoaderApplicationInterface.md](LoaderApplicationInterface.md) document for more
|
||||
information on this.
|
||||
|
||||
##### Device Extensions
|
||||
|
||||
As with instance extensions, a device extension is a set of Vulkan device
|
||||
functions extending the Vulkan language.
|
||||
More information about device extensions can be found later in this document.
|
||||
|
||||
|
||||
### Dispatch Tables and Call Chains
|
||||
|
||||
Vulkan uses an object model to control the scope of a particular action or
|
||||
operation.
|
||||
The object to be acted on is always the first parameter of a Vulkan call and is
|
||||
a dispatchable object (see Vulkan specification section 2.3 Object Model).
|
||||
Under the covers, the dispatchable object handle is a pointer to a structure,
|
||||
which in turn, contains a pointer to a dispatch table maintained by the loader.
|
||||
This dispatch table contains pointers to the Vulkan functions appropriate to
|
||||
that object.
|
||||
|
||||
There are two types of dispatch tables the loader maintains:
|
||||
- Instance Dispatch Table
|
||||
- Created in the loader during the call to `vkCreateInstance`
|
||||
- Device Dispatch Table
|
||||
- Created in the loader during the call to `vkCreateDevice`
|
||||
|
||||
At that time the application and the system can each specify optional layers to
|
||||
be included.
|
||||
The loader will initialize the specified layers to create a call chain for each
|
||||
Vulkan function and each entry of the dispatch table will point to the first
|
||||
element of that chain.
|
||||
Thus, the loader builds an instance call chain for each `VkInstance` that is
|
||||
created and a device call chain for each `VkDevice` that is created.
|
||||
|
||||
When an application calls a Vulkan function, this typically will first hit a
|
||||
*trampoline* function in the loader.
|
||||
These *trampoline* functions are small, simple functions that jump to the
|
||||
appropriate dispatch table entry for the object they are given.
|
||||
Additionally, for functions in the instance call chain, the loader has an
|
||||
additional function, called a *terminator*, which is called after all enabled
|
||||
layers to marshall the appropriate information to all available implementations.
|
||||
|
||||
|
||||
#### Instance Call Chain Example
|
||||
|
||||
For example, the diagram below represents what happens in the call chain for
|
||||
`vkCreateInstance`.
|
||||
After initializing the chain, the loader calls into the first layer's
|
||||
`vkCreateInstance`, which will call the next layer's `vkCreateInstance
|
||||
before finally terminating in the loader again where it will call
|
||||
every implementation's `vkCreateInstance`.
|
||||
This allows every enabled layer in the chain to set up what it needs based on
|
||||
the `VkInstanceCreateInfo` structure from the application.
|
||||
|
||||
![Instance Call Chain](./images/loader_instance_chain.png)
|
||||
|
||||
This also highlights some of the complexity the loader must manage when using
|
||||
instance call chains.
|
||||
As shown here, the loader's *terminator* must aggregate information to and from
|
||||
multiple implementations when they are present.
|
||||
This implies that the loader has to be aware of any instance-level extensions
|
||||
which work on a `VkInstance` to aggregate them correctly.
|
||||
|
||||
|
||||
#### Device Call Chain Example
|
||||
|
||||
Device call chains are created in `vkCreateDevice` and are generally simpler
|
||||
because they deal with only a single device.
|
||||
This allows for the specific implementation exposing this device to always be
|
||||
the *terminator* of the chain.
|
||||
|
||||
![Loader Device Call Chain](./images/loader_device_chain_loader.png)
|
||||
<br/>
|
||||
|
||||
|
||||
## Elevated Privilege Caveats
|
||||
|
||||
To ensure that the system is safe from exploitation, Vulkan applications which
|
||||
are run with elevated privileges are restricted from certain operations, such
|
||||
as reading environment variables from unsecure locations or searching for
|
||||
files in user controlled paths.
|
||||
This is done to ensure that an application running with elevated privileges does
|
||||
not run using components that were not installed in the proper approved
|
||||
locations.
|
||||
|
||||
The loader uses platform-specific mechanisms (such as `secure_getenv` and its
|
||||
equivalents) for querying sensitive environment variables to avoid accidentally
|
||||
using untrusted results.
|
||||
|
||||
These behaviors also result in ignoring certain environment variables, such as:
|
||||
|
||||
* `VK_ICD_FILENAMES`
|
||||
* `VK_LAYER_PATH`
|
||||
* `XDG_CONFIG_HOME` (Linux/Mac-specific)
|
||||
* `XDG_DATA_HOME` (Linux/Mac-specific)
|
||||
|
||||
For more information on the affected search paths, refer to
|
||||
[Layer Discovery](LoaderLayerInterface.md#layer-discovery) and
|
||||
[Implementation Discovery](LoaderImplementationInterface.md#implementation-discovery).
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
|
||||
## Application Interface to the Loader
|
||||
|
||||
The Application interface to the Vulkan loader is now detailed in the
|
||||
[LoaderApplicationInterface.md](LoaderApplicationInterface.md) document found in
|
||||
the same directory as this file.
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
|
||||
## Layer Interface with the Loader
|
||||
|
||||
The Layer interface to the Vulkan loader is detailed in the
|
||||
[LoaderLayerInterface.md](LoaderLayerInterface.md) document found in the same
|
||||
directory as this file.
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
|
||||
## Implementation Interface With the Loader
|
||||
|
||||
The Implementation interface to the Vulkan loader is detailed in the
|
||||
[LoaderImplementationInterface.md](LoaderImplementationInterface.md) document
|
||||
found in the same directory as this file.
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
|
||||
## Table of Debug Environment Variables
|
||||
|
||||
The following are all the Debug Environment Variables available for use with the
|
||||
Loader.
|
||||
These are referenced throughout the text, but collected here for ease of
|
||||
discovery.
|
||||
|
||||
<table style="width:100%">
|
||||
<tr>
|
||||
<th>Environment Variable</th>
|
||||
<th>Behavior</th>
|
||||
<th>Example Format</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><small><i>VK_ICD_FILENAMES</i></small></td>
|
||||
<td>Force the loader to use the specific ICD JSON files.
|
||||
The value contains a list of delimited full path listings to
|
||||
implementation JSON Manifest files.<br/>
|
||||
<b>NOTE:</b> If a global path to the JSON file is not used, issues
|
||||
may be encountered.<br/>
|
||||
<b>Ignored when running Vulkan application in executing with
|
||||
elevated privileges.</b>
|
||||
See <a href="#elevated-privilege-caveats">Elevated Privilege Caveats</a>
|
||||
for more information.
|
||||
</td>
|
||||
<td><small>export<br/>
|
||||
VK_ICD_FILENAMES=<br/>
|
||||
<folder_a>/intel.json:<folder_b>/amd.json
|
||||
<br/> <br/>
|
||||
set<br/>
|
||||
VK_ICD_FILENAMES=<br/>
|
||||
<folder_a>\nvidia.json;<folder_b>\mesa.json
|
||||
</small>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><small><i>VK_INSTANCE_LAYERS</i></small></td>
|
||||
<td>Force the loader to add the given layers to the list of Enabled layers
|
||||
normally passed into <b>vkCreateInstance</b>.
|
||||
These layers are added first, and the loader will remove any duplicate
|
||||
layers that appear in both this list as well as that passed into
|
||||
<i>ppEnabledLayerNames</i>.
|
||||
</td>
|
||||
<td><small>export<br/>
|
||||
VK_INSTANCE_LAYERS=<br/>
|
||||
<layer_a>;<layer_b><br/><br/>
|
||||
set<br/>
|
||||
VK_INSTANCE_LAYERS=<br/>
|
||||
<layer_a>;<layer_b></small>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><small><i>VK_LAYER_PATH</i></small></td>
|
||||
<td>Override the loader's standard Layer library search folders and use the
|
||||
provided delimited folders to search for explicit layer manifest files.
|
||||
<br/>
|
||||
<b>Ignored when running Vulkan application in executing with
|
||||
elevated privileges.</b>
|
||||
See <a href="#elevated-privilege-caveats">Elevated Privilege Caveats</a>
|
||||
for more information.
|
||||
</td>
|
||||
<td><small>export<br/>
|
||||
VK_LAYER_PATH=<br/>
|
||||
<path_a>;<path_b><br/><br/>
|
||||
set<br/>
|
||||
VK_LAYER_PATH=<br/>
|
||||
<path_a>;<path_b></small>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><small><i>VK_LOADER_DISABLE_INST_EXT_FILTER</i></small></td>
|
||||
<td>Disable the filtering out of instance extensions that the loader doesn't
|
||||
know about.
|
||||
This will allow applications to enable instance extensions exposed by
|
||||
implementations but that the loader has no support for.<br/>
|
||||
<b>NOTE:</b> This may cause the loader or application to crash.</td>
|
||||
<td><small>export<br/>
|
||||
VK_LOADER_DISABLE_INST_EXT_FILTER=1<br/><br/>
|
||||
set<br/>
|
||||
VK_LOADER_DISABLE_INST_EXT_FILTER=1</small>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><small><i>VK_LOADER_DEBUG</i></small></td>
|
||||
<td>Enable loader debug messages using a comma-delimited list of level
|
||||
options. These options are:<br/>
|
||||
* error (only errors)<br/>
|
||||
* warn (only warnings)<br/>
|
||||
* info (only info)<br/>
|
||||
* debug (only debug)<br/>
|
||||
* layer (layer-specific output)<br/>
|
||||
* implement (implementation-specific output)<br/>
|
||||
* all (report out all messages)<br/><br/>
|
||||
To enable multiple options (outside of "all") like info, warning and
|
||||
error messages, set the value to "error,warn,info".
|
||||
</td>
|
||||
<td><small>export<br/>
|
||||
VK_LOADER_DEBUG=all<br/>
|
||||
<br/>
|
||||
set<br/>
|
||||
VK_LOADER_DEBUG=warn</small>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
## Glossary of Terms
|
||||
|
||||
<table style="width:100%">
|
||||
<tr>
|
||||
<th>Field Name</th>
|
||||
<th>Field Value</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Android Loader</td>
|
||||
<td>The loader designed to work primarily for the Android OS.
|
||||
This is generated from a different code base than the desktop loader.
|
||||
But, in all important aspects, it should be functionally equivalent.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Desktop Loader</td>
|
||||
<td>The loader designed to work on Windows, Linux and macOS.
|
||||
This is generated from a different
|
||||
<a href="https://github.com/KhronosGroup/Vulkan-Loader">code base</a>
|
||||
than the Android loader.
|
||||
But in all important aspects, it should be functionally equivalent.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Core Function</td>
|
||||
<td>A function that is already part of the Vulkan core specification and not
|
||||
an extension. <br/>
|
||||
For example, <b>vkCreateDevice()</b>.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Device Call Chain</td>
|
||||
<td>The call chain of functions followed for device functions.
|
||||
This call chain for a device function is usually as follows: first the
|
||||
application calls into a loader trampoline, then the loader trampoline
|
||||
calls enabled layers, and the final layer calls into the implementation
|
||||
specific to the device. <br/>
|
||||
See the
|
||||
<a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
|
||||
Chains</a> section for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Device Function</td>
|
||||
<td>A device function is any Vulkan function which takes a <i>VkDevice</i>,
|
||||
<i>VkQueue</i>, <i>VkCommandBuffer</i>, or any child of these, as its
|
||||
first parameter. <br/><br/>
|
||||
Some Vulkan device functions are: <br/>
|
||||
<b>vkQueueSubmit</b>, <br/>
|
||||
<b>vkBeginCommandBuffer</b>, <br/>
|
||||
<b>vkCreateEvent</b>. <br/><br/>
|
||||
See the <a href="#instance-versus-device">Instance Versus Device</a>
|
||||
section for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Discovery</td>
|
||||
<td>The process of the loader searching for implementation and layer files
|
||||
to set up the internal list of Vulkan objects available.<br/>
|
||||
On <i>Windows/Linux/macOS</i>, the discovery process typically focuses on
|
||||
searching for Manifest files.<br/>
|
||||
On <i>Android</i>, the process focuses on searching for library files.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Dispatch Table</td>
|
||||
<td>An array of function pointers (including core and possibly extension
|
||||
functions) used to step to the next entity in a call chain.
|
||||
The entity could be the loader, a layer or an implementation.<br/>
|
||||
See <a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
|
||||
Chains</a> for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Extension</td>
|
||||
<td>A concept of Vulkan used to expand the core Vulkan functionality.
|
||||
Extensions may be IHV-specific, platform-specific, or more broadly
|
||||
available. <br/>
|
||||
Always first query if an extension exists, and enable it during
|
||||
<b>vkCreateInstance</b> (if it is an instance extension) or during
|
||||
<b>vkCreateDevice</b> (if it is a device extension) before attempting
|
||||
to use it. <br/>
|
||||
Extensions will always have an author prefix or suffix modifier to every
|
||||
structure, enumeration entry, command entry-point, or define that is
|
||||
associated with it.
|
||||
For example, `KHR` is the prefix for Khronos authored extensions and
|
||||
will also be found on structures, enumeration entries, and commands
|
||||
associated with those extensions.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Extension Function</td>
|
||||
<td>A function that is defined as part of an extension and not part of the
|
||||
Vulkan core specification. <br/>
|
||||
As with the extension the function is defined as part of, it will have a
|
||||
suffix modifier indicating the author of the extension.<br/>
|
||||
Some example extension suffixes include:<br/>
|
||||
<b>KHR</b> - For Khronos authored extensions, <br/>
|
||||
<b>EXT</b> - For multi-company authored extensions, <br/>
|
||||
<b>AMD</b> - For AMD authored extensions, <br/>
|
||||
<b>ARM</b> - For ARM authored extensions, <br/>
|
||||
<b>NV</b> - For Nvidia authored extensions.<br/>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ICD</td>
|
||||
<td>Acronym for "Installable Client Driver".
|
||||
These are drivers that are provided by IHVs to interact with the
|
||||
hardware they provide. <br/>
|
||||
These are the most common type of Vulkan implementations. <br/>
|
||||
See <a href="#installable-client-drivers">Installable Client Drivers</a>
|
||||
section for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>IHV</td>
|
||||
<td>Acronym for an "Independent Hardware Vendor".
|
||||
Typically the company that built the underlying hardware technology
|
||||
that is being used. <br/>
|
||||
A typical examples for a Graphics IHV include (but not limited to):
|
||||
AMD, ARM, Imagination, Intel, Nvidia, Qualcomm
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Implementation</td>
|
||||
<td>The underlying library which provides support for the Vulkan API.
|
||||
This support can be implemented as either an ICD, API translation
|
||||
library, or pure software.<br/>
|
||||
See <a href="#implementations">Implementations</a> section for more
|
||||
information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Instance Call Chain</td>
|
||||
<td>The call chain of functions followed for instance functions.
|
||||
This call chain for an instance function is usually as follows: first
|
||||
the application calls into a loader trampoline, then the loader
|
||||
trampoline calls enabled layers, the final layer calls a loader
|
||||
terminator, and the loader terminator calls all available
|
||||
implementations. <br/>
|
||||
See the <a href="#dispatch-tables-and-call-chains">Dispatch Tables and
|
||||
Call Chains</a> section for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Instance Function</td>
|
||||
<td>An instance function is any Vulkan function which takes as its first
|
||||
parameter either a <i>VkInstance</i> or a <i>VkPhysicalDevice</i> or
|
||||
nothing at all. <br/><br/>
|
||||
Some Vulkan instance functions are:<br/>
|
||||
<b>vkEnumerateInstanceExtensionProperties</b>, <br/>
|
||||
<b>vkEnumeratePhysicalDevices</b>, <br/>
|
||||
<b>vkCreateInstance</b>, <br/>
|
||||
<b>vkDestroyInstance</b>. <br/><br/>
|
||||
See the <a href="#instance-versus-device">Instance Versus Device</a>
|
||||
section for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Layer</td>
|
||||
<td>Layers are optional components that augment the Vulkan system.
|
||||
They can intercept, evaluate, and modify existing Vulkan functions on
|
||||
their way from the application down to the implementation.<br/>
|
||||
See the <a href="#layers">Layers</a> section for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Layer Library</td>
|
||||
<td>The <b>Layer Library</b> is the group of all layers the loader is able
|
||||
to discover.
|
||||
These may include both implicit and explicit layers.
|
||||
These layers are available for use by applications unless disabled in
|
||||
some way.
|
||||
For more info, see
|
||||
<a href="LoaderLayerInterface.md#layer-layer-discovery">Layer Discovery
|
||||
</a>.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Loader</td>
|
||||
<td>The middleware program which acts as the mediator between Vulkan
|
||||
applications, Vulkan layers, and Vulkan implementations.<br/>
|
||||
See <a href="#the-loader">The Loader</a> section for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Manifest Files</td>
|
||||
<td>Data files in JSON format used by the desktop loader.
|
||||
These files contain specific information for either a
|
||||
<a href="LoaderLayerInterface.md#layer-manifest-file-format">Layer</a>
|
||||
or an
|
||||
<a href="LoaderImplementationInterface.md#icd-manifest-file-format">Implementation</a>
|
||||
and define necessary information such as where to find files and default
|
||||
settings.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Terminator Function</td>
|
||||
<td>The last function in the instance call chain above the implementation
|
||||
and owned by the loader.
|
||||
This function is required in the instance call chain because all
|
||||
instance functionality must be communicated to all implementations
|
||||
capable of receiving the call. <br/>
|
||||
See <a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
|
||||
Chains</a> for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Trampoline Function</td>
|
||||
<td>The first function in an instance or device call chain owned by the
|
||||
loader which handles the set up and proper call chain walk using the
|
||||
appropriate dispatch table.
|
||||
On device functions (in the device call chain) this function can
|
||||
actually be skipped.<br/>
|
||||
See <a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
|
||||
Chains</a> for more information.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>WSI Extension</td>
|
||||
<td>Acronym for Windowing System Integration.
|
||||
A Vulkan extension targeting a particular Windowing system and designed
|
||||
to interface between the Windowing system and Vulkan.<br/>
|
||||
See
|
||||
<a href="LoaderApplicationInterface.md#wsi-extensions">WSI Extensions</a>
|
||||
for more information.
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
1748
docs/LoaderLayerInterface.md
Normal file
BIN
docs/images/function_device_chain.png
Normal file
After Width: | Height: | Size: 128 KiB |
BIN
docs/images/function_instance_chain.png
Normal file
After Width: | Height: | Size: 206 KiB |
BIN
docs/images/high_level_loader.png
Normal file
After Width: | Height: | Size: 194 KiB |
BIN
docs/images/loader_device_chain_app.png
Normal file
After Width: | Height: | Size: 36 KiB |
BIN
docs/images/loader_device_chain_loader.png
Normal file
After Width: | Height: | Size: 48 KiB |
BIN
docs/images/loader_device_surface_chain.png
Normal file
After Width: | Height: | Size: 56 KiB |
BIN
docs/images/loader_instance_chain.png
Normal file
After Width: | Height: | Size: 78 KiB |
BIN
docs/images/loader_layer_order.png
Normal file
After Width: | Height: | Size: 106 KiB |
BIN
docs/images/loader_layer_order_calls.png
Normal file
After Width: | Height: | Size: 109 KiB |
6
docs/images/svgs/README.txt
Normal file
@ -0,0 +1,6 @@
|
||||
The svg image files in this folder were created using Inkscape 1.0.1 and
|
||||
are all copyrighted by Valve Corporation and LunarG, Inc under the
|
||||
Apache 2.0 license (as define in the top LICENSE.txt file of this repo).
|
||||
|
||||
For updating, simply load into Inkscape, modify, and save the resulting
|
||||
images to the folder above this one.
|
2802
docs/images/svgs/function_device_chain.svg
Executable file
After Width: | Height: | Size: 114 KiB |
3481
docs/images/svgs/function_instance_chain.svg
Executable file
After Width: | Height: | Size: 148 KiB |
2919
docs/images/svgs/high_level_loader.svg
Executable file
After Width: | Height: | Size: 164 KiB |
1027
docs/images/svgs/loader_device_chain_app.svg
Executable file
After Width: | Height: | Size: 51 KiB |
1081
docs/images/svgs/loader_device_chain_loader.svg
Executable file
After Width: | Height: | Size: 57 KiB |
1237
docs/images/svgs/loader_device_surface_chain.svg
Normal file
After Width: | Height: | Size: 66 KiB |
1479
docs/images/svgs/loader_instance_chain.svg
Executable file
After Width: | Height: | Size: 82 KiB |
1766
docs/images/svgs/loader_layer_order.svg
Executable file
After Width: | Height: | Size: 93 KiB |
1661
docs/images/svgs/loader_layer_order_calls.svg
Normal file
After Width: | Height: | Size: 93 KiB |
@ -1,5 +1,5 @@
|
||||
# Loader Specification and Interfaces
|
||||
See LoaderAndLayerInterface.md for detailed documentation.
|
||||
See the [LoaderInterfaceArchitecture.md](../docs/LoaderInterfaceArchitecture.md) for detailed documentation.
|
||||
|
||||
# Building
|
||||
Builds for Linux, Windows, and MacOS are supported via CMake. See top level BUILD.md file.
|
||||
|
Before Width: | Height: | Size: 113 KiB |
Before Width: | Height: | Size: 118 KiB |
Before Width: | Height: | Size: 131 KiB |
Before Width: | Height: | Size: 138 KiB |
Before Width: | Height: | Size: 90 KiB |
Before Width: | Height: | Size: 94 KiB |
Before Width: | Height: | Size: 34 KiB |
Before Width: | Height: | Size: 35 KiB |
Before Width: | Height: | Size: 38 KiB |
Before Width: | Height: | Size: 39 KiB |
Before Width: | Height: | Size: 48 KiB |
Before Width: | Height: | Size: 49 KiB |
Before Width: | Height: | Size: 56 KiB |
Before Width: | Height: | Size: 58 KiB |