slang-shaders/spec/SHADER_SPEC.md

Vulkan GLSL RetroArch shader system

This document is a draft of RetroArch's new GPU shader system. It will outline the features in the new shader subsystem and describe details for how it will work in practice.

In addition this document will contain various musings on why certain design choices are made and which compromised have been made to arrive at the conclusion. This is mostly for discussing and deliberation while the new system is under development.

Introduction

Target shader languages

• Vulkan
• GL 2.x (legacy desktop)
• GL 3.x+ (modern desktop)
• GLES2 (legacy mobile)
• GLES3 (modern mobile)
• (HLSL, potentially)
• (Metal, potentially)

RetroArch is still expected to run on GLES2 and GL2 systems. GL2 is mostly not relevant any longer, but GLES2 is certainly a very relevant platform still and having GLES2 compatibility makes GL2 very easy. We therefore want to avoid speccing out a design which deliberately ruins GLES2 compatibility.

However, we also do not want to artificially limit ourselves to shader features which are only available in GLES2. There are many shader builtins for example which only work in GLES3/GL3 and we should not hold back support in these cases. When we want to consider GLES2 compat we should not spec out high level features which do not make much sense in the context of GLES2.

Why a new spec?

The current shader subsystem in RetroArch is quite mature with a large body of shaders written for it. While it has served us well, it is not forward-compatible.

The current state of writing high-level shading languages that work "everywhere" is very challenging. There was no good ready-made solution for this. Up until now, we have relied on nVidia Cg to serve as a basic foundation for shaders, but Cg has been discontinued for years and is closed source. This is very problematic since Cg is not a forward compatible platform. It has many warts which are heavily tied in to legacy APIs and systems. For this reason, we cannot use Cg for newer APIs such as Vulkan and potentially D3D12 and Metal.

Cg cross compilation to GLSL is barely working and it is horribly unmaintainable with several unfixable issues. The output is so horribly mangled and unoptimized that it is clearly not the approach we should be taking. We also cannot do the Cg transform in runtime on mobile due to lack of open source Cg runtime, so there's that as well.

Another alternative is to write straight-up GLSL, but this too has some severe problems. All the different GL versions and GLSL variants are different enough that it becomes painful to write portable GLSL code that works without modification. Examples include:

• varying/attribute vs in/out (legacy vs modern)
• precision qualifiers (GLSL vs ESSL)
• texture2D vs texture (legacy vs modern)
• Lack of standard support for #include to reduce copy-pasta

The problem really is that GLSL shaders are dependent on the runtime GL version, which makes it very annoying and hard to test all shader variants.

We do not want to litter every shader with heaps of #ifdefs everywhere to combat this problem. We also want to avoid having to write pseudo-GLSL with some text based replacement behind the scenes.

Vulkan GLSL as the portable solution

Fortunately, there is now a forward looking and promising solution to our problems. Vulkan GLSL is a GLSL dialect designed for Vulkan and SPIR-V intermediate representation. The good part is that we can use whatever GLSL version we want when writing shaders, as it is decoupled from the GL runtime.

In runtime, we can have a vendor-neutral mature compiler, https://github.com/KhronosGroup/glslang which compiles our Vulkan GLSL to SPIR-V. Using https://github.com/KhronosGroup/SPIRV-Cross, we can then do reflection on the SPIR-V binary to deduce our filter chain layout. We can also disassemble back to our desired GLSL dialect in the GL backend based on which GL version we're running, which effectively means we can completely sidestep all our current problems with a pure GLSL based shading system.

Another upside of this is that we no longer have to deal with vendor-specific quirks in the GLSL frontend. A common problem when people write for nVidia is that people mistakingly use float2/float3/float4 types from Cg/HLSL, which is supported as an extension in their GLSL frontend.

Why not SPIR-V directly?

This was considered, but there are several convenience problems with having a shading spec around pure SPIR-V. The first problem is metadata. In GLSL, we can quite easily extend with custom #pragmas or similar, but there is no trivial way to do this in SPIR-V outside writing custom tools to emit special metadata as debug information or similar with OpSource.

We could also have this metadata outside in a separate file, but juggling more files means more churn, which we should try to avoid. The other problem is convenience. If RetroArch only accepts SPIR-V, we would need an explicit build step outside RetroArch first before we could test a shader. This gets very annoying during shader development, so it is clear that we need to support GLSL anyways, making SPIR-V support kinda redundant.

The main argument for supporting SPIR-V would be to allow new shading languages to be used. This is a reasonable thing to consider, which is why the goal is to not design ourselves into a corner where it's only Vulkan GLSL that can possibly work down the line. We are open to the idea that new shading languages that target SPIR-V will emerge.

Warts in old shader system

While the old shader system is functional it has some severe warts which have accumulated over time. In hindsight, some of the early design decisions were misguided and need to be properly fixed.

Forced POT with padding

This is arguably the largest wart of them all. The original reason behind this design decision was caused by a misguided effort to combat FP precision issues with texture sampling. The idea at the time was to avoid cases where nearest neighbor sampling at texel edges would cause artifacts. This is a typical case when textures are scaled with non-integer factors. However, the problem to begin with is naive nearest neighbor and non-integer scaling factors, and not FP precision. It was pure luck that POT tended to give better results with broken shaders, but we should not make this mistake again. POT padding has some severe issues which are not just cleanliness related either.

Technically, GLES2 doesn't require non-POT support, but in practice, all GPUs support this.

No proper UV wrapping

Since the texture "ends" at UV coords < 1.0, we cannot properly use sampler wrapping modes. We can only fake CLAMP_TO_BORDER by padding with black color, but this filtering mode is not available by default in GLES2 and even GLES3! CLAMP_TO_BORDER isn't necessarily what we want either. CLAMP_TO_EDGE is usually a far more sane default.

Extra arguments for actual width vs. texture width

With normalized coordinates we need to think in both real resolution (e.g. 320x240) vs. POT padded resolutions (512x512) to deal with normalized UV coords. This complicates things massively and we were passing an insane amount of attributes and varyings to deal with this because the ratios between the two needn't be the same for two different textures.

Arbitrary limits

The way the old shader system deals with limits is quite naive. There is a hard limit of 8 when referencing other passes and older frames. There is no reason why we should have arbitrary limits like these. Part of the reason is C where dealing with dynamic memory is more painful than is should be so it was easier to take the lazy way out.

Tacked on format handling

In more complex shaders we need to consider more than just the plain RGBA8_UNORM format. The old shader system tacked on these things after the fact by adding booleans for SRGB and FP support, but this obviously doesn't scale. This point does get problematic since GLES2 has terrible support for render target formats, but we should allow complex shaders to use complex RT formats and rather just allow some shader presets to drop GLES2 compat.

PASS vs PASSPREV

Ugly. We do not need two ways to access previous passes, the actual solution is to have aliases for passes instead and access by name.

Inconsistencies in parameter passing

MVP matrices are passed in with weird conventions in the Cg spec, and its casing is weird. The source texture is passed with magic TEXUNIT0 semantic while other textures are passed via uniform struct members, etc. This is the result of tacking on feature support slowly over time without proper forethought.

High level Overview

The RetroArch shader format outlines a filter chain/graph, a series of shader passes which operate on previously generated data to produce a final result. The goal is for every individual pass to access information from all previous shader passes, even across frames, easily.

• The filter chain specifies a number of shader passes to be executed one after the other.
• Each pass renders a full-screen quad to a texture of a certain resolution and format.
• The resolution can be dependent on external information.
• All filter chains begin at an input texture, which is created by a libretro core or similar.
• All filter chains terminate by rendering to the "backbuffer".

The backbuffer is somewhat special since the resolution of it cannot be controlled by the shader. It can also not be fed back into the filter chain later because the frontend (here RetroArch) will render UI elements and such on top of the final pass output.

Let's first look at what we mean by filter chains and how far we can expand this idea.

Simplest filter chain

The simplest filter chain we can specify is a single pass.

(Input) -> [ Shader Pass #0 ] -> (Backbuffer)

In this case there are no offscreen render targets necessary since our input is rendered directly to screen.

Multiple passes

A trivial extension is to keep our straight line view of the world where each pass looks at the previous output.

(Input) -> [ Shader Pass #0 ] -> (Framebuffer) -> [ Shader Pass #1 ] -> (Backbuffer)

Framebuffer here might have a different resolution than both Input and Backbuffer. A very common scenario for this is separable filters where we first scale horizontally, then vertically.

Multiple passes and multiple inputs

There is no reason why we should restrict ourselves to a straight-line view.

     /------------------------------------------------\
    /                                                  v
(Input) -> [ Shader Pass #0 ] -> (Framebuffer #0) -> [ Shader Pass #1 ] -> (Backbuffer)

In this scenario, we have two inputs to shader pass #1, both the original, untouched input as well as the result of a pass in-between. All the inputs to a pass can have different resolutions. We have a way to query the resolution of individual textures to allow highly controlled sampling.

We are now at a point where we can express an arbitrarily complex filter graph, but we can do better. For certain effects, time (or rather, results from earlier frames) can be an important factor.

Multiple passes, multiple inputs, with history

We now extend our filter graph, where we also have access to information from earlier frames. Note that this is still a causal filter system.

Frame N:        (Input     N, Input N - 1, Input N - 2) -> [ Shader Pass #0 ] -> (Framebuffer     N, Framebuffer N - 1, Input N - 3) -> [ Shader Pass #1 ] -> (Backbuffer)
Frame N - 1:    (Input N - 1, Input N - 2, Input N - 3) -> [ Shader Pass #0 ] -> (Framebuffer N - 1, Framebuffer N - 2, Input N - 4) -> [ Shader Pass #1 ] -> (Backbuffer)
Frame N - 2:    (Input N - 2, Input N - 3, Input N - 4) -> [ Shader Pass #0 ] -> (Framebuffer N - 2, Framebuffer N - 3, Input N - 5) -> [ Shader Pass #1 ] -> (Backbuffer)

For framebuffers we can read the previous frame's framebuffer. We don't really need more than one frame of history since we have a feedback effect in place. Just like IIR filters, the "response" of such a feedback in the filter graph gives us essentially "infinite" history back in time, although it is mostly useful for long-lasting blurs and ghosting effects. Supporting more than one frame of feedback would also be extremely memory intensive since framebuffers tend to be much higher resolution than their input counterparts. One frame is also a nice "clean" limit. Once we go beyond just 1, the floodgate opens to arbitrary numbers, which we would want to avoid. It is also possible to fake as many feedback frames of history we want anyways, since we can copy a feedback frame to a separate pass anyways which effectively creates a "shift register" of feedback framebuffers in memory.

Input textures can have arbitrary number of textures as history (just limited by memory). They cannot feedback since the filter chain cannot render into it, so it effectively is finite response (FIR).

For the very first frames, frames with frame N < 0 are transparent black (all values 0).

No POT padding

No texture in the filter chain is padded at any time. It is possible for resolutions in the filter chain to vary over time which is common with certain emulated systems. In this scenarios, the textures and framebuffers are simply resized appropriately. Older frames still keep their old resolution in the brief moment that the resolution is changing.

It is very important that shaders do not blindly sample with nearest filter with any scale factor. If naive nearest neighbor sampling is to be used, shaders must make sure that the filter chain is configured with integer scaling factors so that ambiguous texel-edge sampling is avoided.

Deduce shader inputs by reflection

We want to have as much useful information in the shader source as possible. We want to avoid having to explicitly write out metadata in shaders whereever we can. The biggest hurdle to overcome is how we describe our pipeline layout. The pipeline layout contains information about how we access resources such as uniforms and textures. There are three main types of inputs in this shader system.

• Texture samplers (sampler2D)
• Look-up textures for static input data
• Uniform data describing dimensions of textures
• Uniform ancillary data for render target dimensions, backbuffer target dimensions, frame count, etc
• Uniform user-defined parameters
• Uniform MVP for vertex shader

Deduction by name

There are two main approaches to deduce what a sampler2D uniform wants to sample from. The first way is to explicitly state somewhere else what that particular sampler needs, e.g.

uniform sampler2D geeWhatAmI;

// Metadata somewhere else
SAMPLER geeWhatAmI = Input[-2]; // Input frame from 2 frames ago

The other approach is to have built-in identifiers which correspond to certain textures.

// Source here being defined as the texture from previous framebuffer pass or the input texture if this is the first pass in the chain.
uniform sampler2D Source;

In SPIR-V, we can use OpName to describe these names, so we do not require the original Vulkan GLSL source to perform this reflection. We use this approach throughout the specification. An identifier is mapped to an internal meaning (semantic). The shader backend looks at these semantics and constructs a filter chain based on all shaders in the chain.

Identifiers can also have user defined meaning, either as an alias to existing identifiers or mapping to user defined parameters.

Combining vertex and fragment into a single shader file

One strength of Cg is its ability to contain multiple shader stages in the same .cg file. This is very convenient since we always want to consider vertex and fragment together. This is especially needed when trying to mix and match shaders in a GUI window for example. We don't want to require users to load first a vertex shader, then fragment manually.

GLSL however does not support this out of the box. This means we need to define a light-weight system for preprocessing one GLSL source file into multiple stages.

Should we make vertex optional?

In most cases, the vertex shader will remain the same. This leaves us with the option to provide a "default" vertex stage if the shader stage is not defined.

#include support

With complex filter chains there is a lot of oppurtunity to reuse code. We therefore want light support for the #include directive.

User parameter support

Since we already have a "preprocessor" of sorts, we can also trivially extend this idea with user parameters. In the shader source we can specify which uniform inputs are user controlled, GUI visible name, their effective range, etc.

Lookup textures

A handy feature to have is reading from lookup textures. We can specify that some sampler inputs are loaded from a PNG file on disk as a plain RGBA8 texture.

Do we want to support complex reinterpretation?

There could be valid use cases for supporting other formats than plain RGBA8_UNORM. SRGB and UINT might be valid cases as well and maybe even 2x16-bit, 1x32-bit integer formats.

Lookup buffers

Do we want to support lookup buffers as UBOs as well? This wouldn't be doable in GLES2, but it could be useful as a more modern feature. If the LUT is small enough, we could realize it via plain old uniforms as well perhaps.

This particular feature could be very interesting for generic polyphase lookup banks with different LUT files for different filters.

Vulkan GLSL specification

This part of the spec considers how Vulkan GLSL shaders are written. The frontend uses the glslang frontend to compile GLSL sources. This ensures that we do not end up with vendor-specific extensions. The #version string should be as recent as possible, e.g. #version 450 or #version 310 es. It is recommended to use 310 es since it allows mediump which can help on mobile. Note that after the Vulkan GLSL is turned into SPIR-V, the original #version string does not matter anymore. Also note that SPIR-V cannot be generated from legacy shader versions such as #version 100 (ES 2.0) or #version 120 (GL 2.1).

The frontend will use reflection on the resulting SPIR-V file in order to deduce what each element in the UBO or what each texture means. The main two types of data passed to shaders are read-only and can be classified as:

• uniform sampler2D: This is used for input textures, framebuffer results and lookup-textures.
• uniform Block { };: This is used for any constant data which is passed to the shader.
• layout(push_constant) uniform Push {} name;: This is used for any push constant data which is passed to the shader.

Resource usage rules

Certain rules must be adhered to in order to make it easier for the frontend to dynamically set up bindings to resources.

• All resources must be using descriptor set #0, or don't use layout(set = #N) at all.
• layout(binding = #N) must be declared for all UBOs and sampler2Ds.
• All resources must use different bindings.
• There can be only one UBO.
• There can be only use push constant block.
• It is possible to have one regular UBO and one push constant UBO.
• If a UBO is used in both vertex and fragment, their binding number must match.
• If a UBO is used in both vertex and fragment, members with the same name must have the same offset/binary interface. This problem is easily avoided by having the same UBO visible to both vertex and fragment as "common" code.
• If a push constant block is used in both vertex and fragment, members with the same name must have the same offset/binary interface.
• sampler2D cannot be used in vertex, although the size parameters of samplers can be used in vertex.
• Other resource types such as SSBOs, images, atomic counters, etc, etc, are not allowed.
• Every member of the UBOs and push constant blocks as well as every texture must be meaningful to the frontend in some way, or an error is generated.

Initial preprocess of slang files

The very first line of a .slang file must contain a #version statement.

The first process which takes place is dealing with #include statements. A slang file is preprocessed by scanning through the slang and resolving all #include statements. The include process does not consider any preprocessor defines or conditional expressions. The include path must always be relative, and it will be relative to the file path of the current file. Nested includes are allowed, but includes in a cycle are undefined as preprocessor guards are not considered.

E.g.:

#include "common.inc"

After includes have been resolved, the frontend scans through all lines of the shader and considers #pragma statements. These pragmas build up ancillary reflection information and otherwise meaningful metadata.

#pragma stage

This pragma controls which part of a .slang file are visible to certain shader stages. Currently, two variants of this pragma are supported:

• #pragma stage vertex
• #pragma stage fragment

If no #pragma stage has been encountered yet, lines of code in a shader belong to all shader stages. If a #pragma stage statement has been encountered, that stage is considered active, and the following lines of shader code will only be used when building source for that particular shader stage. A new #pragma stage can override which stage is active.

#pragma name

This pragma lets a shader set its identifier. This identifier can be used to create simple aliases for other passes.

E.g.:

#pragma name HorizontalPass

#pragma format

This pragma controls the format of the framebuffer which this shader will render to. The default render target format is R8G8B8A8_UNORM.

Supported render target formats are listed below. From a portability perspective, please be aware that GLES2 has abysmal render target format support, and GLES3/GL3 may have restricted floating point render target support.

If rendering to uint/int formats, make sure your fragment shader output target is uint/int.

8-bit

• R8_UNORM
• R8_UINT
• R8_SINT
• R8G8_UNORM
• R8G8_UINT
• R8G8_SINT
• R8G8B8A8_UNORM
• R8G8B8A8_UINT
• R8G8B8A8_SINT
• R8G8B8A8_SRGB

10-bit

• A2B10G10R10_UNORM_PACK32
• A2B10G10R10_UINT_PACK32

16-bit

• R16_UINT
• R16_SINT
• R16_SFLOAT
• R16G16_UINT
• R16G16_SINT
• R16G16_SFLOAT
• R16G16B16A16_UINT
• R16G16B16A16_SINT
• R16G16B16A16_SFLOAT

32-bit

• R32_UINT
• R32_SINT
• R32_SFLOAT
• R32G32_UINT
• R32G32_SINT
• R32G32_SFLOAT
• R32G32B32A32_UINT
• R32G32B32A32_SINT
• R32G32B32A32_SFLOAT

E.g.:

#pragma format R16_SFLOAT

Example slang shader

#version 450 // 450 or 310 es are recommended

layout(set = 0, binding = 0, std140) uniform UBO
{
   mat4 MVP;
   vec4 SourceSize;
   float ColorMod;
};

#pragma name StockShader
#pragma format R8G8B8A8_UNORM
#pragma parameter ColorMod "Color intensity" 1.0 0.1 2.0 0.1

#pragma stage vertex
layout(location = 0) in vec4 Position;
layout(location = 1) in vec2 TexCoord;
layout(location = 0) out vec2 vTexCoord;
void main()
{
   gl_Position = MVP * Position;
   vTexCoord = TexCoord;
}

#pragma stage fragment
layout(location = 0) in vec2 vTexCoord;
layout(location = 0) out vec4 FragColor;
layout(binding = 1) uniform sampler2D Source;
void main()
{
   FragColor = texture(Source, vTexCoord) * ColorMod;
}

Porting guide from legacy Cg spec