P3: Vulkan: draw decals, and draw them like OpenGL does (#7879)

* Vulkan: bind all four model-matrix attribute locations for decals

The Vulkan shader input mask had one bit per listed attribute, so the
pipeline dropped the MATRIX4 attributes at locations 9-11 and the decal
vertex shader read three of its four instance-matrix columns from unbound
inputs. No decal was ever drawn in Vulkan.

The mask now covers four locations for VATTRIB_MODEL_MATRIX. It is built
by shader_vertex_input_mask() in shader_types so both Vulkan call sites
share it and a unit test covers it without a Vulkan build.

Fixes #7866

* Vulkan: build the decal pipeline with the material's shader flags

The decal pipeline config set the shader type but not the flags, so the
decal shader variant was always compiled without USE_NORMAL_MAP and fell
back to screen-space derivative normals for angle rejection. Far more of
the decal volume passed the angle test than in OpenGL, which lit the
decals brighter and wider. The pipeline now takes the flags the material
reports, as the other draw paths already do.

* Review fixes: exact input-mask test, decal flag polarity test, mask width check

Pin the full mask per shader type instead of the presence of the matrix
bits, so an expansion of any other attribute fails; pin that a decal
without a normal map is the one that reads the scene normal buffer; add a
compile-time bound on the mask width; trim two comments.

---------

Co-authored-by: Dustinspace217 <[email protected]>

code/graphics/shader_types.cpp

@@ -162,6 +162,19 @@ static ShaderVariantInfo SHADER_VARIANTS[] = {
 	{SDR_TYPE_SHADOW_MAP_GEN, true, SDR_FLAG_SHADOW_FALLBACK, "GEOMETRY_FALLBACK", {}, "Use a geometry shader to select the cascade layer instead of writing gl_Layer from the vertex shader"}
 };
 
+static_assert(VATTRIB_MODEL_MATRIX + 4 <= 32, "vertex input mask is 32 bits wide");
+
+uint32_t shader_vertex_input_mask(const ShaderTypeInfo& info)
+{
+	uint32_t mask = 0;
+	for (auto attr : info.attributes) {
+		// The Vulkan pipeline drops every location the mask omits, so a mat4 needs all four of its bits.
+		const uint32_t locations = (attr == VATTRIB_MODEL_MATRIX) ? 4u : 1u;
+		mask |= ((1u << locations) - 1u) << attr;
+	}
+	return mask;
+}
+
 const ShaderTypeInfo* shader_get_type_info(shader_type type)
 {
 	for (auto & i : SHADER_TYPES) {

code/graphics/shader_types.h

@@ -55,6 +55,14 @@ const ShaderTypeInfo* shader_get_type_info(shader_type type);
 // Lookup helper: find variant info by (type, flag). Returns nullptr if not found.
 const ShaderVariantInfo* shader_get_variant_info(shader_type type, int flag);
 
+/**
+ * @brief Bitmask of the vertex attribute locations a shader type reads.
+ *
+ * One bit per location; VATTRIB_MODEL_MATRIX contributes bits 8-11 because a
+ * mat4 input occupies four consecutive locations.
+ */
+uint32_t shader_vertex_input_mask(const ShaderTypeInfo& info);
+
 // Iterate all variants matching type whose flag bit is set in flags.
 void shader_for_each_active_variant(shader_type type, unsigned int flags, const std::function<void(const ShaderVariantInfo&)>& fn);
 

code/graphics/vulkan/VulkanDeferred.cpp

@@ -959,6 +959,7 @@ void vulkan_render_decals(decal_material* material_info,
 	// Build pipeline config for decal rendering
 	PipelineConfig config;
 	config.shaderType = SDR_TYPE_DECAL;
+	config.shaderFlags = material_info->get_shader_flags();
 	config.primitiveType = prim_type;
 	config.depthMode = material_info->get_depth_mode();
 	config.depthWriteEnabled = false;

code/graphics/vulkan/VulkanShader.cpp

@@ -127,10 +127,7 @@ void VulkanShaderManager::recompileAllShaders(const std::function<void(size_t, s
 						nprintf(("vulkan", "VulkanShaderManager: Failed to create vertex module: %s\n", e.what()));
 					}
 				}
-				shader.vertexInputMask = 0;
-				for (auto attr : typeInfo->attributes) {
-					shader.vertexInputMask |= (1u << attr);
-				}
+				shader.vertexInputMask = shader_vertex_input_mask(*typeInfo);
 
 				SCP_string fragFile = typeInfo->frag;
 				auto fragSpirv = m_compiler->compile(fragFile, vk::ShaderStageFlagBits::eFragment, type, flags, requiresRaytracing);
@@ -216,10 +213,7 @@ int VulkanShaderManager::loadShader(shader_type type, unsigned int flags)
 				vertFile.c_str(), e.what()));
 		}
 	}
-	shader.vertexInputMask = 0;
-	for (auto attr : typeInfo->attributes) {
-		shader.vertexInputMask |= (1u << attr);
-	}
+	shader.vertexInputMask = shader_vertex_input_mask(*typeInfo);
 
 	// Compile fragment shader
 	SCP_string fragFile = typeInfo->frag;