gfx/vulkan: fix three heap-corruption bugs in driver and common layer + CI

Three independent issues found during a focused audit of gfx/drivers/
vulkan.c and gfx/common/vulkan_common.c, all in the same memory-safety
class (writes past compile-time-sized arrays or undersized
allocations).  Submitting together because they share the same audit
context and none stands alone particularly well.  Adds AddressSanitizer
regression tests for each under samples/gfx/, wired into a new
.github/workflows/Linux-samples-gfx.yml CI job.

* vulkan_find_device_extensions: heap overflow
  (gfx/common/vulkan_common.c)

  The required-extension list is currently written twice -- once via
  memcpy at the top of the body, then again in a per-element loop
  that appears to be left over from an earlier refactor.  The
  instance-side counterpart vulkan_find_instance_extensions only
  logs in its loop, confirming the divergence.  The redundant write
  consumes more slots in the caller's `enabled` buffer than the slot
  accounting expects.

  Inside vulkan_context_create_device_wrapper the buffer is sized
  for (info.enabledExtensionCount + ARRAY_SIZE(required) +
  ARRAY_SIZE(optional)) entries, but the function actually writes
  (enabledExtensionCount + 2*required + optional) entries
  worst-case.  With current values (1 required, 3 optional) that is
  a one-element heap overflow at the end of the malloc'd block
  whenever a libretro core uses the Vulkan HW context-negotiation
  interface and the GPU supports at least one optional extension.
  The other call site in vulkan_context_init_device uses a stack
  buffer of size 8 and stays inside it today, but only by a margin
  of three.

  The duplicate writes also produce duplicate entries in
  ppEnabledExtensionNames passed to vkCreateDevice, which the spec
  forbids (VUID-VkDeviceCreateInfo-ppEnabledExtensionNames-01840);
  strict validation layers will have been complaining.

  Drop the per-element loop -- the early vulkan_find_extensions()
  check at the top of the function already validates that all
  required extensions are present, so the per-element re-check has
  never done any work.  Keep the debug log.

* vulkan_create_swapchain: OOB writes from unclamped image count
  (gfx/common/vulkan_common.c)

  vulkan_create_swapchain calls vkGetSwapchainImagesKHR twice --
  first to query the count, then to fill swapchain_images[] -- and
  never clamps the count between the two calls.  swapchain_images,
  swapchain_fences, the four swapchain_*_semaphores arrays,
  readback.staging[] and vk->swapchain[] are all sized at compile
  time to VULKAN_MAX_SWAPCHAIN_IMAGES (8).  If a driver returns
  more images than that the second vkGetSwapchainImagesKHR writes
  past swapchain_images[], and every loop bounded by
  num_swapchain_images (~12 sites across init/deinit/textures/
  buffers/descriptor pools/command buffers/readback and direct
  vk->swapchain[i] accesses) walks past its array.

  Two clamps.  First, cap desired_swapchain_images to
  VULKAN_MAX_SWAPCHAIN_IMAGES before vkCreateSwapchainKHR so we
  never ask a well-behaved driver for more images than we can
  hold.  Second, clamp num_swapchain_images between the two
  vkGetSwapchainImagesKHR calls to handle drivers that return
  more images than were requested -- the spec permits this.

* vulkan_create_texture: 32-bit overflow in staging-buffer sizing
  (gfx/drivers/vulkan.c)

  buffer_width = width * bpp and buffer_info.size = buffer_width *
  height are both unsigned * unsigned in 32-bit before being
  widened to VkDeviceSize on assignment.  With dimensions large
  enough to wrap (e.g. width=65536, height=16385, bpp=4 ->
  0x1_0004_0000 wraps to 0x40000), the staging buffer is allocated
  at the wrapped (small) size while the per-row upload memcpy loop
  later in the same function walks the unmodified width x height.
  The mapped region is smaller than what the loop writes, so the
  memcpy walks past the mapping into adjacent heap memory.

  Reachable from libretro cores supplying oversized
  retro_framebuffer dimensions and from vulkan_load_texture /
  vulkan_set_texture_frame, which take dimensions originating in
  image decoders.

  Compute the staging-buffer size in 64-bit, and widen the upload
  loop's stride and per-row copy size to size_t.  VkDeviceSize is
  already 64-bit so the assignment to buffer_info.size is now
  correct end-to-end.  The loop's pointer math was already
  implicitly widened on 64-bit hosts but is now unconditionally
  correct on 32-bit hosts (3DS, Vita, PSP, Wii, Wii U, older
  Android, 32-bit Windows) as well.

* samples/gfx/, .github/workflows/Linux-samples-gfx.yml

  Three regression tests, one per fix above, following the pattern
  established by the security regression tests under samples/tasks/
  (archive_name_safety_test, http_method_match_test,
  video_shader_wildcard_test, input_remap_bounds_test,
  bsv_replay_bounds_test, bps_patch_bounds_test).  Each test keeps
  a verbatim copy of the relevant post-fix predicate, demarcated by
  `=== verbatim copy ===` markers; an `IMPORTANT: ... must follow`
  comment documents the convention.  Plain C, asserts manually,
  exits nonzero on failure -- matches the harness the existing
  Linux-samples-{tasks,libretro-{common,db}}-samples.yml workflows
  understand.

  vulkan_extension_count_test (5 cases): create_device_wrapper-
  shaped caller, init_device-shaped caller, GPU supporting no
  optional extensions, GPU missing a required extension, and the
  prepushed-via-negotiation-interface case.  Verified pre/post-
  patch discrimination: under the pre-fix double-write shape the
  create_device_wrapper case ASan-aborts with heap-buffer-overflow.

  vulkan_swapchain_clamp_test (5 cases): well-behaved 3-image
  driver, at-capacity boundary, 9-image (MAX+1) driver,
  pathological 64-image driver, and the request-side cap across
  six input values.  Verified pre/post-patch discrimination: under
  the pre-fix no-clamp shape the 9-image case ASan-aborts on the
  second vkGetSwapchainImagesKHR fill.

  vulkan_texture_size_test (6 cases): typical 320x240, 4K RGBA,
  the smoking-gun 65536x16385x4 wrap case, an assorted-shapes
  table, the per-row upload-loop strides under an ASan-instrumented
  buffer, and a 64-bit-result sanity check.  Verified pre/post-
  patch discrimination: under the pre-fix 32-bit arithmetic the
  overflow cases produce arithmetic mismatches against the 64-bit
  reference.

  Linux-samples-gfx.yml is a near-verbatim copy of
  Linux-samples-tasks.yml, runs each test under
  -fsanitize=address with a per-step explanation of what the test
  is regression-testing and the `verbatim copy must follow`
  reminder.  ~3 seconds per test on Ubuntu 24 with system gcc.
This commit is contained in:
U-DESKTOP-SPFP6AQ\twistedtechre committed 2026-04-28 14:13:31 +02:00
1 parent 2d7b7edec6
commit da5e650ccf
9 files changed
+1378 -16

No files matched your search

+109
View File
@@ -0,0 +1,109 @@
name: CI Linux samples/gfx
on:
push:
branches:
- master
pull_request:
branches:
- master
workflow_dispatch:
permissions:
contents: read
env:
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
jobs:
samples-gfx:
name: Build and run samples/gfx
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Install dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y build-essential
- name: Checkout
uses: actions/checkout@v3
- name: Build and run vulkan_extension_count_test (ASan)
shell: bash
working-directory: samples/gfx/vulkan_extension_count
run: |
set -eu
# Regression test for the heap-overflow fix in
# gfx/common/vulkan_common.c::vulkan_find_device_extensions.
# Pre-fix the function appended the required-extension list
# twice -- once via memcpy at the top of the body, then
# again in a per-element loop -- consuming
# (count_initial + 2*num_required + num_optional) slots in
# the caller's buffer. vulkan_context_create_device_wrapper
# sized its malloc for (count_initial + num_required +
# num_optional) entries, so a libretro core using the Vulkan
# HW context-negotiation interface against a GPU exposing at
# least one optional extension hit a one-element heap-buffer-
# overflow (8 bytes on 64-bit) at the end of the malloc'd
# block. Build under AddressSanitizer so any reintroduction
# of the duplicate write is caught at the bounds level. If
# vulkan_common.c amends the append-to-enabled[] block, the
# verbatim copy in vulkan_extension_count_test.c must follow.
make clean all SANITIZER=address
test -x vulkan_extension_count_test
timeout 60 ./vulkan_extension_count_test
echo "[pass] vulkan_extension_count_test"
- name: Build and run vulkan_swapchain_clamp_test (ASan)
shell: bash
working-directory: samples/gfx/vulkan_swapchain_clamp
run: |
set -eu
# Regression test for the unclamped-swapchain-image-count
# fix in gfx/common/vulkan_common.c::vulkan_create_swapchain.
# Pre-fix the two vkGetSwapchainImagesKHR calls (count
# query + image fill) had no clamp between them, so a driver
# returning more than VULKAN_MAX_SWAPCHAIN_IMAGES (8) images
# on the second call wrote past context.swapchain_images[8]
# and every loop bounded by num_swapchain_images walked past
# its compile-time-sized companion array (~12 such loops
# across init/deinit/textures/buffers/descriptor pools/
# command buffers/readback and direct vk->swapchain[i]
# uses). Build under AddressSanitizer so any reintroduction
# of either the request-side or the post-create clamp is
# caught at the bounds level. If vulkan_common.c amends
# the cap or the post-create clamp, the verbatim copies in
# vulkan_swapchain_clamp_test.c must follow.
make clean all SANITIZER=address
test -x vulkan_swapchain_clamp_test
timeout 60 ./vulkan_swapchain_clamp_test
echo "[pass] vulkan_swapchain_clamp_test"
- name: Build and run vulkan_texture_size_test (ASan)
shell: bash
working-directory: samples/gfx/vulkan_texture_size
run: |
set -eu
# Regression test for the 32-bit-overflow fix in
# gfx/drivers/vulkan.c::vulkan_create_texture's staging-
# buffer sizing. Pre-fix the buffer_info.size calculation
# (buffer_width * height) was unsigned*unsigned in 32-bit
# before being widened to VkDeviceSize on assignment. With
# dimensions large enough to wrap (e.g. 65536x16385x4 ->
# 0x1_0004_0000 truncates to 0x40000), the staging buffer
# was malloc'd at the wrapped (small) size while the per-row
# upload memcpy loop walked the full width x height,
# writing past the mapped region into adjacent heap memory.
# Reachable from libretro cores supplying oversized
# retro_framebuffer dimensions and from vulkan_load_texture
# / vulkan_set_texture_frame. Build under AddressSanitizer
# so any reintroduction of the 32-bit arithmetic is caught
# at the bounds level. If vulkan.c amends the size
# calculation or upload-loop strides, the verbatim copies
# in vulkan_texture_size_test.c must follow.
make clean all SANITIZER=address
test -x vulkan_texture_size_test
timeout 60 ./vulkan_texture_size_test
echo "[pass] vulkan_texture_size_test"
+27 -13
View File
@@ -432,21 +432,12 @@ static bool vulkan_find_device_extensions(VkPhysicalDevice gpu,
goto end;
}
/* Required extensions: presence already validated by the
* vulkan_find_extensions() check above. Append in one shot. */
memcpy((void*)(enabled + count), exts, num_exts * sizeof(*exts));
count += num_exts;
for (i = 0; i < num_exts; i++)
{
if (vulkan_find_extensions(&exts[i], 1, properties, property_count))
{
RARCH_DBG("[Vulkan] Device extension supported: %s.\n", exts[i]);
enabled[count++] = exts[i];
}
else
{
RARCH_DBG("[Vulkan] Device extension NOT supported: %s.\n", exts[i]);
}
}
RARCH_DBG("[Vulkan] Device extension supported: %s.\n", exts[i]);
count += num_exts;
for (i = 0; i < num_optional_exts; i++)
{
@@ -2300,6 +2291,15 @@ bool vulkan_create_swapchain(gfx_ctx_vulkan_data_t *vk,
&& (desired_swapchain_images > surface_properties.maxImageCount))
desired_swapchain_images = surface_properties.maxImageCount;
/* Cap our request to what we can actually hold. Per-image arrays
* (swapchain_images, swapchain_fences, the various semaphore
* arrays, vk->swapchain[], readback.staging[]) are all sized to
* VULKAN_MAX_SWAPCHAIN_IMAGES at compile time. The post-create
* fill below also clamps defensively in case a driver returns
* more images than requested. */
if (desired_swapchain_images > VULKAN_MAX_SWAPCHAIN_IMAGES)
desired_swapchain_images = VULKAN_MAX_SWAPCHAIN_IMAGES;
if (surface_properties.supportedTransforms
& VK_SURFACE_TRANSFORM_IDENTITY_BIT_KHR)
pre_transform = VK_SURFACE_TRANSFORM_IDENTITY_BIT_KHR;
@@ -2415,6 +2415,20 @@ bool vulkan_create_swapchain(gfx_ctx_vulkan_data_t *vk,
vkGetSwapchainImagesKHR(vk->context.device, vk->swapchain,
&vk->context.num_swapchain_images, NULL);
/* Even after capping minImageCount above, drivers may legally
* return more images than requested. Clamp before the fill call
* so we don't write past swapchain_images[] and so every
* downstream loop bounded by num_swapchain_images stays inside
* its compile-time-sized array. */
if (vk->context.num_swapchain_images > VULKAN_MAX_SWAPCHAIN_IMAGES)
{
RARCH_WARN("[Vulkan] Swapchain returned %u images, clamping to %u.\n",
vk->context.num_swapchain_images,
(unsigned)VULKAN_MAX_SWAPCHAIN_IMAGES);
vk->context.num_swapchain_images = VULKAN_MAX_SWAPCHAIN_IMAGES;
}
vkGetSwapchainImagesKHR(vk->context.device, vk->swapchain,
&vk->context.num_swapchain_images, vk->context.swapchain_images);
+15 -3
View File
@@ -1017,6 +1017,7 @@ static struct vk_texture vulkan_create_texture(vk_t *vk,
enum vk_texture_type type)
{
unsigned i;
uint64_t buffer_size_64;
uint32_t buffer_width;
struct vk_texture tex;
VkImageCreateInfo info;
@@ -1051,11 +1052,18 @@ static struct vk_texture vulkan_create_texture(vk_t *vk,
/* Align stride to 4 bytes to make sure we can use compute shader uploads without too many problems. */
buffer_width = width * vulkan_format_to_bpp(format);
buffer_width = (buffer_width + 3u) & ~3u;
/* Compute the buffer size in 64-bit. width*bpp*height as a 32-bit
* unsigned would wrap for sufficiently large dimensions (e.g. an
* upscaled shader render target chain), leaving the staging buffer
* underallocated relative to the upload memcpy loop further down
* and producing a heap overflow on the host side. VkDeviceSize is
* 64-bit; widen the math to match. */
buffer_size_64 = (uint64_t)buffer_width * (uint64_t)height;
buffer_info.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
buffer_info.pNext = NULL;
buffer_info.flags = 0;
buffer_info.size = buffer_width * height;
buffer_info.size = buffer_size_64;
buffer_info.usage = 0;
buffer_info.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
buffer_info.queueFamilyIndexCount = 0;
@@ -1365,14 +1373,18 @@ static struct vk_texture vulkan_create_texture(vk_t *vk,
const uint8_t *src = NULL;
void *ptr = NULL;
unsigned bpp = vulkan_format_to_bpp(tex.format);
unsigned stride = tex.width * bpp;
/* Source stride and per-row copy size in size_t to keep
* the pointer math and memcpy length safe even when
* width*bpp would otherwise wrap a 32-bit unsigned. */
size_t stride = (size_t)tex.width * (size_t)bpp;
size_t row_bytes = (size_t)width * (size_t)bpp;
vkMapMemory(device, tex.memory, tex.offset, tex.size, 0, &ptr);
dst = (uint8_t*)ptr;
src = (const uint8_t*)initial;
for (y = 0; y < tex.height; y++, dst += tex.stride, src += stride)
memcpy(dst, src, width * bpp);
memcpy(dst, src, row_bytes);
if ( (tex.flags & VK_TEX_FLAG_NEED_MANUAL_CACHE_MANAGEMENT)
&& (tex.memory != VK_NULL_HANDLE))
@@ -0,0 +1,25 @@
TARGET := vulkan_extension_count_test
SOURCES := vulkan_extension_count_test.c
OBJS := $(SOURCES:.c=.o)
CFLAGS += -Wall -pedantic -std=gnu99 -g -O0
ifneq ($(SANITIZER),)
CFLAGS := -fsanitize=$(SANITIZER) -fno-omit-frame-pointer $(CFLAGS)
LDFLAGS := -fsanitize=$(SANITIZER) $(LDFLAGS)
endif
all: $(TARGET)
%.o: %.c
$(CC) -c -o $@ $< $(CFLAGS)
$(TARGET): $(OBJS)
$(CC) -o $@ $^ $(LDFLAGS)
clean:
rm -f $(TARGET) $(OBJS)
.PHONY: clean
@@ -0,0 +1,447 @@
/* Copyright (C) 2010-2026 The RetroArch team
*
* ---------------------------------------------------------------------------------------
* The following license statement only applies to this file (vulkan_extension_count_test.c).
* ---------------------------------------------------------------------------------------
*
* Permission is hereby granted, free of charge,
* to any person obtaining a copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
* INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
/* Regression test for the heap-overflow fix in
* gfx/common/vulkan_common.c::vulkan_find_device_extensions.
*
* Pre-fix the function appended the required-extension list to
* the caller's `enabled` buffer twice -- once via memcpy at the
* top of the body, then again in a per-element loop:
*
* memcpy((void*)(enabled + count), exts, num_exts * sizeof(*exts));
* count += num_exts;
* for (i = 0; i < num_exts; i++)
* if (vulkan_find_extensions(&exts[i], 1, properties, ...))
* enabled[count++] = exts[i]; // <-- duplicate
* for (i = 0; i < num_optional_exts; i++)
* if (vulkan_find_extensions(&optional_exts[i], 1, ...))
* enabled[count++] = optional_exts[i];
*
* Net writes worst-case: count_initial + 2*num_required +
* num_optional.
*
* The caller in vulkan_context_create_device_wrapper sized its
* malloc for count_initial + num_required + num_optional entries:
*
* const char **device_extensions = malloc(
* (info.enabledExtensionCount
* + ARRAY_SIZE(vulkan_device_extensions)
* + ARRAY_SIZE(vulkan_optional_device_extensions))
* * sizeof(const char *));
*
* With current ARRAY_SIZE values (1 required, 3 optional) and a
* GPU exposing all optional extensions, the second pass of writes
* placed one extra string-pointer past the end of the malloc'd
* block -- 8 bytes on a 64-bit host. Reachable whenever a
* libretro core uses the Vulkan HW context-negotiation interface
* (iface->create_device / create_device2), which is the standard
* path for cores using Vulkan HW rendering.
*
* The instance-side cousin vulkan_find_instance_extensions only
* logs in its loop -- the divergence indicates the device-side
* loop was a stale leftover from a refactor.
*
* The duplicate writes also produced duplicate entries in
* ppEnabledExtensionNames passed to vkCreateDevice, which the
* spec forbids (VUID-VkDeviceCreateInfo-
* ppEnabledExtensionNames-01840).
*
* Fix: drop the per-element loop entirely; rely on the early
* vulkan_find_extensions(exts, num_exts, ...) check at the top
* of the function, which already validates that all required
* extensions are present.
*
* IMPORTANT: this test keeps a verbatim copy of the post-fix
* append-to-enabled[] block from vulkan_common.c. If the
* function amends, the copy below must follow. Convention used
* by archive_name_safety_test, http_method_match_test,
* video_shader_wildcard_test, input_remap_bounds_test,
* bsv_replay_bounds_test, bps_patch_bounds_test in
* samples/tasks/.
*/
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
/* --- Mock the parts of the Vulkan extension-property API the
* function under test actually consults. In production
* vulkan_find_extensions walks an array of
* VkExtensionProperties returned by
* vkEnumerateDeviceExtensionProperties; for the test we feed
* a flat array of strings and report which of the queried
* names match. The test-side mock only needs to be
* consistent with how the post-fix code calls it. --- */
struct mock_ext_props {
const char **names;
unsigned count;
};
static bool mock_find_extensions(const char * const *exts, unsigned num_exts,
const struct mock_ext_props *props)
{
unsigned i, j;
for (i = 0; i < num_exts; i++)
{
bool found = false;
for (j = 0; j < props->count; j++)
{
if (strcmp(exts[i], props->names[j]) == 0)
{
found = true;
break;
}
}
if (!found)
return false;
}
return true;
}
/* === verbatim copy of the post-fix append block from
* gfx/common/vulkan_common.c::vulkan_find_device_extensions.
* The early "all required extensions present" check at the
* top of the function is folded into the caller's pre-flight
* here so the test exercises the same write pattern. If
* vulkan_common.c amends this block, the copy must follow. === */
static bool find_device_extensions_postfix(
const char **enabled, unsigned *inout_enabled_count,
const char **exts, unsigned num_exts,
const char **optional_exts, unsigned num_optional_exts,
const struct mock_ext_props *props)
{
unsigned i;
unsigned count = *inout_enabled_count;
if (!mock_find_extensions(exts, num_exts, props))
return false;
/* Required extensions: presence already validated by the
* vulkan_find_extensions() check above. Append in one shot. */
memcpy((void*)(enabled + count), exts, num_exts * sizeof(*exts));
for (i = 0; i < num_exts; i++)
(void)exts[i]; /* would be RARCH_DBG in production */
count += num_exts;
for (i = 0; i < num_optional_exts; i++)
{
if (mock_find_extensions(&optional_exts[i], 1, props))
enabled[count++] = optional_exts[i];
}
*inout_enabled_count = count;
return true;
}
/* === end verbatim copy === */
static int failures = 0;
/* Probe: caller sized for required + optional only; run the
* post-fix append against it. ASan instruments the malloc'd
* region exactly, so a regression to the pre-fix double-write
* trips heap-buffer-overflow. Caller layout matches
* vulkan_context_create_device_wrapper. */
static void test_create_device_wrapper_caller(void)
{
/* Simulates: 1 required, 3 optional, GPU exposes everything,
* info.enabledExtensionCount initially 0 (nothing pre-pushed
* by the negotiation interface). */
static const char *required[] = { "VK_KHR_swapchain" };
static const char *optional[] = {
"VK_KHR_sampler_mirror_clamp_to_edge",
"VK_EXT_full_screen_exclusive",
"VK_KHR_portability_subset"
};
static const char *gpu_exts[] = {
"VK_KHR_swapchain",
"VK_KHR_sampler_mirror_clamp_to_edge",
"VK_EXT_full_screen_exclusive",
"VK_KHR_portability_subset",
"VK_EXT_some_other_ext" /* GPU has extras; must not be picked up */
};
const unsigned num_req = sizeof(required) / sizeof(required[0]);
const unsigned num_opt = sizeof(optional) / sizeof(optional[0]);
const unsigned cap = 0 + num_req + num_opt; /* caller's malloc */
const char **enabled = (const char **)malloc(cap * sizeof(const char *));
unsigned count = 0;
struct mock_ext_props props = { gpu_exts, sizeof(gpu_exts)/sizeof(gpu_exts[0]) };
unsigned i, expected;
bool ok;
if (!enabled)
{
printf("[ERROR] malloc failed\n");
failures++;
return;
}
ok = find_device_extensions_postfix(enabled, &count,
required, num_req,
optional, num_opt,
&props);
if (!ok)
{
printf("[ERROR] post-fix predicate rejected an all-supported GPU\n");
failures++;
free((void*)enabled);
return;
}
/* Post-fix writes exactly num_req + num_opt slots when the GPU
* exposes everything. Pre-fix would have written 2*num_req +
* num_opt -- one past the malloc'd region. */
expected = num_req + num_opt;
if (count != expected)
{
printf("[ERROR] count = %u, expected %u (pre-fix would be %u)\n",
count, expected, 2 * num_req + num_opt);
failures++;
free((void*)enabled);
return;
}
/* No duplicates -- the spec forbids them in
* ppEnabledExtensionNames. Pre-fix had VK_KHR_swapchain
* appearing twice. */
for (i = 0; i < count; i++)
{
unsigned j;
for (j = i + 1; j < count; j++)
{
if (strcmp(enabled[i], enabled[j]) == 0)
{
printf("[ERROR] duplicate extension '%s' in enabled list\n",
enabled[i]);
failures++;
free((void*)enabled);
return;
}
}
}
printf("[SUCCESS] create_device_wrapper caller: %u entries, no overflow, no duplicates\n",
count);
free((void*)enabled);
}
/* Probe: stack-buffer caller (vulkan_context_init_device uses
* `const char *enabled_device_extensions[8]`). Capacity 8 vs.
* worst-case post-fix writes of num_req + num_opt = 4. This
* exercise documents the second call site -- the buffer fits
* comfortably today, but only by a margin of 4. */
static void test_init_device_caller(void)
{
static const char *required[] = { "VK_KHR_swapchain" };
static const char *optional[] = {
"VK_KHR_sampler_mirror_clamp_to_edge",
"VK_EXT_full_screen_exclusive",
"VK_KHR_portability_subset"
};
static const char *gpu_exts[] = {
"VK_KHR_swapchain",
"VK_KHR_sampler_mirror_clamp_to_edge",
"VK_EXT_full_screen_exclusive",
"VK_KHR_portability_subset"
};
const unsigned num_req = sizeof(required) / sizeof(required[0]);
const unsigned num_opt = sizeof(optional) / sizeof(optional[0]);
/* Mirror the production stack array literally. */
const char *enabled[8] = {0};
unsigned count = 0;
struct mock_ext_props props = { gpu_exts, sizeof(gpu_exts)/sizeof(gpu_exts[0]) };
bool ok;
ok = find_device_extensions_postfix(enabled, &count,
required, num_req,
optional, num_opt,
&props);
if (!ok || count != num_req + num_opt)
{
printf("[ERROR] init_device caller: ok=%d count=%u expected=%u\n",
(int)ok, count, num_req + num_opt);
failures++;
return;
}
if (count > sizeof(enabled) / sizeof(enabled[0]))
{
printf("[ERROR] init_device caller: count=%u > stack capacity 8\n",
count);
failures++;
return;
}
printf("[SUCCESS] init_device caller: %u entries fits in stack[8]\n", count);
}
/* Probe: GPU exposes none of the optional extensions. Post-fix
* writes exactly num_req slots; the caller's malloc still fits
* the optional set's worth of slack as required by the function
* signature, but only the required ones are populated. */
static void test_no_optional_supported(void)
{
static const char *required[] = { "VK_KHR_swapchain" };
static const char *optional[] = {
"VK_KHR_sampler_mirror_clamp_to_edge",
"VK_EXT_full_screen_exclusive",
"VK_KHR_portability_subset"
};
static const char *gpu_exts[] = {
"VK_KHR_swapchain" /* required only, no optional */
};
const unsigned num_req = 1;
const unsigned num_opt = 3;
const unsigned cap = num_req + num_opt;
const char **enabled = (const char **)malloc(cap * sizeof(const char *));
unsigned count = 0;
struct mock_ext_props props = { gpu_exts, 1 };
bool ok;
if (!enabled) { printf("[ERROR] malloc\n"); failures++; return; }
ok = find_device_extensions_postfix(enabled, &count,
required, num_req, optional, num_opt, &props);
if (!ok || count != num_req)
{
printf("[ERROR] no-optional case: ok=%d count=%u expected=%u\n",
(int)ok, count, num_req);
failures++;
}
else
printf("[SUCCESS] no-optional supported: %u entries (required only)\n",
count);
free((void*)enabled);
}
/* Probe: GPU is missing a required extension. Post-fix returns
* false up front, no writes happen. Caller gets to free
* everything cleanly. */
static void test_required_missing_rejects(void)
{
static const char *required[] = { "VK_KHR_swapchain" };
static const char *optional[] = {
"VK_KHR_sampler_mirror_clamp_to_edge"
};
static const char *gpu_exts[] = {
"VK_EXT_some_other_ext" /* no swapchain support */
};
const unsigned cap = 1 + 1;
const char **enabled = (const char **)malloc(cap * sizeof(const char *));
unsigned count = 0;
struct mock_ext_props props = { gpu_exts, 1 };
bool ok;
if (!enabled) { printf("[ERROR] malloc\n"); failures++; return; }
ok = find_device_extensions_postfix(enabled, &count,
required, 1, optional, 1, &props);
if (ok)
{
printf("[ERROR] missing required extension was accepted\n");
failures++;
}
else if (count != 0)
{
printf("[ERROR] missing-required path mutated count to %u\n", count);
failures++;
}
else
printf("[SUCCESS] missing required extension rejected, count untouched\n");
free((void*)enabled);
}
/* Probe: caller pre-pushes some entries via the negotiation
* interface (info.enabledExtensionCount > 0). Post-fix appends
* starting from `count`, doesn't disturb pre-pushed entries. */
static void test_with_prepushed_entries(void)
{
static const char *required[] = { "VK_KHR_swapchain" };
static const char *optional[] = {
"VK_KHR_sampler_mirror_clamp_to_edge",
"VK_EXT_full_screen_exclusive"
};
static const char *gpu_exts[] = {
"VK_KHR_swapchain",
"VK_KHR_sampler_mirror_clamp_to_edge",
"VK_EXT_full_screen_exclusive"
};
const unsigned num_req = 1;
const unsigned num_opt = 2;
const unsigned prepushed = 2;
const unsigned cap = prepushed + num_req + num_opt;
const char **enabled = (const char **)malloc(cap * sizeof(const char *));
unsigned count = prepushed;
struct mock_ext_props props = { gpu_exts, 3 };
bool ok;
if (!enabled) { printf("[ERROR] malloc\n"); failures++; return; }
/* Pre-push two entries the way create_device_wrapper does. */
enabled[0] = "VK_KHR_external_memory";
enabled[1] = "VK_KHR_external_semaphore";
ok = find_device_extensions_postfix(enabled, &count,
required, num_req, optional, num_opt, &props);
if (!ok || count != prepushed + num_req + num_opt)
{
printf("[ERROR] prepushed case: ok=%d count=%u expected=%u\n",
(int)ok, count, prepushed + num_req + num_opt);
failures++;
}
else if ( strcmp(enabled[0], "VK_KHR_external_memory") != 0
|| strcmp(enabled[1], "VK_KHR_external_semaphore") != 0)
{
printf("[ERROR] prepushed case: prepended entries clobbered\n");
failures++;
}
else
printf("[SUCCESS] prepushed entries preserved, %u entries total\n", count);
free((void*)enabled);
}
int main(void)
{
test_create_device_wrapper_caller();
test_init_device_caller();
test_no_optional_supported();
test_required_missing_rejects();
test_with_prepushed_entries();
if (failures)
{
printf("\n%d vulkan_extension_count test(s) failed\n", failures);
return 1;
}
printf("\nAll vulkan_extension_count regression tests passed.\n");
return 0;
}
@@ -0,0 +1,25 @@
TARGET := vulkan_swapchain_clamp_test
SOURCES := vulkan_swapchain_clamp_test.c
OBJS := $(SOURCES:.c=.o)
CFLAGS += -Wall -pedantic -std=gnu99 -g -O0
ifneq ($(SANITIZER),)
CFLAGS := -fsanitize=$(SANITIZER) -fno-omit-frame-pointer $(CFLAGS)
LDFLAGS := -fsanitize=$(SANITIZER) $(LDFLAGS)
endif
all: $(TARGET)
%.o: %.c
$(CC) -c -o $@ $< $(CFLAGS)
$(TARGET): $(OBJS)
$(CC) -o $@ $^ $(LDFLAGS)
clean:
rm -f $(TARGET) $(OBJS)
.PHONY: clean
@@ -0,0 +1,350 @@
/* Copyright (C) 2010-2026 The RetroArch team
*
* ---------------------------------------------------------------------------------------
* The following license statement only applies to this file (vulkan_swapchain_clamp_test.c).
* ---------------------------------------------------------------------------------------
*
* Permission is hereby granted, free of charge,
* to any person obtaining a copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
* INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
/* Regression test for the unclamped-swapchain-image-count fix in
* gfx/common/vulkan_common.c::vulkan_create_swapchain.
*
* The function calls vkGetSwapchainImagesKHR twice -- once to
* query the count, once to fill the swapchain_images[] array:
*
* vkGetSwapchainImagesKHR(device, swapchain, &count, NULL);
* vkGetSwapchainImagesKHR(device, swapchain, &count, images);
*
* Pre-fix, no clamp existed between the two calls. context.
* swapchain_images, swapchain_fences, the four
* swapchain_*_semaphores arrays, readback.staging[] and
* vk->swapchain[] are all sized at compile time to
* VULKAN_MAX_SWAPCHAIN_IMAGES (8). If a driver returned more
* images than that on the count query, the second call wrote
* past swapchain_images[8], and every loop bounded by
* num_swapchain_images (~12 sites across init/deinit/textures/
* buffers/descriptor pools/command buffers/readback and direct
* vk->swapchain[i] access) walked past its compile-time-sized
* array.
*
* The Vulkan spec permits the driver to return more images than
* minImageCount requested (drivers may use triple-buffer-plus
* strategies, the spec only requires the minimum); on fast
* MAILBOX drivers under high refresh rates this is observed in
* practice.
*
* Fix: cap desired_swapchain_images to VULKAN_MAX_SWAPCHAIN_IMAGES
* before vkCreateSwapchainKHR, so well-behaved drivers are never
* asked for more than we can hold; AND clamp the count between
* the two vkGetSwapchainImagesKHR calls, to handle drivers that
* return more images than requested.
*
* IMPORTANT: this test keeps a verbatim copy of both clamp
* predicates from vulkan_common.c. If vulkan_create_swapchain
* amends them, the copies below must follow. Convention used
* by the security regression tests in samples/tasks/.
*/
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
/* Match the production constant exactly. Defined in
* gfx/common/vulkan_common.h. If that header changes, this
* mirror must follow. */
#define VULKAN_MAX_SWAPCHAIN_IMAGES 8
/* Match the production layout of `struct vulkan_context` for the
* fields the post-fix predicate touches. We don't model the
* neighbouring fields literally -- ASan instruments the
* containing malloc, so an OOB write off swapchain_images[]
* trips heap-buffer-overflow regardless of layout. */
struct mock_vk_context {
/* Mock VkImage handles -- pointer-sized opaque. In production
* VkImage is a 64-bit dispatchable handle. */
void *swapchain_images[VULKAN_MAX_SWAPCHAIN_IMAGES];
uint32_t num_swapchain_images;
};
/* Mock driver: configurable count returned on first call.
* In production this is vkGetSwapchainImagesKHR. */
struct mock_driver {
uint32_t reported_image_count;
/* Magic sentinel pointer the driver "writes" to slot N to
* make OOB-write probing observable in non-ASan builds. */
void *fill_pattern;
};
static void mock_get_swapchain_images(struct mock_driver *drv,
uint32_t *count_io, void **out_images)
{
if (out_images == NULL)
{
/* Count-query call: report the driver's full image count. */
*count_io = drv->reported_image_count;
return;
}
/* Fill call: write min(driver_count, *count_io) entries.
* The Vulkan spec is that the count parameter is in/out --
* caller passes their array capacity, driver writes up to
* that many entries and updates *count to the number
* actually written. */
{
uint32_t to_write = drv->reported_image_count;
uint32_t i;
if (to_write > *count_io)
to_write = *count_io;
for (i = 0; i < to_write; i++)
out_images[i] = drv->fill_pattern;
*count_io = to_write;
}
}
/* === verbatim copy of the post-fix request-cap from
* gfx/common/vulkan_common.c::vulkan_create_swapchain (the
* block immediately after the maxImageCount clamp). If the
* production function amends this clamp, the copy must
* follow. === */
static uint32_t cap_desired_request(uint32_t desired_swapchain_images)
{
if (desired_swapchain_images > VULKAN_MAX_SWAPCHAIN_IMAGES)
desired_swapchain_images = VULKAN_MAX_SWAPCHAIN_IMAGES;
return desired_swapchain_images;
}
/* === end verbatim copy === */
/* === verbatim copy of the post-fix returned-count clamp from
* vulkan_create_swapchain, the block between the two
* vkGetSwapchainImagesKHR calls. If the production function
* amends this clamp, the copy must follow. === */
static void clamp_returned_count(struct mock_vk_context *ctx)
{
if (ctx->num_swapchain_images > VULKAN_MAX_SWAPCHAIN_IMAGES)
{
/* RARCH_WARN in production. */
ctx->num_swapchain_images = VULKAN_MAX_SWAPCHAIN_IMAGES;
}
}
/* === end verbatim copy === */
static int failures = 0;
/* Drives the post-fix two-call sequence end-to-end against a
* mock driver, returning the final num_swapchain_images and
* filling ctx->swapchain_images[]. Mirrors the production
* sequence exactly.
*
* Pre-fix, when reported_count > VULKAN_MAX_SWAPCHAIN_IMAGES the
* second mock call would write reported_count entries into the
* 8-slot array -- ASan trips heap-buffer-overflow. Post-fix,
* the clamp ensures the second call writes at most 8.
*
* The function operates on `ctx` allocated by the caller; the
* caller is malloc'd at exactly sizeof(struct mock_vk_context)
* so ASan can flag any write outside that struct's memory.
* (In production swapchain_images sits in a larger struct, so
* an OOB write would corrupt sibling fields; here it would
* corrupt heap metadata or whatever follows the malloc.) */
static void run_postfix_sequence(struct mock_vk_context *ctx,
struct mock_driver *drv)
{
/* First call: query count. ctx->num_swapchain_images becomes
* the driver-reported value, which may exceed our capacity. */
ctx->num_swapchain_images = 0; /* must initialise before use */
mock_get_swapchain_images(drv, &ctx->num_swapchain_images, NULL);
/* Post-fix clamp -- closes the OOB write on the next call. */
clamp_returned_count(ctx);
/* Second call: fill. The driver may legally write up to
* *count_io entries; the clamp above ensures count_io is
* at most VULKAN_MAX_SWAPCHAIN_IMAGES. */
mock_get_swapchain_images(drv, &ctx->num_swapchain_images,
ctx->swapchain_images);
}
/* Probe: well-behaved driver returns 3 images for a request of
* 3. Common path on Mesa with default settings. No clamp
* fires; final count is 3. */
static void test_well_behaved_three_images(void)
{
struct mock_vk_context *ctx = (struct mock_vk_context *)
calloc(1, sizeof(*ctx));
struct mock_driver drv = { 3, (void*)0xCAFE };
if (!ctx) { printf("[ERROR] calloc\n"); failures++; return; }
run_postfix_sequence(ctx, &drv);
if (ctx->num_swapchain_images != 3)
{
printf("[ERROR] well-behaved 3-image driver: got count=%u, expected 3\n",
ctx->num_swapchain_images);
failures++;
}
else if ( ctx->swapchain_images[0] != (void*)0xCAFE
|| ctx->swapchain_images[2] != (void*)0xCAFE)
{
printf("[ERROR] swapchain_images[] not populated correctly\n");
failures++;
}
else
printf("[SUCCESS] well-behaved 3-image driver: count=3, images filled\n");
free(ctx);
}
/* Probe: driver returns exactly VULKAN_MAX_SWAPCHAIN_IMAGES.
* Boundary case -- clamp must NOT fire. */
static void test_at_capacity_boundary(void)
{
struct mock_vk_context *ctx = (struct mock_vk_context *)
calloc(1, sizeof(*ctx));
struct mock_driver drv = { VULKAN_MAX_SWAPCHAIN_IMAGES, (void*)0xBEEF };
if (!ctx) { printf("[ERROR] calloc\n"); failures++; return; }
run_postfix_sequence(ctx, &drv);
if (ctx->num_swapchain_images != VULKAN_MAX_SWAPCHAIN_IMAGES)
{
printf("[ERROR] at-capacity boundary: count=%u, expected %u\n",
ctx->num_swapchain_images,
(unsigned)VULKAN_MAX_SWAPCHAIN_IMAGES);
failures++;
}
else
printf("[SUCCESS] at-capacity boundary: count=%u, no spurious clamp\n",
ctx->num_swapchain_images);
free(ctx);
}
/* Probe: driver returns 9 images (MAX + 1). Pre-fix, the
* second mock call would write 9 entries into a slot-8 array,
* trampling whatever sits past it. Post-fix, the clamp drops
* the count to 8 before the second call. ASan provides the
* actual bounds check; we additionally verify the final count
* truthfully reports 8. */
static void test_driver_returns_nine_images(void)
{
struct mock_vk_context *ctx = (struct mock_vk_context *)
calloc(1, sizeof(*ctx));
struct mock_driver drv = { 9, (void*)0xDEAD };
if (!ctx) { printf("[ERROR] calloc\n"); failures++; return; }
run_postfix_sequence(ctx, &drv);
if (ctx->num_swapchain_images != VULKAN_MAX_SWAPCHAIN_IMAGES)
{
printf("[ERROR] over-cap driver: count=%u, expected clamped to %u\n",
ctx->num_swapchain_images,
(unsigned)VULKAN_MAX_SWAPCHAIN_IMAGES);
failures++;
}
else
printf("[SUCCESS] driver returned 9 images, clamped to %u, no OOB write\n",
ctx->num_swapchain_images);
free(ctx);
}
/* Probe: pathological driver returns 64 images. Stress case
* for the post-fix clamp -- pre-fix would be a 56-slot OOB write
* (64 - 8) into adjacent heap memory. Post-fix, count becomes 8. */
static void test_driver_returns_many_images(void)
{
struct mock_vk_context *ctx = (struct mock_vk_context *)
calloc(1, sizeof(*ctx));
struct mock_driver drv = { 64, (void*)0xABCD };
if (!ctx) { printf("[ERROR] calloc\n"); failures++; return; }
run_postfix_sequence(ctx, &drv);
if (ctx->num_swapchain_images != VULKAN_MAX_SWAPCHAIN_IMAGES)
{
printf("[ERROR] 64-image driver: count=%u, expected clamped to %u\n",
ctx->num_swapchain_images,
(unsigned)VULKAN_MAX_SWAPCHAIN_IMAGES);
failures++;
}
else
printf("[SUCCESS] driver returned 64 images, clamped to %u, no large OOB\n",
ctx->num_swapchain_images);
free(ctx);
}
/* Probe: request-side cap. cap_desired_request takes a
* user-configured value (settings->uints.video_max_swapchain_
* images) and clips it to the array capacity. Verify the
* boundary behaviour. */
static void test_request_cap(void)
{
struct {
uint32_t input;
uint32_t expected;
const char *desc;
} cases[] = {
{ 1, 1, "input=1 (under cap)" },
{ 3, 3, "input=3 (typical)" },
{ 8, 8, "input=8 (at cap)" },
{ 9, 8, "input=9 (over cap)" },
{ 64, 8, "input=64 (way over)" },
{ (uint32_t)-1, 8, "input=UINT32_MAX (extreme)" }
};
const unsigned n_cases = sizeof(cases) / sizeof(cases[0]);
unsigned i;
bool any_fail = false;
for (i = 0; i < n_cases; i++)
{
uint32_t got = cap_desired_request(cases[i].input);
if (got != cases[i].expected)
{
printf("[ERROR] cap_desired_request(%s): got %u, expected %u\n",
cases[i].desc, got, cases[i].expected);
failures++;
any_fail = true;
}
}
if (!any_fail)
printf("[SUCCESS] cap_desired_request: all %u cases clamp correctly\n",
n_cases);
}
int main(void)
{
test_well_behaved_three_images();
test_at_capacity_boundary();
test_driver_returns_nine_images();
test_driver_returns_many_images();
test_request_cap();
if (failures)
{
printf("\n%d vulkan_swapchain_clamp test(s) failed\n", failures);
return 1;
}
printf("\nAll vulkan_swapchain_clamp regression tests passed.\n");
return 0;
}
+25
View File
@@ -0,0 +1,25 @@
TARGET := vulkan_texture_size_test
SOURCES := vulkan_texture_size_test.c
OBJS := $(SOURCES:.c=.o)
CFLAGS += -Wall -pedantic -std=gnu99 -g -O0
ifneq ($(SANITIZER),)
CFLAGS := -fsanitize=$(SANITIZER) -fno-omit-frame-pointer $(CFLAGS)
LDFLAGS := -fsanitize=$(SANITIZER) $(LDFLAGS)
endif
all: $(TARGET)
%.o: %.c
$(CC) -c -o $@ $< $(CFLAGS)
$(TARGET): $(OBJS)
$(CC) -o $@ $^ $(LDFLAGS)
clean:
rm -f $(TARGET) $(OBJS)
.PHONY: clean
@@ -0,0 +1,355 @@
/* Copyright (C) 2010-2026 The RetroArch team
*
* ---------------------------------------------------------------------------------------
* The following license statement only applies to this file (vulkan_texture_size_test.c).
* ---------------------------------------------------------------------------------------
*
* Permission is hereby granted, free of charge,
* to any person obtaining a copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
* INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
/* Regression test for the 32-bit-overflow fix in
* gfx/drivers/vulkan.c::vulkan_create_texture's staging-buffer
* sizing.
*
* Pre-fix the buffer-size calculation read:
*
* uint32_t buffer_width;
* buffer_width = width * vulkan_format_to_bpp(format);
* buffer_width = (buffer_width + 3u) & ~3u;
* buffer_info.size = buffer_width * height;
*
* Both multiplications are unsigned*unsigned in 32-bit, before
* the result is implicitly widened to VkDeviceSize (uint64_t)
* on the assignment to buffer_info.size. With dimensions large
* enough to wrap (e.g. width=65536, height=16385, bpp=4 ->
* 0x1_0004_0000 truncates to 0x40000), the staging buffer was
* allocated at the wrapped (small) size while the per-row
* upload memcpy loop later in the same function walked the
* full width x height. The mapped region was smaller than
* what the loop wrote, so the memcpy walked past the mapping
* into adjacent heap memory.
*
* Reachable from libretro cores supplying oversized
* retro_framebuffer dimensions (vulkan_get_current_sw_framebuffer
* passes through to vulkan_create_texture), and from
* vulkan_load_texture / vulkan_set_texture_frame, which take
* dimensions originating in image decoders. Particularly
* relevant on 32-bit platforms (3DS, Vita, PSP, Wii, Wii U,
* older Android, 32-bit Windows) where the implicit widening
* to size_t at the next stage doesn't help either.
*
* Fix: compute the staging-buffer size in 64-bit, and widen
* the upload loop's stride and per-row copy size to size_t.
*
* IMPORTANT: this test keeps verbatim copies of the post-fix
* size calculation and upload-loop arithmetic from
* gfx/drivers/vulkan.c. If vulkan_create_texture amends them,
* the copies below must follow. Convention used by the
* security regression tests in samples/tasks/.
*/
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
/* Minimal mirror of the bpp lookup from vulkan.c. Only the
* formats actually used by the production path matter. */
static unsigned vulkan_format_to_bpp_mock(unsigned format_id)
{
switch (format_id)
{
case 0: return 4; /* B8G8R8A8_UNORM */
case 1: return 2; /* R5G6B5_UNORM_PACK16 */
case 2: return 2; /* B4G4R4A4 etc. */
case 3: return 1; /* R8_UNORM */
default: return 0;
}
}
/* === verbatim copy of the post-fix size calculation from
* gfx/drivers/vulkan.c::vulkan_create_texture. If the
* production function amends this arithmetic, the copy
* must follow. === */
static uint64_t compute_buffer_size(unsigned width, unsigned height,
unsigned format_id)
{
uint64_t buffer_size_64;
uint32_t buffer_width;
buffer_width = width * vulkan_format_to_bpp_mock(format_id);
buffer_width = (buffer_width + 3u) & ~3u;
buffer_size_64 = (uint64_t)buffer_width * (uint64_t)height;
return buffer_size_64;
}
/* === end verbatim copy === */
/* === verbatim copy of the post-fix upload-loop size calculations
* from vulkan_create_texture, the STREAMED/STAGING case. The
* production loop is:
* size_t stride = (size_t)tex.width * (size_t)bpp;
* size_t row_bytes = (size_t)width * (size_t)bpp;
* for (y = 0; y < tex.height; ...; src += stride)
* memcpy(dst, src, row_bytes);
* If vulkan.c amends, this copy must follow. === */
static void compute_upload_strides(unsigned tex_width, unsigned width,
unsigned bpp, size_t *stride_out, size_t *row_bytes_out)
{
*stride_out = (size_t)tex_width * (size_t)bpp;
*row_bytes_out = (size_t)width * (size_t)bpp;
}
/* === end verbatim copy === */
static int failures = 0;
/* Probe: typical small dimensions, no risk of any overflow.
* Verifies the post-fix arithmetic agrees with the obvious
* calculation. */
static void test_typical_dimensions(void)
{
unsigned w = 320, h = 240, fmt = 0; /* RGBA8888 */
uint64_t got = compute_buffer_size(w, h, fmt);
uint64_t expected = (uint64_t)((w * 4 + 3) & ~3u) * (uint64_t)h;
if (got != expected)
{
printf("[ERROR] typical 320x240: got %llu, expected %llu\n",
(unsigned long long)got, (unsigned long long)expected);
failures++;
}
else
printf("[SUCCESS] typical 320x240: size=%llu bytes\n",
(unsigned long long)got);
}
/* Probe: 4096x2160 (4K) RGBA8888 -- common modern resolution.
* Result is 35 MiB-ish, well below 32-bit wrap, but exercises
* a realistically-large legitimate case to confirm the cast
* widening doesn't reject anything that should pass. */
static void test_4k_rgba(void)
{
unsigned w = 4096, h = 2160, fmt = 0;
uint64_t got = compute_buffer_size(w, h, fmt);
uint64_t expected = (uint64_t)((w * 4 + 3) & ~3u) * (uint64_t)h;
if (got != expected)
{
printf("[ERROR] 4K RGBA: got %llu, expected %llu\n",
(unsigned long long)got, (unsigned long long)expected);
failures++;
}
else
printf("[SUCCESS] 4K RGBA 4096x2160: size=%llu bytes (%.1f MiB)\n",
(unsigned long long)got, (double)got / (1024.0 * 1024.0));
}
/* Probe: dimensions that wrap a 32-bit unsigned multiplication.
* width=65536, height=16385, bpp=4: 65536*4 = 262144 (fine);
* 262144*16385 = 0x1_0004_0000 (33 bits). Pre-fix this
* truncated to 0x40000 (256 KiB). Post-fix it stays
* 0x100040000 = 4_295_229_440. This is the smoking-gun test:
* verify the result exceeds UINT32_MAX. */
static void test_overflow_wrap_case(void)
{
const unsigned w = 65536, h = 16385, fmt = 0;
const uint32_t prefix_buffer_width = (w * 4 + 3) & ~3u;
/* What pre-fix arithmetic would have produced (kept locally
* for the assertion only -- never used to size anything). */
const uint32_t prefix_size_truncated = prefix_buffer_width * h;
const uint64_t expected_full = (uint64_t)prefix_buffer_width * (uint64_t)h;
uint64_t got = compute_buffer_size(w, h, fmt);
if (got != expected_full)
{
printf("[ERROR] 65536x16385: got %llu, expected %llu\n",
(unsigned long long)got,
(unsigned long long)expected_full);
failures++;
return;
}
if (got <= UINT32_MAX)
{
printf("[ERROR] 65536x16385 should produce %llu (>UINT32_MAX), got %llu\n",
(unsigned long long)expected_full,
(unsigned long long)got);
failures++;
return;
}
printf("[SUCCESS] 65536x16385x4 = %llu bytes (pre-fix would truncate to %u)\n",
(unsigned long long)got, prefix_size_truncated);
}
/* Probe: multiple wrap-triggering shapes. Each pair has
* w*bpp*h > UINT32_MAX. */
static void test_assorted_overflow_shapes(void)
{
struct {
unsigned w, h, fmt;
const char *desc;
} cases[] = {
{ 65536, 16385, 0, "65536x16385 RGBA8888" },
{ 32768, 32769, 0, "32768x32769 RGBA8888" },
{ 65536, 65535, 1, "65536x65535 RGB565 (2bpp)" },
{ 65536, 65536, 0, "65536x65536 RGBA8888 (16 GiB)" }
};
const unsigned n = sizeof(cases) / sizeof(cases[0]);
unsigned i;
bool any_fail = false;
for (i = 0; i < n; i++)
{
const uint32_t bpp = vulkan_format_to_bpp_mock(cases[i].fmt);
const uint32_t bw = (cases[i].w * bpp + 3) & ~3u;
const uint64_t expected = (uint64_t)bw * (uint64_t)cases[i].h;
uint64_t got = compute_buffer_size(cases[i].w, cases[i].h, cases[i].fmt);
if (got != expected)
{
printf("[ERROR] %s: got %llu, expected %llu\n",
cases[i].desc,
(unsigned long long)got,
(unsigned long long)expected);
failures++;
any_fail = true;
}
}
if (!any_fail)
printf("[SUCCESS] all %u overflow shapes computed correctly in 64-bit\n",
n);
}
/* Probe: per-row upload arithmetic. Mimics the inner loop's
* memcpy length and source-stride calculation under an
* AddressSanitizer-instrumented allocation that's exactly the
* computed buffer size.
*
* If the post-fix `(size_t)w * (size_t)bpp` is reverted to
* `unsigned w * unsigned bpp`, the row_bytes value would wrap
* to a small number, the memcpy would copy too little, and the
* per-row pointer math (dst += stride) would also wrap. Test
* with a moderately large but not absurd shape to keep the
* malloc tractable. */
static void test_upload_loop_strides(void)
{
/* 2048x16 RGBA8888 -- 128 KiB total, easy to alloc.
* Each row is 8192 bytes, 16 rows. */
const unsigned w = 2048, h = 16;
const unsigned bpp = 4;
const size_t expected_stride = (size_t)w * (size_t)bpp;
const size_t total_size = expected_stride * (size_t)h;
size_t stride, row_bytes;
uint8_t *buf;
uint8_t *src;
unsigned y;
compute_upload_strides(w, w, bpp, &stride, &row_bytes);
if (stride != expected_stride || row_bytes != expected_stride)
{
printf("[ERROR] upload strides: stride=%zu row_bytes=%zu, expected %zu\n",
stride, row_bytes, expected_stride);
failures++;
return;
}
/* Allocate destination + source at the computed size. ASan
* instruments both; the per-row memcpy walks every byte in
* the destination and reads every byte in the source. */
buf = (uint8_t*)malloc(total_size);
src = (uint8_t*)malloc(total_size);
if (!buf || !src)
{
printf("[ERROR] malloc failed (size=%zu)\n", total_size);
free(buf); free(src);
failures++;
return;
}
memset(src, 0xA5, total_size);
memset(buf, 0, total_size);
/* Mirror the production upload loop. */
{
uint8_t *dst = buf;
const uint8_t *srcptr = src;
for (y = 0; y < h; y++, dst += stride, srcptr += stride)
memcpy(dst, srcptr, row_bytes);
}
/* Spot-check first and last byte and each row boundary --
* if the pointer math wrapped, these would be wrong. */
if ( buf[0] != 0xA5
|| buf[total_size - 1] != 0xA5
|| buf[(h - 1) * expected_stride] != 0xA5
|| buf[(h - 1) * expected_stride + 1] != 0xA5)
{
printf("[ERROR] upload loop produced inconsistent buffer contents\n");
failures++;
}
else
printf("[SUCCESS] upload loop: %u rows of %zu bytes, no OOB writes\n",
h, row_bytes);
free(buf);
free(src);
}
/* Probe: confirm the harness arithmetic itself isn't silently
* collapsing to 32-bit. 16 GiB result expected. */
static void test_size_holds_full_64bit(void)
{
uint64_t got = compute_buffer_size(65536, 65536, 0);
uint64_t expected = (uint64_t)((65536u * 4u + 3u) & ~3u) * 65536ull;
if (got != expected)
{
printf("[ERROR] 65536x65536 RGBA: got %llu, harness expected %llu\n",
(unsigned long long)got,
(unsigned long long)expected);
failures++;
return;
}
/* 65536 * 4 = 262144, * 65536 = 17_179_869_184 = 16 GiB. */
if (got != 17179869184ull)
{
printf("[ERROR] 65536x65536 RGBA: expected 16 GiB exactly, got %llu\n",
(unsigned long long)got);
failures++;
return;
}
printf("[SUCCESS] size holds full 64-bit: 65536x65536 = 16 GiB\n");
}
int main(void)
{
test_typical_dimensions();
test_4k_rgba();
test_overflow_wrap_case();
test_assorted_overflow_shapes();
test_upload_loop_strides();
test_size_holds_full_64bit();
if (failures)
{
printf("\n%d vulkan_texture_size test(s) failed\n", failures);
return 1;
}
printf("\nAll vulkan_texture_size regression tests passed.\n");
return 0;
}