Every RHMAP_*_STR insert stores a private strdup() of the key. For
config_file that copy is pure overhead: the key handed to the map is
always the 'key' field of a struct config_entry_list that the config
file itself owns and that outlives the map at every site --
config_file_deinitialize tears the entry list down after the map,
config_unset removes the map slot before freeing the key, and both
config_file_add_child_list and config_append_file pilfer the donor's
entries before freeing the donor.
Add an opt-in RHMAP_BORROW_KEYS(map) that flips a per-map flag, so
inserts store the caller's pointer and teardown leaves it alone. The
flag lives in the header and is only read on insert, delete and free,
so the lookup path is untouched. Default behaviour is unchanged for
every existing user.
nested_list is the one other string-keyed user in tree and it was
keying off the caller's transient token rather than the copy it had
just made one line above. That is correct today only because the map
duplicates the key; make it key off its own copy so the call site is
right regardless, and bail cleanly if that copy failed to allocate
instead of hashing a NULL key.
Measured on a 1700-entry retroarch.cfg, load plus one
config_get_array() per key, 100 iterations, 5 runs each:
before 63.2 - 64.6 ms (mean 63.8)
after 52.8 - 54.9 ms (mean 54.0) -15.4%
cachegrind (--D1=49152,12,64 --LL=1048576,16,64):
D1 misses 124,714 -> 82,556 -34%
D refs 5.62M -> 4.53M -19%
Ir 14.67M -> 11.8M -20%
Allocation count for the same workload drops from 13,671 to 10,271,
exactly two per key. Identical lookup hit counts before and after;
valgrind reports no leaks and no errors.
Before this change 57% of D1 read misses on a config load were inside
malloc/free rather than in config code at all.
The RBUF (stretchy buffer) macros had two latent OOM crash
paths and one buffer-overflow path, affecting every RBUF user
in the tree (24 RBUF_PUSH call sites, 20+ RBUF_RESIZE sites
across gfx, menu, input/bsv, network, playlist, tasks,
core_updater_list, m3u_file, nested_list).
=== The three bugs ===
Pre-patch macros:
#define RBUF_FIT(b, n) \
((size_t)(n) <= RBUF_CAP(b) ? 0 \
: (*(void**)(&(b)) = rbuf__grow((b), (n), sizeof(*(b)))))
#define RBUF_PUSH(b, val) \
(RBUF_FIT((b), 1 + RBUF_LEN(b)), \
(b)[RBUF__HDR(b)->len++] = (val))
#define RBUF_RESIZE(b, sz) \
(RBUF_FIT((b), (sz)), \
((b) ? RBUF__HDR(b)->len = (sz) : 0))
And rbuf__grow returns:
* NULL on first-time (buf==NULL) malloc failure
* buf unchanged on realloc failure for non-NULL buf
BUG 1: first-time PUSH to NULL buf, malloc fails
int *buf = NULL;
/* malloc fails */
RBUF_PUSH(buf, 42);
/* expands to: rbuf__grow(NULL, 1, sizeof(int)) returns NULL
* -> b becomes NULL
* -> (b)[RBUF__HDR(b)->len++] = val
* = NULL[((rbuf__hdr*)NULL)-1].len++ = ...
* NULL-deref */
BUG 2: subsequent PUSH past cap, realloc fails
/* buf has len=cap=16, realloc to 32 fails */
RBUF_PUSH(buf, 42);
/* rbuf__grow returns original buf unchanged
* -> b stays same, cap still 16, len still 16
* -> (b)[16] = val <-- writes past end of allocation
* -> len++ <-- len now 17, exceeds cap=16
*
* Subsequent PUSHes also write past end, each further
* bumping len. Iteration over 0..LEN(b) reads past the
* allocation. Memory-corruption territory. */
BUG 3: RESIZE to grow, realloc fails
/* buf has len=2, cap=16 */
RBUF_RESIZE(buf, 100);
/* rbuf__grow returns unchanged buf, cap still 16
* -> RBUF__HDR(b)->len = 100 <-- len now 100, cap=16
*
* Caller iterates 0..100 reading past the 16-elem
* allocation. */
=== The fix ===
RBUF_FIT and rbuf__grow are left untouched - their contracts
are fine. The fix is in RBUF_PUSH and RBUF_RESIZE: after
RBUF_FIT, re-check 'RBUF_CAP(b) >= needed'. If that's false,
the grow failed; skip the write / len bump entirely.
New RBUF_PUSH (do-while idiom, statement-only):
#define RBUF_PUSH(b, val) \
do { \
RBUF_FIT((b), 1 + RBUF_LEN(b)); \
if (RBUF_CAP(b) >= 1 + RBUF_LEN(b)) \
(b)[RBUF__HDR(b)->len++] = (val); \
} while (0)
New RBUF_RESIZE (still an expression):
#define RBUF_RESIZE(b, sz) \
(RBUF_FIT((b), (sz)), \
((b) && RBUF_CAP(b) >= (size_t)(sz) \
? RBUF__HDR(b)->len = (sz) : 0))
Walkthrough for each pre-patch bug:
* First-time PUSH + malloc fail: RBUF_FIT makes b=NULL.
RBUF_CAP(NULL)=0. 0 >= 1 is false; no write. b stays
NULL, len=0. Silent no-op, no crash.
* PUSH at cap + realloc fail: RBUF_FIT leaves b unchanged,
cap unchanged (say 16, and LEN=16). 16 >= 17 is false;
no write, no len bump. Data intact, silent no-op.
* RESIZE to grow + realloc fail: RBUF_FIT leaves cap
unchanged. cap(16) >= sz(100) false; len not assigned.
buf valid at old len/cap. Silent no-op.
=== Statement-form do-while(0) compatibility ===
Changing RBUF_PUSH from expression to do-while(0) makes it
statement-only. Audited all 24 call sites in the tree: every
one uses RBUF_PUSH as a trailing-semicolon statement. None
uses it as an expression (no 'x = RBUF_PUSH(...)').
The do-while(0) idiom is the canonical C statement-macro form
precisely so it parses correctly as the body of an
unbraced if/else/while/for. The five call sites in the tree
that do appear in unbraced if/else bodies (gfx_animation.c:850,
852; menu_explore.c:395; menu_explore.c:643;
network/drivers_wifi/connmanctl.c:120) compile correctly
because 'do {} while(0);' is syntactically one statement, and
the trailing semicolon from the call site terminates the
do-while naturally.
RBUF_RESIZE remains an expression (conditional operator form)
to preserve any current expression-context use, though a
tree-wide grep shows all existing call sites are also
statement-context.
=== Verification ===
Unit-tested with a malloc/realloc wrapper that can be toggled
to fail on demand:
* Test A (first-push OOM): buf stays NULL, len=0 — pre-patch
segfaulted, post-patch clean.
* Test B (push at cap, OOM): len=16 cap=16 unchanged, data
intact — pre-patch wrote past allocation.
* Test C (RESIZE grow, OOM): len and cap unchanged at old
values — pre-patch set len=100 with cap=16.
Also compile-tested all direct consumers:
* gfx/gfx_animation.c
* input/bsv/uint32s_index.c
* menu/menu_explore.c
* network/drivers_wifi/connmanctl.c, nmcli.c
* core_updater_list.c
* libretro-common/formats/m3u/m3u_file.c
* libretro-common/lists/nested_list.c
* playlist.c
* tasks/task_content.c
All compile clean with no RBUF-related warnings or errors.
=== Related prior fix ===
Commit 1c7e114 (menu + file_list ...) documents this exact
issue in its commit message under 'Out of scope' and applies
a local workaround in menu_explore.c's ex_arena_grow by
refactoring to avoid the vulnerable RBUF_PUSH-on-failure path.
This patch is the tree-wide fix that makes that workaround
unnecessary, though we leave the menu_explore.c defensive
code in place as it is still correct and has additional
semantics (NULL-propagation through ex_arena_alloc).
=== Thread-safety ===
RBUF operations have always been single-owner (the caller is
responsible for its own serialisation). This patch doesn't
change that. The new do-while form is still single-expression
from the concurrency viewpoint.
=== Reachability ===
OOM during RBUF_PUSH / RBUF_RESIZE is reachable from user-
driven paths:
* playlist growth (every content addition, scan)
* menu explore indexing (every Explore menu open/rescan)
* gfx animation queueing (every animated UI transition)
* core_updater_list population (every online core update fetch)
* wifi scan results (every scan)
* input/bsv uint32 indexing (movie record/replay)
All of these are user-reachable and have been silently fragile
against OOM; this patch closes the gap at the foundation.
Add attribute to avoid the warnings "rbuf__grow' defined but not used [-Wunused-function]" (GCC/Clang) and "unreferenced local function has been removed" (MSVC).
Instead of initial 12MB memory allocation, use a dynamically growing array.
This shares the container structure (now called rbuf) with the explore menu.
`menu_animation_update` enumerates `menu_animation.list` to process each
`tween`. It was observed that some tweens execute a callback that
pushes more animations via `menu_animation_push`. During the push, if
the tween `list` does not have enough space, a `realloc` occurs,
potentially invalidating the existing list. The remaining pointer access
in menu_animation_update is therefore invalid. Best case is the memory
is unused and thus does not affect the program. Worst case is memory
corruption.