Files
RetroArch/tools/thread_read_audit.py
Davey HughesandLibretroAdmin 08e5b72a88 Scan from the command line: exit status, Manual Scan options, playlist refresh (#19977)
* file_path: stop at the end of the path after '~'

fill_pathname_expand_special() skipped two characters after a leading
'~' or ':', so a bare "~" read past its NUL and "~x" lost the x. It
now skips the prefix and a slash after it, if there is one.

* scan: hand the callback user data and what the scan did

task_push_dbscan() and task_push_manual_content_scan() take user data
for their callback, and the callback's task_data is now a
task_dbscan_result_t: how many entries the scan added, how many its
playlists already held, and how many files matched nothing. A file is
counted by what became of it, so an archive that is expanded is not
counted as unmatched, and a loose scan's unmatched files count as
added. Callers pass NULL for now.

* scan: say in the result when a scan failed

A scan that ends before it runs (bad setup, no valid content, a DAT
that will not load) or ends with a task error now sets the result's
failed flag. A database folder that is missing or unset is treated as
an empty one, so it reports the same "no database" error instead of
finishing as if it had scanned.

* scan: count the files a scan could not read

A file whose scan fails (an unreadable file, a failed lookup) was in
no count. The result now carries an errors count for them. They do
not fail the scan.

* scan: print --scan errors on stderr and totals on stdout

A command-line scan now ends with a totals line on stdout, or with the
reason it failed on stderr. A file it could not read is listed on
stderr. The ++/==/?? marks are coloured only when their stream is a
terminal, so pipes and redirected output get plain text; on Windows
redirected output keeps its marks instead of losing them to
WriteConsole.

* scan: the manual scan finder skips a finished task

A finished scan stays in the task queue while its callback runs, and
the finder that refuses a second scan for the same playlist still
matched it, so a scan pushed from that callback was refused.

* retroarch: run every --scan, then exit with what the scans did

Each --scan is collected while the command line is parsed and run in
turn once it is done, so later options and further --scan flags are
no longer dropped. A path is expanded and made absolute first; one
that does not exist, or a scan that cannot start, is reported on
stderr and the next path still runs. --scan with --cloudsync or
--command is refused.

Without --menu the exit status is 1 if any scan failed, else 0 if any
added an entry, else 2, and a content path given with the scans is
ignored, with a line on stderr. With --menu each finished scan starts
the next, as a second scan is refused while one runs.

* manual_content_scan: clear the database name a refresh did not save

A playlist with no saved database name left the name buffer
uninitialised, and the refresh compared it against every installed
database, so whatever was on the stack could pick a database the
playlist never named.

* scan: Skip Database References names the scan's one playlist

The option was meant to put the playlist's name in each entry's
db_name, even for a database match (#18641), for users who build
their own thumbnail sets. When the scan picks the playlists, db_name
also picks the playlist an entry goes to, so with the option on every
match went to one playlist named after the content folder.

It now applies only when the scan writes one playlist, and gives
that playlist's name, a loose scan's unmatched entries included. With
the target on <Automatic>, or Fully Automatic with the option left on
from a Custom scan, matches go to their databases' playlists as with
it off; the entries of a custom playlist look their thumbnails up
under its name, not the folder's.

* scan: name the scanned folder in the start line

The "Scanning content" line, in the log and in --scan's output, has
named the database folder since the manual scan and the database scan
were merged (2fdf736d51); before, it named the path being scanned. It
names the content folder again.

* manual_content_scan: reset, scan overrides and their apply

The command line's scan options need each scan to start from the
defaults and to lay their values over them, or over a playlist's saved
settings. Add a reset, a record of those options and a function that
applies it through the existing setters: a core named by path, file
name or display name, a database, a playlist, the database check or a
DAT, extensions and the switches, stopping with a reason at the first
that does not resolve.

A core path is normalised before it is compared. A core's file name
matches in the case given first, then ignoring case, on every
platform; a display name matches exactly. Given another database, a
playlist named after its database keeps that name as a custom one, so
a refresh still writes the same playlist.

The core setter's copy step becomes its own function, so a core found
by path or file name can be set. Nothing calls the new functions yet.

* scan: run --scan items through one chain

The --scan paths ran through a loop in retroarch.c, and with --menu
through a second chain of their own. The scan options and playlist
refreshes to come need one runner for every item, from the command
line and from the menu.

task_push_dbscan_chain() runs a list of items one task at a time: a
prep task lists the databases, then each step is pushed from the last
task's callback on the main thread, so nothing waits and nothing
locks. An item that fails is reported and the run goes on; a
cancelled task ends it. retroarch.c builds the items in the first
argument pass and hands them to the chain; without --menu it waits
for the chain's end and exits as before.

Running out of memory prints "out of memory" on stderr: for an item's
path the item fails, and for the chain itself nothing runs, which
without --menu exits 1.

The chain also prints a scan's override errors, and the warnings for
an option that needs one playlist or does nothing for a single file,
which the scan options of the next commit can trigger.

* retroarch: Manual Scan options for --scan

--scan-playlist, --scan-database, --scan-match, --scan-dat, --scan-core,
--scan-exts and the five switches set the Manual Scan settings of the
--scan before them; a --scan without any stays Fully Automatic.

Values are checked in the first pass, so an option before any --scan,
an empty or bad value, or one too long for its setting ends the run
with its reason on stderr and status 1, before the config loads and
before anything is scanned. Cores, databases and DAT files are
resolved when their scan runs, and one that does not resolve fails
only that scan.

* scan: refresh playlists in the chain

A refresh item rescans one playlist with its saved scan settings, as
the menu's Refresh Playlist does, with the item's scan options laid
over them and written back by the scan. A refresh-all item does the
same for every playlist in the playlist folder that has saved
settings, in the order Manage Playlists lists them, histories and
favorites left out.

A refresh item names its playlist by name or by a path in the playlist
folder. A bare "name.lpl" is the playlist folder's file, and an
absolute path is normalised before it is checked against the folder.
The name is looked up in the folder's listing on every platform, the
exact name first, then ignoring case, since macOS and FAT/exFAT cards
ignore case; the refresh then takes the file's own name, so a
database's playlist stays one. Two or more playlists that match only
in other cases fail the item.

The playlist folder walk and each playlist load run in prep tasks: on
the worker with a threaded queue, under the shared I/O window without
one, each pushed from the previous step's callback. A playlist whose
saved content folder is gone is skipped rather than failed. A listing
that runs out of memory fails the refresh-all; a playlist folder that
is missing or cannot be read lists nothing, and the refresh-all ends
with "no playlists with saved scan settings". A refresh's start line
names its content folder; its totals and failure lines name the
playlist.

* retroarch: --scan-refresh and --scan-refresh-all

--scan-refresh=NAME|PATH rescans a playlist with its saved scan
settings, as the menu's Refresh Playlist does, and --scan-refresh-all
rescans every playlist that has them. Both are items of the same run
as --scan: the scan options after one are laid over the saved
settings and written back, and the exit status follows the same rule.

--scan-playlist after a refresh is refused, so a refresh always
updates its own playlist, and an empty --scan-refresh is refused as
the other empty values are. Like --scan, the refresh options cannot be
combined with --cloudsync or --command; the error names whichever of
the three comes first on the command line.

* menu: Refresh All Playlists

Manage Playlists gets Refresh All Playlists as its first entry when it
lists any playlist, so an empty list still says there are no entries.
It rescans every playlist that has saved scan settings, with those
settings, through the same chain as --scan-refresh-all, and ends with
one notification saying how many playlists were refreshed, skipped
(content folder gone) and failed.

While a content scan runs it is refused with a new message; a refused
scan had none. Other languages come through Crowdin.

While a scan chain runs (Refresh All, or a command-line scan with
--menu), Start Scan, a playlist's Refresh, Scan Directory and Scan
File are refused with the same message, as they would run with the
settings the chain last loaded. task_push_dbscan() returns false then
too, which covers the companion and the iOS caller.

* docs: --scan, refresh and the Manual Scan options in the man page

--help lists the options but not what a script needs: the order the
scans run in, the exit status and what the options set. The man page
now has them, next to --cloudsync, and CHANGES.md has the entry.

* scan chain: writes from workers reach the playlist cache by note

A playlist written by a task - the chain's prep parse, a scan, a
playlist-manager task, whatever thread it runs on - leaves a note on a
lock-free stack instead of reading the cached playlist; the main
thread applies the notes when it next decides whether the cache may
stand in, so no worker reads playlist_cached while the main thread
frees or installs it. Without threads or pointer atomics a write
applies at once.

The chain's prep tasks set their handlers by name, and
thread_read_audit.py fails on a handler set from a
retro_task_handler_t parameter, which would hide every handler passed
in. The chain puts the Manual Scan settings it found back when it
ends, and is no longer running when its done callback runs, so that
callback may start another. task_push_dbscan() says why it refuses,
so the menu and the companions show the same reason, and the
companions' status line takes it from companion_core. --scan and
--scan-refresh set the verbosity override, so a config with logging
off cannot detach the console the totals go to, and refuse a path too
long for the chain's buffers. The Refresh All notification passes the
length written, and fill_pathname_expand_special() stops at the end of
a buffer shorter than the home or application directory.

Lanes: playlist_parity_test writes the cached file from a worker,
once at the same size and then while the main thread reuses and frees
the cache; database_scan_test checks the refusal message, the restored
settings and a done callback that starts the next chain;
fill_pathname_test expands into a buffer shorter than the home
directory.

* playlist: header-only parse for the scan chain's prep

The prep task needs a playlist's saved scan settings and nothing else,
yet parsed every entry and, through autofix, could write the file back
from a worker. playlist_parse_begin_header() stops at "items", rebases
the scan record in memory when the base content directory moved, and
marks the playlist so playlist_write_file() refuses it.

Lane: playlist_parity_test header_only (entries skipped, scan dir
rebased, file bytes untouched, old format).

* scan chain: ask the file system nothing on the main thread

The chain's steps ran on the main thread between its tasks and stat'd
and realpath'd there: a --scan path's resolve and stat, a
--scan-refresh path's two realpaths, the content folder's stat, the
saved and override DATs' stat and size, an override core path's
realpath.  On a network share each of those can stall a frame.

- manual_content_scan_fs_read() reads them on any thread, touching
  nothing but its arguments; the _fs variants of the content-folder,
  playlist and override setters take its answers instead of asking.
  The plain setters pass NULL and behave as before, so the menu is
  unchanged.
- A --scan path gets its own prep task (dbscan_path_handler); a
  --scan-refresh resolves its path inside the load task, and the load
  reads the folder, the saved DAT and the overrides' paths once the
  header is parsed.  The worker gets copies of the item's strings.
- The prep's state is freed in one place.

Lane: database_scan_test "fs off main": six items that each stop
before their scan, on a threaded queue, with path_stat, path_is_valid,
path_is_directory, path_get_size and path_resolve_realpath wrapped:
no call on the main thread.

Not moved: the scan task's own push still stats the content folder
and the DAT (manual_content_scan_get_task_config), shared with the
menu.

* samples: the scan chain's lanes hold on Windows and without realpath

From a native Windows run of both tests:

- database_scan_test "fs off main" refreshed f_gone.lpl through
  "pl/../pl/", which only a realpath folds; the sample builds without
  RARCH_INTERNAL, where path_resolve_realpath() folds nothing, so the
  item was rightly not in the playlist folder.  It names the playlist
  by its plain path now, which the check still realpaths on the
  worker.
- "another case, a database's playlist" read the system name type
  after the chain ended, and the chain puts the Manual Scan settings
  back then.  The item callback records each scan's type instead.
- playlist_parity_test's header lane expected "/moved/snes"; autofix
  converts the suffix to the local form, "/moved\snes" on Windows.
- The worker lane's stress read required a core name of every read.
  Windows replaces the file with MoveFileExW, and an open that meets
  the replace fails: that read is an empty playlist, which has none.

---------

Co-authored-by: LibretroAdmin <[email protected]>
2026-11-04 19:34:05 +01:00

637 lines
24 KiB
Python
Executable File

#!/usr/bin/env python3
"""Worker-thread singleton-read audit.
The frontend's state singletons (config_get_ptr(), runloop_state_get_ptr(),
video_state_get_ptr() and the rest) belong to the main thread; a worker
takes what it needs through published snapshots, latched fields or
arguments captured when the work was posted. This walks the shipping
binary's call graph from every worker entry point - threads the tree
creates, task handlers the threaded queue dispatches, and the audio
pipeline's callback - and reports every reachable function that calls
one of the singletons.
Owner-domain reads are the sanctioned exception: a subsystem's own
thread may read its own pointer-stable singleton (the video thread and
video_state_get_ptr()). Those pairs live in
tools/thread_read_allow.list as `entry|reader` lines.
A boundary function crosses to the main thread itself - it defers
off-main callers before any singleton is touched, the way
runloop_msg_queue_push() queues a worker's message for the main
thread's drain - and is declared as an `@boundary name` line. The
walk does not descend into a boundary, and the boundary's own reads
are not charged to its workers: the off-main prefix that makes this
true is what code review holds, so a boundary declaration is a claim
about that function, reviewed like the allowlist pairs.
Entries are found in the source (sthread_create/pthread_create argument
symbols, task->handler assignments, task_set_handler calls) so a new
thread or handler is audited the day it lands. A handler assigned in a
function that also sets RETRO_TASK_FLG_MAIN_THREAD on the task runs on
the main thread and is not an entry, unless its file ever clears that
flag or the same handler is also assigned without it; the call graph comes
from `objdump -d` of the binary, so what is audited is what ships in
that configuration. Functions the configuration compiles out are not
seen: run against the fullest builds available.
Usage:
thread_read_audit.py --binary retroarch [--root DIR] [--allow FILE]
thread_read_audit.py --selftest (needs gcc and objdump)
Exit status: 0 clean, 1 findings (or self-test failure).
"""
import argparse
import collections
import os
import re
import subprocess
import sys
import tempfile
READERS = (
"config_get_ptr",
"runloop_get_flags",
"runloop_state_get_ptr",
"video_state_get_ptr",
"video_driver_get_ptr",
"input_state_get_ptr",
"audio_state_get_ptr",
"menu_state_get_ptr",
"disp_get_ptr",
)
STHREAD_RE = re.compile(
r"\bsthread_create(?:_with_priority)?\s*\(\s*&?([a-z_][a-z_0-9]*)\s*,",
re.S)
PTHREAD_RE = re.compile(
r"\bpthread_create\s*\([^,;]+,[^,;]+,\s*&?([a-z_][a-z_0-9]*)\s*,",
re.S)
HANDLER_RE = re.compile(
r"(?:task->handler\s*=\s*|task_set_handler\s*\([^,]+,\s*)"
r"([a-z_][a-z_0-9]*)")
MAIN_SET_RE = re.compile(
r"flags\s*\|=[^;]*\bRETRO_TASK_FLG_MAIN_THREAD\b"
r"|task_set_flags\s*\([^;]*\bRETRO_TASK_FLG_MAIN_THREAD\b[^;]*,"
r"\s*true\s*\)")
MAIN_CLEAR_RE = re.compile(
r"&=\s*~[^;]*\bRETRO_TASK_FLG_MAIN_THREAD\b"
r"|task_set_flags\s*\([^;]*\bRETRO_TASK_FLG_MAIN_THREAD\b[^;]*,"
r"\s*false\s*\)")
SKIP_DIRS = {".git", "deps", "obj-unix", "pkg", "media", "samples"}
CALLBACK_ENTRIES = ("audio_driver_callback",)
# Driver frame-context functions: under the threaded wrapper these run
# on the video thread while the main thread runs free - the wrapper
# dispatches them through an indirect driver->frame edge the binary
# walk cannot follow, so they are listed as entries directly. Only the
# free-running functions belong here: blocking command handlers (init,
# set_shader, the pokes) execute while the main thread is parked in
# send-and-wait, where a settings read cannot race, and must NOT be
# added or every legal init-time read becomes a false finding. Symbols
# absent from a given build are skipped like any other entry.
#
# Part of the standing gate since the 29-finding backlog it opened
# with was driven to zero: latches and snapshots for the settings,
# rotation, input-sensor, spirv-toggle and HDR-recreate classes,
# and reasoned allowlist entries for the batch-confined and
# vtable-dispatch reads. --frame-context is accepted as a no-op for
# compatibility.
# Audio device callbacks: the function an audio backend calls on its
# own thread - SDL's device thread, libpulse's and PipeWire's loop, the
# CoreAudio render thread, JACK's process thread, the ASIO driver
# thread, the DSP and AX frame interrupts. The backend owns the thread,
# so no sthread_create in this tree names them and the binary walk has
# no edge to follow; they are listed here for the same reason the frame
# context functions are. A driver's init, open and free run on the main
# thread and are not entries - only what the device calls back into.
# Symbols absent from a given build are skipped like any other entry.
DEVICE_CALLBACK_ENTRIES = (
"sdl1_audio_playback_cb",
"sdl2_audio_playback_cb",
"sdl2_microphone_record_cb",
"sdl3_audio_stream_cb",
"coreaudio_audio_write_cb",
"coreaudio_mic_input_cb",
"ja_process_cb",
"ja_sample_rate_cb",
"ja_buffer_size_cb",
"ja_xrun_cb",
"ja_shutdown_cb",
"coreaudio_output_default_listener",
"coreaudio_output_default_listener_old",
"coreaudio_mic_default_listener",
"coreaudio_mic_default_listener_old",
"opensl_callback",
"aaudio_data_cb",
"aaudio_error_cb",
"ctr_dsp_audio_frame_cb",
"gx_audio_dma_callback",
"wiiu_ax_callback",
"audioworklet_process_cb",
"pulse_stream_state_cb",
"pulse_stream_request_cb",
"pulse_stream_latency_update_cb",
"pulse_underrun_update_cb",
"pulse_buffer_attr_cb",
"pwire_playback_process_cb",
"pwire_capture_process_cb",
"pwire_stream_state_changed_cb",
"pwire_mic_stream_state_changed_cb",
"pwire_stream_destroy_cb",
"asio_cb_buffer_switch",
"asio_cb_buffer_switch_time_info",
"asio_cb_sample_rate_changed",
"asio_cb_message",
"xa_voice_on_buffer_end",
)
INCLUDE_FRAME_CONTEXT = True
FRAME_CONTEXT_ENTRIES = (
"gl2_frame",
"gl2_set_texture_frame",
"gl3_frame",
"gl3_set_texture_frame",
"vulkan_frame",
"vulkan_set_texture_frame",
"d3d11_gfx_frame",
"d3d12_gfx_frame",
)
# The context driver callbacks the frame-context functions above call
# through their gfx_ctx_driver_t (ctx_driver->swap_buffers and the
# rest): another indirect edge the binary walk cannot follow, one level
# below the wrapper's. Whatever function a context driver's table puts
# in one of these slots runs inside the frame context, so it is an entry
# too. The slots are found by field name in gfx/video_driver.h and the
# functions in them from every gfx_ctx_driver_t table in the tree.
FRAME_CONTEXT_CTX_SLOTS = (
"update_window_title",
"set_resize",
"swap_buffers",
"bind_hw_render",
)
C_COMMENT_RE = re.compile(r"/\*.*?\*/|//[^\n]*", re.S)
CTX_TABLE_RE = re.compile(
r"\bgfx_ctx_driver_t\s+[A-Za-z_][A-Za-z_0-9]*\s*=\s*\{")
IDENT_RE = re.compile(r"^[A-Za-z_][A-Za-z_0-9]*$")
def first_arm(text):
"""Preprocessor lines dropped, keeping only each conditional's first
arm: a table's #ifdef X / #else arms fill the same slot, so one arm
is enough to number the slots."""
keep, stack = [], []
for line in text.split("\n"):
d = line.strip()
if d.startswith("#"):
word = d[1:].strip().split(None, 1)[0] if d[1:].strip() else ""
if word in ("if", "ifdef", "ifndef"):
stack.append(all(stack) if stack else True)
elif word in ("elif", "else") and stack:
stack[-1] = False
elif word == "endif" and stack:
stack.pop()
continue
if not stack or all(stack):
keep.append(line)
return "\n".join(keep)
def ctx_struct_fields(header_text):
"""gfx_ctx_driver_t's fields, in order."""
m = re.search(r"typedef\s+struct\s+gfx_ctx_driver\s*\{(.*?)\}\s*"
r"gfx_ctx_driver_t\s*;", C_COMMENT_RE.sub(" ", header_text),
re.S)
if not m:
return []
fields = []
for decl in first_arm(m.group(1)).split(";"):
decl = " ".join(decl.split())
if not decl:
continue
f = re.search(r"\(\s*\*\s*([A-Za-z_][A-Za-z_0-9]*)\s*\)", decl)
if not f:
f = re.search(r"([A-Za-z_][A-Za-z_0-9]*)\s*$", decl)
if f:
fields.append(f.group(1))
return fields
def ctx_table_slots(fields, source_text, slots):
"""The functions every gfx_ctx_driver_t table in source_text puts in
the named slots, positional or designated."""
found = set()
text = C_COMMENT_RE.sub(" ", source_text)
for m in CTX_TABLE_RE.finditer(text):
depth, i = 1, m.end()
while i < len(text) and depth:
depth += {"{": 1, "}": -1}.get(text[i], 0)
i += 1
body = first_arm(text[m.end():i - 1])
items, depth, cur = [], 0, []
for ch in body:
if ch in "({[":
depth += 1
elif ch in ")}]":
depth -= 1
if ch == "," and depth == 0:
items.append("".join(cur).strip())
cur = []
else:
cur.append(ch)
if "".join(cur).strip():
items.append("".join(cur).strip())
for pos, item in enumerate(items):
d = re.match(r"^\.([A-Za-z_][A-Za-z_0-9]*)\s*=\s*(.*)$", item, re.S)
if d:
field, value = d.group(1), d.group(2).strip()
elif pos < len(fields):
field, value = fields[pos], item
else:
continue
value = re.sub(r"^\(\s*[A-Za-z_][A-Za-z_0-9 *]*\)\s*", "", value)
if (field in slots and IDENT_RE.match(value)
and value not in ("NULL", "true", "false")):
found.add(value)
return found
def ctx_callback_entries(root):
try:
with open(os.path.join(root, "gfx", "video_driver.h"),
errors="replace") as f:
fields = ctx_struct_fields(f.read())
except OSError:
return set()
found = set()
for path in iter_sources(root):
try:
with open(path, errors="replace") as f:
text = f.read()
except OSError:
continue
if "gfx_ctx_driver_t" in text:
found |= ctx_table_slots(fields, text, FRAME_CONTEXT_CTX_SLOTS)
return found
def iter_sources(root):
for base, dirs, files in os.walk(root):
dirs[:] = [d for d in dirs if d not in SKIP_DIRS]
for f in files:
if f.endswith((".c", ".m", ".mm", ".cpp")):
yield os.path.join(base, f)
def enclosing_function(text, pos):
"""The body of the function containing pos: from the column-0 brace
that opens it to the column-0 brace that closes it."""
start = text.rfind("\n{", 0, pos)
end = text.find("\n}", pos)
return text[start if start >= 0 else 0:end if end >= 0 else len(text)]
def source_entries(root, main_thread=None, forwarded=None):
entries = {}
# handler name -> [path, every assignment flagged main-thread]
handlers = {}
for path in iter_sources(root):
rel = os.path.relpath(path, root)
try:
with open(path, errors="replace") as f:
text = f.read()
except OSError:
continue
for m in STHREAD_RE.finditer(text):
# A create-call forwarding a function-pointer parameter -
# sthread_create(thread_func, ...) inside rthreads.c's own
# sthread_create - names no function. The parameter is
# declared as (*name)( in the same file; a real entry never
# is.
if re.search(r"\(\s*\*\s*%s\s*\)\s*\(" % re.escape(m.group(1)),
text):
continue
entries.setdefault(m.group(1), rel)
for m in PTHREAD_RE.finditer(text):
if re.search(r"\(\s*\*\s*%s\s*\)\s*\(" % re.escape(m.group(1)),
text):
continue
entries.setdefault(m.group(1), rel)
clears = MAIN_CLEAR_RE.search(text) is not None
for m in HANDLER_RE.finditer(text):
# A handler set from a retro_task_handler_t parameter names
# no function, and the handlers its callers pass are never
# entries: the audit cannot see them, so it says so.
if re.search(r"\bretro_task_handler_t\s+%s\b"
% re.escape(m.group(1)), text):
if forwarded is not None:
forwarded.append((rel, m.group(1)))
continue
main = (not clears and MAIN_SET_RE.search(
enclosing_function(text, m.start())) is not None)
site = handlers.setdefault(m.group(1), [rel, True])
site[1] = site[1] and main
for name, (rel, main) in handlers.items():
if main:
if main_thread is not None:
main_thread[name] = rel
else:
entries.setdefault(name, rel)
for name in CALLBACK_ENTRIES:
entries.setdefault(name, "(pipeline callback)")
for name in DEVICE_CALLBACK_ENTRIES:
entries.setdefault(name, "(audio device callback)")
if INCLUDE_FRAME_CONTEXT:
for name in FRAME_CONTEXT_ENTRIES:
entries.setdefault(name, "(threaded frame context)")
for name in sorted(ctx_callback_entries(root)):
entries.setdefault(name, "(context callback in a frame)")
return entries
def call_graph(disasm_lines):
calls = collections.defaultdict(set)
defined = set()
cur = None
head = re.compile(r"^[0-9a-f]+ <([^>]+)>:")
site = re.compile(r"\bcall\s+[0-9a-f]+ <([^>+]+)(?:\+0x[0-9a-f]+)?>")
for line in disasm_lines:
m = head.match(line)
if m:
cur = m.group(1)
defined.add(cur)
continue
m = site.search(line)
if m and cur:
calls[cur].add(m.group(1))
return calls, defined
def audit(calls, defined, entries, allow, boundaries=frozenset()):
findings = []
audited = 0
for entry in sorted(entries):
if entry not in defined:
continue
audited += 1
seen = set()
stack = [entry]
while stack:
f = stack.pop()
if f in seen:
continue
seen.add(f)
if f in boundaries and f != entry:
continue
stack.extend(calls.get(f, ()))
for f in sorted(seen):
if f in boundaries and f != entry:
continue
hit = calls.get(f, set()) & set(READERS)
for r in sorted(hit):
# Two grains: (callee, reader) allowlists one verified
# function wherever it appears - the surgical form the
# list's entries use - and (entry, reader) allowlists a
# reader across a whole root, for roots that own their
# state wholesale.
if (f, r) in allow or (entry, r) in allow:
continue
findings.append((entry, f, r))
return findings, audited
def load_allow(path):
allow = set()
boundaries = set()
if not path or not os.path.exists(path):
return allow, boundaries
with open(path) as f:
for raw in f:
raw = raw.strip()
if not raw or raw.startswith("#"):
continue
if raw.startswith("@boundary"):
parts = raw.split()
if len(parts) == 2:
boundaries.add(parts[1])
continue
parts = raw.split("|")
if len(parts) == 2:
allow.add((parts[0], parts[1]))
return allow, boundaries
def run(binary, root, allow_path, list_unaudited=False,
objdump="objdump"):
main_thread = {}
forwarded = []
entries = source_entries(root, main_thread, forwarded)
try:
out = subprocess.run([objdump, "-d", binary],
capture_output=True, text=True, check=True)
except (OSError, subprocess.CalledProcessError) as e:
print("objdump failed: %s" % e, file=sys.stderr)
return 2
calls, defined = call_graph(out.stdout.splitlines())
allow, boundaries = load_allow(allow_path)
findings, audited = audit(calls, defined, entries, allow, boundaries)
if findings:
print("%d worker-thread singleton read(s). A worker takes its"
% len(findings))
print("state through published snapshots, latched fields or")
print("arguments captured when the work was posted; for a")
print("sanctioned owner-domain read add `entry|reader` to")
print("tools/thread_read_allow.list.")
for entry, fn, reader in findings:
print("%s -> %s calls %s" % (entry, fn, reader))
if forwarded:
print("%d task handler(s) set from a parameter, so the handlers"
% len(forwarded))
print("passed in are not audited. Assign each by name:")
for rel, name in forwarded:
print(" %s: task->handler = %s" % (rel, name))
if list_unaudited:
absent = sorted(e for e in entries if e not in defined)
print("%d source entr%s absent from this binary - compiled out"
% (len(absent), "y" if len(absent) == 1 else "ies"))
print("here, so nothing this run says covers them. Audit them")
print("from a build of their platform or configuration:")
for e in absent:
print(" %s (%s)" % (e, entries[e]))
print("thread read audit: %d entr%s in binary, %d finding(s), "
"%d allowlisted pair(s), %d boundar%s, "
"%d main-thread task handler(s) not entries"
% (audited, "y" if audited == 1 else "ies",
len(findings), len(allow),
len(boundaries), "y" if len(boundaries) == 1 else "ies",
len(main_thread)),
file=sys.stderr)
return 1 if findings or forwarded else 0
FIXTURE = """
#include <pthread.h>
#define RETRO_TASK_FLG_MAIN_THREAD (1 << 5)
typedef struct retro_task retro_task_t;
struct retro_task { void (*handler)(retro_task_t *task); unsigned flags; };
void *config_get_ptr(void) { static int s; return &s; }
static void leaf_reads(void) { config_get_ptr(); }
static void main_handler(retro_task_t *task) { (void)task; leaf_reads(); }
static void pool_handler(retro_task_t *task) { (void)task; leaf_reads(); }
static void push_main(retro_task_t *task)
{
task->handler = main_handler;
task->flags |= RETRO_TASK_FLG_MAIN_THREAD;
}
static void push_pool(retro_task_t *task)
{
task->handler = pool_handler;
}
static void *bad_worker(void *p) { leaf_reads(); return p; }
static void crossing(void) { leaf_reads(); }
static void *deferring_worker(void *p) { crossing(); return p; }
static void *good_worker(void *p) { return p; }
int main(void)
{
pthread_t a, b, c;
retro_task_t m = {0, 0}, w = {0, 0};
push_main(&m);
push_pool(&w);
m.handler(&m);
w.handler(&w);
pthread_create(&a, 0, bad_worker, 0);
pthread_create(&b, 0, good_worker, 0);
pthread_create(&c, 0, deferring_worker, 0);
pthread_join(a, 0); pthread_join(b, 0); pthread_join(c, 0);
return 0;
}
"""
def selftest():
with tempfile.TemporaryDirectory() as td:
src = os.path.join(td, "fix.c")
binp = os.path.join(td, "fix")
with open(src, "w") as f:
f.write(FIXTURE)
# A source entry with no definition in the binary: the shape
# --list-unaudited reports. Not compiled, only discovered.
with open(os.path.join(td, "ghost.c"), "w") as f:
f.write("void *ghost_worker(void *p);\n"
"void ghost_spawn(void *t)\n"
"{ pthread_create(t, 0, ghost_worker, 0); }\n")
# A main-thread task whose file clears the flag again: it may
# run on a worker, so its handler stays an entry.
with open(os.path.join(td, "unpin.c"), "w") as f:
f.write("static void flip_handler(retro_task_t *task);\n"
"static void push_flip(retro_task_t *task)\n"
"{\n"
" task->handler = flip_handler;\n"
" task->flags |= RETRO_TASK_FLG_MAIN_THREAD;\n"
"}\n"
"static void unpin(retro_task_t *task)\n"
"{\n"
" task_set_flags(task, RETRO_TASK_FLG_MAIN_THREAD,"
" false);\n"
"}\n")
# A helper that sets whatever handler it is given hides it.
with open(os.path.join(td, "fwd.c"), "w") as f:
f.write("static void push_any(retro_task_t *task,\n"
" retro_task_handler_t handler)\n"
"{\n"
" task->handler = handler;\n"
"}\n")
if subprocess.run(["gcc", "-O0", src, "-o", binp,
"-lpthread"]).returncode:
print("selftest: fixture build failed")
return 1
forwarded = []
entries = source_entries(td, None, forwarded)
if forwarded != [("fwd.c", "handler")] or "handler" in entries:
print("selftest: FAIL forwarded handler %r" % forwarded)
return 1
out = subprocess.run(["objdump", "-d", binp],
capture_output=True, text=True, check=True)
calls, defined = call_graph(out.stdout.splitlines())
findings, audited = audit(calls, defined, entries, set())
ok = (audited == 4
and any(e == "bad_worker" and r == "config_get_ptr"
for e, _, r in findings)
and any(e == "pool_handler" for e, _, _ in findings)
and "main_handler" not in entries
and "flip_handler" in entries
and any(e == "deferring_worker" for e, _, _ in findings)
and not any(e == "good_worker" for e, _, _ in findings)
and "ghost_worker" in entries
and "ghost_worker" not in defined)
if not ok:
print("selftest: FAIL entries=%d findings=%r"
% (audited, findings))
return 1
fields = ctx_struct_fields(
"typedef struct gfx_ctx_driver {\n"
" void* (*init)(void *video_driver);\n"
" update_window_title_cb update_window_title;\n"
" bool has_windowed;\n"
" void (*swap_buffers)(void*);\n"
" const char *ident;\n"
" void (*bind_hw_render)(void *data, bool enable);\n"
"} gfx_ctx_driver_t;\n")
slots = ctx_table_slots(fields,
"const gfx_ctx_driver_t a = {\n"
" a_init,\n"
"#ifdef HAVE_X\n"
" a_title, /* update_window_title */\n"
"#else\n"
" NULL,\n"
"#endif\n"
" true,\n"
" a_swap,\n"
" \"a\",\n"
" NULL\n"
"};\n"
"gfx_ctx_driver_t b = { .swap_buffers = b_swap, .init = b_init };\n",
FRAME_CONTEXT_CTX_SLOTS)
if slots != {"a_title", "a_swap", "b_swap"}:
print("selftest: FAIL context table slots %r" % sorted(slots))
return 1
allow = {("bad_worker", "config_get_ptr"),
("pool_handler", "config_get_ptr")}
findings, _ = audit(calls, defined, entries, allow,
{"crossing"})
if findings:
print("selftest: FAIL allowlist/boundary did not clear")
return 1
print("selftest: OK")
return 0
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--binary")
ap.add_argument("--root", default=".")
ap.add_argument("--allow", default="tools/thread_read_allow.list")
ap.add_argument("--objdump", default="objdump",
help="objdump to use; e.g. x86_64-w64-mingw32-objdump for a "
"PE binary from the mingw cross build")
ap.add_argument("--selftest", action="store_true")
ap.add_argument("--frame-context", action="store_true",
help="accepted for compatibility; frame-context entries are "
"part of the standing gate")
ap.add_argument("--list-unaudited", action="store_true",
help="also list source entry points absent from "
"this binary (compiled out on this build)")
args = ap.parse_args()
if args.selftest:
sys.exit(selftest())
if not args.binary:
ap.error("--binary is required (or --selftest)")
sys.exit(run(args.binary, args.root, args.allow,
args.list_unaudited, objdump=args.objdump))
if __name__ == "__main__":
main()