mirror of
https://github.com/libretro/RetroArch.git
synced 2026-11-22 10:50:15 -10:00
* netplay: bound the wait for tunnel information Stage 1 of making netplay host init non-blocking: the hang becomes an error, with no new states and no restructuring. Hosting through a relay waits for the lobby server to answer with a tunnel address. The wait was already narrowed to that one query, but it was still unbounded - and it is a round trip to a machine we do not control. A lobby server that is down, blackholed or merely slow left the frontend hung with no way out but killing it. It now gives up after ten seconds and reports failure like any other query, so the caller falls back to direct mode with the warning it already has. Generous on purpose: timing out a query that would have succeeded costs the user their session, while waiting a few seconds longer costs only patience. task_queue_wait_timeout() is the bound. It wraps the caller's condition in one that also goes false at a deadline and hands that to the existing wait, so every queue implementation - regular, threaded and GCD - keeps its own waiting behaviour and no platform-specific code is touched. Adding a variant to the implementation vtable would have meant editing the GCD path blind, which is the shape of a recent break and not worth repeating for this. A timeout alone would have been a bug, not a fix. The query is still in flight when we give up, and its callback writes host_room->mitm_address and mitm_port unconditionally - so it would land after the fallback and publish a tunnel address for a session running direct. Each query now carries a generation token and the callback ignores results that are no longer current; abandoning a query bumps the generation. That also closes the pre-existing case of two queries overlapping. The oracle covers the primitive rather than netplay's HTTP, because that is where the risk is: the netplay side is a few lines routing into an else branch that already existed. Five lanes over both queue implementations assert that a satisfiable wait returns as soon as it is satisfied - not after sitting out the timeout, which would pass a naive check - and that an unsatisfiable one gives up near the deadline rather than early or never. Removing the deadline hangs the lane until CI kills it, which is the behaviour being removed. Verified: builds with networking ENABLED and DISABLED, the latter being the configuration class that broke recently; task_queue.c links for ARM32 without NEON; all nine task oracles still pass, since that file is shared; the new lanes pass plain, under ASan+UBSan+LSan and twice under TSan. Not verified: behaviour against a real unresponsive lobby server. * netplay: split the tunnel query into begin / await / result Stage 2 of making netplay host init non-blocking: a refactor that changes no behaviour, so it can be reviewed on its own before anything moves. netplay_mitm_query() welded "ask the lobby server" to "wait for the answer". Those are now netplay_mitm_query_begin(), netplay_mitm_query_ready(), netplay_mitm_query_await() and netplay_mitm_query_result(), with netplay_mitm_query() calling them back to back at the same call site. Nothing is deferred and nothing moves; the point is that the next step - issuing the query when the user opens the host UI, so the answer is already there when host setup runs - becomes a change of WHERE begin() is called rather than a restructuring of host init. Every path was walked against the original: handle NULL/empty false begin() false custom, decode fails false begin() false custom, decode ok address set, result begin(), await(), result() http push fails pending cleared,false begin() false http ok, reply in time wait, then result await() waits, result() http ok, timeout warn, bump gen, false await() identical One deliberate difference, because the split exposed an edge the old shape hid. The custom-relay path answers synchronously and the old code returned without ever consulting the pending flag - so an earlier query left outstanding could not make it wait. Routed through await() it would have, so the custom path now supersedes any query in flight: it bumps the generation and clears the flag. That also stops a late callback from overwriting the address it just set, which the old code would have allowed. Verified: builds with networking enabled AND disabled; the wait_timeout CI step replayed verbatim; the netplay NAT, task ordering and watchdog oracles still pass. Not verified: behaviour against a real lobby server. * netplay: start the tunnel query when the user commits to hosting Stage 3, and the one that removes the stall the user actually feels. Host setup needs the relay address, and the query for it was issued at the moment of need - so the wait was the full round trip, every time. It is now issued at CMD_EVENT_NETPLAY_ENABLE_HOST, where the user commits to hosting, which is well before the address is wanted: on the content-reload path, before content has even started loading. The round trip overlaps work that was going to happen anyway, so by the time host setup asks, the answer is usually already there and the wait costs nothing. Nothing is deferred and host init is not restructured. The wait stays exactly where it was, as the backstop for the case where the answer has not arrived yet - which is what makes this cheap to reason about and cheap to revert. Prefetching only matters if asking twice is avoided, so the query records the handle it was issued for, and host setup reuses an answer that is already in hand or on its way. It deliberately does NOT reuse a prefetch that came back empty: a transient lobby failure at prefetch time would otherwise doom a session that asking again would have got, and one extra request is the better trade. Every state was walked: no prefetch query issued as before prefetch in flight, same reused, one query, overlapped prefetch answered reused, no wait at all prefetch came back empty re-issued relay changed since re-issued, picks the new server custom relay answered synchronously, reused The prefetch is best effort in the strictest sense: skipped when the relay is off or unconfigured, and if it fails for any reason host setup asks exactly as it did before. It is not speculative - the user has already chosen to host, so no request is made for anyone who merely opens a menu. Verified: the call site sits inside #ifdef HAVE_NETWORKING, computed from the preprocessor nesting rather than eyeballed, and both builds were made - networking enabled AND disabled, the configuration class that broke recently. The wait_timeout CI step was replayed verbatim from the workflow, and the netplay NAT and task ordering oracles pass. Not verified: behaviour against a real lobby server, including whether the overlap is long enough in practice to hide the round trip on the content-reload path. * content close: pin the save wait, and say what it is waiting for Stages 0 and 1 of unblocking content close. Nothing about the wait changes; this establishes what it guarantees and stops it looking like a hang. Stage 0, the oracle lane. Closing content waits for in-flight save and load tasks because their handlers call into core function pointers - retro_serialize via content_get_serialized_data, retro_unserialize via core_unserialize - while runloop_event_deinit_core() unloads the dylib underneath them. A worker mid-call when the library goes away dispatches into freed code. That is why this wait, unlike the others removed in this series, cannot simply be deleted. The lane pins the property the unload depends on: the file is checked incomplete before content_wait_for_save_state_task() and complete after, so a wait that returned early - on a budget boundary, say - fails here rather than becoming a crash on a device. Neutering the wait fails it, as it should. Stage 1, legibility. The wait cannot be shortened without deferring the teardown itself, which is later work, but an unexplained frozen frame and a frame that says "saving state" are very different experiences - and on slow storage this is seconds, not milliseconds. The message is pushed and one frame forced out before blocking, since nothing draws once the wait starts. It fires only when there is something to wait for, using the same condition the wait itself uses, so an ordinary close with no save in flight is untouched and shows nothing. That predicate had to become public to be asked from here; its load counterpart already was. Verified: builds with networking enabled and disabled; the save_state oracle passes, including the new lane, and fails when the wait is neutered. Not verified on a device: whether the message reads well in the case that matters, which is the slow-storage close it exists for. * runloop: introduce the content-closing state, observed by nothing Stage 2 of unblocking content close. A no-op by construction: the state is set and cleared around the teardown that already exists, so while it is true the main thread is inside that teardown and no frame runs to ask about it. It is introduced on its own precisely because the work that makes it observable - returning to the frame loop instead of blocking on the save wait - is a lifecycle change, and this is the piece the rest will key off. Reviewing the two separately means a problem with the lifecycle change is one revert, not an unpicking. A plain bool in runloop_state_t rather than a RUNLOOP_FLAG bit. Bits 0-30 of that word are taken and bit 31 was deliberately vacated to avoid a cross-thread race with the AppIntents entity query; reusing it would undo that reasoning for no gain, and this is main-thread only. The clear is placed at the block's single exit, which was checked rather than assumed: there is no early return or goto between the set and the break, so one clear covers every path. A leaked true would be harmless today and would wedge the frame loop at stage 3, which is exactly the kind of thing worth getting right while it still cannot bite. Verified: nothing reads the state except its own accessor, which nothing calls - grepped rather than asserted, since "no-op" is the entire claim here; builds with networking enabled and disabled; the save_state oracle passes and the wait_timeout CI step was replayed verbatim from the workflow. * task_queue: fix two data races TSan found in CI Two separate races, one mine and one older. Mine, in the oracle: work_pending was cleared by the task HANDLER, which runs on the worker thread, and read by the waiting main thread. A plain bool across two threads - a race on inspection, with no tooling needed to see it. It was also a misrepresentation of the thing under test: the code being modelled, netplay_mitm_query_cb(), clears its flag in the CALLBACK, which runs on the thread pumping the queue. Clearing it there fixes the race and makes the test match what it claims to model. The older one is in the queue itself. retro_task_threaded_reset() set RETRO_TASK_FLG_CANCELLED under running_lock, while the worker reads task->flags under property_lock - two locks for one field. It needed a reset to land while a worker was mid-task, which is why it sat undisturbed until a lane that cancels an unfinished task turned up. The flags are now set under property_lock as well. Nesting is safe: property_lock is never held while any other lock is taken anywhere in this file, checked by walking every lock/unlock pair, so it can only be an inner lock and no cycle exists. Both are load-bearing, measured rather than assumed: 40 TSan runs report 0 races with both fixes, and reverting the queue fix alone brings it back at 6/40. On the reporting of this: the commit that introduced the oracle said it passed twice under TSan. That was true - I ran it, three times including the CI replay, and it passed each time. The race reproduces about once in twelve runs here, so three clean runs were about a 78% chance of missing it. The claim was accurate and the verification was not adequate, which is the more useful way to say it. Repeat counts for anything threaded are now sized against how rare the failure can be, not against how many runs feel like enough - and a shared mutable flag written from a handler should have been caught by reading it, which is where this should have stopped. Verified: 40 TSan runs clean on the changed lane; 8 each on the other threaded oracles, since task_queue.c is shared by all of them; every task oracle passes on a plain build; desktop build clean; the CI step replayed verbatim from the workflow. * runloop: don't run the core while content is closing Stage 3 of unblocking content close. Still inert, and deliberately so: it establishes what the frame loop must do while the closing state is set, before anything can set it with frames running. retro_run() must not be entered once closing has begun, because the teardown unloads the library that function lives in. The guard therefore has to exist before the wait moves out of the close path, not with it - otherwise the change that lets frames run would be introducing both the opportunity and the protection at once. That it cannot fire today was checked rather than assumed: both core_run() call sites are inside runloop_iterate(), and the content close block in CMD_EVENT_CORE_DEINIT reaches neither - it calls no iterate, no core_run and no run_ahead, so the main thread cannot be in the frame loop while the flag is set. It polls input and presents a cached frame rather than returning bare, matching the netplay-paused case immediately below, and for the same reason: a frame that simply stops being produced reads as a hang, which is the whole complaint being worked on here. Verified: builds with networking enabled and disabled; the save_state oracle passes, including the lane that pins the close wait; the wait_timeout CI step replayed verbatim from the workflow. Worth stating before stage 4: at that point this guard stops being inert, and video_driver_cached_frame() will be running while a core is being torn down. That is the first thing to look at if the frame loop misbehaves during a close, and it is why the harness lane for content close belongs before that change rather than after it. * runloop: fix C89 violation in core_run The content-closing guard was placed before netplay_preframe's declaration, so that declaration followed a statement: runloop.c:8808: error: ISO C90 forbids mixed declarations and code netplay_preframe is now declared with the other locals and assigned where it was, so the guard still returns before netplay's pre-frame call and the declarations stay at the top of the block. My own standing rule is that every C file gets the strict C89 gate before commit, and I did not run it on this one - the ordinary desktop build does not use those flags, so it compiled clean and I took that as sufficient. It is not, and the gate exists precisely because it is not. Verified with the flags the C89 build actually uses, on runloop.c with networking both enabled and disabled, and with a full C89_BUILD=1 compile of the tree; reinstating the original ordering reproduces the reported error, so the check is real rather than vacuous. * samples: drive the content-closing guard against the real frame loop Stage 4 will stop blocking on the save wait and let frames run while a close finishes. The guard that makes that safe - core_run() refusing to enter retro_run() while content_closing is set - is already in place, and is at present completely unexercised: closing is synchronous, so no frame runs during it, and the guard would go live in the same change that first creates the situation it protects against. If it were wrong, the symptom would be a call into an unloaded dylib. So it is driven here first, directly: the real core_run(), in a real frontend brought up headless, with the flag set. Links the shipping objects with only main() replaced; nothing is stubbed. Three assertions, and the first exists to stop the others passing vacuously: with the flag CLEAR the core must actually be entered, or this harness would be proving nothing about a core_run() that never reaches retro_run() in this environment anyway. Then with the flag set it must not be entered, and once cleared it must resume - a sticky guard would leave the frame loop dead after a close. A second lane runs 240 frames with the flag set, the shape of a close on slow storage, checking core_run() returns cleanly every time rather than faulting or wedging. Removing the guard fails both lanes with exactly the message they exist to produce. It joins the existing menu-harness workflow, which already makes the non-Qt build these harnesses need - with Qt enabled ui_qt.o defines main and the link collides - so it adds about two seconds rather than another full build. The step was replayed verbatim from the workflow before commit, and the menu harness alongside it still passes. Verified: strict C89 gate on runloop.c with networking on and off, which is the check I skipped last time and should not have. * retroarch: extract the content-close teardown into its own function Preparation for removing the close freeze, and nothing else: the sixty lines that run after the save/load wait move into command_event_finish_content_deinit(), still called from exactly where they used to run. They have to become resumable. The wait above them is the freeze this work is about, and removing it means this half runs later - from the frame loop, once the tasks have finished - so it needs to be callable from somewhere other than the middle of a switch case. Doing the extraction on its own makes the next change a question of WHEN it is called rather than what it does, which is the difference between a revert and an unpicking if that goes wrong. Every local is re-derived from its accessor rather than passed in, so the function carries no dependency on the caller's frame - which is what lets it be called from the frame loop later. The extracted body was compared statement by statement against the original block: 53 statements, zero differences. That check was worth writing, because the first attempt at this anchored on a comment that appears twice in the file and silently swallowed an unrelated case - it failed to compile, but a subtler overshoot would not have. Verified: strict C89 gate on retroarch.c with networking on and off; builds with Qt and headless; both full-program harnesses pass, and the save_state oracle with them. --------- Co-authored-by: LibretroAdmin <[email protected]>
231 lines
7.4 KiB
C
231 lines
7.4 KiB
C
/* Oracle for task_queue_wait_timeout().
|
|
*
|
|
* task_queue_wait() waits indefinitely, which is correct for work
|
|
* that is certain to finish and wrong for anything depending on
|
|
* something outside the machine. The netplay MITM query is the
|
|
* motivating case: it waits on an HTTP round trip to the lobby
|
|
* server, and a server that never answers hangs the frontend with no
|
|
* way out.
|
|
*
|
|
* The bound is implemented by wrapping the caller's condition in one
|
|
* that also goes false at a deadline, so each queue implementation -
|
|
* regular, threaded, GCD - keeps its own waiting behaviour and no
|
|
* platform-specific code is involved. What has to hold:
|
|
*
|
|
* completes - a wait whose work finishes returns as soon as it
|
|
* does, reports success, and does NOT sit out the
|
|
* remaining timeout.
|
|
* times out - a wait whose condition never clears returns anyway,
|
|
* reports failure, and takes about the timeout rather
|
|
* than forever.
|
|
* no-op - a condition already false returns immediately.
|
|
* threaded - the same holds on the threaded queue, where the
|
|
* waiting is a different implementation.
|
|
*
|
|
* The timing assertions are deliberately loose at the top end: this
|
|
* runs on shared CI machines and the point is "bounded, not
|
|
* indefinite", not a precise duration. The lower bounds are tight,
|
|
* because returning EARLY would silently break the caller.
|
|
*/
|
|
|
|
#include <stdio.h>
|
|
#include <stdlib.h>
|
|
#include <string.h>
|
|
|
|
#include <boolean.h>
|
|
#include <queues/task_queue.h>
|
|
#include <features/features_cpu.h>
|
|
#include <retro_timers.h>
|
|
|
|
static unsigned failures = 0;
|
|
|
|
#define CHECK(cond, ...) \
|
|
do { \
|
|
if (!(cond)) \
|
|
{ \
|
|
fprintf(stderr, "FAIL %s:%d: ", __FILE__, __LINE__); \
|
|
fprintf(stderr, __VA_ARGS__); \
|
|
fprintf(stderr, "\n"); \
|
|
failures++; \
|
|
} \
|
|
} while (0)
|
|
|
|
#define TIMEOUT_USEC 300000 /* 0.3s - long enough to measure */
|
|
|
|
/* ------------------------------------------------------------------ */
|
|
/* A task that finishes after N ticks, and one that never finishes */
|
|
/* ------------------------------------------------------------------ */
|
|
|
|
static int ticks_remaining;
|
|
static bool work_pending;
|
|
|
|
/* Handlers run on the WORKER thread under the threaded queue, so
|
|
* nothing here touches state the waiting thread reads. The flag is
|
|
* cleared in the callback below - which is where the code being
|
|
* modelled clears it, netplay_mitm_query_cb(), and callbacks run on
|
|
* the thread pumping the queue.
|
|
*
|
|
* Clearing it here was both a data race and a misrepresentation of
|
|
* the thing under test. */
|
|
static void finite_handler(retro_task_t *task)
|
|
{
|
|
if (ticks_remaining > 0)
|
|
{
|
|
ticks_remaining--;
|
|
return;
|
|
}
|
|
task_set_flags(task, RETRO_TASK_FLG_FINISHED, true);
|
|
}
|
|
|
|
static void endless_handler(retro_task_t *task)
|
|
{
|
|
/* Models a network wait: the task stays in the queue, and the
|
|
* thing the caller is waiting for never arrives. */
|
|
}
|
|
|
|
/* Runs on the thread pumping the queue - the waiting thread - just
|
|
* as the netplay query callback does. */
|
|
static void cb(retro_task_t *task, void *task_data, void *user_data,
|
|
const char *error)
|
|
{
|
|
work_pending = false;
|
|
}
|
|
|
|
/* The caller's own condition, as a netplay-style "is it still
|
|
* pending" flag rather than a queue lookup. */
|
|
static bool still_pending(void *data)
|
|
{
|
|
return work_pending;
|
|
}
|
|
|
|
static bool push(void (*handler)(retro_task_t*))
|
|
{
|
|
retro_task_t *task = task_init();
|
|
if (!task)
|
|
return false;
|
|
task->handler = handler;
|
|
task->callback = cb;
|
|
return task_queue_push(task);
|
|
}
|
|
|
|
/* ------------------------------------------------------------------ */
|
|
/* Lanes */
|
|
/* ------------------------------------------------------------------ */
|
|
|
|
static void lane_completes_before_timeout(bool threaded)
|
|
{
|
|
unsigned had = failures;
|
|
retro_time_t started, elapsed;
|
|
bool ok;
|
|
|
|
task_queue_init(threaded, NULL);
|
|
ticks_remaining = 3;
|
|
work_pending = true;
|
|
|
|
CHECK(push(finite_handler), "push failed");
|
|
|
|
started = cpu_features_get_time_usec();
|
|
ok = task_queue_wait_timeout(still_pending, NULL, TIMEOUT_USEC);
|
|
elapsed = cpu_features_get_time_usec() - started;
|
|
|
|
task_queue_deinit();
|
|
|
|
CHECK(ok, "%s: reported failure for work that finished",
|
|
threaded ? "threaded" : "regular");
|
|
CHECK(!work_pending, "%s: work did not actually finish",
|
|
threaded ? "threaded" : "regular");
|
|
/* The point of the bound is that it does not delay the ordinary
|
|
* case; sitting out the full timeout would be a regression that
|
|
* still "passes" a pass/fail check. */
|
|
CHECK(elapsed < (TIMEOUT_USEC / 2),
|
|
"%s: took %lldus for work that finished immediately - the "
|
|
"wait is sitting out its timeout instead of returning",
|
|
threaded ? "threaded" : "regular", (long long)elapsed);
|
|
|
|
if (failures == had)
|
|
fprintf(stderr, "[pass] completes-before-timeout (%s, %lldus)\n",
|
|
threaded ? "threaded" : "regular", (long long)elapsed);
|
|
}
|
|
|
|
static void lane_times_out(bool threaded)
|
|
{
|
|
unsigned had = failures;
|
|
retro_time_t started, elapsed;
|
|
bool ok;
|
|
|
|
task_queue_init(threaded, NULL);
|
|
work_pending = true;
|
|
|
|
CHECK(push(endless_handler), "push failed");
|
|
|
|
started = cpu_features_get_time_usec();
|
|
ok = task_queue_wait_timeout(still_pending, NULL, TIMEOUT_USEC);
|
|
elapsed = cpu_features_get_time_usec() - started;
|
|
|
|
/* Abandon the endless task so deinit is not left holding it. */
|
|
task_queue_reset();
|
|
task_queue_check();
|
|
task_queue_deinit();
|
|
|
|
CHECK(!ok, "%s: reported success though the work never finished",
|
|
threaded ? "threaded" : "regular");
|
|
/* Tight lower bound: giving up early would cut short a round trip
|
|
* that was about to succeed. */
|
|
CHECK(elapsed >= (TIMEOUT_USEC - (TIMEOUT_USEC / 10)),
|
|
"%s: gave up after %lldus, well before the %dus timeout",
|
|
threaded ? "threaded" : "regular", (long long)elapsed,
|
|
TIMEOUT_USEC);
|
|
/* Loose upper bound: the machine is shared, the claim is only
|
|
* that this terminates rather than hangs. */
|
|
CHECK(elapsed < (TIMEOUT_USEC * 10),
|
|
"%s: took %lldus against a %dus timeout - not bounded",
|
|
threaded ? "threaded" : "regular", (long long)elapsed,
|
|
TIMEOUT_USEC);
|
|
|
|
if (failures == had)
|
|
fprintf(stderr, "[pass] times-out (%s, %lldus)\n",
|
|
threaded ? "threaded" : "regular", (long long)elapsed);
|
|
}
|
|
|
|
static void lane_already_done(void)
|
|
{
|
|
unsigned had = failures;
|
|
retro_time_t started, elapsed;
|
|
bool ok;
|
|
|
|
task_queue_init(false, NULL);
|
|
work_pending = false;
|
|
|
|
started = cpu_features_get_time_usec();
|
|
ok = task_queue_wait_timeout(still_pending, NULL, TIMEOUT_USEC);
|
|
elapsed = cpu_features_get_time_usec() - started;
|
|
|
|
task_queue_deinit();
|
|
|
|
CHECK(ok, "reported failure for a condition already false");
|
|
CHECK(elapsed < (TIMEOUT_USEC / 2),
|
|
"waited %lldus for something already finished",
|
|
(long long)elapsed);
|
|
|
|
if (failures == had)
|
|
fprintf(stderr, "[pass] already-done lane\n");
|
|
}
|
|
|
|
int main(void)
|
|
{
|
|
lane_completes_before_timeout(false);
|
|
lane_times_out(false);
|
|
lane_already_done();
|
|
|
|
lane_completes_before_timeout(true);
|
|
lane_times_out(true);
|
|
|
|
if (failures)
|
|
{
|
|
fprintf(stderr, "FAIL wait_timeout_test: %u failures\n", failures);
|
|
return 1;
|
|
}
|
|
fprintf(stderr, "PASS wait_timeout_test\n");
|
|
return 0;
|
|
}
|