Backups go under the assets directory, wherever saves live. A rename
fails when those are on different volumes, or on Android when saves are
in a SAF folder, and the file then was not backed up: non-destructive
mode gave up on replacing it.
Fall back to copying the file and removing the original. If it cannot
be removed, drop the copy and report the failure, as the caller relies
on the local file being gone. cloudsync_commit_test covers both a
replaced and a removed file with such a rename refused.
There is one cloud sync at a time, and a sync asked for while another
ran was turned away with a log line and nothing on screen. Content
loads and closes start network-bound syncs, so Sync Now, Resolve:
Keep Local and Resolve: Keep Server chosen during one did nothing.
A sync asked for without a pusher waiting on it is owed now, and
starts from the running sync's callback once it is through. Asked
for more than once, it runs once, with the conflict resolution asked
for last; a resolution wins over a plain sync. A pusher that waits on
its own sync is still told it was not run. The finder passes over a
finished sync, which stays in the queue while its callback runs, so
the follow-up started from there is not taken for a running one.
Frame thread only, where every push and the callback run: no lock.
cloudsync_pacing_test gains a follow-up lane: Keep Local and then
Sync Now during a sync are taken, keep the resolution, and one more
sync runs once the first is through. It fails on the old code.
A blocking driver's calls reach the worker on a many-producer stack and
its results come back on another; the worker parks on an eventcount
when the sync is still open and the queue is empty, and leaves once it
ends. A call queued as the worker leaves either starts a new worker or
is taken by the leaving one. cloud_sync_deinit() waits for the worker
on an eventcount, still bounded by its timeout.
Without pointer atomics the calls run directly, as without threads.
The async test queues a sync from the worker's leaving point.
Wait for the mock fetch callback without running the task consumer, so a completed download cannot commit before the fixture marks the save live. Keep the download pacing assertion under the same ordering. Use a local temporary directory and native path separators so the fixture also runs on Windows.
The sample and test builds used full debug info. -g1 keeps what
sanitizer reports and backtraces need - function names, files and line
numbers - and drops the variable and type information nobody reads in
CI. Objects get about 30% smaller (rh265.o under ASan+UBSan: 5.1MB to
3.7MB), so the ccache caches hold more of each job's build and are
quicker to restore and save.
Run the budgeted sync handler on the content thread while blocking network calls retain their worker. This keeps ownership checks, backups, replacements and deletions ordered with savefile lifetime changes.
Keep a local save in place until its download is complete. If the completion cannot be handed to the handler, discard the temporary file and carry the manifests forward instead of committing on the callback thread.
Exercise threaded and unthreaded queues, both replacement modes, ownership acquired during transfer, backup contents and failed completion allocation.
Samples compiled shared tree sources in place, so whichever sample ran
next linked those objects with the flags they were built with, as
unity_identity did with mic_handshake's ASan objects. Every sample that
builds a source from outside its directory now includes
sample_objdir.mk and maps those objects under its own obj/; its own
objects stay where they were. The rule for the mapped objects has a
slash in its target, so make 3.81 does not strip the directory.
tools/sample_objdir_check.py dry-runs every sample Makefile and fails
on an object written outside the sample; the Samples-Objdir-Check
workflow runs it.
A fetch is written beside its file and moved into place once complete,
unless a core has taken the file since the fetch began. A transfer that
fails part-way leaves the local file as it was.
The cloud_sync_capture hook copies the settings a driver's calls read,
on the main thread as the sync's task is pushed, where the task takes
its own copy too. SMB and NFS read the share, the account, the
timeouts and the destructive flag from that copy rather than from the
live settings: their calls run on a worker, read some of them after a
network call returns, and one still blocked past the exit wait would
otherwise read settings the frontend has freed.
cloudsync_async_test checks the hook runs once on the calling thread
and that a driver without one is left alone.
cloud_sync_deinit() moves the calls on to a new generation. A call made
before it that is still running when the wait gives up is disposed of by
the worker once it returns - its file closed, nothing handed back - so
its handler never runs for a task that is gone, and a sync begun later
never sees it.
cloud_sync_poll() takes the caller's budget callback and runs the oldest
finished result unconditionally, then more only while the budget allows;
the cloud sync task polls inside its shared I/O window, so a burst of
results is spread over as many runs as the window asks. NULL runs them
all. cloudsync_async_test covers the paced poll and a sync begun while an
abandoned call is still stuck.
Whether completion callbacks hand manifest entries and downloads over
on lock-free lists or under tcs_manifest_lock is decided in one place,
TCS_LOCK_FREE: lock-free where the atomics backend has pointer
operations, unless TASK_CLOUDSYNC_FORCE_LOCK asks for the locked path.
Every backend a CI runner has provides pointer atomics, so the locked
path was never built or run. samples/tasks/cloudsync builds the
manifest-adds and pacing tests with FORCE_LOCK=1 in a new lane of the
tasks workflow, plain, under ASan/UBSan and under TSan, where an append
or hand-over the lock does not cover is a reported race.
The current manifest is built by walking the synced directories with
retro_opendir/retro_readdir, one entry per unit of the shared per-frame
I/O window - at least one per run - instead of reading every directory
in one run of the handler, which with Threaded Tasks off is the frame
loop. The walk keeps a stack of open directories in the sync's state
and lists what dir_list_new() listed: files under the root
recursively, hidden ones included, directories left out, a
subdirectory that cannot be opened skipped. A root that cannot be read
still stops the sync.
cloudsync_pacing_test walks two roots of 3000 files with nested,
hidden and ignored entries, compares the result with dir_list_new()'s
and checks the grant per run and the unreadable root.
A finished download is handed to the task thread instead of being
hashed whole in the fetch callback, which runs on the main thread for
WebDAV, S3 and Google Drive and on the handler's thread for SMB and
NFS. The task hashes queued downloads 256 KiB at a time inside the
same per-frame I/O window as the diff - one window per run for both -
and only then records the fetch and drops it from the in-flight count.
The hand-over is lock-free where pointer atomics exist, under the
manifest lock otherwise.
The path buffers of the directory walk, the local-file backup and the
local manifest commit are on the heap, one block per call, keeping
every frame in task_cloudsync.c under 4 KiB. cloudsync_pacing_test
covers a 9 MiB download reported from another thread.
The diff runs as many steps per handler run as the shared per-frame I/O
window grants, at least one chunk of work each run, with no more than
four transfers in flight. The current file is hashed 256 KiB at a time
between steps and carried over to the next run when the window closes,
so a large save, memory card or system file no longer holds one run of
the handler - the frame loop with Threaded Tasks off - and small files
go several to a run instead of one per frame.
The state is released by the task's cleanup when it retires; every
manifest owns its entries' hashes, an entry added to the updated
manifests taking a copy. samples/tasks/cloudsync/cloudsync_pacing_test
checks the pacing against a stub window and runs a sync to retirement
on the real task queue under LeakSanitizer.
SMB and NFS are flagged CLOUD_SYNC_DRIVER_FLG_BLOCKING: their calls
block on the network until done. With threads, cloud_sync_*() queues
them for one worker per sync, which runs them in call order, and
returns at once; cloud_sync_poll(), called first by the task handler,
runs each completion handler on the handler's own thread. A slow or
unreachable server no longer holds up the thread driving the sync -
the frame loop itself with Threaded Tasks off. Each call is one
allocation with its names copied; the worker leaves once the sync ends
or fails to begin. Without threads the calls are made directly.
On exit, after the task drain, cloud_sync_deinit() drops the calls not
started (closing an upload's file) and waits at most a second for the
one under way, so no driver code runs into the teardown.
samples/tasks/cloudsync/cloudsync_async_test covers ordering, threads,
copying, refusals, worker turnover and the exit path in CI.
Completion callbacks for fetches, uploads and deletes add entries to
the two updated manifests from whatever thread the driver calls back
on, as the task thread does for the entries it settles itself. Each
append took a process-wide lock.
An entry is now a record pushed on a lock-free list in the sync's own
state, and the manifests are the task thread's alone: it folds the
records in when the diff is over and nothing is in flight, which is
the first time a manifest is read. A callback pushes before it drops
`waiting`, so the fold sees every record. The manifest is sorted
before it is written, so the order the records arrive in does not
matter. A sync that ends early folds them in before freeing the
manifests, so what was added is freed as it was.
An append that could not allocate used to write through a list that
had not grown; it now marks the sync failed. Backends without pointer
atomics keep the lock.
cloudsync_manifest_adds_test adds entries from four threads and the
task's own at once and checks each is in its manifest once, with its
own hash, under ThreadSanitizer and ASan.
Three places trust data that comes from outside RetroArch:
- replay_set_serialized_data reads the embedded replay length from the
save state. A negative value, or one shorter than the replay header,
was passed on to the seek and write that restore the replay. Refuse
such states.
- natt_parse_desc_node walks the IGD description XML a router serves.
On a node without children it recursed with a NULL child and then
read child->next, so a leaf node or an empty root crashed it. It
also recursed only into childless nodes, so a <service> nested in
<root>/<device>/<serviceList>, the normal UPnP shape, was never
found. It now always recurses into children and checks the
serviceType and controlURL text before using it.
- task_cloud_sync_fetch_server_file took the local path from a
server-supplied manifest key as strchr(key, '/') + 1. A key without
'/' read from address 1, and a key containing ".." or an absolute
path was joined onto the sync directory, so a hostile server could
write outside it. Such keys are now refused and counted as a
failure.
The description walk moves to network/natt_desc.c and the key check
to tasks/task_cloudsync_path.c, so the new sample tests
(samples/tasks/natt_desc, and cloudsync_path_safety_test beside the
existing cloudsync test) compile the shipped code directly. Both are
added to Linux-samples-tasks.yml.
Tested on Linux: full build; touched files also compile with
C89_BUILD=1 without warnings. The workflow's commands for both new
steps pass, and the existing cloudsync test still passes under
SANITIZER=address. With the old parser or the old key handling
substituted, each test segfaults.
Co-authored-by: Claude Opus 5.5 <[email protected]>
dir_list_new() returns NULL for a root it cannot open, so a sync
directory that does not exist - an unset system path, a mount that is
not there - dereferenced NULL while the manifest of local state was
being built. The listing result is now checked, and every caller of the
directory map handles its absence.
Skipping the directory would have been worse than the crash it replaced.
An empty listing is a statement that every file the directory used to
hold is gone, and the diff answers that by removing the server's copies;
a directory that could not be read says nothing at all, so the two have
to be told apart. Where the local state cannot be established, the sync
stops with a failure rather than working from a manifest that is missing
files it never looked for.
The samples/tasks cloudsync test covers that distinction directly, since
the whole fix rests on it.
cloudsync_savefile_owner_test pins the two properties the startup
rollback fix rests on: that save file ownership tracks the save file
list exactly, from the point content load builds it to the point the
deinit chain frees it, with matching that ignores case; and that
file_list_search() resolves a full portable manifest key to its own
entry rather than to a longer key that shares it as a prefix, which is
what lets the carry-forward recover the hash the last sync recorded.
The predicate reads a file-static list in save.c, so the test keeps a
verbatim copy of it and links the real file_list_search(). The CI Linux
samples/tasks workflow builds and runs it under AddressSanitizer
alongside the other task regression lanes.