mirror of
https://github.com/libretro/RetroArch.git
synced 2026-11-22 10:50:15 -10:00
codex/android-native-streaming
82
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
7f88ccd7ea |
task_cloudsync: hash a download where it lies
A download comes back on the stream it was written through, opened to read and write, which cannot lend its bytes: hashing it read the whole file back into the 256 KB hash buffer. A download of a hash chunk or more is now opened again to read with the mapping hint, after a flush of the stream that wrote it, and hashed through filestream_view out of the page cache the write just filled, with no copy. A smaller one is a single read, cheaper than another open, and keeps its stream. Where the platform has no mapping the new stream reads as the old one did. |
||
|
|
267d4453b5 |
cloud sync: back up by copying when a rename cannot reach the backup
Backups go under the assets directory, wherever saves live. A rename fails when those are on different volumes, or on Android when saves are in a SAF folder, and the file then was not backed up: non-destructive mode gave up on replacing it. Fall back to copying the file and removing the original. If it cannot be removed, drop the copy and report the failure, as the caller relies on the local file being gone. cloudsync_commit_test covers both a replaced and a removed file with such a rename refused. |
||
|
|
45c64b3c73 |
Fix clang warnings in the Windows build
The Win64 build's -Wshorten-64-to-32 narrowings are made explicit where the value is bounded: socket()/accept() handles kept as RetroArch's int descriptors, ai_addrlen passed as socklen_t, UDP and pipe lengths, COM/D3D counts and shader frame counters, WASAPI/XAudio2 buffer sizes. net_socket clamps each send/recv to Winsock's int count on _WIN32 and lets the existing loops move the rest; the remote-input descriptors are plain ints on every platform, as socket_init returns them. xaudio29.h declares its interfaces through a macro that, in C, leaves out DECLARE_INTERFACE's second typedef of the forward-declared names. |
||
|
|
b143f37aa4 |
task_cloudsync: Sync Now and Resolve during a sync run after it
There is one cloud sync at a time, and a sync asked for while another ran was turned away with a log line and nothing on screen. Content loads and closes start network-bound syncs, so Sync Now, Resolve: Keep Local and Resolve: Keep Server chosen during one did nothing. A sync asked for without a pusher waiting on it is owed now, and starts from the running sync's callback once it is through. Asked for more than once, it runs once, with the conflict resolution asked for last; a resolution wins over a plain sync. A pusher that waits on its own sync is still told it was not run. The finder passes over a finished sync, which stays in the queue while its callback runs, so the follow-up started from there is not taken for a running one. Frame thread only, where every push and the callback run: no lock. cloudsync_pacing_test gains a follow-up lane: Keep Local and then Sync Now during a sync are taken, keep the resolution, and one more sync runs once the first is through. It fails on the old code. |
||
|
|
fa0ceb469f |
file_stream: filestream_view(), a read that lends instead of copying
filestream_view() points at up to len octets at the stream position and moves past them, as filestream_read() would after copying them: from the gets/getc lookahead, from the file's mapping, from a frontend VFS's mapping, or from an smb:// or nfs:// stream's read-ahead window. -1 lends nothing and moves nothing, and the caller reads as before. The scan CRC step, sha1_calculate(), the cloud sync MD5 and filestream_matches_buf() take what can be lent where it lies and read the rest, so a hash over a share with read-ahead on no longer copies what the prefetcher already holds; on a mapped file they consume the mapping as before. A mapping over 4 GiB is hashed in blocks rather than passed as one length the hashers truncate. vfs_view_test covers the lookahead, mapped, unmapped, update-stream and frontend-mapping cases and sha1_calculate(); the smb:// and nfs:// thread tests read a file through lends against its contents with read-ahead off and on. |
||
|
|
7361e44c2d |
Cloud sync: serialize save mutations with content ownership
Run the budgeted sync handler on the content thread while blocking network calls retain their worker. This keeps ownership checks, backups, replacements and deletions ordered with savefile lifetime changes. Keep a local save in place until its download is complete. If the completion cannot be handed to the handler, discard the temporary file and carry the manifests forward instead of committing on the callback thread. Exercise threaded and unthreaded queues, both replacement modes, ownership acquired during transfer, backup contents and failed completion allocation. |
||
|
|
250e2d1b48 | save: write states, SRAM and disk index through a temporary file (#19629) | ||
|
|
5221eea01f |
cloudsync: a download lands only whole, and never under a running core
A fetch is written beside its file and moved into place once complete, unless a core has taken the file since the fetch began. A transfer that fails part-way leaves the local file as it was. |
||
|
|
9312a11000 | cli: Add a --cloudsync command to sync the Cloud from the CLI (#19504) | ||
|
|
0e464bdf41 |
cloudsync: a blocking driver's calls read settings copied as the sync is pushed
The cloud_sync_capture hook copies the settings a driver's calls read, on the main thread as the sync's task is pushed, where the task takes its own copy too. SMB and NFS read the share, the account, the timeouts and the destructive flag from that copy rather than from the live settings: their calls run on a worker, read some of them after a network call returns, and one still blocked past the exit wait would otherwise read settings the frontend has freed. cloudsync_async_test checks the hook runs once on the calling thread and that a driver without one is left alone. |
||
|
|
c7f72914a9 |
cloudsync: a result past the exit wait is the worker's, and polls are paced
cloud_sync_deinit() moves the calls on to a new generation. A call made before it that is still running when the wait gives up is disposed of by the worker once it returns - its file closed, nothing handed back - so its handler never runs for a task that is gone, and a sync begun later never sees it. cloud_sync_poll() takes the caller's budget callback and runs the oldest finished result unconditionally, then more only while the budget allows; the cloud sync task polls inside its shared I/O window, so a burst of results is spread over as many runs as the window asks. NULL runs them all. cloudsync_async_test covers the paced poll and a sync begun while an abandoned call is still stuck. |
||
|
|
072a70bc75 |
cloudsync: CI covers the locked hand-over path
Whether completion callbacks hand manifest entries and downloads over on lock-free lists or under tcs_manifest_lock is decided in one place, TCS_LOCK_FREE: lock-free where the atomics backend has pointer operations, unless TASK_CLOUDSYNC_FORCE_LOCK asks for the locked path. Every backend a CI runner has provides pointer atomics, so the locked path was never built or run. samples/tasks/cloudsync builds the manifest-adds and pacing tests with FORCE_LOCK=1 in a new lane of the tasks workflow, plain, under ASan/UBSan and under TSan, where an append or hand-over the lock does not cover is a reported race. |
||
|
|
ce2be126f2 |
cloudsync: the local files are listed a few entries per run
The current manifest is built by walking the synced directories with retro_opendir/retro_readdir, one entry per unit of the shared per-frame I/O window - at least one per run - instead of reading every directory in one run of the handler, which with Threaded Tasks off is the frame loop. The walk keeps a stack of open directories in the sync's state and lists what dir_list_new() listed: files under the root recursively, hidden ones included, directories left out, a subdirectory that cannot be opened skipped. A root that cannot be read still stops the sync. cloudsync_pacing_test walks two roots of 3000 files with nested, hidden and ignored entries, compares the result with dir_list_new()'s and checks the grant per run and the unreadable root. |
||
|
|
328f65a195 |
cloudsync: downloads are hashed by the task, a chunk at a time
A finished download is handed to the task thread instead of being hashed whole in the fetch callback, which runs on the main thread for WebDAV, S3 and Google Drive and on the handler's thread for SMB and NFS. The task hashes queued downloads 256 KiB at a time inside the same per-frame I/O window as the diff - one window per run for both - and only then records the fetch and drops it from the in-flight count. The hand-over is lock-free where pointer atomics exist, under the manifest lock otherwise. The path buffers of the directory walk, the local-file backup and the local manifest commit are on the heap, one block per call, keeping every frame in task_cloudsync.c under 4 KiB. cloudsync_pacing_test covers a 9 MiB download reported from another thread. |
||
|
|
b77f298622 |
cloudsync: the diff is paced by the I/O window; a sync releases its state
The diff runs as many steps per handler run as the shared per-frame I/O window grants, at least one chunk of work each run, with no more than four transfers in flight. The current file is hashed 256 KiB at a time between steps and carried over to the next run when the window closes, so a large save, memory card or system file no longer holds one run of the handler - the frame loop with Threaded Tasks off - and small files go several to a run instead of one per frame. The state is released by the task's cleanup when it retires; every manifest owns its entries' hashes, an entry added to the updated manifests taking a copy. samples/tasks/cloudsync/cloudsync_pacing_test checks the pacing against a stub window and runs a sync to retirement on the real task queue under LeakSanitizer. |
||
|
|
5e968f8b99 |
cloudsync: blocking drivers run on a worker, results come back by poll
SMB and NFS are flagged CLOUD_SYNC_DRIVER_FLG_BLOCKING: their calls block on the network until done. With threads, cloud_sync_*() queues them for one worker per sync, which runs them in call order, and returns at once; cloud_sync_poll(), called first by the task handler, runs each completion handler on the handler's own thread. A slow or unreachable server no longer holds up the thread driving the sync - the frame loop itself with Threaded Tasks off. Each call is one allocation with its names copied; the worker leaves once the sync ends or fails to begin. Without threads the calls are made directly. On exit, after the task drain, cloud_sync_deinit() drops the calls not started (closing an upload's file) and waits at most a second for the one under way, so no driver code runs into the teardown. samples/tasks/cloudsync/cloudsync_async_test covers ordering, threads, copying, refusals, worker turnover and the exit path in CI. |
||
|
|
9274caf034 |
cloudsync/nfs: never lose a save to a failed transfer
A download is written to <file>.rafetching and renamed over the local file only once complete, so a failed open or a read failing part way leaves the save as it was; the task skips leftover .rafetching files. Only the server answering NOENT counts as "not there": an unanswered open, stat or unlink fails rather than reading as a missing file or a finished delete. An upload is one RENAME over the old file; with destructive sync off the old file is first kept as <path>-<time>, then -2 to -9 within one second, and a backup that cannot be made or a stat with no answer fails the upload instead of replacing unguarded. Names that do not fit are refused rather than truncated, empty components in nfs_subdir are dropped, and a directory another client makes at the same moment is accepted. Path buffers and the transfer buffer sit in one scratch block allocated per session, keeping every frame small. samples/network/nfs_backup runs the driver against an in-memory export in the samples CI workflow. |
||
|
|
8fc5e52783 |
cloud sync: entries reach the updated manifests without a lock
Completion callbacks for fetches, uploads and deletes add entries to the two updated manifests from whatever thread the driver calls back on, as the task thread does for the entries it settles itself. Each append took a process-wide lock. An entry is now a record pushed on a lock-free list in the sync's own state, and the manifests are the task thread's alone: it folds the records in when the diff is over and nothing is in flight, which is the first time a manifest is read. A callback pushes before it drops `waiting`, so the fold sees every record. The manifest is sorted before it is written, so the order the records arrive in does not matter. A sync that ends early folds them in before freeing the manifests, so what was added is freed as it was. An append that could not allocate used to write through a list that had not grown; it now marks the sync failed. Backends without pointer atomics keep the lock. cloudsync_manifest_adds_test adds entries from four threads and the task's own at once and checks each is in its manifest once, with its own hash, under ThreadSanitizer and ASan. |
||
|
|
f539ccd3d6 |
task_queue: flags and progress are atomics, off property_lock
task_get_flags(), task_set_flags(), task_get_progress() and task_set_progress() each took the process-wide property_lock, so every handler polling for cancellation and every progress update queued behind every other task's, and behind the progress report formatting a title under the same lock. The queue itself took it again to cancel, to settle a task and to test for a main-thread or detachable one. retro_task_t::flags and ::progress are retro_atomic_int_t now. Reads are an acquire load, progress a release store, and flag updates an atomic fetch-or / fetch-and, so concurrent updates of different bits cannot lose one another. Only a backend whose read-modify-write is not atomic (RETRO_ATOMIC_LOCK_FREE undefined) still takes property_lock, and only around the flag update. Title and error stay under property_lock: their buffers are freed and replaced, which an atomic pointer would not make safe. The snapshot and the progress report read the flags before the strings, so a task seen finished is seen with the error it set before finishing. Code that touched the two fields on a task already pushed goes through the accessors: the progress callbacks, the cloud sync handler, the emscripten HTTP handler and the cheevos badge tasks. Setting them on a task not yet pushed is unchanged. task_queue_snapshot_test covers both properties: bits flipped from two threads with a third held, and a finished task always snapshotting with its error. retro_task_threaded_gather drops from 8 lock acquisitions to 6 and the budget reference moves with it. |
||
|
|
4ee62a575f |
security: validate replay, UPnP and cloud sync input (#19628)
Three places trust data that comes from outside RetroArch: - replay_set_serialized_data reads the embedded replay length from the save state. A negative value, or one shorter than the replay header, was passed on to the seek and write that restore the replay. Refuse such states. - natt_parse_desc_node walks the IGD description XML a router serves. On a node without children it recursed with a NULL child and then read child->next, so a leaf node or an empty root crashed it. It also recursed only into childless nodes, so a <service> nested in <root>/<device>/<serviceList>, the normal UPnP shape, was never found. It now always recurses into children and checks the serviceType and controlURL text before using it. - task_cloud_sync_fetch_server_file took the local path from a server-supplied manifest key as strchr(key, '/') + 1. A key without '/' read from address 1, and a key containing ".." or an absolute path was joined onto the sync directory, so a hostile server could write outside it. Such keys are now refused and counted as a failure. The description walk moves to network/natt_desc.c and the key check to tasks/task_cloudsync_path.c, so the new sample tests (samples/tasks/natt_desc, and cloudsync_path_safety_test beside the existing cloudsync test) compile the shipped code directly. Both are added to Linux-samples-tasks.yml. Tested on Linux: full build; touched files also compile with C89_BUILD=1 without warnings. The workflow's commands for both new steps pass, and the existing cloudsync test still passes under SANITIZER=address. With the old parser or the old key handling substituted, each test segfaults. Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
|
d0c0b52693 |
cloud sync: progress reaches 100% when a sync toggle is off
Server and local manifest entries under a directory whose sync toggle is off are carried forward rather than synced, and the diff steps over them in one go. Progress now leaves them out of both the done and the total counts, and is taken after each diff step rather than before it, so the step that ends the diff also lands the bar on 100%. Previously a disabled directory sorting last (thumbnails after saves and states) was skipped in the final step and the bar stayed at the in-scope share, e.g. 5% (#19606). The directory-scope test moves into task_cloud_sync_key_in_scope(), shared by the skip and by the count taken when the diff starts, so the two cannot disagree. |
||
|
|
67e3aef27d |
webdav: back up the server's copy before an upload replaces it
With destructive sync off, a WebDAV delete moves the server's file into deleted/, but an upload overwrote it in place: the setting protected against the server's copy being removed and not against it being replaced, which is the more likely way to lose a save (an older or broken save from another device overwriting a good one). The code carried a TODO/FIXME for exactly this. The local side already keeps cloud_backups/ before a fetch replaces a local file. An upload now copies the server's file to deleted/<path>-<yymmdd- hhmmss>, the name a delete uses, before the PUT. COPY rather than MOVE, so the server keeps its file until the PUT replaces it, and an upload that fails after the backup leaves the server as it was. A 404 means there is nothing to back up yet; any other failure stops the upload rather than overwrite without a backup. The COPY and the PUT each get their own Digest retry. The server manifest is rewritten every sync and is not backed up; its name moves to cloud_sync_driver.h as CLOUD_SYNC_SERVER_MANIFEST so the driver can recognise it. Adds task_push_webdav_copy() beside task_push_webdav_move() in both HTTP backends, sharing the Destination-header code. samples/network/webdav_backup includes webdav.c and plays the server request by request: backup before PUT, into deleted/ named as a delete names it, 404 not an error, a failed backup stopping the upload with one failure report, destructive sync and the manifest not backed up, and a Digest retry each for COPY and PUT. Against the old driver 5 checks fail. Runs in CI Linux samples/network under ASan+UBSan and TSan; the other WebDAV harnesses gain the new stub. |
||
|
|
d997fe013b |
cloudsync: stop refetching files the server does not have
A file listed in the server manifest but missing on the server (such as one uploaded under a name cut short at '#' before the WebDAV path encoding fix) came back from the driver as success with no file. task_cloud_sync_fetch_cb() logged that as "Failed to write file from server", kept the entry in the server manifest and failed the sync, so the same fetch failed on every sync after. That result was ambiguous in the drivers too: WebDAV and Google Drive also returned success with no file when the download could not be written locally, and when the file on the server was empty. Both now hand back an open RFILE for any found file, empty included, and report a local open or short write as a failure. Success with no file now means only "not on the server", which is what SMB, S3 and iCloud Drive already meant by it. task_cloud_sync_fetch_cb() then drops such an entry from the server manifest and marks it for upload, leaving the local file and local manifest alone: if this device has the file, the next sync sees it as new and uploads it, which repairs the server. Since an empty server manifest now arrives as an empty RFILE rather than none, treat an empty one as no manifest, as an empty body was before; rjson rejects empty input. |
||
|
|
182c6cb6c6 |
cloudsync: write manifest.local through a temporary file
The local manifest was rewritten in place, so a crash, a full disk or a power cut part way through left it empty or truncated. The last fix makes the next sync discard such a file, but that costs the sync its record of the previous state, and every differing file then goes through conflict resolution. Write manifest.local.tmp and rename it over manifest.local only once it is complete. filestream_rename() replaces an existing destination on every platform, including Windows. If the write or the rename fails, the temporary file is removed and the old manifest is left in place, which is the safe outcome task_cloud_sync_commit_local_manifest() already describes: the next sync re-examines the same files. libretro VFS has no fsync, so on filesystems that do not order a rename after the data it replaces, a power cut can still lose the new manifest; the old one, or the discard path, covers that. |
||
|
|
4ae32cee45 |
Fix WebDAV cloud sync crashes and encoded save paths (#19598)
* cloudsync: reject invalid manifests Check the JSON parser result before sorting a manifest. Discard partial manifests and stop the sync with an error instead of dereferencing an incomplete entry. * webdav: handle missing files and encode sync paths Treat HTTP 404 as a missing file without writing the server's error page to disk. Encode sync paths separately from the WebDAV base URL so filenames containing '#' are sent as filenames, not fragments. * webdav: keep '/' as separator in deleted/ backup path fill_pathname_join_special(dest, "deleted", path) appends PATH_DEFAULT_SLASH when the directory has no slash of its own. On Windows that is a backslash, which net_http_urlencode then sends as %5C, so every backup MOVE targeted "deleted%5C<path>" instead of the deleted/ collection. Pass "deleted/" so the existing '/' is kept. * cloudsync: recover from an invalid local manifest manifest.local is rewritten in place, so a crash, full disk or power loss during the write can leave it empty or truncated. rjson rejects empty input, so with the new manifest validation every later sync stopped with "Invalid manifest" and the only way out was switching sync drivers, which is what deletes the file. Log a warning, delete the bad local manifest and continue as a first sync. With no local manifest, files that differ between local and server go through conflict resolution instead of being overwritten. An invalid server manifest is still fatal. --------- Co-authored-by: LibretroAdmin <[email protected]> |
||
|
|
6107ebdc78 |
cloudsync: the directory map is built transactionally
The map constructor checked string_list_new and nothing after it: each append went unchecked and was followed by elems[size - 1], which on a failed append indexes the previous entry or underflows an empty list, and each strdup went unchecked, leaving an appended entry whose userdata - the directory the manifest walks - is NULL. Every entry now goes through one helper that copies the path first, appends second, and attaches the copy only once both exist; any failure frees the whole list, userdata included, and the constructor returns NULL, which its caller already handles. Verified: the cloudsync suite under plain, ASan+UBSan and TSan, the full build with the declaration gate, and the 91-lane matrix. |
||
|
|
9a8f4d815f |
cloudsync: the worker syncs with what the push captured
The handler runs on the threaded task queue's worker, and it read the sync toggles, the destructive flag, the core-assets directory, the save and state directories and the config path from the live settings and runloop mid-sync - torn reads if the menu writes a path while a sync runs. The push captures all of it into the task's state on the main thread: the destructive flag and the core-assets directory as fields, the directory map as a list the task owns. The per-task list also retires the process-wide directory-map cache and its staleness handling: a fresh list per sync is a handful of strdups, and it cannot disagree with the toggles the sync was pushed under. tools/thread_read_audit.py --binary retroarch: 39 findings become 30, and task_cloud_sync_task_handler no longer appears. |
||
|
|
f1a65ed467 |
cloud sync: a directory that cannot be listed stops the sync
dir_list_new() returns NULL for a root it cannot open, so a sync directory that does not exist - an unset system path, a mount that is not there - dereferenced NULL while the manifest of local state was being built. The listing result is now checked, and every caller of the directory map handles its absence. Skipping the directory would have been worse than the crash it replaced. An empty listing is a statement that every file the directory used to hold is gone, and the diff answers that by removing the server's copies; a directory that could not be read says nothing at all, so the two have to be told apart. Where the local state cannot be established, the sync stops with a failure rather than working from a manifest that is missing files it never looked for. The samples/tasks cloudsync test covers that distinction directly, since the whole fix rests on it. |
||
|
|
875e3c9a25 |
cloud sync: a backup that did not happen fails the operation closed
Non-destructive mode promises a local copy of anything it displaces, so task_cloud_sync_backup_file() now reports whether the file actually reached the backup directory, and the fetch and delete paths act on that: the local file keeps its place, the sync is marked as having failed, and both manifests carry forward the hashes they already hold so the next run sees the same difference and tries again. Destructive deletes check their own result the same way. task_cloud_sync_carry_manifests_forward() holds that manifest bookkeeping in one place, shared with the live-savefile deferral. The backup lookup also searches the current manifest by the full portable key, which is what the manifest is keyed on, rather than by the path below the directory prefix. |
||
|
|
c986f7ba1b |
cloud sync: leave save files alone while a core owns them
A loaded core holds its save RAM in memory and writes it back over the file on disk when the content closes, so for as long as the content is up the file is not the authoritative copy. content_savefile_is_live() answers whether a path belongs to that set, and cloud sync consults it before replacing or removing a local file: where the running core owns the file the operation is deferred and both manifests carry forward the hashes they already hold. The next sync runs once the deinit chain has written save RAM out, sees the real difference between the two sides and resolves it, raising a conflict where both of them moved. This matters most when RetroArch is launched with content on the command line, because the core loads its save RAM well before the startup sync has a server manifest to diff against, and a file fetched into place behind it is overwritten from core memory at unload and then uploaded in place of the server's copy. |
||
|
|
b12d1d0684 |
strl: include compat/strl.h where strlcpy_lit() is used
The literal conversion left 39 files calling strlcpy_lit() without including the header that declares it. They built anyway: the files were already calling strlcpy() without including compat/strl.h either, and on glibc 2.38 and later that resolves to the C library's own strlcpy(), so nothing ever forced the include. strlcpy_lit() has no such fallback -- it is an implicit declaration that compiles and then fails to link. RetroArch itself did not notice, because its translation units pull the header in through something else. The standalone samples do not, and samples/tasks/database is where it surfaced: scan_begin_budget_test.c: undefined reference to `strlcpy_lit' Adding the include to each of them fixes that and removes the reliance on the accidental libc fallback for strlcpy() at the same time. Checked by building every sample rather than the one that failed: 42 under samples/, 38 under libretro-common/samples and libretro-db/samples, with libxrandr, libdrm and libvulkan headers installed so the X11, KMS and Vulkan ones actually compile instead of being skipped. All build except samples/gfx/font_lifecycle, which fails identically on a clean tree (undefined reference to video_driver_thread_wrapper_active in its HAVE_THREADS variant) and is unrelated. The sample binaries were run as their workflows run them, including "make fixtures" for gfx_thumbnail_preview and a long enough timeout for the 1000000-iteration SPSC stress in gfx_thumbnail_status_atomic: all pass. samples/tasks/database builds, passes "test -x database_task" and exits 1 with its usage line, which is what the smoke step expects. |
||
|
|
b9be38fd55 |
strl: copy string literals without scanning them at runtime
strlcpy() begins with strlen() of its source. It lives in another translation unit, so with no LTO that scan happens at runtime even when the source is a literal whose length the compiler already measured, and the call itself cannot be folded away either. strlcpy_lit() passes the length over as sizeof(). When the destination size is also a constant -- sizeof(buf), the usual case -- nothing is left but the stores memcpy() compiles to: strlcpy "core_options" sizeof(buf) 7.01 ns strlcpy_lit "core_options" sizeof(buf) 0.42 ns strlcpy 41-char literal sizeof(buf) 7.74 ns strlcpy_lit 41-char literal sizeof(buf) 0.71 ns strlcpy "core_options" runtime len 7.11 ns strlcpy_lit "core_options" runtime len 0.70 ns The destination bound is kept. When the literal does not fit it defers to strlcpy(), so the copy truncates as before rather than overrunning; a form that took only the literal would have turned every one of these sites into an overflow waiting for a small buffer. The literal is concatenated with "" so a non-literal source is a compile error rather than a sizeof() that silently measures a pointer. 818 sites across 104 files were converted by parsing arguments rather than by pattern-matching text: only a three-argument call whose second argument is a bare string literal is rewritten, so a macro that happens to expand to a literal is left alone. Line endings are preserved, so the CRLF files carry only their own changed lines. test/compat/test_strl.c compares the two against each other over every destination size from 0 past the literal's length, for literals that are empty, single-character, path-shaped, adjacently concatenated and escape-bearing, checking the return, the terminator and the whole buffer so an overrun past the terminator is caught. ASan+UBSan+LSan and TSan both 4/4. The full menu_displaylist_build_list walk over every displaylist type is byte-identical to before the change (md5 a2a081bcac09cdc8d72d67efd14ac40e). |
||
|
|
fe5dd9217e |
rjson: move file I/O out of the codec
The contract for libretro-common/formats is that a codec imposes no file I/O through its interface: parsing and writing run over caller-supplied buffers or caller-supplied callbacks, and how bytes reach the parser - filestream, intfstream, or anything else - is the caller's decision, not the format's. rjson violated it four ways: rjson_open_stream / rjson_open_rfile and their writer equivalents lived inside rjson.c, coupling the codec translation unit to both stream layers for the sake of four thin shims. The shims move to a new opt-in, header-only adapter (formats/rjson_stream.h) as static INLINE functions with names that say what they couple to: rjson_open_intfstream / rjson_open_filestream and the rjsonwriter equivalents, preserving the size-scaled io block heuristic. Including the adapter couples the including translation unit - and only it - to the stream layers; rjson.c and rjson.h now reference no I/O at all, and a user with their own I/O wires rjson_open_user / rjsonwriter_open_user directly, exactly as the in-tree callers now demonstrate. All sixteen call sites across the nine in-tree callers migrate mechanically (core_info, net-retropad, disk_index_file, the two test input drivers, menu_explore, playlist, runtime_file, cloudsync). One latent bug surfaced: rjson.c used INLINE while reaching retro_inline.h only transitively through the stream headers it no longer includes; it now includes it explicitly. Verified: rjson.c passes the strict C89 gate standalone (it could not before - the missing include was masked); full Linux build links with HAVE_QT=1; strict gates clean on the migrated C89 callers; net-retropad translation unit syntax-checked; no remaining references to the removed constructors anywhere in the tree. |
||
|
|
ef2b482a1b |
task_cloudsync: four defects that make a failed sync look like a clean one
None of these were reported. All four were found while building a two-device iCloud test rig, and each one turns a failure into silence. 1. task_cloud_sync_directory_map() built its list once per process and never reconsidered it. Under CLOUD_SYNC_MODE_AUTOMATIC a sync fires at startup, so the map was already populated before the user could reach the menu, and turning "Sync: Saves/States" on afterwards had no effect until RetroArch was restarted - the sync ran to completion having considered zero files, which is indistinguishable from a sync with nothing to do. The list is now rebuilt when any of the four settings it derives from has changed. string_list_free() releases elems[i].userdata as well as elems[i].data, so the strdup'd directory strings hung off it are not leaked. 2. task_cloud_sync_update_manifest_cb() ignored its `success` argument: it logged "Uploading updated manifest succeeded" unconditionally and never set ->failures, so a failed server-manifest upload finished the task titled "Cloud Sync finished". Every other completion callback in this file honours the flag. 3. The local manifest was written before the server manifest upload was known to have landed. A failed upload therefore left the local manifest asserting hashes the server had never been told about, and the next sync reads that as local == current and local != server, concludes the server changed, and fetches - so a transient upload failure could pull an older server copy over a newer local file. The write now happens in the completion callback on success, or directly when there is nothing to upload. When it does not happen, the next sync re-examines the same files and re-uploads: wasteful and safe rather than quiet and lossy. 4. task_cloud_sync_end_handler() guarded task->state for NULL and then dereferenced sync_state->start_time, ->uploads and ->downloads outside the guard. Moved inside it. Verified on two iPhones against the iCloud driver, CloudKit development environment: - (1) toggling "Sync: Saves/States" off and on, then "Sync: Configuration Files" on, within a single session moved the reported disk-state count 1 -> 0 -> 1 -> 2 with no restart, so the free-and-rebuild ran three times and picked up a different directory the third time. - (2) and (3) together, with icloud_update() temporarily failing every manifest upload: the sync logged "Uploading updated manifest failed.", finished titled "Cloud Sync finished with failures", and wrote no local manifest. The following sync re-examined and re-uploaded the same file rather than treating the server's stale copy as newer. - (3) on the success path: "Wrote .../manifest.local" now follows "Uploading updated manifest succeeded" rather than preceding the upload. - A full round trip afterwards - upload on one device, download on the other, both converging to 0 uploaded / 0 downloaded - to confirm the reordering did not disturb the ordinary path. - (4) is not reachable from the UI and is verified by inspection only. Sanitizers: Address and Undefined Behavior clean on device across the sequence above, including all three directory-map rebuilds. Thread Sanitizer was not run - it is unavailable for arm64 iOS devices, and the simulator, where it is available, does not currently build for x86_64. LeakSanitizer does not exist on Apple platforms. (3) is the change that would want TSan: it moves the local-manifest write from the task thread into a driver completion callback, which under the iCloud driver runs on a CloudKit thread. |
||
|
|
0b96149251 |
File I/O: mapped-view accessor, allocation-free file/buffer comparison, and the stack-safety work that fell out of it (#19330)
* file_stream/vfs: read files that report no size, and expose the mapping
Two file-I/O defects found by auditing the read paths, plus the
construct the second one was missing.
filestream_read_file() sized its allocation from
filestream_get_size() and read exactly that many bytes, so a file
reporting zero came back empty - with a *success* return,
indistinguishable from a genuinely empty file. Every procfs entry
stats as zero, so this was not a corner case: the /proc/apm battery
probe in platform_unix.c, the /proc/modules sg check in cdrom.c and
the dingux /proc/jz/battery reader all silently read nothing and
reported success. mem_stats.c reads /proc/meminfo through a raw
open() and read() into a stack buffer, which is what routing around
this looks like when the helper is not fixed.
A stat size is a hint, not a contract - synthetic filesystems have
no length until read, and a growing file's is stale the moment it
is taken - so with no usable hint, read to EOF. Verified against
real procfs: /proc/cpuinfo and /proc/meminfo now deliver their
contents where they previously delivered an empty string. A
genuinely empty file still reads empty and a missing file still
fails, which are the two cases the fix could plausibly have broken.
RETRO_VFS_FILE_ACCESS_HINT_FREQUENT_ACCESS has always mmap'd the
file, but the mapping was reachable only from inside
retro_vfs_file_read_impl(), which memcpy's out of it - so asking for
a map bought a copy of data that was already addressable, and the
point of the hint was lost. Only two call sites in the tree use it
at all. Add filestream_get_mapped_ptr() (and the _impl behind it):
a borrowed read-only view of the whole file, valid until close, NULL
whenever there is no mapping - no HAVE_MMAP, no hint, a write mode,
or a frontend-supplied VFS, which is driven entirely through the
callbacks and has no mapping this side of them. NULL is a normal
answer, so callers keep their copying path and treat this as a fast
lane.
On top of it, filestream_matches_buf(): does this file hold exactly
these bytes? It settles a size mismatch without reading, compares
in place against the map where there is one, falls back to a 64 KiB
window otherwise, stops at the first differing byte, and allocates
nothing. The window is sized to the VFS's own stdio buffer because
below that each read is staged through it - reintroducing the copy
the mapped path exists to avoid, which cost 41% in the first version
of this.
save.c's SRAM dirty-check is the first consumer. It read the whole
save file into a fresh allocation purely to memcmp and free it, and
checked the size only afterwards - so a save that had changed size,
the one case where the answer is free, still paid a full-size malloc
and a full-file read. Cores with megabytes of save RAM paid that on
every save, against the memory budget that is scarcest on the
handheld targets. Measured, unchanged/changed and peak extra heap:
128 KiB -50% / -66% 128 KiB -> 0
2 MiB -47% / -94% 2 MiB -> 0
8 MiB -49% / -99% 8 MiB -> 0
32 MiB -80% / -100% 32 MiB -> 0
The compressed lane is unchanged: rzipstream has a real header, so
it never had the size problem and has no map to compare against.
Oracle: samples/file/vfs/vfs_mapped_ptr_test, added to the samples
run allowlist, plus an explicit MMAP=0 CI lane. Both lanes are
needed and the Makefile now makes HAVE_MMAP a variable to get them:
with mappings, matches_buf compares in one memcmp and its windowed
fallback never executes, so the window-boundary cases in the default
run assert on code that did not run. The test says which lane it
took. Verified to fail with the size-0 fix reverted (procfs reads
empty), with the window advance off by one (MMAP=0 only - the mapped
lane does not notice), and with the size gate removed. The dir's
Makefile also gained the conventional SANITIZER opt-in block, absent
until now, so the workflow's SANITIZER= was being silently ignored
for both tests here.
Verified: full build and link clean; save.c compiles clean in both
the HAVE_COMPRESSION and -UHAVE_COMPRESSION lanes; griffin-shape
single-TU compile of the touched libretro-common files clean;
ASan/UBSan/LeakSan and TSan clean on both MMAP lanes; C89 -pedantic
-Wdeclaration-after-statement clean on file_stream.c (the eight
findings in vfs_implementation.c are pre-existing, identical before
and after); workflow YAML parses; the data_transfer and image-decode
oracles added earlier still pass.
Noted and not claimed: the hint gate in
retro_vfs_file_get_mapped_ptr_impl() is redundant by construction -
'mapped' is only ever set under the hint - so removing it changes
nothing observable and the oracle does not discriminate it. It is
kept to match the read and seek paths, which gate the same way.
* vfs: stop zeroing the per-open stdio buffer
Every buffered open allocated its stdio buffer with calloc, so each
one cleared 64 KiB (128 KiB on WiiU, which was already exempt by
using memalign) before handing it to setvbuf. Nothing ever reads
those bytes: stream->buf is allocated, passed to setvbuf, and freed,
and stdio fills it before it is read from. The zeroing was pure
cost.
It was also the single dearest part of opening a small file. 3000
opens of 256-byte files, the shape a library scan or a menu full of
thumbnails produces - thousands of opens, few bytes each:
filestream_read_file 3.2 us -> 1.8 us per file
open + 64-byte header 3.1 us -> 1.8 us per file
44% off, from not clearing a buffer the read never touches, and 187
MiB less allocator traffic per 3000 files. The buffer size itself
is unchanged and stays justified by the write measurements recorded
above it; this is only about how the allocation is obtained.
The regression this could introduce is stale bytes surfacing through
a read, so vfs_mapped_ptr_test grew a lane for it: alternating opens
of two files with different lengths and distinct fills, read to EOF
in 4093-byte steps so the final read of each is short and leaves the
buffer partly filled, verifying every byte and that reads past EOF
deliver nothing rather than residue.
That lane is a guard rather than a discriminator - it cannot fail on
this change, because the change is safe by construction - so it was
checked the other way round: a build that deliberately memsets the
buffer to 0xDE before setvbuf still passes every content check,
which is the positive evidence that stdio never surfaces these bytes
to a reader.
Verified: full build and link clean; griffin-shape single-TU compile
clean; ASan/UBSan/LeakSan and TSan clean on both MMAP lanes of the
oracle.
* task_cloudsync: hash through filestream, and hash the whole file
task_cloud_sync_md5_rfile() was the only place in the tree that read
a memory mapping straight out of the VFS handle - filestream_get_vfs
_handle() and then hfile->mapped - which is a task reaching past the
layer that owns the mapping into a struct it does not, and which
answers nothing at all for a frontend-supplied VFS.
filestream_get_mapped_ptr() asks the same question through
filestream, and NULL stays a normal answer, so the read loop remains
the general path.
Two things fall out of doing it that way.
The 256 KB read buffer was allocated before the branch, so every
hash taken from a mapping malloc'd and freed a quarter of a megabyte
without a byte of it being touched. It now belongs to the branch
that uses it.
More importantly the two branches did not hash the same bytes. The
mapped branch always covered the whole file; the read branch started
wherever the stream happened to be. Two of the three callers hand
this a freshly opened stream, but the third takes one from a fetch
callback, so on a platform with mappings and a non-zero position the
same content produced two different manifest entries depending on
which branch ran. A partial hash is not a useful answer to "what is
this file", so the read path is squared up with the mapped one - seek
to the start first - rather than the other way round.
The oracle's map-versus-read agreement check covered only the first
4096 bytes, which is exactly the prefix a whole-file consumer like
this would pass while still producing a different digest. It now
compares the entire file, and is verified to fail both with the
reported mapping length off by one and with the mapping base off by
one.
Verified: full build and link clean; task_cloudsync.c compiles clean
with HAVE_CLOUDSYNC and HAVE_NETWORKING; ASan/UBSan/LeakSan and TSan
clean on both MMAP lanes of the oracle; the eight C89 findings in
vfs_implementation.c are pre-existing and unchanged.
* rzip_stream/save: compare compressed saves in place too
The SRAM dirty-check got a size-first, allocation-free compare in a
recent commit, but only on the uncompressed lane - and
HAVE_COMPRESSION is the default build, so the lane that actually
ships kept decompressing the entire save into a fresh buffer to
memcmp and free it, and kept checking the size only afterwards. The
measured win applied to the minority configuration.
rzipstream_matches_buf() is the counterpart: the uncompressed size
comes from the rzip header, so a save that changed size is settled
before anything is decompressed; otherwise it decompresses a chunk
at a time into a fixed window and stops at the first difference.
There is no mapped fast path here and there cannot be - the bytes
being compared do not exist on disk in that form - so what this
avoids is the whole-file allocation rather than the copy.
save.c now uses the matching construct in each lane, and the two
lanes are one block rather than two divergent copies.
One deliberate asymmetry, found by the test rather than by
inspection: a zero-byte file never matches here, not even against
zero bytes, where filestream_matches_buf() does match it. An empty
file has no rzip header, so rzipstream_open() cannot open it and
there is nothing to compare - and false is the answer that serves
the caller, since writing then replaces it with a valid file. My
first version of the oracle asserted the filestream behaviour and
failed; the construct was right and the expectation was wrong. It
is now asserted explicitly, and documented in the header, so the
difference stays deliberate rather than surprising anyone moving a
call site between the two.
Oracle: samples/streams/rzip/rzip_matches_buf_test, added to the
samples run allowlist. rzip reads are decompressions, so the chunk
loop re-enters an inflate stream rather than walking a buffer, and
its boundaries do not line up with the compressor's chunking - a
difference on a read boundary, on an rzip chunk boundary, and
between them are three different offsets, and all three are checked,
on both compressed and uncompressed input since rzip accepts both
through the same entry point. Verified to fail with the chunk
advance off by one (2 failures) and with the size gate removed (4).
The dir's Makefile also gained the conventional SANITIZER opt-in
block, absent until now, so the workflow's SANITIZER= was being
silently ignored for rzip_chunk_size_test as well.
Verified: full build and link clean; save.c compiles clean in both
the HAVE_COMPRESSION and -UHAVE_COMPRESSION lanes; ASan/UBSan/
LeakSan and TSan clean; C89 -pedantic -Wdeclaration-after-statement
clean on rzip_stream.c; workflow YAML parses; rzip_chunk_size_test
still passes.
* rjson: release the grown buffers in rjson_parse_quick, and let CI see it
Fifteen sample Makefiles had no SANITIZER opt-in block. The samples
workflow passes SANITIZER=address,undefined to every sample
directory, so the variable was accepted and ignored and those
targets built unsanitized - while the workflow's own comment names
rbmp_test and cdrom_cuesheet_overflow_test among the tests that
"use ASan as the regression discriminator for the bound-check they
verify". Two of the tests it named were in that set: their bound
checks had nothing behind them.
Adding the block to all fifteen turned up a real leak.
rjson_parse_quick() keeps its rjson_t on the stack, but the two
buffers inside it are not on the stack: the string buffer grows onto
the heap for a long string token, and the nesting stack grows for
deep nesting. Neither was released on either return path.
rjson_free() cannot be used there because it frees the handle too,
so the buffer release is split out into _rjson_free_buffers() and
called from both.
This is not only a leak. The one caller in the tree is
netplay_room_parse.c, parsing lobby responses, so the input is a
remote server's and both growths are its choice - string tokens up
to _RJSON_MAX_SIZE, nesting up to the 50-level ceiling - leaking
per refresh at whatever size the other end picked.
Coverage for both halves, since the fix has two:
test_parser_long_string() already forced string growth and is what
LeakSanitizer caught; nothing forced stack growth, because the
inline stack holds 10 levels and every existing case is shallower.
Both of parse_quick's returns are now exercised, because both
leaked: 40 levels grows the stack and parses, 200 grows it to the
ceiling and then fails, which is the path a hostile input takes.
Verified against the reverted fix, which reports three distinct
leaks - the string and one per nesting lane.
Verified: all fifteen newly-sanitized directories build and their
tests pass under ASan/UBSan/LeakSan (28 sample binaries run clean,
including every previously-unsanitized one); full RetroArch build
and link clean; griffin-shape compile of rjson.c clean; C89
-pedantic -Wdeclaration-after-statement clean on rjson.c.
* CI: run the seven sample tests that were only ever compiled
Nine sample targets were built by the samples workflow and never
executed. Two of them should not be: rzip is a CLI tool that prints
usage and exits non-zero without arguments, and http_test performs
real HTTP requests against an external host, so running it would
make the job depend on a third party being up. The other seven are
self-contained regression tests that exit 0 on success -
data_transfer_source/prefix/window_test, audio_transfer_window_test,
rjpeg_test, r7z_lzma_props_test and vcdiff_test - and all seven pass
under ASan/UBSan/LeakSan. They are now in the run allowlist.
The reason they were lost is worth fixing separately from the fact
that they were. A target absent from RUN_TARGETS printed
"[skip-run] not in run allowlist" and carried on, which reads
identically whether the omission was deliberate or an oversight - so
adding a test to a directory that already had one, and forgetting
the allowlist, produced a test that compiled forever and ran never,
silently. That is the same failure shape as the SANITIZER blocks in
the previous commit: CI reporting coverage it did not have.
So the skip is now explicit. NORUN_TARGETS lists the two that are
deliberately not run, each with its reason, and a target in neither
list is an error naming the target and both lists. Verified by
removing vcdiff_test from the allowlist, which fails the job with
exactly one error identifying it.
Verified: the workflow's own script run against the tree with
MAKE_ARGS as CI sets it - 33 directories built, 41 targets run, zero
unclassified. The one failure in that run is retro_spsc_test timing
out at 60s, which is this single-core container rather than the
change: it passes with a longer timeout, and CI runners are not
single-core.
* vfs: define the mapped-ptr accessor on UWP, and check backend parity
The UWP build fails to link:
griffin.obj : error LNK2001: unresolved external symbol
retro_vfs_file_get_mapped_ptr_impl
vfs_implementation_uwp.cpp does not extend vfs_implementation.c, it
replaces it - the UWP project compiles the .cpp and not the .c - so
adding the accessor to the C backend left UWP without it, while
file_stream.c, which both builds share, calls it unconditionally.
Every other target linked fine.
That backend never maps a file: retro_vfs_file_open_impl() there
nulls 'mapped' and clears RETRO_VFS_FILE_ACCESS_HINT_FREQUENT_ACCESS
unconditionally. So the definition is just NULL and a zero length,
which is the documented "no mapping here" answer that every caller
already has to handle. It still has to be written out: a missing
definition is a link error, not a fallback.
Checked the rest of the surface rather than only the symbol that
broke - the two backends now agree on all 22 retro_vfs_*_impl
functions, with no other gaps.
The failure mode is worth a guard. It compiles everywhere, links
nowhere but UWP, and surfaces at the end of a long MSVC build on a
runner most contributors do not have - so the next function added to
the C backend breaks it again the same way. tools/vfs_backend_parity
.py compares the two backends by symbol and fails on anything the C
backend defines and the UWP one does not, as a source-level check
costing nothing on Linux, wired into the libretro-common tests
workflow. Verified by deleting the definition again, which reports
exactly the missing symbol.
Verified: C linkage confirmed empirically rather than assumed - the
definition compiled in a C++ TU emits an unmangled
retro_vfs_file_get_mapped_ptr_impl matching the undefined symbol
file_stream.c asks for, and a harness that mimics the UWP
arrangement (real VFS with its C definition removed, the symbol
supplied from a C++ TU) links and runs, having first reproduced the
LNK2001 without it. Full build and link clean; the
vfs_mapped_ptr_test oracle passes on both MMAP lanes; workflow YAML
parses.
* streams: size the compare windows by the stack, not by the stdio buffer
filestream_matches_buf() and rzipstream_matches_buf() each held a
64 KiB buffer on the stack. sthread_create() gives Vita threads a
0x10000 stack - 65536 bytes - and -fstack-usage puts the frame at
65648, so the function overflows the whole thread stack on its own,
before any caller frame. Both are libretro-common API, so a caller
on a spawned thread is not hypothetical, and an overflow there is
not a slow path.
4 KiB instead: 65648 -> 4208 bytes measured.
What that costs is small and lands in the right place. A window at
or above the VFS's own 64 KiB stdio buffer does read better, because
those reads go straight to the destination instead of being staged
through it - on the unchanged case, which reads everything, 4 KiB
against 64 KiB is 30.9 us vs 27.0 us on a 512 KiB file and 831 us vs
774 us on 8 MiB. Seven to fourteen per cent, and only on this
branch: where the platform maps the file the comparison does not
copy at all, so the targets that pay are the ones without mappings,
which are the same ones with the small stacks.
The comment justifying 64 KiB was also wrong to state it as
established. It generalised from an earlier whole-file-slurp
against streaming measurement, where the 41% gap was syscall count
rather than window size; window sizes were never measured until now,
and the real figure is the seven to fourteen above. The numbers in
both comments are now the measured ones.
Both oracles had their boundary cases aimed at 64 KiB, so with a
4 KiB window they were no longer testing a boundary at all - just
more mid-file checks under a boundary's name, which is the same
shape as asserting on a code path that did not run. They now target
the real window and its neighbours, with the tie to the
implementation written down so the next resize does not silently
detach them again.
Verified: full build and link clean; both oracles pass under
ASan/UBSan/LeakSan and TSan, on both MMAP lanes for the filestream
one; C89 -pedantic -Wdeclaration-after-statement clean on both TUs;
VFS backend parity check passes.
* streams: size the compare buffers for the smallest thread stack
The previous commit fixed these against the Vita stack, which is
0x10000, and stopped there. It is not the smallest: PSP and GX
threads get 8 KiB (STACKSIZE in psp_pthread.h and gx_pthread.h), and
3DS 32 KiB. A 4 KiB local is 51% of an 8 KiB stack before the
caller's frame, which is not a margin.
Both buffers are 1 KiB now. Measured with -fstack-usage, the frames
are 1136 and 1120 bytes - 14% of 8 KiB, against 51% at the previous
size and 801% at the original one.
The cost is real and lands where it should. Larger reads are
faster, and at or above the VFS's own 64 KiB stdio buffer they skip
it entirely: on the unchanged case, which reads every byte, 1 KiB
against 4 KiB against 64 KiB is 1753 / 1432 / 1153 us on an 8 MiB
file. The platforms that pay it are the ones with no memory mapping
- where there is a mapping, neither function copies at all - and
those are the same platforms with the 8 KiB stacks. It runs when a
save is written, not in a frame.
Both sizes are now named constants in the headers, and both oracles
take their boundary offsets from those names rather than repeating
the number. The boundary cases have already been aimed at a stale
value once, in the previous commit, and a boundary case pointed at
the wrong offset is not a boundary case - it is another mid-file
check under a boundary's name, which passes while testing nothing.
Verified: full build and link clean; both oracles pass under
ASan/UBSan/LeakSan and TSan, on both MMAP lanes for the filestream
one; C89 -pedantic -Wdeclaration-after-statement clean on both TUs;
VFS backend parity check passes.
Noted, not changed: the same -fstack-usage sweep across
libretro-common finds config_file_io.c's config_file_write() at
16432 bytes, twice an 8 KiB thread stack, from a fixed
char buf[0x4000] handed to setvbuf. Unlike the archive functions
that also looked large, it does not shrink on those targets - their
frames are dominated by PATH_MAX_LENGTH, which is 512 there rather
than 2048, and drop under 4 KiB once rebuilt with it. Whether that
matters depends on whether config writes ever run off the main
thread on PSP or GX, which I have not traced, so it is flagged
rather than guessed at.
* config_file: move the write buffer off the stack, and gate frame sizes
config_file_write() held its 16 KiB stdio buffer as a local, giving
it a 16432-byte frame by -fstack-usage. PSP and GX threads get 8 KiB
of stack (STACKSIZE in rthreads/psp_pthread.h and gx_pthread.h), so
that is twice the whole stack before the caller's frame.
The previous commit flagged this without tracing it. Traced now, and
it does reach a thread: input_autoconfigure_connect_handler is a task
handler, so it runs on the task thread whenever the queue is
threaded, and it calls ..._scan_config_files_external ->
..._index_write -> config_file_write. That is the path taken when a
gamepad is plugged in.
The buffer is heap now, freed after fclose because it has to outlive
every write through the stream, and skipped entirely if the
allocation fails - the C library default is used instead, as in
retro_vfs_file_open_impl(), so memory pressure costs write speed
rather than the config. Frame: 16432 -> 48 bytes.
tools/stack_budget.py measures every libretro-common translation unit
with -fstack-usage and fails on frames over 2 KiB, which leaves most
of an 8 KiB stack for callers. It builds with -DPSP deliberately:
PATH_MAX_LENGTH is 512 on those targets rather than 2048, and a
host-shaped measurement overstates anything holding a path buffer -
several archive functions look like 6 KiB on a desktop build and are
under 4 KiB there. Wired into the libretro-common tests workflow.
The 37 frames already over budget are recorded in an allowlist rather
than fixed. Almost all are codec and compression inner loops (rvp9,
rh264, raac, ropus, rflac, rzstd, rvorbis) and chd, whose working
sets are large by nature, whose call graphs have not been traced to a
thread on those targets, and several of which may not be built for
them at all. The list is unaudited and says so: this is a ratchet,
not a clean bill of health. Each entry pins the size measured today,
so an allowlisted frame that grows is still caught.
Verified against three deliberate regressions, because the first
version of this tool passed them all while measuring nothing: gcc
writes the .su file alongside the object named by -o, not alongside
the source, so every translation unit silently yielded no frames and
the check reported a clean tree. It now fails outright if nothing
was measured. With that fixed it catches a new oversized frame
(the 64 KiB compare window, 65648 bytes), this commit's own fix
reverted (16448), and an allowlisted frame grown past its recorded
size (config_file_dump at 12400 against 4208).
Verified: full build and link clean; config_file_test passes under
ASan/LeakSan, which exercises the write path; C89 -pedantic
-Wdeclaration-after-statement clean on config_file_io.c; VFS backend
parity check passes; workflow YAML parses.
* tools: correct the 8 KiB stack citation, and add call-chain analysis
The stack-budget work cited "PSP and GX threads get 8 KiB, see
STACKSIZE in psp_pthread.h and gx_pthread.h" in five places. Half of
that is wrong: nothing includes psp_pthread.h. rthreads.c reaches
for gx_pthread.h under GEKKO and ctr_pthread.h under _3DS, and PSP
falls through to plain pthreads with no stack attribute set, so the
8 KiB figure comes from GEKKO alone - with 32 KiB on 3DS and 0x10000
on Vita. The sizing decisions that figure drove are unaffected; the
attribution was not, and a comment that sends a reader to an
unreferenced header is worse than none.
The larger point is that per-frame checking cannot see the shape of
the problem. tools/stack_budget.py measures one frame at a time,
which is necessary and not sufficient: a frame runs on top of
everything that called it. config_file_write() was caught only
because its own frame held the 16 KiB; its caller,
input_autoconfigure_connect_handler, measures 1488 bytes and looks
fine. Ten 1.5 KiB frames in a row pass every per-frame check and
still overflow 8 KiB.
tools/stack_chain.py sums frames along the call graph GCC emits with
-fcallgraph-info=su and reports the deepest chain from each root,
task handlers being the roots that matter - a threaded task queue
runs them on a worker rather than the main thread.
It is deliberately not wired into CI. It is a screening tool, not a
verdict: the deepest path may be infeasible, indirect calls are
invisible so a real chain can be deeper than anything it reports, and
it only measures what the given configuration compiles. That last
one is not academic - the same roots measure 73 KiB with a host
config.h and 20 KiB with a console-shaped feature set, and neither
figure describes a platform anybody built. A check reporting numbers
that need a real target configuration to mean anything earns an
allowlist and then gets ignored.
Run against a psp1-shaped feature set, two task handlers exceed
GEKKO's 8 KiB by a wide margin, both through image decode:
task_overlay_handler at 20008 bytes and task_file_load_handler at
18568, each ending in rwebp_process_image -> alph_decode. GEKKO sets
HAVE_THREADS, so those handlers do run on the 8 KiB worker there.
Not acted on here: confirming it needs a real Wii or GameCube build
configuration and a look at whether those chains are feasible, which
is a different piece of work from this branch.
Verified: full build and link clean; stack_budget and vfs parity
checks pass; both oracles pass under ASan/UBSan/LeakSan, on both MMAP
lanes for the filestream one; C89 -pedantic
-Wdeclaration-after-statement clean on the three touched TUs;
stack_chain.py exercised against a populated call graph and against
both its error paths.
* rwebp: move the 16 KiB Huffman scratch off the stack
vh_build() held int sorted[4096] as a local - 16 KiB in one frame,
twice the 8 KiB a GEKKO thread gets (STACKSIZE in
rthreads/gx_pthread.h). It is reachable from two task handlers,
which run on a worker rather than the main thread whenever the task
queue is threaded:
task_file_load_handler -> task_image_load_handler ->
task_image_process -> image_transfer_process ->
rwebp_process_image -> ... -> vh_build
task_overlay_handler -> ... -> image_texture_load_internal ->
image_transfer_process -> rwebp_process_image -> ... -> vh_build
Wii and GameCube set HAVE_RWEBP and HAVE_THREADS both, so a WebP
thumbnail or overlay decoded on the task thread overflows the stack
there. BIG_STACK does not help: it enlarges the initial stack
through bootstrap/gx/rvl.ld, while spawned threads still come from
OSCreateThread with STACKSIZE.
The array is heap now, and uint16_t rather than int - every value in
it is a symbol index, bounded by the ns > 4096 rejection at the top,
and every read of it was already masked with 0xFFFF. The early
returns become goto fail so the allocation is released on every exit.
Measured with the real wii feature set, taken from
'make -f Makefile.griffin platform=wii -n' rather than guessed at:
vh_build frame 16784 -> 400 bytes
task_overlay_handler 19976 -> 7064 bytes worst-case chain
task_file_load_handler 18568 -> 5656 bytes worst-case chain
Every task handler chain is now under 8 KiB, but task_overlay_handler
at 7064 is 86% of it and that is not a margin. The next thing added
anywhere under image decode puts it over, and nothing in CI will say
so - tools/stack_chain.py is a diagnostic, deliberately not a gate,
for the reasons in its docstring.
Found with that tool. It is exactly the case tools/stack_budget.py
cannot see: vh_build is in libretro-common, so the per-frame check
covers it, but at 16784 it sat under the 'recorded rather than fixed'
list of codec frames that were already over budget when that check
was written - large working sets, call graphs not traced to a task
thread. This one traces.
Verified: full build and link clean; the image_decode_fuzz WebP lane
decodes identically before and after (259 attempts, 69 decoded) under
ASan/UBSan/LeakSan and TSan; image_decode_slice_test passes;
stack_budget and vfs parity checks pass; C89 -pedantic
-Wdeclaration-after-statement clean.
* stack_budget: drop the stale vh_build entry, and test the allowlist's premise
Two problems with the allowlist, one urgent.
vh_build was recorded at 16784 bytes. The previous commit moved its
scratch array to the heap and the frame is 400 now, so the entry was
silently licensing a forty-twofold regression back to where it
started. Removed - the check catches a return to a 4096-entry local
again, verified by putting one back, which reports vh_build at 8592
and exits non-zero.
The rest of the list was exempted on an assumption written down as
one: that those codec and compression frames do not reach a task
thread on the small-stack targets, and that several are not built
there at all. There is now a tool for testing that, so it has been
tested - against the feature sets those targets really use, taken
from 'make -f Makefile.griffin platform=wii -n' and Makefile.ctr
rather than guessed at:
wii (8 KiB threads, GEKKO) 2 of 37 entries reachable from a task
handler: config_file_dump and vh_build.
The other 35 are not compiled there.
ctr (32 KiB threads) 3 of 37: the same two plus
mem_stats_proc_meminfo.
So the premise holds for the codec entries on those two targets, and
the comment now says that with its basis rather than asserting it.
It is still not a general claim, and the text says so too: indirect
calls are invisible to the call-graph walk, and a target built with a
different feature set can reach further than either of these.
Worth noting alongside it: config_file_dump is reachable on both, at
4208 bytes, inside the input_autoconfigure_connect_handler chain that
totals 5800 of GEKKO's 8192. Under budget, but 71% of the stack on a
path that a gamepad being plugged in takes.
Verified: full build and link clean; the ratchet's failure path
exercised in both directions; vfs parity check passes; the WebP fuzz
lane still decodes identically (259 attempts, 69 decoded) under
ASan/UBSan/LeakSan.
* stack: fix oversized frames on principle, not on current reachability
The allowlist exempted frames on the reasoning that nothing reaches
them from a task thread, or that the codec is not built on a target
with a small stack. Both are statements about today, and both have
already been wrong once: config_file_write() was exempt on exactly
that basis until its caller turned out to be a task handler.
Platforms gain features and call graphs gain edges, so the question
for a large frame is not "can this overflow today" but "does this
need to be on the stack at all".
Two more that did not.
chd_read_header_core_file() held a whole chd_file as a local - the
struct embeds the working state of every codec it can use, huff,
zlib, cdzl and lzma - and zeroed it in full, to read a header that
touches one member. 57504 bytes -> 32. No small-stack target builds
CHD today, which is exactly the reasoning being retired here.
rd_gen_lengths() held the Huffman length generator's scratch as
locals: 2*288 nodes of weight, left, right and depth for the
two-queue tree, plus three 288-entry sort arrays. 12768 -> 352, by
moving them into struct rdeflate, which is calloc'd once per stream,
so they cost nothing extra to keep and are no longer re-established
per call.
Both entries are dropped from the allowlist, along with the stale
one this pass also turned up.
The check itself was under-measuring. Its define set was narrower
than any real build, and HAVE_7ZIP is the case that proved it:
without it, libchdr_chd.c's largest frame is 8320 bytes; with it,
57504. A seven-fold difference in shipped code, under a define that
wii, ctr and every desktop build set - so the check was measuring
code no target compiles and missing code every target does. The
define set now includes 7ZIP, CHD and the image decoders.
The allowlist comment states the rule, and records what is left as
debt rather than as exemption: mostly codec inner loops whose working
sets are large by nature, which is not the same as
necessarily-on-the-stack - most could hold their scratch in the
decoder state, which is allocated once per stream. They are recorded
because that is per-codec work with real regression risk, not because
they are safe.
Verified: full build and link clean; 12 sample tests covering
deflate, zip, zstd, PNG round-trip and decode corpus, CHD, rzip, cdfs
and 7z all pass under ASan/UBSan/LeakSan, plus the image decode fuzz
lane; stack_budget and vfs parity checks pass; the C89 findings in
both touched TUs are pre-existing and unchanged in count.
* rzstd: move the FSE compression tables into the encoder's scratch
rzstd_emit_block() held its three FSE compression tables as locals.
Each carries a 1<<accuracy_log entry table plus two 256-entry delta
arrays, so three of them are most of a 9440-byte frame - and
rzstd_encode() already owns heap scratch for the match finder, so
they belong beside it, allocated once per call rather than once per
block.
The frame is gone entirely: with the tables out of it the function is
small enough that GCC inlines it, and the largest frame left in the
translation unit is 2368 bytes.
This is the codec debt from the previous commit, taken in the order
it can be proved rather than the order of size. The vp9 entries are
larger - rvp9_build_inter_pred_hbd is 51456 - but nothing in the tree
decodes VP9 video in a test, and changing a decoder inner loop with
no way to check the output is how a silent corruption ships. zstd
has a round trip, so zstd goes first.
Proved rather than smoke-tested: the same 40 inputs - five content
shapes (ramp, random, runs, text, mixed) across eight lengths from 0
to 300000 bytes - compressed with the encoder before and after, and
the compressed bytes are byte-identical in every case, digest for
digest. Every case also decodes back to its source. That is a
stronger statement than "the tests still pass": the encoder's output
did not change at all.
Worth recording because it nearly went unnoticed: archive_zstd_test
does pass, but it covers content_size header guards and never
compresses anything, so it would not have caught an encoder
regression. Picking an entry "because it has a test" is not the same
as the test covering the thing being changed.
Verified: compressed output byte-identical across all 40 cases;
round trip correct for each; harness clean under ASan/UBSan/LeakSan;
full build and link clean; archive_zstd_test and both rzip tests
pass; C89 -pedantic -Wdeclaration-after-statement clean, unchanged
from before; stack_budget passes with the entry removed.
* lrc_hash: hash through the mapping, and get the block buffer off the stack
sha1_calculate() read the file into unsigned char buff[4096], a
local, which made it a 4304-byte frame - over half of the 8 KiB a
GEKKO thread gets (STACKSIZE in rthreads/gx_pthread.h) - to hash a
file.
It now asks for RETRO_VFS_FILE_ACCESS_HINT_FREQUENT_ACCESS and takes
filestream_get_mapped_ptr(): where the platform maps the file the
whole thing is already addressable and there is nothing to copy, so
it is one SHA1Input() call over the mapping. Where there is no
mapping - NULL is a normal answer - it falls back to the same block
loop with the buffer on the heap. Hashing a whole file dwarfs one
allocation either way. 4304 -> 224 bytes.
This is the second consumer of that accessor, after task_cloudsync's
MD5, and the same shape: mapped fast path, copying path kept as the
general case.
Proved rather than smoke-tested. Both lanes were built before and
after and their output diffed: 25 cases, three content shapes across
eight lengths chosen to straddle the 4096 block - empty, sub-block,
exact multiples, and off-by-one either side - plus a missing file
that must still be rejected.
HAVE_MMAP on digests identical before and after
HAVE_MMAP off digests identical before and after
the two lanes also agree with each other
That last line is the one worth having. The mapped path is new code
that the default build takes every time, so the copying path it
replaces would otherwise stop being exercised - and "the tests pass"
would mean the tests pass on one of the two paths.
Verified: both lanes byte-identical to the previous implementation
and to each other; both clean under ASan/UBSan/LeakSan; the existing
libcheck hash suite passes (3 checks); full build and link clean;
C89 -pedantic -Wdeclaration-after-statement unchanged at 26
pre-existing findings; stack_budget passes with the entry removed.
* file_stream: move filestream_vscanf's scan window to the heap
char buf[4096] as a local made this a 4368-byte frame, over half of
the 8 KiB a GEKKO thread gets (STACKSIZE in rthreads/gx_pthread.h).
4368 -> 272.
Shrinking the buffer would have been the cheaper change and the wrong
one: the window is how far a single conversion may reach, so a
smaller one fails differently on long input rather than merely more
slowly. The harness below pins that - with 4094 bytes of padding
before a number, the value is truncated at the window edge and only
its first digit is read, and at 4200 the conversion fails outright.
Both behaviours are preserved exactly; either would have changed
silently under a smaller buffer.
The size is now FILESTREAM_SCANF_WINDOW in the header, so a test can
drive input either side of it rather than assume a value - the same
reason the compare window below it is named.
rfscanf() has no in-tree callers; it is an fscanf() shim for cores,
which may well call it in a parse loop, so one allocation per call
rather than a per-call malloc of something larger. Beside the read
this function already performs, it is nothing.
Verified by diffing behaviour before and after across conversion
kinds and input shapes: integers, negatives, strings, literal
prefixes, field widths, hex, float, partial matches, empty input,
three successive calls on one stream, and padding either side of the
window. Return codes, parsed values and - the part that matters -
the resulting stream position are identical in every case. The
position is the thing a bug would hide in: this function seeks back
to just past what it consumed, so an off-by-one there is invisible
in the parsed output and breaks the next call.
Caught during that: the first version of this leaked the buffer on
every call. The final block has a filestream_seek() between va_end()
and return, so the pattern the free was added to did not match, and
it landed only on the early-out path. LeakSanitizer reported 9
allocations for 9 calls. The free now sits after the seek, which
needs bufiter - buf and so has to run first.
Verified: behaviour identical to the previous implementation across
every case; clean under ASan/UBSan/LeakSan; full build and link
clean; C89 -pedantic -Wdeclaration-after-statement clean, unchanged;
stack_budget, vfs parity, both MMAP lanes of vfs_mapped_ptr_test and
data_transfer_pool_scrub_test all pass.
---------
Co-authored-by: libretroadmin <>
|
||
|
|
887c0282a5 |
Migrate remaining single-word mutex/intrinsic sites to retro_atomic; fix two data races in task_cloudsync (#19326)
* task_cloudsync: fix data races on the phase and in-flight counters sync_state had two cross-thread race conditions around its async transfer bookkeeping: 1. `waiting` had an inconsistent locking protocol: every decrement performed by a cloud_sync_* completion callback took tcs_running_lock, but all three increments on the dispatch side, the dispatch-side = 0 / = 1 stores in the phase handlers, and the dispatch-failure rollbacks ran with no lock at all. A driver is free to invoke its completion callback from another thread (the threaded task queue runs a worker), at which point an unlocked ++ racing a locked -- is a data race on a plain uint32_t. 2. `phase` was written by completion callbacks (begin/manifest/end handlers) with no lock, while the task handler's poll read it - under the lock, but a lock only orders accesses that both take it, so the write side raced the poll regardless. This one is older than the counter problem and reproduces on the unpatched file: driving the pristine TU with a completion-on-another-thread driver, TSan flags begin_handler's phase store against the poll's read immediately. Both fields become retro_atomic_int_t behind small accessors: release store / acq_rel RMW on the writer side, acquire loads in the poll and the phase dispatch switch. A completion callback publishes its side effects (failures, need_manifest_uploaded, manifest-list appends) before its release phase-store/counter-drop, and the task thread's acquire loads pair with them, reproducing the ordering the old lock provided on the one path it actually covered while closing the paths it did not. The mutex is not removed: it still serialises concurrent appends to updated_server_manifest / updated_local_manifest from completion callbacks, and is renamed tcs_manifest_lock to say so. The lists are only read after the counter drains, so the read side stays lock-free. `waiting` also changes from uint32_t to int: an underflow bug now reads as a negative count (poll proceeds, bug observable) instead of wrapping to ~4 billion and wedging the sync in the waiting state forever. Verification: - Full TU builds via make (gcc 13, HAVE_THREADS) and a -std=c89 -pedantic pass. - Real-TU sanitizer sweep: this exact file linked against real libretro-common (task_queue, file_list, rjson, filestream, MD5) with only the cloud driver faked - begin/read/update/free/end each complete on their own thread after jitter. 20 iterations x 24-file thumbnail sync, full BEGIN..END phase walk with the >4 concurrent-upload DIFF gate exercised. Clean under ASan+UBSan+LSan and under TSan (3 runs). Baseline control: the unpatched TU under the same harness trips TSan on the phase race described above. - Not exercised against a live network backend; a synced run on a configured webdav build is still advisable before tagging. * opensl: give buffered_blocks real acquire/release semantics buffered_blocks was a volatile unsigned mutated with raw __sync_fetch_and_sub / __sync_fetch_and_add but *read* with plain volatile loads: the nonblock full-queue check, the cond-wait predicate in sl_write, and sl_write_avail's occupancy report all lacked acquire semantics, and the initial buf_count store was plain. This driver only ever runs on Android, i.e. always weakly-ordered ARM/AArch64, where a volatile load orders nothing. Migrate the field to retro_atomic_int_t: acq_rel fetch_sub in the engine-thread callback, acq_rel fetch_add on enqueue, acquire loads at all three read sites, release store for the initial count, and an explicit retro_atomic_int_init after the calloc (zero-fill is not a portable atomic initializer). This also removes the last raw __sync_* intrinsics in the audio drivers, matching the consolidation retro_atomic.h's header comment already claims for this file. No locking changes: sl->lock / sl->cond remain, the wait-predicate bounding logic is untouched, and the write_avail arithmetic is byte-identical with the load substituted. Compile-tested with aarch64-linux-gnu-gcc against the AOSP OpenSL ES headers: baseline and patched parse with identical warning profiles. Not runtime-tested - no Android device/NDK in this environment; needs a smoke run on device (blocking and nonblock paths) before tagging. * play_feature_delivery: lock-free Play-Store-build query play_feature_delivery_enabled() is called frequently, often in loops (its own comment says so - that is why the Java query result is cached at all), yet every call took enabled_lock just to read two bools implementing a lazy-init cache. Collapse enabled / enabled_set into a single retro_atomic_int_t holding -1 (not yet queried) / 0 / 1, statically initialised to -1. The hot path is now one acquire load and no lock. On a miss the (idempotent, side-effect-free) Java query runs and the result is release-stored; a cold race can at worst issue the query twice, with both threads storing the same value. enabled_lock is deleted from the struct, the static initialiser, init() and deinit(). status_lock is untouched: it guards a multi-field status struct including a string and stays a mutex. Compile-tested with aarch64-linux-gnu-gcc against AOSP NDK headers: baseline and patched both parse clean. Not runtime-tested - Android JNI path, no device in this environment; needs a Play Store build smoke test before tagging. * xaudio: migrate the buffers counter to retro_atomic The buffers counter was InterlockedIncrement/Decrement on the RMW side but plain volatile loads on every read: the full-queue spin in xa_write and both XAUDIO2_WRITE_AVAILABLE call sites. MSVC volatile carries acquire/release only under /volatile:ms (the x86/x64 default); on ARM64 the default is /volatile:iso, so on Windows-on-ARM the producer's reads ordered nothing against the engine thread's OnBufferEnd. Migrate to retro_atomic_int_t: acq_rel fetch_sub in both callback paths (C++ member and C vtable), acq_rel fetch_add on submit, acquire load in the full-queue spin, retro_atomic_int_init after the C-path calloc (the C++ path initializes in the mem-initializer list). The XAUDIO2_WRITE_AVAILABLE macro read the counter twice per evaluation, so the occupancy report could mix two different counter values; it is replaced by xaudio2_write_available(), which reads once with acquire and keeps the bufptr-aware arithmetic. Verification: - mingw-w64 gcc (C mode): baseline and patched both parse clean. - mingw-w64 g++ (C++ mode): baseline and patched fail with the same 8 pre-existing harness errors (mingw's xaudio2.h lacks the C-macro shims in C++ mode); none touch this change. - Pattern harness (real retro_atomic + rthreads, ring cap 8, 200k ops, read-once write_avail probed concurrently): clean under ASan+UBSan+LSan and under TSan (2 runs). Negative control: with the acquire load degraded back to a plain volatile read, TSan reports the race immediately. Not runtime-tested on Windows; needs an xaudio smoke run (blocking and nonblock) before tagging. * xaudio: include retro_atomic.h unconditionally Fixes the CI Windows MXE build breakage (i686 and x64): audio/drivers/xaudio.c:193:4: error: unknown type name 'retro_atomic_int_t' The retro_atomic.h include added by the buffers-counter migration was nested inside the HAVE_MMDEVICE / HAVE_THREADS conditional. The MXE configs define no HAVE_MMDEVICE (mmdevice_common.c is compiled via the Makefile's WASAPI grouping, but the C define is not set), so the include was dead there and retro_atomic_int_t only resolved where some other header happened to provide retro_atomic.h transitively. Every other CI environment's config-dependent header graph supplied that chain; MXE's leaner config does not. Hoist the include to the unconditional top-level include block: a header a TU's own code depends on must be included by that TU, not inherited from whatever the active config happens to drag in. Verified: the file now parses clean with i686-w64-mingw32-gcc and x86_64-w64-mingw32-gcc using a minimal define set and no config.h - i.e. with no transitive help available at all - and the full configure+make i686-mingw build compiles the object. --------- Co-authored-by: LibretroAdmin <[email protected]> |
||
|
|
1a800507a4 |
task_cloudsync: honour the manifest writer's failure
task_cloud_sync_write_updated_manifest() discarded rjsonwriter_free()'s return value. That call performs the final flush, so a short write - a full disk, an I/O error - was reported as "Wrote ..." and the function returned the open file as though it held a complete manifest. The caller then rewinds that file and uploads it as the new server manifest, or closes it as the new local one, so a truncated manifest could be published as authoritative and would fail to parse on the next sync. Return NULL on writer failure instead, which is the same signal the function already uses when the file cannot be opened - the caller handles it (logs, sets failures, ends the phase). Verified with a harness reproducing the write sequence - open READ_WRITE, write the JSON, close, parse it back - across shrink and grow cycles: 9/9 checks clean under ASan+UBSan with strict leak checking, TSan, UBSan at -O2, and MSan. Compiles clean with HAVE_CLOUDSYNC defined; C89 pedantic clean. Note for anyone reading the open flags: the file is opened READ_WRITE without RETRO_VFS_FILE_ACCESS_UPDATE_EXISTING, which discards the previous contents, so a manifest that shrinks does not inherit the tail of the longer one it replaces. The harness asserts that property, since the writer depends on it silently and adding UPDATE_EXISTING to those flags would quietly start corrupting manifests. |
||
|
|
a55e42c540 |
task_cloudsync: do not seek a NULL manifest stream
task_cloud_sync_update_manifests() calls task_cloud_sync_write_updated_manifest() for the server manifest and passes the result straight into filestream_seek(). That function returns NULL when filestream_open() fails or when rjsonwriter_open_rfile() cannot be attached, and filestream_seek() dereferences stream->hfile with no NULL check, so the sync worker takes a NULL dereference at offset 0 of filestream_seek. Observed on iOS 26.5 as EXC_BAD_ACCESS (SIGSEGV) at 0x0 with the crashing frame reported as 'filestream_seek + 0' inlined into task_cloud_sync_update_manifests, on the gcd_worker task thread. The local manifest write immediately above already guards its result with 'if (file)'. Guard the server manifest the same way and fail the task the way a rejected upload does, rather than crashing. Note that sync_state->waiting is deliberately not set before the early return: it is only raised once the upload has actually been handed to cloud_sync_update(). |
||
|
|
abb72220eb |
cloudsync: hash files in 256 KB reads, not 4 KB
The same shape as the intfstream CRC fix, found by sweeping for the pattern: the MD5 of a synced file fed a 4 KB stack buffer, which on a large savestate is tens of thousands of read calls of overhead on what the mmap fast path above it already proves is a bulk operation. A 256 KB heap buffer, allocated once per file on this background path, matches the CRC fix's shape and reasoning. |
||
|
|
50f3dd8d17 |
libretro-common/queues: add task_free_error + document set/free protocol
task_set_title() and task_set_error() take ownership of a heap pointer and rely on the task system to free it at completion, but neither setter frees the *previous* value -- so calling either one twice on the same task without an intervening free leaks the prior string. task_free_title() has long existed for the title side of this protocol; the error side had no symmetric helper at all, forcing callers that legitimately needed to update the error mid- task to either guard with task_get_error() or just live with the leak. This commit: * Adds task_free_error(), symmetric to the pre-existing task_free_title() (both lock property_lock under HAVE_THREADS, free if non-NULL, NULL out the field). * Documents the free-then-set protocol clearly in queues/task_queue.h on both setters, both struct fields, and cross-references the helpers via @see -- the previous "@warning This does not free the existing X" line was correct but easy to miss, and three sites in tasks/task_cloudsync.c missed it. * Fixes those three task_cloudsync.c violators (lines 107, 1281, 1331). task_cloudsync sets the initial title at line 1439 via direct assignment, then later replaces it via task_set_title in three callbacks without calling task_free_title first. Each replacement leaks the prior strdup. * Adds a regression test under libretro-common/samples/queues/ task_queue_title_error_test/ exercising both helpers and the free-then-set protocol (single-set, replace-via-free, free-on- null no-op, double-free no-op, 50-iteration stress chain). Wires it into the existing libretro-common-samples CI workflow, so it builds under -fsanitize=address,undefined and LSan flags any future regression to the protocol. The test compiles task_queue.c without HAVE_THREADS so the slock / scond / sthread paths drop out cleanly -- the title/error protocol is identical with and without threading; the locks just serialise it. The single remaining external dependency (cpu_features_get_time_usec) is provided as a trivial stub since no test path drives the queue through gather(). Verified pre-patch: the test fails to link without task_free_error (undefined reference), confirming it genuinely exercises the new symbol rather than passing trivially. Verified post-patch: ASan + UBSan + LSan clean, all ten cases pass. |
||
|
|
1f6c709ae6 |
tasks: extend window/taskbar progress coverage
Build on the task_window_progress_cb helper introduced in
|
||
|
|
a70ab4dc1a | Make --disable-threads work again | ||
|
|
f2c598f60a |
remove most of the remaining string_is_empty calls
also reduce calls to path_get |
||
|
|
2effea9b34 | More string_is_empty removal pt. 3 | ||
|
|
cb58dbc1fe | Use string_list_append_n where appropriate | ||
|
|
d3284ca9f3 | cloud sync: conflict resolution (#18527) | ||
|
|
151d49311d | Some CXX_BUILD buildfixes | ||
|
|
4c07e26649 | Code analysis fixes - fix some warnings and prevent garbage data | ||
|
|
b2ca569521 | Logging unification cleanups | ||
|
|
12599993c0 | apple: speed up cloud sync |