mirror of
https://github.com/libretro/RetroArch.git
synced 2026-11-07 08:36:02 -10:00
Davey HughesandLibretroAdmin
08e5b72a88
* file_path: stop at the end of the path after '~'
fill_pathname_expand_special() skipped two characters after a leading
'~' or ':', so a bare "~" read past its NUL and "~x" lost the x. It
now skips the prefix and a slash after it, if there is one.
* scan: hand the callback user data and what the scan did
task_push_dbscan() and task_push_manual_content_scan() take user data
for their callback, and the callback's task_data is now a
task_dbscan_result_t: how many entries the scan added, how many its
playlists already held, and how many files matched nothing. A file is
counted by what became of it, so an archive that is expanded is not
counted as unmatched, and a loose scan's unmatched files count as
added. Callers pass NULL for now.
* scan: say in the result when a scan failed
A scan that ends before it runs (bad setup, no valid content, a DAT
that will not load) or ends with a task error now sets the result's
failed flag. A database folder that is missing or unset is treated as
an empty one, so it reports the same "no database" error instead of
finishing as if it had scanned.
* scan: count the files a scan could not read
A file whose scan fails (an unreadable file, a failed lookup) was in
no count. The result now carries an errors count for them. They do
not fail the scan.
* scan: print --scan errors on stderr and totals on stdout
A command-line scan now ends with a totals line on stdout, or with the
reason it failed on stderr. A file it could not read is listed on
stderr. The ++/==/?? marks are coloured only when their stream is a
terminal, so pipes and redirected output get plain text; on Windows
redirected output keeps its marks instead of losing them to
WriteConsole.
* scan: the manual scan finder skips a finished task
A finished scan stays in the task queue while its callback runs, and
the finder that refuses a second scan for the same playlist still
matched it, so a scan pushed from that callback was refused.
* retroarch: run every --scan, then exit with what the scans did
Each --scan is collected while the command line is parsed and run in
turn once it is done, so later options and further --scan flags are
no longer dropped. A path is expanded and made absolute first; one
that does not exist, or a scan that cannot start, is reported on
stderr and the next path still runs. --scan with --cloudsync or
--command is refused.
Without --menu the exit status is 1 if any scan failed, else 0 if any
added an entry, else 2, and a content path given with the scans is
ignored, with a line on stderr. With --menu each finished scan starts
the next, as a second scan is refused while one runs.
* manual_content_scan: clear the database name a refresh did not save
A playlist with no saved database name left the name buffer
uninitialised, and the refresh compared it against every installed
database, so whatever was on the stack could pick a database the
playlist never named.
* scan: Skip Database References names the scan's one playlist
The option was meant to put the playlist's name in each entry's
db_name, even for a database match (#18641), for users who build
their own thumbnail sets. When the scan picks the playlists, db_name
also picks the playlist an entry goes to, so with the option on every
match went to one playlist named after the content folder.
It now applies only when the scan writes one playlist, and gives
that playlist's name, a loose scan's unmatched entries included. With
the target on <Automatic>, or Fully Automatic with the option left on
from a Custom scan, matches go to their databases' playlists as with
it off; the entries of a custom playlist look their thumbnails up
under its name, not the folder's.
* scan: name the scanned folder in the start line
The "Scanning content" line, in the log and in --scan's output, has
named the database folder since the manual scan and the database scan
were merged (2fdf736d51); before, it named the path being scanned. It
names the content folder again.
* manual_content_scan: reset, scan overrides and their apply
The command line's scan options need each scan to start from the
defaults and to lay their values over them, or over a playlist's saved
settings. Add a reset, a record of those options and a function that
applies it through the existing setters: a core named by path, file
name or display name, a database, a playlist, the database check or a
DAT, extensions and the switches, stopping with a reason at the first
that does not resolve.
A core path is normalised before it is compared. A core's file name
matches in the case given first, then ignoring case, on every
platform; a display name matches exactly. Given another database, a
playlist named after its database keeps that name as a custom one, so
a refresh still writes the same playlist.
The core setter's copy step becomes its own function, so a core found
by path or file name can be set. Nothing calls the new functions yet.
* scan: run --scan items through one chain
The --scan paths ran through a loop in retroarch.c, and with --menu
through a second chain of their own. The scan options and playlist
refreshes to come need one runner for every item, from the command
line and from the menu.
task_push_dbscan_chain() runs a list of items one task at a time: a
prep task lists the databases, then each step is pushed from the last
task's callback on the main thread, so nothing waits and nothing
locks. An item that fails is reported and the run goes on; a
cancelled task ends it. retroarch.c builds the items in the first
argument pass and hands them to the chain; without --menu it waits
for the chain's end and exits as before.
Running out of memory prints "out of memory" on stderr: for an item's
path the item fails, and for the chain itself nothing runs, which
without --menu exits 1.
The chain also prints a scan's override errors, and the warnings for
an option that needs one playlist or does nothing for a single file,
which the scan options of the next commit can trigger.
* retroarch: Manual Scan options for --scan
--scan-playlist, --scan-database, --scan-match, --scan-dat, --scan-core,
--scan-exts and the five switches set the Manual Scan settings of the
--scan before them; a --scan without any stays Fully Automatic.
Values are checked in the first pass, so an option before any --scan,
an empty or bad value, or one too long for its setting ends the run
with its reason on stderr and status 1, before the config loads and
before anything is scanned. Cores, databases and DAT files are
resolved when their scan runs, and one that does not resolve fails
only that scan.
* scan: refresh playlists in the chain
A refresh item rescans one playlist with its saved scan settings, as
the menu's Refresh Playlist does, with the item's scan options laid
over them and written back by the scan. A refresh-all item does the
same for every playlist in the playlist folder that has saved
settings, in the order Manage Playlists lists them, histories and
favorites left out.
A refresh item names its playlist by name or by a path in the playlist
folder. A bare "name.lpl" is the playlist folder's file, and an
absolute path is normalised before it is checked against the folder.
The name is looked up in the folder's listing on every platform, the
exact name first, then ignoring case, since macOS and FAT/exFAT cards
ignore case; the refresh then takes the file's own name, so a
database's playlist stays one. Two or more playlists that match only
in other cases fail the item.
The playlist folder walk and each playlist load run in prep tasks: on
the worker with a threaded queue, under the shared I/O window without
one, each pushed from the previous step's callback. A playlist whose
saved content folder is gone is skipped rather than failed. A listing
that runs out of memory fails the refresh-all; a playlist folder that
is missing or cannot be read lists nothing, and the refresh-all ends
with "no playlists with saved scan settings". A refresh's start line
names its content folder; its totals and failure lines name the
playlist.
* retroarch: --scan-refresh and --scan-refresh-all
--scan-refresh=NAME|PATH rescans a playlist with its saved scan
settings, as the menu's Refresh Playlist does, and --scan-refresh-all
rescans every playlist that has them. Both are items of the same run
as --scan: the scan options after one are laid over the saved
settings and written back, and the exit status follows the same rule.
--scan-playlist after a refresh is refused, so a refresh always
updates its own playlist, and an empty --scan-refresh is refused as
the other empty values are. Like --scan, the refresh options cannot be
combined with --cloudsync or --command; the error names whichever of
the three comes first on the command line.
* menu: Refresh All Playlists
Manage Playlists gets Refresh All Playlists as its first entry when it
lists any playlist, so an empty list still says there are no entries.
It rescans every playlist that has saved scan settings, with those
settings, through the same chain as --scan-refresh-all, and ends with
one notification saying how many playlists were refreshed, skipped
(content folder gone) and failed.
While a content scan runs it is refused with a new message; a refused
scan had none. Other languages come through Crowdin.
While a scan chain runs (Refresh All, or a command-line scan with
--menu), Start Scan, a playlist's Refresh, Scan Directory and Scan
File are refused with the same message, as they would run with the
settings the chain last loaded. task_push_dbscan() returns false then
too, which covers the companion and the iOS caller.
* docs: --scan, refresh and the Manual Scan options in the man page
--help lists the options but not what a script needs: the order the
scans run in, the exit status and what the options set. The man page
now has them, next to --cloudsync, and CHANGES.md has the entry.
* scan chain: writes from workers reach the playlist cache by note
A playlist written by a task - the chain's prep parse, a scan, a
playlist-manager task, whatever thread it runs on - leaves a note on a
lock-free stack instead of reading the cached playlist; the main
thread applies the notes when it next decides whether the cache may
stand in, so no worker reads playlist_cached while the main thread
frees or installs it. Without threads or pointer atomics a write
applies at once.
The chain's prep tasks set their handlers by name, and
thread_read_audit.py fails on a handler set from a
retro_task_handler_t parameter, which would hide every handler passed
in. The chain puts the Manual Scan settings it found back when it
ends, and is no longer running when its done callback runs, so that
callback may start another. task_push_dbscan() says why it refuses,
so the menu and the companions show the same reason, and the
companions' status line takes it from companion_core. --scan and
--scan-refresh set the verbosity override, so a config with logging
off cannot detach the console the totals go to, and refuse a path too
long for the chain's buffers. The Refresh All notification passes the
length written, and fill_pathname_expand_special() stops at the end of
a buffer shorter than the home or application directory.
Lanes: playlist_parity_test writes the cached file from a worker,
once at the same size and then while the main thread reuses and frees
the cache; database_scan_test checks the refusal message, the restored
settings and a done callback that starts the next chain;
fill_pathname_test expands into a buffer shorter than the home
directory.
* playlist: header-only parse for the scan chain's prep
The prep task needs a playlist's saved scan settings and nothing else,
yet parsed every entry and, through autofix, could write the file back
from a worker. playlist_parse_begin_header() stops at "items", rebases
the scan record in memory when the base content directory moved, and
marks the playlist so playlist_write_file() refuses it.
Lane: playlist_parity_test header_only (entries skipped, scan dir
rebased, file bytes untouched, old format).
* scan chain: ask the file system nothing on the main thread
The chain's steps ran on the main thread between its tasks and stat'd
and realpath'd there: a --scan path's resolve and stat, a
--scan-refresh path's two realpaths, the content folder's stat, the
saved and override DATs' stat and size, an override core path's
realpath. On a network share each of those can stall a frame.
- manual_content_scan_fs_read() reads them on any thread, touching
nothing but its arguments; the _fs variants of the content-folder,
playlist and override setters take its answers instead of asking.
The plain setters pass NULL and behave as before, so the menu is
unchanged.
- A --scan path gets its own prep task (dbscan_path_handler); a
--scan-refresh resolves its path inside the load task, and the load
reads the folder, the saved DAT and the overrides' paths once the
header is parsed. The worker gets copies of the item's strings.
- The prep's state is freed in one place.
Lane: database_scan_test "fs off main": six items that each stop
before their scan, on a threaded queue, with path_stat, path_is_valid,
path_is_directory, path_get_size and path_resolve_realpath wrapped:
no call on the main thread.
Not moved: the scan task's own push still stats the content folder
and the DAT (manual_content_scan_get_task_config), shared with the
menu.
* samples: the scan chain's lanes hold on Windows and without realpath
From a native Windows run of both tests:
- database_scan_test "fs off main" refreshed f_gone.lpl through
"pl/../pl/", which only a realpath folds; the sample builds without
RARCH_INTERNAL, where path_resolve_realpath() folds nothing, so the
item was rightly not in the playlist folder. It names the playlist
by its plain path now, which the check still realpaths on the
worker.
- "another case, a database's playlist" read the system name type
after the chain ended, and the chain puts the Manual Scan settings
back then. The item callback records each scan's type instead.
- playlist_parity_test's header lane expected "/moved/snes"; autofix
converts the suffix to the local form, "/moved\snes" on Windows.
- The worker lane's stress read required a core name of every read.
Windows replaces the file with MoveFileExW, and an open that meets
the replace fails: that read is an empty playlist, which has none.
---------
Co-authored-by: LibretroAdmin <[email protected]>