Files
RetroArch/samples/tasks/database/main.c
Davey HughesandLibretroAdmin 08e5b72a88 Scan from the command line: exit status, Manual Scan options, playlist refresh (#19977)
* file_path: stop at the end of the path after '~'

fill_pathname_expand_special() skipped two characters after a leading
'~' or ':', so a bare "~" read past its NUL and "~x" lost the x. It
now skips the prefix and a slash after it, if there is one.

* scan: hand the callback user data and what the scan did

task_push_dbscan() and task_push_manual_content_scan() take user data
for their callback, and the callback's task_data is now a
task_dbscan_result_t: how many entries the scan added, how many its
playlists already held, and how many files matched nothing. A file is
counted by what became of it, so an archive that is expanded is not
counted as unmatched, and a loose scan's unmatched files count as
added. Callers pass NULL for now.

* scan: say in the result when a scan failed

A scan that ends before it runs (bad setup, no valid content, a DAT
that will not load) or ends with a task error now sets the result's
failed flag. A database folder that is missing or unset is treated as
an empty one, so it reports the same "no database" error instead of
finishing as if it had scanned.

* scan: count the files a scan could not read

A file whose scan fails (an unreadable file, a failed lookup) was in
no count. The result now carries an errors count for them. They do
not fail the scan.

* scan: print --scan errors on stderr and totals on stdout

A command-line scan now ends with a totals line on stdout, or with the
reason it failed on stderr. A file it could not read is listed on
stderr. The ++/==/?? marks are coloured only when their stream is a
terminal, so pipes and redirected output get plain text; on Windows
redirected output keeps its marks instead of losing them to
WriteConsole.

* scan: the manual scan finder skips a finished task

A finished scan stays in the task queue while its callback runs, and
the finder that refuses a second scan for the same playlist still
matched it, so a scan pushed from that callback was refused.

* retroarch: run every --scan, then exit with what the scans did

Each --scan is collected while the command line is parsed and run in
turn once it is done, so later options and further --scan flags are
no longer dropped. A path is expanded and made absolute first; one
that does not exist, or a scan that cannot start, is reported on
stderr and the next path still runs. --scan with --cloudsync or
--command is refused.

Without --menu the exit status is 1 if any scan failed, else 0 if any
added an entry, else 2, and a content path given with the scans is
ignored, with a line on stderr. With --menu each finished scan starts
the next, as a second scan is refused while one runs.

* manual_content_scan: clear the database name a refresh did not save

A playlist with no saved database name left the name buffer
uninitialised, and the refresh compared it against every installed
database, so whatever was on the stack could pick a database the
playlist never named.

* scan: Skip Database References names the scan's one playlist

The option was meant to put the playlist's name in each entry's
db_name, even for a database match (#18641), for users who build
their own thumbnail sets. When the scan picks the playlists, db_name
also picks the playlist an entry goes to, so with the option on every
match went to one playlist named after the content folder.

It now applies only when the scan writes one playlist, and gives
that playlist's name, a loose scan's unmatched entries included. With
the target on <Automatic>, or Fully Automatic with the option left on
from a Custom scan, matches go to their databases' playlists as with
it off; the entries of a custom playlist look their thumbnails up
under its name, not the folder's.

* scan: name the scanned folder in the start line

The "Scanning content" line, in the log and in --scan's output, has
named the database folder since the manual scan and the database scan
were merged (2fdf736d51); before, it named the path being scanned. It
names the content folder again.

* manual_content_scan: reset, scan overrides and their apply

The command line's scan options need each scan to start from the
defaults and to lay their values over them, or over a playlist's saved
settings. Add a reset, a record of those options and a function that
applies it through the existing setters: a core named by path, file
name or display name, a database, a playlist, the database check or a
DAT, extensions and the switches, stopping with a reason at the first
that does not resolve.

A core path is normalised before it is compared. A core's file name
matches in the case given first, then ignoring case, on every
platform; a display name matches exactly. Given another database, a
playlist named after its database keeps that name as a custom one, so
a refresh still writes the same playlist.

The core setter's copy step becomes its own function, so a core found
by path or file name can be set. Nothing calls the new functions yet.

* scan: run --scan items through one chain

The --scan paths ran through a loop in retroarch.c, and with --menu
through a second chain of their own. The scan options and playlist
refreshes to come need one runner for every item, from the command
line and from the menu.

task_push_dbscan_chain() runs a list of items one task at a time: a
prep task lists the databases, then each step is pushed from the last
task's callback on the main thread, so nothing waits and nothing
locks. An item that fails is reported and the run goes on; a
cancelled task ends it. retroarch.c builds the items in the first
argument pass and hands them to the chain; without --menu it waits
for the chain's end and exits as before.

Running out of memory prints "out of memory" on stderr: for an item's
path the item fails, and for the chain itself nothing runs, which
without --menu exits 1.

The chain also prints a scan's override errors, and the warnings for
an option that needs one playlist or does nothing for a single file,
which the scan options of the next commit can trigger.

* retroarch: Manual Scan options for --scan

--scan-playlist, --scan-database, --scan-match, --scan-dat, --scan-core,
--scan-exts and the five switches set the Manual Scan settings of the
--scan before them; a --scan without any stays Fully Automatic.

Values are checked in the first pass, so an option before any --scan,
an empty or bad value, or one too long for its setting ends the run
with its reason on stderr and status 1, before the config loads and
before anything is scanned. Cores, databases and DAT files are
resolved when their scan runs, and one that does not resolve fails
only that scan.

* scan: refresh playlists in the chain

A refresh item rescans one playlist with its saved scan settings, as
the menu's Refresh Playlist does, with the item's scan options laid
over them and written back by the scan. A refresh-all item does the
same for every playlist in the playlist folder that has saved
settings, in the order Manage Playlists lists them, histories and
favorites left out.

A refresh item names its playlist by name or by a path in the playlist
folder. A bare "name.lpl" is the playlist folder's file, and an
absolute path is normalised before it is checked against the folder.
The name is looked up in the folder's listing on every platform, the
exact name first, then ignoring case, since macOS and FAT/exFAT cards
ignore case; the refresh then takes the file's own name, so a
database's playlist stays one. Two or more playlists that match only
in other cases fail the item.

The playlist folder walk and each playlist load run in prep tasks: on
the worker with a threaded queue, under the shared I/O window without
one, each pushed from the previous step's callback. A playlist whose
saved content folder is gone is skipped rather than failed. A listing
that runs out of memory fails the refresh-all; a playlist folder that
is missing or cannot be read lists nothing, and the refresh-all ends
with "no playlists with saved scan settings". A refresh's start line
names its content folder; its totals and failure lines name the
playlist.

* retroarch: --scan-refresh and --scan-refresh-all

--scan-refresh=NAME|PATH rescans a playlist with its saved scan
settings, as the menu's Refresh Playlist does, and --scan-refresh-all
rescans every playlist that has them. Both are items of the same run
as --scan: the scan options after one are laid over the saved
settings and written back, and the exit status follows the same rule.

--scan-playlist after a refresh is refused, so a refresh always
updates its own playlist, and an empty --scan-refresh is refused as
the other empty values are. Like --scan, the refresh options cannot be
combined with --cloudsync or --command; the error names whichever of
the three comes first on the command line.

* menu: Refresh All Playlists

Manage Playlists gets Refresh All Playlists as its first entry when it
lists any playlist, so an empty list still says there are no entries.
It rescans every playlist that has saved scan settings, with those
settings, through the same chain as --scan-refresh-all, and ends with
one notification saying how many playlists were refreshed, skipped
(content folder gone) and failed.

While a content scan runs it is refused with a new message; a refused
scan had none. Other languages come through Crowdin.

While a scan chain runs (Refresh All, or a command-line scan with
--menu), Start Scan, a playlist's Refresh, Scan Directory and Scan
File are refused with the same message, as they would run with the
settings the chain last loaded. task_push_dbscan() returns false then
too, which covers the companion and the iOS caller.

* docs: --scan, refresh and the Manual Scan options in the man page

--help lists the options but not what a script needs: the order the
scans run in, the exit status and what the options set. The man page
now has them, next to --cloudsync, and CHANGES.md has the entry.

* scan chain: writes from workers reach the playlist cache by note

A playlist written by a task - the chain's prep parse, a scan, a
playlist-manager task, whatever thread it runs on - leaves a note on a
lock-free stack instead of reading the cached playlist; the main
thread applies the notes when it next decides whether the cache may
stand in, so no worker reads playlist_cached while the main thread
frees or installs it. Without threads or pointer atomics a write
applies at once.

The chain's prep tasks set their handlers by name, and
thread_read_audit.py fails on a handler set from a
retro_task_handler_t parameter, which would hide every handler passed
in. The chain puts the Manual Scan settings it found back when it
ends, and is no longer running when its done callback runs, so that
callback may start another. task_push_dbscan() says why it refuses,
so the menu and the companions show the same reason, and the
companions' status line takes it from companion_core. --scan and
--scan-refresh set the verbosity override, so a config with logging
off cannot detach the console the totals go to, and refuse a path too
long for the chain's buffers. The Refresh All notification passes the
length written, and fill_pathname_expand_special() stops at the end of
a buffer shorter than the home or application directory.

Lanes: playlist_parity_test writes the cached file from a worker,
once at the same size and then while the main thread reuses and frees
the cache; database_scan_test checks the refusal message, the restored
settings and a done callback that starts the next chain;
fill_pathname_test expands into a buffer shorter than the home
directory.

* playlist: header-only parse for the scan chain's prep

The prep task needs a playlist's saved scan settings and nothing else,
yet parsed every entry and, through autofix, could write the file back
from a worker. playlist_parse_begin_header() stops at "items", rebases
the scan record in memory when the base content directory moved, and
marks the playlist so playlist_write_file() refuses it.

Lane: playlist_parity_test header_only (entries skipped, scan dir
rebased, file bytes untouched, old format).

* scan chain: ask the file system nothing on the main thread

The chain's steps ran on the main thread between its tasks and stat'd
and realpath'd there: a --scan path's resolve and stat, a
--scan-refresh path's two realpaths, the content folder's stat, the
saved and override DATs' stat and size, an override core path's
realpath.  On a network share each of those can stall a frame.

- manual_content_scan_fs_read() reads them on any thread, touching
  nothing but its arguments; the _fs variants of the content-folder,
  playlist and override setters take its answers instead of asking.
  The plain setters pass NULL and behave as before, so the menu is
  unchanged.
- A --scan path gets its own prep task (dbscan_path_handler); a
  --scan-refresh resolves its path inside the load task, and the load
  reads the folder, the saved DAT and the overrides' paths once the
  header is parsed.  The worker gets copies of the item's strings.
- The prep's state is freed in one place.

Lane: database_scan_test "fs off main": six items that each stop
before their scan, on a threaded queue, with path_stat, path_is_valid,
path_is_directory, path_get_size and path_resolve_realpath wrapped:
no call on the main thread.

Not moved: the scan task's own push still stats the content folder
and the DAT (manual_content_scan_get_task_config), shared with the
menu.

* samples: the scan chain's lanes hold on Windows and without realpath

From a native Windows run of both tests:

- database_scan_test "fs off main" refreshed f_gone.lpl through
  "pl/../pl/", which only a realpath folds; the sample builds without
  RARCH_INTERNAL, where path_resolve_realpath() folds nothing, so the
  item was rightly not in the playlist folder.  It names the playlist
  by its plain path now, which the check still realpaths on the
  worker.
- "another case, a database's playlist" read the system name type
  after the chain ended, and the chain puts the Manual Scan settings
  back then.  The item callback records each scan's type instead.
- playlist_parity_test's header lane expected "/moved/snes"; autofix
  converts the suffix to the local form, "/moved\snes" on Windows.
- The worker lane's stress read required a core name of every read.
  Windows replaces the file with MoveFileExW, and an open that meets
  the replace fails: that read is an empty playlist, which has none.

---------

Co-authored-by: LibretroAdmin <[email protected]>
2026-11-04 19:34:05 +01:00

279 lines
8.7 KiB
C

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <queues/task_queue.h>
#include <compat/strl.h>
#include <lists/dir_list.h>
#include "../../../core_info.h"
#include "../../../tasks/tasks_internal.h"
#define SCAN_TIMEOUT_SECONDS 120
#include <time.h>
#include <retro_timers.h>
#include "../../../list_special.h"
#include "../../../manual_content_scan.h"
#include "../../../configuration.h"
#include "../../../verbosity.h"
static bool loop_active = true;
/* Stubs for symbols referenced by the retroarch-tree sources we pull
* in. The real definitions live in intl/msg_hash_us.c and
* configuration.c, but those files transitively require RARCH_INTERNAL
* which drags in the entire frontend subsystem. This sample only
* exercises task_push_dbscan; none of these symbols are actually
* invoked on the path through task_push_dbscan / task_queue_check.
*
* These take enum msg_hash_enums now that configuration.h - included
* for settings_t - brings msg_hash.h with it. They used to be
* declared with int, which matched at the link level but conflicts
* once the real prototypes are visible. */
int msg_hash_get_help_us_enum(enum msg_hash_enums msg, char *s, size_t len)
{
(void)msg;
if (s && len)
s[0] = '\0';
return 0;
}
const char *msg_hash_to_str_us(enum msg_hash_enums msg)
{
(void)msg;
return "";
}
/* The string-table index builder (added by the msg_hash strtab refactor).
* With msg_hash_to_str_us() stubbed above, the index is never consulted, so
* this can be an empty stub rather than linking intl/msg_hash_us.c. */
void msg_hash_us_index_init(void)
{
}
settings_t *config_get_ptr(void)
{
/* A real settings_t, not a zeroed blob.
*
* The scan reads settings->paths.directory_playlist before it does
* anything else and refuses to start if it is empty - so a stub that
* returned zeros meant no task was ever created, the completion
* callback never fired, and this sample sat in its loop until the CI
* runner killed it. The playlist directory argument it accepts on
* the command line went nowhere.
*
* Static so it is zero-initialised; main() fills in the one field
* the scan actually consults. */
static settings_t settings;
return &settings;
}
/* Additional stubs for retroarch-core symbols referenced transitively.
* None of these are exercised on the dbscan path; they're link-time
* stubs to avoid pulling in retroarch.c, runloop.c, frontend drivers,
* and the UI/video subsystems. */
void runloop_msg_queue_push(const char *msg, size_t len,
unsigned prio, unsigned duration,
bool flush, char *title, unsigned icon, unsigned category)
{
(void)msg; (void)len; (void)prio; (void)duration;
(void)flush; (void)title; (void)icon; (void)category;
}
bool retroarch_override_setting_is_set(unsigned enum_idx, void *data)
{
(void)enum_idx; (void)data;
return false;
}
void ui_companion_driver_notify_refresh(void)
{
}
void video_display_server_set_window_progress(int progress, bool finished)
{
(void)progress; (void)finished;
}
/* task_database.c's progress_cb is now the shared task_window_progress_cb,
* whose definition lives in tasks/task_file_transfer.c. Pulling that
* file in would drag in nbio, the audio mixer, and the image-task
* machinery, none of which the dbscan path exercises. Stub it here
* to satisfy the linker; the function is never called on this code
* path because no progress_cb is invoked unless a worker thread
* publishes progress, and this sample never reaches that state. */
void task_window_progress_cb(retro_task_t *task)
{
(void)task;
}
/* dir_list_new_special lives in retroarch.c, which we cannot link
* without dragging in the world.
*
* Returning NULL here meant the scan never enumerated its databases,
* so it could not finish, and everything downstream of a completed
* scan - notably the task teardown - went unexercised. The scanner
* only asks for DIR_LIST_DATABASES, which retroarch.c answers with a
* plain "rdb" listing, so that much is worth providing. */
struct string_list *dir_list_new_special(const char *input_dir,
enum dir_list_type type, const char *filter, bool show_hidden_files)
{
(void)filter;
if (type == DIR_LIST_DATABASES)
return dir_list_new(input_dir, "rdb", false, show_hidden_files,
false, false);
return NULL;
}
static void main_msg_queue_push(retro_task_t *task,
const char *msg,
unsigned prio, unsigned duration,
bool flush)
{
(void)task;
fprintf(stderr, "MSGQ: %s\n", msg);
}
/*
* return codes -
* graceful exit: 1
* normal exit: 0
* error exit: -1
*/
static bool scan_completed = false;
static bool scan_errored = false;
static void main_db_cb(retro_task_t *task,
void *task_data, void *user_data, const char *err)
{
(void)task;
(void)task_data;
(void)user_data;
if (err && *err)
{
fprintf(stderr, "scan reported an error: %s\n", err);
scan_errored = true;
}
scan_completed = true;
loop_active = false;
}
int main(int argc, char *argv[])
{
const char *db_dir = NULL;
const char *core_info_dir = NULL;
const char *core_dir = NULL;
const char *input_dir = NULL;
const char *playlist_dir = NULL;
#if defined(_WIN32)
const char *exts = "dll";
#elif defined(__MACH__)
const char *exts = "dylib";
#else
const char *exts = "so";
#endif
if (argc < 6)
{
fprintf(stderr, "Usage: %s <database dir> <core dir> <core info dir> <input dir> <playlist dir>\n", argv[0]);
return 1;
}
db_dir = argv[1];
core_dir = argv[2];
core_info_dir = argv[3];
input_dir = argv[4];
playlist_dir = argv[5];
fprintf(stderr, "RDB database dir: %s\n", db_dir);
fprintf(stderr, "Core dir: %s\n", core_dir);
fprintf(stderr, "Core info dir: %s\n", core_info_dir);
fprintf(stderr, "Input dir: %s\n", input_dir);
fprintf(stderr, "Playlist dir: %s\n", playlist_dir);
/* The scanner traces its state through RARCH_LOG/RARCH_DBG; without
* this the sample runs blind. */
verbosity_enable();
retro_main_log_file_init(NULL, false);
#ifdef HAVE_THREADS
task_queue_init(true /* threaded enable */, main_msg_queue_push);
#else
task_queue_init(false /* threaded enable */, main_msg_queue_push);
#endif
{
/* core_info_list_new() writes through this unconditionally, so
* it cannot be NULL - passing one crashed the sample before it
* reached the scan. */
bool cache_supported = false;
core_info_init_list(core_info_dir, core_dir, exts, true, false,
&cache_supported);
}
/* The scan reads its playlist and database directories from the
* settings. Left unset, the database path is empty, the scan has
* nothing to match against and never produces a result:
*
* [Scanner] ""...
*
* is that empty path being logged. */
strlcpy(config_get_ptr()->paths.directory_playlist, playlist_dir,
sizeof(config_get_ptr()->paths.directory_playlist));
strlcpy(config_get_ptr()->paths.path_content_database, db_dir,
sizeof(config_get_ptr()->paths.path_content_database));
/* A scan needs a system name to build its playlist from. Without
* one manual_content_scan_get_task_config refuses and no task is
* created - which is what this sample used to do, silently, before
* spinning in the loop below until CI killed it. */
if (!manual_content_scan_set_menu_system_name(
MANUAL_CONTENT_SCAN_SYSTEM_NAME_CUSTOM, "SampleScan"))
{
fprintf(stderr, "could not set a system name for the scan\n");
goto done;
}
/* The return value matters: false means no task exists, so nothing
* will ever call main_db_cb and the loop below would never end. */
if (!task_push_dbscan(input_dir, main_db_cb, NULL))
{
fprintf(stderr, "task_push_dbscan refused to start a scan\n");
goto done;
}
/* Bounded, so a scanner that stalls is a reportable failure rather
* than a job that hangs until the runner times out. */
{
time_t started = time(NULL);
while (loop_active)
{
task_queue_check();
if (difftime(time(NULL), started) > SCAN_TIMEOUT_SECONDS)
{
fprintf(stderr, "scan did not finish within %d seconds\n",
SCAN_TIMEOUT_SECONDS);
break;
}
retro_sleep(1);
}
}
done:
if (!scan_completed)
fprintf(stderr, "FAILED: the scan never ran to completion\n");
else if (scan_errored)
fprintf(stderr, "FAILED: the scan completed with an error\n");
else
fprintf(stderr, "PASS: scan completed\n");
core_info_deinit_list();
task_queue_deinit();
return (scan_completed && !scan_errored) ? 0 : 1;
}