From 40a5f4e6ae47793d02799ca8bd1a2b24e240faee Mon Sep 17 00:00:00 2001 From: Davey Hughes Date: Wed, 28 Oct 2026 15:36:20 -0700 Subject: [PATCH] scan: look archives up by their first member's CRC first (#19937) * scan: look archives up by their first member's CRC first A zip, 7z, apk, rar, zst or bz2 was hashed whole before its first member's CRC was fetched. Cartridge databases key on ROM CRCs, so a one-ROM-per-zip set always matches by the member: the whole read, the main cost of scanning zips on a slow share, was wasted. The databases are now walked first for the member's CRC alone. Where it is stored (zip, 7z, RAR with CRC32) that costs one directory read; a zst, bz2 or BLAKE2 RAR member is decoded for it, as before. Only when that misses is the archive read whole, or a reader's whole result taken, and the databases walked again for its own CRC, which the arcade databases key on. The archive is expanded only after both miss. Each walk's size check uses its own size: the member's, then the archive's. By default the core gate looks in a database only when a core claims the file's extension. Stock cartridge cores do not list zip, so a cartridge zip was read whole and matched only once expanded. The first walk now also takes the databases claimed for the first member's extension. A hit there is held while the walk finishes the databases claimed for the archive's name: a match in one of those is written for the archive; otherwise the archive is expanded without the whole read. With "Scan Inside Archives" off nothing expands, so the first walk keeps to the archive's name. The parallel readers likewise start an archive with its first member's CRC, size and extension, and hand them to the scan. They hash archives whole only as far as the scan grants: a member-CRC miss grants one whole read ahead, each further miss in a row twice the last, up to 16 at a time, and a member-CRC match takes back what is left. A lone unknown cartridge then costs one wasted read, while an arcade folder soon has the readers 16 ahead. While the scan first walks a database's index in a member pass mid streak, the readers may keep 16 ahead, so the walk does not leave them idle. A grant goes first to the nearest archive ahead that has only its member CRC. Without a reader's whole result the scan hashes an archive it needs whole itself, in steps, like a file over the cutoff. An archive with no CRC for its first entry, such as a zip or 7z with a directory first, skipped every database, so its own CRC was never compared. It now goes straight to the whole walk, and readers always hash it whole. In first-match mode the walk's early exit leads to the whole walk too, while that has not run. The CRCs, sizes and walk are cleared at each file's start. A file that errored left them to the next, and an archive after it was looked up by the stale CRC. The serial is cleared with them: an archive matched after a file that errored in its serial lookup was written under that file's serial. Every database's member CRC is now checked before any whole CRC; each database used to check the whole CRC first. That differs only when a zip's first member matches a cartridge record and the whole zip an arcade record. The cartridge record now wins, written for the zip itself where the core gate lets the zip's name into the cartridge database, otherwise through the expansion. With "Scan Inside Archives" off nothing expands, so in that second case the arcade record still wins. * samples/tasks/database: write an rdb count over 127 as a uint A record count from 128 up is written as a msgpack uint32. The read-ahead lanes size their sets from the reader window, 64 files per reader, so they write 131 to 259 records; a fixint byte past 127 decodes as a map or a bool and the database is rejected on two- and three-core hosts. * scan: keep a refused crc index build refused A database whose crc index build is refused is flagged, and neither the size probe nor the CRC lookup starts another build for it. The index budget only shrinks, so a retry could only walk the database again to the same refusal, once per content file. The database scan test gains a cap on index builds and a lane that scans plain files with every build refused: one query walk each. * scan: look archives of several members up by either CRC An archive whose header lists more than one entry, an arcade set as a rule, is read whole first, and each database takes its own CRC or its first member's. A set whose first member is a cartridge ROM is written for its arcade record rather than expanded into the cartridge playlist, and an arcade folder walks no database claimed only for a member's extension. Readers read such an archive whole without a grant. The member pass stays for one-member archives, where a cartridge zip needs no whole read. file_archive_get_file_crc32_size_and_ext() reports the entry count the backends already take from the archive's header. A database claimed only for the member's extension joins the member pass only while it has or may get a crc index; unindexed, the lookup is a full walk the expansion repeats. A reader's member CRC survives a whole read that fails or is over the cutoff, so the scan reads only the archive itself. Lanes: an arcade set with a cartridge ROM first, an arcade folder beside a cartridge database, sets whose reader reads fail, and cartridge zips against an unindexed database. * archive: read a mapped zip's directory in place When the archive is mapped, zip_parse_file_init() finds the end of central directory record, reads the ZIP64 records and walks the central directory where they lie; only the context is allocated. Unmapped archives read them through the stream. The scan's member pass opens every zip for its directory, so this is its hot path. archive_zip64_test counts reads through the stream and checks that, mapped, a plain zip with a comment and a ZIP64 one give their first member's CRC without any. --------- Co-authored-by: LibretroAdmin --- libretro-common/file/archive_file.c | 13 + libretro-common/file/archive_file_zlib.c | 125 +- libretro-common/include/file/archive_file.h | 18 + .../samples/file/archive_zip64/Makefile | 4 +- .../file/archive_zip64/archive_zip64_test.c | 73 + samples/tasks/database/Makefile | 15 +- samples/tasks/database/database_scan_test.c | 2769 ++++++++++++++++- samples/tasks/database/spend_window.h | 4 +- tasks/task_database.c | 488 ++- 9 files changed, 3332 insertions(+), 177 deletions(-) diff --git a/libretro-common/file/archive_file.c b/libretro-common/file/archive_file.c index ca69a3978f..d4d3f98814 100644 --- a/libretro-common/file/archive_file.c +++ b/libretro-common/file/archive_file.c @@ -1015,6 +1015,13 @@ uint32_t file_archive_get_file_crc32(const char *path) * file found inside is used. **/ uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size) +{ + return file_archive_get_file_crc32_size_and_ext(path, size, NULL, 0, + NULL); +} + +uint32_t file_archive_get_file_crc32_size_and_ext(const char *path, + uint64_t *size, char *ext, size_t len, unsigned *entries) { file_archive_transfer_t state; struct archive_extract_userdata userdata = {0}; @@ -1080,6 +1087,8 @@ uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size) break; } + if (entries) + *entries = found ? state.step_total : 0; file_archive_parse_file_iterate_stop(&state); /* Report nothing rather than whichever entry the walk stopped on: @@ -1088,9 +1097,13 @@ uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size) if (!found) { *size = 0; + if (ext && len) + *ext = '\0'; return 0; } + if (ext && len) + strlcpy(ext, path_get_extension(userdata.current_file_path), len); *size = userdata.size; return userdata.crc; } diff --git a/libretro-common/file/archive_file_zlib.c b/libretro-common/file/archive_file_zlib.c index 9519571628..b397c33f2f 100644 --- a/libretro-common/file/archive_file_zlib.c +++ b/libretro-common/file/archive_file_zlib.c @@ -1370,9 +1370,18 @@ static int zip_parse_file_init(file_archive_transfer_t *state, const char *file) { uint8_t footer_buf[1024]; - uint8_t *footer = footer_buf; + uint8_t loc_buf[20], rec_buf[56]; + /* A mapped archive is read where it lies: the end record, the ZIP64 + * records and the central directory alike. */ +#ifdef VFS_HAVE_FILE_MAPPING + uint8_t *map = state->archive_mmap_data; +#else + uint8_t *map = NULL; +#endif + const uint8_t *footer = footer_buf; int64_t read_pos = state->archive_size; int64_t read_block = MIN(read_pos, (ssize_t)sizeof(footer_buf)); + int64_t eocd_at = 0; int64_t directory_size, directory_offset; uint64_t entries; zip_context_t *zip_context = NULL; @@ -1384,31 +1393,53 @@ static int zip_parse_file_init(file_archive_transfer_t *state, /* Find the end of central directory record by scanning * the file from the end towards the beginning. */ - for (;;) + if (map) { - if (--footer < footer_buf) + for (footer = map + (size_t)state->archive_size - 22; ; footer--) { - if (read_pos <= 0) - return -1; /* reached beginning of file */ - - /* Read 21 bytes of overlaps except on the first block. */ - if (read_pos == state->archive_size) - read_pos = read_pos - read_block; - else - read_pos = MAX(read_pos - read_block + 21, 0); - - /* Seek to read_pos and read read_block bytes. */ - filestream_seek(state->archive_file, read_pos, RETRO_VFS_SEEK_POSITION_START); - if (filestream_read(state->archive_file, footer_buf, read_block) != read_block) + if (read_le(footer, 4) == END_OF_CENTRAL_DIR_SIGNATURE) + { + unsigned comment_len = read_le(footer + 20, 2); + if ((int64_t)(footer - map) + 22 + comment_len + == state->archive_size) + break; + } + if (footer == map) return -1; - - footer = footer_buf + read_block - 22; } - if (read_le(footer, 4) == END_OF_CENTRAL_DIR_SIGNATURE) + eocd_at = (int64_t)(footer - map); + } + else + { + for (;;) { - unsigned comment_len = read_le(footer + 20, 2); - if (read_pos + (footer - footer_buf) + 22 + comment_len == state->archive_size) - break; /* found it! */ + if (--footer < footer_buf) + { + if (read_pos <= 0) + return -1; /* reached beginning of file */ + + /* Read 21 bytes of overlaps except on the first block. */ + if (read_pos == state->archive_size) + read_pos = read_pos - read_block; + else + read_pos = MAX(read_pos - read_block + 21, 0); + + /* Seek to read_pos and read read_block bytes. */ + filestream_seek(state->archive_file, read_pos, RETRO_VFS_SEEK_POSITION_START); + if (filestream_read(state->archive_file, footer_buf, read_block) != read_block) + return -1; + + footer = footer_buf + read_block - 22; + } + if (read_le(footer, 4) == END_OF_CENTRAL_DIR_SIGNATURE) + { + unsigned comment_len = read_le(footer + 20, 2); + if (read_pos + (footer - footer_buf) + 22 + comment_len == state->archive_size) + { + eocd_at = read_pos + (footer - footer_buf); + break; /* found it! */ + } + } } } @@ -1423,23 +1454,35 @@ static int zip_parse_file_init(file_archive_transfer_t *state, || directory_size == 0xFFFFFFFFu || directory_offset == 0xFFFFFFFFu) { - uint8_t loc[20], rec[56]; - int64_t at = read_pos + (footer - footer_buf); + const uint8_t *loc = loc_buf, *rec = rec_buf; + int64_t at = eocd_at; uint64_t rec_off; if (at < 20) return -1; - filestream_seek(state->archive_file, at - 20, - RETRO_VFS_SEEK_POSITION_START); - if ( filestream_read(state->archive_file, loc, 20) != 20 - || read_le(loc, 4) != ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIGNATURE) + if (map) + loc = map + (size_t)(at - 20); + else + { + filestream_seek(state->archive_file, at - 20, + RETRO_VFS_SEEK_POSITION_START); + if (filestream_read(state->archive_file, loc_buf, 20) != 20) + return -1; + } + if (read_le(loc, 4) != ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIGNATURE) return -1; rec_off = read_le64(loc + 8); if (rec_off > (uint64_t)(at - 20) || (uint64_t)(at - 20) - rec_off < 56) return -1; - filestream_seek(state->archive_file, (int64_t)rec_off, - RETRO_VFS_SEEK_POSITION_START); - if ( filestream_read(state->archive_file, rec, 56) != 56 - || read_le(rec, 4) != ZIP64_END_OF_CENTRAL_DIR_SIGNATURE) + if (map) + rec = map + (size_t)rec_off; + else + { + filestream_seek(state->archive_file, (int64_t)rec_off, + RETRO_VFS_SEEK_POSITION_START); + if (filestream_read(state->archive_file, rec_buf, 56) != 56) + return -1; + } + if (read_le(rec, 4) != ZIP64_END_OF_CENTRAL_DIR_SIGNATURE) return -1; entries = read_le64(rec + 32); if ( read_le64(rec + 40) > (uint64_t)state->archive_size @@ -1469,13 +1512,16 @@ static int zip_parse_file_init(file_archive_transfer_t *state, return -1; /* This is a ZIP file, allocate one block of memory for both the - * context and the entire directory, then read the directory. + * context and the entire directory, then read the directory; a + * mapped one is walked in place. */ - zip_context = (zip_context_t*)malloc(sizeof(zip_context_t) + (size_t)directory_size); + zip_context = (zip_context_t*)malloc(sizeof(zip_context_t) + + (map ? 0 : (size_t)directory_size)); if (!zip_context) return -1; zip_context->state = state; - zip_context->directory = (uint8_t*)(zip_context + 1); + zip_context->directory = map ? map + (size_t)directory_offset + : (uint8_t*)(zip_context + 1); zip_context->directory_entry = zip_context->directory; zip_context->directory_end = zip_context->directory + (size_t)directory_size; zip_context->zstream = NULL; @@ -1502,11 +1548,14 @@ static int zip_parse_file_init(file_archive_transfer_t *state, zip_context->csize = 0; zip_context->usize = 0; - filestream_seek(state->archive_file, directory_offset, RETRO_VFS_SEEK_POSITION_START); - if (filestream_read(state->archive_file, zip_context->directory, directory_size) != directory_size) + if (!map) { - free(zip_context); - return -1; + filestream_seek(state->archive_file, directory_offset, RETRO_VFS_SEEK_POSITION_START); + if (filestream_read(state->archive_file, zip_context->directory, directory_size) != directory_size) + { + free(zip_context); + return -1; + } } state->context = zip_context; diff --git a/libretro-common/include/file/archive_file.h b/libretro-common/include/file/archive_file.h index af1b20b4dc..4671e70b8d 100644 --- a/libretro-common/include/file/archive_file.h +++ b/libretro-common/include/file/archive_file.h @@ -325,6 +325,24 @@ uint32_t file_archive_get_file_crc32(const char *path); **/ uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size); +/** + * file_archive_get_file_crc32_size_and_ext: + * @path : filename path of archive + * @size : size of the file inside the archive + * @ext : that file's extension, or "" when + * not found; may be NULL + * @len : size of @ext in bytes + * @entries : the archive's entry count as its + * header gives it, or 0 when not + * found; may be NULL + * + * Returns: CRC32 of the specified file in the archive, otherwise 0. + * If no path within the archive is specified, the first + * file found inside is used. + **/ +uint32_t file_archive_get_file_crc32_size_and_ext(const char *path, + uint64_t *size, char *ext, size_t len, unsigned *entries); + extern const struct file_archive_file_backend zlib_backend; extern const struct file_archive_file_backend sevenzip_backend; extern const struct file_archive_file_backend zstd_backend; diff --git a/libretro-common/samples/file/archive_zip64/Makefile b/libretro-common/samples/file/archive_zip64/Makefile index 98c989b77e..31932b0398 100644 --- a/libretro-common/samples/file/archive_zip64/Makefile +++ b/libretro-common/samples/file/archive_zip64/Makefile @@ -56,8 +56,10 @@ $(SAMPLE_OBJDIR)/%.o: $$(call sample_src,$$*).c Makefile @mkdir -p $(@D) $(CC) -c -o $@ $< $(CPPFLAGS) $(CFLAGS) +# The wrap counts reads through the stream: mapped, the directory is +# read in place. $(TARGET): $(OBJS) - $(CC) -o $@ $^ $(LDFLAGS) + $(CC) -o $@ $^ $(LDFLAGS) -Wl,--wrap=filestream_read check: $(TARGET) ./$(TARGET) diff --git a/libretro-common/samples/file/archive_zip64/archive_zip64_test.c b/libretro-common/samples/file/archive_zip64/archive_zip64_test.c index 25a3de561f..ec157c29db 100644 --- a/libretro-common/samples/file/archive_zip64/archive_zip64_test.c +++ b/libretro-common/samples/file/archive_zip64/archive_zip64_test.c @@ -73,6 +73,16 @@ static unsigned failures; +/* Linked with --wrap=filestream_read: reads through the stream. */ +static unsigned stream_reads; + +int64_t __real_filestream_read(RFILE *stream, void *data, int64_t len); +int64_t __wrap_filestream_read(RFILE *stream, void *data, int64_t len) +{ + stream_reads++; + return __real_filestream_read(stream, data, len); +} + static void fail(const char *what, const char *detail) { printf("[FAIL] %s: %s\n", what, detail); @@ -328,6 +338,66 @@ static bool listed(const char *archive, unsigned n) return ok; } +/* @archive's first member's CRC and size: @crc and @n, and, mapped, + * taken from the archive where it lies, with no read through the + * stream for the end record, the ZIP64 records or the directory. */ +static void check_in_place(const char *what, const char *archive, + uint32_t crc, uint64_t n) +{ + uint64_t size = 0; + uint32_t got; + + stream_reads = 0; + got = file_archive_get_file_crc32_and_size(archive, &size); + if (got != crc || size != n) + fail(what, "first member's CRC and size"); +#ifdef HAVE_MMAP + else if (stream_reads) + fail(what, "read through the stream while mapped"); +#endif +} + +/* One stored member and an archive comment: the end record is found + * past the comment. */ +static void test_plain_in_place(void) +{ + static const uint8_t note[] = "no ZIP64 here\n"; + static uint8_t zip[512]; + struct member m; + size_t at, cd; + + m.name = "plain.txt"; + m.data = note; + m.method = 0; + m.usize = m.csize = sizeof(note) - 1; + m.big_sizes = false; + m.big_offset = false; + m.offset = 0; + m.crc = encoding_crc32(0, note, sizeof(note) - 1); + + at = local_header(zip, &m); + memcpy(zip + at, note, sizeof(note) - 1); + at += sizeof(note) - 1; + cd = at; + at += central_header(zip + at, &m); + memset(zip + at, 0, 22); + put32(zip + at, 0x06054b50u); + put16(zip + at + 8, 1); + put16(zip + at + 10, 1); + put32(zip + at + 12, (uint32_t)(at - cd)); + put32(zip + at + 16, (uint32_t)cd); + put16(zip + at + 20, 3); + memcpy(zip + at + 22, "PK!", 3); + at += 22 + 3; + + if (!write_file(ZIP_PATH, zip, at)) + fail("plain", "cannot write"); + else + check_in_place("plain, first member in place", ZIP_PATH, m.crc, + m.usize); + remove(ZIP_PATH); +} + static void test_small(void) { uint8_t *stored = (uint8_t*)malloc(STORED_LEN); @@ -382,6 +452,8 @@ static void test_small(void) fail("ZIP64", "cannot write"); if (!listed(ZIP_PATH, 3)) fail("ZIP64", "listed"); + check_in_place("ZIP64, first member in place", ZIP_PATH, m[0].crc, + STORED_LEN); check_member("ZIP64, stored", ZIP_PATH, "stored.bin", stored, STORED_LEN); check_member("ZIP64, deflated", ZIP_PATH, "deflate.bin", plain, DEFLATE_LEN); @@ -555,6 +627,7 @@ static void test_big(void) int main(void) { + test_plain_in_place(); test_small(); test_big(); if (failures) diff --git a/samples/tasks/database/Makefile b/samples/tasks/database/Makefile index 8d2d905404..bfa7b171ae 100644 --- a/samples/tasks/database/Makefile +++ b/samples/tasks/database/Makefile @@ -241,11 +241,22 @@ $(INDEX_TEST)$(EXE_EXT): $(INDEX_TEST_OBJECTS) # Builds its own databases, so it needs nothing but a writable dir. # The wraps count the scanner's database walks and parks, slow chosen # reads and tell the scan's own from its readers', spend its window, -# count and refuse reader pushes, and count queue worker starts. +# count and refuse reader pushes, count queue worker starts, count an +# archive's stored-CRC reads, fail a chosen serial's query, order the +# database and content lists, count reader passes, hold chosen lookups +# and hold a chosen index build in one step. SCAN_TEST_WRAPS := -Wl,--wrap=libretrodb_open -Wl,--wrap=task_queue_park \ -Wl,--wrap=intfstream_crc_step -Wl,--wrap=rm3u_is_m3u \ -Wl,--wrap=task_nbio_slice_open -Wl,--wrap=task_nbio_slice_close \ - -Wl,--wrap=task_queue_push + -Wl,--wrap=task_queue_push \ + -Wl,--wrap=file_archive_get_file_crc32_size_and_ext \ + -Wl,--wrap=bin_to_hex_alloc -Wl,--wrap=dir_list_new \ + -Wl,--wrap=task_queue_is_threaded \ + -Wl,--wrap=database_info_list_new_crc \ + -Wl,--wrap=database_info_index_job_new \ + -Wl,--wrap=database_info_dir_init_from_list \ + -Wl,--wrap=database_info_index_job_step \ + -Wl,--wrap=task_nbio_slice_within_budget ifeq ($(HAVE_THREADS), 1) SCAN_TEST_WRAPS += -Wl,--wrap=sthread_create endif diff --git a/samples/tasks/database/database_scan_test.c b/samples/tasks/database/database_scan_test.c index 569fc1ad78..db5fe80c4b 100644 --- a/samples/tasks/database/database_scan_test.c +++ b/samples/tasks/database/database_scan_test.c @@ -50,6 +50,7 @@ #endif #include "../../../core_info.h" +#include "../../../database_info.h" #include "../../../tasks/tasks_internal.h" #include "../../../manual_content_scan.h" #include "../../../list_special.h" @@ -215,6 +216,52 @@ struct string_list *dir_list_new_special(const char *input_dir, return NULL; } +/* Linked with --wrap: database listings in name order while db_order + * is 1, in reverse while it is -1. Set only between scans. */ +static int db_order; + +struct string_list *__real_dir_list_new(const char *dir, const char *ext, + bool include_dirs, bool include_hidden, bool include_compressed, + bool recursive); +struct string_list *__wrap_dir_list_new(const char *dir, const char *ext, + bool include_dirs, bool include_hidden, bool include_compressed, + bool recursive) +{ + struct string_list *list = __real_dir_list_new(dir, ext, include_dirs, + include_hidden, include_compressed, recursive); + size_t i; + + if (!list || !db_order || !ext || strcmp(ext, "rdb")) + return list; + dir_list_sort(list, false); + for (i = 0; db_order < 0 && i < list->size / 2; i++) + { + struct string_list_elem e; + + e = list->elems[i]; + list->elems[i] = list->elems[list->size - 1 - i]; + list->elems[list->size - 1 - i] = e; + } + return list; +} + +/* Linked with --wrap: the scan's content list in name order while + * content_sorted is set, set only between scans. */ +static bool content_sorted; + +database_info_handle_t *__real_database_info_dir_init_from_list( + enum database_type type, struct string_list *list); +database_info_handle_t *__wrap_database_info_dir_init_from_list( + enum database_type type, struct string_list *list) +{ + database_info_handle_t *db = __real_database_info_dir_init_from_list( + type, list); + + if (db && content_sorted) + dir_list_sort(list, false); + return db; +} + static int failures = 0; static int checks = 0; @@ -230,11 +277,16 @@ static void check(int ok, const char *what, const char *detail) /* Linked with --wrap=libretrodb_open. An index build opens its * database once and walks it once, so this counts the walks. */ static unsigned rdb_opens = 0; +/* Opens of databases whose path holds open_name. */ +static const char *open_name; +static unsigned named_opens; int __real_libretrodb_open(const char *path, libretrodb_t *db, bool write); int __wrap_libretrodb_open(const char *path, libretrodb_t *db, bool write) { rdb_opens++; + if (open_name && path && strstr(path, open_name)) + named_opens++; return __real_libretrodb_open(path, db, write); } @@ -242,6 +294,9 @@ int __wrap_libretrodb_open(const char *path, libretrodb_t *db, bool write) #ifndef SCAN_READERS_MAX #define SCAN_READERS_MAX 4 #endif +#ifndef SCAN_READER_AHEAD_MAX +#define SCAN_READER_AHEAD_MAX 16 +#endif /* The readers a scan on the threaded queue pushes on this host. */ static unsigned readers_expected(void) @@ -290,34 +345,80 @@ static void scan_tick_open(void) void __real_task_nbio_slice_close(nbio_budget_t *b); void __wrap_task_nbio_slice_close(nbio_budget_t *b) { - retro_atomic_store_release_size(&scan_tick_tid, 0); __real_task_nbio_slice_close(b); + retro_atomic_store_release_size(&scan_tick_tid, 0); +} + +/* Linked with --wrap: outside the scan's tick only a reader's handler + * asks, once a call. Those calls are counted from the scan's first + * whole read on (its first miss grants just before), and the count + * kept as the scan first reads a stream of watch_size bytes itself + * (-1: it read none), with the microseconds since the last call then + * (-1: no call yet). + * Whole reads are counted as each begins: the scan's, readers' and + * readers' of watch_size bytes. Reset by run_scan(). */ +static retro_atomic_int_t rd_armed; +static retro_atomic_int_t rd_passes; +static retro_atomic_int_t rd_last; +static retro_time_t rd_t0; +static int rd_watch_passes; +static int rd_watch_gap; +static retro_atomic_int_t scan_wholes; +static retro_atomic_int_t rd_wholes; +static retro_atomic_int_t rd_watch_wholes; + +bool __real_task_queue_is_threaded(void); +bool __wrap_task_queue_is_threaded(void) +{ + if ( retro_atomic_load_acquire_int(&rd_armed) + && retro_atomic_load_acquire_size(&scan_tick_tid) + != (size_t)current_tid()) + { + retro_atomic_fetch_add_int(&rd_passes, 1); + retro_atomic_store_release_int(&rd_last, + (int)(cpu_features_get_time_usec() - rd_t0)); + } + return __real_task_queue_is_threaded(); } /* A stream of slow_size bytes sleeps slow_ms before its first quantum; - * slow_in counts those sleeps begun, slow_back those ended, and - * slow_on_scan those on the thread that last ran the scan: a reader - * there holds every other task of the queue, the scan too. Reads of - * streams of watch_size bytes are counted by who makes them, a - * reader's on the main thread apart, and those of begun_size bytes as - * each begins; so are reads begun once cb_fired is set. A reader's + * slow_in counts those sleeps begun, slow_back those ended (slow_begun + * holds reader_begun as each of the first four ends), slow_by_scan + * those in the scan's tick and slow_on_scan those on the thread that + * last ran the scan: a reader there holds every other task of the + * queue, the scan too. Reads of streams of watch_size bytes are + * counted by who makes them, a reader's on the main thread apart, and + * those of begun_size bytes as each begins (reader_begun those a + * reader begins); so are reads begun once cb_fired is set. A reader's * read of a stream of fail_size bytes fails, counted in reader_fails. - * All set only between scans. */ + * The scan's first quantum of a stream of hold_size bytes waits, at + * most HOLD_USEC, until hold_want reads of begun_size bytes have begun, + * a slow one if slow_size is set and gate_n readers are in the gate + * (below). All set only between scans. */ +#define HOLD_USEC 10000000 + static uint32_t slow_size; static unsigned slow_ms; static retro_atomic_int_t slow_in; static retro_atomic_int_t slow_back; +static int slow_begun[4]; static retro_atomic_int_t slow_on_scan; +static retro_atomic_int_t slow_by_scan; static uint32_t watch_size; static retro_atomic_int_t watch_scan; static retro_atomic_int_t watch_reader; static retro_atomic_int_t watch_inline; static uint32_t begun_size; static retro_atomic_int_t reads_begun; +static retro_atomic_int_t reader_begun; static retro_atomic_int_t cb_fired; static retro_atomic_int_t reads_after_cb; static uint32_t fail_size; static retro_atomic_int_t reader_fails; +static uint32_t hold_size; +static int hold_want; +static int gate_n; +static retro_atomic_int_t gate_in; #ifdef HAVE_THREADS static uintptr_t main_tid; #endif @@ -332,19 +433,58 @@ int64_t __wrap_intfstream_crc_step(intfstream_internal_t *intf, bool scan = retro_atomic_load_acquire_size(&scan_tick_tid) == (size_t)current_tid(); + if (first && scan) + { + retro_atomic_fetch_add_int(&scan_wholes, 1); + retro_atomic_store_release_int(&rd_armed, 1); + if ( watch_size && size == (int64_t)watch_size + && rd_watch_passes < 0) + { + rd_watch_passes = retro_atomic_load_acquire_int(&rd_passes); + rd_watch_gap = -1; + if (rd_watch_passes) + rd_watch_gap = (int)(cpu_features_get_time_usec() - rd_t0) + - retro_atomic_load_acquire_int(&rd_last); + } + } + else if (first) + { + retro_atomic_fetch_add_int(&rd_wholes, 1); + if (watch_size && size == (int64_t)watch_size) + retro_atomic_fetch_add_int(&rd_watch_wholes, 1); + } if (first && retro_atomic_load_acquire_int(&cb_fired)) retro_atomic_fetch_add_int(&reads_after_cb, 1); if (slow_size && size == (int64_t)slow_size && first) { + int k; + if (retro_atomic_load_acquire_size(&scan_thread_tid) == (size_t)current_tid()) retro_atomic_fetch_add_int(&slow_on_scan, 1); + if (scan) + retro_atomic_fetch_add_int(&slow_by_scan, 1); retro_atomic_fetch_add_int(&slow_in, 1); retro_sleep(slow_ms); - retro_atomic_fetch_add_int(&slow_back, 1); + if ((k = retro_atomic_fetch_add_int(&slow_back, 1)) < 4) + slow_begun[k] = retro_atomic_load_acquire_int(&reader_begun); + } + if (hold_size && size == (int64_t)hold_size && first && scan) + { + retro_time_t until = cpu_features_get_time_usec() + HOLD_USEC; + + while ( ( retro_atomic_load_acquire_int(&reads_begun) < hold_want + || (slow_size && !retro_atomic_load_acquire_int(&slow_in)) + || retro_atomic_load_acquire_int(&gate_in) < gate_n) + && cpu_features_get_time_usec() < until) + retro_sleep(1); } if (begun_size && size == (int64_t)begun_size && first) + { retro_atomic_fetch_add_int(&reads_begun, 1); + if (!scan) + retro_atomic_fetch_add_int(&reader_begun, 1); + } if (watch_size && size == (int64_t)watch_size) { if (scan) @@ -366,11 +506,236 @@ int64_t __wrap_intfstream_crc_step(intfstream_internal_t *intf, return __real_intfstream_crc_step(intf, accumulator, max_bytes); } +/* Linked with --wrap: the scan's lookups in a CRC index. On the + * threaded queue, one of look_crc[k] (k < look_n) waits, at most + * HOLD_USEC, until look_want[k] whole reads by readers have begun, of + * watch_size bytes with look_watch, and with look_files no more than + * that many files less the scan's own. look_hits counts the holds; one + * that times out counts in look_late and ends the scan's holds; + * look_end[k] keeps the count the last of look_crc[k] ended at. All + * set only between scans. */ +#define LOOK_MAX 40 + +static uint32_t look_crc[LOOK_MAX]; +static int look_want[LOOK_MAX]; +static int look_end[LOOK_MAX]; +static unsigned look_n; +static bool look_watch; +static int look_files; +static retro_atomic_int_t look_hits; +static retro_atomic_int_t look_late; + +database_info_list_t *__real_database_info_list_new_crc( + database_info_crc_index_t *idx, const char *rdb_path, + uint32_t crc, uint32_t archive_crc, unsigned fields); +database_info_list_t *__wrap_database_info_list_new_crc( + database_info_crc_index_t *idx, const char *rdb_path, + uint32_t crc, uint32_t archive_crc, unsigned fields) +{ + unsigned k; + + for (k = 0; k < look_n; k++) + if ( crc == look_crc[k] + && !retro_atomic_load_acquire_int(&look_late) + && __real_task_queue_is_threaded()) + { + retro_atomic_int_t *n = look_watch + ? &rd_watch_wholes : &rd_wholes; + int want = look_want[k]; + retro_time_t until = cpu_features_get_time_usec() + + HOLD_USEC; + + if ( look_files + && want > look_files + - retro_atomic_load_acquire_int(&scan_wholes)) + want = look_files - retro_atomic_load_acquire_int(&scan_wholes); + retro_atomic_fetch_add_int(&look_hits, 1); + while ( retro_atomic_load_acquire_int(n) < want + && cpu_features_get_time_usec() < until) + retro_sleep(1); + if (retro_atomic_load_acquire_int(n) < want) + retro_atomic_store_release_int(&look_late, 1); + look_end[k] = retro_atomic_load_acquire_int(n); + break; + } + return __real_database_info_list_new_crc(idx, rdb_path, crc, + archive_crc, fields); +} + +/* Linked with --wrap: index builds. On the threaded queue, the build + * of a database whose path holds walk_name waits, at most HOLD_USEC, + * as it begins until walk_want whole reads by readers have begun, and + * at its first step until walk_more more have, or with walk_more 0 + * sleeps WALK_IDLE_MS, time for readers to begin any they may; until + * it ends the scan's window does not run out, so it steps in one call. + * walk_from and walk_to keep the readers' whole reads begun as it + * began and as it ended; walk_late is set by a wait that ran out. + * All set only between scans. */ +#define WALK_IDLE_MS 300 + +static const char *walk_name; +static int walk_want; +static int walk_more; +static database_info_index_job_t *walk_job; +static int walk_steps; +static int walk_from; +static int walk_to; +static bool walk_late; + +static void walk_hold(int want) +{ + retro_time_t until = cpu_features_get_time_usec() + HOLD_USEC; + + while ( retro_atomic_load_acquire_int(&rd_wholes) < want + && cpu_features_get_time_usec() < until) + retro_sleep(1); + if (retro_atomic_load_acquire_int(&rd_wholes) < want) + walk_late = true; +} + +database_info_index_job_t *__real_database_info_index_job_new( + const char *rdb_path, size_t max_bytes, unsigned want); +/* When set, the most every index build may take: 1 refuses them all, + * as a spent or too small budget does. */ +static size_t index_cap; + +database_info_index_job_t *__wrap_database_info_index_job_new( + const char *rdb_path, size_t max_bytes, unsigned want) +{ + database_info_index_job_t *job = __real_database_info_index_job_new( + rdb_path, index_cap ? index_cap : max_bytes, want); + + if ( job && walk_name && strstr(rdb_path, walk_name) + && __real_task_queue_is_threaded()) + { + walk_hold(walk_want); + walk_job = job; + walk_steps = 0; + walk_from = retro_atomic_load_acquire_int(&rd_wholes); + } + return job; +} + +bool __real_database_info_index_job_step(database_info_index_job_t *job, + unsigned records, size_t entries); +bool __wrap_database_info_index_job_step(database_info_index_job_t *job, + unsigned records, size_t entries) +{ + bool done; + + if (job == walk_job && !walk_steps++ && !walk_late) + { + if (walk_more) + walk_hold(walk_from + walk_more); + else + retro_sleep(WALK_IDLE_MS); + } + done = __real_database_info_index_job_step(job, records, entries); + if (done && job == walk_job) + { + walk_to = retro_atomic_load_acquire_int(&rd_wholes); + walk_job = NULL; + } + return done; +} + +bool __real_task_nbio_slice_within_budget(void *ud, size_t avail, + size_t len); +bool __wrap_task_nbio_slice_within_budget(void *ud, size_t avail, + size_t len) +{ + if (walk_job) + return true; + return __real_task_nbio_slice_within_budget(ud, avail, len); +} + +/* Appends the reader passes and the holds to a check's detail. */ +static void passes_detail(char *s) +{ + sprintf(s + strlen(s), "; passes %d, %d %dus before the scan's; held %d%s", + retro_atomic_load_acquire_int(&rd_passes), rd_watch_passes, + rd_watch_gap, retro_atomic_load_acquire_int(&look_hits), + retro_atomic_load_acquire_int(&look_late) ? ", late" : ""); +} + +/* Linked with --wrap: stored-CRC reads (an archive's directory) of + * paths holding stored_name, counted as each returns. A read of + * dir_hold's waits first, at most HOLD_USEC, until hold_dirs have. A + * read whose first member is gate_size bytes waits after, at most + * HOLD_USEC, in the gate: until gate_n are in and gate_slow slow + * reads have begun. One on the thread that last ran the scan, which + * gate_on_scan counts, waits only for the first: the scan cannot run. + * All set only between scans. */ +static const char *stored_name; +static retro_atomic_int_t stored_reads; +static const char *dir_hold; +static int hold_dirs; +static uint64_t gate_size; +static int gate_slow; +static retro_atomic_int_t gate_on_scan; + +uint32_t __real_file_archive_get_file_crc32_size_and_ext(const char *path, + uint64_t *size, char *ext, size_t len, unsigned *entries); +uint32_t __wrap_file_archive_get_file_crc32_size_and_ext(const char *path, + uint64_t *size, char *ext, size_t len, unsigned *entries) +{ + uint32_t crc; + + if (dir_hold && path && strstr(path, dir_hold)) + { + retro_time_t until = cpu_features_get_time_usec() + HOLD_USEC; + + while ( retro_atomic_load_acquire_int(&stored_reads) < hold_dirs + && cpu_features_get_time_usec() < until) + retro_sleep(1); + } + crc = __real_file_archive_get_file_crc32_size_and_ext(path, size, + ext, len, entries); + if (gate_n && size && *size == gate_size) + { + retro_time_t until = cpu_features_get_time_usec() + HOLD_USEC; + bool on_scan = retro_atomic_load_acquire_size(&scan_thread_tid) + == (size_t)current_tid(); + + if (on_scan) + retro_atomic_fetch_add_int(&gate_on_scan, 1); + retro_atomic_fetch_add_int(&gate_in, 1); + while ( ( retro_atomic_load_acquire_int(&gate_in) < gate_n + || ( !on_scan + && retro_atomic_load_acquire_int(&slow_in) < gate_slow)) + && cpu_features_get_time_usec() < until) + retro_sleep(1); + } + if (stored_name && path && strstr(path, stored_name)) + retro_atomic_fetch_add_int(&stored_reads, 1); + return crc; +} + +/* Linked with --wrap: the serial query fails, as on a failed + * allocation, for the serial hex_fail names, set only between scans. */ +static const char *hex_fail; +static retro_atomic_int_t hex_fails; + +char *__real_bin_to_hex_alloc(const uint8_t *data, size_t len); +char *__wrap_bin_to_hex_alloc(const uint8_t *data, size_t len) +{ + if ( hex_fail && len == strlen(hex_fail) + && !memcmp(data, hex_fail, len)) + { + retro_atomic_fetch_add_int(&hex_fails, 1); + return NULL; + } + return __real_bin_to_hex_alloc(data, len); +} + /* The last time the scanner asked rm3u_is_m3u() about each order file. * Its START step asks for every file, in content-list order, after the - * directory walk. */ -static unsigned order_stamp; -static unsigned order_last[2]; + * directory walk. wa_last does the same for whole_ahead_lane()'s and + * hit_clears_lane()'s wa_.zip, k < wa_count, not their members. */ +static unsigned order_stamp; +static unsigned order_last[2]; +static unsigned *wa_last; +static unsigned wa_count; bool __real_rm3u_is_m3u(const char *path); bool __wrap_rm3u_is_m3u(const char *path) @@ -382,6 +747,13 @@ bool __wrap_rm3u_is_m3u(const char *path) else if (strstr(path, "o_b.rd")) order_last[1] = ++order_stamp; } + if ( wa_last && path && strstr(path, "whole_ahead") + && !path_contains_compressed_file(path)) + { + unsigned k = (unsigned)strtoul(path_basename(path) + 3, NULL, 10); + if (k < wa_count) + wa_last[k] = ++order_stamp; + } return __real_rm3u_is_m3u(path); } @@ -581,7 +953,10 @@ static int write_db_set(const char *path, const db_rec_t *recs, bfixmap(&meta, 1); bfixstr(&meta, "count"); - bbyte(&meta, (uint8_t)n); + if (n < 0x80) + bbyte(&meta, (uint8_t)n); + else + buint32(&meta, n); off = 16 + (uint64_t)body.len; memcpy(hdr, "RARCHDB", 7); @@ -795,18 +1170,30 @@ static int write_cia(const char *path, uint32_t crc) return ok; } -/* A one-entry zip whose member @data (@clen octets) is stored with - * @method and stands for @len octets with crc32 @crc. Written by hand - * because the point is to exercise the scanner's archive handling, not - * to depend on a fixture checked into the tree. */ -static int write_zip_method(const char *path, const char *member, - unsigned method, const uint8_t *data, uint32_t clen, uint32_t len, - uint32_t crc) +/* A zip entry standing for @len octets with crc32 @crc: @data itself + * when @method is 0 (stored), else @data is @clen octets compressed + * with @method. A name ending in '/' is a directory: no data, and a + * stored CRC of 0. */ +typedef struct +{ + const char *name; + const uint8_t *data; + uint32_t len; + uint32_t crc; + unsigned method; + uint32_t clen; +} zip_entry_t; + +/* A zip of @n entries. Written by hand because the point is to + * exercise the scanner's archive handling, not to depend on a fixture + * checked into the tree. */ +static int write_zip_entries(const char *path, const zip_entry_t *e, + unsigned n) { FILE *f = fopen(path, "wb"); - uint16_t nlen = (uint16_t)strlen(member); uint8_t h[46]; - uint32_t local_off = 0; + uint32_t local_off = 0, dir_size = 0; + unsigned i; if (!f) return 0; @@ -814,36 +1201,51 @@ static int write_zip_method(const char *path, const char *member, #define PUT16(p, v) do { (p)[0] = (uint8_t)(v); (p)[1] = (uint8_t)((v) >> 8); } while (0) #define PUT32(p, v) do { (p)[0] = (uint8_t)(v); (p)[1] = (uint8_t)((v) >> 8); (p)[2] = (uint8_t)((v) >> 16); (p)[3] = (uint8_t)((v) >> 24); } while (0) - /* local file header */ - memset(h, 0, 30); - PUT32(h, 0x04034b50); PUT16(h + 4, 20); PUT16(h + 6, 0); - PUT16(h + 8, method); - PUT16(h + 10, 0); PUT16(h + 12, 0); - PUT32(h + 14, crc); PUT32(h + 18, clen); PUT32(h + 22, len); - PUT16(h + 26, nlen); PUT16(h + 28, 0); - fwrite(h, 1, 30, f); - fwrite(member, 1, nlen, f); - fwrite(data, 1, clen, f); + /* local file headers */ + for (i = 0; i < n; i++) + { + uint16_t nlen = (uint16_t)strlen(e[i].name); + uint32_t clen = e[i].method ? e[i].clen : e[i].len; + + memset(h, 0, 30); + PUT32(h, 0x04034b50); PUT16(h + 4, 20); PUT16(h + 6, 0); + PUT16(h + 8, e[i].method); + PUT16(h + 10, 0); PUT16(h + 12, 0); + PUT32(h + 14, e[i].crc); PUT32(h + 18, clen); PUT32(h + 22, e[i].len); + PUT16(h + 26, nlen); PUT16(h + 28, 0); + fwrite(h, 1, 30, f); + fwrite(e[i].name, 1, nlen, f); + if (clen) + fwrite(e[i].data, 1, clen, f); + } /* central directory */ - local_off = 0; - memset(h, 0, 46); - PUT32(h, 0x02014b50); PUT16(h + 4, 20); PUT16(h + 6, 20); - PUT16(h + 8, 0); PUT16(h + 10, method); - PUT16(h + 12, 0); PUT16(h + 14, 0); - PUT32(h + 16, crc); PUT32(h + 20, clen); PUT32(h + 24, len); - PUT16(h + 28, nlen); PUT16(h + 30, 0); PUT16(h + 32, 0); - PUT16(h + 34, 0); PUT16(h + 36, 0); PUT32(h + 38, 0); - PUT32(h + 42, local_off); - fwrite(h, 1, 46, f); - fwrite(member, 1, nlen, f); + for (i = 0; i < n; i++) + { + uint16_t nlen = (uint16_t)strlen(e[i].name); + uint32_t clen = e[i].method ? e[i].clen : e[i].len; + + memset(h, 0, 46); + PUT32(h, 0x02014b50); PUT16(h + 4, 20); PUT16(h + 6, 20); + PUT16(h + 8, 0); PUT16(h + 10, e[i].method); + PUT16(h + 12, 0); PUT16(h + 14, 0); + PUT32(h + 16, e[i].crc); PUT32(h + 20, clen); PUT32(h + 24, e[i].len); + PUT16(h + 28, nlen); PUT16(h + 30, 0); PUT16(h + 32, 0); + PUT16(h + 34, 0); PUT16(h + 36, 0); + PUT32(h + 38, e[i].name[nlen - 1] == '/' ? 0x10 : 0); + PUT32(h + 42, local_off); + fwrite(h, 1, 46, f); + fwrite(e[i].name, 1, nlen, f); + local_off += 30 + nlen + clen; + dir_size += 46 + nlen; + } /* end of central directory */ memset(h, 0, 22); PUT32(h, 0x06054b50); - PUT16(h + 8, 1); PUT16(h + 10, 1); - PUT32(h + 12, 46 + nlen); - PUT32(h + 16, 30 + nlen + clen); + PUT16(h + 8, n); PUT16(h + 10, n); + PUT32(h + 12, dir_size); + PUT32(h + 16, local_off); fwrite(h, 1, 22, f); fclose(f); #undef PUT16 @@ -851,6 +1253,23 @@ static int write_zip_method(const char *path, const char *member, return 1; } +/* A one-entry zip whose member @data (@clen octets) is stored with + * @method and stands for @len octets with crc32 @crc. */ +static int write_zip_method(const char *path, const char *member, + unsigned method, const uint8_t *data, uint32_t clen, uint32_t len, + uint32_t crc) +{ + zip_entry_t e; + + e.name = member; + e.data = data; + e.len = len; + e.crc = crc; + e.method = method; + e.clen = clen; + return write_zip_entries(path, &e, 1); +} + /* The same with the member stored as it is */ static int write_zip(const char *path, const char *member, const uint8_t *data, uint32_t len, uint32_t crc) @@ -868,6 +1287,86 @@ static int write_bytes(const char *path, const uint8_t *d, size_t n) return (fclose(f) == 0) && ok; } +#ifdef HAVE_7ZIP +/* A 7z NUMBER below 0x4000. */ +static unsigned put_7z_num(uint8_t *p, uint32_t v) +{ + if (v < 0x80) + { + p[0] = (uint8_t)v; + return 1; + } + p[0] = (uint8_t)(0x80 | (v >> 8)); + p[1] = (uint8_t)v; + return 2; +} + +/* A 7z of one member stored by the copy coder, its CRC in the header + * as a zip's is in its directory. */ +static int write_7z(const char *path, const char *member, + const uint8_t *data, uint32_t len, uint32_t crc) +{ + static const uint8_t sig[8] = { '7', 'z', 0xBC, 0xAF, 0x27, 0x1C, + 0, 4 }; + uint8_t s[32], h[128]; + unsigned n = 0; + size_t i; + FILE *f; + int ok; + + h[n++] = 0x01; /* header */ + h[n++] = 0x04; /* streams */ + h[n++] = 0x06; /* one packed stream, at 0 */ + h[n++] = 0x00; + h[n++] = 0x01; + h[n++] = 0x09; + n += put_7z_num(h + n, len); + h[n++] = 0x00; + h[n++] = 0x07; /* one folder of one copy coder */ + h[n++] = 0x0B; + h[n++] = 0x01; + h[n++] = 0x00; + h[n++] = 0x01; + h[n++] = 0x01; + h[n++] = 0x00; + h[n++] = 0x0C; + n += put_7z_num(h + n, len); + h[n++] = 0x00; + h[n++] = 0x08; /* its CRC */ + h[n++] = 0x0A; + h[n++] = 0x01; + put_le32(h + n, crc); + n += 4; + h[n++] = 0x00; + h[n++] = 0x00; + h[n++] = 0x05; /* one file, its name UTF-16 */ + h[n++] = 0x01; + h[n++] = 0x11; + n += put_7z_num(h + n, (uint32_t)(1 + 2 * (strlen(member) + 1))); + h[n++] = 0x00; + for (i = 0; i <= strlen(member); i++) + { + h[n++] = (uint8_t)member[i]; + h[n++] = 0; + } + h[n++] = 0x00; + h[n++] = 0x00; + + memset(s, 0, sizeof(s)); + memcpy(s, sig, sizeof(sig)); + put_le32(s + 12, len); /* next header: offset, size, CRC */ + put_le32(s + 20, n); + put_le32(s + 28, crc_of(h, n)); + put_le32(s + 8, crc_of(s + 12, 20)); + if (!(f = fopen(path, "wb"))) + return 0; + ok = fwrite(s, 1, 32, f) == 32 + && fwrite(data, 1, len, f) == len + && fwrite(h, 1, n, f) == n; + return (fclose(f) == 0) && ok; +} +#endif + static int file_contains(const char *path, const char *needle) { int64_t len = 0; @@ -913,6 +1412,17 @@ static bool run_scan(const char *dir) loop_active = true; scan_completed = false; + retro_atomic_store_release_int(&rd_armed, 0); + retro_atomic_store_release_int(&rd_passes, 0); + retro_atomic_store_release_int(&rd_last, 0); + rd_t0 = cpu_features_get_time_usec(); + rd_watch_passes = -1; + rd_watch_gap = 0; + retro_atomic_store_release_int(&scan_wholes, 0); + retro_atomic_store_release_int(&rd_wholes, 0); + retro_atomic_store_release_int(&rd_watch_wholes, 0); + retro_atomic_store_release_int(&look_hits, 0); + retro_atomic_store_release_int(&look_late, 0); if (!task_push_dbscan(dir, scan_cb)) return false; @@ -1995,7 +2505,7 @@ static void set_rec(db_rec_t *rec, const char *name, uint32_t crc, #define MIX_BIG_TWO 131072 /* Two files over the cutoff, an archive matched by its own CRC, one - * expanded into its member, and a cue that prunes its track. */ + * matched by its member's stored CRC, and a cue that prunes its track. */ static void mix_lane(const char *root, const char *pl_dir, bool threaded) { char db[512], dir[512], p[1024], lpl[1024], what[64]; @@ -2212,6 +2722,2089 @@ fixture: check(0, "fixture", "could not write the archives"); } +#define MF_MEMBER_LEN 1024 + +/* @len bytes of a pattern from @seed, brought to crc32 @crc. */ +static void fill_forced(uint8_t *d, uint32_t len, unsigned seed, + uint32_t crc) +{ + uint32_t i; + + for (i = 0; i < len - 4; i++) + d[i] = (uint8_t)((i * seed + 3) & 0xFF); + crc_force(d, len - 4, crc, d + len - 4); +} + +/* @rec named @name, keyed by the crc32 and size of the file at @path. */ +static bool set_whole_rec(db_rec_t *rec, const char *name, + const char *path) +{ + uint8_t *d = NULL; + int64_t len = 0; + + if (!filestream_read_file(path, (void**)&d, &len) || !d) + return false; + set_rec(rec, name, crc_of(d, (size_t)len), (uint32_t)len); + free(d); + return true; +} + +/* An archive is looked up by its first member's stored CRC, then by + * its own, and expanded only when both miss; a stored CRC of 0 (a + * directory first) goes straight to its own. On the regular queue + * nothing reads ahead, so a member match reads no archive whole. */ +static void member_first_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static uint8_t d[4][MF_MEMBER_LEN]; + char db[512], dir[512], p[1024], cart_lpl[1024]; + char arcade_lpl[1024], what[64], detail[64], crc[2][16]; + db_rec_t cart[2], arcade[2]; + zip_entry_t e[2]; + uint32_t zip_size = 0; + int reads, stored; + bool started; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db, "%s/member_first_db", root); + sprintf(dir, "%s/member_first", root); + path_mkdir(db); + path_mkdir(dir); + + /* Matched by its member; its own CRC matches nothing. */ + fill_forced(d[0], MF_MEMBER_LEN, 41, 0x0BF10001u); + memset(e, 0, sizeof(e)); + e[0].name = "member.rd"; + e[0].data = d[0]; + e[0].len = MF_MEMBER_LEN; + e[0].crc = 0x0BF10001u; + sprintf(p, "%s/mf_member.zip", dir); + if (!write_zip_entries(p, e, 1)) + goto fixture; + zip_size = (uint32_t)path_get_size(p); + set_rec(&cart[0], "MF Member", 0x0BF10001u, MF_MEMBER_LEN); + + /* Its first member matches nothing, its own CRC does: not expanded, + * though its second member would match. */ + fill_forced(d[1], MF_MEMBER_LEN, 43, 0x0BF100A1u); + e[0].name = "a1.rd"; + e[0].data = d[1]; + e[0].crc = 0x0BF100A1u; + e[1].name = "a2.rd"; + e[1].data = d[0]; + e[1].len = MF_MEMBER_LEN; + e[1].crc = 0x0BF10001u; + sprintf(p, "%s/mf_arcade.zip", dir); + if ( !write_zip_entries(p, e, 2) + || !set_whole_rec(&arcade[0], "MF Arcade", p)) + goto fixture; + + /* A directory first: no stored CRC, so only its own CRC can match. */ + e[0].name = "d/"; + e[0].data = NULL; + e[0].len = 0; + e[0].crc = 0; + e[1].name = "d/x.rd"; + e[1].data = d[1]; + e[1].crc = 0x0BF100A1u; + sprintf(p, "%s/mf_dirfirst.zip", dir); + if ( !write_zip_entries(p, e, 2) + || !set_whole_rec(&arcade[1], "MF Dir First", p)) + goto fixture; + + /* Neither its first member nor its own CRC matches: expanded, its + * second member does. */ + fill_forced(d[2], MF_MEMBER_LEN, 47, 0x0BF100A2u); + fill_forced(d[3], MF_MEMBER_LEN, 53, 0x0BF10003u); + e[0].name = "a.rd"; + e[0].data = d[2]; + e[0].len = MF_MEMBER_LEN; + e[0].crc = 0x0BF100A2u; + e[1].name = "b.rd"; + e[1].data = d[3]; + e[1].crc = 0x0BF10003u; + sprintf(p, "%s/mf_multi.zip", dir); + if (!write_zip_entries(p, e, 2)) + goto fixture; + set_rec(&cart[1], "MF Multi B", 0x0BF10003u, MF_MEMBER_LEN); + + sprintf(p, "%s/Test MF Cart.rdb", db); + if (!write_db_set(p, cart, 2)) + goto fixture; + sprintf(p, "%s/Test MF Arcade.rdb", db); + if (!write_db_set(p, arcade, 2)) + goto fixture; + + sprintf(cart_lpl, "%s/Test MF Cart.lpl", pl_dir); + sprintf(arcade_lpl, "%s/Test MF Arcade.lpl", pl_dir); + filestream_delete(cart_lpl); + filestream_delete(arcade_lpl); + use_queue(threaded); + use_db(db); + retro_atomic_store_release_int(&reads_begun, 0); + retro_atomic_store_release_int(&stored_reads, 0); + begun_size = zip_size; + stored_name = "mf_dirfirst.zip"; + started = run_scan(dir) && scan_completed; + begun_size = 0; + stored_name = NULL; + reads = retro_atomic_load_acquire_int(&reads_begun); + stored = retro_atomic_load_acquire_int(&stored_reads); + + sprintf(crc[0], "%08X|crc", (unsigned)cart[0].crc); + sprintf(what, "archive matched by its member's CRC %s", q); + check( started && file_contains(cart_lpl, "MF Member") + && file_contains(cart_lpl, crc[0]), what, "mf_member.zip"); + if (!threaded) + { + sprintf(detail, "%d whole reads", reads); + check(started && reads == 0, "a member match reads no archive whole", + detail); + } + sprintf(crc[1], "%08X|crc", (unsigned)arcade[0].crc); + sprintf(what, "archive matched by its own CRC %s", q); + check( started && file_contains(arcade_lpl, "MF Arcade") + && file_contains(arcade_lpl, crc[1]) + && !file_contains(cart_lpl, "mf_arcade.zip#"), what, + "mf_arcade.zip, not expanded"); + sprintf(what, "no stored CRC: matched by its own %s", q); + check(started && file_contains(arcade_lpl, "MF Dir First"), what, + "mf_dirfirst.zip"); + sprintf(detail, "%d directory reads", stored); + sprintf(what, "no stored CRC: its directory read once %s", q); + check(started && stored == 1, what, detail); + sprintf(what, "expanded after both CRCs miss %s", q); + check( started && file_contains(cart_lpl, "MF Multi B") + && file_contains(cart_lpl, "mf_multi.zip#b.rd"), what, + "mf_multi.zip#b.rd"); + return; + +fixture: + check(0, "fixture", "could not write the member-first set"); +} + +#define GATE_ZIPS 4 +/* One member named rom.<2 letters>; two named .ar. */ +#define GATE_ZIP_LEN (30 + 6 + MF_MEMBER_LEN + 46 + 6 + 22) +#define ARC_ZIP_LEN (2 * (30 + 4 + MF_MEMBER_LEN + 46 + 4) + 22) + +/* @dir/@file of one member, @member, forced to @crc, and its record + * @rec, named @name. */ +static bool write_gate_zip(const char *dir, const char *file, + const char *member, const char *name, uint32_t crc, db_rec_t *rec) +{ + static uint8_t d[MF_MEMBER_LEN]; + char p[1024]; + + fill_forced(d, MF_MEMBER_LEN, 59 + (crc & 0xF), crc); + sprintf(p, "%s/%s", dir, file); + set_rec(rec, name, crc, MF_MEMBER_LEN); + return write_zip(p, member, d, MF_MEMBER_LEN, crc) != 0; +} + +/* Cartridge zips whose database's core lists the member's extension + * but not zip, with "Scan without core match" off: matched by the + * member without a whole read, and written as x.zip#member, as when + * expanded. */ +static void cart_gate_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static const char *const file[GATE_ZIPS] = { + "cg0.zip", "cg1.zip", "cg2.zip", "cg3.zip" }; + static const char *const name[GATE_ZIPS] = { + "Gate CG0", "Gate CG1", "Gate CG2", "Gate CG3" }; + char db[512], dir[512], p[1024], lpl[1024]; + char what[64], detail[64], want[32]; + db_rec_t rec[GATE_ZIPS]; + unsigned k; + int reads; + bool started, all = true; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db, "%s/cart_gate_db", root); + sprintf(dir, "%s/cart_gate", root); + path_mkdir(db); + path_mkdir(dir); + for (k = 0; k < GATE_ZIPS; k++) + if (!write_gate_zip(dir, file[k], "rom.ct", name[k], + 0x0CA70000u + k, &rec[k])) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, rec, GATE_ZIPS)) + goto fixture; + + sprintf(lpl, "%s/Test Cart Gate.lpl", pl_dir); + filestream_delete(lpl); + use_queue(threaded); + use_db(db); + retro_atomic_store_release_int(&reads_begun, 0); + begun_size = GATE_ZIP_LEN; + started = run_scan(dir) && scan_completed; + begun_size = 0; + reads = retro_atomic_load_acquire_int(&reads_begun); + + for (k = 0; k < GATE_ZIPS; k++) + { + sprintf(want, "%s#rom.ct", file[k]); + sprintf(p, "%08X|crc", (unsigned)rec[k].crc); + all = all && file_contains(lpl, want) && file_contains(lpl, p); + } + sprintf(what, "cart zip: matched as x.zip#member %s", q); + check(started && all, what, "cg0.zip#rom.ct to cg3.zip#rom.ct"); + sprintf(what, "cart zip: no archive read whole %s", q); + sprintf(detail, "%d whole reads", reads); + check(started && reads == 0, what, detail); + return; + +fixture: + check(0, "fixture", "could not write the cart gate set"); +} + +/* Where the database takes the archive's own name, with "Scan without + * core match" on or a core listing zip claiming it, a member match is + * written for x.zip, unexpanded and not read whole, whether or not + * that core lists the member's extension. */ +static void base_paths_lane(const char *root, const char *pl_dir, + bool threaded) +{ + char db[2][512], dir[2][512], p[1024], lpl[2][1024]; + char what[64], detail[64], crc[3][16]; + db_rec_t rec[3]; + unsigned k; + int reads[2]; + bool started[2]; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db[0], "%s/base_on_db", root); + sprintf(dir[0], "%s/base_on", root); + sprintf(db[1], "%s/base_zip_db", root); + sprintf(dir[1], "%s/base_zip", root); + for (k = 0; k < 2; k++) + { + path_mkdir(db[k]); + path_mkdir(dir[k]); + } + if ( !write_gate_zip(dir[0], "bp0.zip", "rom.ct", "Gate BP0", + 0x0CA70010u, &rec[0]) + || !write_gate_zip(dir[1], "bz0.zip", "rom.cz", "Gate BZ0", + 0x0CA70011u, &rec[1]) + || !write_gate_zip(dir[1], "bz1.zip", "rom.cy", "Gate BZ1", + 0x0CA70012u, &rec[2])) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db[0]); + if (!write_db_set(p, rec, 1)) + goto fixture; + sprintf(p, "%s/Test Cart Zip.rdb", db[1]); + if (!write_db_set(p, rec + 1, 2)) + goto fixture; + for (k = 0; k < 3; k++) + sprintf(crc[k], "%08X|crc", (unsigned)rec[k].crc); + sprintf(lpl[0], "%s/Test Cart Gate.lpl", pl_dir); + sprintf(lpl[1], "%s/Test Cart Zip.lpl", pl_dir); + + use_queue(threaded); + for (k = 0; k < 2; k++) + { + filestream_delete(lpl[k]); + use_db(db[k]); + config_get_ptr()->bools.scan_without_core_match = !k; + retro_atomic_store_release_int(&reads_begun, 0); + begun_size = GATE_ZIP_LEN; + started[k] = run_scan(dir[k]) && scan_completed; + begun_size = 0; + reads[k] = retro_atomic_load_acquire_int(&reads_begun); + } + config_get_ptr()->bools.scan_without_core_match = false; + + sprintf(what, "without core match: a member match is x.zip %s", q); + check( started[0] && file_contains(lpl[0], "bp0.zip\"") + && file_contains(lpl[0], crc[0]) + && !file_contains(lpl[0], "bp0.zip#"), what, + "bp0.zip, not expanded"); + sprintf(what, "zip-listing core: a member match is x.zip %s", q); + check( started[1] && file_contains(lpl[1], "bz0.zip\"") + && file_contains(lpl[1], "bz1.zip\"") + && file_contains(lpl[1], crc[1]) && file_contains(lpl[1], crc[2]) + && !file_contains(lpl[1], ".zip#"), what, + "bz0.zip and bz1.zip, not expanded"); + sprintf(what, "x.zip member matches: no archive read whole %s", q); + sprintf(detail, "%d and %d whole reads", reads[0], reads[1]); + check(started[0] && started[1] && !reads[0] && !reads[1], what, detail); + return; + +fixture: + check(0, "fixture", "could not write the base paths set"); +} + +/* An arcade zip: no core claims its first member's extension, and its + * own CRC is in a database a core listing zip claims. The member pass + * misses, even where a database holds the member, and the whole pass + * writes x.zip. */ +static void arc_gate_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static uint8_t d[2][MF_MEMBER_LEN]; + char db[512], dir[512], p[1024], lpl[2][1024]; + char what[64], detail[64], crc[16]; + db_rec_t arc, cart; + zip_entry_t e[2]; + int reads; + bool started; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db, "%s/arc_gate_db", root); + sprintf(dir, "%s/arc_gate", root); + path_mkdir(db); + path_mkdir(dir); + fill_forced(d[0], MF_MEMBER_LEN, 61, 0x0CA70020u); + fill_forced(d[1], MF_MEMBER_LEN, 67, 0x0CA70021u); + memset(e, 0, sizeof(e)); + e[0].name = "a.ar"; + e[0].data = d[0]; + e[0].len = MF_MEMBER_LEN; + e[0].crc = 0x0CA70020u; + e[1].name = "b.ar"; + e[1].data = d[1]; + e[1].len = MF_MEMBER_LEN; + e[1].crc = 0x0CA70021u; + sprintf(p, "%s/ag0.zip", dir); + if ( !write_zip_entries(p, e, 2) + || !set_whole_rec(&arc, "Gate AG0", p)) + goto fixture; + set_rec(&cart, "Gate AG0 Member", 0x0CA70020u, MF_MEMBER_LEN); + sprintf(p, "%s/Test Arc Gate.rdb", db); + if (!write_db_set(p, &arc, 1)) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, &cart, 1)) + goto fixture; + + sprintf(lpl[0], "%s/Test Arc Gate.lpl", pl_dir); + sprintf(lpl[1], "%s/Test Cart Gate.lpl", pl_dir); + filestream_delete(lpl[0]); + filestream_delete(lpl[1]); + use_queue(threaded); + use_db(db); + retro_atomic_store_release_int(&reads_begun, 0); + begun_size = ARC_ZIP_LEN; + started = run_scan(dir) && scan_completed; + begun_size = 0; + reads = retro_atomic_load_acquire_int(&reads_begun); + + sprintf(crc, "%08X|crc", (unsigned)arc.crc); + sprintf(what, "arcade zip: matched whole as x.zip %s", q); + check( started && file_contains(lpl[0], "ag0.zip\"") + && file_contains(lpl[0], crc) + && !file_contains(lpl[0], "ag0.zip#") + && !file_contains(lpl[1], "ag0.zip"), what, + "ag0.zip, not expanded"); + if (!threaded) + { + sprintf(detail, "%d whole reads", reads); + check(started && reads == 1, "arcade zip: read whole once", detail); + } + return; + +fixture: + check(0, "fixture", "could not write the arcade gate set"); +} + +/* An arcade set whose first member is a cartridge ROM: its extension is + * claimed and its CRC is in the cartridge database, and the set's own + * CRC is in a database a core listing zip claims. An archive of + * several members takes either CRC in each database, so x.zip is + * written for the arcade database and nothing is expanded. */ +static void arc_cart_first_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static uint8_t d[2][MF_MEMBER_LEN]; + char db[512], dir[512], p[1024], lpl[2][1024]; + char what[64], detail[64], crc[16]; + db_rec_t arc, cart; + zip_entry_t e[2]; + int reads; + bool started; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db, "%s/arc_cart_first_db", root); + sprintf(dir, "%s/arc_cart_first", root); + path_mkdir(db); + path_mkdir(dir); + fill_forced(d[0], MF_MEMBER_LEN, 73, 0x0CA70030u); + fill_forced(d[1], MF_MEMBER_LEN, 79, 0x0CA70031u); + memset(e, 0, sizeof(e)); + e[0].name = "a.ct"; + e[0].data = d[0]; + e[0].len = MF_MEMBER_LEN; + e[0].crc = 0x0CA70030u; + e[1].name = "b.ar"; + e[1].data = d[1]; + e[1].len = MF_MEMBER_LEN; + e[1].crc = 0x0CA70031u; + sprintf(p, "%s/ac0.zip", dir); + if ( !write_zip_entries(p, e, 2) + || !set_whole_rec(&arc, "Gate AC0", p)) + goto fixture; + set_rec(&cart, "Gate AC0 Cart", 0x0CA70030u, MF_MEMBER_LEN); + sprintf(p, "%s/Test Arc Gate.rdb", db); + if (!write_db_set(p, &arc, 1)) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, &cart, 1)) + goto fixture; + + sprintf(lpl[0], "%s/Test Arc Gate.lpl", pl_dir); + sprintf(lpl[1], "%s/Test Cart Gate.lpl", pl_dir); + filestream_delete(lpl[0]); + filestream_delete(lpl[1]); + use_queue(threaded); + use_db(db); + retro_atomic_store_release_int(&reads_begun, 0); + begun_size = ARC_ZIP_LEN; + started = run_scan(dir) && scan_completed; + begun_size = 0; + reads = retro_atomic_load_acquire_int(&reads_begun); + + sprintf(crc, "%08X|crc", (unsigned)arc.crc); + sprintf(what, "arcade set, cart ROM first: x.zip %s", q); + check( started && file_contains(lpl[0], "ac0.zip\"") + && file_contains(lpl[0], crc) + && !file_contains(lpl[0], "ac0.zip#") + && !file_contains(lpl[1], "ac0.zip"), what, + "ac0.zip in the arcade playlist, not expanded"); + sprintf(what, "arcade set, cart ROM first: read whole once %s", q); + sprintf(detail, "%d whole reads", reads); + check(started && reads == 1, what, detail); + return; + +fixture: + check(0, "fixture", "could not write the arcade set"); +} + +/* Arcade sets whose readers' whole reads fail: the scan reads them + * whole itself and keeps each reader's member CRC, so every archive's + * directory is read once. */ +static void reader_fail_arcade_lane(const char *root, const char *pl_dir) +{ + static uint8_t d[2][MF_MEMBER_LEN]; + static char names[GATE_ZIPS][16]; + char db[512], dir[512], p[1024], lpl[1024], want[32]; + char detail[64]; + db_rec_t arc[GATE_ZIPS]; + zip_entry_t e[2]; + unsigned k; + int fails, stored; + bool started, all = true; + + if (!readers_expected()) + { + check(1, "failed arcade reads: member CRC kept", "no readers"); + return; + } + sprintf(db, "%s/arc_fail_db", root); + sprintf(dir, "%s/arc_fail", root); + path_mkdir(db); + path_mkdir(dir); + memset(e, 0, sizeof(e)); + e[0].name = "a.ar"; + e[1].name = "b.ar"; + for (k = 0; k < GATE_ZIPS; k++) + { + fill_forced(d[0], MF_MEMBER_LEN, 89 + k, 0x0CA70060u + 2 * k); + fill_forced(d[1], MF_MEMBER_LEN, 97 + k, 0x0CA70061u + 2 * k); + e[0].data = d[0]; + e[0].len = MF_MEMBER_LEN; + e[0].crc = 0x0CA70060u + 2 * k; + e[1].data = d[1]; + e[1].len = MF_MEMBER_LEN; + e[1].crc = 0x0CA70061u + 2 * k; + sprintf(p, "%s/rf%u.zip", dir, k); + sprintf(names[k], "Gate RF%u", k); + if ( !write_zip_entries(p, e, 2) + || !set_whole_rec(&arc[k], names[k], p)) + goto fixture; + } + sprintf(p, "%s/Test Arc Gate.rdb", db); + if (!write_db_set(p, arc, GATE_ZIPS)) + goto fixture; + + sprintf(lpl, "%s/Test Arc Gate.lpl", pl_dir); + filestream_delete(lpl); + use_queue(true); + use_db(db); + retro_atomic_store_release_int(&reader_fails, 0); + retro_atomic_store_release_int(&stored_reads, 0); + fail_size = ARC_ZIP_LEN; + stored_name = "/arc_fail/"; + started = run_scan(dir) && scan_completed; + fail_size = 0; + stored_name = NULL; + fails = retro_atomic_load_acquire_int(&reader_fails); + stored = retro_atomic_load_acquire_int(&stored_reads); + + for (k = 0; k < GATE_ZIPS; k++) + { + sprintf(want, "rf%u.zip\"", k); + all = all && file_contains(lpl, want); + } + check(started && all, "failed arcade reads: matched whole as x.zip", + "rf0.zip to rf3.zip"); + sprintf(detail, "%d reader reads failed, %d directory reads", fails, + stored); + check( started && fails == GATE_ZIPS && stored == GATE_ZIPS, + "failed arcade reads: member CRC kept", detail); + return; + +fixture: + check(0, "fixture", "could not write the failed arcade set"); +} + +/* Writes @n plain .ct files of MF_MEMBER_LEN bytes into @dir, the k-th + * with CRC @crc + k, and their records. */ +static bool write_plain_ct(const char *dir, unsigned n, uint32_t crc, + db_rec_t *rec) +{ + static uint8_t d[MF_MEMBER_LEN]; + static char names[GATE_ZIPS][16]; + char p[1024]; + unsigned k; + + for (k = 0; k < n; k++) + { + FILE *f; + + fill_forced(d, MF_MEMBER_LEN, 83 + k, crc + k); + sprintf(p, "%s/up%u.ct", dir, k); + if (!(f = fopen(p, "wb"))) + return false; + fwrite(d, 1, MF_MEMBER_LEN, f); + fclose(f); + sprintf(names[k], "Gate UP%u", k); + set_rec(&rec[k], names[k], crc + k, MF_MEMBER_LEN); + } + return true; +} + +/* Plain files against a database whose crc index build is refused: + * the refusal is kept, so each file after the first costs its one + * query walk and no further build attempt. */ +static void unindexed_plain_lane(const char *root, const char *pl_dir) +{ + char db[512], dir[2][512], p[1024], lpl[1024], detail[64]; + db_rec_t rec[GATE_ZIPS]; + unsigned k, opens[2]; + bool started[2]; + + sprintf(db, "%s/unindexed_plain_db", root); + sprintf(dir[0], "%s/unindexed_plain_1", root); + sprintf(dir[1], "%s/unindexed_plain_4", root); + path_mkdir(db); + path_mkdir(dir[0]); + path_mkdir(dir[1]); + if ( !write_plain_ct(dir[1], GATE_ZIPS, 0x0CA70050u, rec) + || !write_plain_ct(dir[0], 1, 0x0CA70050u, rec)) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, rec, GATE_ZIPS)) + goto fixture; + sprintf(lpl, "%s/Test Cart Gate.lpl", pl_dir); + use_queue(false); + use_db(db); + + /* One file, then four: what the last three add. */ + for (k = 0; k < 2; k++) + { + filestream_delete(lpl); + index_cap = 1; + rdb_opens = 0; + started[k] = run_scan(dir[k]) && scan_completed; + opens[k] = rdb_opens; + index_cap = 0; + } + check( started[0] && started[1] && file_contains(lpl, "Gate UP0") + && file_contains(lpl, "Gate UP3"), + "unindexed: plain files matched", "Gate UP0 to Gate UP3"); + sprintf(detail, "%u opens for 1 file, %u for %u", opens[0], opens[1], + GATE_ZIPS); + check( started[0] && started[1] && opens[1] >= opens[0] + && opens[1] - opens[0] <= GATE_ZIPS - 1, + "unindexed: a refused index is not built again", detail); + return; + +fixture: + check(0, "fixture", "could not write the unindexed plain set"); +} + +/* Arcade sets whose first members carry a claimed cartridge extension + * and match nothing there: the sets are matched whole and the + * cartridge database, claimed only for that extension, is never + * opened. */ +static void arc_folder_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static uint8_t d[2][MF_MEMBER_LEN]; + static char names[GATE_ZIPS][16]; + char db[512], dir[512], p[1024], lpl[1024], want[32]; + char what[64], detail[64]; + db_rec_t arc[GATE_ZIPS], cart; + zip_entry_t e[2]; + unsigned k, opens; + bool started, all = true; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db, "%s/arc_folder_db", root); + sprintf(dir, "%s/arc_folder", root); + path_mkdir(db); + path_mkdir(dir); + memset(e, 0, sizeof(e)); + e[0].name = "a.ct"; + e[1].name = "b.ar"; + for (k = 0; k < GATE_ZIPS; k++) + { + fill_forced(d[0], MF_MEMBER_LEN, 101 + k, 0x0CA70070u + 2 * k); + fill_forced(d[1], MF_MEMBER_LEN, 107 + k, 0x0CA70071u + 2 * k); + e[0].data = d[0]; + e[0].len = MF_MEMBER_LEN; + e[0].crc = 0x0CA70070u + 2 * k; + e[1].data = d[1]; + e[1].len = MF_MEMBER_LEN; + e[1].crc = 0x0CA70071u + 2 * k; + sprintf(p, "%s/af%u.zip", dir, k); + sprintf(names[k], "Gate AF%u", k); + if ( !write_zip_entries(p, e, 2) + || !set_whole_rec(&arc[k], names[k], p)) + goto fixture; + } + set_rec(&cart, "Gate AF Cart", 0x0CA7007Fu, MF_MEMBER_LEN); + sprintf(p, "%s/Test Arc Gate.rdb", db); + if (!write_db_set(p, arc, GATE_ZIPS)) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, &cart, 1)) + goto fixture; + + sprintf(lpl, "%s/Test Arc Gate.lpl", pl_dir); + filestream_delete(lpl); + use_queue(threaded); + use_db(db); + open_name = "Test Cart Gate"; + named_opens = 0; + started = run_scan(dir) && scan_completed; + opens = named_opens; + open_name = NULL; + + for (k = 0; k < GATE_ZIPS; k++) + { + sprintf(want, "af%u.zip\"", k); + all = all && file_contains(lpl, want); + } + sprintf(what, "arcade folder: sets matched whole %s", q); + check(started && all, what, "af0.zip to af3.zip"); + sprintf(what, "arcade folder: no cartridge database walked %s", q); + sprintf(detail, "%u opens", opens); + check(started && opens == 0, what, detail); + return; + +fixture: + check(0, "fixture", "could not write the arcade folder"); +} + +/* Cartridge zips whose database is claimed only for the member's + * extension, with every index build refused. The member pass leaves + * that database out, so each zip costs the walks the expansion makes + * and no more. */ +static void unindexed_member_lane(const char *root, const char *pl_dir) +{ + static const char *const file[GATE_ZIPS] = { + "ui0.zip", "ui1.zip", "ui2.zip", "ui3.zip" }; + static const char *const name[GATE_ZIPS] = { + "Gate UI0", "Gate UI1", "Gate UI2", "Gate UI3" }; + char db[512], dir[2][512], p[1024], lpl[1024], want[32]; + char detail[64]; + db_rec_t rec[GATE_ZIPS]; + unsigned k, opens[2]; + bool started[2], all = true; + + sprintf(db, "%s/unindexed_db", root); + sprintf(dir[0], "%s/unindexed_1", root); + sprintf(dir[1], "%s/unindexed_4", root); + path_mkdir(db); + path_mkdir(dir[0]); + path_mkdir(dir[1]); + for (k = 0; k < GATE_ZIPS; k++) + if ( !write_gate_zip(dir[1], file[k], "rom.ct", name[k], + 0x0CA70040u + k, &rec[k]) + || (!k && !write_gate_zip(dir[0], file[k], "rom.ct", name[k], + 0x0CA70040u + k, &rec[k]))) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, rec, GATE_ZIPS)) + goto fixture; + sprintf(lpl, "%s/Test Cart Gate.lpl", pl_dir); + use_queue(false); + use_db(db); + + /* One zip, then four: what the last three add. */ + for (k = 0; k < 2; k++) + { + filestream_delete(lpl); + index_cap = 1; + rdb_opens = 0; + started[k] = run_scan(dir[k]) && scan_completed; + opens[k] = rdb_opens; + index_cap = 0; + } + for (k = 0; k < GATE_ZIPS; k++) + { + sprintf(want, "%s#rom.ct", file[k]); + all = all && file_contains(lpl, want); + } + check(started[0] && started[1] && all, + "unindexed: cart zips matched as x.zip#member", + "ui0.zip#rom.ct to ui3.zip#rom.ct"); + sprintf(detail, "%u opens for 1 zip, %u for %u", opens[0], opens[1], + GATE_ZIPS); + check( started[0] && started[1] && opens[1] >= opens[0] + && opens[1] - opens[0] <= GATE_ZIPS - 1, + "unindexed: one database walk per later zip", detail); + return; + +fixture: + check(0, "fixture", "could not write the unindexed set"); +} + +/* Scans @dir with @db's two databases in name order, then in reverse: + * @want with @crc in @pl[0]'s playlist, nothing of @file in @pl[1]'s, + * and @file read whole @whole times. */ +static void two_db_scan(const char *pl_dir, const char *dir, + const char *db, const char *const pl[2], const char *file, + const char *want, const char *crc, const char *lane, int whole, + bool threaded) +{ + char lpl[2][1024], what[128], detail[64]; + unsigned k; + int reads; + bool started; + const char *q = threaded ? "(threaded)" : "(regular)"; + + for (k = 0; k < 2; k++) + sprintf(lpl[k], "%s/%s.lpl", pl_dir, pl[k]); + use_queue(threaded); + use_db(db); + for (k = 0; k < 2; k++) + { + filestream_delete(lpl[0]); + filestream_delete(lpl[1]); + db_order = k ? -1 : 1; + retro_atomic_store_release_int(&reads_begun, 0); + begun_size = GATE_ZIP_LEN; + started = run_scan(dir) && scan_completed; + begun_size = 0; + db_order = 0; + reads = retro_atomic_load_acquire_int(&reads_begun); + + sprintf(what, "%s, %s order %s", lane, k ? "reverse" : "name", q); + check( started && file_contains(lpl[0], want) + && file_contains(lpl[0], crc) && !file_contains(lpl[1], file), + what, want); + sprintf(what, "%s, %s order: %s %s", lane, k ? "reverse" : "name", + whole ? "read whole once" : "not read whole", q); + sprintf(detail, "%d whole reads", reads); + check(started && reads == whole, what, detail); + } +} + +/* A cartridge zip's first member is in a database claimed only for + * it and in one a core listing zip claims: x.zip in the latter, as + * when not looked up by the member, whichever comes first. */ +static void sibling_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static const char *const pl[2] = { "Test Cart Zip", "Test Cart Gate" }; + char db[512], dir[512], p[1024], crc[16]; + db_rec_t rec; + unsigned k; + + sprintf(db, "%s/sibling_db", root); + sprintf(dir, "%s/sibling", root); + path_mkdir(db); + path_mkdir(dir); + if (!write_gate_zip(dir, "sb0.zip", "rom.ct", "Gate SB0", + 0x0CA70030u, &rec)) + goto fixture; + for (k = 0; k < 2; k++) + { + sprintf(p, "%s/%s.rdb", db, pl[k]); + if (!write_db_set(p, &rec, 1)) + goto fixture; + } + sprintf(crc, "%08X|crc", (unsigned)rec.crc); + two_db_scan(pl_dir, dir, db, pl, "sb0.zip", "sb0.zip\"", crc, + "sibling: x.zip where zip is claimed", 0, threaded); + return; + +fixture: + check(0, "fixture", "could not write the sibling set"); +} + +/* The first member in a database claimed only for it, the archive's + * own CRC in one a core listing zip claims: expanded either way. */ +static void dual_match_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static const char *const pl[2] = { "Test Cart Gate", "Test Arc Gate" }; + char db[512], dir[512], p[1024], crc[16]; + db_rec_t rec[2]; + + sprintf(db, "%s/dual_match_db", root); + sprintf(dir, "%s/dual_match", root); + path_mkdir(db); + path_mkdir(dir); + sprintf(p, "%s/dm0.zip", dir); + if ( !write_gate_zip(dir, "dm0.zip", "rom.ct", "Gate DM0 Member", + 0x0CA70040u, &rec[0]) + || !set_whole_rec(&rec[1], "Gate DM0", p)) + goto fixture; + sprintf(p, "%s/%s.rdb", db, pl[0]); + if (!write_db_set(p, &rec[0], 1)) + goto fixture; + sprintf(p, "%s/%s.rdb", db, pl[1]); + if (!write_db_set(p, &rec[1], 1)) + goto fixture; + sprintf(crc, "%08X|crc", (unsigned)rec[0].crc); + two_db_scan(pl_dir, dir, db, pl, "dm0.zip", "dm0.zip#rom.ct", crc, + "dual match: x.zip#member", 0, threaded); + return; + +fixture: + check(0, "fixture", "could not write the dual match set"); +} + +/* With "Scan Inside Archives" off nothing expands, so the member pass + * looks only where zip is claimed: the dual match is x.zip there, and + * a cartridge zip whose core lists only its member is added nowhere + * and read whole. */ +static void archives_off_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static const char *const pl[2] = { "Test Arc Gate", "Test Cart Gate" }; + bool *search_archives = manual_content_scan_get_search_archives_ptr(); + char db[2][512], dir[2][512], p[1024], lpl[1024]; + char what[64], detail[64], crc[16]; + db_rec_t rec[3]; + unsigned k; + int reads; + bool started; + const char *q = threaded ? "(threaded)" : "(regular)"; + + for (k = 0; k < 2; k++) + { + sprintf(db[k], "%s/archives_off_db%u", root, k); + sprintf(dir[k], "%s/archives_off%u", root, k); + path_mkdir(db[k]); + path_mkdir(dir[k]); + } + sprintf(p, "%s/ao0.zip", dir[0]); + if ( !search_archives + || !write_gate_zip(dir[0], "ao0.zip", "rom.ct", "Gate AO0 Member", + 0x0CA70050u, &rec[0]) + || !set_whole_rec(&rec[1], "Gate AO0", p) + || !write_gate_zip(dir[1], "ao1.zip", "rom.ct", "Gate AO1", + 0x0CA70051u, &rec[2])) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db[0]); + if (!write_db_set(p, &rec[0], 1)) + goto fixture; + sprintf(p, "%s/Test Arc Gate.rdb", db[0]); + if (!write_db_set(p, &rec[1], 1)) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db[1]); + if (!write_db_set(p, &rec[2], 1)) + goto fixture; + + /* Automatic always searches archives. */ + manual_content_scan_set_menu_scan_method( + MANUAL_CONTENT_SCAN_METHOD_CUSTOM); + *search_archives = false; + sprintf(crc, "%08X|crc", (unsigned)rec[1].crc); + two_db_scan(pl_dir, dir[0], db[0], pl, "ao0.zip", "ao0.zip\"", crc, + "archives off: dual match x.zip", 1, threaded); + + sprintf(lpl, "%s/Test Cart Gate.lpl", pl_dir); + filestream_delete(lpl); + use_queue(threaded); + use_db(db[1]); + retro_atomic_store_release_int(&reads_begun, 0); + begun_size = GATE_ZIP_LEN; + started = run_scan(dir[1]) && scan_completed; + begun_size = 0; + reads = retro_atomic_load_acquire_int(&reads_begun); + *search_archives = true; + manual_content_scan_set_menu_scan_method( + MANUAL_CONTENT_SCAN_METHOD_AUTOMATIC); + + sprintf(what, "archives off: cart zip added nowhere %s", q); + check(started && !file_contains(lpl, "ao1.zip"), what, "ao1.zip"); + sprintf(what, "archives off: cart zip read whole %s", q); + sprintf(detail, "%d whole reads", reads); + check(started && reads == 1, what, detail); + return; + +fixture: + check(0, "fixture", "could not write the archives off sets"); +} + +/* Each run of zips is the window and a few more long. */ +#define WA_EXTRA 3 + +static bool files_equal(const char *a, const char *b) +{ + void *da = NULL, *db = NULL; + int64_t la = 0, lb = 0; + bool eq; + + if ( !filestream_read_file(a, &da, &la) + || !filestream_read_file(b, &db, &lb)) + { + free(da); + return false; + } + eq = la == lb && !memcmp(da, db, (size_t)la); + free(da); + free(db); + return eq; +} + +/* Writes wa_.zip for the file at @pos in content-list order, and + * its record: two cartridge zips (1024-byte member, a 1130-byte zip), + * @n arcade zips (1100, the last two 1200: 1306 bytes), then @n + * cartridge zips (1000, the last two 1024). */ +static bool write_wa_zip(const char *dir, unsigned k, unsigned pos, + unsigned n, char *name, db_rec_t *rec) +{ + static uint8_t d[1200]; + char p[1024]; + zip_entry_t e; + unsigned run = (pos < 2) ? 0 : (pos < 2 + n) ? 1 : 2; + unsigned at = (run == 0) ? pos : (run == 1) ? pos - 2 : pos - 2 - n; + bool tail = run && at >= n - 2; + uint32_t len = (run == 1) ? (tail ? 1200 : 1100) + : (run == 2 && !tail) ? 1000 : 1024; + uint32_t crc = 0x0BF30000u + pos; + + fill_forced(d, len, 61 + pos, crc); + memset(&e, 0, sizeof(e)); + e.name = "m.rd"; + e.data = d; + e.len = len; + e.crc = crc; + sprintf(p, "%s/wa_%03u.zip", dir, k); + sprintf(name, "WA %c%03u", "ABC"[run], at); + if (!write_zip_entries(p, &e, 1)) + return false; + if (run == 1) + return set_whole_rec(rec, name, p); + set_rec(rec, name, crc, len); + return true; +} + +/* Cartridge zips, arcade zips, then cartridge zips, on the threaded + * queue. Readers read only archive directories until an archive + * misses by its member CRC, then archives whole until one matches by + * it. A file the window reaches after a switch is claimed after it: + * the last two arcade zips are read whole by readers alone, the last + * two cartridge zips by no one, nor are the first ones. The playlists + * equal a run without readers. */ +static void whole_ahead_lane(const char *root, const char *pl_dir) +{ + static const char *const pl[2] = { "Test MF Cart", "Test MF Arcade" }; + static char first[32]; + char db[512], dir[512], p[1024], lpl[2][1024], ref[2][1024]; + char detail[128]; + char *names = NULL; + db_rec_t *cart = NULL, *arcade = NULL; + unsigned n = reader_window() + WA_EXTRA; + unsigned i, j, nc, na; + unsigned *pos = NULL; + int begun, by_scan, by_reader, stored; + bool started, same = true; + + if (!readers_expected()) + { + check(1, "whole_ahead: read whole ahead on and off", "no readers"); + return; + } + sprintf(db, "%s/whole_ahead_db", root); + sprintf(dir, "%s/whole_ahead", root); + path_mkdir(db); + path_mkdir(dir); + wa_count = 2 + 2 * n; + names = (char*)calloc(wa_count, 16); + cart = (db_rec_t*)calloc(wa_count, sizeof(*cart)); + arcade = (db_rec_t*)calloc(n, sizeof(*arcade)); + pos = (unsigned*)calloc(wa_count, sizeof(*pos)); + wa_last = (unsigned*)calloc(wa_count, sizeof(*wa_last)); + if (!names || !cart || !arcade || !pos || !wa_last) + goto fixture; + for (j = 0; j < 2; j++) + { + sprintf(lpl[j], "%s/%s.lpl", pl_dir, pl[j]); + sprintf(ref[j], "%s/%s ref.lpl", pl_dir, pl[j]); + } + use_db(db); + use_queue(false); + + /* Twice: in name order, to learn the list's, then in that. */ + for (j = 0; j < 2; j++) + { + nc = na = 0; + for (i = 0; i < wa_count; i++) + { + db_rec_t *rec = (pos[i] - 2 < n) ? &arcade[na++] : &cart[nc++]; + if (!write_wa_zip(dir, i, pos[i], n, names + i * 16, rec)) + goto fixture; + if (pos[i] == n - 1) + look_crc[0] = rec->crc; + } + sprintf(p, "%s/%s.rdb", db, pl[0]); + if (!write_db_set(p, cart, nc)) + goto fixture; + sprintf(p, "%s/%s.rdb", db, pl[1]); + if (!write_db_set(p, arcade, na)) + goto fixture; + filestream_delete(lpl[0]); + filestream_delete(lpl[1]); + if (j) + break; + if (!run_scan(dir) || !scan_completed) + goto fixture; + for (i = 0; i < wa_count; i++) + { + unsigned k; + if (!wa_last[i]) + goto fixture; + for (k = 0; k < wa_count; k++) + if (wa_last[k] < wa_last[i]) + pos[i]++; + if (!pos[i]) + sprintf(first, "wa_%03u.zip", i); + } + } + free(wa_last); + wa_last = NULL; + + filestream_delete(ref[0]); + filestream_delete(ref[1]); + if (!run_scan(dir) || !scan_completed) + goto fixture; + for (j = 0; j < 2; j++) + if (filestream_rename(lpl[j], ref[j])) + goto fixture; + + use_queue(true); + retro_atomic_store_release_int(&reads_begun, 0); + retro_atomic_store_release_int(&watch_scan, 0); + retro_atomic_store_release_int(&watch_reader, 0); + retro_atomic_store_release_int(&stored_reads, 0); + begun_size = 30 + 4 + 1024 + 46 + 4 + 22; + watch_size = 30 + 4 + 1200 + 46 + 4 + 22; + stored_name = first; + /* The arcade zip before the last two, looked up by its own CRC, + * waits for readers to begin them. */ + look_want[0] = 2; + look_n = 1; + look_watch = true; + started = run_scan(dir) && scan_completed; + look_n = 0; + look_watch = false; + begun_size = 0; + watch_size = 0; + stored_name = NULL; + begun = retro_atomic_load_acquire_int(&reads_begun); + by_scan = retro_atomic_load_acquire_int(&watch_scan); + by_reader = retro_atomic_load_acquire_int(&watch_reader); + stored = retro_atomic_load_acquire_int(&stored_reads); + for (j = 0; j < 2; j++) + same = same && files_equal(lpl[j], ref[j]); + + sprintf(detail, "%d whole reads", begun); + check(started && begun == 0, + "whole_ahead: cartridge zips read no archive whole", detail); + sprintf(detail, "scan %d, readers %d", by_scan, by_reader); + passes_detail(detail); + check( started && by_scan == 0 && by_reader > 0 + && retro_atomic_load_acquire_int(&look_hits) + && !retro_atomic_load_acquire_int(&look_late), + "whole_ahead: readers read the arcade zips whole", detail); + sprintf(detail, "%d directory reads", stored); + check(started && stored == 1, + "whole_ahead: a reader's directory read is the scan's", detail); + sprintf(p, "WA C%03u", n - 1); + check( started && same && file_contains(lpl[0], "WA A000") + && file_contains(lpl[0], p) && file_contains(lpl[1], "WA B000"), + "whole_ahead: the playlists of a run without readers", + same ? "equal" : "differ"); + free(names); + free(cart); + free(arcade); + free(pos); + return; + +fixture: + free(wa_last); + wa_last = NULL; + free(names); + free(cart); + free(arcade); + free(pos); + check(0, "fixture", "could not write or scan the whole_ahead set"); +} + +/* Writes wa_.zip for the file at @pos in content-list order, and + * its record: @n arcade zips by their own CRC (a 900-byte member, the + * last two 1000: 1106 bytes), then @n cartridge zips by their member, + * rom.ct (900, the last two 1024: 1134 bytes). */ +static bool write_hc_zip(const char *dir, unsigned k, unsigned pos, + unsigned n, db_rec_t *rec) +{ + static uint8_t d[MF_MEMBER_LEN]; + char p[1024]; + bool arc = pos < n; + bool tail = (arc ? pos : pos - n) >= n - 2; + uint32_t len = !tail ? 900 : arc ? 1000 : MF_MEMBER_LEN; + uint32_t crc = 0x0CA71000u + pos; + + fill_forced(d, len, 71 + pos, crc); + sprintf(p, "%s/wa_%03u.zip", dir, k); + if (!write_zip(p, arc ? "a.ar" : "rom.ct", d, len, crc)) + return false; + if (arc) + return set_whole_rec(rec, "HC Arcade", p); + set_rec(rec, "HC Cart", crc, len); + return true; +} + +/* Arcade zips by their own CRC, then cartridge zips by their first + * member in a database claimed only for it, on the threaded queue. + * The first arcade zip's member miss turns read whole ahead on, the + * first cartridge zip's member hit turns it off. The last two of + * each run are past the window from the switch: the arcade ones are + * read whole by readers, the cartridge ones by no one. */ +static void hit_clears_lane(const char *root, const char *pl_dir) +{ + char db[512], dir[512], p[1024], lpl[1024], detail[128]; + db_rec_t *arc = NULL, *cart = NULL; + unsigned *pos = NULL; + unsigned n = reader_window() + WA_EXTRA; + unsigned i, j, k, na, nc; + int begun, by_reader; + bool started; + + if (!readers_expected()) + { + check(1, "member hit: read whole ahead off", "no readers"); + return; + } + sprintf(db, "%s/whole_ahead_hit_db", root); + sprintf(dir, "%s/whole_ahead_hit", root); + path_mkdir(db); + path_mkdir(dir); + wa_count = 2 * n; + arc = (db_rec_t*)calloc(wa_count, sizeof(*arc)); + cart = (db_rec_t*)calloc(wa_count, sizeof(*cart)); + pos = (unsigned*)calloc(wa_count, sizeof(*pos)); + wa_last = (unsigned*)calloc(wa_count, sizeof(*wa_last)); + if (!arc || !cart || !pos || !wa_last) + goto fixture; + sprintf(lpl, "%s/Test Cart Gate.lpl", pl_dir); + use_db(db); + use_queue(false); + + /* Twice: in name order, to learn the list's, then in that. */ + for (j = 0; j < 2; j++) + { + na = nc = 0; + for (i = 0; i < wa_count; i++) + { + if (!write_hc_zip(dir, i, pos[i], n, + (pos[i] < n) ? &arc[na++] : &cart[nc++])) + goto fixture; + if (pos[i] == n - 3) + look_crc[0] = arc[na - 1].crc; + } + sprintf(p, "%s/Test Arc Gate.rdb", db); + if (!write_db_set(p, arc, na)) + goto fixture; + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, cart, nc)) + goto fixture; + filestream_delete(lpl); + sprintf(p, "%s/Test Arc Gate.lpl", pl_dir); + filestream_delete(p); + if (j) + break; + if (!run_scan(dir) || !scan_completed) + goto fixture; + for (i = 0; i < wa_count; i++) + { + if (!wa_last[i]) + goto fixture; + for (k = 0; k < wa_count; k++) + if (wa_last[k] < wa_last[i]) + pos[i]++; + } + } + free(wa_last); + wa_last = NULL; + + use_queue(true); + retro_atomic_store_release_int(&reads_begun, 0); + retro_atomic_store_release_int(&watch_scan, 0); + retro_atomic_store_release_int(&watch_reader, 0); + begun_size = GATE_ZIP_LEN; + watch_size = 1106; + /* The arcade zip before the last two, looked up by its own CRC, + * waits for readers to begin them. */ + look_want[0] = 2; + look_n = 1; + look_watch = true; + started = run_scan(dir) && scan_completed; + look_n = 0; + look_watch = false; + begun_size = 0; + watch_size = 0; + begun = retro_atomic_load_acquire_int(&reads_begun); + by_reader = retro_atomic_load_acquire_int(&watch_reader); + + sprintf(detail, "%d reader reads", by_reader); + passes_detail(detail); + check( started && by_reader > 0 + && retro_atomic_load_acquire_int(&look_hits) + && !retro_atomic_load_acquire_int(&look_late), + "member hit: readers read the last arcade zips whole", detail); + sprintf(detail, "%d whole reads", begun); + check(started && begun == 0, + "member hit: the last cartridge zips read whole by no one", detail); + sprintf(detail, "%u cartridge zips", nc); + check( started && file_contains(lpl, ".zip#rom.ct") + && file_contains(lpl, "HC Cart"), + "member hit: cartridge zips matched as x.zip#member", detail); + free(arc); + free(cart); + free(pos); + return; + +fixture: + free(wa_last); + wa_last = NULL; + free(arc); + free(cart); + free(pos); + check(0, "fixture", "could not write or scan the member hit set"); +} + +#define RA_FILES 8 + +/* Writes wa_.zip for the file at @pos in content-list order, and + * its record, by its own CRC: the first a 1206-byte zip, the second + * 1306 bytes with no stored CRC, the rest 1130. */ +static bool write_ra_zip(const char *dir, unsigned k, unsigned pos, + db_rec_t *rec) +{ + static uint8_t d[1200]; + static char names[RA_FILES][16]; + char p[1024]; + zip_entry_t e; + uint32_t len = (pos == 0) ? 1100 : (pos == 1) ? 1200 : 1024; + + fill_forced(d, len, 71 + pos, 0x0BF40000u + pos); + memset(&e, 0, sizeof(e)); + e.name = "m.rd"; + e.data = d; + e.len = len; + e.crc = (pos == 1) ? 0 : 0x0BF40000u + pos; + sprintf(p, "%s/wa_%03u.zip", dir, k); + sprintf(names[k], "RA %03u", pos); + return write_zip_entries(p, &e, 1) && set_whole_rec(rec, names[k], p); +} + +/* An arcade folder smaller than the window, on the threaded queue. + * Readers read every directory before the first archive misses by + * its member CRC; then they take back as many archives as the misses + * grant them and read them whole. An archive read whole already (the + * second: no stored CRC) is not read again, nor by the scan while a + * reader holds it (read slowly), on whichever worker. Test only: the + * first's directory read waits for the others', and the scan's read + * of it whole waits for the read its miss grants to begin. Which + * archive that is is the readers' to pick. The playlist equals a run + * without readers. */ +static void rearm_lane(const char *root, const char *pl_dir) +{ + static db_rec_t rec[RA_FILES]; + static unsigned last[RA_FILES]; + static char first[32]; + char db[512], dir[512], p[1024], lpl[1024], ref[1024]; + char detail[64]; + unsigned pos[RA_FILES]; + unsigned i, j, k; + int begun, slow, by_scan, by_reader; + bool started, same; + + if (!readers_expected()) + { + check(1, "rearm: read whole after the first miss", "no readers"); + return; + } + sprintf(db, "%s/whole_ahead_rearm_db", root); + sprintf(dir, "%s/whole_ahead_rearm", root); + sprintf(lpl, "%s/Test MF Arcade.lpl", pl_dir); + sprintf(ref, "%s/Test MF Arcade ra.lpl", pl_dir); + path_mkdir(db); + path_mkdir(dir); + sprintf(p, "%s/Test MF Arcade.rdb", db); + memset(pos, 0, sizeof(pos)); + memset(last, 0, sizeof(last)); + wa_count = RA_FILES; + wa_last = last; + use_db(db); + use_queue(false); + + /* Twice: in name order, to learn the list's, then in that. */ + for (j = 0; j < 2; j++) + { + for (i = 0; i < RA_FILES; i++) + if (!write_ra_zip(dir, i, pos[i], &rec[i])) + goto fixture; + if (!write_db_set(p, rec, RA_FILES)) + goto fixture; + filestream_delete(lpl); + if (j) + break; + if (!run_scan(dir) || !scan_completed) + goto fixture; + for (i = 0; i < RA_FILES; i++) + { + if (!last[i]) + goto fixture; + for (k = 0; k < RA_FILES; k++) + if (last[k] < last[i]) + pos[i]++; + if (!pos[i]) + sprintf(first, "wa_%03u.zip", i); + } + } + wa_last = NULL; + + filestream_delete(ref); + if (!run_scan(dir) || !scan_completed || filestream_rename(lpl, ref)) + goto fixture; + + use_queue(true); + retro_atomic_store_release_int(&reads_begun, 0); + retro_atomic_store_release_int(&watch_scan, 0); + retro_atomic_store_release_int(&watch_reader, 0); + retro_atomic_store_release_int(&slow_in, 0); + retro_atomic_store_release_int(&slow_back, 0); + retro_atomic_store_release_int(&slow_by_scan, 0); + retro_atomic_store_release_int(&stored_reads, 0); + stored_name = "whole_ahead_rearm"; + dir_hold = first; + hold_dirs = RA_FILES - 1; + begun_size = 30 + 4 + 1024 + 46 + 4 + 22; + watch_size = begun_size; + slow_size = 30 + 4 + 1200 + 46 + 4 + 22; + slow_ms = 1000; + hold_size = 30 + 4 + 1100 + 46 + 4 + 22; + hold_want = 1; + started = run_scan(dir) && scan_completed; + begun_size = 0; + watch_size = 0; + slow_size = 0; + slow_ms = 0; + hold_size = 0; + stored_name = NULL; + dir_hold = NULL; + begun = retro_atomic_load_acquire_int(&reads_begun); + slow = retro_atomic_load_acquire_int(&slow_in); + by_scan = retro_atomic_load_acquire_int(&watch_scan); + by_reader = retro_atomic_load_acquire_int(&watch_reader); + same = files_equal(lpl, ref); + + sprintf(detail, "scan %d, readers %d", by_scan, by_reader); + check( started && by_reader > 0 + && !retro_atomic_load_acquire_int(&slow_by_scan), + "rearm: readers read the zips after the first", detail); + sprintf(detail, "%d + %d whole reads", begun, slow); + check(started && begun == RA_FILES - 2 && slow == 1, + "rearm: each zip read whole once", detail); + check( started && same && file_contains(lpl, "RA 000") + && file_contains(lpl, "RA 007"), + "rearm: the playlist of a run without readers", + same ? "equal" : "differ"); + return; + +fixture: + wa_last = NULL; + check(0, "fixture", "could not write or scan the rearm set"); +} + +/* The read ahead lanes' zips, by role: K a cartridge zip its member's + * stored CRC matches, F the same 50 bytes longer, U one nothing + * matches, N an unknown zip with no stored CRC over the readers' + * cutoff, which the scan reads whole itself, M the same 200 bytes + * shorter, A an arcade zip by its own CRC, B the same 100 bytes + * longer, G an A whose member is a cart gate ROM, H a zip whose member + * the gate database holds and S a 7z of a gate ROM nothing matches. */ +#define AH_GATE_CRC 0x0CBFFFFFu +#define AH_FILES 40 +#define AH_BIG 66000 +#define AH_SLOW_MS 300 + +static uint32_t ah_member_len(char role) +{ + switch (role) + { + case 'N': return AH_BIG; + case 'M': return AH_BIG - 200; + case 'F': return 1050; + case 'A': + case 'G': + case 'H': + case 'S': return 1100; + case 'B': return 1200; + default: break; + } + return 1000; +} + +/* The zip's size: one member, m.rd or m.ct. */ +static uint32_t ah_len(char role) +{ + return 30 + 4 + ah_member_len(role) + 46 + 4 + 22; +} + +/* Writes wa_.zip, or .7z for an S, as the file at @pos in + * content-list order. */ +static bool write_ah_zip(const char *dir, unsigned k, unsigned pos, + char role) +{ + static uint8_t d[AH_BIG]; + char p[1024]; + zip_entry_t e; + uint32_t crc = (role == 'H') ? AH_GATE_CRC : 0x0CB00000u + pos; + + fill_forced(d, ah_member_len(role), 83 + pos, crc); + memset(&e, 0, sizeof(e)); + e.name = (role == 'G' || role == 'H' || role == 'S') ? "m.ct" : "m.rd"; + e.data = d; + e.len = ah_member_len(role); + e.crc = (role == 'N' || role == 'M') ? 0 : crc; + sprintf(p, "%s/wa_%03u.%s", dir, k, (role == 'S') ? "zip" : "7z"); + filestream_delete(p); + sprintf(p, "%s/wa_%03u.%s", dir, k, (role == 'S') ? "7z" : "zip"); +#ifdef HAVE_7ZIP + if (role == 'S') + return write_7z(p, e.name, d, e.len, crc) != 0; +#endif + return write_zip_entries(p, &e, 1) != 0; +} + +/* Scans zips of @roles, in content-list order, on the threaded queue; + * true when every K, F, A, B and G is in its playlist. The first + * zip's directory read waits for the others', so a reader has read + * each directory by the scan's first miss. With @gate, the readers + * instead wait in the F zips' (one each: the last), until all are in + * and @gate slow reads have begun, and the scan's read of the first N + * waits for them to be in. A stream of @slow bytes sleeps AH_SLOW_MS + * before its first quantum, time for the readers to take what they + * may: an N after each miss. Reads of @begun-byte zips are counted as + * each begins, those of @watch-byte ones by who makes them. */ +static bool ah_scan(const char *root, const char *pl_dir, const char *tag, + const char *roles, uint32_t slow, uint32_t begun, uint32_t watch, + int gate) +{ + static unsigned last[AH_FILES]; + static char names[AH_FILES][16]; + static char first[32]; + static db_rec_t cart[AH_FILES], arc[AH_FILES]; + char db[512], dir[512], p[1024], lpl[2][1024]; + unsigned pos[AH_FILES]; + unsigned n = (unsigned)strlen(roles); + unsigned i, j, k, nc = 0, na = 0; + bool ok; + + if (n > AH_FILES) + return false; + sprintf(db, "%s/whole_ahead_%s_db", root, tag); + sprintf(dir, "%s/whole_ahead_%s", root, tag); + sprintf(lpl[0], "%s/Test MF Cart.lpl", pl_dir); + sprintf(lpl[1], "%s/Test MF Arcade.lpl", pl_dir); + path_mkdir(db); + path_mkdir(dir); + memset(pos, 0, sizeof(pos)); + memset(last, 0, sizeof(last)); + wa_count = n; + wa_last = last; + use_db(db); + use_queue(false); + + /* Twice: in name order, to learn the list's, then in that. */ + for (j = 0; j < 2; j++) + { + nc = na = 0; + for (i = 0; i < n; i++) + { + char role = roles[pos[i]]; + + if (!write_ah_zip(dir, i, pos[i], role)) + goto fail; + sprintf(names[pos[i]], "AH %c%03u", role, pos[i]); + if (role == 'K' || role == 'F') + set_rec(&cart[nc++], names[pos[i]], 0x0CB00000u + pos[i], + ah_member_len(role)); + else if (role == 'A' || role == 'B' || role == 'G') + { + sprintf(p, "%s/wa_%03u.zip", dir, i); + if (!set_whole_rec(&arc[na++], names[pos[i]], p)) + goto fail; + } + } + sprintf(p, "%s/Test MF Cart.rdb", db); + if (!write_db_set(p, cart, nc)) + goto fail; + sprintf(p, "%s/Test MF Arcade.rdb", db); + if (!write_db_set(p, arc, na)) + goto fail; + if (strpbrk(roles, "GHS")) + { + db_rec_t gate; + + set_rec(&gate, "AH gate", AH_GATE_CRC, ah_member_len('G')); + sprintf(p, "%s/Test Cart Gate.rdb", db); + if (!write_db_set(p, &gate, 1)) + goto fail; + } + filestream_delete(lpl[0]); + filestream_delete(lpl[1]); + if (j) + break; + if (!run_scan(dir) || !scan_completed) + goto fail; + for (i = 0; i < n; i++) + { + if (!last[i]) + goto fail; + for (k = 0; k < n; k++) + if (last[k] < last[i]) + pos[i]++; + if (!pos[i]) + sprintf(first, "wa_%03u.", i); + } + } + wa_last = NULL; + + use_queue(true); + retro_atomic_store_release_int(&reads_begun, 0); + retro_atomic_store_release_int(&reader_begun, 0); + retro_atomic_store_release_int(&watch_scan, 0); + retro_atomic_store_release_int(&watch_reader, 0); + retro_atomic_store_release_int(&slow_in, 0); + retro_atomic_store_release_int(&slow_back, 0); + retro_atomic_store_release_int(&stored_reads, 0); + retro_atomic_store_release_int(&gate_in, 0); + retro_atomic_store_release_int(&gate_on_scan, 0); + memset(slow_begun, 0, sizeof(slow_begun)); + stored_name = dir; + dir_hold = gate ? NULL : first; + hold_dirs = (int)n - 1; + gate_size = ah_member_len('F'); + gate_slow = gate; + gate_n = gate ? (int)readers_expected() : 0; + hold_size = gate ? ah_len('N') : 0; + hold_want = 0; + begun_size = begun; + watch_size = watch; + slow_size = slow; + slow_ms = AH_SLOW_MS; + ok = run_scan(dir) && scan_completed; + begun_size = 0; + watch_size = 0; + slow_size = 0; + slow_ms = 0; + gate_n = 0; + hold_size = 0; + stored_name = NULL; + dir_hold = NULL; + for (i = 0; i < nc; i++) + ok = ok && file_contains(lpl[0], cart[i].name); + for (i = 0; i < na; i++) + ok = ok && file_contains(lpl[1], arc[i].name); + return ok; + +fail: + wa_last = NULL; + return false; +} + +/* One cartridge zip nothing matches among known ones: its member miss + * lets the readers read one zip whole ahead, the next zip's member hit + * no more. Which one is theirs to pick. */ +static void ahead_lone_lane(const char *root, const char *pl_dir) +{ + char detail[64]; + bool ok; + int begun; + + if (!readers_expected()) + { + check(1, "read ahead: one after a lone miss", "no readers"); + return; + } + ok = ah_scan(root, pl_dir, "lone", "KUNKKKKKKKK", + ah_len('N'), ah_len('K'), 0, 0); + begun = retro_atomic_load_acquire_int(&reader_begun); + sprintf(detail, "%d reader reads", begun); + check(ok && begun == 1, "read ahead: one after a lone miss", detail); +} + +/* Misses in a row let the readers read 1, 2, then 4 zips whole ahead, + * up to SCAN_READER_AHEAD_MAX each. After each miss the scan reads + * an N whole, slowly, so the readers take all they may first. */ +static void ahead_double_lane(const char *root, const char *pl_dir) +{ + char detail[64]; + bool ok; + int want[3]; + int i, g, sum, begun; + + if (!readers_expected()) + { + check(1, "read ahead: misses in a row double it", "no readers"); + return; + } + for (i = 0, g = 1, sum = 0; i < 3; i++, g *= 2) + want[i] = sum += (g < SCAN_READER_AHEAD_MAX) + ? g : SCAN_READER_AHEAD_MAX; + ok = ah_scan(root, pl_dir, "double", "KUNUNUNKKKKKKKKKK", + ah_len('N'), ah_len('K'), 0, 0); + begun = retro_atomic_load_acquire_int(&reader_begun); + sprintf(detail, "%d, %d, %d, %d reader reads (want %d, %d, %d)", + slow_begun[0], slow_begun[1], slow_begun[2], begun, + want[0], want[1], want[2]); + check( ok && slow_begun[0] == want[0] && slow_begun[1] == want[1] + && slow_begun[2] == want[2] && begun == want[2], + "read ahead: misses in a row double it", detail); +} + +/* Miss, hit, miss: a member hit ends the run, so the second miss lets + * the readers read one zip whole ahead again, not two. */ +static void ahead_streak_lane(const char *root, const char *pl_dir) +{ + char detail[64]; + bool ok; + int begun; + + if (!readers_expected()) + { + check(1, "read ahead: a member hit ends the run", "no readers"); + return; + } + ok = ah_scan(root, pl_dir, "streak", "KUNKUNKKKKKK", + ah_len('N'), ah_len('K'), 0, 0); + begun = retro_atomic_load_acquire_int(&reader_begun); + sprintf(detail, "%d reader reads", begun); + check(ok && begun == 2, "read ahead: a member hit ends the run", + detail); +} + +/* @head, then an F for each reader: the last zips, which the readers + * wait in with @gate. */ +static void ah_gate_roles(char *roles, size_t len, const char *head) +{ + size_t n = strlen(head); + unsigned r = readers_expected(); + + if (n + r >= len) + r = 0; + memcpy(roles, head, n); + memset(roles + n, 'F', r); + roles[n + r] = '\0'; +} + +/* Each reader's last zip, an F far ahead, holds only its stored CRC + * when the first miss grants one read whole: it goes to the scan's + * next zip (a K), not to an F. The M between them are not read + * ahead. */ +static void ahead_next_lane(const char *root, const char *pl_dir) +{ + char roles[64], detail[64]; + bool ok; + int next_reads, far_reads; + + if (!readers_expected()) + { + check(1, "read ahead: the scan's next first", "no readers"); + return; + } + ah_gate_roles(roles, sizeof(roles), "NNUNKMMMMMMMMMMMMMMMMMMMMMMMM"); + ok = ah_scan(root, pl_dir, "next", roles, ah_len('N'), + ah_len('K'), ah_len('F'), 1); + next_reads = retro_atomic_load_acquire_int(&reader_begun); + far_reads = retro_atomic_load_acquire_int(&watch_reader); + sprintf(detail, "next %d, far %d reader reads", next_reads, + far_reads); + check(ok && next_reads == 1 && !far_reads, + "read ahead: the scan's next first", detail); +} + +/* Three misses grant seven reads whole while the readers wait in their + * F, and a member hit follows: once out, they read no F. A reader on + * the scan's worker cannot wait there (the scan would not run): it + * may read its own F, while the others' are held. */ +static void ahead_hit_count_lane(const char *root, const char *pl_dir) +{ + char roles[64], detail[80]; + bool ok, on_scan; + int far_reads; + + if (!readers_expected()) + { + check(1, "read ahead: a member hit takes back what is left", + "no readers"); + return; + } + ah_gate_roles(roles, sizeof(roles), "NUUUKN"); + ok = ah_scan(root, pl_dir, "hitcount", roles, ah_len('N'), + ah_len('F'), 0, 2); + far_reads = retro_atomic_load_acquire_int(&reader_begun); + on_scan = retro_atomic_load_acquire_int(&gate_on_scan) > 0; + sprintf(detail, "%d reader reads of an F%s", far_reads, + on_scan ? ", one reader on the scan's worker" : ""); + check(ok && far_reads <= (on_scan ? 1 : 0), + "read ahead: a member hit takes back what is left", detail); +} + +/* Arcade zips, each a member miss: the readers read whole ahead as the + * misses add up, so past the first 16 the scan reads none itself. + * Test only: each zip's lookup waits for the readers to begin every + * read the misses before it granted. */ +static void ahead_arcade_lane(const char *root, const char *pl_dir) +{ + char detail[128]; + bool ok; + int begun, by_scan, by_reader, granted = 0, grant = 1; + unsigned k; + + if (!readers_expected()) + { + check(1, "read ahead: arcade zips by readers", "no readers"); + return; + } + for (k = 0; k < 40; k++) + { + look_crc[k] = 0x0CB00000u + k; + look_want[k] = granted; + granted += grant; + grant = (grant * 2 < SCAN_READER_AHEAD_MAX) + ? grant * 2 : SCAN_READER_AHEAD_MAX; + } + look_n = 40; + look_files = 40; + ok = ah_scan(root, pl_dir, "arcade", + "AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBBBBBBB", 0, ah_len('B'), + ah_len('B'), 0); + look_n = 0; + look_files = 0; + begun = retro_atomic_load_acquire_int(&reads_begun); + by_scan = retro_atomic_load_acquire_int(&watch_scan); + by_reader = retro_atomic_load_acquire_int(&watch_reader); + sprintf(detail, "%d whole reads: scan %d, readers %d", begun, by_scan, + by_reader); + passes_detail(detail); + check( ok && begun == 24 && by_scan == 0 && by_reader > 0 + && retro_atomic_load_acquire_int(&look_hits) >= 40 + && !retro_atomic_load_acquire_int(&look_late), + "read ahead: arcade zips by readers", detail); +} + +/* Two arcade zips miss by their members, then a G, whose member lets + * in a database not walked yet: while the scan walks it, the readers + * read whole ahead as far as they may, and the G's miss after grants + * what it would have. Test only: the second zip's lookup waits for + * the first's grant to be taken, the walk for the readers to begin + * every read granted before it, then as many more as they may, and the + * lookup of the A after the N for them to begin the G's. */ +static void ahead_walk_lane(const char *root, const char *pl_dir) +{ + char detail[160]; + bool ok; + bool late; + int g2 = (2 < SCAN_READER_AHEAD_MAX) ? 2 : SCAN_READER_AHEAD_MAX; + int g3 = (4 < SCAN_READER_AHEAD_MAX) ? 4 : SCAN_READER_AHEAD_MAX; + int want; + + if (!readers_expected()) + { + check(1, "read ahead: whole while the scan walks", "no readers"); + check(1, "read ahead: a walk grants nothing after", "no readers"); + return; + } + walk_name = "Test Cart Gate"; + walk_want = 1 + g2; + walk_more = SCAN_READER_AHEAD_MAX; + walk_late = false; + walk_from = -1; + walk_to = -1; + want = walk_want + SCAN_READER_AHEAD_MAX + g3; + look_crc[0] = 0x0CB00001u; + look_want[0] = 1; + look_crc[1] = 0x0CB00004u; + look_want[1] = want; + look_end[1] = -1; + look_n = 2; + ok = ah_scan(root, pl_dir, "walk", + "AAGNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", ah_len('N'), + ah_len('A'), 0, 0); + look_n = 0; + walk_name = NULL; + walk_job = NULL; + late = walk_late || retro_atomic_load_acquire_int(&look_late); + sprintf(detail, "readers %d as it began, %d as it ended", walk_from, + walk_to); + passes_detail(detail); + strcat(detail, walk_late ? ", walk late" : ""); + check( ok && !late && retro_atomic_load_acquire_int(&look_hits) > 0 + && walk_from == walk_want, + "read ahead: whole while the scan walks", detail); + sprintf(detail, "%d reader reads by the A after the N (want %d)%s", + look_end[1], want, late ? ", late" : ""); + check(ok && !late && look_end[1] == want, + "read ahead: a walk grants nothing after", detail); +} + +/* Two 7z miss by their gate ROMs, which let in only the gate database, + * then a zip walks the arcade one: H after its member hit there, N in + * the whole pass, its own CRC read at once. Neither walk lets the + * readers read ahead. Test only: files and databases go in name + * order, the second 7z's lookup waits for the first's grant to be + * taken, and each walk for the readers to begin every read granted + * before it, then gives them time to begin more. */ +static void ahead_walk_none_lane(const char *root, const char *pl_dir) +{ + static const char *const what[2] = { + "read ahead: none for a walk after a member hit", + "read ahead: none for a walk in the whole pass" }; +#ifdef HAVE_7ZIP + static const char *const roles[2] = { + "SSHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "SSNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" }; + char detail[160], lpl[1024]; + bool ok; + unsigned k; + + for (k = 0; k < 2; k++) + { + if (!readers_expected()) + { + check(1, what[k], "no readers"); + continue; + } + sprintf(lpl, "%s/Test Cart Gate.lpl", pl_dir); + filestream_delete(lpl); + walk_name = "Test MF Arcade"; + walk_want = 1 + ((2 < SCAN_READER_AHEAD_MAX) + ? 2 : SCAN_READER_AHEAD_MAX); + walk_more = 0; + walk_late = false; + walk_from = -1; + walk_to = -1; + look_crc[0] = 0x0CB00001u; + look_want[0] = 1; + look_n = 1; + db_order = 1; + content_sorted = true; + ok = ah_scan(root, pl_dir, + k ? "walk_whole" : "walk_hit", roles[k], 0, 0, 0, 0) + && (k || file_contains(lpl, "AH gate")); + look_n = 0; + db_order = 0; + content_sorted = false; + walk_name = NULL; + walk_job = NULL; + filestream_delete(lpl); + sprintf(detail, "readers %d as it began, %d as it ended%s", + walk_from, walk_to, walk_late ? ", walk late" : ""); + check( ok && !walk_late && !retro_atomic_load_acquire_int(&look_late) + && walk_from == walk_want && walk_to == walk_from, what[k], + detail); + } +#else + check(1, what[0], "no 7z"); + check(1, what[1], "no 7z"); +#endif +} + +#define SE_SHARED 33 + +/* A file that errors leaves nothing to the next: a cue whose serial + * query fails (more records share its serial than the index holds, so + * it takes the query), then an archive matched by its member, which + * must be written as a crc match, not under the cue's serial. */ +static void serial_error_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static const char *const str[] = { + "PSX.EXE;1", "SLUS_001.52;1", "SYSTEM.CNF;1", + "BOOT = cdrom:\\SLUS_001.52;1" + }; + static const unsigned at[] = { 52407, 52517, 52577, 54120 }; + static uint8_t d[MF_MEMBER_LEN]; + char db[512], dir[512], p[1024], lpl[1024], what[64]; + char detail[64], crc[16]; + db_rec_t disc[SE_SHARED], cart; + zip_entry_t e; + unsigned i; + int fails; + bool started; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db, "%s/serial_error_db", root); + sprintf(dir, "%s/serial_error", root); + path_mkdir(db); + path_mkdir(dir); + + sprintf(p, "%s/a_err.bin", dir); + if (!write_ps1_track(p, str, at, ARRAY_SIZE(str), 0x5E1AE000u)) + goto fixture; + sprintf(p, "%s/a_err.cue", dir); + if (!write_cue(p, "a_err.bin")) + goto fixture; + for (i = 0; i < SE_SHARED; i++) + { + set_rec(&disc[i], "SE Shared", 0x5E1AE100u + i, PS1_TRACK_SIZE); + disc[i].serial = "SLUS-00152"; + } + sprintf(p, "%s/Test SE Disc.rdb", db); + if (!write_db_set(p, disc, SE_SHARED)) + goto fixture; + + fill_forced(d, MF_MEMBER_LEN, 59, 0x0BF1E001u); + memset(&e, 0, sizeof(e)); + e.name = "cart.rd"; + e.data = d; + e.len = MF_MEMBER_LEN; + e.crc = 0x0BF1E001u; + sprintf(p, "%s/b_cart.zip", dir); + if (!write_zip_entries(p, &e, 1)) + goto fixture; + set_rec(&cart, "SE Cart", 0x0BF1E001u, MF_MEMBER_LEN); + sprintf(p, "%s/Test MF Cart.rdb", db); + if (!write_db_set(p, &cart, 1)) + goto fixture; + + sprintf(lpl, "%s/Test MF Cart.lpl", pl_dir); + filestream_delete(lpl); + use_queue(threaded); + use_db(db); + retro_atomic_store_release_int(&hex_fails, 0); + hex_fail = "SLUS-00152"; + started = run_scan(dir) && scan_completed; + hex_fail = NULL; + fails = retro_atomic_load_acquire_int(&hex_fails); + + sprintf(crc, "%08X|crc", (unsigned)cart.crc); + sprintf(detail, "%d failed queries", fails); + sprintf(what, "after an error: the next file by its CRC %s", q); + check( started && fails == 1 && file_contains(lpl, "SE Cart") + && file_contains(lpl, crc) && !file_contains(lpl, "|serial"), + what, detail); + return; + +fixture: + check(0, "fixture", "could not write the serial error set"); +} + +/* First-match mode: two archives matched only by their own CRCs, in + * the database listed second. The first match moves that database to + * the front; the other archive's member pass then misses it, and its + * whole pass must still try it rather than stop at the first-match + * exit. */ +static void first_match_whole_lane(const char *root, const char *pl_dir, + bool threaded) +{ + static const char *const db_names[2] = { "Test FM One", "Test FM Two" }; + static uint8_t d[2][MF_MEMBER_LEN]; + char db[512], dir[512], p[1024], lpl[1024], what[64]; + db_rec_t recs[2]; + zip_entry_t e; + struct string_list *dbs; + unsigned i, second; + bool started; + const char *q = threaded ? "(threaded)" : "(regular)"; + + sprintf(db, "%s/first_match_db", root); + sprintf(dir, "%s/first_match", root); + path_mkdir(db); + path_mkdir(dir); + /* Both databases first, so the listing below is the scanner's. */ + for (i = 0; i < 2; i++) + { + sprintf(p, "%s/%s.rdb", db, db_names[i]); + if (!write_db(p, "FM Other", 0x0BF200FFu, 64)) + goto fixture; + } + if ( !(dbs = dir_list_new(db, "rdb", false, false, false, false)) + || dbs->size != 2) + { + if (dbs) + string_list_free(dbs); + goto fixture; + } + second = strstr(dbs->elems[1].data, db_names[1]) ? 1 : 0; + string_list_free(dbs); + + for (i = 0; i < 2; i++) + { + static const char *const names[2] = { "FM Whole 1", "FM Whole 2" }; + + fill_forced(d[i], MF_MEMBER_LEN, 59 + 2 * i, 0x0BF200A1u + i); + memset(&e, 0, sizeof(e)); + e.name = "m.rd"; + e.data = d[i]; + e.len = MF_MEMBER_LEN; + e.crc = 0x0BF200A1u + i; + sprintf(p, "%s/fm_%u.zip", dir, i + 1); + if ( !write_zip_entries(p, &e, 1) + || !set_whole_rec(&recs[i], names[i], p)) + goto fixture; + } + sprintf(p, "%s/%s.rdb", db, db_names[second]); + if (!write_db_set(p, recs, 2)) + goto fixture; + + sprintf(lpl, "%s/%s.lpl", pl_dir, db_names[second]); + filestream_delete(lpl); + use_queue(threaded); + use_db(db); + manual_content_scan_set_menu_scan_method( + MANUAL_CONTENT_SCAN_METHOD_CUSTOM); + manual_content_scan_set_menu_scan_db_select( + MANUAL_CONTENT_SCAN_SELECT_DB_AUTO_FIRST_MATCH, NULL); + started = run_scan(dir) && scan_completed; + manual_content_scan_set_menu_scan_db_select( + MANUAL_CONTENT_SCAN_SELECT_DB_AUTO, NULL); + manual_content_scan_set_menu_scan_method( + MANUAL_CONTENT_SCAN_METHOD_AUTOMATIC); + + sprintf(what, "first match: both archives by their own CRC %s", q); + check( started && file_contains(lpl, "FM Whole 1") + && file_contains(lpl, "FM Whole 2"), what, db_names[second]); + return; + +fixture: + check(0, "fixture", "could not write the first-match set"); +} + #define CUE_TRACK_SIZE 5000 /* No reader is sent a cue's or a gdi's track: they sort first, and @@ -2593,8 +5186,8 @@ int main(int argc, char **argv) * is the only coverage of that path. */ const uint32_t crc_c = 0x0C0FFEE0u; const uint32_t sz_c = 6144; - /* Inside an archive, so the scanner has to expand it and build a - * "archive#member" path for the entry. */ + /* Inside an archive: the CRC its directory stores for the member, + * which the scanner matches without expanding the archive. */ const uint32_t crc_d = 0xD15CD15Cu; const uint32_t sz_d = 2048; /* Claimed by a second core that supports only ".gam", so the @@ -2672,7 +5265,7 @@ int main(int argc, char **argv) } /* The archive itself checksums to something else entirely, so a - * match can only come from expanding it. */ + * match can only come from its member's stored CRC. */ { uint8_t *d = (uint8_t*)malloc(sz_d); uint32_t i; @@ -2759,7 +5352,8 @@ int main(int argc, char **argv) "display_name = \"Reader Test\"\n" "corename = \"ReaderTest\"\n" "supported_extensions = \"rd|zip|cue|bin\"\n" - "database = \"Test Reader|Test Order One|Test Order Two|Test Mix|Test Switch\"\n"); + "database = \"Test Reader|Test Order One|Test Order Two|Test Mix|Test Switch|" + "Test MF Cart|Test MF Arcade|Test FM One|Test FM Two\"\n"); fclose(f); } sprintf(p, "%s/reader_libretro.so", core_dir); @@ -2778,6 +5372,33 @@ int main(int argc, char **argv) } sprintf(p, "%s/archive_libretro.so", core_dir); { FILE *f = fopen(p, "wb"); if (f) { fputs("\177ELF", f); fclose(f); } } + /* Cartridge cores list their ROMs' extensions, most without zip; + * arcade cores list zip. */ + { + static const char *const gate[3][3] = { + { "cartgate", "ct", "Test Cart Gate" }, + { "cartzip", "cz|zip", "Test Cart Zip" }, + { "arcgate", "zip", "Test Arc Gate" } }; + unsigned g; + + for (g = 0; g < 3; g++) + { + FILE *f; + + sprintf(p, "%s/%s_libretro.info", info_dir, gate[g][0]); + if (!(f = fopen(p, "w"))) + { check(0, "fixture", "could not write core info"); return 1; } + fprintf(f, + "display_name = \"%s\"\n" + "corename = \"%s\"\n" + "supported_extensions = \"%s\"\n" + "database = \"%s\"\n", + gate[g][0], gate[g][0], gate[g][1], gate[g][2]); + fclose(f); + sprintf(p, "%s/%s_libretro.so", core_dir, gate[g][0]); + if ((f = fopen(p, "wb"))) { fputs("\177ELF", f); fclose(f); } + } + } #ifdef HAVE_THREADS task_queue_init(true, msgq_push); @@ -2792,8 +5413,8 @@ int main(int argc, char **argv) strlcpy(config_get_ptr()->paths.path_content_database, db_dir, sizeof(config_get_ptr()->paths.path_content_database)); - /* Without this the scanner never opens an archive, and the entry - * inside is invisible. */ + /* Without this the scanner never expands an archive that matches + * nothing, and the entries inside are invisible. */ { bool *search_archives = manual_content_scan_get_search_archives_ptr(); if (search_archives) @@ -2833,10 +5454,8 @@ int main(int argc, char **argv) sprintf(p, "%s/Test Zip.lpl", pl_dir); check(file_contains(p, "Zipped The Game"), "archive member matched", "Zipped The Game"); - /* The member is what gets checksummed and matched, but the entry - * records the archive: that is the path a frontend loads. Assert - * it rather than the "archive#member" form the scan builds - * internally, which does not reach the playlist. */ + /* The member's stored CRC is what matches, but the entry records + * the archive: that is the path a frontend loads. */ check( file_contains(p, "04_bundle.zip") && !file_contains(p, "#inner.bin"), "entry records the archive, not the member", "04_bundle.zip"); @@ -3601,6 +6220,42 @@ int main(int argc, char **argv) mix_lane(root, pl_dir, false); archive_lane(root, pl_dir, true); archive_lane(root, pl_dir, false); + member_first_lane(root, pl_dir, true); + member_first_lane(root, pl_dir, false); + cart_gate_lane(root, pl_dir, true); + cart_gate_lane(root, pl_dir, false); + base_paths_lane(root, pl_dir, true); + base_paths_lane(root, pl_dir, false); + arc_gate_lane(root, pl_dir, true); + arc_gate_lane(root, pl_dir, false); + arc_cart_first_lane(root, pl_dir, true); + arc_cart_first_lane(root, pl_dir, false); + arc_folder_lane(root, pl_dir, true); + arc_folder_lane(root, pl_dir, false); + unindexed_plain_lane(root, pl_dir); + reader_fail_arcade_lane(root, pl_dir); + unindexed_member_lane(root, pl_dir); + sibling_lane(root, pl_dir, true); + sibling_lane(root, pl_dir, false); + dual_match_lane(root, pl_dir, true); + dual_match_lane(root, pl_dir, false); + archives_off_lane(root, pl_dir, true); + archives_off_lane(root, pl_dir, false); + first_match_whole_lane(root, pl_dir, true); + first_match_whole_lane(root, pl_dir, false); + serial_error_lane(root, pl_dir, true); + serial_error_lane(root, pl_dir, false); + whole_ahead_lane(root, pl_dir); + hit_clears_lane(root, pl_dir); + rearm_lane(root, pl_dir); + ahead_lone_lane(root, pl_dir); + ahead_double_lane(root, pl_dir); + ahead_streak_lane(root, pl_dir); + ahead_next_lane(root, pl_dir); + ahead_hit_count_lane(root, pl_dir); + ahead_arcade_lane(root, pl_dir); + ahead_walk_lane(root, pl_dir); + ahead_walk_none_lane(root, pl_dir); cue_lane(root, reader_db, false); cue_lane(root, reader_db, true); strlcpy(config_get_ptr()->paths.path_content_database, reader_db, diff --git a/samples/tasks/database/spend_window.h b/samples/tasks/database/spend_window.h index 0ea3f87a57..bc9fb14da7 100644 --- a/samples/tasks/database/spend_window.h +++ b/samples/tasks/database/spend_window.h @@ -10,15 +10,15 @@ * window does not hold: it resets on the wall clock, and a reset * between the charge and the scanner's open grants the whole window. */ static bool window_spend_next; -/* Called after each open, when set. */ +/* Called before each open, when set. */ static void (*window_open_hook)(void); void __real_task_nbio_slice_open(nbio_budget_t *b); void __wrap_task_nbio_slice_open(nbio_budget_t *b) { - __real_task_nbio_slice_open(b); if (window_open_hook) window_open_hook(); + __real_task_nbio_slice_open(b); if (window_spend_next) { window_spend_next = false; diff --git a/tasks/task_database.c b/tasks/task_database.c index 7a529b8b64..719c0a0dee 100644 --- a/tasks/task_database.c +++ b/tasks/task_database.c @@ -177,7 +177,10 @@ enum db_state_flags_enum * walks a time. */ DB_STATE_FLAG_SIZE_CHECKED = (1 << 4), /* A core claiming this database matches archive members. */ - DB_STATE_FLAG_ARCHIVE_MEMBER = (1 << 5) + DB_STATE_FLAG_ARCHIVE_MEMBER = (1 << 5), + /* Its crc index build was refused. The budget only shrinks, so a + * retry would walk the database again to the same answer. */ + DB_STATE_FLAG_NO_CRC_INDEX = (1 << 6) }; #ifdef HAVE_LIBRETRODB @@ -254,6 +257,8 @@ typedef struct database_state_handle uint64_t archive_size; char archive_name[512]; /* TODO/FIXME - check size */ char serial[4096]; /* TODO/FIXME - check size */ + /* The first member's extension, which gates the member pass. */ + char member_ext[16]; /* One entry per database in 'list', allocated to list->size: * list_index is bounded only by list->size - the number of .rdb * files in the database directory - so a fixed @@ -294,6 +299,13 @@ typedef struct database_state_handle /* The index build in flight, run by task_database_index_step() * until index_ready, then taken by the lookup that began it. */ database_info_index_job_t *index_job; + /* The lookup's pass, and whether archive_crc holds the archive's + * own CRC or its read is under way. */ + uint8_t crc_pass; /* enum db_crc_pass */ + bool whole_done; + /* The member pass hit where only member_ext is claimed; it expands + * unless one claimed for the archive's own name matches too. */ + bool member_hit; bool index_ready; } database_state_handle_t; @@ -311,6 +323,18 @@ enum db_crc_fail DB_CRC_FAIL_ZERO /* the CRC is 0, so no entry matches */ }; +/* What the CRC lookup compares. A one-member archive is looked up by + * its member's CRC, then read whole and looked up by its own only if + * that misses. An archive of several members, an arcade set as a + * rule, is read whole first and each database takes either CRC. */ +enum db_crc_pass +{ + DB_CRC_PASS_FILE = 0, /* crc: any other name */ + DB_CRC_PASS_MEMBER, /* crc, the first member's */ + DB_CRC_PASS_WHOLE, /* archive_crc */ + DB_CRC_PASS_BOTH /* crc or archive_crc: several members */ +}; + /* None on Wii U: its newlib and libfat take no locks, and without * libfat its SD layer serialises every call. None on Vita: Vita3K's * sceIo tables take no lock. */ @@ -337,6 +361,11 @@ enum db_crc_fail #ifndef SCAN_READER_CUTOFF #define SCAN_READER_CUTOFF (128 * 1024 * 1024) #endif +/* Archives the readers may read whole ahead of the scan at most. Each + * member miss in a row grants 1, 2, 4 ... more. */ +#ifndef SCAN_READER_AHEAD_MAX +#define SCAN_READER_AHEAD_MAX 16 +#endif #ifdef SCAN_HAVE_READERS enum scan_slot_state @@ -345,19 +374,27 @@ enum scan_slot_state SCAN_SLOT_READY, /* for a reader to claim */ SCAN_SLOT_IN_FLIGHT, SCAN_SLOT_OK, + SCAN_SLOT_MEMBER_OK, /* an archive's member CRC, not read whole */ SCAN_SLOT_BIG, SCAN_SLOT_FAIL }; /* A file handed out. The scan writes it while EMPTY, its reader while - * IN_FLIGHT, and each publishes with the state. */ + * IN_FLIGHT, and each publishes with the state. For an archive the + * reader takes member_crc and member_size first: its first member's + * CRC and size. */ typedef struct scan_slot { uint64_t size; + uint64_t member_size; size_t index; /* in the content list */ char *path; /* PATH_MAX_LENGTH, its own copy */ uint32_t crc; + uint32_t member_crc; retro_atomic_int_t state; /* enum scan_slot_state */ + char member_ext[16]; + bool archive; + bool multi; /* an archive of several members */ } scan_slot_t; /* Shared by the scan and its readers; the last reference frees it. @@ -372,6 +409,7 @@ typedef struct scan_readers unsigned seq; /* scan: slots filled */ unsigned take; /* scan: slots given back */ unsigned count; /* scan: readers pushed */ + unsigned grant; /* scan: the last miss's, 0 after a hit */ uint32_t ident; /* the scan's */ retro_atomic_int_t filled; /* seq, for the readers */ retro_atomic_int_t claim; /* the next slot to claim */ @@ -379,6 +417,12 @@ typedef struct scan_readers retro_atomic_int_t live; /* readers not yet gone */ retro_atomic_int_t quit; retro_atomic_int_t refs; + /* Archives the readers may still read whole ahead, beyond their + * member CRCs: granted when an archive misses by its member CRC, + * taken back when one matches by it. A hint: each slot's state + * says what it holds. */ + retro_atomic_int_t whole_ahead; + retro_atomic_int_t rearm; /* the next slot to take back */ bool off; /* scan: the readers are gone or going */ } scan_readers_t; @@ -387,6 +431,7 @@ typedef struct scan_reader { scan_readers_t *blk; scan_slot_t *slot; /* the file in hand */ + scan_slot_t *own; /* its last MEMBER_OK */ intfstream_t *fd; int64_t left; uint32_t acc; @@ -1060,6 +1105,21 @@ static void task_database_crc_close(database_state_handle_t *db_state) db_state->crc_done = false; } +/* No serial, CRC, size or pass of one file is left for the next. */ +static void task_database_crc_reset(database_state_handle_t *db_state) +{ + task_database_crc_close(db_state); + db_state->serial[0] = '\0'; + db_state->crc = 0; + db_state->archive_crc = 0; + db_state->size = 0; + db_state->archive_size = 0; + db_state->crc_pass = DB_CRC_PASS_FILE; + db_state->whole_done = false; + db_state->member_hit = false; + db_state->member_ext[0] = '\0'; +} + /* Opens @name's CRC span for task_database_crc_step() to hash over * the steps that follow, and records its size. */ static bool task_database_crc_begin(database_state_handle_t *db_state, @@ -1176,12 +1236,62 @@ static bool scan_reader_eligible(const char *path) return false; } +/* One of the whole reads ahead the scan granted, if any are left. */ +static bool scan_reader_credit(scan_readers_t *blk) +{ + int n; + + while ((n = retro_atomic_load_acquire_int(&blk->whole_ahead)) > 0) + if (retro_atomic_cas_int(&blk->whole_ahead, n, n - 1)) + return true; + return false; +} + +/* While whole reads ahead are granted, a slot that holds only a member + * CRC, taken back to read whole: the next from the scan's on, in + * order, then this reader's last (it may have missed the grant). */ +static bool scan_reader_rearm(scan_reader_t *rd) +{ + scan_readers_t *blk = rd->blk; + + while (retro_atomic_load_acquire_int(&blk->whole_ahead) > 0) + { + scan_slot_t *slot = rd->own; + int k = retro_atomic_load_acquire_int(&blk->rearm); + + if (k < retro_atomic_load_acquire_int(&blk->claim)) + { + if (!retro_atomic_cas_int(&blk->rearm, k, k + 1)) + continue; + slot = &blk->win[(size_t)k % blk->window]; + } + else if (!slot) + return false; + else + rd->own = NULL; + if (!retro_atomic_cas_int(&slot->state, SCAN_SLOT_MEMBER_OK, + SCAN_SLOT_IN_FLIGHT)) + continue; + /* The slot first, then a credit: with none left it goes back. */ + if (scan_reader_credit(blk)) + { + rd->slot = slot; + return true; + } + retro_atomic_store_release_int(&slot->state, SCAN_SLOT_MEMBER_OK); + task_queue_wake(blk->ident); + } + return false; +} + /* The next filled slot, in order; one the scan took back is passed - * over. False when none is filled. */ + * over. One to take back comes first. False when none is filled. */ static bool scan_reader_claim(scan_reader_t *rd) { scan_readers_t *blk = rd->blk; + if (scan_reader_rearm(rd)) + return true; for (;;) { int c = retro_atomic_load_acquire_int(&blk->claim); @@ -1281,6 +1391,27 @@ static void scan_reader_handler(retro_task_t *task) rd->parked = false; retro_atomic_fetch_sub_int(&blk->idle, 1); } + /* One taken back has its member CRC. */ + if (rd->slot->archive && !rd->slot->member_crc) + { + unsigned entries = 0; + + rd->slot->member_crc = file_archive_get_file_crc32_size_and_ext( + rd->slot->path, &rd->slot->member_size, + rd->slot->member_ext, sizeof(rd->slot->member_ext), + &entries); + rd->slot->multi = entries > 1; + /* Only the member's CRC unless a whole read is granted; the + * scan needs the archive whole when it has several members + * or no member CRC. */ + if ( rd->slot->member_crc && !rd->slot->multi + && !scan_reader_credit(blk)) + { + rd->own = rd->slot; + scan_reader_publish(rd, SCAN_SLOT_MEMBER_OK); + return; + } + } if (!(rd->fd = task_database_crc_open(rd->slot->path, TASK_DATABASE_CRC_FILE, &rd->left, &size))) { @@ -1341,6 +1472,8 @@ static void scan_readers_start(retro_task_t *task, retro_atomic_int_init(&blk->claim, 0); retro_atomic_int_init(&blk->idle, 0); retro_atomic_int_init(&blk->quit, 0); + retro_atomic_int_init(&blk->whole_ahead, 0); + retro_atomic_int_init(&blk->rearm, 0); retro_atomic_int_init(&blk->live, (int)n); retro_atomic_int_init(&blk->refs, (int)n + 1); for (r = 0; r < n; r++) @@ -1438,9 +1571,14 @@ static void scan_readers_fill(manual_scan_handle_t *manual_scan) { scan_slot_t *slot = &blk->win[blk->seq % blk->window]; - slot->index = blk->fill; - slot->size = 0; - slot->crc = 0; + slot->index = blk->fill; + slot->size = 0; + slot->crc = 0; + slot->member_crc = 0; + slot->member_ext[0] = '\0'; + slot->multi = false; + slot->archive = extension_to_file_type(path_get_extension(path)) + == FILE_TYPE_COMPRESSED; /* A copy: the list moves and is pruned under the readers. */ strlcpy(slot->path, path, PATH_MAX_LENGTH); retro_atomic_store_release_int(&slot->state, SCAN_SLOT_READY); @@ -1488,14 +1626,34 @@ static enum scan_take scan_readers_take(manual_scan_handle_t *manual_scan, return SCAN_TAKE_WAIT; state = SCAN_SLOT_EMPTY; } + /* A reader may take it back to read whole. */ + if ( state == SCAN_SLOT_MEMBER_OK + && !retro_atomic_cas_int(&slot->state, SCAN_SLOT_MEMBER_OK, + SCAN_SLOT_EMPTY)) + return SCAN_TAKE_WAIT; blk->take++; - if (slot->index == i && state == SCAN_SLOT_OK) + if ( slot->index == i + && ( state == SCAN_SLOT_OK || state == SCAN_SLOT_MEMBER_OK + || (slot->archive && slot->member_crc))) { - if (extension_to_file_type(path_get_extension(name)) - == FILE_TYPE_COMPRESSED) + if (slot->archive) { - db_state->archive_crc = slot->crc; - db_state->archive_size = slot->size; + db_state->crc = slot->member_crc; + db_state->size = slot->member_size; + db_state->crc_pass = slot->multi + ? DB_CRC_PASS_BOTH : DB_CRC_PASS_MEMBER; + strlcpy(db_state->member_ext, slot->member_ext, + sizeof(db_state->member_ext)); + /* BIG or FAIL: the scan reads it whole itself. */ + if (state == SCAN_SLOT_OK) + { + db_state->archive_crc = slot->crc; + db_state->archive_size = slot->size; + db_state->whole_done = true; + /* No member CRC: only its own can match. */ + if (!slot->member_crc) + db_state->crc_pass = DB_CRC_PASS_WHOLE; + } } else { @@ -1517,6 +1675,65 @@ static enum scan_take scan_readers_take(manual_scan_handle_t *manual_scan, blk->fill = i + 1; return SCAN_TAKE_NONE; } + +/* Adds whole reads ahead, up to SCAN_READER_AHEAD_MAX. With some + * granted, readers also take back what they have only the member CRC + * of, from the scan's next file on; any parked on a full window + * wake. */ +static void scan_readers_grant(scan_readers_t *blk, int add) +{ + int had, n; + + /* Readers only take: a count of 0 stays 0 until this adds. */ + do + { + had = retro_atomic_load_acquire_int(&blk->whole_ahead); + if (had >= SCAN_READER_AHEAD_MAX) + return; + n = had + add; + if (n > SCAN_READER_AHEAD_MAX) + n = SCAN_READER_AHEAD_MAX; + /* Before the count: a reader that sees it sees the cursor. */ + if (!had) + retro_atomic_store_release_int(&blk->rearm, (int)blk->take); + } while (!retro_atomic_cas_int(&blk->whole_ahead, had, n)); + if (!had) + scan_readers_wake(blk); +} + +/* A member miss grants the readers whole reads ahead, twice the last + * miss's, up to SCAN_READER_AHEAD_MAX; a member hit takes back what is + * left. */ +static void scan_readers_whole_ahead(manual_scan_handle_t *manual_scan, + int miss) +{ + scan_readers_t *blk = manual_scan->readers; + + if (!blk) + return; + if (!miss) + { + blk->grant = 0; + if (retro_atomic_load_acquire_int(&blk->whole_ahead)) + retro_atomic_store_release_int(&blk->whole_ahead, 0); + return; + } + blk->grant = blk->grant ? blk->grant * 2 : 1; + if (blk->grant > SCAN_READER_AHEAD_MAX) + blk->grant = SCAN_READER_AHEAD_MAX; + scan_readers_grant(blk, (int)blk->grant); +} + +/* While the scan walks a database in a member pass mid miss streak, + * the readers may read whole up to the cap. The streak's grant is + * left as it is. */ +static void scan_readers_top_up(manual_scan_handle_t *manual_scan) +{ + scan_readers_t *blk = manual_scan->readers; + + if (blk && blk->grant) + scan_readers_grant(blk, SCAN_READER_AHEAD_MAX); +} #endif static void task_database_index_close(database_state_handle_t *db_state) @@ -1577,11 +1794,9 @@ static int task_database_iterate_playlist( { case FILE_TYPE_COMPRESSED: #ifdef HAVE_COMPRESSION - db->type = DATABASE_TYPE_CRC_LOOKUP; - /* first check crc of archive itself */ - return task_database_crc_begin(db_state, name, - TASK_DATABASE_CRC_FILE, DB_CRC_DEST_ARCHIVE, - DB_CRC_FAIL_ERROR); + db->type = DATABASE_TYPE_CRC_LOOKUP; + db_state->crc_pass = DB_CRC_PASS_MEMBER; + break; #else break; #endif @@ -1748,16 +1963,7 @@ static enum scan_verdict database_info_list_iterate_end_no_match( db_state->list_index = 0; db_state->entry_index = 0; - db_state->size = 0; - db_state->archive_size = 0; - db_state->serial[0] = '\0'; - task_database_crc_close(db_state); - - if (db_state->crc != 0) - db_state->crc = 0; - - if (db_state->archive_crc != 0) - db_state->archive_crc = 0; + task_database_crc_reset(db_state); return archive_added ? SCAN_VERDICT_ARCHIVE_CONTENTS_ADDED : SCAN_VERDICT_NO_DB_MATCH; } @@ -1917,16 +2123,15 @@ static enum scan_verdict database_info_list_iterate_found_match( archive_name)) RARCH_ERR("[Scanner] Failed to add result for: \"%s\".\n", entry_lbl); +#ifdef SCAN_HAVE_READERS + if (db_state->crc_pass == DB_CRC_PASS_MEMBER) + scan_readers_whole_ahead(_db, 0); +#endif database_info_list_free(db_state->info); free(db_state->info); db_state->info = NULL; - db_state->crc = 0; - db_state->archive_crc = 0; - db_state->size = 0; - db_state->archive_size = 0; - db_state->serial[0] = '\0'; - task_database_crc_close(db_state); + task_database_crc_reset(db_state); /* Move database to start since we are likely to match against it again */ @@ -2074,13 +2279,18 @@ static bool task_database_fill_db_min_max(database_state_handle_t *db_state, * lookup reads the database once. */ if ( rdb && *rdb && !db_state->crc_index[db_state->list_index] + && !(db_state->flags[db_state->list_index] + & DB_STATE_FLAG_NO_CRC_INDEX) && db_state->index_budget) { database_info_crc_index_t *ci = NULL; if (!task_database_index_take(db_state, &ci, serial, &has_serial)) return false; - if (ci) + if (!ci) + db_state->flags[db_state->list_index] |= + DB_STATE_FLAG_NO_CRC_INDEX; + else { size_t used = database_info_crc_index_bytes(ci); db_state->index_budget = (used < db_state->index_budget) @@ -2185,10 +2395,26 @@ static bool task_database_fill_db_min_max(database_state_handle_t *db_state, return true; } +/* An installed core claims the current database for @ext; one that + * matches archive members, only for a member. */ +static bool task_database_claimed(database_state_handle_t *db_state, + const char *ext, bool member) +{ + return string_list_find_elem( + db_state->claim_exts[db_state->list_index], ext) + && ( member + || !(db_state->flags[db_state->list_index] + & DB_STATE_FLAG_ARCHIVE_MEMBER)); +} + /* Passes over the run of databases, from the current one, that no - * installed core claims @name for. True when it passed over any. */ + * installed core claims @name for, nor @member_ext when given and the + * database has or may still get a crc index: unindexed, a member + * lookup is a whole walk the expansion repeats. True when it passed + * over any. */ static bool task_database_skip_unclaimed(database_state_handle_t *db_state, - const char *name, bool path_contains_compressed_file) + const char *name, const char *member_ext, + bool path_contains_compressed_file) { /* Every database the gate refuses is passed over in this one * step, so a file costs a step per database it can match rather @@ -2197,11 +2423,14 @@ static bool task_database_skip_unclaimed(database_state_handle_t *db_state, bool skipped = false; while ( db_state->list_index < db_state->list->size - && ( !string_list_find_elem( - db_state->claim_exts[db_state->list_index], ext) - || ( !path_contains_compressed_file - && (db_state->flags[db_state->list_index] - & DB_STATE_FLAG_ARCHIVE_MEMBER)))) + && !task_database_claimed(db_state, ext, + path_contains_compressed_file) + && !( member_ext + && task_database_claimed(db_state, member_ext, true) + && ( db_state->crc_index[db_state->list_index] + || ( !(db_state->flags[db_state->list_index] + & DB_STATE_FLAG_NO_CRC_INDEX) + && db_state->index_budget)))) { database_info_list_iterate_next(db_state); skipped = true; @@ -2210,6 +2439,43 @@ static bool task_database_skip_unclaimed(database_state_handle_t *db_state, return skipped; } +/* The member pass missed, or its first member has no CRC: the + * databases again from the first, for the archive's own CRC. The + * steps that follow read the archive whole unless a reader has. */ +static enum scan_verdict task_database_whole_pass( + database_state_handle_t *db_state, const char *name) +{ + database_info_list_free(db_state->info); + free(db_state->info); + db_state->info = NULL; + db_state->list_index = 0; + db_state->entry_index = 0; + db_state->crc_pass = DB_CRC_PASS_WHOLE; + if ( !db_state->whole_done + && !task_database_crc_begin(db_state, name, TASK_DATABASE_CRC_FILE, + DB_CRC_DEST_ARCHIVE, DB_CRC_FAIL_ERROR)) + return SCAN_VERDICT_ERROR; + return SCAN_VERDICT_CONTINUE; +} + +/* Found by its first member where the archive's own name would not + * be looked up: matched as the members the expansion adds, without + * reading the archive whole. */ +static enum scan_verdict task_database_member_hit( + manual_scan_handle_t *_db, database_state_handle_t *db_state, + database_info_handle_t *db, const char *name, + bool path_contains_compressed_file) +{ +#ifdef SCAN_HAVE_READERS + scan_readers_whole_ahead(_db, 0); +#endif + database_info_list_free(db_state->info); + free(db_state->info); + db_state->info = NULL; + return database_info_list_iterate_end_no_match(_db, db, db_state, name, + path_contains_compressed_file); +} + static enum scan_verdict task_database_iterate_crc_lookup( manual_scan_handle_t *_db, database_state_handle_t *db_state, @@ -2223,25 +2489,58 @@ static enum scan_verdict task_database_iterate_crc_lookup( || ( _db->flags & DB_HANDLE_FLAG_USE_FIRST_MATCH_ONLY && db_state->list_index > 0 && db_state->flags[0] & DB_STATE_FLAG_MATCHED)) + { + if (db_state->list && db_state->crc_pass == DB_CRC_PASS_MEMBER) + { + if (db_state->member_hit) + return task_database_member_hit(_db, db_state, db, name, + path_contains_compressed_file); +#ifdef SCAN_HAVE_READERS + scan_readers_whole_ahead(_db, 1); +#endif + return task_database_whole_pass(db_state, name); + } return database_info_list_iterate_end_no_match(_db, db, db_state, name, path_contains_compressed_file); + } /* Archive did not contain a CRC for this entry, * or the file is empty. */ - if (!db_state->crc) + if (!db_state->crc && db_state->crc_pass != DB_CRC_PASS_WHOLE) { + unsigned entries = 0; #ifdef DEBUG RARCH_DBG("[Scanner] Extra crc check 1: %x %d / %x %d %s\n", db_state->crc, db_state->size, db_state->archive_crc, db_state->archive_size, path_contains_compressed_file ? "compressed:true" : "compressed:false"); #endif - db_state->crc = file_archive_get_file_crc32_and_size(name, &db_state->size); + db_state->crc = file_archive_get_file_crc32_size_and_ext(name, + &db_state->size, db_state->member_ext, + sizeof(db_state->member_ext), &entries); #ifdef DEBUG RARCH_DBG("[Scanner] Extra crc check 2: %x %d / %x %d.\n", db_state->crc, db_state->size, db_state->archive_crc, db_state->archive_size); #endif if (!db_state->crc) + { + /* No CRC for the first member (a directory, or a decode that + * failed): only the archive's own CRC can match. */ + if (db_state->crc_pass == DB_CRC_PASS_MEMBER) + return task_database_whole_pass(db_state, name); return database_info_list_iterate_next(db_state); + } + if (entries > 1 && db_state->crc_pass == DB_CRC_PASS_MEMBER) + db_state->crc_pass = DB_CRC_PASS_BOTH; + } + + /* Several members: read whole before the first lookup. */ + if (db_state->crc_pass == DB_CRC_PASS_BOTH && !db_state->whole_done) + { + db_state->whole_done = true; + if (!task_database_crc_begin(db_state, name, TASK_DATABASE_CRC_FILE, + DB_CRC_DEST_ARCHIVE, DB_CRC_FAIL_ERROR)) + return SCAN_VERDICT_ERROR; + return SCAN_VERDICT_CONTINUE; } /* The core-info gate is an in-memory string check; the size gate @@ -2251,9 +2550,15 @@ static enum scan_verdict task_database_iterate_crc_lookup( * database in the directory was read end to end once per scan, * including the ones no installed core claims and which can never * match - 200 MB of .rdb reads to scan a 70 MB set with the - * shipped database directory. Ask the cheap question first. */ + * shipped database directory. Ask the cheap question first. + * The member pass asks it for the first member too, as its + * expansion will, until one matches; not when nothing expands. */ if ( !(_db->flags & DB_HANDLE_FLAG_SCAN_WITHOUT_CORE_MATCH) && task_database_skip_unclaimed(db_state, name, + ( db_state->crc_pass == DB_CRC_PASS_MEMBER + && !db_state->member_hit + && _db->task_config->search_archives) + ? db_state->member_ext : NULL, path_contains_compressed_file)) return SCAN_VERDICT_CONTINUE; @@ -2264,32 +2569,25 @@ static enum scan_verdict task_database_iterate_crc_lookup( if (db_state->min_sizes[db_state->list_index] > 0) { - /* Examining zip file main entry (archive size filled, but no indication of compressed file) */ - if ( !path_contains_compressed_file && db_state->archive_size > 0) - { - if ( ( db_state->min_sizes[db_state->list_index] > (int64_t) db_state->archive_size - && db_state->min_sizes[db_state->list_index] > (int64_t) db_state->size ) - || ( db_state->max_sizes[db_state->list_index] < (int64_t) db_state->archive_size - && db_state->max_sizes[db_state->list_index] < (int64_t) db_state->size )) - { -#ifdef DEBUG - RARCH_DBG("[Scanner] Skipping DB, neither archive nor uncompressed size %ld/%ld is in range.\n", - db_state->archive_size, db_state->size); -#endif - return database_info_list_iterate_next(db_state); - } - } - /* Any other case (non-archive file, or a file inside the archive */ - else if ( db_state->size > 0 - && ( db_state->min_sizes[db_state->list_index] > (int64_t) db_state->size - || db_state->max_sizes[db_state->list_index] < (int64_t) db_state->size)) + int64_t lo = db_state->min_sizes[db_state->list_index]; + int64_t hi = db_state->max_sizes[db_state->list_index]; + uint64_t size = (db_state->crc_pass == DB_CRC_PASS_WHOLE) + ? db_state->archive_size : db_state->size; + + if ( db_state->crc_pass == DB_CRC_PASS_BOTH + && db_state->archive_size >= (uint64_t)lo + && db_state->archive_size <= (uint64_t)hi) + size = db_state->archive_size; + if ( size > 0 + && ( db_state->min_sizes[db_state->list_index] > (int64_t) size + || db_state->max_sizes[db_state->list_index] < (int64_t) size)) { #ifdef DEBUG - RARCH_DBG("[Scanner] Skipping DB, file size %ld not in range.\n", db_state->size); + RARCH_DBG("[Scanner] Skipping DB, file size %ld not in range.\n", size); #endif return database_info_list_iterate_next(db_state); } - else if (db_state->size == 0) + else if (size == 0) { #ifdef DEBUG RARCH_DBG("[Scanner] Zero-length file, skipping database match\n"); @@ -2305,6 +2603,8 @@ static enum scan_verdict task_database_iterate_crc_lookup( const char *rdb = db_state->list->elems[ db_state->list_index].data; database_info_list_t *hits = NULL; + uint32_t crc = (db_state->crc_pass == DB_CRC_PASS_WHOLE) + ? db_state->archive_crc : db_state->crc; query[0] = '\0'; @@ -2321,13 +2621,18 @@ static enum scan_verdict task_database_iterate_crc_lookup( if (rdb && *rdb) { if ( !db_state->crc_index[db_state->list_index] + && !(db_state->flags[db_state->list_index] + & DB_STATE_FLAG_NO_CRC_INDEX) && db_state->index_budget) { database_info_crc_index_t *ci = NULL; if (!task_database_index_take(db_state, &ci, NULL, NULL)) return SCAN_VERDICT_CONTINUE; - if (ci) + if (!ci) + db_state->flags[db_state->list_index] |= + DB_STATE_FLAG_NO_CRC_INDEX; + else { size_t used = database_info_crc_index_bytes(ci); db_state->index_budget = (used < db_state->index_budget) @@ -2339,8 +2644,8 @@ static enum scan_verdict task_database_iterate_crc_lookup( if (db_state->crc_index[db_state->list_index]) hits = database_info_list_new_crc( db_state->crc_index[db_state->list_index], rdb, - db_state->crc, db_state->archive_crc, - DB_EXTRACT_SCAN_FIELDS); + crc, (db_state->crc_pass == DB_CRC_PASS_BOTH) + ? db_state->archive_crc : 0, DB_EXTRACT_SCAN_FIELDS); } if (hits) @@ -2352,12 +2657,18 @@ static enum scan_verdict task_database_iterate_crc_lookup( } db_state->info = hits; } - else + else if (db_state->crc_pass == DB_CRC_PASS_BOTH) { snprintf(query, sizeof(query), "{crc:or(b\"%08lX\",b\"%08lX\")}", - (unsigned long)db_state->crc, + (unsigned long)crc, (unsigned long)db_state->archive_crc); + database_info_list_iterate_new(db_state, query); + } + else + { + snprintf(query, sizeof(query), "{crc:b\"%08lX\"}", + (unsigned long)crc); database_info_list_iterate_new(db_state, query); } @@ -2382,17 +2693,32 @@ static enum scan_verdict task_database_iterate_crc_lookup( database_info_t *db_info_entry = &db_state->info->list[db_state->entry_index]; - /* When scanning an archive, "first" file crc32 is also checked. */ if (db_info_entry->crc32) { - if (db_state->archive_crc == db_info_entry->crc32) - return database_info_list_iterate_found_match( - _db, - db_state, db, NULL); - if (db_state->crc == db_info_entry->crc32) + if ( db_state->crc_pass == DB_CRC_PASS_WHOLE + || db_state->crc_pass == DB_CRC_PASS_BOTH) + { + if (db_state->archive_crc == db_info_entry->crc32) + return database_info_list_iterate_found_match( + _db, + db_state, db, NULL); + } + if ( db_state->crc_pass != DB_CRC_PASS_WHOLE + && db_state->crc == db_info_entry->crc32) + { + if ( db_state->crc_pass == DB_CRC_PASS_MEMBER + && !(_db->flags & DB_HANDLE_FLAG_SCAN_WITHOUT_CORE_MATCH) + && !task_database_claimed(db_state, + path_get_extension(name), + path_contains_compressed_file)) + { + db_state->member_hit = true; + return database_info_list_iterate_next(db_state); + } return database_info_list_iterate_found_match( _db, db_state, db, archive_entry); + } } } @@ -2514,7 +2840,7 @@ static int task_database_iterate_serial_lookup( /* 3DS images, unlike discs, keep the crc lookup's core gate. */ if ( !(_db->flags & DB_HANDLE_FLAG_SCAN_WITHOUT_CORE_MATCH) && extension_to_file_type(path_get_extension(name)) == FILE_TYPE_3DS - && task_database_skip_unclaimed(db_state, name, + && task_database_skip_unclaimed(db_state, name, NULL, path_contains_compressed_file)) return SCAN_VERDICT_CONTINUE; @@ -2787,7 +3113,14 @@ static int task_database_iterate( if (db_state->crc_fd) return task_database_crc_step(db_state, b); if (db_state->index_job && !db_state->index_ready) + { +#ifdef SCAN_HAVE_READERS + if ( db_state->crc_pass == DB_CRC_PASS_MEMBER + && !db_state->member_hit) + scan_readers_top_up(_db); +#endif return task_database_index_step(db_state, b); + } #ifdef DEBUG RARCH_DBG("[Scanner] Type %d, \"%s\" against \"%s\".\n", db->type, name, database_info_get_current_name(db_state)); /* list_index == list->size is a valid state here (every database @@ -2851,7 +3184,7 @@ static void task_database_cleanup_state(database_state_handle_t *db_state) if (db_state->buf) free(db_state->buf); db_state->buf = NULL; - task_database_crc_close(db_state); + task_database_crc_reset(db_state); task_database_index_close(db_state); } #endif @@ -4162,6 +4495,7 @@ static enum scan_step task_manual_scan_iterate_step(retro_task_t *task, * same failing step every tick. Give up on this * file and go to the next one. */ RARCH_ERR("[Scanner] Scanning of content unexpectedly failed for \"%s\"\n", content_path); + task_database_crc_reset(dbstate); manual_scan->status = DATABASE_SCAN_ITERATE_NEXT; break; case SCAN_VERDICT_CONTINUE: