scan: look archives up by their first member's CRC first (#19937)

* scan: look archives up by their first member's CRC first

A zip, 7z, apk, rar, zst or bz2 was hashed whole before its first
member's CRC was fetched.  Cartridge databases key on ROM CRCs, so a
one-ROM-per-zip set always matches by the member: the whole read, the
main cost of scanning zips on a slow share, was wasted.

The databases are now walked first for the member's CRC alone.  Where
it is stored (zip, 7z, RAR with CRC32) that costs one directory read;
a zst, bz2 or BLAKE2 RAR member is decoded for it, as before.  Only
when that misses is the archive read whole, or a reader's whole result
taken, and the databases walked again for its own CRC, which the
arcade databases key on.  The archive is expanded only after both
miss.  Each walk's size check uses its own size: the member's, then
the archive's.

By default the core gate looks in a database only when a core claims
the file's extension.  Stock cartridge cores do not list zip, so a
cartridge zip was read whole and matched only once expanded.  The
first walk now also takes the databases claimed for the first member's
extension.  A hit there is held while the walk finishes the databases
claimed for the archive's name: a match in one of those is written for
the archive; otherwise the archive is expanded without the whole
read.  With "Scan Inside Archives" off nothing expands, so the first
walk keeps to the archive's name.

The parallel readers likewise start an archive with its first member's
CRC, size and extension, and hand them to the scan.  They hash
archives whole only as far as the scan grants: a member-CRC miss
grants one whole read ahead, each further miss in a row twice the
last, up to 16 at a time, and a member-CRC match takes back what is
left.  A lone unknown cartridge then costs one wasted read, while an
arcade folder soon has the readers 16 ahead.  While the scan first
walks a database's index in a member pass mid streak, the readers may
keep 16 ahead, so the walk does not leave them idle.  A grant goes
first to the nearest archive ahead that has only its member CRC.
Without a reader's whole result the scan hashes an archive it needs
whole itself, in steps, like a file over the cutoff.

An archive with no CRC for its first entry, such as a zip or 7z with
a directory first, skipped every database, so its own CRC was never
compared.  It now goes straight to the whole walk, and readers always
hash it whole.  In first-match mode the walk's early exit leads to the
whole walk too, while that has not run.

The CRCs, sizes and walk are cleared at each file's start.  A file
that errored left them to the next, and an archive after it was
looked up by the stale CRC.  The serial is cleared with them: an
archive matched after a file that errored in its serial lookup was
written under that file's serial.

Every database's member CRC is now checked before any whole CRC;
each database used to check the whole CRC first.  That differs only
when a zip's first member matches a cartridge record and the whole
zip an arcade record.  The cartridge record now wins, written for the
zip itself where the core gate lets the zip's name into the cartridge
database, otherwise through the expansion.  With "Scan Inside
Archives" off nothing expands, so in that second case the arcade
record still wins.

* samples/tasks/database: write an rdb count over 127 as a uint

A record count from 128 up is written as a msgpack uint32.  The
read-ahead lanes size their sets from the reader window, 64 files per
reader, so they write 131 to 259 records; a fixint byte past 127
decodes as a map or a bool and the database is rejected on two- and
three-core hosts.

* scan: keep a refused crc index build refused

A database whose crc index build is refused is flagged, and neither
the size probe nor the CRC lookup starts another build for it.  The
index budget only shrinks, so a retry could only walk the database
again to the same refusal, once per content file.

The database scan test gains a cap on index builds and a lane that
scans plain files with every build refused: one query walk each.

* scan: look archives of several members up by either CRC

An archive whose header lists more than one entry, an arcade set as a
rule, is read whole first, and each database takes its own CRC or its
first member's.  A set whose first member is a cartridge ROM is
written for its arcade record rather than expanded into the cartridge
playlist, and an arcade folder walks no database claimed only for a
member's extension.  Readers read such an archive whole without a
grant.  The member pass stays for one-member archives, where a
cartridge zip needs no whole read.

file_archive_get_file_crc32_size_and_ext() reports the entry count
the backends already take from the archive's header.  A database
claimed only for the member's extension joins the member pass only
while it has or may get a crc index; unindexed, the lookup is a full
walk the expansion repeats.  A reader's member CRC survives a whole
read that fails or is over the cutoff, so the scan reads only the
archive itself.

Lanes: an arcade set with a cartridge ROM first, an arcade folder
beside a cartridge database, sets whose reader reads fail, and
cartridge zips against an unindexed database.

* archive: read a mapped zip's directory in place

When the archive is mapped, zip_parse_file_init() finds the end of
central directory record, reads the ZIP64 records and walks the
central directory where they lie; only the context is allocated.
Unmapped archives read them through the stream.  The scan's member
pass opens every zip for its directory, so this is its hot path.

archive_zip64_test counts reads through the stream and checks that,
mapped, a plain zip with a comment and a ZIP64 one give their first
member's CRC without any.

---------

Co-authored-by: LibretroAdmin <[email protected]>

libretro-common/file/archive_file.c

@@ -1015,6 +1015,13 @@ uint32_t file_archive_get_file_crc32(const char *path)
  * file found inside is used.
  **/
 uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size)
+{
+   return file_archive_get_file_crc32_size_and_ext(path, size, NULL, 0,
+         NULL);
+}
+
+uint32_t file_archive_get_file_crc32_size_and_ext(const char *path,
+      uint64_t *size, char *ext, size_t len, unsigned *entries)
 {
    file_archive_transfer_t state;
    struct archive_extract_userdata userdata        = {0};
@@ -1080,6 +1087,8 @@ uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size)
          break;
    }
 
+   if (entries)
+      *entries = found ? state.step_total : 0;
    file_archive_parse_file_iterate_stop(&state);
 
    /* Report nothing rather than whichever entry the walk stopped on:
@@ -1088,9 +1097,13 @@ uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size)
    if (!found)
    {
       *size = 0;
+      if (ext && len)
+         *ext = '\0';
       return 0;
    }
 
+   if (ext && len)
+      strlcpy(ext, path_get_extension(userdata.current_file_path), len);
    *size = userdata.size;
    return userdata.crc;
 }

libretro-common/file/archive_file_zlib.c

@@ -1370,9 +1370,18 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
       const char *file)
 {
    uint8_t footer_buf[1024];
-   uint8_t *footer = footer_buf;
+   uint8_t loc_buf[20], rec_buf[56];
+   /* A mapped archive is read where it lies: the end record, the ZIP64
+    * records and the central directory alike. */
+#ifdef VFS_HAVE_FILE_MAPPING
+   uint8_t *map = state->archive_mmap_data;
+#else
+   uint8_t *map = NULL;
+#endif
+   const uint8_t *footer = footer_buf;
    int64_t read_pos = state->archive_size;
    int64_t read_block = MIN(read_pos, (ssize_t)sizeof(footer_buf));
+   int64_t eocd_at = 0;
    int64_t directory_size, directory_offset;
    uint64_t entries;
    zip_context_t *zip_context = NULL;
@@ -1384,31 +1393,53 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
    /* Find the end of central directory record by scanning
     * the file from the end towards the beginning.
     */
-   for (;;)
+   if (map)
    {
-      if (--footer < footer_buf)
+      for (footer = map + (size_t)state->archive_size - 22; ; footer--)
       {
-         if (read_pos <= 0)
-            return -1; /* reached beginning of file */
-
-         /* Read 21 bytes of overlaps except on the first block. */
-         if (read_pos == state->archive_size)
-            read_pos = read_pos - read_block;
-         else
-            read_pos = MAX(read_pos - read_block + 21, 0);
-
-         /* Seek to read_pos and read read_block bytes. */
-         filestream_seek(state->archive_file, read_pos, RETRO_VFS_SEEK_POSITION_START);
-         if (filestream_read(state->archive_file, footer_buf, read_block) != read_block)
+         if (read_le(footer, 4) == END_OF_CENTRAL_DIR_SIGNATURE)
+         {
+            unsigned comment_len = read_le(footer + 20, 2);
+            if ((int64_t)(footer - map) + 22 + comment_len
+                  == state->archive_size)
+               break;
+         }
+         if (footer == map)
             return -1;
-
-         footer = footer_buf + read_block - 22;
       }
-      if (read_le(footer, 4) == END_OF_CENTRAL_DIR_SIGNATURE)
+      eocd_at = (int64_t)(footer - map);
+   }
+   else
+   {
+      for (;;)
       {
-         unsigned comment_len = read_le(footer + 20, 2);
-         if (read_pos + (footer - footer_buf) + 22 + comment_len == state->archive_size)
-            break; /* found it! */
+         if (--footer < footer_buf)
+         {
+            if (read_pos <= 0)
+               return -1; /* reached beginning of file */
+
+            /* Read 21 bytes of overlaps except on the first block. */
+            if (read_pos == state->archive_size)
+               read_pos = read_pos - read_block;
+            else
+               read_pos = MAX(read_pos - read_block + 21, 0);
+
+            /* Seek to read_pos and read read_block bytes. */
+            filestream_seek(state->archive_file, read_pos, RETRO_VFS_SEEK_POSITION_START);
+            if (filestream_read(state->archive_file, footer_buf, read_block) != read_block)
+               return -1;
+
+            footer = footer_buf + read_block - 22;
+         }
+         if (read_le(footer, 4) == END_OF_CENTRAL_DIR_SIGNATURE)
+         {
+            unsigned comment_len = read_le(footer + 20, 2);
+            if (read_pos + (footer - footer_buf) + 22 + comment_len == state->archive_size)
+            {
+               eocd_at = read_pos + (footer - footer_buf);
+               break; /* found it! */
+            }
+         }
       }
    }
 
@@ -1423,23 +1454,35 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
          || directory_size   == 0xFFFFFFFFu
          || directory_offset == 0xFFFFFFFFu)
    {
-      uint8_t loc[20], rec[56];
-      int64_t at = read_pos + (footer - footer_buf);
+      const uint8_t *loc = loc_buf, *rec = rec_buf;
+      int64_t at = eocd_at;
       uint64_t rec_off;
       if (at < 20)
          return -1;
-      filestream_seek(state->archive_file, at - 20,
-            RETRO_VFS_SEEK_POSITION_START);
-      if (     filestream_read(state->archive_file, loc, 20) != 20
-            || read_le(loc, 4) != ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIGNATURE)
+      if (map)
+         loc = map + (size_t)(at - 20);
+      else
+      {
+         filestream_seek(state->archive_file, at - 20,
+               RETRO_VFS_SEEK_POSITION_START);
+         if (filestream_read(state->archive_file, loc_buf, 20) != 20)
+            return -1;
+      }
+      if (read_le(loc, 4) != ZIP64_END_OF_CENTRAL_DIR_LOCATOR_SIGNATURE)
          return -1;
       rec_off = read_le64(loc + 8);
       if (rec_off > (uint64_t)(at - 20) || (uint64_t)(at - 20) - rec_off < 56)
          return -1;
-      filestream_seek(state->archive_file, (int64_t)rec_off,
-            RETRO_VFS_SEEK_POSITION_START);
-      if (     filestream_read(state->archive_file, rec, 56) != 56
-            || read_le(rec, 4) != ZIP64_END_OF_CENTRAL_DIR_SIGNATURE)
+      if (map)
+         rec = map + (size_t)rec_off;
+      else
+      {
+         filestream_seek(state->archive_file, (int64_t)rec_off,
+               RETRO_VFS_SEEK_POSITION_START);
+         if (filestream_read(state->archive_file, rec_buf, 56) != 56)
+            return -1;
+      }
+      if (read_le(rec, 4) != ZIP64_END_OF_CENTRAL_DIR_SIGNATURE)
          return -1;
       entries = read_le64(rec + 32);
       if (     read_le64(rec + 40) > (uint64_t)state->archive_size
@@ -1469,13 +1512,16 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
       return -1;
 
    /* This is a ZIP file, allocate one block of memory for both the
-    * context and the entire directory, then read the directory.
+    * context and the entire directory, then read the directory; a
+    * mapped one is walked in place.
     */
-   zip_context = (zip_context_t*)malloc(sizeof(zip_context_t) + (size_t)directory_size);
+   zip_context = (zip_context_t*)malloc(sizeof(zip_context_t)
+         + (map ? 0 : (size_t)directory_size));
    if (!zip_context)
       return -1;
    zip_context->state             = state;
-   zip_context->directory         = (uint8_t*)(zip_context + 1);
+   zip_context->directory         = map ? map + (size_t)directory_offset
+                                        : (uint8_t*)(zip_context + 1);
    zip_context->directory_entry   = zip_context->directory;
    zip_context->directory_end     = zip_context->directory + (size_t)directory_size;
    zip_context->zstream           = NULL;
@@ -1502,11 +1548,14 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
    zip_context->csize             = 0;
    zip_context->usize             = 0;
 
-   filestream_seek(state->archive_file, directory_offset, RETRO_VFS_SEEK_POSITION_START);
-   if (filestream_read(state->archive_file, zip_context->directory, directory_size) != directory_size)
+   if (!map)
    {
-      free(zip_context);
-      return -1;
+      filestream_seek(state->archive_file, directory_offset, RETRO_VFS_SEEK_POSITION_START);
+      if (filestream_read(state->archive_file, zip_context->directory, directory_size) != directory_size)
+      {
+         free(zip_context);
+         return -1;
+      }
    }
 
    state->context = zip_context;

libretro-common/include/file/archive_file.h

@@ -325,6 +325,24 @@ uint32_t file_archive_get_file_crc32(const char *path);
  **/
 uint32_t file_archive_get_file_crc32_and_size(const char *path, uint64_t *size);
 
+/**
+ * file_archive_get_file_crc32_size_and_ext:
+ * @path                         : filename path of archive
+ * @size                         : size of the file inside the archive
+ * @ext                          : that file's extension, or "" when
+ *                                 not found; may be NULL
+ * @len                          : size of @ext in bytes
+ * @entries                      : the archive's entry count as its
+ *                                 header gives it, or 0 when not
+ *                                 found; may be NULL
+ *
+ * Returns: CRC32 of the specified file in the archive, otherwise 0.
+ * If no path within the archive is specified, the first
+ * file found inside is used.
+ **/
+uint32_t file_archive_get_file_crc32_size_and_ext(const char *path,
+      uint64_t *size, char *ext, size_t len, unsigned *entries);
+
 extern const struct file_archive_file_backend zlib_backend;
 extern const struct file_archive_file_backend sevenzip_backend;
 extern const struct file_archive_file_backend zstd_backend;

libretro-common/samples/file/archive_zip64/Makefile

@@ -56,8 +56,10 @@ $(SAMPLE_OBJDIR)/%.o: $$(call sample_src,$$*).c Makefile
 	@mkdir -p $(@D)
 	$(CC) -c -o $@ $< $(CPPFLAGS) $(CFLAGS)
 
+# The wrap counts reads through the stream: mapped, the directory is
+# read in place.
 $(TARGET): $(OBJS)
-	$(CC) -o $@ $^ $(LDFLAGS)
+	$(CC) -o $@ $^ $(LDFLAGS) -Wl,--wrap=filestream_read
 
 check: $(TARGET)
 	./$(TARGET)

libretro-common/samples/file/archive_zip64/archive_zip64_test.c

@@ -73,6 +73,16 @@
 
 static unsigned failures;
 
+/* Linked with --wrap=filestream_read: reads through the stream. */
+static unsigned stream_reads;
+
+int64_t __real_filestream_read(RFILE *stream, void *data, int64_t len);
+int64_t __wrap_filestream_read(RFILE *stream, void *data, int64_t len)
+{
+   stream_reads++;
+   return __real_filestream_read(stream, data, len);
+}
+
 static void fail(const char *what, const char *detail)
 {
    printf("[FAIL] %s: %s\n", what, detail);
@@ -328,6 +338,66 @@ static bool listed(const char *archive, unsigned n)
    return ok;
 }
 
+/* @archive's first member's CRC and size: @crc and @n, and, mapped,
+ * taken from the archive where it lies, with no read through the
+ * stream for the end record, the ZIP64 records or the directory. */
+static void check_in_place(const char *what, const char *archive,
+      uint32_t crc, uint64_t n)
+{
+   uint64_t size = 0;
+   uint32_t got;
+
+   stream_reads = 0;
+   got          = file_archive_get_file_crc32_and_size(archive, &size);
+   if (got != crc || size != n)
+      fail(what, "first member's CRC and size");
+#ifdef HAVE_MMAP
+   else if (stream_reads)
+      fail(what, "read through the stream while mapped");
+#endif
+}
+
+/* One stored member and an archive comment: the end record is found
+ * past the comment. */
+static void test_plain_in_place(void)
+{
+   static const uint8_t note[] = "no ZIP64 here\n";
+   static uint8_t zip[512];
+   struct member m;
+   size_t at, cd;
+
+   m.name   = "plain.txt";
+   m.data   = note;
+   m.method = 0;
+   m.usize  = m.csize = sizeof(note) - 1;
+   m.big_sizes  = false;
+   m.big_offset = false;
+   m.offset = 0;
+   m.crc    = encoding_crc32(0, note, sizeof(note) - 1);
+
+   at  = local_header(zip, &m);
+   memcpy(zip + at, note, sizeof(note) - 1);
+   at += sizeof(note) - 1;
+   cd  = at;
+   at += central_header(zip + at, &m);
+   memset(zip + at, 0, 22);
+   put32(zip + at, 0x06054b50u);
+   put16(zip + at + 8, 1);
+   put16(zip + at + 10, 1);
+   put32(zip + at + 12, (uint32_t)(at - cd));
+   put32(zip + at + 16, (uint32_t)cd);
+   put16(zip + at + 20, 3);
+   memcpy(zip + at + 22, "PK!", 3);
+   at += 22 + 3;
+
+   if (!write_file(ZIP_PATH, zip, at))
+      fail("plain", "cannot write");
+   else
+      check_in_place("plain, first member in place", ZIP_PATH, m.crc,
+            m.usize);
+   remove(ZIP_PATH);
+}
+
 static void test_small(void)
 {
    uint8_t *stored  = (uint8_t*)malloc(STORED_LEN);
@@ -382,6 +452,8 @@ static void test_small(void)
       fail("ZIP64", "cannot write");
    if (!listed(ZIP_PATH, 3))
       fail("ZIP64", "listed");
+   check_in_place("ZIP64, first member in place", ZIP_PATH, m[0].crc,
+         STORED_LEN);
    check_member("ZIP64, stored", ZIP_PATH, "stored.bin", stored, STORED_LEN);
    check_member("ZIP64, deflated", ZIP_PATH, "deflate.bin", plain,
          DEFLATE_LEN);
@@ -555,6 +627,7 @@ static void test_big(void)
 
 int main(void)
 {
+   test_plain_in_place();
    test_small();
    test_big();
    if (failures)

samples/tasks/database/Makefile

@@ -241,11 +241,22 @@ $(INDEX_TEST)$(EXE_EXT): $(INDEX_TEST_OBJECTS)
 # Builds its own databases, so it needs nothing but a writable dir.
 # The wraps count the scanner's database walks and parks, slow chosen
 # reads and tell the scan's own from its readers', spend its window,
-# count and refuse reader pushes, and count queue worker starts.
+# count and refuse reader pushes, count queue worker starts, count an
+# archive's stored-CRC reads, fail a chosen serial's query, order the
+# database and content lists, count reader passes, hold chosen lookups
+# and hold a chosen index build in one step.
 SCAN_TEST_WRAPS := -Wl,--wrap=libretrodb_open -Wl,--wrap=task_queue_park \
 	-Wl,--wrap=intfstream_crc_step -Wl,--wrap=rm3u_is_m3u \
 	-Wl,--wrap=task_nbio_slice_open -Wl,--wrap=task_nbio_slice_close \
-	-Wl,--wrap=task_queue_push
+	-Wl,--wrap=task_queue_push \
+	-Wl,--wrap=file_archive_get_file_crc32_size_and_ext \
+	-Wl,--wrap=bin_to_hex_alloc -Wl,--wrap=dir_list_new \
+	-Wl,--wrap=task_queue_is_threaded \
+	-Wl,--wrap=database_info_list_new_crc \
+	-Wl,--wrap=database_info_index_job_new \
+	-Wl,--wrap=database_info_dir_init_from_list \
+	-Wl,--wrap=database_info_index_job_step \
+	-Wl,--wrap=task_nbio_slice_within_budget
 ifeq ($(HAVE_THREADS), 1)
 SCAN_TEST_WRAPS += -Wl,--wrap=sthread_create
 endif

samples/tasks/database/spend_window.h

@@ -10,15 +10,15 @@
  * window does not hold: it resets on the wall clock, and a reset
  * between the charge and the scanner's open grants the whole window. */
 static bool window_spend_next;
-/* Called after each open, when set. */
+/* Called before each open, when set. */
 static void (*window_open_hook)(void);
 
 void __real_task_nbio_slice_open(nbio_budget_t *b);
 void __wrap_task_nbio_slice_open(nbio_budget_t *b)
 {
-   __real_task_nbio_slice_open(b);
    if (window_open_hook)
       window_open_hook();
+   __real_task_nbio_slice_open(b);
    if (window_spend_next)
    {
       window_spend_next = false;

tasks/task_database.c

@@ -177,7 +177,10 @@ enum db_state_flags_enum
     * walks a time. */
    DB_STATE_FLAG_SIZE_CHECKED             = (1 << 4),
    /* A core claiming this database matches archive members. */
-   DB_STATE_FLAG_ARCHIVE_MEMBER           = (1 << 5)
+   DB_STATE_FLAG_ARCHIVE_MEMBER           = (1 << 5),
+   /* Its crc index build was refused.  The budget only shrinks, so a
+    * retry would walk the database again to the same answer. */
+   DB_STATE_FLAG_NO_CRC_INDEX             = (1 << 6)
 };
 
 #ifdef HAVE_LIBRETRODB
@@ -254,6 +257,8 @@ typedef struct database_state_handle
    uint64_t archive_size;
    char archive_name[512]; /* TODO/FIXME - check size */
    char serial[4096];      /* TODO/FIXME - check size */
+   /* The first member's extension, which gates the member pass. */
+   char member_ext[16];
    /* One entry per database in 'list', allocated to list->size:
     * list_index is bounded only by list->size - the number of .rdb
     * files in the database directory - so a fixed
@@ -294,6 +299,13 @@ typedef struct database_state_handle
    /* The index build in flight, run by task_database_index_step()
     * until index_ready, then taken by the lookup that began it. */
    database_info_index_job_t *index_job;
+   /* The lookup's pass, and whether archive_crc holds the archive's
+    * own CRC or its read is under way. */
+   uint8_t crc_pass;  /* enum db_crc_pass */
+   bool whole_done;
+   /* The member pass hit where only member_ext is claimed; it expands
+    * unless one claimed for the archive's own name matches too. */
+   bool member_hit;
    bool index_ready;
 } database_state_handle_t;
 
@@ -311,6 +323,18 @@ enum db_crc_fail
    DB_CRC_FAIL_ZERO        /* the CRC is 0, so no entry matches */
 };
 
+/* What the CRC lookup compares.  A one-member archive is looked up by
+ * its member's CRC, then read whole and looked up by its own only if
+ * that misses.  An archive of several members, an arcade set as a
+ * rule, is read whole first and each database takes either CRC. */
+enum db_crc_pass
+{
+   DB_CRC_PASS_FILE = 0,  /* crc: any other name */
+   DB_CRC_PASS_MEMBER,    /* crc, the first member's */
+   DB_CRC_PASS_WHOLE,     /* archive_crc */
+   DB_CRC_PASS_BOTH       /* crc or archive_crc: several members */
+};
+
 /* None on Wii U: its newlib and libfat take no locks, and without
  * libfat its SD layer serialises every call.  None on Vita: Vita3K's
  * sceIo tables take no lock. */
@@ -337,6 +361,11 @@ enum db_crc_fail
 #ifndef SCAN_READER_CUTOFF
 #define SCAN_READER_CUTOFF (128 * 1024 * 1024)
 #endif
+/* Archives the readers may read whole ahead of the scan at most.  Each
+ * member miss in a row grants 1, 2, 4 ... more. */
+#ifndef SCAN_READER_AHEAD_MAX
+#define SCAN_READER_AHEAD_MAX 16
+#endif
 
 #ifdef SCAN_HAVE_READERS
 enum scan_slot_state
@@ -345,19 +374,27 @@ enum scan_slot_state
    SCAN_SLOT_READY,       /* for a reader to claim */
    SCAN_SLOT_IN_FLIGHT,
    SCAN_SLOT_OK,
+   SCAN_SLOT_MEMBER_OK,   /* an archive's member CRC, not read whole */
    SCAN_SLOT_BIG,
    SCAN_SLOT_FAIL
 };
 
 /* A file handed out.  The scan writes it while EMPTY, its reader while
- * IN_FLIGHT, and each publishes with the state. */
+ * IN_FLIGHT, and each publishes with the state.  For an archive the
+ * reader takes member_crc and member_size first: its first member's
+ * CRC and size. */
 typedef struct scan_slot
 {
    uint64_t           size;
+   uint64_t           member_size;
    size_t             index;   /* in the content list */
    char              *path;    /* PATH_MAX_LENGTH, its own copy */
    uint32_t           crc;
+   uint32_t           member_crc;
    retro_atomic_int_t state;   /* enum scan_slot_state */
+   char               member_ext[16];
+   bool               archive;
+   bool               multi;   /* an archive of several members */
 } scan_slot_t;
 
 /* Shared by the scan and its readers; the last reference frees it.
@@ -372,6 +409,7 @@ typedef struct scan_readers
    unsigned           seq;      /* scan: slots filled */
    unsigned           take;     /* scan: slots given back */
    unsigned           count;    /* scan: readers pushed */
+   unsigned           grant;    /* scan: the last miss's, 0 after a hit */
    uint32_t           ident;    /* the scan's */
    retro_atomic_int_t filled;   /* seq, for the readers */
    retro_atomic_int_t claim;    /* the next slot to claim */
@@ -379,6 +417,12 @@ typedef struct scan_readers
    retro_atomic_int_t live;     /* readers not yet gone */
    retro_atomic_int_t quit;
    retro_atomic_int_t refs;
+   /* Archives the readers may still read whole ahead, beyond their
+    * member CRCs: granted when an archive misses by its member CRC,
+    * taken back when one matches by it.  A hint: each slot's state
+    * says what it holds. */
+   retro_atomic_int_t whole_ahead;
+   retro_atomic_int_t rearm;    /* the next slot to take back */
    bool               off;      /* scan: the readers are gone or going */
 } scan_readers_t;
 
@@ -387,6 +431,7 @@ typedef struct scan_reader
 {
    scan_readers_t *blk;
    scan_slot_t    *slot;   /* the file in hand */
+   scan_slot_t    *own;    /* its last MEMBER_OK */
    intfstream_t   *fd;
    int64_t         left;
    uint32_t        acc;
@@ -1060,6 +1105,21 @@ static void task_database_crc_close(database_state_handle_t *db_state)
    db_state->crc_done = false;
 }
 
+/* No serial, CRC, size or pass of one file is left for the next. */
+static void task_database_crc_reset(database_state_handle_t *db_state)
+{
+   task_database_crc_close(db_state);
+   db_state->serial[0]     = '\0';
+   db_state->crc           = 0;
+   db_state->archive_crc   = 0;
+   db_state->size          = 0;
+   db_state->archive_size  = 0;
+   db_state->crc_pass      = DB_CRC_PASS_FILE;
+   db_state->whole_done    = false;
+   db_state->member_hit    = false;
+   db_state->member_ext[0] = '\0';
+}
+
 /* Opens @name's CRC span for task_database_crc_step() to hash over
  * the steps that follow, and records its size. */
 static bool task_database_crc_begin(database_state_handle_t *db_state,
@@ -1176,12 +1236,62 @@ static bool scan_reader_eligible(const char *path)
    return false;
 }
 
+/* One of the whole reads ahead the scan granted, if any are left. */
+static bool scan_reader_credit(scan_readers_t *blk)
+{
+   int n;
+
+   while ((n = retro_atomic_load_acquire_int(&blk->whole_ahead)) > 0)
+      if (retro_atomic_cas_int(&blk->whole_ahead, n, n - 1))
+         return true;
+   return false;
+}
+
+/* While whole reads ahead are granted, a slot that holds only a member
+ * CRC, taken back to read whole: the next from the scan's on, in
+ * order, then this reader's last (it may have missed the grant). */
+static bool scan_reader_rearm(scan_reader_t *rd)
+{
+   scan_readers_t *blk = rd->blk;
+
+   while (retro_atomic_load_acquire_int(&blk->whole_ahead) > 0)
+   {
+      scan_slot_t *slot = rd->own;
+      int          k    = retro_atomic_load_acquire_int(&blk->rearm);
+
+      if (k < retro_atomic_load_acquire_int(&blk->claim))
+      {
+         if (!retro_atomic_cas_int(&blk->rearm, k, k + 1))
+            continue;
+         slot = &blk->win[(size_t)k % blk->window];
+      }
+      else if (!slot)
+         return false;
+      else
+         rd->own = NULL;
+      if (!retro_atomic_cas_int(&slot->state, SCAN_SLOT_MEMBER_OK,
+            SCAN_SLOT_IN_FLIGHT))
+         continue;
+      /* The slot first, then a credit: with none left it goes back. */
+      if (scan_reader_credit(blk))
+      {
+         rd->slot = slot;
+         return true;
+      }
+      retro_atomic_store_release_int(&slot->state, SCAN_SLOT_MEMBER_OK);
+      task_queue_wake(blk->ident);
+   }
+   return false;
+}
+
 /* The next filled slot, in order; one the scan took back is passed
- * over.  False when none is filled. */
+ * over.  One to take back comes first.  False when none is filled. */
 static bool scan_reader_claim(scan_reader_t *rd)
 {
    scan_readers_t *blk = rd->blk;
 
+   if (scan_reader_rearm(rd))
+      return true;
    for (;;)
    {
       int          c = retro_atomic_load_acquire_int(&blk->claim);
@@ -1281,6 +1391,27 @@ static void scan_reader_handler(retro_task_t *task)
          rd->parked = false;
          retro_atomic_fetch_sub_int(&blk->idle, 1);
       }
+      /* One taken back has its member CRC. */
+      if (rd->slot->archive && !rd->slot->member_crc)
+      {
+         unsigned entries = 0;
+
+         rd->slot->member_crc = file_archive_get_file_crc32_size_and_ext(
+               rd->slot->path, &rd->slot->member_size,
+               rd->slot->member_ext, sizeof(rd->slot->member_ext),
+               &entries);
+         rd->slot->multi      = entries > 1;
+         /* Only the member's CRC unless a whole read is granted; the
+          * scan needs the archive whole when it has several members
+          * or no member CRC. */
+         if (   rd->slot->member_crc && !rd->slot->multi
+             && !scan_reader_credit(blk))
+         {
+            rd->own = rd->slot;
+            scan_reader_publish(rd, SCAN_SLOT_MEMBER_OK);
+            return;
+         }
+      }
       if (!(rd->fd = task_database_crc_open(rd->slot->path,
             TASK_DATABASE_CRC_FILE, &rd->left, &size)))
       {
@@ -1341,6 +1472,8 @@ static void scan_readers_start(retro_task_t *task,
    retro_atomic_int_init(&blk->claim,  0);
    retro_atomic_int_init(&blk->idle,   0);
    retro_atomic_int_init(&blk->quit,   0);
+   retro_atomic_int_init(&blk->whole_ahead, 0);
+   retro_atomic_int_init(&blk->rearm,  0);
    retro_atomic_int_init(&blk->live,   (int)n);
    retro_atomic_int_init(&blk->refs,   (int)n + 1);
    for (r = 0; r < n; r++)
@@ -1438,9 +1571,14 @@ static void scan_readers_fill(manual_scan_handle_t *manual_scan)
       {
          scan_slot_t *slot = &blk->win[blk->seq % blk->window];
 
-         slot->index = blk->fill;
-         slot->size  = 0;
-         slot->crc   = 0;
+         slot->index         = blk->fill;
+         slot->size          = 0;
+         slot->crc           = 0;
+         slot->member_crc    = 0;
+         slot->member_ext[0] = '\0';
+         slot->multi         = false;
+         slot->archive       = extension_to_file_type(path_get_extension(path))
+               == FILE_TYPE_COMPRESSED;
          /* A copy: the list moves and is pruned under the readers. */
          strlcpy(slot->path, path, PATH_MAX_LENGTH);
          retro_atomic_store_release_int(&slot->state, SCAN_SLOT_READY);
@@ -1488,14 +1626,34 @@ static enum scan_take scan_readers_take(manual_scan_handle_t *manual_scan,
             return SCAN_TAKE_WAIT;
          state = SCAN_SLOT_EMPTY;
       }
+      /* A reader may take it back to read whole. */
+      if (   state == SCAN_SLOT_MEMBER_OK
+          && !retro_atomic_cas_int(&slot->state, SCAN_SLOT_MEMBER_OK,
+               SCAN_SLOT_EMPTY))
+         return SCAN_TAKE_WAIT;
       blk->take++;
-      if (slot->index == i && state == SCAN_SLOT_OK)
+      if (   slot->index == i
+          && (   state == SCAN_SLOT_OK || state == SCAN_SLOT_MEMBER_OK
+              || (slot->archive && slot->member_crc)))
       {
-         if (extension_to_file_type(path_get_extension(name))
-               == FILE_TYPE_COMPRESSED)
+         if (slot->archive)
          {
-            db_state->archive_crc  = slot->crc;
-            db_state->archive_size = slot->size;
+            db_state->crc      = slot->member_crc;
+            db_state->size     = slot->member_size;
+            db_state->crc_pass = slot->multi
+               ? DB_CRC_PASS_BOTH : DB_CRC_PASS_MEMBER;
+            strlcpy(db_state->member_ext, slot->member_ext,
+                  sizeof(db_state->member_ext));
+            /* BIG or FAIL: the scan reads it whole itself. */
+            if (state == SCAN_SLOT_OK)
+            {
+               db_state->archive_crc  = slot->crc;
+               db_state->archive_size = slot->size;
+               db_state->whole_done   = true;
+               /* No member CRC: only its own can match. */
+               if (!slot->member_crc)
+                  db_state->crc_pass  = DB_CRC_PASS_WHOLE;
+            }
          }
          else
          {
@@ -1517,6 +1675,65 @@ static enum scan_take scan_readers_take(manual_scan_handle_t *manual_scan,
       blk->fill = i + 1;
    return SCAN_TAKE_NONE;
 }
+
+/* Adds whole reads ahead, up to SCAN_READER_AHEAD_MAX.  With some
+ * granted, readers also take back what they have only the member CRC
+ * of, from the scan's next file on; any parked on a full window
+ * wake. */
+static void scan_readers_grant(scan_readers_t *blk, int add)
+{
+   int had, n;
+
+   /* Readers only take: a count of 0 stays 0 until this adds. */
+   do
+   {
+      had = retro_atomic_load_acquire_int(&blk->whole_ahead);
+      if (had >= SCAN_READER_AHEAD_MAX)
+         return;
+      n   = had + add;
+      if (n > SCAN_READER_AHEAD_MAX)
+         n = SCAN_READER_AHEAD_MAX;
+      /* Before the count: a reader that sees it sees the cursor. */
+      if (!had)
+         retro_atomic_store_release_int(&blk->rearm, (int)blk->take);
+   } while (!retro_atomic_cas_int(&blk->whole_ahead, had, n));
+   if (!had)
+      scan_readers_wake(blk);
+}
+
+/* A member miss grants the readers whole reads ahead, twice the last
+ * miss's, up to SCAN_READER_AHEAD_MAX; a member hit takes back what is
+ * left. */
+static void scan_readers_whole_ahead(manual_scan_handle_t *manual_scan,
+      int miss)
+{
+   scan_readers_t *blk = manual_scan->readers;
+
+   if (!blk)
+      return;
+   if (!miss)
+   {
+      blk->grant = 0;
+      if (retro_atomic_load_acquire_int(&blk->whole_ahead))
+         retro_atomic_store_release_int(&blk->whole_ahead, 0);
+      return;
+   }
+   blk->grant = blk->grant ? blk->grant * 2 : 1;
+   if (blk->grant > SCAN_READER_AHEAD_MAX)
+      blk->grant = SCAN_READER_AHEAD_MAX;
+   scan_readers_grant(blk, (int)blk->grant);
+}
+
+/* While the scan walks a database in a member pass mid miss streak,
+ * the readers may read whole up to the cap.  The streak's grant is
+ * left as it is. */
+static void scan_readers_top_up(manual_scan_handle_t *manual_scan)
+{
+   scan_readers_t *blk = manual_scan->readers;
+
+   if (blk && blk->grant)
+      scan_readers_grant(blk, SCAN_READER_AHEAD_MAX);
+}
 #endif
 
 static void task_database_index_close(database_state_handle_t *db_state)
@@ -1577,11 +1794,9 @@ static int task_database_iterate_playlist(
    {
       case FILE_TYPE_COMPRESSED:
 #ifdef HAVE_COMPRESSION
-         db->type = DATABASE_TYPE_CRC_LOOKUP;
-         /* first check crc of archive itself */
-         return task_database_crc_begin(db_state, name,
-               TASK_DATABASE_CRC_FILE, DB_CRC_DEST_ARCHIVE,
-               DB_CRC_FAIL_ERROR);
+         db->type           = DATABASE_TYPE_CRC_LOOKUP;
+         db_state->crc_pass = DB_CRC_PASS_MEMBER;
+         break;
 #else
          break;
 #endif
@@ -1748,16 +1963,7 @@ static enum scan_verdict database_info_list_iterate_end_no_match(
 
    db_state->list_index   = 0;
    db_state->entry_index  = 0;
-   db_state->size         = 0;
-   db_state->archive_size = 0;
-   db_state->serial[0]    = '\0';
-   task_database_crc_close(db_state);
-
-   if (db_state->crc != 0)
-      db_state->crc = 0;
-
-   if (db_state->archive_crc != 0)
-      db_state->archive_crc = 0;
+   task_database_crc_reset(db_state);
 
    return archive_added ? SCAN_VERDICT_ARCHIVE_CONTENTS_ADDED : SCAN_VERDICT_NO_DB_MATCH;
 }
@@ -1917,16 +2123,15 @@ static enum scan_verdict database_info_list_iterate_found_match(
                          archive_name))
       RARCH_ERR("[Scanner] Failed to add result for: \"%s\".\n", entry_lbl);
 
+#ifdef SCAN_HAVE_READERS
+   if (db_state->crc_pass == DB_CRC_PASS_MEMBER)
+      scan_readers_whole_ahead(_db, 0);
+#endif
    database_info_list_free(db_state->info);
    free(db_state->info);
 
    db_state->info         = NULL;
-   db_state->crc          = 0;
-   db_state->archive_crc  = 0;
-   db_state->size         = 0;
-   db_state->archive_size = 0;
-   db_state->serial[0]    = '\0';
-   task_database_crc_close(db_state);
+   task_database_crc_reset(db_state);
 
    /* Move database to start since we are likely to match against it
       again */
@@ -2074,13 +2279,18 @@ static bool task_database_fill_db_min_max(database_state_handle_t *db_state,
     * lookup reads the database once. */
    if (   rdb && *rdb
        && !db_state->crc_index[db_state->list_index]
+       && !(db_state->flags[db_state->list_index]
+          & DB_STATE_FLAG_NO_CRC_INDEX)
        && db_state->index_budget)
    {
       database_info_crc_index_t *ci = NULL;
 
       if (!task_database_index_take(db_state, &ci, serial, &has_serial))
          return false;
-      if (ci)
+      if (!ci)
+         db_state->flags[db_state->list_index] |=
+            DB_STATE_FLAG_NO_CRC_INDEX;
+      else
       {
          size_t used = database_info_crc_index_bytes(ci);
          db_state->index_budget = (used < db_state->index_budget)
@@ -2185,10 +2395,26 @@ static bool task_database_fill_db_min_max(database_state_handle_t *db_state,
    return true;
 }
 
+/* An installed core claims the current database for @ext; one that
+ * matches archive members, only for a member. */
+static bool task_database_claimed(database_state_handle_t *db_state,
+      const char *ext, bool member)
+{
+   return string_list_find_elem(
+            db_state->claim_exts[db_state->list_index], ext)
+      && (   member
+          || !(db_state->flags[db_state->list_index]
+             & DB_STATE_FLAG_ARCHIVE_MEMBER));
+}
+
 /* Passes over the run of databases, from the current one, that no
- * installed core claims @name for.  True when it passed over any. */
+ * installed core claims @name for, nor @member_ext when given and the
+ * database has or may still get a crc index: unindexed, a member
+ * lookup is a whole walk the expansion repeats.  True when it passed
+ * over any. */
 static bool task_database_skip_unclaimed(database_state_handle_t *db_state,
-      const char *name, bool path_contains_compressed_file)
+      const char *name, const char *member_ext,
+      bool path_contains_compressed_file)
 {
    /* Every database the gate refuses is passed over in this one
     * step, so a file costs a step per database it can match rather
@@ -2197,11 +2423,14 @@ static bool task_database_skip_unclaimed(database_state_handle_t *db_state,
    bool skipped    = false;
 
    while (   db_state->list_index < db_state->list->size
-          && (   !string_list_find_elem(
-                     db_state->claim_exts[db_state->list_index], ext)
-              || (   !path_contains_compressed_file
-                  && (db_state->flags[db_state->list_index]
-                     & DB_STATE_FLAG_ARCHIVE_MEMBER))))
+          && !task_database_claimed(db_state, ext,
+               path_contains_compressed_file)
+          && !(   member_ext
+               && task_database_claimed(db_state, member_ext, true)
+               && (   db_state->crc_index[db_state->list_index]
+                   || (   !(db_state->flags[db_state->list_index]
+                            & DB_STATE_FLAG_NO_CRC_INDEX)
+                       && db_state->index_budget))))
    {
       database_info_list_iterate_next(db_state);
       skipped = true;
@@ -2210,6 +2439,43 @@ static bool task_database_skip_unclaimed(database_state_handle_t *db_state,
    return skipped;
 }
 
+/* The member pass missed, or its first member has no CRC: the
+ * databases again from the first, for the archive's own CRC.  The
+ * steps that follow read the archive whole unless a reader has. */
+static enum scan_verdict task_database_whole_pass(
+      database_state_handle_t *db_state, const char *name)
+{
+   database_info_list_free(db_state->info);
+   free(db_state->info);
+   db_state->info        = NULL;
+   db_state->list_index  = 0;
+   db_state->entry_index = 0;
+   db_state->crc_pass    = DB_CRC_PASS_WHOLE;
+   if (   !db_state->whole_done
+       && !task_database_crc_begin(db_state, name, TASK_DATABASE_CRC_FILE,
+            DB_CRC_DEST_ARCHIVE, DB_CRC_FAIL_ERROR))
+      return SCAN_VERDICT_ERROR;
+   return SCAN_VERDICT_CONTINUE;
+}
+
+/* Found by its first member where the archive's own name would not
+ * be looked up: matched as the members the expansion adds, without
+ * reading the archive whole. */
+static enum scan_verdict task_database_member_hit(
+      manual_scan_handle_t *_db, database_state_handle_t *db_state,
+      database_info_handle_t *db, const char *name,
+      bool path_contains_compressed_file)
+{
+#ifdef SCAN_HAVE_READERS
+   scan_readers_whole_ahead(_db, 0);
+#endif
+   database_info_list_free(db_state->info);
+   free(db_state->info);
+   db_state->info = NULL;
+   return database_info_list_iterate_end_no_match(_db, db, db_state, name,
+         path_contains_compressed_file);
+}
+
 static enum scan_verdict task_database_iterate_crc_lookup(
       manual_scan_handle_t *_db,
       database_state_handle_t *db_state,
@@ -2223,25 +2489,58 @@ static enum scan_verdict task_database_iterate_crc_lookup(
        || ( _db->flags & DB_HANDLE_FLAG_USE_FIRST_MATCH_ONLY &&
             db_state->list_index > 0 &&
             db_state->flags[0] & DB_STATE_FLAG_MATCHED))
+   {
+      if (db_state->list && db_state->crc_pass == DB_CRC_PASS_MEMBER)
+      {
+         if (db_state->member_hit)
+            return task_database_member_hit(_db, db_state, db, name,
+                  path_contains_compressed_file);
+#ifdef SCAN_HAVE_READERS
+         scan_readers_whole_ahead(_db, 1);
+#endif
+         return task_database_whole_pass(db_state, name);
+      }
       return database_info_list_iterate_end_no_match(_db, db, db_state, name,
             path_contains_compressed_file);
+   }
 
    /* Archive did not contain a CRC for this entry,
     * or the file is empty. */
-   if (!db_state->crc)
+   if (!db_state->crc && db_state->crc_pass != DB_CRC_PASS_WHOLE)
    {
+      unsigned entries = 0;
 #ifdef DEBUG
       RARCH_DBG("[Scanner] Extra crc check 1: %x %d / %x %d %s\n",
             db_state->crc, db_state->size, db_state->archive_crc, db_state->archive_size,
             path_contains_compressed_file ? "compressed:true" : "compressed:false");
 #endif
-      db_state->crc = file_archive_get_file_crc32_and_size(name, &db_state->size);
+      db_state->crc = file_archive_get_file_crc32_size_and_ext(name,
+            &db_state->size, db_state->member_ext,
+            sizeof(db_state->member_ext), &entries);
 #ifdef DEBUG
       RARCH_DBG("[Scanner] Extra crc check 2: %x %d / %x %d.\n",
             db_state->crc, db_state->size, db_state->archive_crc, db_state->archive_size);
 #endif
       if (!db_state->crc)
+      {
+         /* No CRC for the first member (a directory, or a decode that
+          * failed): only the archive's own CRC can match. */
+         if (db_state->crc_pass == DB_CRC_PASS_MEMBER)
+            return task_database_whole_pass(db_state, name);
          return database_info_list_iterate_next(db_state);
+      }
+      if (entries > 1 && db_state->crc_pass == DB_CRC_PASS_MEMBER)
+         db_state->crc_pass = DB_CRC_PASS_BOTH;
+   }
+
+   /* Several members: read whole before the first lookup. */
+   if (db_state->crc_pass == DB_CRC_PASS_BOTH && !db_state->whole_done)
+   {
+      db_state->whole_done = true;
+      if (!task_database_crc_begin(db_state, name, TASK_DATABASE_CRC_FILE,
+            DB_CRC_DEST_ARCHIVE, DB_CRC_FAIL_ERROR))
+         return SCAN_VERDICT_ERROR;
+      return SCAN_VERDICT_CONTINUE;
    }
 
    /* The core-info gate is an in-memory string check; the size gate
@@ -2251,9 +2550,15 @@ static enum scan_verdict task_database_iterate_crc_lookup(
     * database in the directory was read end to end once per scan,
     * including the ones no installed core claims and which can never
     * match - 200 MB of .rdb reads to scan a 70 MB set with the
-    * shipped database directory.  Ask the cheap question first. */
+    * shipped database directory.  Ask the cheap question first.
+    * The member pass asks it for the first member too, as its
+    * expansion will, until one matches; not when nothing expands. */
    if (   !(_db->flags & DB_HANDLE_FLAG_SCAN_WITHOUT_CORE_MATCH)
        && task_database_skip_unclaimed(db_state, name,
+            (   db_state->crc_pass == DB_CRC_PASS_MEMBER
+             && !db_state->member_hit
+             && _db->task_config->search_archives)
+            ? db_state->member_ext : NULL,
             path_contains_compressed_file))
       return SCAN_VERDICT_CONTINUE;
 
@@ -2264,32 +2569,25 @@ static enum scan_verdict task_database_iterate_crc_lookup(
 
    if (db_state->min_sizes[db_state->list_index] > 0)
    {
-      /* Examining zip file main entry (archive size filled, but no indication of compressed file) */
-      if ( !path_contains_compressed_file && db_state->archive_size > 0)
-      {
-         if (       ( db_state->min_sizes[db_state->list_index] > (int64_t) db_state->archive_size
-                   && db_state->min_sizes[db_state->list_index] > (int64_t) db_state->size )
-              || (    db_state->max_sizes[db_state->list_index] < (int64_t) db_state->archive_size
-                   && db_state->max_sizes[db_state->list_index] < (int64_t) db_state->size ))
-         {
-#ifdef DEBUG
-            RARCH_DBG("[Scanner] Skipping DB, neither archive nor uncompressed size %ld/%ld is in range.\n",
-                  db_state->archive_size, db_state->size);
-#endif
-            return database_info_list_iterate_next(db_state);
-         }
-      }
-      /* Any other case (non-archive file, or a file inside the archive */
-      else if (         db_state->size > 0
-                && (    db_state->min_sizes[db_state->list_index] > (int64_t) db_state->size
-                     || db_state->max_sizes[db_state->list_index] < (int64_t) db_state->size))
+      int64_t  lo   = db_state->min_sizes[db_state->list_index];
+      int64_t  hi   = db_state->max_sizes[db_state->list_index];
+      uint64_t size = (db_state->crc_pass == DB_CRC_PASS_WHOLE)
+         ? db_state->archive_size : db_state->size;
+
+      if (   db_state->crc_pass == DB_CRC_PASS_BOTH
+          && db_state->archive_size >= (uint64_t)lo
+          && db_state->archive_size <= (uint64_t)hi)
+         size = db_state->archive_size;
+      if (         size > 0
+                && (    db_state->min_sizes[db_state->list_index] > (int64_t) size
+                     || db_state->max_sizes[db_state->list_index] < (int64_t) size))
       {
 #ifdef DEBUG
-         RARCH_DBG("[Scanner] Skipping DB, file size %ld not in range.\n", db_state->size);
+         RARCH_DBG("[Scanner] Skipping DB, file size %ld not in range.\n", size);
 #endif
          return database_info_list_iterate_next(db_state);
       }
-      else if (db_state->size == 0)
+      else if (size == 0)
       {
 #ifdef DEBUG
          RARCH_DBG("[Scanner] Zero-length file, skipping database match\n");
@@ -2305,6 +2603,8 @@ static enum scan_verdict task_database_iterate_crc_lookup(
       const char *rdb            = db_state->list->elems[
          db_state->list_index].data;
       database_info_list_t *hits = NULL;
+      uint32_t crc               = (db_state->crc_pass == DB_CRC_PASS_WHOLE)
+         ? db_state->archive_crc : db_state->crc;
 
       query[0] = '\0';
 
@@ -2321,13 +2621,18 @@ static enum scan_verdict task_database_iterate_crc_lookup(
       if (rdb && *rdb)
       {
          if (   !db_state->crc_index[db_state->list_index]
+             && !(db_state->flags[db_state->list_index]
+                & DB_STATE_FLAG_NO_CRC_INDEX)
              && db_state->index_budget)
          {
             database_info_crc_index_t *ci = NULL;
 
             if (!task_database_index_take(db_state, &ci, NULL, NULL))
                return SCAN_VERDICT_CONTINUE;
-            if (ci)
+            if (!ci)
+               db_state->flags[db_state->list_index] |=
+                  DB_STATE_FLAG_NO_CRC_INDEX;
+            else
             {
                size_t used = database_info_crc_index_bytes(ci);
                db_state->index_budget = (used < db_state->index_budget)
@@ -2339,8 +2644,8 @@ static enum scan_verdict task_database_iterate_crc_lookup(
          if (db_state->crc_index[db_state->list_index])
             hits = database_info_list_new_crc(
                   db_state->crc_index[db_state->list_index], rdb,
-                  db_state->crc, db_state->archive_crc,
-                  DB_EXTRACT_SCAN_FIELDS);
+                  crc, (db_state->crc_pass == DB_CRC_PASS_BOTH)
+                  ? db_state->archive_crc : 0, DB_EXTRACT_SCAN_FIELDS);
       }
 
       if (hits)
@@ -2352,12 +2657,18 @@ static enum scan_verdict task_database_iterate_crc_lookup(
          }
          db_state->info = hits;
       }
-      else
+      else if (db_state->crc_pass == DB_CRC_PASS_BOTH)
       {
          snprintf(query, sizeof(query),
                "{crc:or(b\"%08lX\",b\"%08lX\")}",
-               (unsigned long)db_state->crc,
+               (unsigned long)crc,
                (unsigned long)db_state->archive_crc);
+         database_info_list_iterate_new(db_state, query);
+      }
+      else
+      {
+         snprintf(query, sizeof(query), "{crc:b\"%08lX\"}",
+               (unsigned long)crc);
 
          database_info_list_iterate_new(db_state, query);
       }
@@ -2382,17 +2693,32 @@ static enum scan_verdict task_database_iterate_crc_lookup(
       database_info_t *db_info_entry =
          &db_state->info->list[db_state->entry_index];
 
-      /* When scanning an archive, "first" file crc32 is also checked. */
       if (db_info_entry->crc32)
       {
-         if (db_state->archive_crc == db_info_entry->crc32)
-            return database_info_list_iterate_found_match(
-                  _db,
-                  db_state, db, NULL);
-         if (db_state->crc == db_info_entry->crc32)
+         if (   db_state->crc_pass == DB_CRC_PASS_WHOLE
+             || db_state->crc_pass == DB_CRC_PASS_BOTH)
+         {
+            if (db_state->archive_crc == db_info_entry->crc32)
+               return database_info_list_iterate_found_match(
+                     _db,
+                     db_state, db, NULL);
+         }
+         if (   db_state->crc_pass != DB_CRC_PASS_WHOLE
+             && db_state->crc == db_info_entry->crc32)
+         {
+            if (   db_state->crc_pass == DB_CRC_PASS_MEMBER
+                && !(_db->flags & DB_HANDLE_FLAG_SCAN_WITHOUT_CORE_MATCH)
+                && !task_database_claimed(db_state,
+                     path_get_extension(name),
+                     path_contains_compressed_file))
+            {
+               db_state->member_hit = true;
+               return database_info_list_iterate_next(db_state);
+            }
             return database_info_list_iterate_found_match(
                   _db,
                   db_state, db, archive_entry);
+         }
       }
    }
 
@@ -2514,7 +2840,7 @@ static int task_database_iterate_serial_lookup(
    /* 3DS images, unlike discs, keep the crc lookup's core gate. */
    if (   !(_db->flags & DB_HANDLE_FLAG_SCAN_WITHOUT_CORE_MATCH)
        && extension_to_file_type(path_get_extension(name)) == FILE_TYPE_3DS
-       && task_database_skip_unclaimed(db_state, name,
+       && task_database_skip_unclaimed(db_state, name, NULL,
             path_contains_compressed_file))
       return SCAN_VERDICT_CONTINUE;
 
@@ -2787,7 +3113,14 @@ static int task_database_iterate(
    if (db_state->crc_fd)
       return task_database_crc_step(db_state, b);
    if (db_state->index_job && !db_state->index_ready)
+   {
+#ifdef SCAN_HAVE_READERS
+      if (   db_state->crc_pass == DB_CRC_PASS_MEMBER
+          && !db_state->member_hit)
+         scan_readers_top_up(_db);
+#endif
       return task_database_index_step(db_state, b);
+   }
 #ifdef DEBUG
    RARCH_DBG("[Scanner] Type %d, \"%s\" against \"%s\".\n", db->type, name, database_info_get_current_name(db_state));
    /* list_index == list->size is a valid state here (every database
@@ -2851,7 +3184,7 @@ static void task_database_cleanup_state(database_state_handle_t *db_state)
    if (db_state->buf)
       free(db_state->buf);
    db_state->buf = NULL;
-   task_database_crc_close(db_state);
+   task_database_crc_reset(db_state);
    task_database_index_close(db_state);
 }
 #endif
@@ -4162,6 +4495,7 @@ static enum scan_step task_manual_scan_iterate_step(retro_task_t *task,
                    * same failing step every tick.  Give up on this
                    * file and go to the next one. */
                   RARCH_ERR("[Scanner] Scanning of content unexpectedly failed for \"%s\"\n", content_path);
+                  task_database_crc_reset(dbstate);
                   manual_scan->status = DATABASE_SCAN_ITERATE_NEXT;
                   break;
                case SCAN_VERDICT_CONTINUE: