scan: read an archive whole only when a database may hold its size (#19944)

* scan: read an archive's directory once for all its passes

The scan fetched an archive's first member's CRC, size and extension
through a helper that stops at the first entry and reports how many
the header counts, and its readers did the same.  One walk now reads
the whole directory, in the readers as in the scan: every member's
stored CRC, size and name, kept for each of the archive's passes.
Its transfer state is on the heap, out of the readers' frames.  The
walk replaces the helper in the archive passes; the helper stays for
its other users.

The walk decodes the first member for a CRC where the archive stores
none, a zstd stream or a RAR 5 member hashed with BLAKE2, so it is
looked up as before.  Later members are not decoded and keep a CRC of
0.  A .bz2 is decoded to be listed at all.

Folders are not members: an archive of a folder and one file has one
member, that file, and is looked up by its CRC first.

* zip: report a TorrentZip end comment

A zip whose end-of-central-directory comment is TORRENTZIPPED- and
eight hex digits was written by TorrentZip, as arcade sets are.  The
zip reader already reads that record to find the directory, so it now
says so in the transfer state, for a caller that wants the hint.  The
CRC in the comment is not checked: nothing depends on it.

* database_info, libretro-db: archive record sizes beside the crc index

The crc index build walks each database once, off the main thread.
In the same walk it now also collects the size of every record whose
rom_name ends in an archive extension (zip, 7z, apk, rar, zst), kept
in one sorted array keyed by size and type, so a caller can ask
whether a database may hold an archive of a given type and size.  A
record of that type with no size, or an index built without the
sizes, answers yes.  The sizes count against the index budget.

The stepped database scan reports one string field for this:
libretrodb_scan_begin() and its callback gain a name field.  A
record's rom_name can also be extracted on request.

* scan: look every archive member up from the directory

An archive whose first member missed was read whole and, when that
missed too, expanded: each member went through the databases again,
by the stored CRC the directory already held.  An archive of several
members was read whole before any lookup, even where no database
could hold it, so disc zips and console sets were read whole again.

One member: its stored CRC is looked up first, as before.  A hit in a
database claimed only for the member's type is held, and every
member's CRC is looked up next.  On a miss the archive is read whole
only when a database whose cores open its type may hold it by its
exact size; its own CRC is then looked up there, and where that is
not asked or misses, every member's.

Several members: the same size question comes first.  Each database
whose cores open the type is then asked for the first member's CRC,
and for the archive's own CRC only when the size let it be read
whole, so an arcade record still wins and no database claimed only
for a member's type is walked before it.  Where neither matches,
every member's stored CRC is looked up.

A database may hold the archive when it has a record whose rom_name
is of the archive's type and size, one of that type with no size, or
no sizes at all.  Every member is looked up with the gates and size
checks the expansion uses.  A hit names the archive itself only when
it is the one record hit, that record is named for this archive and
its database's cores open the type; other hits are written as the
expansion wrote them, archive#member.  The expansion stays for scans
that keep unmatched files, for archives whose directory cannot be
read, and where nothing hit and a member stores no CRC, which only
the expansion unpacks for one.

Readers read an archive whole ahead only after a whole match, twice
as many after each such match up to the cap; any other verdict takes
the rest back.  A TorrentZip set of several members is still read
whole by a reader with no grant while some database opens zip.  An
archive over the readers' cutoff is handed back before a grant is
spent on it.

This narrows "several members are read whole first" to "read whole
when a database may hold it by its size".  A record whose crc is an
archive's own CRC but whose rom_name is not of the archive's type, or
whose size is not the file's, is no longer matched; each of FBNeo's
596 sets carries its zip's size.  A zip of a folder and one ROM takes
the one-member order.

* database_scan_test: rar archives through every pass

A RAR 4 archive of stored entries, written by hand as the zips are,
takes the passes zip and 7z take: a first member's hit where a core
opens rar names the archive, a later member's elsewhere is held as
archive#member, and the archive is read whole only while a database
whose cores open rar holds a rar record of its size.

* database_scan_test: define the fixture byte writers once

* archive_file: clear the TorrentZip flag for every archive

The flag was cleared only by the zip reader, so a transfer reused for
a 7z, a rar or an archive with no backend kept what the last zip said.
It is now cleared before any backend opens the archive.

* scan: readers read TorrentZip sets whole while the scan asks for that

Readers read every TorrentZip set of several members whole while some
database opens zip, but where no database may hold a set by its size
the scan drops that CRC, so a folder of sets no database holds was
read whole for nothing.  Each TorrentZip set the scan rules on now
says whether its own CRC was asked, and the readers read the next ones
whole only while it was.

---------

Co-authored-by: libretroadmin <[email protected]>

database_info.c

@@ -227,6 +227,7 @@ char *bin_to_hex_alloc(const uint8_t *data, size_t len)
 #define DB_EXTRACT_SIZE     (1 << 3)
 #define DB_EXTRACT_MD5      (1 << 4)
 #define DB_EXTRACT_SHA1     (1 << 5)
+#define DB_EXTRACT_ROM_NAME (1 << 6)
 
 /* Combination used by the scanner */
 #define DB_EXTRACT_SCAN_FIELDS \
@@ -449,8 +450,7 @@ static int database_cursor_iterate(libretrodb_cursor_t *cur,
             }
             else if (memcmp(str, "rom_name", 8) == 0)
             {
-               /* rom_name is not used anywhere in codebase,
-                * but is frequently added to DB */
+               /* The filtered read takes rom_name; this one skips it. */
             }
             break;
 
@@ -685,6 +685,19 @@ static int database_info_fill_from_dom(struct rmsgpack_dom_value *item,
             }
             break;
 
+         case 8:
+            if (   (fields & DB_EXTRACT_ROM_NAME)
+                && memcmp(str, "rom_name", 8) == 0)
+            {
+               if (val->type == RDT_STRING)
+               {
+                  const char *vs = val->val.string.buff;
+                  if (vs && *vs)
+                     db_info->rom_name = strdup(vs);
+               }
+            }
+            break;
+
          default:
             break;
       }
@@ -1021,6 +1034,13 @@ struct database_info_crc_index
    uint64_t             size_min;
    uint64_t             size_max;
    bool                 have_size;
+   /* (size << 3) | kind of each record named for an archive, sorted,
+    * when arc_have; bit kind of arc_unsized is set when a record of
+    * that kind has no size.  See db_arc_kind(). */
+   uint64_t            *arc_keys;
+   size_t               arc_count;
+   uint8_t              arc_unsized;
+   bool                 arc_have;
    /* One buffered stream on the database, opened on the first hit
     * and kept for the index's lifetime.  A hit reads one record at a
     * known offset; opening the file twice per hit (libretrodb_open
@@ -1186,6 +1206,98 @@ static database_info_crc_index_t *db_crc_index_finish(
    return idx;
 }
 
+/* The archive types a rom_name can end in.  A kind is the place here
+ * plus one; 0 is none. */
+static const char *const db_arc_exts[] = { "zip", "7z", "apk", "rar", "zst" };
+
+static unsigned db_arc_kind(const char *ext)
+{
+   unsigned i;
+
+   if (!ext || !*ext)
+      return 0;
+   for (i = 0; i < sizeof(db_arc_exts) / sizeof(db_arc_exts[0]); i++)
+      if (string_is_equal_noncase(ext, db_arc_exts[i]))
+         return i + 1;
+   return 0;
+}
+
+static int db_arc_by_key(const void *a, const void *b)
+{
+   uint64_t x = *(const uint64_t*)a;
+   uint64_t y = *(const uint64_t*)b;
+   if (x < y) return -1;
+   if (x > y) return  1;
+   return 0;
+}
+
+struct db_arc_build
+{
+   uint64_t *keys;
+   size_t    count;
+   size_t    capacity;
+   size_t    max_bytes;
+   uint8_t   unsized;
+   bool      failed;
+};
+
+static void db_arc_collect(struct db_arc_build *b, const char *name,
+      size_t name_len, const uint64_t *size)
+{
+   const char *dot;
+   unsigned    kind;
+
+   if (   !name_len
+       || !(dot = strrchr(name, '.'))
+       || !(kind = db_arc_kind(dot + 1)))
+      return;
+   /* A size the key cannot hold counts as none: read whole. */
+   if (!size || (*size >> 61))
+   {
+      b->unsized |= (uint8_t)(1u << kind);
+      return;
+   }
+   if (b->count == b->capacity)
+   {
+      size_t    cap = b->capacity ? b->capacity * 2 : 256;
+      uint64_t *tmp;
+
+      if (   (b->max_bytes && cap * sizeof(*b->keys) > b->max_bytes)
+          || !(tmp = (uint64_t*)realloc(b->keys, cap * sizeof(*tmp))))
+      {
+         b->failed = true;
+         return;
+      }
+      b->keys     = tmp;
+      b->capacity = cap;
+   }
+   b->keys[b->count++] = (*size << 3) | kind;
+}
+
+bool database_info_crc_index_may_hold_archive(
+      const database_info_crc_index_t *idx, const char *ext, uint64_t size)
+{
+   unsigned kind = db_arc_kind(ext);
+   uint64_t key;
+   size_t   lo   = 0;
+   size_t   hi;
+
+   if (   !idx || !idx->arc_have || !kind || !size || (size >> 61)
+       || (idx->arc_unsized & (1u << kind)))
+      return true;
+   key = (size << 3) | kind;
+   hi  = idx->arc_count;
+   while (lo < hi)
+   {
+      size_t mid = lo + ((hi - lo) / 2);
+      if (idx->arc_keys[mid] < key)
+         lo = mid + 1;
+      else
+         hi = mid;
+   }
+   return lo < idx->arc_count && idx->arc_keys[lo] == key;
+}
+
 bool database_info_crc_index_size_range(
       const database_info_crc_index_t *idx, int64_t *min, int64_t *max)
 {
@@ -1200,7 +1312,8 @@ size_t database_info_crc_index_bytes(const database_info_crc_index_t *idx)
 {
    if (!idx)
       return 0;
-   return idx->count * sizeof(*idx->entries) + sizeof(*idx);
+   return idx->count * sizeof(*idx->entries)
+      + idx->arc_count * sizeof(*idx->arc_keys) + sizeof(*idx);
 }
 
 void database_info_crc_index_free(database_info_crc_index_t *idx)
@@ -1209,6 +1322,7 @@ void database_info_crc_index_free(database_info_crc_index_t *idx)
       return;
    db_index_stream_close(idx->fd);
    free(idx->entries);
+   free(idx->arc_keys);
    free(idx->rdb_path);
    free(idx);
 }
@@ -1462,21 +1576,23 @@ static database_info_serial_index_t *db_serial_index_finish(
    return idx;
 }
 
-/* Both builds, fed by one walk.  Each may grow to the whole budget, so
- * a build for the serial lookup can hold twice it until the split at
- * the end. */
+/* The three builds, fed by one walk.  Each may grow to the whole
+ * budget, so a build of all three can hold three times it until the
+ * split at the end. */
 struct db_index_build
 {
    struct db_crc_build    crc;
    struct db_serial_build serial;
+   struct db_arc_build    arc;
    bool                   want_crc;
    bool                   want_serial;
+   bool                   want_arc;
    bool                   has_serial;
 };
 
 static int db_index_collect(void *ctx, const uint8_t *key, size_t key_len,
-      const uint8_t *serial, size_t serial_len, uint64_t offset,
-      const uint64_t *aux)
+      const uint8_t *serial, size_t serial_len, const char *name,
+      size_t name_len, uint64_t offset, const uint64_t *aux)
 {
    struct db_index_build *b = (struct db_index_build*)ctx;
 
@@ -1484,6 +1600,8 @@ static int db_index_collect(void *ctx, const uint8_t *key, size_t key_len,
       b->has_serial = true;
    if (b->want_crc && !b->crc.failed)
       db_crc_collect(&b->crc, key, key_len, offset, aux);
+   if (b->want_arc && !b->arc.failed)
+      db_arc_collect(&b->arc, name, name_len, aux);
    if (b->want_serial && !b->serial.failed)
       db_serial_collect(&b->serial, serial, serial_len, offset, NULL);
 
@@ -1578,6 +1696,7 @@ enum db_index_phase
    DB_INDEX_WALK = 0,
    DB_INDEX_SORT_CRC,
    DB_INDEX_SORT_SERIAL,
+   DB_INDEX_SORT_ARC,
    DB_INDEX_DONE,
    DB_INDEX_FAILED
 };
@@ -1606,6 +1725,9 @@ database_info_index_job_t *database_info_index_job_new(
 
    job->b.want_crc         = (want & DATABASE_INFO_INDEX_CRC)    != 0;
    job->b.want_serial      = (want & DATABASE_INFO_INDEX_SERIAL) != 0;
+   job->b.want_arc         = job->b.want_crc
+      && (want & DATABASE_INFO_INDEX_ARCHIVE_SIZES) != 0;
+   job->b.arc.max_bytes    = max_bytes;
    job->b.crc.max_bytes    = max_bytes;
    job->b.serial.max_bytes = max_bytes;
    job->max_bytes          = max_bytes;
@@ -1622,7 +1744,8 @@ database_info_index_job_t *database_info_index_job_new(
             job->b.want_crc ? "crc" : NULL,
             (want & (DATABASE_INFO_INDEX_SERIAL
                    | DATABASE_INFO_INDEX_HAS_SERIAL)) ? "serial" : NULL,
-            job->b.want_crc ? "size" : NULL);
+            job->b.want_crc ? "size" : NULL,
+            job->b.want_arc ? "rom_name" : NULL);
       libretrodb_close(db);
    }
    libretrodb_free(db);
@@ -1680,6 +1803,20 @@ bool database_info_index_job_step(database_info_index_job_t *job,
             if (!done)
                return false;
          }
+         job->phase = DB_INDEX_SORT_ARC;
+         return false;
+      case DB_INDEX_SORT_ARC:
+         if (b->want_arc && !b->arc.failed)
+         {
+            void *base = b->arc.keys;
+            bool  done = db_sort_step(&job->sort, &base, b->arc.count,
+                  b->arc.capacity, sizeof(*b->arc.keys), db_arc_by_key,
+                  entries, &b->arc.failed);
+
+            b->arc.keys = (uint64_t*)base;
+            if (!done)
+               return false;
+         }
          job->phase = DB_INDEX_DONE;
          return true;
       default:
@@ -1706,6 +1843,19 @@ void database_info_index_job_finish(database_info_index_job_t *job,
 
       if (crc && b->want_crc && !b->crc.failed)
          *crc = db_crc_index_finish(&b->crc, job->rdb_path);
+      /* Sizes that would go over the budget are dropped, and the archives
+       * they would have gated are read whole. */
+      if (   crc && *crc && b->want_arc && !b->arc.failed
+          && (   !job->max_bytes
+              ||   database_info_crc_index_bytes(*crc)
+                 + b->arc.count * sizeof(*b->arc.keys) <= job->max_bytes))
+      {
+         (*crc)->arc_keys    = b->arc.keys;
+         (*crc)->arc_count   = b->arc.count;
+         (*crc)->arc_unsized = b->arc.unsized;
+         (*crc)->arc_have    = true;
+         b->arc.keys         = NULL;
+      }
 
       /* The serial index gets what the crc index leaves. */
       if (job->max_bytes && crc && *crc)
@@ -1738,6 +1888,7 @@ void database_info_index_job_free(database_info_index_job_t *job)
    free(job->sort.tmp);
    free(job->b.crc.entries);
    free(job->b.serial.entries);
+   free(job->b.arc.keys);
    free(job->rdb_path);
    free(job);
 }

database_info.h

@@ -146,6 +146,7 @@ typedef struct
 #define DB_EXTRACT_SIZE     (1 << 3)
 #define DB_EXTRACT_MD5      (1 << 4)
 #define DB_EXTRACT_SHA1     (1 << 5)
+#define DB_EXTRACT_ROM_NAME (1 << 6)
 
 /* Preset used by the ROM scanner */
 #define DB_EXTRACT_SCAN_FIELDS \
@@ -179,6 +180,14 @@ size_t database_info_crc_index_count(const database_info_crc_index_t *idx);
 bool database_info_crc_index_size_range(
       const database_info_crc_index_t *idx, int64_t *min, int64_t *max);
 
+/* False only when every record whose rom_name ends in ".@ext" carries
+ * a size and none is @size.  True when the index was built without
+ * those sizes, so a caller reads such an archive as it would without
+ * an index; also for a type it keeps no sizes for, an unknown @size of
+ * 0 and a @size of 2^61 or more. */
+bool database_info_crc_index_may_hold_archive(
+      const database_info_crc_index_t *idx, const char *ext, uint64_t size);
+
 /* Records whose crc is @crc or @archive_crc, in file order, with
  * @fields extracted - the same list database_info_list_new_filtered()
  * returns for "{crc:or(...)}".  NULL if the lookup could not be
@@ -213,7 +222,10 @@ enum database_info_index_want
    DATABASE_INFO_INDEX_CRC        = (1 << 0),
    DATABASE_INFO_INDEX_SERIAL     = (1 << 1),
    /* Whether any record carries a serial, without the serial index. */
-   DATABASE_INFO_INDEX_HAS_SERIAL = (1 << 2)
+   DATABASE_INFO_INDEX_HAS_SERIAL = (1 << 2),
+   /* With CRC: the sizes of the records named for an archive, for
+    * database_info_crc_index_may_hold_archive(). */
+   DATABASE_INFO_INDEX_ARCHIVE_SIZES = (1 << 3)
 };
 
 /* Builds the crc index of a database and its serial index in one
@@ -222,10 +234,12 @@ enum database_info_index_want
 typedef struct database_info_index_job database_info_index_job_t;
 
 /* NULL when @rdb_path cannot be opened or @want names no index.
- * @max_bytes caps the crc index's entry table and the serial index
- * gets what the crc index leaves; an index that cannot be built whole
- * comes back NULL, because a partial index would miss matches without
- * saying so.  Zero means no limit. */
+ * @max_bytes caps the crc index's entry table; its archive sizes, then
+ * the serial index, get what that leaves.  An index that cannot be
+ * built whole comes back NULL, because a partial index would miss
+ * matches without saying so; sizes that do not fit are dropped, and
+ * the archives they would have gated are read whole.  Zero means no
+ * limit. */
 database_info_index_job_t *database_info_index_job_new(
       const char *rdb_path, size_t max_bytes, unsigned want);
 

libretro-common/file/archive_file.c

@@ -182,6 +182,7 @@ static int file_archive_parse_file_init(file_archive_transfer_t *state,
    char path[PATH_MAX_LENGTH];
    char *last                 = NULL;
 
+   state->torrentzip = false;
    strlcpy(path, file, sizeof(path));
 
    if ((last = (char*)path_get_archive_delim(path)))

libretro-common/file/archive_file_zlib.c

@@ -1366,6 +1366,13 @@ static int64_t zip_file_read(
    return (int64_t)decomp.size;
 }
 
+/* TorrentZip names itself in the end record's comment: TORRENTZIPPED-
+ * and eight hex digits. */
+static bool zip_comment_torrentzip(const uint8_t *comment, unsigned len)
+{
+   return len == 22 && !memcmp(comment, "TORRENTZIPPED-", 14);
+}
+
 static int zip_parse_file_init(file_archive_transfer_t *state,
       const char *file)
 {
@@ -1386,6 +1393,8 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
    uint64_t entries;
    zip_context_t *zip_context = NULL;
 
+   state->torrentzip = false;
+
    /* Minimal ZIP file size is 22 bytes */
    if (read_block < 22)
       return -1;
@@ -1402,7 +1411,11 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
             unsigned comment_len = read_le(footer + 20, 2);
             if ((int64_t)(footer - map) + 22 + comment_len
                   == state->archive_size)
+            {
+               state->torrentzip = zip_comment_torrentzip(footer + 22,
+                     comment_len);
                break;
+            }
          }
          if (footer == map)
             return -1;
@@ -1437,6 +1450,11 @@ static int zip_parse_file_init(file_archive_transfer_t *state,
             if (read_pos + (footer - footer_buf) + 22 + comment_len == state->archive_size)
             {
                eocd_at = read_pos + (footer - footer_buf);
+               /* Only a comment inside this block is looked at. */
+               state->torrentzip =
+                     (int64_t)(footer - footer_buf) + 22 + comment_len
+                        <= read_block
+                  && zip_comment_torrentzip(footer + 22, comment_len);
                break; /* found it! */
             }
          }

libretro-common/include/file/archive_file.h

@@ -110,6 +110,9 @@ typedef struct file_archive_transfer
     * that parks a member and lets the scan continue is counted when it
     * is found; one that parks and stops is counted when it drains. */
    bool     pending_counted;
+   /* The zip's end record carries a TorrentZip comment.  Set by the zip
+    * reader when it opens the archive, false for any other. */
+   bool     torrentzip;
 } file_archive_transfer_t;
 
 typedef struct

libretro-common/samples/file/archive_file/archive_zip_test.c

@@ -1244,6 +1244,109 @@ static void test_entry_source_bounds(void)
    printf("[SUCCESS] entry sources open only for members inside the archive\n");
 }
 
+/* TorrentZip ends a zip with the comment TORRENTZIPPED- and a CRC; the
+ * reader says so whether the end record is read or mapped. */
+static void tz_check(const char *label, const char *comment, bool lend,
+      bool want)
+{
+   static const uint8_t payload[4] = { 1, 2, 3, 4 };
+   uint8_t      buf[256];
+   const char  *tmp_path  = "rarch_zip_tz_test.zip";
+   uint32_t     crc       = encoding_crc32(0, payload, sizeof(payload));
+   size_t       clen      = comment ? strlen(comment) : 0;
+   size_t       len, cd_start;
+   bool         returnerr = true;
+   file_archive_transfer_t         *state    = (file_archive_transfer_t*)
+      calloc(1, sizeof(*state));
+   struct archive_extract_userdata *userdata =
+      (struct archive_extract_userdata*)calloc(1, sizeof(*userdata));
+
+   if (!state || !userdata)
+      abort();
+
+   len       = write_stored_lfh(buf, "a.bin", payload, sizeof(payload), crc);
+   cd_start  = len;
+   len      += write_stored_cfh(buf + len, "a.bin", sizeof(payload), crc, 0);
+   len      += write_eocd(buf + len, 1, (uint32_t)(len - cd_start),
+         (uint32_t)cd_start);
+   put_u16(buf + len - 2, (uint16_t)clen);
+   if (clen)
+      memcpy(buf + len, comment, clen);
+   len      += clen;
+   write_file(tmp_path, buf, len);
+   if (lend)
+      lend_vfs(buf, (int64_t)len);
+
+   state->type = ARCHIVE_TRANSFER_INIT;
+   file_archive_parse_file_iterate(state, &returnerr, tmp_path, NULL,
+         NULL, userdata);
+   if (state->type != ARCHIVE_TRANSFER_ITERATE || state->torrentzip != want)
+   {
+      printf("[FAIL] %s: torrentzip %d, want %d\n", label,
+            (int)state->torrentzip, (int)want);
+      failures++;
+   }
+   else
+      printf("[SUCCESS] %s\n", label);
+   file_archive_parse_file_iterate_stop(state);
+   if (lend)
+      lend_vfs(NULL, 0);
+   remove(tmp_path);
+   free(state);
+   free(userdata);
+}
+
+static void test_torrentzip_comment(void)
+{
+   tz_check("TorrentZip comment, read", "TORRENTZIPPED-5C56B2DD",
+         false, true);
+   tz_check("other comment, read", "RVZSTD-250F05A8", false, false);
+   tz_check("long TorrentZip comment, read", "TORRENTZIPPED-5C56B2DD0",
+         false, false);
+   tz_check("22-byte other comment, read", "TORRENTZIPPXD-5C56B2DD",
+         false, false);
+   tz_check("no comment, read", NULL, false, false);
+#ifdef VFS_HAVE_FILE_MAPPING
+   tz_check("TorrentZip comment, mapped", "TORRENTZIPPED-5C56B2DD",
+         true, true);
+   tz_check("other comment, mapped", "RVZSTD-250F05A8", true, false);
+   tz_check("long TorrentZip comment, mapped", "TORRENTZIPPED-5C56B2DD0",
+         true, false);
+   tz_check("22-byte other comment, mapped", "TORRENTZIPPXD-5C56B2DD",
+         true, false);
+#endif
+}
+
+/* A transfer the zip reader last opened a TorrentZip with says no such
+ * thing of the next archive, whichever backend opens it. */
+static void test_torrentzip_not_carried(void)
+{
+   bool returnerr = false;
+   file_archive_transfer_t *state =
+      (file_archive_transfer_t*)calloc(1, sizeof(*state));
+   struct archive_extract_userdata *userdata =
+      (struct archive_extract_userdata*)calloc(1, sizeof(*userdata));
+
+   if (!state || !userdata)
+      abort();
+   write_file("tz_carried.7z", (const uint8_t*)"7z", 2);
+   state->torrentzip = true;
+   state->type       = ARCHIVE_TRANSFER_INIT;
+   file_archive_parse_file_iterate(state, &returnerr, "tz_carried.7z",
+         NULL, NULL, userdata);
+   if (state->torrentzip)
+   {
+      printf("[FAIL] TorrentZip flag carried over to the next archive\n");
+      failures++;
+   }
+   else
+      printf("[SUCCESS] TorrentZip flag not carried over\n");
+   file_archive_parse_file_iterate_stop(state);
+   remove("tz_carried.7z");
+   free(state);
+   free(userdata);
+}
+
 int main(void)
 {
    test_truncated_entry();
@@ -1259,6 +1362,8 @@ int main(void)
    test_member_crc_mismatch();
    test_member_lent();
    test_entry_source_bounds();
+   test_torrentzip_comment();
+   test_torrentzip_not_carried();
 
    if (failures)
    {

libretro-db/libretrodb.c

@@ -694,19 +694,53 @@ static int rmsgpack_read_uint_inplace(intfstream_t *fd, uint64_t *out)
    return -1;
 }
 
+/**
+ * rmsgpack_read_str_inplace:
+ *
+ * Read a MsgPack string value into a caller buffer, NUL-terminated.
+ * Returns -1 and leaves the stream positioned after the value when it
+ * is not a string or does not fit, so the caller can carry on
+ * scanning.
+ */
+static int rmsgpack_read_str_inplace(intfstream_t *fd, char *buf,
+      size_t buf_size, uint64_t *len)
+{
+   uint8_t type = 0;
+   int32_t n;
+
+   if (intfstream_read(fd, &type, 1) != 1)
+      return -1;
+   if (intfstream_seek(fd, -1, RETRO_VFS_SEEK_POSITION_CURRENT) < 0)
+      return -1;
+   if (   (type < _MPF_FIXSTR || type >= _MPF_NIL)
+       && type != _MPF_STR8 && type != _MPF_STR16 && type != _MPF_STR32)
+   {
+      rmsgpack_skip_value(fd);
+      return -1;
+   }
+   /* One too long is stepped over by the reader. */
+   if ((n = rmsgpack_read_key_string(fd, buf, buf_size)) < 0)
+      return -1;
+   *len = (uint64_t)n;
+   return 0;
+}
+
 struct libretrodb_scan
 {
    intfstream_t *fd;
    const char   *field;
    const char   *field2;
    const char   *aux_field;
+   const char   *name_field;
    size_t        field_len;
    size_t        field2_len;
    size_t        aux_len;
+   size_t        name_len;
 };
 
 libretrodb_scan_t *libretrodb_scan_begin(libretrodb_t *db,
-      const char *field, const char *field2, const char *aux_field)
+      const char *field, const char *field2, const char *aux_field,
+      const char *name_field)
 {
    libretrodb_scan_t *scan;
 
@@ -721,9 +755,12 @@ libretrodb_scan_t *libretrodb_scan_begin(libretrodb_t *db,
       scan->field2_len = strlen(field2);
    if (aux_field && *aux_field)
       scan->aux_len    = strlen(aux_field);
+   if (name_field && *name_field)
+      scan->name_len   = strlen(name_field);
    scan->field     = field;
    scan->field2    = field2;
    scan->aux_field = aux_field;
+   scan->name_field = name_field;
 
    if (!scan->field_len && !scan->field2_len)
    {
@@ -767,9 +804,11 @@ int libretrodb_scan_step(libretrodb_scan_t *scan, unsigned max_records,
    const char   *field;
    const char   *field2;
    const char   *aux_field;
+   const char   *name_field;
    size_t        field_len;
    size_t        field2_len;
    size_t        aux_len;
+   size_t        name_len;
    unsigned      n;
    int           rv = -1;
 
@@ -783,6 +822,8 @@ int libretrodb_scan_step(libretrodb_scan_t *scan, unsigned max_records,
    field_len  = scan->field_len;
    field2_len = scan->field2_len;
    aux_len    = scan->aux_len;
+   name_field = scan->name_field;
+   name_len   = scan->name_len;
 
    /* Every record is read from its start to its end, so the stream's
     * position is all that carries over to the next call. */
@@ -795,10 +836,13 @@ int libretrodb_scan_step(libretrodb_scan_t *scan, unsigned max_records,
       int      have_key     = 0;
       int      have_key2    = 0;
       int      have_aux     = 0;
+      int      have_name    = 0;
       uint8_t  key_val[LIBRETRODB_MAX_KEY_SIZE];
       uint8_t  key2_val[LIBRETRODB_MAX_KEY_SIZE];
+      char     name_val[LIBRETRODB_MAX_KEY_SIZE];
       uint64_t key_val_len  = 0;
       uint64_t key2_val_len = 0;
+      uint64_t name_val_len = 0;
       uint64_t aux_val      = 0;
 
       if (record_start < 0)
@@ -819,6 +863,7 @@ int libretrodb_scan_step(libretrodb_scan_t *scan, unsigned max_records,
       have_key  = 0;
       have_key2 = 0;
       have_aux  = 0;
+      have_name = 0;
 
       for (i = 0; i < map_len; i++)
       {
@@ -859,6 +904,17 @@ int libretrodb_scan_step(libretrodb_scan_t *scan, unsigned max_records,
             continue;
          }
 
+         if (     name_len
+               && !have_name
+               && (size_t)key_len == name_len
+               && memcmp(key_buf, name_field, name_len) == 0)
+         {
+            if (rmsgpack_read_str_inplace(fd, name_val, sizeof(name_val),
+                     &name_val_len) == 0)
+               have_name = 1;
+            continue;
+         }
+
          if (     aux_len
                && !have_aux
                && (size_t)key_len == aux_len
@@ -879,7 +935,7 @@ int libretrodb_scan_step(libretrodb_scan_t *scan, unsigned max_records,
        * record rather than only the ones it can index.  Reporting only
        * on the key gave a range narrower than the data, which is the
        * wrong direction for a range used to decide what to skip. */
-      if (have_key || have_key2 || have_aux)
+      if (have_key || have_key2 || have_name || have_aux)
       {
          int64_t resume = intfstream_tell(fd);
 
@@ -887,6 +943,8 @@ int libretrodb_scan_step(libretrodb_scan_t *scan, unsigned max_records,
                   have_key ? (size_t)key_val_len : 0,
                   have_key2 ? key2_val : NULL,
                   have_key2 ? (size_t)key2_val_len : 0,
+                  have_name ? name_val : NULL,
+                  have_name ? (size_t)name_val_len : 0,
                   (uint64_t)record_start, have_aux ? &aux_val : NULL))
             stop = 1;
 

libretro-db/libretrodb.h

@@ -58,10 +58,11 @@ int libretrodb_find_entry(libretrodb_t *db, const char *index_name,
         const void *key, struct rmsgpack_dom_value *out);
 
 /* Reported once per record that carries the scanned field, the second
- * field or the companion one: the binary values of the two fields, the
- * offset of the record, and the companion numeric value when present
- * (NULL otherwise).  A field the record lacks comes as NULL with a
- * length of zero.
+ * field, the name field or the companion one: the binary values of the
+ * two fields, the name field's string (NUL-terminated), the offset of
+ * the record, and the companion numeric value when present (NULL
+ * otherwise).  A field the record lacks comes as NULL with a length of
+ * zero.
  *
  * A record that has only the companion is reported with @key NULL and
  * @key_len zero, so a caller accumulating over the companion sees
@@ -72,19 +73,22 @@ int libretrodb_find_entry(libretrodb_t *db, const char *index_name,
  * Returning non-zero stops the scan. */
 typedef int (*libretrodb_scan2_cb)(void *ctx, const uint8_t *key,
       size_t key_len, const uint8_t *key2, size_t key2_len,
-      uint64_t offset, const uint64_t *aux);
+      const char *name, size_t name_len, uint64_t offset,
+      const uint64_t *aux);
 
 /* Walks a database a run of records at a time, reporting @field and
  * @field2, either of which may be NULL but not both, so a caller
  * indexing both walks it once.  @aux_field may be NULL; naming one
  * collects a numeric field in the same pass and reports the records
- * carrying only it.  The scan reads through a stream of its own, so
- * @db may be closed once it has begun.  The field names are not
- * copied and must outlive the scan. */
+ * carrying only it.  @name_field may be NULL; naming one reports a
+ * string field, one too long for a key as absent.  The scan reads
+ * through a stream of its own, so @db may be closed once it has begun.
+ * The field names are not copied and must outlive the scan. */
 typedef struct libretrodb_scan libretrodb_scan_t;
 
 libretrodb_scan_t *libretrodb_scan_begin(libretrodb_t *db,
-      const char *field, const char *field2, const char *aux_field);
+      const char *field, const char *field2, const char *aux_field,
+      const char *name_field);
 
 /* Reports up to @max_records records, which must be non-zero.
  * Returns 1 while records remain, 0 once the walk is complete or @cb

libretro-db/samples/libretrodb/libretrodb_parser_test.c

@@ -842,12 +842,14 @@ static scan_ent_t scan_ents[64];
 static size_t     scan_n;
 
 static int scan_collect(void *ctx, const uint8_t *key, size_t key_len,
-      const uint8_t *key2, size_t key2_len, uint64_t offset,
-      const uint64_t *aux)
+      const uint8_t *key2, size_t key2_len, const char *name,
+      size_t name_len, uint64_t offset, const uint64_t *aux)
 {
    (void)ctx;
    (void)key2;
    (void)key2_len;
+   (void)name;
+   (void)name_len;
    if (key_len != 4 || scan_n >= sizeof(scan_ents) / sizeof(scan_ents[0]))
       return 0;
    scan_ents[scan_n].have_aux = (aux != NULL);
@@ -863,7 +865,7 @@ static int scan_collect(void *ctx, const uint8_t *key, size_t key_len,
 static int scan_all(libretrodb_t *db)
 {
    int                rv   = -1;
-   libretrodb_scan_t *scan = libretrodb_scan_begin(db, "crc", NULL, "size");
+   libretrodb_scan_t *scan = libretrodb_scan_begin(db, "crc", NULL, "size", NULL);
 
    if (scan)
       do
@@ -1021,7 +1023,7 @@ static void case_scan_steps(const char *dir)
       if (scan_all(db) == 0)
          n_whole = scan_n;
       memcpy(whole, scan_ents, n_whole * sizeof(*whole));
-      scan = libretrodb_scan_begin(db, "crc", NULL, "size");
+      scan = libretrodb_scan_begin(db, "crc", NULL, "size", NULL);
       libretrodb_close(db);
    }
    libretrodb_free(db);
@@ -1052,7 +1054,7 @@ static void case_scan_steps(const char *dir)
    scan = NULL;
    if ((db = libretrodb_new()) && libretrodb_open(path, db, false) == 0)
    {
-      scan = libretrodb_scan_begin(db, "crc", NULL, "size");
+      scan = libretrodb_scan_begin(db, "crc", NULL, "size", NULL);
       libretrodb_close(db);
    }
    libretrodb_free(db);
@@ -1062,6 +1064,88 @@ static void case_scan_steps(const char *dir)
    check(rv == 1, "stepped scan ended part way", "one record of five");
 }
 
+static char   name_got[8][16];
+static int    name_have[8];
+static size_t name_n;
+
+static int name_collect(void *ctx, const uint8_t *key, size_t key_len,
+      const uint8_t *key2, size_t key2_len, const char *name,
+      size_t name_len, uint64_t offset, const uint64_t *aux)
+{
+   (void)ctx;
+   (void)key;
+   (void)key2;
+   (void)key2_len;
+   (void)offset;
+   (void)aux;
+   if (key_len != 4 || name_n >= 8)
+      return 0;
+   name_have[name_n] = (name != NULL);
+   name_got[name_n][0] = '\0';
+   if (name && name_len < sizeof(name_got[0]))
+      strcpy(name_got[name_n], name);
+   name_n++;
+   return 0;
+}
+
+/* A name field that is no string, or too long for a key, is absent,
+ * and the scan carries on to the key after it. */
+static void case_name_field(const char *dir)
+{
+   char               path[512];
+   static char        longstr[300];
+   buf_t              body, meta;
+   uint8_t            crc[4] = { 1, 2, 3, 4 };
+   unsigned           i;
+   int                rv = -1, ok;
+   libretrodb_t      *db = libretrodb_new();
+   libretrodb_scan_t *scan = NULL;
+
+   memset(&body, 0, sizeof(body));
+   memset(&meta, 0, sizeof(meta));
+   memset(longstr, 'x', sizeof(longstr));
+   for (i = 0; i < 4; i++)
+   {
+      bfixmap(&body, 2);
+      bfixstr(&body, "rom_name");
+      if (i == 0)
+         bfixstr(&body, "set.zip");
+      else if (i == 1)
+         buint8(&body, 7);
+      else if (i == 2)
+         bstr32(&body, longstr, sizeof(longstr));
+      else
+         bfixstr(&body, "x");
+      bfixstr(&body, "crc");
+      bbin(&body, crc, 4);
+   }
+   bnil(&body);
+   meta_count(&meta, 1);
+   sprintf(path, "%s/name_field.rdb", dir);
+   write_rdb(path, &body, &meta);
+   bfree(&body);
+   bfree(&meta);
+
+   begin("name field: string, else absent");
+   name_n = 0;
+   if (db && libretrodb_open(path, db, false) == 0)
+   {
+      scan = libretrodb_scan_begin(db, "crc", NULL, NULL, "rom_name");
+      libretrodb_close(db);
+   }
+   libretrodb_free(db);
+   while (scan && (rv = libretrodb_scan_step(scan, 4, name_collect,
+               NULL)) == 1)
+      ;
+   libretrodb_scan_end(scan);
+   ok =    rv == 0 && name_n == 4
+        && name_have[0] && !strcmp(name_got[0], "set.zip")
+        && !name_have[1] && !name_have[2]
+        && name_have[3] && !strcmp(name_got[3], "x");
+   check(ok, "name field: string, else absent",
+         ok ? "non-string and too long: absent, key kept" : "wrong");
+}
+
 int main(int argc, char **argv)
 {
    const char *dir = (argc > 1) ? argv[1] : "/tmp";
@@ -1084,6 +1168,7 @@ int main(int argc, char **argv)
    case_index_round_trip(dir);
    case_field_scan(dir);
    case_scan_steps(dir);
+   case_name_field(dir);
 
    printf("\n%d checks, %d failures\n", checks, failures);
    return failures ? 1 : 0;

samples/tasks/database/Makefile

@@ -247,21 +247,24 @@ $(INDEX_TEST)$(EXE_EXT): $(INDEX_TEST_OBJECTS)
 # The wraps count the scanner's database walks and parks, slow chosen
 # reads and tell the scan's own from its readers', spend its window,
 # count and refuse reader pushes, count queue worker starts, count an
-# archive's stored-CRC reads, fail a chosen serial's query, order the
-# database and content lists, count reader passes, hold chosen lookups
-# and hold a chosen index build in one step.
+# archive's stored-CRC and directory reads, fail a chosen serial's
+# query, order the database and content lists, count reader passes,
+# hold chosen lookups, count an archive's member hits a step and hold
+# a chosen index build in one step.
 SCAN_TEST_WRAPS := -Wl,--wrap=libretrodb_open -Wl,--wrap=task_queue_park \
 	-Wl,--wrap=intfstream_crc_step -Wl,--wrap=rm3u_is_m3u \
 	-Wl,--wrap=task_nbio_slice_open -Wl,--wrap=task_nbio_slice_close \
 	-Wl,--wrap=task_queue_push \
-	-Wl,--wrap=file_archive_get_file_crc32_size_and_ext \
+	-Wl,--wrap=file_archive_get_file_crc32_and_size \
+	-Wl,--wrap=file_archive_parse_file_iterate \
 	-Wl,--wrap=bin_to_hex_alloc -Wl,--wrap=dir_list_new \
 	-Wl,--wrap=task_queue_is_threaded \
 	-Wl,--wrap=database_info_list_new_crc \
 	-Wl,--wrap=database_info_index_job_new \
 	-Wl,--wrap=database_info_dir_init_from_list \
 	-Wl,--wrap=database_info_index_job_step \
-	-Wl,--wrap=task_nbio_slice_within_budget
+	-Wl,--wrap=task_nbio_slice_within_budget \
+	-Wl,--wrap=fill_pathname_join_delim
 ifeq ($(HAVE_THREADS), 1)
 SCAN_TEST_WRAPS += -Wl,--wrap=sthread_create
 endif

samples/tasks/database/database_index_test.c

@@ -311,6 +311,66 @@ static int write_db_sizes(const char *path, int count, int zero_at,
    return write_rdb(path, &body);
 }
 
+/* Records named for archives: set.zip of 300 bytes, set.7z of 400 and
+ * a ROM's name of 500; with @unsized, an arc.zip with no size too. */
+static int write_db_archives(const char *path, bool unsized)
+{
+   static const char *const roms[3] = { "set.zip", "set.7z", "game.bin" };
+   buf_t body;
+   int   i;
+
+   memset(&body, 0, sizeof(body));
+   for (i = 0; i < 3; i++)
+   {
+      char    name[32];
+      uint8_t crc[4];
+
+      crc[0] = 0xA0; crc[1] = 0; crc[2] = 0; crc[3] = (uint8_t)i;
+      sprintf(name, "Archive %d", i);
+      bfixmap(&body, 4);
+      bfixstr(&body, "name");     bfixstr(&body, name);
+      bfixstr(&body, "rom_name"); bfixstr(&body, roms[i]);
+      bfixstr(&body, "crc");      bbin(&body, crc, 4);
+      bfixstr(&body, "size");     buint16(&body, (uint16_t)(300 + 100 * i));
+   }
+   if (unsized)
+   {
+      uint8_t crc[4];
+
+      crc[0] = 0xA0; crc[1] = 0; crc[2] = 0; crc[3] = 9;
+      bfixmap(&body, 3);
+      bfixstr(&body, "name");     bfixstr(&body, "Unsized");
+      bfixstr(&body, "rom_name"); bfixstr(&body, "arc.zip");
+      bfixstr(&body, "crc");      bbin(&body, crc, 4);
+   }
+   return write_rdb(path, &body);
+}
+
+/* @count records, each named for a zip of its own, sizes from 1000 up. */
+static int write_db_zips(const char *path, int count)
+{
+   buf_t body;
+   int   i;
+
+   memset(&body, 0, sizeof(body));
+   for (i = 0; i < count; i++)
+   {
+      char    name[16], rom[16];
+      uint8_t crc[4];
+
+      crc[0] = 0xB0; crc[1] = 0;
+      crc[2] = (uint8_t)(i >> 8); crc[3] = (uint8_t)i;
+      sprintf(name, "Zip %04d", i);
+      sprintf(rom,  "z%04d.zip", i);
+      bfixmap(&body, 4);
+      bfixstr(&body, "name");     bfixstr(&body, name);
+      bfixstr(&body, "rom_name"); bfixstr(&body, rom);
+      bfixstr(&body, "crc");      bbin(&body, crc, 4);
+      bfixstr(&body, "size");     buint16(&body, (uint16_t)(1000 + i));
+   }
+   return write_rdb(path, &body);
+}
+
 /* The crc of record @i of write_db_many(): scattered, so that the
  * index has to sort them, and every hundredth repeats the one before
  * it. */
@@ -428,6 +488,19 @@ static database_info_serial_index_t *serial_index_new(const char *path,
    return idx;
 }
 
+static database_info_crc_index_t *crc_index_want(const char *path,
+      size_t max_bytes, unsigned want)
+{
+   database_info_crc_index_t *idx = NULL;
+   database_info_index_job_t *job = database_info_index_job_new(path,
+         max_bytes, DATABASE_INFO_INDEX_CRC | want);
+
+   while (job && !database_info_index_job_step(job, 4096, 65536))
+      ;
+   database_info_index_job_finish(job, &idx, NULL, NULL);
+   return idx;
+}
+
 /* Builds @path's indexes in one walk under @max and as two builds, the
  * crc index first and the serial index on what it leaves.  False when
  * the two disagree on which come back; *@crc_only says the one walk
@@ -747,6 +820,97 @@ static void check_index_job_edges(const char *dir)
    free(path);
 }
 
+/* The sizes of the records named for an archive, beside the crc index:
+ * an archive of a type and size no such record has is not held. */
+static void check_archive_sizes(const char *dir)
+{
+   char path[512], path_u[512], detail[64];
+   database_info_crc_index_t *idx;
+   int  i;
+
+   sprintf(path,   "%s/index_archives.rdb",         dir);
+   sprintf(path_u, "%s/index_archives_unsized.rdb", dir);
+   if (!write_db_archives(path, false) || !write_db_archives(path_u, true))
+   {
+      check(0, "archive size databases", "could not be written");
+      return;
+   }
+
+   if (!(idx = crc_index_want(path, 0, DATABASE_INFO_INDEX_ARCHIVE_SIZES)))
+      check(0, "archive size index built", "no index");
+   else
+   {
+      check(database_info_crc_index_may_hold_archive(idx, "zip", 300),
+            "archive sizes: a zip record's size is held", "set.zip 300");
+      check(database_info_crc_index_may_hold_archive(idx, "ZIP", 300),
+            "archive sizes: the type in any case", "ZIP 300");
+      check(!database_info_crc_index_may_hold_archive(idx, "zip", 301),
+            "archive sizes: another size is not", "zip 301");
+      check(!database_info_crc_index_may_hold_archive(idx, "zip", 400),
+            "archive sizes: another type's size is not", "zip 400");
+      check(database_info_crc_index_may_hold_archive(idx, "7z", 400),
+            "archive sizes: 7z kept apart", "set.7z 400");
+      check(!database_info_crc_index_may_hold_archive(idx, "zip", 500),
+            "archive sizes: a ROM's name is no archive", "game.bin 500");
+      check(database_info_crc_index_may_hold_archive(idx, "chd", 12345),
+            "archive sizes: a type not kept may be held", "chd");
+      check(database_info_crc_index_may_hold_archive(idx, "zip", 0),
+            "archive sizes: an unknown size may be held", "zip 0");
+      database_info_crc_index_free(idx);
+   }
+
+   if (!(idx = crc_index_want(path, 0, 0)))
+      check(0, "index without archive sizes built", "no index");
+   else
+   {
+      check(database_info_crc_index_may_hold_archive(idx, "zip", 301),
+            "archive sizes: an index without them may hold any",
+            "zip 301");
+      database_info_crc_index_free(idx);
+   }
+
+   if (!(idx = crc_index_want(path_u, 0, DATABASE_INFO_INDEX_ARCHIVE_SIZES)))
+      check(0, "unsized archive index built", "no index");
+   else
+   {
+      check(database_info_crc_index_may_hold_archive(idx, "zip", 999),
+            "archive sizes: an unsized zip record may hold any zip",
+            "arc.zip");
+      check(!database_info_crc_index_may_hold_archive(idx, "7z", 999),
+            "archive sizes: only for its own type", "7z 999");
+      database_info_crc_index_free(idx);
+   }
+
+   check(database_info_crc_index_may_hold_archive(NULL, "zip", 1),
+         "archive sizes: no index may hold any", "NULL");
+
+   /* 1024 records fill the crc table, so a budget of the index without
+    * sizes holds the table and the sizes' build, but not both kept. */
+   sprintf(path_u, "%s/index_archives_many.rdb", dir);
+   if (!write_db_zips(path_u, 1024))
+   {
+      check(0, "archive size budget database", "could not be written");
+      return;
+   }
+   for (i = 0; i < 2; i++)
+   {
+      database_info_crc_index_t *full = crc_index_want(path_u, 0,
+            i ? DATABASE_INFO_INDEX_ARCHIVE_SIZES : 0);
+      size_t max = database_info_crc_index_bytes(full);
+
+      database_info_crc_index_free(full);
+      idx = crc_index_want(path_u, max, DATABASE_INFO_INDEX_ARCHIVE_SIZES);
+      sprintf(detail, "zip 999 under %u bytes", (unsigned)max);
+      if (i)
+         check(idx && !database_info_crc_index_may_hold_archive(idx, "zip",
+                  999), "archive sizes: kept within the budget", detail);
+      else
+         check(idx && database_info_crc_index_may_hold_archive(idx, "zip",
+                  999), "archive sizes: dropped over the budget", detail);
+      database_info_crc_index_free(idx);
+   }
+}
+
 int main(int argc, char **argv)
 {
    const char *dir = (argc > 1) ? argv[1] : "/tmp";
@@ -897,6 +1061,7 @@ int main(int argc, char **argv)
    check_one_walk(dir, path_a);
    check_index_job(dir);
    check_index_job_edges(dir);
+   check_archive_sizes(dir);
 
    printf("\n%d checks, %d failures\n", checks, failures);
    return failures ? 1 : 0;