mirror of
https://github.com/libretro/RetroArch.git
synced 2026-11-07 08:36:02 -10:00
Davey Hughesandlibretroadmin
770391e8d5
* scan: read an archive's directory once for all its passes The scan fetched an archive's first member's CRC, size and extension through a helper that stops at the first entry and reports how many the header counts, and its readers did the same. One walk now reads the whole directory, in the readers as in the scan: every member's stored CRC, size and name, kept for each of the archive's passes. Its transfer state is on the heap, out of the readers' frames. The walk replaces the helper in the archive passes; the helper stays for its other users. The walk decodes the first member for a CRC where the archive stores none, a zstd stream or a RAR 5 member hashed with BLAKE2, so it is looked up as before. Later members are not decoded and keep a CRC of 0. A .bz2 is decoded to be listed at all. Folders are not members: an archive of a folder and one file has one member, that file, and is looked up by its CRC first. * zip: report a TorrentZip end comment A zip whose end-of-central-directory comment is TORRENTZIPPED- and eight hex digits was written by TorrentZip, as arcade sets are. The zip reader already reads that record to find the directory, so it now says so in the transfer state, for a caller that wants the hint. The CRC in the comment is not checked: nothing depends on it. * database_info, libretro-db: archive record sizes beside the crc index The crc index build walks each database once, off the main thread. In the same walk it now also collects the size of every record whose rom_name ends in an archive extension (zip, 7z, apk, rar, zst), kept in one sorted array keyed by size and type, so a caller can ask whether a database may hold an archive of a given type and size. A record of that type with no size, or an index built without the sizes, answers yes. The sizes count against the index budget. The stepped database scan reports one string field for this: libretrodb_scan_begin() and its callback gain a name field. A record's rom_name can also be extracted on request. * scan: look every archive member up from the directory An archive whose first member missed was read whole and, when that missed too, expanded: each member went through the databases again, by the stored CRC the directory already held. An archive of several members was read whole before any lookup, even where no database could hold it, so disc zips and console sets were read whole again. One member: its stored CRC is looked up first, as before. A hit in a database claimed only for the member's type is held, and every member's CRC is looked up next. On a miss the archive is read whole only when a database whose cores open its type may hold it by its exact size; its own CRC is then looked up there, and where that is not asked or misses, every member's. Several members: the same size question comes first. Each database whose cores open the type is then asked for the first member's CRC, and for the archive's own CRC only when the size let it be read whole, so an arcade record still wins and no database claimed only for a member's type is walked before it. Where neither matches, every member's stored CRC is looked up. A database may hold the archive when it has a record whose rom_name is of the archive's type and size, one of that type with no size, or no sizes at all. Every member is looked up with the gates and size checks the expansion uses. A hit names the archive itself only when it is the one record hit, that record is named for this archive and its database's cores open the type; other hits are written as the expansion wrote them, archive#member. The expansion stays for scans that keep unmatched files, for archives whose directory cannot be read, and where nothing hit and a member stores no CRC, which only the expansion unpacks for one. Readers read an archive whole ahead only after a whole match, twice as many after each such match up to the cap; any other verdict takes the rest back. A TorrentZip set of several members is still read whole by a reader with no grant while some database opens zip. An archive over the readers' cutoff is handed back before a grant is spent on it. This narrows "several members are read whole first" to "read whole when a database may hold it by its size". A record whose crc is an archive's own CRC but whose rom_name is not of the archive's type, or whose size is not the file's, is no longer matched; each of FBNeo's 596 sets carries its zip's size. A zip of a folder and one ROM takes the one-member order. * database_scan_test: rar archives through every pass A RAR 4 archive of stored entries, written by hand as the zips are, takes the passes zip and 7z take: a first member's hit where a core opens rar names the archive, a later member's elsewhere is held as archive#member, and the archive is read whole only while a database whose cores open rar holds a rar record of its size. * database_scan_test: define the fixture byte writers once * archive_file: clear the TorrentZip flag for every archive The flag was cleared only by the zip reader, so a transfer reused for a 7z, a rar or an archive with no backend kept what the last zip said. It is now cleared before any backend opens the archive. * scan: readers read TorrentZip sets whole while the scan asks for that Readers read every TorrentZip set of several members whole while some database opens zip, but where no database may hold a set by its size the scan drops that CRC, so a folder of sets no database holds was read whole for nothing. Each TorrentZip set the scan rules on now says whether its own CRC was asked, and the readers read the next ones whole only while it was. --------- Co-authored-by: libretroadmin <[email protected]>
libretrodb
A small read only database Mainly to be used by retroarch
Usage
Files specified later in the chain will override earlier ones if the same key exists multiple times.
- To list out the content of a db
libretrodb_tool <db file> list - To create an index
libretrodb_tool <db file> create-index <index name> <field name> - To find an entry with an index
libretrodb_tool <db file> find <index name> <value>
Compiling a single DAT into a single RDB with c_converter
git clone https://github.com/libretro/libretro-super.git
cd libretro-super
./libretro-fetch.sh retroarch
cd retroarch
./configure
cd libretro-db
c_converter "NAME_OF_RDB_FILE.rdb" "NAME_OF_SOURCE_DAT.dat"
Compiling multiple DATs into a single RDB with c_converter
Specify rom.crc as the second parameter to use CRC as the unique fingerprint.
git clone https://github.com/libretro/libretro-super.git
cd libretro-super
./libretro-fetch.sh retroarch
cd retroarch
./configure
cd libretro-db
c_converter "NAME_OF_RDB_FILE.rdb" "rom.crc" "NAME_OF_SOURCE_DAT_1.dat" "NAME_OF_SOURCE_DAT_2.dat" "NAME_OF_SOURCE_DAT_3.dat"
Compiling all RDBs with libretro-build-database.sh
This approach builds and uses the c_converter program to compile the databases
git clone https://github.com/libretro/libretro-super.git
cd libretro-super
./libretro-fetch.sh retroarch
./libretro-build-database.sh
Lua DAT file converter
To convert a dat file use:
dat_converter <db file> <dat file>
If you want to merge multiple dat files you need to run:
dat_converter <db file> <match key> <dat file> ...
for example:
dat_converter snes.rdb rom.crc snes1.dat snes2.dat
Query examples
Some examples of queries you can use with libretrodbtool:
- Glob pattern matching Usecase : Search for all games starting with 'Street Fighter' in the 'name' field (glob pattern matching)
libretrodb_tool <db file> find "{'name':glob('Street Fighter*')}"
- Combined number matching query Usecase: Search for all games released on October 1995.
libretrodb_tool <db file> find "{'releasemonth':10,'releaseyear':1995}"
- Hash matching query Usecase: Search for any game matching a given crc32, in this case Soul Blazer (USA) for the SNES. Also works with serial, md5, and sha1.
libretrodb_tool <db file> find "{'crc':b'31B965DB'}"
- Names only search Usecase: Search for all games released on October 1995, wont print checksums, filename or rom size, only the game name.
libretrodb_tool <db file> get-names "{'releasemonth':10,'releaseyear':1995}"
Writing Lua converters
In order to write you own converter you must have a lua file that implements the following functions:
-- this function gets called before the db is created and should validate the
-- arguments and set up the ground for db insertion
function init(...)
local args = {...}
local script_name = args[1]
end
-- this is in iterator function. It is called before each insert.
-- the function should return a table for insertion or nil when there are no
-- more records to insert.
function get_value()
return {
key = "value", -- will be saved as string
num = 3, -- will be saved as int
bin = binary("some string"), -- will be saved as binary
unum = uint(3), -- will be saved as uint
some_bool = true, -- will be saved as bool
}
end