A sprite reading the buffer it draws into, linear, from beside where
it draws, is a blur in place: its texture is held for the render pass
and read as the buffer was before it, rather than the pass ending at
every column whose page the last one wrote. The rule matched a UV box
exactly two texels larger than the draw on both axes at a bias of one;
a blur sampling an axis on texel centres, or at the texture's edge,
has fewer extra texels there, and one reaching a few texels aside has
a larger bias. The rule now takes zero to two extra texels an axis and
a bias up to four.
A game blooming through such a blur, a column of 32 at a time: 80
render passes a frame become 33, with 2130 of every 2400 passes
before holding one to three primitives.
tests/swdraw blur_in_place: a buffer blurred in place a column at a
time every frame, brightened until white and read back on every
renderer, in at most two render passes a frame on paraLLEl-GS.
A target drawn at another buffer width has its pages moved to that
width's places first, a copy a page. When the draw keeps nothing of
the target (no blend reading it, no mask, no test) and covers every
page at the new width, what is moved is overwritten at once;
RelayoutTarget now takes that and leaves the new texture unwritten
instead. A game redrawing a buffer at two widths in turn every frame:
32 image copies a frame on the Vulkan renderer become none, the
pictures the same.
tests/swdraw width_overwritten: a buffer drawn four pages wide and
then five every frame, each draw covering all forty pages, then read
as a texture: the same picture on the software renderer, paraLLEl-GS
and the Vulkan renderer, and no image copy a frame on the latter.
A colour draw at another buffer width over a depth buffer's pages had
the whole depth converted to colour at the depth's width, and the
converted target then laid out again at the draw's width with a copy
a run of pages. RelayoutTargetWidth now takes the texture the pages
come from and the conversion shader, and converts each run straight
into its place; the full-size conversion and the copies after it are
gone. Taken when the target has no dirty rects (they are in the old
layout) and both are at one scale; otherwise as before.
A game filling its depth buffer's RGB through a narrow colour frame
every frame: 62 image copies a frame on the Vulkan renderer become
32, the pictures the same.
tests/swdraw depth_narrowed: a depth buffer ten pages wide drawn every
frame, a colour draw two pages wide with its alpha masked over its
pages, and the pages read back as a texture at that width: the same
readback and picture on the software renderer, paraLLEl-GS and the
Vulkan renderer, and no image copy a frame on the latter.
The AVX2 setup code made the Gouraud colour steps of a primitive in
scratch slots of the scanline temporaries, among them temp.uv_minmax.
That slot is not a temporary: with a constant LOD (TEX1.LCM, or UV
coordinates, or every level past MXL) BeginDraw puts the mip level's
clamp there once for the whole draw, and the scanline code masks every
texel coordinate with it. A primitive after the first found a colour
step there instead - a small negative number for a ramp - and a repeated
texture's coordinates, masked by that, reached megabytes past the level:
a racing game's race start read 8 MB past an 8x8 level and the
rasterizer thread died in the fetch.
The steps go in temp.lod instead, which the scanline code writes before
it reads in the per-pixel LOD path and never reads in the constant one.
tests/swdraw: mip_shaded, a mipmapped PSMT8 texture repeated across
Gouraud triangles at a constant LOD of 1. The swdraw lane holds its
picture on four rasterizers to the one on one, and the vknegotiate lane
holds the software renderer's, on one and on four, to Vulkan's and
paraLLEl-GS's; before, the three differed.
Every change of texture allocated and wrote a TFX texture descriptor set,
and every change of target a target set: 462 sets a frame for 580 draws
in a racing game, as its draws cycle through their textures, each a
vkAllocateDescriptorSets and a vkUpdateDescriptorSets (and on a driver
allocating sets from the heap, a malloc).
The sets made on a command buffer are kept in a table keyed by what they
bind (the two views, the sampler, the layouts), cleared with the pool
when the command buffer comes round again; a draw whose binding an
earlier draw on the buffer had takes that set. The same game now makes
157 sets a frame, every frame hash-identical.
tests/swdraw: texture_alternate, sixteen sprites a frame reading two
textures in turn. tests/vknegotiate counts the Vulkan renderer's
descriptor sets a steady frame: at most eight (five: the two textures,
the target, the readback and the present), where a set a draw made
nineteen.
Three things were made anew every frame a game streams its textures:
The texture slab kept only power-of-two images up to 1024 a side, and
the extents textures are cut to by their clamps rarely are: 34x113,
640x448 and the like, a hundred VkImages a frame in a racing game, each
made and destroyed with its views. The slab now keys on the exact
dimensions, levels and layers, any size up to 4096 a side, in buckets
walked newest first; the byte budget and its flush stay as they were.
Granite's staging pool kept 32 blocks of 64 KB, two megabytes, where a
frame's uploads ran to three; and an upload larger than a block took a
block of its own that the pool never kept, so every texture past 64 KB
made a VkBuffer for itself. The pool keeps 256 blocks' worth of bytes
and keeps oversized blocks within that budget, handing one back to the
next request it fits; a plain request still takes a plain block.
The scratch buffer a frame's render passes fill (32 MB, mostly coarse
tile lists) was dropped when full and a new one made. A full one is set
aside with the timeline value whose signal says the GPU is done with it,
and the next scratch of its domain takes it back once that value is
reached; up to four are kept.
In the racing game a steady frame on paraLLEl-GS went from 100 images
and 9 buffers created to none, every frame hash-identical.
tests/swdraw: stream_clamped, a 1024x120 texture streamed six times a
frame (each upload larger than a staging block) and read through a
region clamp 34x113. tests/vknegotiate counts a steady frame's images
and buffers on paraLLEl-GS: none.
InvalidateVideoMem took every colour draw over a source's pages as
rewriting them, whatever the draw's frame mask kept. A game that keeps a
PSMT8H texture in the top byte of its depth buffer's words and draws RGB
over those pages through a 32-bit frame with the alpha masked lost the
texture every frame: the source was dropped and made again, its memory
hashed (1 MB a frame here) for the hash cache to find the texture it
already had; past the cache's size, uploaded again.
The frame mask of the draw doing the write now reaches the invalidation,
and a 32-bit write whose mask keeps every bit a source reads (the top
byte for 8H, its nibbles for 4HL and 4HH, the low three bytes for 24-bit)
leaves that source alone, in the per-page walk and in the same-base
removals.
tests/swdraw: alpha_under_depth_big, the alpha uploaded as PSMT8H and
read as a texture too large for the hash cache, the masked colour draw
one page wide at the depth buffer's base over its first page.
tests/vknegotiate counts the Vulkan renderer's uploads over a steady
frame: one (the buffer's RGB the depth draw dirtied), where the dropped
texture's page made two.
Each render pass bound its textures through a bindless descriptor set
of its own, allocated from the pool with the pass's count; a driver
that backs every variable-count set with its layout's whole capacity
(lavapipe: four megabytes a set) then paid that for every pass, and a
loading screen's hundreds of passes a second grew past memory within a
minute. A pool now holds one set of every descriptor it has, allocated
once, and each pass takes a range of it, written at its place while
the set stays bound (update after bind); the shader indexes its
textures from the pass's base (GlobalConstants.bindless_base). A
descriptor-buffer pool already worked this way. Granite's
BindlessAllocator, unused, goes.
tests/vknegotiate counts descriptor set allocations (descsets) and
checks a steady frame makes none.
A sky's clouds, read through a palette the game rebuilds in a target
every frame, came out as a hatched blotch at 4x and were right at
native. Three things at scale, each pinned by a case:
A target preloaded from local memory was upscaled with a linear
filter (the bilinear upload hack, on by default and through the GameDB
for a hundred titles), so a palette entry's block of fragments held a
gradient of its neighbours. A pixel of local memory is now a block of
the scale's fragments, as the GS has it; the hack, its config and its
GameDB fix go.
A palette copied from a target is one value an entry: its native
pixel's, read at the place among the pixel's fragments where the draw
at scale put the native sample (NativeSample: the first fragment's
centre on the kept grid, the pixel's middle under the Normal and
Native offsets, between two fragments at an even scale, with a linear
fetch). The copy that kept every fragment as a sample of its own, and
let each fragment of a draw read its own, gave every fragment a
different entry; the readback at scale took the block's middle for
every draw, half a pixel off on the kept grid, and takes the same
place now.
A point is drawn at the pixel its vertex is in, whatever the vertex's
fraction: its centre for a sized point, which spans half the size
either way, its origin for one expanded into a quad; a sized point at
scale was centred on the pixel's corner and covered a quarter of it.
tests/swdraw: palette_bilinear, palette_bilinear_drawn and
palette_shaded, native and at 4x on both renderers in
tests/vknegotiate, which runs palette_engulfed at 4x too;
tests/gsclut/clut_from_target checks the entry's sample.
A sky blended by the alpha of a gradient drawn into a small buffer,
then taken by the wide buffer around it, came out grey from a fresh
start: the wide buffer's draws had masked alpha, so its alpha was not
valid, and the first read needing it laid local memory's flat alpha
over the whole buffer, the gradient's page included. Four things in
CopyTargetPagesInto, and one each side of it:
A channel the older target holds that the new one has not drawn is
the new one's from the copy on: its other pages take local memory's
first, as a preloaded target has them, and the channel is marked
valid, so a read of it finds the copied page rather than refilling
it. The alpha range grows over the older target's, or a blend by the
copied page's alpha was skipped as flat. Both keep their alpha as the
GS does before the copy; scaled, an alpha past 0x80 (a palette's)
cannot be. The pages taken count as drawn, and the target is valid
over them, so a readback carries them to local memory - the older
target, parked, leaves them to it.
CreateTarget scaled a new target's alpha by setting the flag over
whatever it held; it now scales the texture, and only where the alpha
allows.
A state load dropped every target and marked only the pages whose
bytes the state changed as restored, so a target made again over an
unchanged page started blank past its first draw. TargetsRestored
marks the dropped targets' pages, whether the state changed them or
not.
tests/swdraw: engulfed_alpha, on both renderers in tests/vknegotiate
and the software renderer in tests/swdraw, with a savestate round
trip.
A sky drawn once into the alpha of a wide buffer, its palette drawn
into a small one, and the gradient under it drawn into half a page,
came out white or black from a fresh start, and black again from a
state the hardware renderer had saved. Four things in the texture
cache, all of the same kind - a page of local memory holds several
targets' pixels, a channel each, and the cache treated it as one
target's:
A target leaving the cache skipped any page a transfer had written,
whatever the transfer wrote: a 24-bit Z laid over the sky's pages by a
depth buffer of another width marked their RGB dirty, and the alpha
holding the sky was never written to local memory. A dirty rect now
takes away only the channels it was written with, in DropKeepsPage and
in the read's own dirty check, and the page stamps a read and a drop
leave - m_page_read_draw and DropBatch::page_draw - are per channel, so
a read of a target's RGB leaves the page's alpha to whoever holds it. A
dropped target claims only the channels it has drawn.
A target made over an older one of another width left the older one
in the lists and started blank where its pages were; the palette
lookup, going by the drawn rect's bounds, then read the blank page.
CopyTargetPagesInto carries the older target's pages into the new one
page for page, as RelayoutTargetWidth lays them out, and parks the
older one; a display target, which spans the buffers drawn inside it,
is left out.
A texture based at a block inside a page of a target was read from
local memory, which had never seen it. The GS adds each texel's block
to the base, so a read kept to one page whose blocks share no bit with
the base's block in it is the target's texels from that block's place
on: LookupSource serves it from the target, at that offset.
A state the hardware renderer saved wrote every target back whole, a
depth buffer's conversion over the sky's alpha; ReadbackAll now goes by
the drop rule, and a target made over pages a state restored, which
the cache cannot know the GPU had, starts from local memory
(mem_restored, TakeRestoredPages).
tests/swdraw: alpha_dirtied_parked, palette_engulfed and
block_offset_read pin the four, on both hardware renderers and across a
state saved by one and loaded.
A colour target whose first pages a new target takes over (PreloadTarget
copies them into the new one and let the old target go) is now parked
as a target dropped by age is: its pages go to local memory when they
are read. A page a later target holds is written for the channels that
target does not hold: a 24-bit target, or one drawn with its alpha
masked, leaves the page's alpha to the older target. Without this, a
buffer whose alpha is laid down once and read every frame as PSMT8H
through a palette, under a 24-bit depth buffer and effects drawn with
their alpha masked, read zeros once an effect was drawn at the depth
buffer's base: the sky of a game that keeps its clouds in that alpha.
A state taken for another instance to load writes the targets back to
local memory first, so it holds what the GPU drew; a state for this
instance (runahead) keeps them on the GPU as before.
tests/swdraw's alpha_under_depth, _wide and _strip pin the three
layouts on the software renderer, Vulkan and paraLLEl-GS, through a
state save and load; the harness takes 16 kicks.
A colour draw at the base of a depth buffer laid out at another buffer
width gets the depth buffer's contents laid out at its own width, page
for page as local memory holds them, before it draws. A striped double
clear also covers a 32-bit colour with its alpha masked under a 24-bit
Z: both write the same three bytes, so the clear fills the whole page
column in colour and keeps the alpha.
Together these give a bloom pass drawn a page column at a time
the bright pass the GS draws, where it came out as bright bands
a column wide, at every scale.
tests/swdraw/column_passes draws such a pass twice over a depth buffer
ten pages wide and reads back the second row of pages, on the software
renderer, Vulkan and paraLLEl-GS.
A local to host transfer is read out of local memory when the FIFO is
first read, as the other renderers do. A read that takes all of it
gets the pixels written straight into its buffer; one that takes a
part, or a new transfer that comes first, reads it into the FIFO's own
buffer, from which reads are served as before.
tests/swdraw/readback_parts reads a transfer in two parts with a packet
between them, and upload_clut_between has a palette load read the
first half of an upload before the second arrives; both run on Vulkan
and paraLLEl-GS in tests/vknegotiate, and on the software renderer.
On a device with a real texture barrier (GL_ARB_texture_barrier or
NV_texture_barrier), a batch drawn as runs samples its render target
directly, as one draw in groups with a barrier before each: a primitive
starts a group when it reads where the group wrote, writes where it
read, or, blending, writes where it wrote. Nothing is copied. A batch
with a primitive reading what it writes, a second alpha pass, colour
clip or primitive-ID DATE reads a copy of the whole target instead,
refreshed run by run; other devices copy as before.
tests/gsautoflush/runs_groups pins the grouping against drawing the
primitives one by one. ASan/UBSan and TSan clean on the gsautoflush
suite; the feedback cases draw the GS's picture on GL and Vulkan.
Where auto flush would end a batch -- a primitive reading what an
earlier one of the batch drew -- the batch now keeps going, recording a
segment boundary instead (up to 64 a batch). The hardware renderer
draws the whole batch with one texture cache lookup and one state
setup, as a draw per run of primitives between boundaries: before each
run after the first, the copy of the target the draw samples is
refreshed over just the texels that run reads, from what the runs
before it drew. Every backend's RenderHW and CopyRect take it as is;
nothing is backend specific.
The renderer says per batch whether it can (a texture at the frame,
same format and width, colour, unmasked, no hack paths); when it
cannot, or on the software renderer, the batch is handed to Draw one
segment at a time, each with its own vertices and trace, as flushing
at each boundary did. Auto flush's detection looks back over the
current segment only, against what the segments before it drew. A
draw whose setup merged or split the batch's primitives draws each of
its primitives as a run of its own, which still sees every write
before it.
On a batch of 7200 sprites chained across 1500 boundaries, the GS
thread's time on llvmpipe goes from 126 to 86 ms; a batch with no
feedback costs what it did.
tests/swdraw/feedback_runs: four chains, six sprites long, interleaved
into one batch that splits five times. The software renderer's memory
and the OpenGL and Vulkan renderers' picture must hold the seed halved
six times in every chain's last block; drawn without refreshing the
copy between runs, it is black.
Two swdraw cases: three sprites in one batch each reading, at half, the
block the one before wrote a page to its left, the third holding the
seed halved three times; and three sprites over one area each reading
exactly its own pixels, the area halved three times. The software
renderer's GS memory must hold those words (swdraw), and the OpenGL and
Vulkan renderers must draw the same picture, the one those words make
(glnegotiate). Without auto flush, the first case's second and third
blocks come out black on both.
On drivers with no copy_image, CopyRect copies with glBlitFramebuffer,
whose destination rectangle took the source rect's offset on top of the
one it was given, so a copy from a rect not at the origin landed that
far right and down of where it was asked to. The blit now writes at the
destination offset, as the copy_image paths and the other devices do.
tests/gscopyrect/copyrect_dest pins where each path's copy lands.
A primitive that samples the frame nearest, each vertex's texel within a
texel of its pixel and inside any region clamp -- a sprite, or a
triangle writing alpha alone -- reads only the pixels it writes. The
hardware renderer draws a batch of those reading the target in place,
with a barrier between overlapping primitives, which sees every earlier
write; auto flush no longer ends such a batch at an overlap. Every
primitive of a batch is classified, its first included, and one that
reads elsewhere, after an overlap was let through, flushes before it.
A batch an overlap was let through for is drawn in place. The software
renderer, and a device without texture barriers or framebuffer fetch,
flush as before.
tests/gsautoflush/pixel_feedback pins which overlaps stay in a batch.
The vertex kick handlers come in two sets, plain and auto-flush, and
which one a draw kicks through is chosen when a register that decides it
is written -- TEX0 and TEX2, TEST, FRAME and ZBUF of the current context,
PRIM and PRMODE -- and changes only when the answer does: the draw state
reads what it draws. Every other draw kicks through the plain handlers,
which do not ask, where before every kick asked whenever auto flush was
on (by default in the software renderer, and in the hardware renderer
for every game the database turned it on for).
Auto flush is now always on, for every primitive, in both renderers.
The hardware level setting, the software switch, their core options and
the GameDB fix and its entries go.
tests/gsautoflush/autoflush_select pins the choice against the state
after every register write.
A CLUT load skips bringing local memory up to date only where the draw
will read the palette from the target holding it on the GPU: CSM1 with
its CSA in the first sixteen blocks. A CSM2 load, or a 16-bit palette
past block 15, falls back on the palette built from local memory, so
the target holding its blocks is now read back before the load reads
memory.
tests/gsclut/clut_load_sync pins the choice against GSClut::Read32's.
A target dropped by age leaves the lists but keeps its texture. What it
drew goes to local memory, by the same rule as a drop, only when that
memory is read: an EE readback, a CLUT load, a local copy, a source or a
target's dirty area loading from memory, or a primitive drawn on the
CPU. A write over its pages retires them instead, and one left with
nothing local memory lacks goes. Parked targets are the first to go
over the memory budget, and are all written back when the cache is read
back whole.
A target nothing reads again now costs no readback and no wait.
tests/gstc/park_flush pins the rule against the memory each read should
see.
The colour target holding the pages a constant Z writes now gets the
value by a fill on the GPU, from a one-texel target cleared to it, in
place of having those pages uploaded again from local memory; local
memory still gets the Z, and every other target over the pages reloads
them from there. The pages go where the target's own layout puts them,
and pages past its width within its buffer width widen it.
A draw on an existing target's pages past that target's width, within
its buffer width, widens it the same way rather than going into a
target of its own. A widened target keeps the width for the next target
made at its address, so it is sized once.
On the GS a page keeps what was last written to it until something
writes it again, and a game can read it any time later. A target the
cache drops by age or over its memory budget now writes the pages of
what it drew since its last read back to local memory first, except
pages a transfer has written since, pages a target drawn after it still
holds, and pages holding newer pixels: those a later target dropped in
the same pass takes, and those a read has put there, which the cache
stamps per page with the draw they came from. The drops of one pass are
written out together and deleted after.
tests/gstc/drop_writeback runs random histories of draws, transfers and
drops against the memory they should leave.
A sprite with ZTST ALWAYS and one Z writes that Z into memory in the Z
layout. When the whole pages it writes belong to a colour target drawn
after any depth buffer there, and the draw's own colour is on other
pages, the Z is written into local memory and those pages of the colour
target are refreshed from it, with no depth buffer made for the draw.
A buffer one page column wide drawn in turn at two neighbouring pages,
each draw laying the next column's starting value down as Z, now works
as drawn.
GSC_BurnoutGames, which collapsed such a pass into one full-screen draw,
is removed; its GameDB entries keep the sky and cloud handling through
GSC_BlackAndBurnoutSky. tests/gstc/zwrite_colour pins the rule against
the GS's PSMCT32 and PSMZ32 addressing.
FileSystem::FindFiles filled a std::vector of entries holding
std::string names, walking the tree with std::string paths; HostFS.h
pulled <string> and <vector> into the sixteen files that include it,
most of them for the path separator alone.
HostFS.cpp is now HostFS.c, C89, with the same walk and filters:
FileSystem_FindFiles fills a FILESYSTEM_FIND_RESULTS array whose names
it mallocs, FileSystem_FreeFindResults frees them, and KEEP_ARRAY keeps
what is there as before. The header is C with extern "C" for its C++
includers. The IOP host directory listing owns its results and walks
them by index; the BIOS search and the BIOS core option free theirs
when done. An unused path-append template went with the .cpp.
A tree built before this has HostFS.o made from the .cpp and a depfile
naming it, which stops the build ("No rule to make target") or, with
the depfile gone, links the C++ object. The Makefile now drops, before
reading the depfiles, the depfile and object of any C source whose
depfile still names a .cpp, so a rename from C++ to C rebuilds.
tests/hostfs builds HostFS.c as C89 and walks a real folder through
libretro-common: files and folders as the flags select, hidden entries
only when asked, a wildcard, an exact name, full and relative paths,
the recursive walk not entering a hidden folder unless asked, KEEP_ARRAY,
a broken link left out, a missing folder, and the results freed and
reused. The same checks pass against the C++ version it replaces.
With the automatic game fixes off, an entry that sets eeDivRoundMode
and not eeRoundMode was reported by eeRoundMode's name: the mode left
at MaxCount indexed the four-entry name table one past its end, and
the pointer read there was printed as a string. The message for
applying it printed eeRoundMode's number the same way. Both now name
the division mode.
tests/patch's dbload lane applies every entry's fixes with the
automatic fixes on and off, as a VM start does, so a log line that
indexes a table with a mode the entry left unset shows up in the
sanitizer runs; it reported this one under UBSan.
The database was a std::unordered_map<std::string, GameEntry>, each
entry holding std::strings, std::vectors of fixes, hacks, memory card
filters and dynamic patches, and its own std::unordered_map of patch
blocks, with member functions over them and std::optional through the
parser.
Now the file is read into a few pools shared by every entry (text,
game fixes, speed hacks, GS fixes, patch blocks, dynamic patch
entries); while it is read an entry names its runs by offset, and
once the pools are final the entries are made with pointers into them.
Serials are found through an open-addressed index, built as the file
is read so a serial repeated in another case is still refused before
it is parsed. Memory card filters are stored already joined with '/'.
GameDatabase_FindGame/FindPatch/ApplyGameFixes/ApplyGSHardwareFixes
replace the namespace and members; DynamicPatch is a pair of pointer
and count runs; ParseSpeedHackName takes a pointer and a length. The
GS fixes skipped under manual hacks were also collected into a string
nothing read, which is gone.
Checked by dumping every one of the 12807 entries through both the old
and the new code, every field plus what both apply functions do with
the automatic fixes on and off: byte for byte the same, and clean under
ASan, UBSan and LeakSan across an unload and reload. tests/patch's
dbload lane now also loads a small database whose every field is known,
including the eight kinds of line that must be refused, and checks
what each field reads back as, lookups in any case, the default patch
block as fallback only when there is one, and unloading.
GSFunctionMap and GSLocalMemory moved onto the C flat map
(GSFlatMap.c) and GSLocalMemory onto GSPage2Tile.c, but these three
suites build those units by hand and were not given the new sources,
so all three stopped linking. They compile them now; gssearch builds
both as C89 -pedantic with the suite's sanitizer flags, as it does
its own main.c.
The patch store was a std::vector<IniPatch> exported to Counters and
VMManager, each walking it for the places it wanted; a 'bytes'
payload was a std::vector<u8> inside each patch; the dynamic patches
were copies of the game database's vectors; the parser split lines
into std::vector<std::string_view>; the loaders took std::string and
went through std::string buffers, std::optional and FileSystem's
std::vector of results.
Now:
- The patches are a plain array in Patch.cpp, read through
LoadedPatchCount/LoadedPatch, and applied by ApplyLoadedPatches with
a mask of places, in load order as before. A payload is a malloc'd
run with its length, freed by ForgetLoadedPatches, and only
allocated once the line has passed every check.
- The dynamic patches' entries are copied into one array of their
own, a pattern run then a replacement run per patch, so a reload of
the game database cannot leave them dangling.
- The parser splits into five (pointer, length) fields on the stack,
with the stripping and empty-field rules SplitString had, and the
command table takes the same.
- LoadPatchesFromZip and LoadPatchesFromDir take C strings. The zip
member is inflated into a malloc'd buffer. The folder is walked with
retro_opendir directly, with FindFiles's filter (files, hidden ones
included, "*.pnach", sizable) and its order, and each file is read
with filestream_read_file, which terminates it. The "not found"
message no longer has a newline in the middle of the path.
tests/patch gains a lane that links Patch.cpp against the real
libretro-common directory, file and inflate code: pnach files for the
CRC in either case and with suffixes beside another CRC's, a .txt, a
hidden file, a directory named like a pnach and an empty folder; and
archives written by Python's zipfile with the member deflated, stored
under an upper-case name, only under a subfolder, and behind a
comment, plus every truncation and every one-byte corruption of one.
It fails if the CRC match turns case-sensitive.
pcsx2/StringView.h/.cpp held prefix/suffix tests, a lower-casing copy
and the pnach "key = value" split, all on std::string_view. Each had
one or two callers:
- MemoryCardFile's ".bin" checks and pcap_io's device-name prefix are
libretro-common's string_ends_with/string_starts_with, on strings
that are NUL-terminated anyway.
- VMManager::IsElfFileName compares the last four characters to
".elf" without case.
- The game database lower-cases its serial keys with an ASCII loop.
- The pnach reader splits at '=' itself, with the stripping it already
had for the comma-separated fields; that stripping now passes
isspace an unsigned char, as it must for bytes past 0x7f.
The files are gone from the Makefile, CMake and the test builds that
linked them.
Each byte of a 'bytes' payload was parsed as two characters in base
16, and the parse stops at the first non-hex character, so "0g" was
taken as a byte of 0 and the line loaded, where the error branch for
a payload that is not hex was meant to refuse it. Both characters
are now checked.
tests/patch only fed the parser the game database's own lines, which
are all well formed. It now also loads eighteen pnach lines that are
not: spacing around '=' and the commas, a trailing comment, no '=',
a place out of range, signed or not a number, a payload not hex or
not whole bytes, an unknown CPU or operand size, too few fields, and
an address or value past its field. Each must load or be refused as
before, and what loads must carry the place, address, value and
payload written.
pcsx2/ParseNumber.h was a set of templates over std::from_chars,
std::optional and std::string_view. It is now C89: parse_number_u64,
_u32, _u8, _s32, _float and _bool take a pointer and a length, write
*out only on success and return nonzero when they did. The integer
rules are std::from_chars's: no whitespace, '+' or base prefix, a '-'
only for the signed form, letters as digits past 9 in either case,
stop at the first non-digit, and an out-of-range value is no value.
The bool and float forms keep the old word matching and rstrtof_len.
Callers converted: the SettingInfo default/min/max/step helpers,
the pnach line parser (place, address, value, 'bytes' payload) and
the game database's speedHacks, gsHWFixes and patch-CRC fields.
tests/parsenumber builds the header as C89 -pedantic and holds both
the C and C++ builds to std::from_chars on fixed edge cases (range
limits, signs, partial runs, bases 2-36) and 200000 generated runs,
plus the bool and float forms: 1.2M checks. It fails when the
overflow check or the sign handling is broken.
The game-database lane had stopped linking: its stubs for the three
GS lookup functions still had the std::string_view/std::optional
signatures from before GSHwHack.cpp took a pointer and length and
returned s16, so the suite failed on every run.
It also set SANFLAGS to empty before reading SANITIZER, discarding
the flags the CI matrix passes, so the ASan and UBSan jobs ran it
unsanitized. SANFLAGS is now kept and SANITIZER added to it.
With the flags honoured, the TSan build no longer linked:
libretro-common's .c files went through the C++ driver, which gives
retro_procbarrier's __tsan_acquire/__tsan_release declarations C++
linkage. They are built by the C compiler now, as in the core.
swdraw --bios reports the frames its case needs, and glnegotiate runs
both renderers that long (at least 30) rather than 30 for every case,
so what it compares is each case's finished picture: move_width_change,
whose transfer lands late, showed a black frame about one run in twelve
at the 30th. It joins the cases compared, with the CPU sprite lane
taking its case's count too.
A GSString (Common/GSString.h, a C89 growable string kept
NUL-terminated, with its own unit test) is what the OpenGL device
assembles a shader's header, macros and code into, in two kept
buffers; the convert vertex shader's source is one too. The program
cache takes sources as pointer and length, hands the program back
through an out parameter, and keeps its filenames on the stack; the
compute program path, which nothing used, and the pre-link callback,
which nothing passed, go. GLProgram's info logs are malloc'd and
logged; the pcsx2_bad_shader_N.txt file a failed compile wrote into
the working directory is not written any more. The GL debug callback
logs from the message in place.
LoadTextureReplacements is false in every configuration the core
builds, with no option and no game database fix that sets it, so the
replacement loader - a worker thread behind a mutex and a condition
variable, a std::function queue, maps of names to files and textures,
the async-load bookkeeping, the image loaders - never ran. It goes,
with its hooks in the texture cache (the replacement branch of the
hash cache lookup, is_replacement, the replacement memory counter and
InjectHashCacheTexture), the GameChanged plumbing that only reloaded
its map, the three config bits and the textures folder that was made
on disk for nothing.
The on-disk program cache's index is a gs_flat_map from a padding-free
key to the blob's place, and the binary a program is read from or
retrieved into is one byte array kept at the largest, in place of a
vector made per lookup. GetBinary fills that array.
glnegotiate runs a case twice in one directory: the second run must
read the index the first wrote and draw the same picture from the
binaries, which no suite exercised.
The software rasterizer list owns its rasterizers and workers in two
plain arrays, the workers torn down first since they draw on the
rasterizers. The SW renderer's hires read row is a buffer kept at the
widest read; the SW device's MadFields working set is a gs_scratch
kept between frames rather than a vector made and freed per frame, so
GSScratch moves from HW/ to Common/ where both renderers reach it.
A Vulkan texture's linked framebuffers are a GSArray of a small
struct in place of a vector of tuples, and a GL program's uniform
locations a GSArray of GLint.
A target's dirty rectangles are a GSArray behind the same size/index/
erase surface the texture cache walks, with a deep copy for a snapshot's
clone and take() where one target inherits another's. The move remaps
are a GSArray too, and BuildMoveRemap keeps its element records, remap
table, coverage bytes and nibble marks in scratch buffers between
builds instead of allocating and freeing up to w*h records per call.
The CPU sprite renderer's vertices live in a 32-byte-aligned buffer
grown to the largest draw and kept.
glnegotiate gains a lane that draws many_copies with the CPU sprite
renderer on, which no suite exercised: the OpenGL and Vulkan pictures
must match each other and differ from the GPU's.
The GSdx OpenGL renderer had no frontend among the tests: nothing ran it
here, and a change to it went out tried on the Vulkan renderer alone.
tests/glnegotiate is one, as tests/vknegotiate is for Vulkan: an OpenGL
3.3 core context from EGL on a surfaceless display, no window or X
server, so Mesa's llvmpipe serves; the frontend's framebuffer for the
core to draw into (get_current_framebuffer) and its entry points
(get_proc_address), as RetroArch gives them; the frame taken in the
video callback as RETRO_HW_FRAME_BUFFER_VALID. EGL and GL are opened at
run time and declared in the harness, so it builds without their
headers and skips without them.
After the frames the picture is read back, rows top first as the Vulkan
frontend reads an image, hashed the same way (GN_RGB leaves the alpha
out), and dumped on request (GN_DUMP). build.sh boots eleven of
tests/swdraw's cases on it and holds each picture to the Vulkan
renderer's of the same case: on Mesa the two draw them to the pixel
(sprites, points, lines, mipmaps, the depth states, the destination
alpha test, CLUT draws and the dynamic siblings' cases). The 4-bit read
of a target, which Vulkan draws through its GPU view, only has to run.
The suite takes eighteen seconds. The Tests workflow installs EGL and
Mesa's software driver so it runs there.
GSState kept the EE's transfers into local memory since the last frame
(GSUploadQueue) and the runs of non-overlapping sprites of a draw in two
std::vectors, pushed to and read back from the end each transfer and
draw, walked backwards by the texture cache for the target a transfer
may fill, and cut at the front once a frame. They are in a gs_array
(GSArray.h, new, C89): elements of one size in one allocation that
doubles, pushed, erased at an index or from the front, with the type
named at the access. GSState hands the transfers out as a pointer and a
count (DrawTransfers, DrawTransferCount, LastDrawTransfer), and takes
them by PushDrawTransfer, PopDrawTransfer, EraseDrawTransfer and
ClearDrawTransfers; the reverse iterators became index walks, and an
erase in the middle of one goes on with the element before it, as the
iterator arithmetic did. The sprite draw list reaches the device as a
pointer and a count in GSHWDrawConfig (drawlist, drawlist_count) rather
than a pointer to the vector; the Vulkan and OpenGL devices read it so.
The list is reserved for two thousand runs when the state is made, as
it was, so a draw allocates nothing.
tests/gsarray holds the array against a model over random pushes,
erases, cuts and clears, built as C89. tests/swdraw presents the same
pictures on every case, and gstc, gspresent, vknegotiate, gsautoindex,
gsinterlace, savestate and secondload pass as before.
VK_EXT_extended_dynamic_state3 set the blend enable, equation and colour
write mask on the command buffer, and left them out of the TFX keys. On
NVIDIA's driver (an RTX 5090) the draws stopped reaching the targets:
Devil May Cry 3 black but for fragments from a state load, Soulcalibur
III a picture frozen as a fight began while the game ran on. It held with
the pipeline libraries on or off, with the state set before or after the
pipeline's bind as the Khronos sample does, and with a dual-source blend
declared in the pipeline's static state; the same build with the
extension left off (LRPS2_TEST_VK_NO_EDS3) drew both games. Lavapipe,
where every lane had passed, draws correctly either way.
The blend goes back into the pipeline and its key, and the extension is
not asked for. The libraries alone carry what the dynamic blend was for:
a new blend combination links its sibling from a state library in a
fraction of a build, nothing on the render thread.
A library's key has no blend, and BuildTFXPipeline read that as colours
masked: the pixel shader part of a sibling was built without its second
output, while the state part it is linked with may blend through it
(SRC1). The keys carry whether the output is used (TFXPipelineKey,
VKDynamicPS), and BuildTFXPipeline reads it from them alone.
tests/vkpipewarm fails if any run enables the extension, and runs a new
tests/swdraw case, dual_blend, a sprite blended (Cs - Cd) * As + Cd with
an alpha none of its channels equal, through the linked siblings;
tests/swdraw draws it on the software renderer too.
The rasterizer threads each draw their own bands of rows. When a draw's
frame or depth buffer wraps local memory, or the two overlap and one of
them is written, rows a band apart land on one address. Threads then
write those addresses, or read what another writes, in no set order. The
console draws in order, so the rows drawn later win. Under ThreadSanitizer
tests/swdraw's dirty_whole_target reported 16 races in DrawRect on four
rasterizers; its colour is flat, so the picture hid them.
GSRendererSW now marks such a draw from the pages the rect spans
(DrawsOverItself), and GSRasterizerList draws it on the GS thread once
the workers have drawn everything before it, every row in order, with
the first rasterizer's bands set to all of them (DrawWhole). The draws
before and after keep their order on every row. Ridge Racer V's intro
has no such draw.
tests/swdraw's wrap_gradient draws a 640-wide frame 2047 rows high, a
colour ramp, so the bottom rows land on the top rows the display shows.
The swdraw lane holds its picture on four rasterizers against the one
on one. Before, it came out different on every run.
GSLocalMemory kept its GSPixelOffset4s and GSPage2TileMaps in two
std::unordered_maps from a u64. Each is made once and kept until the
local memory goes, so they become GSFlatMaps from the u64 to the
pointer, looked up with gs_key_hash.
A GSPage2TileMap was built through a std::unordered_map of
std::unordered_sets, a std::unordered_map per page and a std::sort, the
first time a texture shape is used. The block walk now writes one key a
block (page, and y:x as before) into a single array. GSPage2Tile.c, new
and C89, radix sorts the keys in three passes of nine bits, then fills
each page's rows and inverted masks as the runs come. The entries and
their order are the same as before. Building a map takes 10 us instead
of 75 us for 1024 blocks, 124 us instead of 760 us for 16384, and
0.8 ms instead of 4.8 ms for 65536: a hitch the first time a large
texture shape is drawn.
GSFunctionMap.h now includes <unordered_map> itself; it had been getting
it through GSLocalMemory.h.
tests/gspage2tile holds the builder against the old containers, kept as
the model, for texture-shaped and random key sets. Ridge Racer V's
intro presents the same 19 frames on the Vulkan renderer, bit for bit.
The texture cache's hash cache was a std::unordered_map, with nodes
from GSNodeAllocator's pool. It becomes a GSStableMap, new in C89. Sources
point at their entries, so an entry can't move the way a GSFlatMap's
values do. Here the slots are a stored hash and a node pointer, sixteen
bytes each, at most a quarter in use, with linear probing. Each node
holds its key next to its value, so a hit reads the slot and the node.
A node knows its slot and its place in a dense list of nodes, so an
erase neither looks it up nor hashes it again (backward shift for the
slots, the last node into its place in the list). A walk goes down the
list and can erase the entry it is at. AgeHashCache does that instead of
erasing through an iterator. Its purge list is a plain array of entries,
grown only when the cache is fuller than it has been. The slots for 800
entries are made when the cache is.
tests/gsstablemap checks the map against a model, with nodes from a
pool and from the allocator, including erases during a walk. With
GSSM_BENCH=1 it times both maps on 40-byte keys at 32 to 800 entries,
best of five, ns per operation, std then stable:
find hit 9-13 vs 6-9 (1.3-1.6x)
find miss 9-11 vs 7-8 (1.1-1.5x)
walk 1.1-3.8 vs 0.6-1.3 per entry (1.2-2.9x)
insert+erase 20-29 vs 15-20 (1.3-1.5x)
Within a few percent of the quarter-load boundary the miss and insert
times are even. Ridge Racer V's intro presents the same 19 frames, bit
for bit, and the steady-state allocation count stays at zero.
The hash cache expands a paletted texture to 32 bits, once for each
palette it is drawn with: four times the indices for an 8-bit texture,
sixteen for a 4-bit one, every time. Ridge Racer V's intro has 27
textures, 7 MB of indices, as 370 MB of expansions. Sources keep them,
so they cannot be purged, and at 1x and 2x they filled the Vulkan heap:
textures were refused and drawn without, a building gone and a face
with only its hair.
The second palette a texture is seen with now makes it an index texture,
shared by every palette and read through them on the GPU, as GPU
palette conversion does for every texture already. A texture drawn with
only one palette stays expanded, as before. Which palette each was
first seen with is kept in a GSFlatMap of 1024 entries, reserved when
the cache is made (gs_flat_map_reserve, new), so nothing is allocated
mid-game; full, it is forgotten at once. Devices that need point samplers
for index textures get them for these too.
Ridge Racer V's intro at 1x and 2x: no texture refused (thousands
before); over 400 frames the pictures are the expanded ones' but for a
few hundred pixels of bilinear rounding, as close to the software
renderer's.
LRPS2_TEST_NO_AUTO_INDEX expands every palette, as before. tests/swdraw's
t4_many_palettes draws a 1024x512 4-bit texture through 48 palettes in a
frame; tests/gsautoindex holds its picture against every palette
expanded, and with a 300 MB heap it refuses nothing and draws the same,
where with every palette expanded it refuses 301 textures (as before
this change) and draws sprites without them. tests/gsflatmap checks the
reserve.
A dynamic sibling reads its selector from push constants. The key was
pushed when the layout changed or the key did, but only by a sibling's
draw: a draw with a pipeline of its own in between took the layout
change and the dirty flag with it, so a sibling drawn after it, with the
same key it pushed before, read whatever was pushed since - a utility
draw's constants, such as the palette conversion's (UpdateCLUTTexture)
or the merge's background colour.
Siblings draw while the warm-up thread builds a key's own pipeline, so
this hit new keys, in the frames after a state was loaded or a scene
first drawn. In Ridge Racer V's intro, text glows drawn with the wrong
selector, and with paletted textures read on the GPU whole backgrounds
went missing; a state saved there and loaded again drew right, its
pipelines built by then.
Whatever asked for the key to be pushed now waits for the next sibling's
draw.
LRPS2_TEST_VK_PIPELINES=mixed draws every other key, in the order first
drawn, with its sibling, the rest with pipelines of their own. tests/
swdraw's sibling_key_after_clut draws a stretched sprite, a 4-bit sprite
whose palette is converted from a frame on the GPU, and the stretched
sprite again, every frame; tests/vkpipewarm holds its mixed picture, and
the other cases' with more than one key, against every pipeline its own
(=sync). Before, the second sprite came out a flat colour.
Interlace drew into its weave/bob, blend and MAD targets whether or not
ResizeRenderTarget could make them. On a heap that refuses (the Vulkan
device's has a ceiling) that read a null texture's size and ended the
core: Ridge Racer V at 2x did, once its paletted textures had filled the
heap.
With no merge there is nothing to deinterlace and the frame is skipped,
as with no merge before. A deinterlacer target that cannot be made shows
the merge as it is, for as long as it cannot; the MAD mode's second
target is made with its first, before either is drawn into.
LRPS2_TEST_VK_HEAP_CEILING sets the Vulkan heap's ceiling in bytes, for
tests of what refuses. tests/gspresent runs the interlaced fades at 8x
under each deinterlace mode with a 420 MB heap: every run refuses
targets and runs on. Before, every one ended in a segfault.
LibretroAdmin