A draw whose pipeline key was new built that pipeline on the render
thread before it drew. On lavapipe that is a 20-90 ms frame per new
key, and on real drivers it is the hitch every first meeting with a
shader costs.
Each TFX fragment module now has a dynamic sibling. It is the same
tfx.glsl compiled with DYNAMIC: the PS_ fields come from a 16-byte key
in push constants instead of specialization constants.
tools/vk_spirv.py packs the key:
- each field gets its selector's width, and none crosses a word;
- the sibling's decode is written into its source;
- the field positions go out as a table that VKTFXDynamicKey packs the
key by.
The sibling pipeline of a key keeps the state the pipeline is built from
and the module axes (VKDynamicPS), so one sibling serves every key of
one fixed state.
On a miss, GetTFXPipelineMissed:
- takes what the warm-up thread has finished;
- if the key's sibling is there, draws with it, pushing the key from
ApplyTFXState, and asks the thread for the key's own pipeline;
- otherwise builds the key's pipeline on the render thread as before,
and asks the thread for the sibling, so the next new key of that
state need not wait.
Draws keep going through the lookup until the key's pipeline lands; it
is then taken (also once a frame) and listed for the next device. Only
pipelines of their own are kept as the last pipeline.
The warm-up thread now stays for the device's life:
- it builds the utility pipelines, then the listed keys, then their
siblings, so a new key of a state the last session knew draws at
once;
- it serves what the render thread asks between items of that, ahead
of them;
- it then sleeps on an eventcount until asked. There are no locks:
jobs go to it in an SPSC ring, results come back in the other.
- At most 64 jobs are out at once.
- The result ring has room for all of them.
- Every job is answered, built or not, so a key never waits forever.
Cost: three dynamic fragment modules, 268 KB of SPIR-V; still far less
than glslang was.
Tests, on lavapipe:
- tests/vkpipewarm adds, for each case, from nothing:
- LRPS2_TEST_VK_PIPELINES=sibling: every draw with a sibling;
- =async: every new key draws with its sibling while the thread
builds its own.
Both present the same picture as the specialized pipelines. The
async run ends in the same state, and its pipelines come from the
thread while siblings drew.
- All 74 tests/swdraw cases at 1x and 2x present master's picture in
the default, sibling and async modes.
- tests/vkpipekey (CI) also checks the sibling selector and the key:
- the sibling is made the same module as its key;
- its packed key unpacks, by the generated table, to the selector's
fields;
- no field holds more than the key's bits for it.
Each of these fails it:
- dropping a field VKDynamicPS keeps;
- shifting the packing by a bit;
- giving tfx.glsl fewer bits for DATE than the selector has.
- tests/vkspirv covers the dynamic modules' patching too.
- ASan/UBSan, and TSan with the renderer, retro_spsc, the eventcount
and rthreads instrumented, are clean over tests/vkpipewarm (all
lanes) and tests/vknegotiate. Driver and MTVU races in
uninstrumented code were suppressed for this run only.
The colour varying, the target read and the outputs are module axes as
before, and the key keeps them, so the sibling never reads what the
pipeline's render pass lacks.
The Vulkan renderer wrote GLSL at run time and compiled it with glslang:
for each TFX selector it assembled a source string of ~60 #defines in
front of tfx.glsl, hashed it, looked it up in an on-disk SPIR-V cache and
compiled what was missing; the utility shaders likewise. glslang was
about 1.6 MB of the core's code, and the warm-up thread could not build a
TFX pipeline whose SPIR-V a session before had not cached.
Now the GLSL lives in pcsx2/GS/Renderers/Vulkan/shaders/ and is compiled
ahead of time by tools/vk_spirv.py into VKShaders.inc, which is
committed (as parallel-dc, swanstation and parallel-n64 do theirs). The
core makes its modules straight from those words.
- Every draw selector field is a specialization constant of tfx.glsl. A
pipeline passes the selector to the driver in VkSpecializationInfo:
no source text, macros, string hashing or compiler. The generator
writes the constant IDs out as an enum (VKSC_*), and a static_assert
catches a constant with no value in the C++.
- What changes a shader's interface is settled in the SPIR-V. There are
two vertex modules (with and without VS_EXPAND) and three fragment
modules (the target not read, read as an input attachment, read as a
sampled image). Each fragment module carries the dual-source output,
gl_PrimitiveID and gl_FragDepth. For a draw that does without one,
VKSpirvPatch.c turns it into a private variable in the words: a draw
that does not clamp depth gets no DepthReplacing (so early depth
testing stays), one without the alpha test passes needs no geometry
capability, and so on. It also drops the colour varying's Flat
decoration for an interpolating draw. That keeps the TFX SPIR-V at
five modules (224 KB) instead of one per combination; all 24 would
be 1.7 MB.
- The utility shaders are one module per entry point.
- Modules are made once per thread as each is first asked for. The
warm-up thread has its own set and builds every listed pipeline; it no
longer leaves TFX ones it has no SPIR-V for to the render thread.
- VKShaderCache is gone. The driver's pipeline cache stays and is
handed the file as read; the stale vulkan_shaders.idx/.bin are
deleted. The pipeline list is tied to the shaders (VKSPV_VERSION),
not SHADER_CACHE_VERSION.
- 3rdparty/glslang is removed from the tree and from both builds.
On lavapipe, all 74 tests/swdraw cases at 1x and 2x present the same
picture as before, with both the CMake and Makefile cores. The text
segment is 1.9 MB smaller, and a cold start (no caches) is about 150 ms
faster.
One behaviour is kept on purpose. VS_PROVOKING_VERTEX_LAST was always
defined (as 0 or 1), so its #ifdef always took the provoking-vertex-last
line end, and the shader still does.
Tests:
- tests/vkspirv (new, in CI):
- tools/vk_spirv.py check fails if a shader changed without the .inc
being regenerated.
- The harness makes every module the renderer can make (three
fragment modules x 16 interfaces, both vertex modules) and reads
each back as SPIR-V. Dropped variables must be private, of the
right pointer type, undecorated and out of the interface; kept
ones unchanged. The geometry capability and depth replacing must
go exactly with their variables. Malformed modules are refused.
- A patcher that leaves the capability, the execution mode, the
decoration or the interface entry, or writes the pointer type
wrong, fails it.
- tests/vknegotiate: the corrupt-index lane is now a pipeline cache with
junk after its header plus an old SPIR-V cache beside it. The run
must go on, and the old cache must be gone; on master it is
rewritten.
- tests/vkpipewarm: the driver change now also stamps the pipeline
cache with another device's UUID, and the cache must be refused.
Upstream already carries equivalents of two of this PR's three commits:
8c81f41 R5900: use the portable FP control register API
-> 1dbdc62 (also covers the same _mm_setcsr use in VUmicro.cpp)
543eed9 cmake: generate the game database without needing sh
-> 8254b70
Conflicts in pcsx2/CMakeLists.txt and tools/gen_gamedb_builtin.cmake are
resolved by taking the upstream versions. What remains from the PR after
this merge is a81d2f5 (Android arm64-v8a/x86_64 CI jobs) and the
keep-in-sync note added to tools/gen_gamedb_builtin.sh.
The gamedb custom command invoked tools/gen_gamedb_builtin.sh via sh,
which does not exist on the MSVC Windows runner (libretro-build-windows-x64,
MSB8066 / exit code 9009). Add tools/gen_gamedb_builtin.cmake, a cmake -P
twin of the shell script (file(READ HEX) + regex, no {n} quantifier or
string(REPEAT) so CMake 3.12 still works), and point the custom command at
it. Output is byte-identical to the .sh generator. The static Makefile rule
still uses the shell script and is unchanged.
libretro-build-windows-x64 fails in the GameDatabaseBuiltin.cpp step:
Generating GameDatabaseBuiltin.cpp from GameIndex.yaml
'sh' is not recognized as an internal or external command
error MSB8066: Custom build ... exited with code 9009
The custom command shells out to tools/gen_gamedb_builtin.sh, and the MSVC
runner has no sh (nor od or awk). Drive the same generation through CMake's
own script mode instead, so the only dependency is the CMake already running
the build.
Verified against the shell version on the shipped GameIndex.yaml: both emit
1819912 bytes, matching the input exactly, and the two byte sequences are
identical. The generated file also passes a compile check.
The .sh stays for the static-Makefile rule, which is POSIX-only anyway; both
now carry a note to keep them in sync.
One CMake wrinkle worth recording: its regex flavour has no {n} repetition,
so the 20-bytes-per-row split is built by repeating the pattern in a loop.
Written the obvious way it silently does nothing and emits the whole array
as a single 9MB line.
Replaces the abandoned two-patch shape whose first patch weighed
11.6MB: editing the checked-in 9.2MB hex-array builtin re-flows
every line after the first change, so the representation itself was
the defect. This single commit removes the seven Superman Returns
VUOverflowHack entries from the shipped yaml (each gameFixes list
held only the hack, so the emptied key goes with the item), deletes
the checked-in generated artifact, and generates it at build time
instead: a POSIX-shell generator (od and awk, zero new
dependencies) proven byte-identical to the previous checked-in
format before the switch, a Makefile rule keyed on the yaml and the
generator, and a gitignore entry. Future database edits diff as
exactly the yaml lines they touch. The one-time cost is this
commit's own size -- deleting a tracked nine-megabyte file
necessarily carries its content -- paid once so it is never paid
again. Clean-tree build fires the generator, the embedded copy
carries no VUOverflow bytes, boot sanity clean; the parser's
tolerance for the retired name stays for third-party databases.
Standalone corpus that drives the real emitter over a broad set of
instruction shapes (ALU reg-reg/reg-imm all sizes + extended regs, every
memory addressing form, SSE with prefixes) and dumps emitted bytes, with
a checked-in golden reference. Serves as the merge gate for the emitter
cursor-passing refactor: behaviour-preserving emitter changes must
reproduce golden.txt byte-for-byte.
GsDump file can be optional in case you just want to launch the plugin config dialog.
Throw with a meaningful error message in case validation layer fails.
- Add simple logging infrastructure and further improve the existing one
- Delegate loading of dlls and dumps into dedicated loader classes
- Adjust user interaction to restrict updates to only relevant parts
- Gsdx dlls (dumps) are only reloaded when a new path for gsdx dlls (dumps) was given. That means dlls are not reloaded just because the dumps path was changed or another dump was selected
- When GSDumpGui can not find the specified settings folder or there is no setting saved so far it will look into the directory of the currently running application (however as before only exactly in the directory and not in subdirectories)
- Further decouple model, view and 'controller' logic.
- Restrict directory checks to application basepath instead of current directory as current directly is changed at too man places
- Avoid some minor memory leaks by disposing some disposable elements
Note: Net framework requirement has been increased to 4.0 to run/compile the application.
- automatically find the next free TCP port yielded by OS
- transfering currently used instance-fixed port to launched clients
- connect each client via given port
See #1637
libretroadmin