With no naomi.zip (naomi2.zip) a NAOMI game stopped at "cannot load
BIOS". A flat image now starts all the same: reios does for it what the
BIOS does. The BIOS is used as before when it is there, and a romset
still needs one.
What that turned out to be - found by letting the real BIOSes
(epr-21576h, and the NAOMI 2's epr-23605c) run up to a game's first
instruction and looking at what they leave, by reading the BIOS's code
for the parts games call, and by tracing Dead or Alive 2 and Virtua
Fighter 4 with the BIOS and without until the two runs went the same
way:
The start. The image's header says what to load where (up to eight
pieces at 0x360) and where the game begins (0x420). The BIOS hands over
with the display in its 640x480 mode, a copy of the header at 0c01f400,
the clock's time at 0c01f130, and registers pointing at both.
The system routines. The BIOS leaves a table of 24 routines at 0c018000
that games call for what every cabinet does alike: the operator's
settings and the check of them, the 28 coin settings, coins into
credits (bonus coins, service credits, the most credits, the meters,
the queue of sounds), what each player can do with the credits there
are, the board's region and the game's serial. They are written here
from what the BIOS's do; a routine's place in the table holds the trap
reios answers. The ones for the GD-ROM's DIMM board and the network
board answer 0 and say so once: no cartridge game seen calls them.
There being no test menu, the settings are the BIOS's defaults: one
coin, one credit.
The interrupt service. A game never takes the processor's vectors: the
vector base stays at the BIOS's 0c000000, and the game writes its
handlers into the BIOS's table at 0c0001c0. The vectors are traps here:
one puts the interrupted program's registers by, in the machine's own
memory so that a saved state has them, and sends the processor to the
handler; the handler returns to the other, which puts them back.
Two things a game looks at in the boot ROM itself:
- Before it will use the routines, a game compares 112 bytes at 1ffd00
with a copy it carries - the BIOS's "COPYRIGHT (C)SEGA ENTERPRISES"
lines and "NAOMI BOOT ROM" - and otherwise goes a way that never
ends. Those bytes are put there. (Both games carry the same copy.)
- Dead or Alive 2's start-up code waits for a word that it reads
through a pointer it has not yet set, which makes it the boot ROM's
word at fc0. It is given the value the BIOS has there.
naomi_cart: bios_loaded is set when a flat image's BIOS was found; it
was only set for romsets, and nothing had asked until now.
Tested with no BIOS in the system directory (RetroArch nightly, glcore,
and the headless harness): Dead or Alive 2, Dead or Alive 2 Millennium
(from its RAR archive) and Virtua Fighter 4 (NAOMI 2) reach their
attract modes; a coin in Virtua Fighter 4 gives CREDIT(S) 1 and PRESS
START BUTTON, as it does with the BIOS; a reset starts the game again;
a state saved at frame 1400 and loaded at 1800 leaves the machine, 800
frames on, where 2200 frames straight leave it but for 270 bytes of
92 MB - and that state was taken inside an interrupt handler. With the
BIOS all three run as before, and the Dreamcast test disc passes.
Not done: romsets (the cartridge would have to be read through its
decryption), the test menu, and so any setting but the defaults. Of the
coin routines only the paths the defaults take have been run: bonus
coins, a chute to each player and service credits are as the BIOS's
code has them, and not exercised.
A flat NAOMI image whose header says "Naomi2" is run as a NAOMI 2, with
the BIOS from naomi2.zip (or .7z). Until now every such image was taken
for a NAOMI 1, and the BIOS answered "this game is not acceptable by
main board" - which is what upstream flycast does with one as well.
What a NAOMI 2 has that a NAOMI does not:
- The geometry processor of the last patch, in area 2: its registers
at 08800000, its command port at 09000000, its 32 MB at 0a000000,
and at 085f6800 and 085f8000 the holly's and the PowerVR's registers
written to both PowerVRs at once (core/hw/pvr/elan_host.cpp). What
it sends goes into the tile accelerator the way a store queue write
does.
- A second PowerVR. The two are one here, as in upstream: one set of
registers, one video memory. What the second has of its own is its
normal interrupt status, at 025f6900: the end of a list and the end
of a render are raised in both, and the game clears each.
- The second PowerVR's video memory is the mirror 32 MB above the
first's, and games write every texture there too. The texture
cache did not watch the mirror ("the mirrors don't seem to be
used"), so a texture changed through it kept its old picture. On a
NAOMI 2 it is watched.
- Frames several times the size of anything a NAOMI sends: the tile
accelerator's buffer and the lists a frame is parsed into are four
times as big on a NAOMI 2, and a buffer kept from another machine
is not reused.
Save states: version V20 adds, on a NAOMI 2 only, the second interrupt
status, the chip's state and memory, and the time its texture transfer
ends. Older states load as before.
"Is this a NAOMI" is asked in a dozen places: SYSTEM_IS_NAOMI() answers
for both.
Tested with a flat image of Virtua Fighter 4 and naomi2.7z, in the
RetroArch nightly of 7 October under Xvfb (Mesa llvmpipe):
- The BIOS starts the game; the attract mode's temple scene and the
demonstration fight are drawn - letterbox, coin text, the stage, the
fighters and their shadows - by the OpenGL and the Vulkan renderer,
per triangle and per pixel, and look alike in all four.
- Against upstream flycast (which lights in vertex shaders), made to
run the same image as a NAOMI 2 by a change of three lines: the same
scenes, the same picture to the eye. The two do not reach a scene on
the same frame, so they were not compared pixel by pixel.
- What the chip sends the tile accelerator over 1500 frames is the
same, byte for byte, from run to run.
- Threaded rendering on and off: 138 pixels of a 1440x1080 frame
differ, where two runs with the same setting differ in 54. Those
are Mesa's: a texture with mirrored repeat comes out differently
from run to run on llvmpipe, and identically with plain repeat.
- A state saved, the game run 50 frames on, the state loaded and the
game run on: 100 frames later its state differs from a straight
run's in no more bytes than two straight runs differ from each
other (about 900 of 92 MB).
- tools/threads/headless.sh, the Dreamcast test disc: passes.
Not done here: the chip's memory is not in the recompiler's direct
mapping, so its accesses go by the handlers; NAOMI 2 games in MAME's
zip form are not in the game table; starting without a BIOS.
The option chose between two ways of treating what a game renders to
a texture: copy it to video memory after every such render, waiting
for the graphics card each time and losing Render To Texture
Upscaling, or leave it on the card and have the games that look for
it in video memory come out wrong. A table switched the first on for
the games known to need it.
Since the last three patches there is one way, and it is right for
both: the picture stays on the card and video memory gets it when it
is first touched there. Nothing is left for the option to choose.
- The core option, in every language, and settings.rend
.RenderToTextureBuffer.
- The copy-after-every-render paths of all four renderers, with what
only they used: the Vulkan drawers' second render target and its
render pass variant, the check for the setting having changed, and
WriteTextureToVRam() (rend/rtt_watch.cpp packs pixels now).
- The table's column, in the console's table and the arcade one, and
the 51 rows that had nothing else in them.
Render To Texture Upscaling stays, and now applies to every game.
A frontend that still has the old option saved ignores it.
The live test passes on gl, glcore and vulkan, the headless test
passes.
On the console what a game renders to a texture is in video memory
afterwards. Here it stays on the graphics card unless "Enable RTT
(Render To Texture) Buffer" is on, which it is not by default; games
that then go on to read that memory, write into it, or use it as a
texture of another shape come out wrong, and a table switches the
copy on for the ones known to.
- The table had 14 games. Upstream's list has 48 product numbers:
the 37 missing here are added - Jet Grind Radio, Shenmue, Grandia
II, Sakura Taisen 1 and 3, Silent Scope, Worms World Party and
others. Two of the 14 were written without the region suffix the
disc has (the European Tony Hawk's Skateboarding and Pro Skater 2)
and matched nothing; the added rows have them as the disc does.
- What the table switched on was switched off again the next time
any core option was changed, because the option's value alone was
written to the setting each time. The table's word now holds.
Builds; the headless test passes. None of these games is here: the
rows are upstream's, not tested.
The 32-bit ARM recompiler knew nothing of the SH4's MMU, so a game
that needs it - the Windows CE ones - could not use it; the commit
before this one gave those games the interpreter. They now get the
recompiler, and that fallback is taken out again.
A block compiled with the MMU on differs in three ways, as it does on
the other recompilers:
- Every access to memory is a call that translates the address. Where
translation fails, the SH4's exception is taken there - registers
saved, PC at the handler - and longjmp() returns to the main loop,
which goes on from the handler. Instructions the interpreter runs
and store queue flushes, which can raise exceptions too, go through
the same. A constant address is still read or written directly when
it is one the MMU does not translate.
- It ends by handing its next PC to the main loop, which looks the
block up: which block an address means depends on the mapping in
force, so blocks are not chained to each other.
- It begins by checking that the PC is its own, and that the FPU is
on if it uses it.
The main loop (ngen_arm.S) has the two entry points again for such
blocks, and a setjmp() where it starts, once a frame.
There is no access through the host's own mapping, as the 64-bit
recompilers have: a 32-bit host has no room to lay a translated
address space out. So with the MMU on every load and store is a call;
slower than without, a great deal faster than the interpreter.
Run headless on a 32-bit ARM core under qemu-arm, the live test's disc
passes with the MMU on - its MMU test, then everything else through
the MMU, including the TLB misses and the illegal instruction that
come back through longjmp() - under both SH4 timings; and without the
MMU as before. The x86-64 headless test passes as before. Built for
a device target and with NO_MMU defined.
Not run with a Windows CE game, and the path for an FPU instruction
with the FPU off is not reached by the disc.
The 32-bit ARM recompiler knows nothing of the MMU. With it on, the
recompiler went on reading and writing at the addresses the program
gave as though they were physical ones, so a Windows CE game, which
lives behind the MMU, could not run on a 32-bit ARM build at all.
Where the core turns its MMU on for a game - a Windows CE disc, or
the "Force Windows CE Mode" option - a 32-bit ARM build now runs the
SH4 with the interpreter, which does the MMU properly, and says so in
the log. That will be slow on such a machine, but it is the game and
not a crash. x86-64 and ARM64, whose recompilers have the MMU, are as
they were.
Seen with the disc of the commit before, run headless on a 32-bit ARM
core under qemu-arm: the MMU run failed with the address not
translated, and passes with this.
"elf" has been among the extensions the core says it takes, and the
HLE BIOS has had a loader for ELF files, but nothing connected them:
the setting that names the file was cleared at start-up and never set,
and an .elf was handed to the disc reader like any other content.
Now it works. Content ending in .elf is not taken for a disc. The HLE
BIOS is used for it whatever the HLE BIOS option says - it is what
loads the file, and the real BIOS would only find an empty drive - and
the drive is left empty, where before an empty drive made the core
switch the HLE BIOS off. The program is loaded into main memory and
started where the file says it starts, which is 0x8C010000 for
anything built the usual way, on every reset.
The loader no longer takes the file's word for where things are.
Before acting on them it checks that the file is long enough to have a
header, is a 32-bit ELF, and has its program header table inside it;
that each segment's bytes lie inside the file; and that where they go,
with the zeroes after them, lies inside main memory. Only segments
marked as loaded are loaded - a stack marker at address 0 used to draw
a warning. A file that fails any of this is refused, and the CPU is
then left idling at the start address, as it is for a missing disc,
and not run into whatever memory holds.
Checked with the live test's program built as an ELF by the compiler
and written as one by hand: both start and draw their scene. Five
damaged files - cut off inside the header, the header table outside
the file, a segment outside the file, one too long for memory, one
placed at the end of memory - are each refused and leave the core up;
before the checks the loader would have copied from or to wherever
they pointed.
The linker was asked which functions of the core no code path reaches
(-ffunction-sections with --gc-sections --print-gc-sections), and each
one it named was then looked up by name across the whole source,
every platform's code and the tools included. These have no caller
anywhere:
MemWrite32 hw/pvr/pvr_mem.cpp
pvr_ReadReg hw/pvr/pvr_regs.cpp
rend_framePending hw/pvr/ta_ctx.cpp
rend_init, rend_resize hw/pvr/Renderer_if.cpp
emit_WriteCodeCache hw/sh4/dyna/driver.cpp
bm_WriteBlockMap hw/sh4/dyna/blockmanager.cpp
dec_Un_rNrN, dec_Un_frNfrM, dec_rz hw/sh4/sh4_opcode_list.cpp
cpu_iWarn hw/sh4/interpr/sh4_opcodes.cpp
get_nvmem_data hw/holly/sb_mem.cpp
aica_Term hw/aica/aica_if.cpp
reios_term reios/reios.cpp
cryptoReset hw/naomi/decrypt.cpp
printtoc imgread/common.cpp
dc_start nullDC.cpp
shutdown_thread_pool rend/TexCache.cpp
gl_DebugOutput rend/gl4/gles.cpp
libPvr_GetRenderTarget,
libPvr_GetRenderSurface libretro/libretro.cpp
VArray2::LockRegion libretro/common.cpp
LogManager::SetEnable log/LogManagerLibretro.cpp
set_user_data_dir,
add_system_config_dir,
add_system_data_dir stdclass.cpp, with the three
variables only they wrote
The two *_Term functions were empty. cryptoReset was not: but the
cartridge decryption it would have cleared starts itself afresh
whenever a key or an address is set.
Left alone, though the linker named them on x86-64: what the ARM,
ARM64 and x86 recompilers and the cached interpreter call (rdv_*,
emit_Write32, emit_SetBaseAddr, bm_GetStaleBlock, the block reference
counting, _vmem_ReadMem8SX32/16SX32, shil_opcode_name), cpu_iNimp,
which a build without an MMU calls, mmu_QACR_SQ, which is called but
inlined so that only its separate copy goes unused, and
archive_num_entries, which tools/archive uses.
libARM_Term, also empty and never called, stays for now: its file has
CRLF line endings, and a patch that touches it does not go through a
plain git am.
A number of settings are given a value once, in LoadSettings(), and
never another: there is no core option behind them and nothing else
writes them. Each is replaced by its value where it was read, and the
branch that value never took is removed.
- dynarec.idleskip, always on: the idle loop detection in the block
decoder is unconditional, and the branch for it being off is gone.
- dynarec.unstable_opt, always off: the store queue shortcut in the
ARM recompiler that it guarded is removed.
- pvr.MaxThreads, always 3: a constant in the texture upscaler.
- pvr.subdivide_transp, always off, and only read inside an #if 0
block of the triangle sorter, which goes with it.
- rend.Clipping, always on: tile clipping no longer asks.
- rend.AutoExtraDepthScale, always on.
- debug.SerialConsole, always off: the serial port's echo to stdout.
- imgread.PatchRegion, never set: the two functions that would have
rewritten a disc's region strings returned at once, and are removed
with the calls to them.
- validate.OpenGlChecks and pvr.Emulation.zMin/zMax were written and
never read.
LoadSettings() also loses a block for builds that are not libretro,
which this core is not, and which set a field that no longer exists.
core/rend/soft/softrend.cpp, a software renderer that used one of
these settings, was not in any makefile. It is deleted, and its
declaration with it.
Neither did anything. Both settings were written from their options
and read nowhere: what they chose between went with the old threaded
renderer, which could run ahead of the picture and had to either wait
for it or drop a frame. Threaded rendering now makes one frame for
each retro_run, the frame non-threaded rendering makes, and presents
it at vblank. The core had already been hiding both options for that
reason.
The options and their settings are removed.
settings.aica.LimitFPS was set to zero and never read.
settings.aica.NoSound was set to zero and read in one place, where a
non-zero value would have thrown every sample away. Sound is always
written.
settings.aica.CDDAMute was never set, so CD audio was always mixed in
and passed to the DSP. The branch for it being muted is gone.
That leaves settings.aica empty, and it is removed.
settings.pvr.Video (ResolutionMode, VSync) and settings.pvr.OSD
(ShowFPS, ShowStats) were declared and never mentioned again.
Each exit of dc_init() tears down the stages it completed, and
plugins_Init() terminates the plugins it brought up before the one that
failed. A content load that fails no longer leaves the Win64 vectored
exception handler, the dynarec unwind table or the reserved address
space registered after the core is unloaded.
The option chose between two mixers. With it off, sound was mixed 32
samples at a time by a version that leaves the sound chip's DSP out:
no reverb, no chorus, none of the effects games route their channels
through, and a channel sent only to the DSP was dropped into the
output as it was instead. With it on, sound is mixed a sample at a
time with the DSP run for each.
The option, its setting and the batching setting that went with it are
removed, and so is the mixer without the DSP. Low-end builds, which
had it off by default, now run the DSP like every other.
Fog and modifier volumes are not enhancements: they are what the
PowerVR2 does, and a game that uses them looks wrong without them, with
no fog where there should be fog and no shadows where there should be
shadows. Both options defaulted to on and there is no reason to offer
"off".
The two options go, and settings.rend.Fog and
settings.rend.ModifierVolumes with them. The OpenGL, OpenGL per-pixel
and Vulkan renderers (both Vulkan drawers) always use the fog mode the
polygon asks for, always keep the fog table up to date, and always draw
the opaque and translucent modifier volumes.
The translated option tables still carry the two entries until they are
next regenerated; they are looked up by the English table's keys, so
they are not shown.
"Framerate" (Full Speed / Normal) chose when the core handed control
back to the frontend: after each rendered frame, or at each vblank. That
choice is gone. Since the threaded rendering rework every retro_run()
runs the machine to the next vblank and returns, with Threaded
Rendering on or off, with the frame deadline behind it for a game that
has the display off. The core never waits for the game to render.
What the option still set, settings.UpdateMode, was read in one place:
rend_end_sch(), and there only in a build made with NO_THREADS=1, which
no platform in the Makefile is. In every other build the option changed
nothing. In that one, its default (Full Speed) skipped the call that
presents the frame at the end of a render.
So the option is removed, with settings.UpdateMode. The per-game
override that forced it, settings.UpdateModeForced, goes too; the
updatemode_type column stays in rom_luts.h, unused, rather than every
row of the table being rewritten. rend_end_sch() presents
unconditionally.
The translated option tables still carry the entry until they are next
regenerated; they are looked up by the English table's keys, so it is
not shown.
Three were left in the core's own sources: the count of custom-texture
load requests in BaseTextureCacheData, the reset request flag, and an
unused <atomic> include in gles.h. They are retro_atomic like the rest;
the mirror of the texture counter in tools/threads/threads_test.cpp goes
the same way. No change in behaviour.
With Threaded Rendering on, nothing tied the emulation to retro_run. The
emulation thread ran free, held back only by the audio it pushed to the
frontend itself, 512 samples at a time, from its own thread, whenever
that call blocked. retro_run waited up to 100 ms for whatever render
turned up and returned a duplicate frame if none did. A render the game
started while the last one was still being drawn was thrown away. So a
retro_run could last any length, carry a new frame or not, and its audio
arrived some other time from some other thread.
Each retro_run now makes exactly one frame, the same one it makes with
Threaded Rendering off:
- retro_run wakes the emulation thread, which runs the machine up to
vblank and goes back to sleep, as dc_run() does in retro_run itself
when rendering is not threaded;
- when the game starts a render the emulation thread hands it to the
libretro thread and sleeps; that thread reads what the frame takes
from the game's memory (display lists, textures, framebuffer), lets
the emulation thread go on, and draws. A render to a texture, which
ends up back in the game's memory, keeps it asleep until it is done;
- every render is drawn. The emulation thread waits for the one before
to be finished; none is dropped;
- at vblank the frame's audio is sent from the libretro thread in one
batch, together with its video.
The renderers read some PVR registers while they draw, by which time the
emulation has moved on and may have changed them. Each render now
carries a copy of the registers made when the game started it, and the
renderers' files read through that (PVR_REGS_FOR_RENDERER), so they draw
from what non-threaded rendering would have read.
EmuBaton (core/libretro/emu_baton.h) is that handshake. There is no
lock, no sleep and no timeout in it: every change of hands is one atomic
operation and a thread with nothing to do sleeps on an eventcount. A
save state from the frontend's task thread takes the machine between
two frames.
The "synchronous rendering" and "delay frame swapping" options have
nothing left to choose and are hidden.
tools/threads/live.sh runs the test disc for the same 300 frames with
Threaded Rendering on and off and requires the same screenshot, byte
for byte, and the same save state (live_same.py leaves out the host
addresses a state carries). The old threaded path fails that: its state
differs from the non-threaded one. tools/threads/run.sh puts the new
EmuBaton under ThreadSanitizer.
The previous change routes non-threaded control-return through vblank in
os_DoEvents, which fixes the reported case (a static screen during disc loading
no longer blocks). But that still depends on the SPG raising vblank: rend_vblank
only runs when pvr_cur_scanline == SPG_VBLANK.vstart, so before video timing is
programmed, or if a title programs vstart out of range or a degenerate
Line_Cycles, vblank need not fire and dc_run() could still run unbounded.
Add a scheduler backstop that forces dc_stop() after at most SH4_MAIN_CLOCK/20
cycles (~2.5 frames), re-armed at every dc_run() entry. Under normal operation
vblank fires every frame, well inside that budget, so it never triggers; if
vblank stalls for any reason, dc_run() still returns and retro_run() is never
starved. Non-threaded only -- in threaded mode dc_run() is meant to loop
continuously on the emu thread, so the backstop is not armed there.
Verified by enumerating every CPU-stop source (dc_stop/Sh4_int_Stop): in
'Normal' mode with no rendering, none fired except the render-tied Present(),
confirming the reported stall; with these two changes control-return no longer
depends on either rendering or a well-behaved SPG.
dc_init() registers a vectored exception handler (AddVectoredExceptionHandler)
and a dynarec unwind function table (RtlAddFunctionTable, keyed to the code
cache) on every content load, but dc_term() removed neither. After Close
Content, dc_term() frees the code cache and vmem while both registrations still
point at that freed memory, so the next stack unwind or access violation -- even
one raised during teardown -- walks freed state inside the OS exception
machinery. That crashes on close/reload and is effectively undebuggable, since
the fault is in the unwinder itself rather than in steppable code. It only bites
on Win64 (VEH + RtlAddFunctionTable); other targets are unaffected.
Add common_libretro_cleanup(), called at the start of dc_term() before anything
is freed, to RemoveVectoredExceptionHandler() and RtlDeleteFunctionTable(). Keep
the VEH handle so it can be removed, make its install idempotent so reloads
don't stack handlers, and have setup_seh() drop any prior registration before
re-adding (the code cache is re-prepared each load).
Diagnosed by inspection; needs a Win64 runtime check to confirm.
emu_in_thread, performed_serialization and reset_requested are written on one
thread and read on the other (emu thread vs. the libretro thread that drives
serialize/reset/unload). They were plain bools, i.e. a data race / UB. Make
them std::atomic<bool>; the mutex handshakes still provide the ordering that
matters, so relaxed semantics are sufficient for these standalone flags.
wait_until_dc_running() and acquire_mainloop_lock() hot-spun for up to five
seconds, burning a full core. The first has no work to do until the emu thread
reaches dc_run; the second must keep kicking the emu thread out of its render
wait but not at spin speed. Back both off by a millisecond per iteration with
retro_sleep(1); timeout and kick semantics are unchanged.
Create maple devices after loading the game so the required arcade
controls are known (AW).
Map ftspeed and maxspeed accel and brake to triggers.
Issue #865
Use file extension to select the right disc driver
Better error reporting, less memory leaks and crashes on invalid content
Force HLE off when booting BIOS
Don't crash if missing BIOS file
Better IP.BIN parsing (wince, vga)
Use wince flag and boot filename for wince detection
Switch to TV composite if vga unsupported
Switch to supported BIOS region if current is unsupported or default
Add default ISP information to flashrom
Use default asyncmap for LCP control packets
Delay reading from picotcp ppp stream when connecting
Accept [PSH,SYN] as valid tcp connect packet
Add PBA Tour Bowling 2001 as wince game
Disable vmem32 for StarLancer and Maximum Pool (req'd for online support)
LibretroAdmin