this was breaking the CI process -_-
simplify libretro makefile (#1)
only build the components needed by libretro
default to building with GCC
see if this can clear some CI hurdles
let runners override toolchain
libretro: build fixes (#2)
libretro: more apple build fixes (#4)
libretro: apple build fixes (#5)
buildfix for iOS/tvOS
libretro: frontend VFS support + Android/Windows/libnx build fixes (#3)
* libretro: route all file access through the frontend VFS
RetroArch can hand the core paths that don't exist on the real
filesystem - the common case being Android SAF content:// URIs, which
only the frontend can open. The core sets need_fullpath, so it opens the
content itself and every one of those loads currently fails.
All of the core's file I/O goes through utf8::ifstream/utf8::ofstream, so
routing those two through a VFS-backed streambuf covers ROMs, patches,
firmware, battery saves, save states, screenshots and recordings:
- Utilities/VfsFile.{h,cpp}: VfsStreamBuf, a std::streambuf on top of
retro_vfs_interface with a stdio fallback for when no frontend VFS is
available, plus VfsIo helpers for stat/mkdir/opendir/remove/rename.
Note that RetroArch's seek() returns 0 rather than the new offset, so
the position is always read back with tell().
- UTF8Util.h: in LIBRETRO builds utf8::ifstream/ofstream are backed by
VfsStreamBuf (the _WIN32 wide-path classes stay for other builds).
- retro_set_environment asks for VFS v3 and falls back to v2, so the
directory calls are used when the frontend has them.
- FolderUtilities uses the VFS for CreateFolder/GetFolders/
GetFilesInFolder when v3 is available, std::filesystem otherwise.
- ZipWriter builds archives in memory and writes them out through the
stream layer - miniz's file API goes straight to stdio.
- The remaining direct std::ifstream/std::ofstream/rename/remove calls
in libretro.cpp and sha1.cpp now go through the same layer.
Verified with a harness that only accepts saf://-prefixed paths (a core
still on stdio fails to load): GB and SNES content loads, and the
RecentGames archive is written back through the VFS. A second harness
that refuses GET_VFS_INTERFACE checks the stdio fallback.
* libretro: fix the Android, Windows and libnx builds
Android (No rule to make target '.../Core/APU.cpp'): Libretro/jni/
Android.mk pulled its file list from Libretro/Makefile.common, a
leftover from the Mesen 1 libretro port that lists sources this tree
doesn't have. Android.mk now derives the sources the same way
Makefile.libretro does (and builds as C++17, not C++11); the two stale
Mesen 1 makefiles are removed.
Windows x64: two failures.
- Lua/unixdgram.c needs sys/un.h. The unix-domain luasocket modules
(unix.c, unixdgram.c, unixstream.c, serial.c) aren't registered by the
core and aren't in the MSVC project either, so they're excluded.
- The core is cross-compiled from Linux, so uname reported Linux and the
build produced a .so with ELF-only link flags. The OS is now taken
from the target triple, and the Windows build gets --no-undefined plus
the winsock/UPnP import libraries.
libnx: the job builds with platform=libnx and then moves
mesen2_libretro_libnx.a, but Makefile.libretro had no such target, so it
built a host .so and the job died on the move. It now includes
switch_rules, compiles with the devkitA64 flags and archives a static
library. Untested - this needs a CI run on the libnx image to confirm the
core itself builds for the Switch.
Also fixes the compiler detection: `$(CXX) --version` matched 'gcc|g++'
by name, so a plain `make -f Makefile.libretro` (CXX=c++) took the clang
path and passed -flto=thin to GCC. It now looks for clang instead.
* libretro: use _wfopen_s in the MSVC build
The Mesen solution compiles with C4996 as an error, so the deprecated
_wfopen broke the standalone Windows build. The stdio fallback now has a
single open helper that uses _wfopen_s on MSVC.
* libretro: keep the core changes behind LIBRETRO, restore the old makefiles
Review feedback on libretro/MesenCE#3: the changes outside the libretro
glue should be conditional so this stays a soft fork, and the deleted
files should come back.
Everything the VFS work touches in shared code is now behind #ifdef
LIBRETRO - the WaveRecorder stream member, sha1's file hash, ZipWriter's
archive writing and FolderUtilities' directory calls all build exactly as
before without it. VfsFile.{h,cpp} is a libretro-only translation unit now
(empty otherwise), so it is out of Utilities.vcxproj again; Makefile.libretro
and Android.mk pick it up by globbing.
The macro is LIBRETRO, which is what Makefile.libretro and Android.mk
define, rather than __LIBRETRO__.
Libretro/Makefile and Libretro/Makefile.common are restored. Android.mk no
longer includes Makefile.common, so the stale Mesen 1 file list is only
dead weight rather than a broken build.
Re-verified after the change: linux-x64 core builds, the saf:// harness
loads GB and SNES content and writes the RecentGames archive through the
VFS, and the no-VFS harness still loads content through stdio.
* libretro: confine the VFS to the libretro layer
Review feedback on libretro/MesenCE#3: keep the VFS out of the core and
wrap the core's existing file access instead.
VfsFile.{h,cpp} moves to Libretro/, so it is not part of Utilities at all
any more. The interception point is utf8::ifstream/ofstream in
UTF8Util.h - the one place the core already routes file access through -
which is the only file in Core/ or Utilities/ that mentions the VFS.
The four sites that used to bypass utf8:: now go through it, which drops
their #ifdef LIBRETRO entirely (and incidentally fixes non-ASCII paths on
Windows, where utf8:: is what the rest of the codebase already uses):
- sha1.cpp opened a raw std::ifstream
- WaveRecorder held a std::ofstream, now a utf8::ofstream opened in place
rather than move-assigned
- ZipWriter used miniz's stdio API; it now drives mz_zip_writer_init with
a write callback backed by an ofstream, and reads added files through
an ifstream. Still streamed, so no in-memory archive
- FolderUtilities keeps three guarded early-returns; the directory walk
itself moved to VfsIo::GetEntries/CreateFolderIfSupported, which return
false when the frontend has no v3 folder calls so the core falls back
to std::filesystem
Makefile.libretro and Android.mk compile Libretro/VfsFile.cpp explicitly.
libretro: fix the Windows and libnx buildbot jobs (#6)
* libretro: lowercase the Windows headers for the MinGW cross-build
The buildbot's libretro-build-windows-x64 job is an MXE MinGW cross-build
running on Linux, where the include lookup is case-sensitive. mingw-w64
ships these headers all-lowercase, so <Windows.h> and <Ws2tcpip.h> are not
found and the job dies on Utilities/PlatformUtilities.cpp:
fatal error: Windows.h: No such file or directory
MSVC is unaffected either way, since the Windows filesystem matches the
name regardless of case.
UTF8Util.cpp keeps its <Windows.h>: that include sits behind #ifdef
_MSC_VER, so the MinGW build never reaches it.
* libretro: don't require IN6ADDR_ANY_INIT on libnx
libretro-build-libnx-aarch64 dies on luasocket:
Lua/inet.c:374:39: error: 'IN6ADDR_ANY_INIT' undeclared
devkitA64's newlib declares in6_addr, sockaddr_in6 and AF_INET6 - the
rest of the IPv6 code around it compiles - but not that macro. Fall back
to zeroing the address, which is what the wildcard address is by
definition and does not depend on how in6_addr is laid out.
libretro doesn't need upnp mapping
hopefully fix win64 builds
remove LUAOBJ from libnx
hopefully get it building on CI
fix typo in ifdef
libretro: fix the remaining Windows and libnx buildbot failures (#7)
* libretro: link winmm on Windows
PlatformUtilities asks for a 1ms timer resolution while a game runs, and
nothing in LIBS provides timeBeginPeriod/timeEndPeriod, so the MXE link ends
on __imp_timeBeginPeriod / __imp_timeEndPeriod. Those two are the only
symbols left unresolved in the buildbot's windows-x64 job.
* libretro: define _DEFAULT_SOURCE on libnx
Utilities/Socket.cpp does not compile there: u_long, fd_set, FD_ZERO,
FD_SET, select and struct timeval are all undeclared. They are all in
newlib, behind __MISC_VISIBLE and __BSD_VISIBLE - and newlib sets both from
_DEFAULT_SOURCE alone, which it auto-defines only when __STRICT_ANSI__ is
absent. CXXFLAGS passes --std=c++17, which defines it.
glibc turns _DEFAULT_SOURCE on whatever the dialect, so the same source
builds everywhere else and only the Switch job sees this.
re-add LUAOBJ to libnx
libretro: fix the windows-x64 link and get libnx past Lua (#8)
* libretro: give the VFS streams a key function
The windows-x64 job links with -flto and stops on multiple definitions of
the vtable, VTT and construction vtables for utf8::ifstream and
utf8::ofstream, one pair per translation unit that includes UTF8Util.h:
BsxCart.o: multiple definition of `vtable for utf8::ifstream';
libretro.o: first defined here
std::istream and std::ostream inherit virtually, so a class derived from
them gets construction vtables and a VTT as well as its own vtable. With no
key function all of it is emitted everywhere into .gnu.linkonce comdats, and
on PE/COFF the group keys come out inconsistent between objects - the same
section is keyed on the vtable in one and on the construction vtable in
another - so the linker keeps both instead of folding them.
Declaring the destructors in the header and defining them in the .cpp makes
those the key functions, so one object emits the lot. The VFS streams are the
only classes here that derive from a virtual base, which is why this is the
job that noticed.
* libretro: build libnx as gnu++17 instead of patching around __STRICT_ANSI__
declarations back for Utilities/Socket.cpp. That was the narrower of the two
options and it only covered newlib: the job now reaches Lua and stops there,
because gcc's own limits.h leaves LLONG_MAX undefined in C++ mode when
__STRICT_ANSI__ is set, and luaconf.h reads that as "no long long":
Lua/luaconf.h:572: error: #error "Compiler does not support 'long long'.
Use option '-DLUA_32BITS' or '-DLUA_C89_NUMBERS'"
Both come from the same place, so take the other option instead. MESENFLAGS
lands after the global --std=c++17 in CXXFLAGS, so this stays scoped to
libnx; every other target keeps the dialect it had.
* Delete mesen2_libretro.so
libretro: give libnx the ipv6_mreq luasocket needs (#9)
libretro-build-libnx-aarch64 is the last red job on the buildbot, and it
dies in luasocket:
Lua/options.c: In function 'opt_ip6_setmembership':
Lua/options.c:399:22: error: storage size of 'val' isn't known
399 | struct ipv6_mreq val;
devkitA64's newlib declares in6_addr, sockaddr_in6 and the
IPV6_ADD_MEMBERSHIP / IPV6_DROP_MEMBERSHIP constants - the two callers a
few lines up compile, and the IN6ADDR_ANY_INIT fallback in inet.c already
leans on in6_addr being there - but it never declares the struct those
two options take.
There is no macro to test a struct tag for, so the shim keys off
__SWITCH__, and it declares luasocket_ipv6_mreq rather than filling in
struct ipv6_mreq so that a devkitA64 that grows the real one later cannot
collide with it. socket.h is where it goes because options.c is built on
every platform and only reaches usocket.h off Windows.
The layout is the one RFC 3493 fixes; checked against glibc, both are 20
bytes with the fields at 0 and 16, so the buffer handed to setsockopt()
is unchanged. Whether the Switch network stack honours IPv6 multicast
joins at runtime is a separate question - this only stops the option from
breaking the build for every other libnx user.
set core option to default to 44.1 khz
this was making everything 55 FPS instead of 60
libretro: give libnx the LLONG_MAX luaconf.h expects (#10)
With the ipv6_mreq shim from #9 in, the libnx job gets past Lua/options.c
and dies further along, in the first C++ translation unit that pulls in
lua.hpp:
In file included from Lua/lua.h:16,
from Lua/lua.hpp:6,
from Core/Debugger/LuaCallHelper.h:3,
from Core/Debugger/LuaCallHelper.cpp:2:
Lua/luaconf.h:572:2: error: #error "Compiler does not support 'long long'."
luaconf.h includes <limits.h> itself and then gates its LUA_INT_LONGLONG
branch on LLONG_MAX. devkitA64's newlib does not make that macro visible to
C++ translation units, so the branch falls through to the #error - while the
C files in the same build, which the -std=gnu++17 in CFLAGS does not apply
to, see it and compile.
The error suggests LUA_32BITS or LUA_C89_NUMBERS, but both change Lua's
integer type, and only on the Switch. GCC always predefines
__LONG_LONG_MAX__, so fill LLONG_MAX, LLONG_MIN and ULLONG_MAX from it
instead, guarded on __SWITCH__ and on the macro actually being absent.
Checked against glibc's limits.h: all three expressions compare equal to the
real LLONG_MAX, LLONG_MIN and ULLONG_MAX, and luaconf.h still parses from
both C and C++ with the block inert off-Switch.
fix linking, rename lib, prep for GB/C
get more consoles working. PCE still not there
fix PCE hucard loading; CD still broken
fix PCE CD loading and firmware. SGB still not quite there
get SGB working
add gitlab recipe for libretro buildbot
This is implemented by pretending every 16 bytes in ROM is a different function, allowing the profiler to display cycle counts for each individual "function" in the UI
Also fixes a missing call to ResetFlags for the merge opcode for FX3
Also fixed a few scenarios where the PPU memory write callbacks did not allow the Lua script to change the value about to be written (because the callback was called too late)
-Fixed getAccessCounters documentation (swapped param order)
-Fixed file watcher not being disabled properly when closing the script window, causing the script to run if changed on the disk, even if the script window is closed
-Prevent file watcher from triggering from writes done by the script window itself (which caused the script to be reloaded 2 or 3 times in a row)
-Fixed UI freezes when using emu.log() on a large table
-Fixed crash when 100k character limit (in the last 500 lines) for script log was reached (out of bounds read/write)
-Added "fps" value to "getState"
-Slightly improved getAccessCounters/getCdlData performance
-Added a new drawPixels API to improve performance when a script needs to draw a lot of individual pixels
The game starts trying to play an audio track, but then cancels it before the seek operation completes and starts loading data instead. This triggered a bug where the cancelled play/seek operation would send a SCSI "good" status message, breaking the SCSI bus' state and causing the game to freeze.
-Some warnings were suppressed (undefined-inline and return-type-c-linkage)
-Warnings as errors was enabled to make CI builds fail when warnings are introduced
-Slightly altered the behavior of SnesCpu::IdleOrRead, but this should match what ares does
Misc fixes:
-Fixed issues with the opcode tooltip not appearing for some GBA instructions (esp. when the condition code ends with an 's', e.g: rsbcs, movls, subcs)
-Fixed byte code not being lower case when the "use lower case" option is enabled
Also, minor bug fixes for WS:
-Fixed tilemap viewer for sometimes using the wrong background color
-Fixed the 4-bit RGB color values in the palette viewer having the R and B values swapped
Also:
-Improved emu.log() to support tables/array and numeric values, instead of only strings.
-Fixed duplicate save state key name for GB
-Fixed TrimRight logic not working properly when the string had a single non-space character followed by spaces (e.g TrimRight("A ") returned "A ")
An additional frame's worth of input lag was added by runahead because of the timing at which it loads/saves the state vs when the inputs are polled (NES core was not affected due to the timing at which it polls controllers compared to other cores)
Also fixed a SNES-specific runahead issue when the game starts a very long DMA before vblank that ends after vblank, causing some frames to not be rendered at all and showing older frames instead (this is a rare scenario, but caused issues in Star Fox)
This also fixes an issue where the dot at H=22 was getting drawn at H=23 (along with the H=23 dot), causing both dots to be drawn on the same cycle. This caused a visual issue in the test_noise test - the first pixel of the grayscale line on the left was not in the being drawn correctly.
Timers, DMAs and the PPU no longer run during sleep (prevents audio DMA from being triggered and executed), and prevents the PPU from triggering DMAs/IRQs during sleep
This also fixes a bug where the GB-style APU channels could stay disabled permanently after leaving sleep mode (because the StopMode flag wasn't being reset to false)
Some games/mappers never update CHR banking, which caused previous changes done for performance to make 0000-1FFF open bus instead of being mapped to CHR RAM by default. The mappings need to be set after UpdatePageSizes is called.
Also fixes presets/clear buttons for NTT, Pachinko & Hypershot, and improves layout for all controller config popups (buttons/borders are aligned, etc.)
-Fixed BG shifter reload timing by moving it to the last dot of every 8-dot sequence (instead of the first dot). Fixes the "BG Serial In" test.
-Changed sprite shifter init/reset logic to allow edge cases where the shifter starts outputting pixels on a different scanline (when rendering was disabled during specific portions of the scanline, etc). Fixes the "Stale Sprite Shift Regs" test.
-Tweaked code in ProcessScanlineImpl, etc. to slightly improve performance
-Fixed missing PPU register mirror reads/writes in event viewer
-The pachinko/subor mouse/ntt/virtual boy controllers could be selected in the UI for 2p/4p adapters, but did not work at all
-Keybindings for the "custom keys" for some controllers were not working when connected to a 2p/4p adapter
-Pachinko controller did not allow the user to configure the keybindings for NES-style buttons
The NES (and AV Famicom) both ignore consecutive reads to 4016/4017 (when the reads on 2 consecutive CPU cycles). This implements that behavior when the corresponding console type is selected.
These changes also simplify the DMA code, because a lot of extra code was in the DMA logic to simulate the same result. Moving this logic to the 4016/4017 reads makes the code simpler and allows it to work properly when a regular instruction reads from 4016/4017 twice in a row, too.
hizzlekizzle