TT_000 is the first overlay from PlayStation 1 that we are now able to
compile from the source and produce a 1:1 binary. This lead me to start
exploring the same overlay from the game Castlevania: Dracula X
Chronicles, which contains a PSP re-build of Symphony of the Night.
This PR adds all the infrastructure to add the same flow for a PSP
matching decomp. Use `export VERSION=pspeu` and then the usual `sotn`
command to splat the overlay, build it and check if it matches. Running
`make extract_disk` should not be necessary as the same ISO used from
`VERSION=hd` is also used for `pspeu`, so you will probably have it
extracted already.
Some important notes about the PSP build:
* The whole PSP build seems to be compiled with `-O0`, which makes it
much easier to decompile
* Having ŧhe PSX, PSP and Saturn builds will allow to easily
cross-reference the code and reduce fake matches
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin` is the HD PSX
build
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PSPBIN/tt_000.bin` has the same
code from the HD build, but for PSP
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PACK_E/TP00.BIN` is the same as
above, but it packs both overlay and graphics. This is the file the PSP
game seems to actually use
* The PSP build uses the Metrowerk CodeWarrior's C compiler, which is
very different from the GCC one used on PSX.
* Thanks to @mkst lead, we found a way to still use the GNU assembler
and linker
* MWCC uses [wibo](https://github.com/decompals/WiBo/), a think
compatibility layer to run Windows CLI tools on Linux. It is much more
lightweight than Wine.
* MWCC does not support the `INCLUDE_ASM` dialect, so the custom
pre-processor `tools/mwcpp` had to be created
* The exact MWCC compiler version is unknown, but I suspect it is `build
147`
* I am not yet sure of any implications for using GNU AS and GNU LD
instead of the MW correspondent tools
* Not all the functions can be correctly disassembled, spimdisasm will
just produce a bunch of `.word 0x________` due to the in-progress effort
of adding the Allegrex-specific opcodes support
---
TO-DO list before marking the PR as ready:
- [X] Add PSP build to the CI
- [x] Add progress reporting to the PSP build
- [x] Integrate source file from `src/servant/tt_000_psp` to
`src/servant/tt_000` to promote the psp build as first-citizen
---
TO-DO in a follow-up PR:
* Figure out what `header` is: can we extract it as assembly code? or
maybe as custom re-compilable asset via splat? Is it a MW stuff or a
Castlevania-specific file?
* Get rid of the last line in `Makefile.psp.mk`
This implements a null backend without SDL2 that can be run in CI and
also serve to show what the current platform-agnostic API is. There is a
core library, which is DRA and the pc supporting code, and two targets,
the SDL2 target, and the null one. null.c is the API that has to be
implemented to make a new target.
Since main is an infinite loop I added a little code to make it exit
after 60 frames.
I don't know what is wrong with the headers in null.c.
`AppleClang` has `-Wimplicit-function-declaration` and `-Wreturn-type`
defined by default. I had to disable them to have parity between
non-Apple default compiler flags.
Technically unimpressive, but I had to go through quite a good amount of
changes. Here is the full changelog to make everything work:
* Ensure `ResetPlatform` is always called to avoid memory leaks
* Add `g_RawVram` to emulate the PS1 VRAM
* ~~The engine will load the optional file `disks/vram.bin`, a RAM dump
from an emulator~~
* SDL2 will create 256x256 textures on-the-fly whenever a specific tpage
is requested via `GetVramTexture`
* The function `GetVramTexture` caches the last called tpage to avoid
tanking the performance
* `GetVramTexture` for only renders 4bpp and 8bpp textures with their
specified palette
* Remove `SDL2_image` as the font is now loaded straight from the VRAM
* Calling `VSync` will call the set callback, which the game uses for
DMA operations
* `MyLoadImage` is not yet implemented, but it is a placeholder to then
interact with `g_RawVram`
* The menu font now uses the texture found in the VRAM
* Plugged a custom version of `LoadFileSim`
* The file `sim_pc.c` is similar to the original game's code but it is
used here to load files from custom paths
* Using F5, F6 or F7 can dump the VRAM content on-screen, respectively
in 16bpp, 8bpp and 4bpp
There are new graphical glitches on the font. In some occasions it
appears black. It seems to be related to a flag in `P_TAG.code`. I plan
to dig into it when I can render entities on screen to avoid potential
mistakes. The same problem is present for the first half of Alucard's
portrait. It seems to be related when a texture is transparent? 🤷
I am not sure why the font is completely corrupted when entering in the
Equip menu. It is hard to understand if I introduced any regression.
Maybe the glitch was always there but it was hidden since I was always
forcing the font texture to be rendered.
EDIT: Implemented `LoadImage`, `SaveImage` and `ClearImage`
This adds cmake and a windows build to the CI. I haven't tested the
result, just got it building. I had to make quite a few changes since
MSVC has a number of differences compared to GCC. I added two cmake
modules to find sdl2 and sdl2_image from here
https://github.com/aminosbh/sdl2-cmake-modules
The dirent stuff is just `#ifdef`ed out for now since I think it's
unused?
I'm keeping the Makefile.pc for the moment until this is more proven. I
wasn't able to figure out how to add -fsanitize=address for some reason
to gcc/clang builds, open to suggestions there. Otherwise I think the
cmake build is more or less the same as the Makefile one.
As per title.
I created a new folder called `assetx/`, that is not cleaned with `make
clean` and that contains assets that are not extracted from the game and
that we want to keep in the repo. I am not sure if we agree on this, so
I will be happy to remove it from the PR if it is an unwanted approach.
`MySetDrawMode` is implemented and it sets the texture to render. The
problem is that currently there is not a batch system in place, meaning
`MyRenderPrimitives` will operate on the latest texture set. This works
as I am using just one texture, but it will not work long-term. The PS1
uses something called Ordering Table (our `g_CurrentOt`) that contains a
list of ordered drawing calls that will be submitted to the GPU via
`DrawOTag(g_CurrentOT)`. A later PR will come to properly handle this.
The PS1 has 32 texture pages (or `tpages`, or `tp`). I am thinking to
store each tpage into 128x128 textures. I am not sure how it will work
long-term or if it would be better to just create a single 1024x512
texture with the entire vram in it. When the tpage is 0x1E the font PNG
is loaded.
In Italian we say "Abbiamo fatto 30, facciamo 31". It literally means
"We come all the way up to 30, it wouldn't cost nothing to push a little
further for 31".
I moved `log.h` into `include/` to easily logging stuff elsewhere and
without weird `#ifdef VERSION_*` by adding `-DNO_LOGS` in the main
Makefile.
This is a bit hacky, but it does the trick. I had to expand `D_80138784`
otherwise I would get a segfault.
Both `SoundInit` and `func_801361F8` are called in the main, so I
removed them. To restore the previous behaviour and isolate the sound
engine, just replace `MainGame()` in the `main.c` with a `SoundInit()`
and then a loop that calls `UpdateGame()`.
Last, but not least, SDL2 is in.
I've been meaning to do this for a long time now.
The analyze_calls script running in the CI creates a .md file which acts
as a directory pointing to all the function graphs that were generated.
It also generates an HTML file. Because Github does not natively render
HTML files, but does render Markdown files, it is preferrable to have
this for the sake of easy viewing and linking within the gh-duplicates
branch.
We need to just move the generated file into the proper directory, which
will hopefully make it show up at
https://github.com/Xeeynamo/sotn-decomp/tree/gh-duplicates. I'm very
hopeful that this will work, and not be yet another "Fix CI" commit...
I also added a space to an output string. Doesn't really matter but
might as well make things look a tiny bit nicer.
I noticed since #642 the artifacts published by the CI have become very
big and that takes a big chunk of execution time. That is solved by
running `make` within `tools/sotn-debugmodule`. Also since the debug
module has no dependencies to be installed, it can run in its own
separate CI to further reduce the execution time, even if the
improvement would be marginal.
This CI is a bit experimental. Compare to the other CI it is triggered
by `pull_request` instead of `pull_request_target`. That allows to
simplify `actions/checkout@v3` with the draw-back CI configurations from
foreign contributors can be injected into the repo. It shouldn't be a
problem as collaborators would need to manually approve the CI execution
for new contributors. Also using `paths:` instead of `paths-ignore:`
should give us more control on what we execute in the first place.
## What
Enhance the existing `Makefile` to build new rules for the Saturn side
of the decomp. This should allow more flexibility when adding new
overlays or when tuning existing rules.
## Changes
I separated part of the Saturn build process in the separate file
`Makefile.saturn.mk`. I realise that naming it `saturn.mk` would have
been enough, but I pre-pended `Makefile.` so it can be found right below
the main `Makefile` when listing files in an alphabetic order. I plan to
do the same with the psx and psp toolchain, therefore you will find
`include Makefile.*.mk` in the main `Makefile`.
I deleted all the game building process done with Docker. Now that we
have an established way to do it natively I think it is no longer
required. We can always run the entire buildchain within a Docker
container instead of having `_native` and `_docker`. Now all the
`_native` references are removed. `build_saturn_native` is now
`build_saturn`.
`check_saturn` is no longer responsible of stripping the ELF into a
binary. That is now part of `build_saturn`.
I removed the references to `_li.o` (e.g. `alucard_li.o`) and used
`.elf` instead, which is closer to how the PSX build chain works. If
`_li.o` was a better preference, please let me know.
I am no longer using `./compile_dosemu.sh`. Instead I am using the new
`$(DOSEMU)` to directly invoke the tool within the Makefile. I have done
that to reduce the amount of dependent files.
I tried minimising duplication as much as possible. We now have a list
of overlays found in `SATURN_OVL_TARGETS`. Each expected output triggers
a series of dependencies so seamlessly build everything. `Makefile` is
smart enough to call `$(SATURN_TOOLCHAIN)` only once. If the game was
already built but just one source or symbol file changed, triggering a
new `build_saturn` will only compile the modified overlay.
The Saturn ADPCM files are now extracted in `assets/` instead of
`build/`. I think `assets/` should contain all the uncompressed and
uncooked files. The list of PCM file is not hardcoded. Instead I am now
using `$(wildcard disks/saturn/SD/*.PCM)`. This now means the tool tries
to convert PCMs from `SDD0.PCM` to `SDF0.PCM` with no success. As the
tool is silently failing I decided to leave it as I wrote it.
## Problems
I rewrote everything thinking about concurrency in mind. But `make -j
build_saturn` gives some unexpected output on `stdout`. I did not dig
too much into it. I suspect it might be dosemu. This is not a stopper as
we were not using `-j` when building the game anyway.
I also noticed doing `VERSION=saturn make build` calls
`mipsel-linux-gnu-ld` for `stage_02`. I suspect it is calling the rule
`$(MAIN_TARGET).elf: $(MAIN_O_FILES)` and simply moving `include
Makefile.*.mk` above it should fix it. But then it would cause the same
problem if I split the PSX rules into their own separate file. We never
used `make build` by setting the env variable `VERSION`, so this is not
either a breaking change or a stopper.
## Post thoughts
I am happy with what I achieved so far. I used the knowledge I
accumulated when maintaining the PSX counterpart. Since I now better
understand how `make` works, I was able to make some better decisions in
the Saturn counterpart. For example triggering a new build when the
symbol list changes is something the PSX build chain lacks of. I think
in the future it would be nice to trigger `make extract` when either the
YAML or the symbol list changes.
This probably fixes the debug module, it crashes instantly when I use
the bat familiar which I'm guessing is related to the compiler
differences found previously. I also added it to the CI since it's often
not able to compile.
[As
mentioned](https://discord.com/channels/1079389589950705684/1135205782703570994/1154948845638254672),
since #595 some tools are not able to pick up some function names as the
symbol names are gone. This PR adds a quick tool to re-generate that
list from a map file and integrates it in the CI flow.
EDIT: CI is failing because the running Linter is from `master`, where
`mapfile-parser` is not installed.
Allows to not hard code the location in-memory of decompiled functions
and imported data if not required. This allows to relocate the
referenced symbols when editing the original code or game data. If you
were getting the offset of those symbols from the symbol list in
`config/` I suggest to use the built `build/us/*.map` file instead.
As suggested by @delta-473, GitHub recently shipped their [caching
mechanic for the
CI](https://github.com/actions/cache/blob/main/caching-strategies.md). I
can think of a few ways it can get used in our advantage:
* Cloning the "dependencies": it should save cloning about 4GB of data
to every commit on `master` and about 1.5GB per PR.
* Using this cache instead of artifacts to export the `.map` files to
calculate the decomp progress
* Same as above but to the reporting step.
This PR just addresses the first point as it is the heaviest burden of
the CI so far and it is what I think a quick win.
This adds a little script to check if the repo is dirty after running
`make format`. This would also catch errors if the format fails. If we
agree with the idea I will commit to master to get this running.
I have rewritten much of the overall logic flow of analyze_calls. This
should make it run faster. Hopefully the program logic is also easier to
understand.
The output of this script is mostly identical to the previous version.
It will now include calls through g_api to a given function, which is a
nice upgrade (for example, a call to g_api.func_800FE97C will now
properly show up on func_800FE97C's call graph).
Important change: I am now using the argparse library to parse
arguments. Running without arguments will generate all the call trees,
and save the files along with the HTML page that links to all of them.
There are two optional arguments:
`--dry` will render all call trees, but will not save them to files. The
bytes of the SVG are created, but are then simply forgotten.
`--ultradry` will perform the analysis of every function, what it calls,
and what calls it, but will stop there without generating any trees.
I believe that `--ultradry` will be sufficient to automatically run in
PRs, since the majority of problems are in the function analysis stage,
and not in the call tree generation stage. Therefore, for the CI, I
would recommend that we set PRs to run with `--ultradry`, and the main
repo to run with no arguments. In the event that we find a situation
where `--ultradry` misses an error that `--dry` catches, we can revert
to using `--dry` on PRs.
For some benchmarking: Normal runs and --dry runs take approximately 30
seconds on my computer. It turns out that writing to files is a
relatively small amount of the execution time. On the other hand,
`--ultradry` takes less than 1 second. This is the main reason I suggest
using `--ultradry` for PRs.
This will hopefully be applicable to the stalled
`weapon0-weapon1-makefile` branch.
Note: The previous run modes of this script (running with command line
interface, or running generating single flow charts one at a time) are
now removed. If anyone was relying on them, I can reimplement them, but
I assume the web interface is more convenient for everyone.
