Had an idea, not sure if this is useful or not but I figured it might
help give an indication for newer people or those not in the Discord
loop if code is already duplicated when looking at the function report.
Here's an example output from the current state (minus the scratches
which didn't seem to load for me locally so I skipped them)
[functions.md](https://github.com/user-attachments/files/17162454/functions.md)
@bismurphy guessing this would be your area of expertise 😄
Ubuntu/Debian and Python recommend using virtual environments for
project-specific Python dependencies and as of Ubuntu 24 this is lightly
enforced when installing packages via pip. This is due to pip and the
system package manager installing files to the same location which may
cause either's internal state to no longer reflect what is actually
installed.
This updates the project to use a Python `virtualenv` for project
dependencies and updates internal scripts to support both global and
virtualenvs, but favors virtualenvs for new workspaces.
All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the
first `python3` in the path. For those with a virtualenv configured,
this will be the Python managed there. For everyone else, this should be
the system Python or whatever other scheme they may have used previously
(assuming `python3` already existed in their `PATH`).
The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and
use `python3` from there if it exists and has been configured. It also
has a new `python-dependencies` target which will configure the venv and
install all python dependnecies there.
The `Dockerfile` has been updated to create an external `.venv` outside
of the project directory. Python's `virtualenv`s are not relocatable and
may hardcode paths which are mounted differently in the container and
host. To deal with differences in paths between the container (which
mounts the host's project directory to `/sotn`) host which may be at an
arbitarary directory the `VENV_PATH` environment variable is used to
override paths in the `Makefile`.
GitHub workflows have been updated to use `.venv`.
Use the following input endpoints `FileOpenRead`, `FileReadToBuf`,
`FileAsString` and `FileUseContent`.
There was a lot of redundant and inconsistent code scattered across all
the functions responsible of opening and processing files. I also
realised the purpose of some File helpers was not entirely clear, so I
spent some effort to document them.
This is a small part of a bigger effort on refactoring `sim_pc.c`. I am
also planning on adding support for paths such as `disk://` to load from
`disks/us/`, `asset://` to load from `asset/`, `card://` to start adding
support for simulated memcards and so on. This is setting the very
basics to plan that out.
This will decompile all the functions but the new `EntityRWarpRoom` from
the RWRP overlay.
Note that as `e_misc.c` has a function with a different castle flag, I
had to use a shared header. Everything else was identical.
---------
Co-authored-by: bismurphy <tjmurphy@mit.edu>
This aims to deprecate all the Splat tools in `tools/splat_ext` in
favour of a more centralised asset manager. This brings the following
advantages:
* Much faster extraction
* Faster build
* Automatically define `static` symbols or unique names whenever
`static` is not possible
* Allow to embed assets into the output binary
* Drastically simplify `Makefile` by removing all the asset build rules
* Avoid situations where it is not possible to extract and build assets
that is not 4-byte aligned
This is achieved by having the splat YAML targeting a normal C file as
data and have an external tool to take care of the following:
1. Extract asset files straight from the overlay binary file into human
readable file in `assets/st/STAGE_NAME`
2. Build assets as header files that go into `src/st/STAGE_NAME` to just
include them from any C file
This requires each stage header to have the following new format: please
see `src/st/nz0/header.c`
Built assets in `src/st` are ignored by Git.
As for now, for simplicity sake, the steps `make extract_assets` and
`make build_assets` are just executed within `make extract` exclusively
for the US version.
I plan to auto-generate files such as `src/st/nz0/tile_data.c`.
For a first iteration I am aiming to handle the following:
* [X] Extract rooms: `assets/st/*/rooms.json`
* [X] Extract room layers: `assets/st/*/entity_layouts.json`
* [X] Extract tilemap data: `assets/st/*/tilemap_*.bin`
* [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json`
* [X] Extract sprites: `assets/st/*/sprites.json`
* [x] Extract entity layouts
* [X] Build rooms: `src/st/*/rooms.h`
* [X] Build room layers: `src/st/*/layers.h`
* [X] Build tilemap data: `src/st/*/tilemap_*.h`
* [X] Build tilemap definitions: `src/st/*/tiledef_*.h`
* [x] Build sprites (aka `g_SpriteBanks`)
* [x] Build entity layouts
* [x] Allow the tool to suggest how to adjust the Splat config for each
overlay
I want the tool to cover the following stages:
* [x] CEN
* [x] DRE
* ~MAD~ I do not think this can be done, it is way too different from
the other overlays
* [x] NO3
* [x] NP3
* [X] NZ0
* [x] ST0
* [X] WRP
* [x] RWRP
* ~WRP (PSP)~ Maybe in a follow-up PR
For a later iteration I plan on extracting and build:
* Entity GFX thingie
* The CLUT thingie in the header
* Uncompressed GFX data
* Cutscene data
* Blueprints
* The `src/config_us.h` thingie
---------
Co-authored-by: Josh Lory <josh.lory@outlook.com>
This reworks enough writes to libsnd/libspu to make it through the call
to SsInitHot. Instead of raw pointers we have read/write_16 functions.
The writes are saved to an array so we can test against output from
mednafen. See `src/pc/psxsdk/expected/_spu_init.txt`. The gold file is
generated by running a real psx psy-q program that calls the same
function. A modified mednafen prints the writes. There are reads to
spustat and other regs that require particular values to be returned, so
I've also imported and hooked up the SPU part of mednafen.
This makes changes to allow libsnd/libspu to compile when
WANT_LIBSND_LLE is set. If turned on it will immediately crash due to
trying to write to the SPU. A build is added to make sure it's
compiling.
Still work in progress. I removed splat as a submodule and started using
it as a pip package instead. Everything is matching but the memory card
icons part in both DRA and SEL. I still have no idea what the issue is.
Once this PR is good to be merged, we can get rid of the splat fork too.
TT_000 is the first overlay from PlayStation 1 that we are now able to
compile from the source and produce a 1:1 binary. This lead me to start
exploring the same overlay from the game Castlevania: Dracula X
Chronicles, which contains a PSP re-build of Symphony of the Night.
This PR adds all the infrastructure to add the same flow for a PSP
matching decomp. Use `export VERSION=pspeu` and then the usual `sotn`
command to splat the overlay, build it and check if it matches. Running
`make extract_disk` should not be necessary as the same ISO used from
`VERSION=hd` is also used for `pspeu`, so you will probably have it
extracted already.
Some important notes about the PSP build:
* The whole PSP build seems to be compiled with `-O0`, which makes it
much easier to decompile
* Having ŧhe PSX, PSP and Saturn builds will allow to easily
cross-reference the code and reduce fake matches
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin` is the HD PSX
build
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PSPBIN/tt_000.bin` has the same
code from the HD build, but for PSP
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PACK_E/TP00.BIN` is the same as
above, but it packs both overlay and graphics. This is the file the PSP
game seems to actually use
* The PSP build uses the Metrowerk CodeWarrior's C compiler, which is
very different from the GCC one used on PSX.
* Thanks to @mkst lead, we found a way to still use the GNU assembler
and linker
* MWCC uses [wibo](https://github.com/decompals/WiBo/), a think
compatibility layer to run Windows CLI tools on Linux. It is much more
lightweight than Wine.
* MWCC does not support the `INCLUDE_ASM` dialect, so the custom
pre-processor `tools/mwcpp` had to be created
* The exact MWCC compiler version is unknown, but I suspect it is `build
147`
* I am not yet sure of any implications for using GNU AS and GNU LD
instead of the MW correspondent tools
* Not all the functions can be correctly disassembled, spimdisasm will
just produce a bunch of `.word 0x________` due to the in-progress effort
of adding the Allegrex-specific opcodes support
---
TO-DO list before marking the PR as ready:
- [X] Add PSP build to the CI
- [x] Add progress reporting to the PSP build
- [x] Integrate source file from `src/servant/tt_000_psp` to
`src/servant/tt_000` to promote the psp build as first-citizen
---
TO-DO in a follow-up PR:
* Figure out what `header` is: can we extract it as assembly code? or
maybe as custom re-compilable asset via splat? Is it a MW stuff or a
Castlevania-specific file?
* Get rid of the last line in `Makefile.psp.mk`
This implements a null backend without SDL2 that can be run in CI and
also serve to show what the current platform-agnostic API is. There is a
core library, which is DRA and the pc supporting code, and two targets,
the SDL2 target, and the null one. null.c is the API that has to be
implemented to make a new target.
Since main is an infinite loop I added a little code to make it exit
after 60 frames.
I don't know what is wrong with the headers in null.c.
`AppleClang` has `-Wimplicit-function-declaration` and `-Wreturn-type`
defined by default. I had to disable them to have parity between
non-Apple default compiler flags.
Technically unimpressive, but I had to go through quite a good amount of
changes. Here is the full changelog to make everything work:
* Ensure `ResetPlatform` is always called to avoid memory leaks
* Add `g_RawVram` to emulate the PS1 VRAM
* ~~The engine will load the optional file `disks/vram.bin`, a RAM dump
from an emulator~~
* SDL2 will create 256x256 textures on-the-fly whenever a specific tpage
is requested via `GetVramTexture`
* The function `GetVramTexture` caches the last called tpage to avoid
tanking the performance
* `GetVramTexture` for only renders 4bpp and 8bpp textures with their
specified palette
* Remove `SDL2_image` as the font is now loaded straight from the VRAM
* Calling `VSync` will call the set callback, which the game uses for
DMA operations
* `MyLoadImage` is not yet implemented, but it is a placeholder to then
interact with `g_RawVram`
* The menu font now uses the texture found in the VRAM
* Plugged a custom version of `LoadFileSim`
* The file `sim_pc.c` is similar to the original game's code but it is
used here to load files from custom paths
* Using F5, F6 or F7 can dump the VRAM content on-screen, respectively
in 16bpp, 8bpp and 4bpp
There are new graphical glitches on the font. In some occasions it
appears black. It seems to be related to a flag in `P_TAG.code`. I plan
to dig into it when I can render entities on screen to avoid potential
mistakes. The same problem is present for the first half of Alucard's
portrait. It seems to be related when a texture is transparent? 🤷
I am not sure why the font is completely corrupted when entering in the
Equip menu. It is hard to understand if I introduced any regression.
Maybe the glitch was always there but it was hidden since I was always
forcing the font texture to be rendered.
EDIT: Implemented `LoadImage`, `SaveImage` and `ClearImage`
This adds cmake and a windows build to the CI. I haven't tested the
result, just got it building. I had to make quite a few changes since
MSVC has a number of differences compared to GCC. I added two cmake
modules to find sdl2 and sdl2_image from here
https://github.com/aminosbh/sdl2-cmake-modules
The dirent stuff is just `#ifdef`ed out for now since I think it's
unused?
I'm keeping the Makefile.pc for the moment until this is more proven. I
wasn't able to figure out how to add -fsanitize=address for some reason
to gcc/clang builds, open to suggestions there. Otherwise I think the
cmake build is more or less the same as the Makefile one.
As per title.
I created a new folder called `assetx/`, that is not cleaned with `make
clean` and that contains assets that are not extracted from the game and
that we want to keep in the repo. I am not sure if we agree on this, so
I will be happy to remove it from the PR if it is an unwanted approach.
`MySetDrawMode` is implemented and it sets the texture to render. The
problem is that currently there is not a batch system in place, meaning
`MyRenderPrimitives` will operate on the latest texture set. This works
as I am using just one texture, but it will not work long-term. The PS1
uses something called Ordering Table (our `g_CurrentOt`) that contains a
list of ordered drawing calls that will be submitted to the GPU via
`DrawOTag(g_CurrentOT)`. A later PR will come to properly handle this.
The PS1 has 32 texture pages (or `tpages`, or `tp`). I am thinking to
store each tpage into 128x128 textures. I am not sure how it will work
long-term or if it would be better to just create a single 1024x512
texture with the entire vram in it. When the tpage is 0x1E the font PNG
is loaded.
In Italian we say "Abbiamo fatto 30, facciamo 31". It literally means
"We come all the way up to 30, it wouldn't cost nothing to push a little
further for 31".
I moved `log.h` into `include/` to easily logging stuff elsewhere and
without weird `#ifdef VERSION_*` by adding `-DNO_LOGS` in the main
Makefile.
This is a bit hacky, but it does the trick. I had to expand `D_80138784`
otherwise I would get a segfault.
Both `SoundInit` and `func_801361F8` are called in the main, so I
removed them. To restore the previous behaviour and isolate the sound
engine, just replace `MainGame()` in the `main.c` with a `SoundInit()`
and then a loop that calls `UpdateGame()`.
Last, but not least, SDL2 is in.
I've been meaning to do this for a long time now.
The analyze_calls script running in the CI creates a .md file which acts
as a directory pointing to all the function graphs that were generated.
It also generates an HTML file. Because Github does not natively render
HTML files, but does render Markdown files, it is preferrable to have
this for the sake of easy viewing and linking within the gh-duplicates
branch.
We need to just move the generated file into the proper directory, which
will hopefully make it show up at
https://github.com/Xeeynamo/sotn-decomp/tree/gh-duplicates. I'm very
hopeful that this will work, and not be yet another "Fix CI" commit...
I also added a space to an output string. Doesn't really matter but
might as well make things look a tiny bit nicer.
I noticed since #642 the artifacts published by the CI have become very
big and that takes a big chunk of execution time. That is solved by
running `make` within `tools/sotn-debugmodule`. Also since the debug
module has no dependencies to be installed, it can run in its own
separate CI to further reduce the execution time, even if the
improvement would be marginal.
This CI is a bit experimental. Compare to the other CI it is triggered
by `pull_request` instead of `pull_request_target`. That allows to
simplify `actions/checkout@v3` with the draw-back CI configurations from
foreign contributors can be injected into the repo. It shouldn't be a
problem as collaborators would need to manually approve the CI execution
for new contributors. Also using `paths:` instead of `paths-ignore:`
should give us more control on what we execute in the first place.
## What
Enhance the existing `Makefile` to build new rules for the Saturn side
of the decomp. This should allow more flexibility when adding new
overlays or when tuning existing rules.
## Changes
I separated part of the Saturn build process in the separate file
`Makefile.saturn.mk`. I realise that naming it `saturn.mk` would have
been enough, but I pre-pended `Makefile.` so it can be found right below
the main `Makefile` when listing files in an alphabetic order. I plan to
do the same with the psx and psp toolchain, therefore you will find
`include Makefile.*.mk` in the main `Makefile`.
I deleted all the game building process done with Docker. Now that we
have an established way to do it natively I think it is no longer
required. We can always run the entire buildchain within a Docker
container instead of having `_native` and `_docker`. Now all the
`_native` references are removed. `build_saturn_native` is now
`build_saturn`.
`check_saturn` is no longer responsible of stripping the ELF into a
binary. That is now part of `build_saturn`.
I removed the references to `_li.o` (e.g. `alucard_li.o`) and used
`.elf` instead, which is closer to how the PSX build chain works. If
`_li.o` was a better preference, please let me know.
I am no longer using `./compile_dosemu.sh`. Instead I am using the new
`$(DOSEMU)` to directly invoke the tool within the Makefile. I have done
that to reduce the amount of dependent files.
I tried minimising duplication as much as possible. We now have a list
of overlays found in `SATURN_OVL_TARGETS`. Each expected output triggers
a series of dependencies so seamlessly build everything. `Makefile` is
smart enough to call `$(SATURN_TOOLCHAIN)` only once. If the game was
already built but just one source or symbol file changed, triggering a
new `build_saturn` will only compile the modified overlay.
The Saturn ADPCM files are now extracted in `assets/` instead of
`build/`. I think `assets/` should contain all the uncompressed and
uncooked files. The list of PCM file is not hardcoded. Instead I am now
using `$(wildcard disks/saturn/SD/*.PCM)`. This now means the tool tries
to convert PCMs from `SDD0.PCM` to `SDF0.PCM` with no success. As the
tool is silently failing I decided to leave it as I wrote it.
## Problems
I rewrote everything thinking about concurrency in mind. But `make -j
build_saturn` gives some unexpected output on `stdout`. I did not dig
too much into it. I suspect it might be dosemu. This is not a stopper as
we were not using `-j` when building the game anyway.
I also noticed doing `VERSION=saturn make build` calls
`mipsel-linux-gnu-ld` for `stage_02`. I suspect it is calling the rule
`$(MAIN_TARGET).elf: $(MAIN_O_FILES)` and simply moving `include
Makefile.*.mk` above it should fix it. But then it would cause the same
problem if I split the PSX rules into their own separate file. We never
used `make build` by setting the env variable `VERSION`, so this is not
either a breaking change or a stopper.
## Post thoughts
I am happy with what I achieved so far. I used the knowledge I
accumulated when maintaining the PSX counterpart. Since I now better
understand how `make` works, I was able to make some better decisions in
the Saturn counterpart. For example triggering a new build when the
symbol list changes is something the PSX build chain lacks of. I think
in the future it would be nice to trigger `make extract` when either the
YAML or the symbol list changes.
This probably fixes the debug module, it crashes instantly when I use
the bat familiar which I'm guessing is related to the compiler
differences found previously. I also added it to the CI since it's often
not able to compile.
