I completely rewrote the cutscene asset handler. Now instead of parsing
the data from the original overlay into a C-like header file, it instead
follows a two-stage process. This works by extracting it in `asset/`
with `make extract_assets`, to then allow modders to modify the file and
build it as a C-like header with `make build_assets`. This also aims to
fix#1701 as the build process takes account of the two-stage process.
I created a framework where each asset type should only make available
the two methods `Extract` and `Build`. The entire transformation process
should be isolated to not create cognitive overload like what we can
find in `build.go`. I would need to migrate all the existing asset types
to properly use this new framework. The old code served well enough to
understand how to build the entire infrastructure, but it needs to be
migrated using the new pattern.
Last, but not least, I renamed `config/assets.us.weapon.yaml` to
`config/assets.us.yaml` as it is now used by all the overlays
Ubuntu/Debian and Python recommend using virtual environments for
project-specific Python dependencies and as of Ubuntu 24 this is lightly
enforced when installing packages via pip. This is due to pip and the
system package manager installing files to the same location which may
cause either's internal state to no longer reflect what is actually
installed.
This updates the project to use a Python `virtualenv` for project
dependencies and updates internal scripts to support both global and
virtualenvs, but favors virtualenvs for new workspaces.
All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the
first `python3` in the path. For those with a virtualenv configured,
this will be the Python managed there. For everyone else, this should be
the system Python or whatever other scheme they may have used previously
(assuming `python3` already existed in their `PATH`).
The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and
use `python3` from there if it exists and has been configured. It also
has a new `python-dependencies` target which will configure the venv and
install all python dependnecies there.
The `Dockerfile` has been updated to create an external `.venv` outside
of the project directory. Python's `virtualenv`s are not relocatable and
may hardcode paths which are mounted differently in the container and
host. To deal with differences in paths between the container (which
mounts the host's project directory to `/sotn`) host which may be at an
arbitarary directory the `VENV_PATH` environment variable is used to
override paths in the `Makefile`.
GitHub workflows have been updated to use `.venv`.
This aims to deprecate all the Splat tools in `tools/splat_ext` in
favour of a more centralised asset manager. This brings the following
advantages:
* Much faster extraction
* Faster build
* Automatically define `static` symbols or unique names whenever
`static` is not possible
* Allow to embed assets into the output binary
* Drastically simplify `Makefile` by removing all the asset build rules
* Avoid situations where it is not possible to extract and build assets
that is not 4-byte aligned
This is achieved by having the splat YAML targeting a normal C file as
data and have an external tool to take care of the following:
1. Extract asset files straight from the overlay binary file into human
readable file in `assets/st/STAGE_NAME`
2. Build assets as header files that go into `src/st/STAGE_NAME` to just
include them from any C file
This requires each stage header to have the following new format: please
see `src/st/nz0/header.c`
Built assets in `src/st` are ignored by Git.
As for now, for simplicity sake, the steps `make extract_assets` and
`make build_assets` are just executed within `make extract` exclusively
for the US version.
I plan to auto-generate files such as `src/st/nz0/tile_data.c`.
For a first iteration I am aiming to handle the following:
* [X] Extract rooms: `assets/st/*/rooms.json`
* [X] Extract room layers: `assets/st/*/entity_layouts.json`
* [X] Extract tilemap data: `assets/st/*/tilemap_*.bin`
* [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json`
* [X] Extract sprites: `assets/st/*/sprites.json`
* [x] Extract entity layouts
* [X] Build rooms: `src/st/*/rooms.h`
* [X] Build room layers: `src/st/*/layers.h`
* [X] Build tilemap data: `src/st/*/tilemap_*.h`
* [X] Build tilemap definitions: `src/st/*/tiledef_*.h`
* [x] Build sprites (aka `g_SpriteBanks`)
* [x] Build entity layouts
* [x] Allow the tool to suggest how to adjust the Splat config for each
overlay
I want the tool to cover the following stages:
* [x] CEN
* [x] DRE
* ~MAD~ I do not think this can be done, it is way too different from
the other overlays
* [x] NO3
* [x] NP3
* [X] NZ0
* [x] ST0
* [X] WRP
* [x] RWRP
* ~WRP (PSP)~ Maybe in a follow-up PR
For a later iteration I plan on extracting and build:
* Entity GFX thingie
* The CLUT thingie in the header
* Uncompressed GFX data
* Cutscene data
* Blueprints
* The `src/config_us.h` thingie
---------
Co-authored-by: Josh Lory <josh.lory@outlook.com>
This took me a lot of manual work. But I think I confirmed a pattern
that will help me to automate all of this for all the next stage
overlays that will be imported in the repo.
I noticed some stages with only one room having more than two layers or
more than two tile definitions, it might be either debugging or unused
content? I did not have time to explore any of that.
I modified the `tiledef` splat extension to greatly minimise the set-up
and noise.
TT_000 is the first overlay from PlayStation 1 that we are now able to
compile from the source and produce a 1:1 binary. This lead me to start
exploring the same overlay from the game Castlevania: Dracula X
Chronicles, which contains a PSP re-build of Symphony of the Night.
This PR adds all the infrastructure to add the same flow for a PSP
matching decomp. Use `export VERSION=pspeu` and then the usual `sotn`
command to splat the overlay, build it and check if it matches. Running
`make extract_disk` should not be necessary as the same ISO used from
`VERSION=hd` is also used for `pspeu`, so you will probably have it
extracted already.
Some important notes about the PSP build:
* The whole PSP build seems to be compiled with `-O0`, which makes it
much easier to decompile
* Having ŧhe PSX, PSP and Saturn builds will allow to easily
cross-reference the code and reduce fake matches
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin` is the HD PSX
build
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PSPBIN/tt_000.bin` has the same
code from the HD build, but for PSP
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PACK_E/TP00.BIN` is the same as
above, but it packs both overlay and graphics. This is the file the PSP
game seems to actually use
* The PSP build uses the Metrowerk CodeWarrior's C compiler, which is
very different from the GCC one used on PSX.
* Thanks to @mkst lead, we found a way to still use the GNU assembler
and linker
* MWCC uses [wibo](https://github.com/decompals/WiBo/), a think
compatibility layer to run Windows CLI tools on Linux. It is much more
lightweight than Wine.
* MWCC does not support the `INCLUDE_ASM` dialect, so the custom
pre-processor `tools/mwcpp` had to be created
* The exact MWCC compiler version is unknown, but I suspect it is `build
147`
* I am not yet sure of any implications for using GNU AS and GNU LD
instead of the MW correspondent tools
* Not all the functions can be correctly disassembled, spimdisasm will
just produce a bunch of `.word 0x________` due to the in-progress effort
of adding the Allegrex-specific opcodes support
---
TO-DO list before marking the PR as ready:
- [X] Add PSP build to the CI
- [x] Add progress reporting to the PSP build
- [x] Integrate source file from `src/servant/tt_000_psp` to
`src/servant/tt_000` to promote the psp build as first-citizen
---
TO-DO in a follow-up PR:
* Figure out what `header` is: can we extract it as assembly code? or
maybe as custom re-compilable asset via splat? Is it a MW stuff or a
Castlevania-specific file?
* Get rid of the last line in `Makefile.psp.mk`
This took me a while. I used a [throw-away
script](https://gist.github.com/Xeeynamo/58da1ff8f3831d0ba5d23da27cbca025)
to help me with the quest, but I still had to manually check every
single YAML subsegment.
I transported over what we were able to decompile from main. For a few
functions I added the signatures and documented existing DRA code,
especially on the sound department (got inspired by the recent @sozud
PRs).
All the `.text` part from `main.exe` is now completely extracted as
C/ASM where appropiate. Almost all the functions have their original
PSY-Q names but 7 of them: `func_80012DBC`, `func_80012F84`,
`func_80017008`, `func_80017078`, `func_8001929C`, `func_80021F0C` and
`func_800286E0`. I did not feel confident enough to rename them, so I
left them be. The rest of the functions I am 99% sure they are all
accurate.
I am now excluding the
[asm/](https://github.com/Xeeynamo/sotn-decomp/tree/master/asm/us/main/psxsdk)
folder from the repo. It was useless.
This research confirms me the game uses the PSY-Q 3.5 libraries, with
the exception of `LIBGPU.LIB`, which is from PSY-Q 3.0 for some unknown
reason.
EDIT: `make format` was not taking care of duplicated symbols in our
symbol list. To speed-up my work, all the duplicate symbols (duplicates
= name AND offset) are now removed.
Previously if either the YAML or the symbol list were changed, we were
forced to either manually delete the `asm` folder of the specific
overlay or to invoke `make clean`, where the latter forced us to
re-extract all the overlays via `make -j extract`.
That changes with this PR. How `make` works is `target: dep1 dep2` where
if one of the dependencies has the last modified date greater than the
target, the rule is triggered again. Previously we were extracting
overlays doing `make extract_stcen`. But since every overlay extraction
generates a linker script I now changed the rules to do `make
build/us/stcen.ld` to extract the same overlay. As I explained above, if
either the YAML or one of the related symbols changes, the linker
modified date will be older and the extraction for that overlay to be
trigger again.
The above allowed me to stop polluting the repo root with the linker
scripts as they are now moved into `build/us` or `build/hd` depending of
what you're trying to build. If you do `make extract` twice in a row you
will be welcomed with a `make: Nothing to be done for 'extract'.`.
There are still some instances where you **might** need `make clean`
beforehand, especially when modifying the `segment` section in the YAML
or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I
want to see if the current solution will be enough.
### Please try removing `make clean` from your workflow, once this is
merged, to quickly detect possible problems with this new approach.
I am also planning to make more substantial changes on our build-chain
like this PR or #660 in the future. I am aiming to small incremental
changes over time in case I break someone's flow or detect design flaws.
I am also considering the breaking changes introduced in make 4.4, which
will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a
deadline to finish all the new build-chain work.
---------
Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
## What is this
Required for #465 . I re-wrote how the build process works for the
`BIN/WEAPON0.BIN` ~~overlay~~ file.
## A bit of a background
On the PSP build I found `WEAPON0.BIN` is just a blob with `f0_000.bin`,
`w0_000.bin`, `f0_001.bin` and so on concatenated as a single file. `f0`
contains the graphics (or CHR) and `w0` contains the actual overlay (or
PRG). The individual overlays share a great amount of functions I
documented in `config/symbols.us.weapon.txt`.
## The problem
Splat creates its own `weapon.ld` linker script, insisting to link all
the overlays as a single binary file. While this is not technically
correct, it creates a bigger problem: symbols cannot be shared. In 58
overlays there are 58 `DestroyEntity`. Other decomps are
circumnavigating the issue by adding a prefix. Following the same
pattern we could have `W000_DestroyEntity`, `W001_DestroyEntity` and so
on. But this is not maintainable and it does not reflect our long-term
vision of having shared code between overlays.
## The solution
I created the dedicated linker script `weapon0.ld` to output the
individual overlay `w0_000.bin` and similar. `f0_000.bin` is also
created from the correspondent PNG. Once all the overlays and graphics
are created I am simply concatenating them back to `WEAPON0.BIN`. This
is much more efficient in terms of separation of concerns,
maintainability, build speed and it also allows to use the same symbol
names across overlays. Also very important, the generated individual
overlays matches the structure of what is found in the PSP build.
## Side note
I thought it was wise to prepare the Makefile and the linker script to
also allow rebasing and compiling for WEAPON1.BIN. The generated file
does not currently match and it will probably need a bit more research.
I left the build infrastructure there. To compile the non-matching
WEAPON1 just modify `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES)` to
`$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES) $(WEAPON1_FILES)`.
# What
Adds a custom debug menu. It is written in C and it is meant to replace
the Bat familar `SERVANT/TT_000.BIN`. Once loaded you can long-press
SELECT+START to soft-reset the game and keep using the debug menu
everywhere, including when playing with Richter or during the credits.
One key requirement to run this is to have an emulator that emulates the
8MB of RAM. This is a key requirement to have the debug module surviving
soft-resets or accessing to the in-game menu. I personally used [PCSX
Redux](https://github.com/grumpycoders/pcsx-redux) to build this module.
I am not sure about the compatibility with other emulators. This does
**NOT** work on real hardware and it is a choice by design. The debug
module is intended to test different areas of the game and help
decompiling. It is not intended to be used in normal gameplay.
# Build
Simply invoke `make disk_debug` to create a disk image of the game in
`build/` with the debug module replacing the Bat familiar.
# Usage
On PCSX Redux go to Configuration, Emulation and tick the box `8MB`.
## Loading the module
You need to enable the Bat Card from the menu. This will load the debug
module from the disk. If you want to re-load the module you need to
select another Familar Card, un-pause, pause again and select the Bat
Card once more.
## The main screen
You will know you have loaded the debug module when you see the blue
rectangle on the top right. You can press R2 to cycle between the menus.
Some menus will temporarily freeze the game, some not. To quickly return
in-game you can either press TRIANGLE or START. To bring back the paused
debug menu just press R2 once again.
## Debug Mode
### Stage
Teleports the player to a different stage. It is not stable and it can
crash often.
### Player
Switches between Alucard and Richter. Currently switching from Richter
to Alucard consistently crashes the game.
### No Clip
Allows to freely move the player within the room and without the
collisions on. Once the flag is enabled from the debug menu, pressing L2
while in-game will temporarily freeze the player movement and make it
immune to collision checking. You can then press the directional buttons
to slowly move the player or you can hold CROSS to move it faster. You
can also use SQUARE or CIRCLE to cycle between the player frames. Press
L2 again to deactivate the NoClip mode.
### Frame by frame
Freezes the game outside the debug module. Press L1 to advance by 1
frame. Hold L2 to put the game in slow-motion.
### Show hitboxes
As shown in the image
### Show debug messages
When the debug menu is un-paused, prints on the top left all the debug
messages from the game itself.
### Show collision layer
This prints the internal collision value for every 16x16 tile on the
screen. Look the CheckCollision function for more information on how
each printed value is used.
### Show draw calls
Shows the maximum GPU resource usage since the game started. Currently
only the `max` option works. The `current` option will not show
anything.
### Show HBlank
Prints the current horizontal blank interrupt count.
## Entity Spawn
Allows to immediately spawn new entities in the current map
### Mode
There are three list of entities, each one with their own ID: DRA, Stage
and RIC. The option RIC is hidden if Richter is not the current playing
character. As the list of entities per stage is maintained manually,
stage entities might not be available for all stages. The `Alloc` shows
how many entities are reserved or actually used. Pressing the SQUARE
button here will destroy all the entities within that range.
### ID
Press Left or Right to cycle between the different IDs available. Some
of them might crash the game immediately once spawned. Press CROSS to
immediately spawn the entity.
### Params
Each entity might have its own parameters. Sometimes the flag 0x8000 is
used, which can be toggled with the SQUARE button. Press CROSS to
immediately spawn the entity.
### Entity preview
Shows the entity before spawning it. This is turned off by default as it
can immediately crash by cycling through the available entity IDs.
### Place entity
Pressing CROSS will allow to move the entity across the screen before
placing it. Press CROSS again to place the entity and return to the
previous screen. Press SQUARE to quickly place multiple entities of the
same type.
## Sound player
There are three macro categories the sound player is split into. For
what is currently known only the sounds within the Kind 3 changes based
on the loaded stage.
### Stop all sounds
This will also disable the SPU IRQ, effectively unlocking the frame
rate.
### Load Stage
Loads a different sound font than the current loaded stage. This can
help to quickly preview and test SFXs from other stages without
necessarily moving the player there.
### Load Servant
Loads the sound font of a specific servant without necessarily equipping
the Familiar Card.
## Castle flags
Preview all the flags used to modify the behaviour of different parts of
the two castles.
### Edit mode
You can move the cursor with the directional buttons and flip the flag
with CROSS. Press L1 or R1 to cycle between the pages. The cursor warps
when reaching the border of the flag grid, allowing a faster navigation.
### View mode
Allows to move between the flags more flexibly.
### Listen mode
Listens for the modified flags while playing. Every time a flag is
modified the offset and its value is registered on the top left up to 4
rows. When all the rows are occupied, new values will just remove the
oldest one. The last modified flag will always be displayed at the
bottom.
---------
Co-authored-by: Alejandro Asenjo Nitti <sonicdcer@users.noreply.github.com>
Co-authored-by: sozud <sozud@users.noreply.github.com>
Fully splits `libsnd` into individual files following the original PSY-Q
SDK file names. The same is done with `libcd` and `libspu` just to
define the boundaries of `libsnd`. I also correctly names **all** the
function symbols from the sound library with their right offset. We
previously had some wrong symbol offsets, leading to the inability to
create their C counterparts with `INCLUDE_ASM`. This PR will start to
open the doors to decompiling `main` or understand better how `DRA`
communicates with `main`.
This took an incredible amount of time as I had to do it manually. I
would have written a script, but there are some variants I would prefer
to handle manually. To achieve this I used the [PSY-Q
signatures](e9e46e7e13/350/LIBSND.LIB.json).
I also discovered the game uses PSY-Q 3.5, which brings some piece of
mind to what they used to develop the game.
After this PR my plan is to:
* Finish to split the other PSY-Q libs
* Move `asm/main/*.s` into their own C files
* Include `main` progress in our reporting
Following the incredible work of @bismurphy in #396, this will integrate
the tool in the CI. The markdown file `gh-duplicates/functions.md` will
have clickable links to navigate right to the SVG call tree of the
correspondent functions.
---------
Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
Very happy to have this working!
Now, when you run `python3 analyze_calls.py ALL`, it will generate the
call graphs for every function in the game, and save as an SVG file in
the directory `tools/generated_graphs`. It will then finish by creating
an `index.html` that links to every one of those graphs. Each graph's
bubbles are clickable to take you to the next graph.
I have deployed an example of the result of this, linked in the sotn
decomp Discord, but because it is hosted on my personal computer, I will
not be linking it here (accessible to the wider internet and crawlers,
etc). It would be great if this script could be worked into the CI to
create a publicly visible list of functions linking to graphs, which
would automatically update whenever the repo gets updated.
As always, be sure to run `make force_extract` before invoking this.
It will be very exciting once this is up and running!
This is the script demonstrated in Discord, with both graphical and
command line interfaces.
I won't say it's particularly good Python, but it gets the job done for
now :)
This traverses all functions in /asm (after doing `make force_extract`),
and logs all functions called by every function. This takes several
seconds, so the result of this traversing is saved to a text file called
sotn_calltree.txt and used for future runs. I've added this to the
gitignore. As mentioned in the comments at the top of the Python file,
this file can be deleted if the user would like to re-generate the call
tree (mainly if symbols have been renamed, which would change the
generated assembly files).
This is very much meant to be a first draft; it may have bugs and it
definitely has ugly, low-quality code (I had to do a whole bunch of
special cases for assembly that calls functions indirectly, for
example), but at least for now this could be a useful starting point.
Hopefully myself or others will make improvements into the future!
This adds some new Makefile commands:
build_saturn_toolchain: Downloads GCCSH, and builds two docker
containers. One container is binutils for SuperH, and the other has
Dosemu.
build_saturn: Copies GCCSH, the source code, and asm into the build
directory and runs the GCCSH via Dosemu in the docker container.
Everything is kept in the same directory since it's hard to use Dosemu
otherwise.
check_saturn: Dumps the object files to binary using the binutils Docker
container, then checks the hashes.
diff_saturn: Compares the binaries and outputs a diff to
build/saturn/$FILENAME-diff.txt. Used like FILENAME=GAME.PRG make
diff_saturn.
I've added the source for game.c and t_bat.c. I've replaced one of the
asm functions in t_bat.c to show that it's possible to match simple
functions. I'm not able to generate linker scripts yet so functions that
rely on symbols from other overlays generally can't be matched yet.
This workflow is not all that clean so I'm open to ideas on improving
it. I tried using various versions of
https://github.com/decompals/old-gcc to try to get a native compiler but
couldn't get a matching build. I'm not 100% sure that this version of
GCCSH is correct but initial results seem promising.
As of right now, it's not possible to run this in CI since the dosemu
container changes the USER and Github won't let you write to disk if you
do that.
https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#user
