As promised, a new tool to automatically generate a splat config for PSP
and PSX overlays for multiple game versions. This first version will
perform the following tasks:
* Create the splat config with data, rodata, text and bss sections to
their right offsets
* Generate a symbol list by cross-referencing duplicate symbols
* Split the overlay by creating the correspondent C file
This tool is not currently:
* Splitting C files
* Importing shared code already decompiled
There is a lot of tech involved, especially to detect the various
sections for the PSX overlays. But I will leave it to you to discover
how the implementation work if you're interested. Hopefully the code is
self-documented enough.
Usage examples:
* `python3 tools/make-config.py mar`
* `VERSION=jp12 python3 tools/make-config.py no2`
* `python3 tools/make-config.py tt_001 --version pspko`
With this tool I aim to add in the project multiple game versions of
TT_001, MAR and RBO3 from #557 and NO0.
A few updates that didn't make it into the first round:
* masks are now identified and expanded
* AND-EQUALS and OR-EQUALS are supported
* `sotn-lint` is run before `clang-format` to avoid having to run the
`format-src` target twice to catch line breaks
* `.h` files are included in `sotn-lint` since they are being used for
shared code
* `sotn` lint ignores the `mednafen` directory
From the [public
announcement](https://discord.com/channels/1079389589950705684/1079395108501331990/1276310695964835910)
on Discord:
> We've been progressing very far with the project and I started
wondering what would be the best course of action to protect the work of
existing and future contributors. I personally consider contributions to
a decompilation project as creative work. Therefore it is my desire to
protect this creative work from being re-claimed by other people. This
is especially true in the light of the ability of modding the existing
decompiled code and create derivative work.
>
> I talked privately with the current most active contributors. And me,
@sozud , @bismurphy and @joshlory so far agreed on pushing through the
AGPLv3 license for all the decompiled code in `sotn-decomp/src`,
excluding any third party code and decompiled code from the PSX SDK. But
I've been wanting to ask all of you if you are fine your code to be
licensed and protected under the AGPLv3 terms.
>
>
https://raw.githubusercontent.com/Xeeynamo/sotn-decomp/89686514916cabd0dd88ae0387f749a889c19e05/src/LICENSE
>
> I put and all the contributors, which can be accessed to the page
https://github.com/Xeeynamo/sotn-decomp/graphs/contributors . Again,
this will work for existing and future contributors. In the license I
put my full name as I will be responsible of any legality. And I
understand not every contributor wants to disclose their identity.
Please give an approval to my latest PR if you do not have any
objection. If not, please write here publicly.
>
> PS. I understand past contributors will either not see the message
above or decide to ignore it. If I do not hear from anyone of the
contributors for more than one week between those who did not interact
with me or the server, I will still go ahead and apply the license. It
is not in my intention to ignore past contributors. But I do not want
contributors who have been inactive for months or years to be a blocker.
Please do not mistake this as an act of malevolence.
---------
Co-authored-by: Joey Murphy <tjmurphy@mit.edu>
Allow to compile weapon overlays as a single C files. The benefit is to
disallow symbols to be exported by statically compile them all but the
header. We would no longer need to use `OVL_EXPORT` as long as the
symbols are marked as `static`. The frames data is also generated as an
array of data that can be just included into their respective C files,
generated by JSON files on the fly.
I changed `animset` to `frameset` as the data does not represent a set
of animations but rather a set of frames, where every frame has multiple
sprites. The information on how the frames are played (e.g. animation)
is found elsewhere.
Following the full list of changes to achieve this project:
* The `animset` from splat is now dummied out
* The `frameset` JSON is now generated by the asset manager
* The `frameset` JSON is now compiled into `src/weapon/w_0xx_y.h`
* The `header.c` has been moved into `shared.h`
* For simplicity I added a `#define g_Animset w_000_1`. I did not want
to hack the asset manager too much to force the name to be `g_Animset`
and I did not want to change all the symbols yet.
* Simplified `weapon_pc.c` as we now only need the exported header
* Always try to build the most up-to-date asset manager: this should
avoid weird errors whenever people pull new changes
As the asset manager now can accept those `config` files, now the CLI
supports the following commands:
* `stage extract`
* `stage build`
* `config extract` (NEW)
* `config build` (NEW)
The config structure is heavily inspired to Splat to maintain
consistency and continuity.
I moved *some* of the PSX-specific stuff out of the common `Makefile`
and into the `Makefile.psx.mk` -- there is still more that could be done
here, but this should be enough to have the correct rules for PSX and
PSP with the desired assembler.
Also threw in a bonus feature where I've removed the `move` macro from
`macro.inc` because `maspsx` does this automagically for you now.
Mostly identical to MAD and PSP. There are four very different functions
from US that needs to be decompiled to be 100% complete:
* [x] `e_misc.c` `EntityMessageBox`
* [x] `e_misc.c` `EntityRelicOrb`
* [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist
* [x] `e_stage_name.c` `EntityStageNamePopup`
These four very different functions are the only ones that use the JP
text instead of the US one. This overlay also lacks of `BlitChar`.
`EntityStageNamePopup` is very similar to the PSP counterpart. I used
@joshlory scratch to match the HD part.
`EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD
overlay). I found some code that crossed between BETA and HD, so I just
put an `#else` after `VERSION_US`. We are starting to reconstruct how
the source code originally evolved across different game builds.
This aims to deprecate all the Splat tools in `tools/splat_ext` in
favour of a more centralised asset manager. This brings the following
advantages:
* Much faster extraction
* Faster build
* Automatically define `static` symbols or unique names whenever
`static` is not possible
* Allow to embed assets into the output binary
* Drastically simplify `Makefile` by removing all the asset build rules
* Avoid situations where it is not possible to extract and build assets
that is not 4-byte aligned
This is achieved by having the splat YAML targeting a normal C file as
data and have an external tool to take care of the following:
1. Extract asset files straight from the overlay binary file into human
readable file in `assets/st/STAGE_NAME`
2. Build assets as header files that go into `src/st/STAGE_NAME` to just
include them from any C file
This requires each stage header to have the following new format: please
see `src/st/nz0/header.c`
Built assets in `src/st` are ignored by Git.
As for now, for simplicity sake, the steps `make extract_assets` and
`make build_assets` are just executed within `make extract` exclusively
for the US version.
I plan to auto-generate files such as `src/st/nz0/tile_data.c`.
For a first iteration I am aiming to handle the following:
* [X] Extract rooms: `assets/st/*/rooms.json`
* [X] Extract room layers: `assets/st/*/entity_layouts.json`
* [X] Extract tilemap data: `assets/st/*/tilemap_*.bin`
* [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json`
* [X] Extract sprites: `assets/st/*/sprites.json`
* [x] Extract entity layouts
* [X] Build rooms: `src/st/*/rooms.h`
* [X] Build room layers: `src/st/*/layers.h`
* [X] Build tilemap data: `src/st/*/tilemap_*.h`
* [X] Build tilemap definitions: `src/st/*/tiledef_*.h`
* [x] Build sprites (aka `g_SpriteBanks`)
* [x] Build entity layouts
* [x] Allow the tool to suggest how to adjust the Splat config for each
overlay
I want the tool to cover the following stages:
* [x] CEN
* [x] DRE
* ~MAD~ I do not think this can be done, it is way too different from
the other overlays
* [x] NO3
* [x] NP3
* [X] NZ0
* [x] ST0
* [X] WRP
* [x] RWRP
* ~WRP (PSP)~ Maybe in a follow-up PR
For a later iteration I plan on extracting and build:
* Entity GFX thingie
* The CLUT thingie in the header
* Uncompressed GFX data
* Cutscene data
* Blueprints
* The `src/config_us.h` thingie
---------
Co-authored-by: Josh Lory <josh.lory@outlook.com>
After building for all supported architectures `make format` was taking
around 30s. This commit includes various changes which get that down to
around 3½s on the same machine.
This changes the `format` target in a few ways:
* filters out files generated by CMake
* runs `clang-format` in parallel
* breaks `format` into 3 separate targets which can be run in parallel
When building the PC version cmake creates a file to test compiler
features. This file is picked up by the pattern which matches files for
`clang-format`. This file took around 20s, filtering it out brought
total format time to around 10s.
Instead of passing a list created by a `find` command directly as the
args to `clang-format` the list is now passed to `xargs` which runs
several processes in parallel, each looking at 10 files (determined by
experimentation). This has the added advantage of ensuring as the source
list grows `ARG_MAX` won't be a concern. This brought total format time
to around 6s.
`format` was broken up into three targets `-src`, `-tools`, and
`-symbols`. These targets can be run in parallel, bringing the total
time down to 3½s.
---------
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
This updates the Makefile so that targets which depend on phony targets
(like `*_dirs`) have direct dependencies on those targets rather than
relying on transient dependencies from dependent targets. Other targets
explicitly create parent directories for their files.
This resolves#739, making it possible to using make-4.4 to build in
parallel.
Still work in progress. I removed splat as a submodule and started using
it as a pip package instead. Everything is matching but the memory card
icons part in both DRA and SEL. I still have no idea what the issue is.
Once this PR is good to be merged, we can get rid of the splat fork too.
This adds some additional context for func_801B9744 and renames globals
to fit their assumed purpose.
Migrated various stream metadata to C definitions.
The makefile changes add a dependency on `sel.h` to all the objects that
get built from the `st/sel` directory.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
The `check` recipe now has dependencies on all of the files it checks
allowing them to be rebuilt if necessary.
Consistent use of `dd` and suppressing `dd`'s output which noise for the
replacement and padding it's doing.
Run 7z in non-interactive mode to avoid confirming overwrite of files
during extraction (equivalent behavior to `VERSION=us` `extract_disk`).
`patch` is a phony target.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
Extract ST/WRP out of #1119 . All the function symbols should have been
cross-referenced. There as some PSX functions missing from PSP and some
new functions from PSP that are not present on PSX (e.g.
`st_init_wrp.c`).
The files `st_debug.c` and `e_breakable.c` are shared between WRP PSX
and WRP PSP. Everything else from PSP is isolated into its own folder. I
had to do some tricks on the YAML config to allow shared code.
`ST_WRP_MERGE = st_debug e_breakable` in the `Makefile` is a bit
annoying as MWCC complains about every single minute detail from the C
source that has been already decompiled for the PSX US build.
`EntityWarpSmallRocks` is matching on PSP but I couldn't extract the
rodata without having a bunch of linker errors. This might be a Splat
issue. I need to investigate further.
`func_psp_09244760` is soooo interesting. The values from `0x11` to
`0x17` matches the Entity IDs that are unique to the WRP overlay. This
aligns to what we have in `typedef enum EntityIDs`.
Overall I am very excited to the recent discoveries from the PSP build!
OK. Here goes. Version 0.0.1 alpha of the MWCC Global Assembly Processor
(mwccgap). It's currently very simple/limited, but it appears to work
for `src/servant/tt_000/10E8.c`.
There is lot more that can be done to improve mwccgap - i.e. supporting
.rodata migration would be a good addition, but let's see how far we can
get with it in it's current state.
Note that the Makefile could do with some improvements - we don't nede
to use mwccgap for any C file that *dont* have INCLUDE_ASM macros (it's
a waste of time) so these could be ignored, i.e. for SSSV I do something
like this to find the files that need fixing up:
```
GLOBAL_ASM_C_FILES := $(shell $(GREP) GLOBAL_ASM $(SRC_DIR) </dev/null 2>/dev/null)
```
.. although this is perhaps too simple given that SOTN has a mix of PSP
and PSX functions (and therefore there may be INCLUDE_ASM for a PSX
function but none for PSP functions...
This replaces several dozen BIOS trampolines from the extracted ASM to
"decompiled" source. These are modeled on the `INCLUDE_ASM` macro, but
generate the instructions necessary for each trampoline directly instead
of importing an extracted source file.
Because these trampolines never return, and GCC 2.6 doesnt appear to
have builtins for leaving off the return jump postamble, these will
likely need to remain assembly.
This also changes the `main.elf` target to depend on `main.ld`, and
undefined symbols files, allowing `make build` to regenerate those files
if necessary.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
In CUE files `FILE` is a stateful, global declaration that applies to
all following `TRACK`s. `sotn-disk` was treating `FILE` declarations as
`TRACK` delimiters which would result in incorrect parsing of CUE files
with a single `FILE`, but multiple tracks.
Now when reading a `FILE` declaration, the path is stored and processing
continues. A `TRACK` declaration will use the previously defined path,
and if a previous track had been started, append that previous track to
the list.
This also builds `sotn-disk` using the local repository instead of
pulling the latest commit from GitHub. The target depends on `sotn-disk`
sources and will rebuild as necessary (or with `make
~/go/bin/sotn-disk`).
As an aside,
[pull/232](https://github.com/Xeeynamo/sotn-decomp/pull/232) ran into
this same error, but fixes in a slightly different way that leaves side
effects that may make supporting things like `INDEX` (for extracting the
placeholder audio, for example) more error prone in the future.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
100% decompiled.
`US` adds some padding to have the file exactly 40KB long. The HD
version does not add any padding, hence why I ended up producing
`tt_000_raw.bin`. I would have used just `tt_000.bin` as a name as found
in `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin`, but on
Windows-based file system it would collide with `TT_000.BIN` due to the
OS having case insensitive names.
I modified `make clean` as I found annoying that `VERSION=hd make clean`
would wipe out `us` build stuff.
The only different function in HD is `ProcessEvent`, which has a weaker
check. Another hint suggesting HD being older than US.
`s32 _unused[26];` added enough padding in the bss section to get an
🆗 . I am pretty sure it is unused data because the final binary is
not aligned by any power of 2.
A side-project I've been working on that helped me understanding some
structures that are now a bit better documented in #1034 . Richter
sprites are extracted as usual. Doing `python3
tools/splat_ext/spritesheet.py decode disks/us/BIN/ARC_F.BIN assets/arc
alucard disks/us/BIN/F_GAME.BIN 1` can extract Alucard sprites too.
There is no repacking yet as I want to avoid using splat for this one.
This is a proof-of-concept for Japanese sotn_strs. The macro `_SJ` is
used to designate them. I only converted one string in config_jp to
serve as a test for the pipeline. I moved sotn_str to a folder and added
some tests. I renamed it since python doesn't like modules with - in the
name. The code could be cleaner and the conversion table could use some
more work but I think this is a reasonable start.
Once decomp.me is updated I will update the SOTN preset to include
`--expand-li`. Let me know if you run into any issues but this builds an
🆗 binary for me.
Needed a break from all the asm. I realize that Makefile changes aren't
always appreciated as they can break things in unexpected ways,
especially if they come from newbies to the project (ahem), so feel free
to close this if it's not needed.
One comment about removing `/**/*.s` and `/**/*.c`: as far as I could
see there were no subdirectories, so didn't seem needed.
Uses a small JSON parsing library called cJSON (MIT license) to parse
the assets `equipment.json` and `accessory.json` and load them in memory
rather than hard coding it or pre-building it.
As per title. I also added a new tool called `dirt_patch`. As I
mentioned in our Discord server there is some left-over data from
previous dev builds in DRA.BIN and potentially elsewhere too. The tool
uses the new file `config/dirt.us.json` which stores a list of patches
to avoid crazy hacks and `#ifdef` for the sake of getting a match. I
hope this tool will not be abused.
Extract the memory card icons out of DRA.BIN and SEL.BIN.
I also took the opportunity to extract `g_MemcardPortMask` into its own
source file `save_mgr_pre.c`. Unfortunately this is required to keep the
original data order. This suggests that most likely the icon palette and
bitmap were baked into the original `save_mgr.c` as byte arrays. I
decided to take a different approach and extracted them as PNGs for
better moddability.
I had to spin-off Splat into a new fork due to some breaking changes on
0.18.0.
In Italian we say "Abbiamo fatto 30, facciamo 31". It literally means
"We come all the way up to 30, it wouldn't cost nothing to push a little
further for 31".
I moved `log.h` into `include/` to easily logging stuff elsewhere and
without weird `#ifdef VERSION_*` by adding `-DNO_LOGS` in the main
Makefile.
This is a bit hacky, but it does the trick. I had to expand `D_80138784`
otherwise I would get a segfault.
Both `SoundInit` and `func_801361F8` are called in the main, so I
removed them. To restore the previous behaviour and isolate the sound
engine, just replace `MainGame()` in the `main.c` with a `SoundInit()`
and then a loop that calls `UpdateGame()`.
Last, but not least, SDL2 is in.
Second last function of TT_000 decompiled.
* Add `printf '\x00' | dd of=$@ bs=1 seek=40959 count=1 conv=notrunc` to
ensure the output binary is 40960 bytes long (useful once BSS is
migrated)
* Add `$(BUILD_DIR)/tt_%.ld` so every time the splat YAML changes, doing
`make tt_000` triggers the extract again
* All the variables from BSS are now in `10E8` and in order. Removing
`extern` from will properly create the BSS section
* Fix 7A4D0.c to not use the familiar entity extension
* Delete the fake symbols `D_80170660` and `D_8017065C`
* Replace all the instances of `ext.generic` as `ext.bat` when possible
NOTES:
* For `SquareRoot12((...) << 0xC) >> 0xC;` I was not able to use the
macro `FLT`
*
Issue was raised on discord, this should correct for it.
Previously assets.py would output raw binary data. This makes it more
difficult for the build chain to understand.
Now, it outputs an assembly file (which is just a big list of `.byte`
entries) which will represent the parsed data, and that assembly data
then gets assembled. Therefore, the symbols now exist.
In order to make this work, I adjusted the format of the splat, so now
we have lines like `[0x7718, assets, accessory, g_AccessoryDefs]` where
the entries are [Address, assets, configuration json file to use, name
of the output symbol].
I think this addresses the issue, but I am more than happy to iterate on
this if needed.
This took me a while. I used a [throw-away
script](https://gist.github.com/Xeeynamo/58da1ff8f3831d0ba5d23da27cbca025)
to help me with the quest, but I still had to manually check every
single YAML subsegment.
I transported over what we were able to decompile from main. For a few
functions I added the signatures and documented existing DRA code,
especially on the sound department (got inspired by the recent @sozud
PRs).
All the `.text` part from `main.exe` is now completely extracted as
C/ASM where appropiate. Almost all the functions have their original
PSY-Q names but 7 of them: `func_80012DBC`, `func_80012F84`,
`func_80017008`, `func_80017078`, `func_8001929C`, `func_80021F0C` and
`func_800286E0`. I did not feel confident enough to rename them, so I
left them be. The rest of the functions I am 99% sure they are all
accurate.
I am now excluding the
[asm/](https://github.com/Xeeynamo/sotn-decomp/tree/master/asm/us/main/psxsdk)
folder from the repo. It was useless.
This research confirms me the game uses the PSY-Q 3.5 libraries, with
the exception of `LIBGPU.LIB`, which is from PSY-Q 3.0 for some unknown
reason.
EDIT: `make format` was not taking care of duplicated symbols in our
symbol list. To speed-up my work, all the duplicate symbols (duplicates
= name AND offset) are now removed.
Failing to decompile [RenderTilemap](https://decomp.me/scratch/WigVS)
made me realise there were a few fields and structures I have always
been suspicious to be part of the same structure. After
cross-referencing overlapping fields from different structures, I was
able to merge `D_80073088`, `g_Camera`, `D_8007309C`, `g_CurrentRoom`
and `g_CurrentRoomTileLayout` into the new `g_Tilemap`.
I was forced to touch the majority of the code-base, which gave me the
opportunity to standardise some field names (e.g. from
`currentRoomTileLayout`, `roomLayout`, `layout`, `t` into `tilemap`),
remove some fake code, redundant code and adjust some symbols.
I started importing the data in SEL also as a proof of concept we can
also import the SOTN-specific formatted strings. For that I integrated
the new tool `tools/sotn-str.py` into the build chain. It works like the
following:
* `./tools/sotn-str.py parse disks/us/ST/SEL/SEL.BIN 27598`: read the
binary content in a specific offset and parse it as a readable Unicode
string.
* `cat src/st/sel/2C048.c | ./tools/sotn-str.py process` converts the
Unicode string into a byte sequence that can be understood by the SOTN
engine.
There is a dirty hack I did to make this work:
```c
#ifndef SOTN_STR
#define _S(x) (x) // Strings processed by tools/sotn-str.py
#endif
```
When invoking `mipsel-linux-gnu-cpp` I am passing -DSOTN_STR,
effectively ignoring `_S` in the pre-processor. Then I am passing it
into `sotn-str` before `iconv` to resolve any string surrounded by `_S`.
I am a bit of afraid this might be annoying when using the permuter. I
tried doing `$(SOTN_STR) -f $< | $(CPP)` but apparently `cpp` does not
work well with content passed via stdin.
When extracting equipment and accessories, we use scripts in
tools/splat_ext called accessory.py and equipment.py, each called with a
different makefile rule. These files have a lot of "data in code", with
function calls matching the structure of the equipment and accessory
structs. They also involve repeating the structure both in the
extracting from the game, and the rebuilding into the compiled
executable.
This PR takes the equipment and accessories, and turns them into a
single generic makefile rule, which calls a new script called
`assets.py`. We change the splat yaml to match. To control the
extraction, we add new _config.json files in the splat_ext. This means
we only need to define the structure of each of these in a single,
localized place.
If we like this new approach (which should be more flexible), I will see
whether it can work for more of the assets in that same area of the
makefile. I will then see about adding extraction of enemies (in
g_EnemyDefs) next.
assets.py is adapted from the old equipment.py with a lot of changes to
make it work more flexibly. I'm hoping it will be a nicer path forward
into the future.
Previously if either the YAML or the symbol list were changed, we were
forced to either manually delete the `asm` folder of the specific
overlay or to invoke `make clean`, where the latter forced us to
re-extract all the overlays via `make -j extract`.
That changes with this PR. How `make` works is `target: dep1 dep2` where
if one of the dependencies has the last modified date greater than the
target, the rule is triggered again. Previously we were extracting
overlays doing `make extract_stcen`. But since every overlay extraction
generates a linker script I now changed the rules to do `make
build/us/stcen.ld` to extract the same overlay. As I explained above, if
either the YAML or one of the related symbols changes, the linker
modified date will be older and the extraction for that overlay to be
trigger again.
The above allowed me to stop polluting the repo root with the linker
scripts as they are now moved into `build/us` or `build/hd` depending of
what you're trying to build. If you do `make extract` twice in a row you
will be welcomed with a `make: Nothing to be done for 'extract'.`.
There are still some instances where you **might** need `make clean`
beforehand, especially when modifying the `segment` section in the YAML
or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I
want to see if the current solution will be enough.
### Please try removing `make clean` from your workflow, once this is
merged, to quickly detect possible problems with this new approach.
I am also planning to make more substantial changes on our build-chain
like this PR or #660 in the future. I am aiming to small incremental
changes over time in case I break someone's flow or detect design flaws.
I am also considering the breaking changes introduced in make 4.4, which
will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a
deadline to finish all the new build-chain work.
---------
Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
## What
Enhance the existing `Makefile` to build new rules for the Saturn side
of the decomp. This should allow more flexibility when adding new
overlays or when tuning existing rules.
## Changes
I separated part of the Saturn build process in the separate file
`Makefile.saturn.mk`. I realise that naming it `saturn.mk` would have
been enough, but I pre-pended `Makefile.` so it can be found right below
the main `Makefile` when listing files in an alphabetic order. I plan to
do the same with the psx and psp toolchain, therefore you will find
`include Makefile.*.mk` in the main `Makefile`.
I deleted all the game building process done with Docker. Now that we
have an established way to do it natively I think it is no longer
required. We can always run the entire buildchain within a Docker
container instead of having `_native` and `_docker`. Now all the
`_native` references are removed. `build_saturn_native` is now
`build_saturn`.
`check_saturn` is no longer responsible of stripping the ELF into a
binary. That is now part of `build_saturn`.
I removed the references to `_li.o` (e.g. `alucard_li.o`) and used
`.elf` instead, which is closer to how the PSX build chain works. If
`_li.o` was a better preference, please let me know.
I am no longer using `./compile_dosemu.sh`. Instead I am using the new
`$(DOSEMU)` to directly invoke the tool within the Makefile. I have done
that to reduce the amount of dependent files.
I tried minimising duplication as much as possible. We now have a list
of overlays found in `SATURN_OVL_TARGETS`. Each expected output triggers
a series of dependencies so seamlessly build everything. `Makefile` is
smart enough to call `$(SATURN_TOOLCHAIN)` only once. If the game was
already built but just one source or symbol file changed, triggering a
new `build_saturn` will only compile the modified overlay.
The Saturn ADPCM files are now extracted in `assets/` instead of
`build/`. I think `assets/` should contain all the uncompressed and
uncooked files. The list of PCM file is not hardcoded. Instead I am now
using `$(wildcard disks/saturn/SD/*.PCM)`. This now means the tool tries
to convert PCMs from `SDD0.PCM` to `SDF0.PCM` with no success. As the
tool is silently failing I decided to leave it as I wrote it.
## Problems
I rewrote everything thinking about concurrency in mind. But `make -j
build_saturn` gives some unexpected output on `stdout`. I did not dig
too much into it. I suspect it might be dosemu. This is not a stopper as
we were not using `-j` when building the game anyway.
I also noticed doing `VERSION=saturn make build` calls
`mipsel-linux-gnu-ld` for `stage_02`. I suspect it is calling the rule
`$(MAIN_TARGET).elf: $(MAIN_O_FILES)` and simply moving `include
Makefile.*.mk` above it should fix it. But then it would cause the same
problem if I split the PSX rules into their own separate file. We never
used `make build` by setting the env variable `VERSION`, so this is not
either a breaking change or a stopper.
## Post thoughts
I am happy with what I achieved so far. I used the knowledge I
accumulated when maintaining the PSX counterpart. Since I now better
understand how `make` works, I was able to make some better decisions in
the Saturn counterpart. For example triggering a new build when the
symbol list changes is something the PSX build chain lacks of. I think
in the future it would be nice to trigger `make extract` when either the
YAML or the symbol list changes.
[As
mentioned](https://discord.com/channels/1079389589950705684/1135205782703570994/1154948845638254672),
since #595 some tools are not able to pick up some function names as the
symbol names are gone. This PR adds a quick tool to re-generate that
list from a map file and integrates it in the CI flow.
EDIT: CI is failing because the running Linter is from `master`, where
`mapfile-parser` is not installed.
Allows to not hard code the location in-memory of decompiled functions
and imported data if not required. This allows to relocate the
referenced symbols when editing the original code or game data. If you
were getting the offset of those symbols from the symbol list in
`config/` I suggest to use the built `build/us/*.map` file instead.
This took me a monumental effort. The PR does:
* Decompile `StoreSaveData` for DRA (US and HD)
* Decompile `StoreSaveData` for SEL
* Match `MemcardFormat` in SEL
* Cross-reference symbols between DRA and SEL
* Bring common code between DRA and SEL to parity
* Discover `save_mgr.c`, `save_mgr2.c` and `save_mgr3.c` have the same
layout in both DRA and SEL
* Deduplicate `save_mgr3.c`
* Import `.data` and `.rodata` from both DRA and SEL
* Convert UTF-8 Japanese characters into Shift-JIS while building
* Minor refactor with less hardcoded magic numbers
I made the following compromises:
* In DRA US `MakeMemcardPath` there is some dirty data at the end of the
string. I had to create a byte array called `HACK_SaveName`. This does
not occur on DRA HD or SEL
* I did not de-duplciate `save_mgr` as there are a good amount of
functions and symbols I have no knowledge about
* I did not decompile `save_mgr2` and it will come as a separate PR.
Following up #454 , #453 and related from @sozud , I started to convert
some previously defined headers into proper shared C code. The advantage
from the header is the C file is completely independent, it will have
its own `.data` and `.rodata` and it will be compiled once.
I also took the opportunity to move `include/st/AnimateEntity.h` to
`src/st/animate_entity.h` for consistency. I think the change is small
enough to be included as part of this PR.
