I don't have the frogress secrets etc. to fully test this but it seemed
to work on a dry run:
```
{'timestamp': 1707176381, 'git_hash': '5f3c92dc729020b7e8a6d7a98ac7549fc697089c', 'categories': {'code': {'main': 19812, 'main/total': 108268}, 'functions': {'main': 201, 'main/total': 534}}}
```
Another disappointing non-match, even closer than #1008 . There is only
one registry swap in one single line. I've been permuting for a day
without success. Putting this here if anyone wants to give it a try:
https://decomp.me/scratch/AuSvv
A side-project I've been working on that helped me understanding some
structures that are now a bit better documented in #1034 . Richter
sprites are extracted as usual. Doing `python3
tools/splat_ext/spritesheet.py decode disks/us/BIN/ARC_F.BIN assets/arc
alucard disks/us/BIN/F_GAME.BIN 1` can extract Alucard sprites too.
There is no repacking yet as I want to avoid using splat for this one.
This is a proof-of-concept for Japanese sotn_strs. The macro `_SJ` is
used to designate them. I only converted one string in config_jp to
serve as a test for the pipeline. I moved sotn_str to a folder and added
some tests. I renamed it since python doesn't like modules with - in the
name. The code could be cleaner and the conversion table could use some
more work but I think this is a reasonable start.
Mostly a proof of concept to extract resources from `BIN/F_GAME.BIN`.
There are tons of hacks to make this work:
* Tweaked png2s to decode binaries into PNGs
* Add a `config/gfx.game.json` that describes how to extract different
sprites out of a binary file
* Coded the palette location by reading the `clut` parameter in the
source code. A clut value of `0x196` corresponds to `"palette": "0x96"`
or `"palette": 150`.
* Coded the bits per pixel for a couple of 8-bit images
* Coded the palette location of those graphics that expects the palette
from DRA.BIN instead of F_GAME.BIN
* For all the undiscovered palette, I called the files `unk` and
defaulted their palette to grey
The expectation is that sprites will slowly be documented and added into
`config/gfx.game.json` by modders and enthusiasts. This does not yet
pack back these PNGs into the binary file. The way these assets are
extracted is probably not final or perfect, but we have to start from
somewhere.
Once decomp.me is updated I will update the SOTN preset to include
`--expand-li`. Let me know if you run into any issues but this builds an
🆗 binary for me.
As per title. I also added a new tool called `dirt_patch`. As I
mentioned in our Discord server there is some left-over data from
previous dev builds in DRA.BIN and potentially elsewhere too. The tool
uses the new file `config/dirt.us.json` which stores a list of patches
to avoid crazy hacks and `#ifdef` for the sake of getting a match. I
hope this tool will not be abused.
Extract the memory card icons out of DRA.BIN and SEL.BIN.
I also took the opportunity to extract `g_MemcardPortMask` into its own
source file `save_mgr_pre.c`. Unfortunately this is required to keep the
original data order. This suggests that most likely the icon palette and
bitmap were baked into the original `save_mgr.c` as byte arrays. I
decided to take a different approach and extracted them as PNGs for
better moddability.
I had to spin-off Splat into a new fork due to some breaking changes on
0.18.0.
After finding a copy of GetFreeDraEntity in RIC, I decided to rename the
function (in both DRA and RIC) to GetFreeEntity. Similarly, the function
right after it is GetFreeEntityReverse (since its logic runs in
reverse).
I also found CreateEntFactoryFromEntity in RIC, so I renamed that to
match the version in DRA. We will need to go through and use the FACTORY
macro for its uses, but I'm not going to tackle that quite yet.
Seems there is a lot of work to be done making RIC catch up to DRA,
we'll see if I keep working on that moving forward, it's neat to find
the ways the two match.
Another function, not sure what it really does besides messing with a
bunch of primitives.
Some clues in the rodata (particularly 00 00 00 00 bytes) indicated that
we should merge these C files, so I did.
An issue with maspsx was discovered in the process of decompiling this
one. I submitted a PR to maspsx which was accepted, and therefore I also
updated maspsx in this PR in order to capture the fix.
For some reason this was included as:
```
#if defined(VERSION_US)
INCLUDE_ASM("asm/us/dra/nonmatchings/627C4", func_801028AC);
#elif defined(VERSION_HD)
INCLUDE_ASM("asm/hd/dra/nonmatchings/627C4", func_801028AC);
#endif
```
But that wasn't actually needed, the functions are identical. Not sure
why it was that way but it's fixed now.
In Italian we say "Abbiamo fatto 30, facciamo 31". It literally means
"We come all the way up to 30, it wouldn't cost nothing to push a little
further for 31".
I moved `log.h` into `include/` to easily logging stuff elsewhere and
without weird `#ifdef VERSION_*` by adding `-DNO_LOGS` in the main
Makefile.
This is a bit hacky, but it does the trick. I had to expand `D_80138784`
otherwise I would get a segfault.
Both `SoundInit` and `func_801361F8` are called in the main, so I
removed them. To restore the previous behaviour and isolate the sound
engine, just replace `MainGame()` in the `main.c` with a `SoundInit()`
and then a loop that calls `UpdateGame()`.
Last, but not least, SDL2 is in.
Occasional maintenance:
* asm-differ: latest commit
* m2c: latest commit
* maspsx: latest commit
* spimdisasm: 0.18.0 latest
* splat: 0.17.3
The latest version of splat is 0.19.1 but a breaking change in
[0.18.0](https://github.com/ethteck/splat/pull/294) is preventing me to
upgrade further ([discussion
here](https://discord.com/channels/710646040331681844/813939516385525790/1172978921000669274))
Some YAML were malformed and since splat 0.17.0 there are additional
checks to ensure they are compliant. There are also new checks that
prevents a malformed symbol list including duplicates, which I fixed
too.
This has been one of the most weird functions I ever decompiled. I will
share a few tricks I learnt. In short, I tried inlining as much as
possible by removing all the temps.
---
```c
switch (self->step) {
...
case 2:
isEntityAlive = 0;
if (self->step != 2) {
return;
}
```
This was the output from M2C. As there is no way that `self->step` is
different than `2`, deleting had no effect on the matching.
---
```c
var_v0 = self->step;
switch (var_v0) {
case 1:
...
if (statement) {
self->step++;
}
...
D_SOME_VARIABLE = var_v0;
}
```
This was another very weird one. I couldn't understand why
`D_SOME_VARIABLE` was assigned that way much further down the `case 1`.
The way I fixed it is that `var_v0` was always `1` due to `case 1:`. By
doing `D_SOME_VARIABLE = 1` I got a match.
---
```c
temp_a0_2 = D_80174C2C + 1;
...
D_80174C2C = temp_a0_2 & -(temp_a0_2 < 0x10);
```
To understand this madness I used a random C compiler I found online and
tested in a `for` loop what's the output for all the given `temp_a0_2`.
It seemed the value never changed but over `16` the value was always 0.
I logically re-written that statement into something that made logically
more sense for me and it matched, even if it looks very different from
the original:
```c
D_80174C2C++;
D_80174C2C = D_80174C2C >= 16 ? 0 : D_80174C2C;
```
Fairly straightforward one here I think.
Unfortunately the subweapon name does not appear in the definition
(actually... are subweapon names in the game at all? I don't recall them
being in there, I think it's all just in the manual?) so it's hard to
map these definitions to the subweapons. Might be good to think about
whether we could insert comments somehow. Anyway, this works for now and
might help with collecting the subweapons in one place to understand the
remaining members of the definition.
Occasional maintenance:
* asm-differ: latest commit
* m2c: latest commit
* maspsx: latest commit
* spimdisasm: 0.18.0 latest
* splat: 0.17.3
The latest version of splat is 0.19.1 but a breaking change in
[0.18.0](https://github.com/ethteck/splat/pull/294) is preventing me to
upgrade further ([discussion
here](https://discord.com/channels/710646040331681844/813939516385525790/1172978921000669274))
Some YAML were malformed and since splat 0.17.0 there are additional
checks to ensure they are compliant. There are also new checks that
prevents a malformed symbol list including duplicates, which I fixed
too.
I've been meaning to do this for a long time now.
The analyze_calls script running in the CI creates a .md file which acts
as a directory pointing to all the function graphs that were generated.
It also generates an HTML file. Because Github does not natively render
HTML files, but does render Markdown files, it is preferrable to have
this for the sake of easy viewing and linking within the gh-duplicates
branch.
We need to just move the generated file into the proper directory, which
will hopefully make it show up at
https://github.com/Xeeynamo/sotn-decomp/tree/gh-duplicates. I'm very
hopeful that this will work, and not be yet another "Fix CI" commit...
I also added a space to an output string. Doesn't really matter but
might as well make things look a tiny bit nicer.
Issue was raised on discord, this should correct for it.
Previously assets.py would output raw binary data. This makes it more
difficult for the build chain to understand.
Now, it outputs an assembly file (which is just a big list of `.byte`
entries) which will represent the parsed data, and that assembly data
then gets assembled. Therefore, the symbols now exist.
In order to make this work, I adjusted the format of the splat, so now
we have lines like `[0x7718, assets, accessory, g_AccessoryDefs]` where
the entries are [Address, assets, configuration json file to use, name
of the output symbol].
I think this addresses the issue, but I am more than happy to iterate on
this if needed.
`func_8011AC3C` appears to be a very important function when it comes to
spawning entities. From what I can tell, it seems to be responsible for
creating all of the entities listed in `asm/us/dra/data/CF4C.data.s`
from 800AD0C4 through 800AD1D0, which is the game's main entity updating
function lookup table.
When func_8011AC3C is called, it uses its `self.params` to index into a
table which contains spawning information for entities. For example, if
func_8011AC3C is called with `self.params` of 0x21, it will create a
child entity with entityID of 25, which is the entity for the Hellfire
spell (which I recently decompiled and will submit as a PR soon).
I think func_8011AC3C is going to be useful for understanding entity
spawning logic, so I have gone ahead and taken the data table that it
uses at D_800AD1D4, and created it as an asset which will now be
extracted by splat. This way, we can more directly view the data, rather
than just seeing it as a giant array of u8 values. I am hoping that
having this available as json will be helpful for understanding the
entity creation logic. Overall, this PR is meant as a stepping stone to
fully understanding func_8011AC3C and what it does to create the
entities that it creates.
Doing this has helped me understand one struct member's purpose, which
is to set the ID of the child entity, so I've added that to the struct,
both the entity extension for func_8011AC3C and the struct which this is
an array of, which for now is Unkstruct_800AD1D4.
This took me a while. I used a [throw-away
script](https://gist.github.com/Xeeynamo/58da1ff8f3831d0ba5d23da27cbca025)
to help me with the quest, but I still had to manually check every
single YAML subsegment.
I transported over what we were able to decompile from main. For a few
functions I added the signatures and documented existing DRA code,
especially on the sound department (got inspired by the recent @sozud
PRs).
All the `.text` part from `main.exe` is now completely extracted as
C/ASM where appropiate. Almost all the functions have their original
PSY-Q names but 7 of them: `func_80012DBC`, `func_80012F84`,
`func_80017008`, `func_80017078`, `func_8001929C`, `func_80021F0C` and
`func_800286E0`. I did not feel confident enough to rename them, so I
left them be. The rest of the functions I am 99% sure they are all
accurate.
I am now excluding the
[asm/](https://github.com/Xeeynamo/sotn-decomp/tree/master/asm/us/main/psxsdk)
folder from the repo. It was useless.
This research confirms me the game uses the PSY-Q 3.5 libraries, with
the exception of `LIBGPU.LIB`, which is from PSY-Q 3.0 for some unknown
reason.
EDIT: `make format` was not taking care of duplicated symbols in our
symbol list. To speed-up my work, all the duplicate symbols (duplicates
= name AND offset) are now removed.
Failing to decompile [RenderTilemap](https://decomp.me/scratch/WigVS)
made me realise there were a few fields and structures I have always
been suspicious to be part of the same structure. After
cross-referencing overlapping fields from different structures, I was
able to merge `D_80073088`, `g_Camera`, `D_8007309C`, `g_CurrentRoom`
and `g_CurrentRoomTileLayout` into the new `g_Tilemap`.
I was forced to touch the majority of the code-base, which gave me the
opportunity to standardise some field names (e.g. from
`currentRoomTileLayout`, `roomLayout`, `layout`, `t` into `tilemap`),
remove some fake code, redundant code and adjust some symbols.
This pull request fixes typo python function helper texts in
`tools/splat_ext/spritesheet.py`. I kindly request the repository
maintainers to review and merge it. Thanks! ❤️
Fixes a very annoying issue it has been bothering me for a while. Since
the introduction of the sotn-str tool in #663 the line output has been
wrong and I failed to realise it immediately. This caused issues like
`differ` reporting the wrong C line number for the correspondent
assembly output, wrong debug info and so on.
Decompiles HandleMenu. The HD version is not yet in and I will submit it
with another commit in this same PR. I wanted to collect some feedback
before decompiling the HD counterpart.
This is some code I wrote a while ago and I think it might make sense to
add it to `decompile.py`.
This function takes in an assembly file, and will scan through it for
any jump tables. It will then find them in the asm folder, and move the
jumptable information into the assembly file for the function. This
means that users do not have to seek out rodata and paste it into the
asm file manually.
As an example, the user can now run `dec EntityAlucard` and it will
successfully decompile the function and insert it to the .c file,
because now when it passes the assembly to m2c, the rodata is there at
the bottom of the assembly function. Overall this has been a big
convenience for me and I hope it will be helpful for others.
I started importing the data in SEL also as a proof of concept we can
also import the SOTN-specific formatted strings. For that I integrated
the new tool `tools/sotn-str.py` into the build chain. It works like the
following:
* `./tools/sotn-str.py parse disks/us/ST/SEL/SEL.BIN 27598`: read the
binary content in a specific offset and parse it as a readable Unicode
string.
* `cat src/st/sel/2C048.c | ./tools/sotn-str.py process` converts the
Unicode string into a byte sequence that can be understood by the SOTN
engine.
There is a dirty hack I did to make this work:
```c
#ifndef SOTN_STR
#define _S(x) (x) // Strings processed by tools/sotn-str.py
#endif
```
When invoking `mipsel-linux-gnu-cpp` I am passing -DSOTN_STR,
effectively ignoring `_S` in the pre-processor. Then I am passing it
into `sotn-str` before `iconv` to resolve any string surrounded by `_S`.
I am a bit of afraid this might be annoying when using the permuter. I
tried doing `$(SOTN_STR) -f $< | $(CPP)` but apparently `cpp` does not
work well with content passed via stdin.
Now that assets.py exists, it's a whole lot easier to extract assets!
In order to make g_EnemyDefs get added to assets/dra/enemydefs.json,
only a few changes are needed.
1) Realign the bounds of the splat to add a separate line for the end of
g_EnemyDefs. g_EnemyDefs is exactly 400 elements in size. We have a new
data segment at 0xC780.
2) Change the yaml to extract it as "assets", named enemydefs.
3) Previously we would output name_resolved and desc_resolved to the
.json assets; since g_EnemyDefs don't have a description, we now move
that to an if statement that first checks if it exists in the config
file.
4) Create the config file for enemydefs!
Overall this creates a very smooth workflow for extracting more assets.
The biggest limitation we have with this approach is that the config
file is selected based on the name in the third space in the yaml (for
example, `[0x4B04, assets, equipment]` loads
`tools/splat_ext/equipment_config.json` by name). Therefore entries like
`layoutobj` which is used many times in `splat.us.stwrp.yaml` are not
possible to extract using this method, since there are many of them
which use the same config. I'll see if I can work out a solution to
this. Ultimately we're using the same `self.name` property to control
both what config file gets loaded, and what file we output to in
`assets`, but there's no reason those need to be the same parameter. I
just need to figure out how to pass an extra variable from the splat
into the parsing script (`assets.py`). I'll work it out somehow at some
point, and then we'll be even more flexible. For now, it's nice to be
able to browse all the enemies in the game!
---------
Co-authored-by: Luciano Ciccariello <Xeeynamo@users.noreply.github.com>
A small, but hopefully useful, update to maspsx. Per
https://github.com/mkst/maspsx/issues/26 maspsx did not handle `nop`
injection when assembly macros immediately followed a load instruction.
The latest change adds some basic support - i.e. it fixes this known
issue with Frogger.
I've tested it against the master branch of sotn-decomp and got the 🆗
When extracting equipment and accessories, we use scripts in
tools/splat_ext called accessory.py and equipment.py, each called with a
different makefile rule. These files have a lot of "data in code", with
function calls matching the structure of the equipment and accessory
structs. They also involve repeating the structure both in the
extracting from the game, and the rebuilding into the compiled
executable.
This PR takes the equipment and accessories, and turns them into a
single generic makefile rule, which calls a new script called
`assets.py`. We change the splat yaml to match. To control the
extraction, we add new _config.json files in the splat_ext. This means
we only need to define the structure of each of these in a single,
localized place.
If we like this new approach (which should be more flexible), I will see
whether it can work for more of the assets in that same area of the
makefile. I will then see about adding extraction of enemies (in
g_EnemyDefs) next.
assets.py is adapted from the old equipment.py with a lot of changes to
make it work more flexibly. I'm hoping it will be a nicer path forward
into the future.
As part of an upcoming PR, I would like to call decompile.py from an
external Python script. Therefore, to make this easy, I have moved the
main behaviors of decompile.py to a dedicated `main()` function, which
gets called with the arguments from `argparse`. The parsing, rather than
being out in the open in the file, is handled in the `if __name__==
__main__` block, since we shouldn't need to do parsing when calling
externally.
This change had one small breaking result, that `check_injected_code`
wasn't working since it was relying on `func` being an accessible global
variable. Now `func` is passed as a function argument, so it will not
error out. Otherwise, this PR should hopefully be completely "invisible"
when using decompile.py as it was used previously, but should open the
door to better external calling.
## What
Enhance the existing `Makefile` to build new rules for the Saturn side
of the decomp. This should allow more flexibility when adding new
overlays or when tuning existing rules.
## Changes
I separated part of the Saturn build process in the separate file
`Makefile.saturn.mk`. I realise that naming it `saturn.mk` would have
been enough, but I pre-pended `Makefile.` so it can be found right below
the main `Makefile` when listing files in an alphabetic order. I plan to
do the same with the psx and psp toolchain, therefore you will find
`include Makefile.*.mk` in the main `Makefile`.
I deleted all the game building process done with Docker. Now that we
have an established way to do it natively I think it is no longer
required. We can always run the entire buildchain within a Docker
container instead of having `_native` and `_docker`. Now all the
`_native` references are removed. `build_saturn_native` is now
`build_saturn`.
`check_saturn` is no longer responsible of stripping the ELF into a
binary. That is now part of `build_saturn`.
I removed the references to `_li.o` (e.g. `alucard_li.o`) and used
`.elf` instead, which is closer to how the PSX build chain works. If
`_li.o` was a better preference, please let me know.
I am no longer using `./compile_dosemu.sh`. Instead I am using the new
`$(DOSEMU)` to directly invoke the tool within the Makefile. I have done
that to reduce the amount of dependent files.
I tried minimising duplication as much as possible. We now have a list
of overlays found in `SATURN_OVL_TARGETS`. Each expected output triggers
a series of dependencies so seamlessly build everything. `Makefile` is
smart enough to call `$(SATURN_TOOLCHAIN)` only once. If the game was
already built but just one source or symbol file changed, triggering a
new `build_saturn` will only compile the modified overlay.
The Saturn ADPCM files are now extracted in `assets/` instead of
`build/`. I think `assets/` should contain all the uncompressed and
uncooked files. The list of PCM file is not hardcoded. Instead I am now
using `$(wildcard disks/saturn/SD/*.PCM)`. This now means the tool tries
to convert PCMs from `SDD0.PCM` to `SDF0.PCM` with no success. As the
tool is silently failing I decided to leave it as I wrote it.
## Problems
I rewrote everything thinking about concurrency in mind. But `make -j
build_saturn` gives some unexpected output on `stdout`. I did not dig
too much into it. I suspect it might be dosemu. This is not a stopper as
we were not using `-j` when building the game anyway.
I also noticed doing `VERSION=saturn make build` calls
`mipsel-linux-gnu-ld` for `stage_02`. I suspect it is calling the rule
`$(MAIN_TARGET).elf: $(MAIN_O_FILES)` and simply moving `include
Makefile.*.mk` above it should fix it. But then it would cause the same
problem if I split the PSX rules into their own separate file. We never
used `make build` by setting the env variable `VERSION`, so this is not
either a breaking change or a stopper.
## Post thoughts
I am happy with what I achieved so far. I used the knowledge I
accumulated when maintaining the PSX counterpart. Since I now better
understand how `make` works, I was able to make some better decisions in
the Saturn counterpart. For example triggering a new build when the
symbol list changes is something the PSX build chain lacks of. I think
in the future it would be nice to trigger `make extract` when either the
YAML or the symbol list changes.
This probably fixes the debug module, it crashes instantly when I use
the bat familiar which I'm guessing is related to the compiler
differences found previously. I also added it to the CI since it's often
not able to compile.
[As
mentioned](https://discord.com/channels/1079389589950705684/1135205782703570994/1154948845638254672),
since #595 some tools are not able to pick up some function names as the
symbol names are gone. This PR adds a quick tool to re-generate that
list from a map file and integrates it in the CI flow.
EDIT: CI is failing because the running Linter is from `master`, where
`mapfile-parser` is not installed.
The former g_BlinkTimer continues running during pauses, and may be used
for more things than just blinks, so it will now be g_Timer.
A previously unidentified variable is a timer which only runs when the
main game is running (when things are unpaused). When loading a save, it
starts as zero. It does not restart when doing things like entering a
new area in the game. This will be called g_GameTimer, since it's the
timer that only runs when in-game. It appears to increment once per
frame.
I have renamed these, changed every instance in which they are used, and
also cleaned up a few instances I found where we had patterns like
`if((variable / value) * value == variable)`, which equates to
`if(variable % value == 0)` which I think makes much more sense. I'm not
sure if I found all those cases, but at least I found a few.
