I also deduped the common shared servant functions except
ProcessSfxState. That one has a jumptable and I didn't want to mess with
multiple files/.rodata while just adding a new overlay.
Experimental piece of work I feel confident enough to submit. A few
things to take in consideration:
* The new `pfn_entity_updates.h` header aims to use the PSP variant of
`PfnEntityUpdates` and the two `g_pStObjLayout`, I just tweaked a bit
the code from `stage.h`
* Add overlay prefix to the PSX version of `PfnEntityUpdates` and the
two `g_pStObjLayout`
* Changed `wrp/create_entity` to `rwrp/create_entity` in RWRP because we
need the `RWRP_` prefix to be issued
* Remodernized the WRP header
* I am still not sure what's wrong with `g_HeartDropArray` in `warp.c`.
I am pretty sure it is a PSX bug. I opted for the PSP version
There's a `// PfnEntityUpdates = OVL_EXPORT(EntityUpdates);`. If you
uncomment and resolve the multiple definitions you will have the two
overlays linked. I still need to figure out how to handle the
`InitializeEntity` parameters in `e_init` without going mad with the
`#ifdef`.
Takeaways from the Xbox 360 build:
* `EntityBreakable` is different in every overlay, so we can use the
`OVL_EXPORT` in our decomp
* Every other function seem to be de-duplciated. For those functions
that are slightly different between overlays, the developers of the Xbox
port opted for `if (g_StageId == XYZ)`
Takeaways from the PSP build:
* god bless the psp
I completely rewrote the cutscene asset handler. Now instead of parsing
the data from the original overlay into a C-like header file, it instead
follows a two-stage process. This works by extracting it in `asset/`
with `make extract_assets`, to then allow modders to modify the file and
build it as a C-like header with `make build_assets`. This also aims to
fix#1701 as the build process takes account of the two-stage process.
I created a framework where each asset type should only make available
the two methods `Extract` and `Build`. The entire transformation process
should be isolated to not create cognitive overload like what we can
find in `build.go`. I would need to migrate all the existing asset types
to properly use this new framework. The old code served well enough to
understand how to build the entire infrastructure, but it needs to be
migrated using the new pattern.
Last, but not least, I renamed `config/assets.us.weapon.yaml` to
`config/assets.us.yaml` as it is now used by all the overlays
This extends upon my work in display_texture.py. I have used this for
identifying a few entities already. display_texture is suited for
viewing prims, while this one is better for entities that are rendered
by RenderEntities.
I think there's enough documentation at the top of the function for
others to use it. Hopefully it can be helpful, especially as it evolves
over time.
At some point this could be converted to an editor in addition to a
viewer, but for now the viewing capability is the core focus.
Also rewrote some of DRA's sprite data (this should probably be in
sotn-assets though) to fit the standard format so that this tool could
parse the data.
I was reviewing some of the animations pulled out by sotn-assets.
Speicfically, I found that in `src/st/no3/sprite_banks.h` (a file
created by sotn-assets locally, which does not exist directly in this
repo), we have:
```
extern s16* sprites_no3_0;
extern s16* sprites_no3_1;
extern s16* sprites_no3_2;
extern s16* sprites_no3_3;
extern s16* sprites_no3_4;
extern s16* sprites_no3_5;
extern s16* sprites_no3_6;
extern s16* sprites_no3_7;
extern s16* sprites_no3_8;
static s16* spriteBanks[] = {
0,
&sprites_no3_0,
&sprites_no3_3,
&sprites_no3_1,
&sprites_no3_2,
&sprites_no3_4,
&sprites_no3_5,
&sprites_no3_6,
&sprites_no3_7,
&sprites_no3_8,
```
However, those are not the right pointer types. `sprites_no3_#` should
have a type of `s16* []`, so I reworked the sotn-assets tool to use the
right pointers for each of these things. Now the sprite_banks.h file
looks like:
```
extern s16* sprites_no3_0[];
extern s16* sprites_no3_1[];
extern s16* sprites_no3_2[];
extern s16* sprites_no3_3[];
extern s16* sprites_no3_4[];
extern s16* sprites_no3_5[];
extern s16* sprites_no3_6[];
extern s16* sprites_no3_7[];
extern s16* sprites_no3_8[];
static s16** spriteBanks[] = {
0,
sprites_no3_0,
sprites_no3_3,
sprites_no3_1,
sprites_no3_2,
sprites_no3_4,
sprites_no3_5,
sprites_no3_6,
sprites_no3_7,
sprites_no3_8,
```
Which more accurately represents the data within.
Naturally, since this PR mostly just changes the tool, the real changes
are in the tool's output, so feel free to clone this branch and
investigate the sprite_banks.h file if you want to see anything further
about it.
DRA's equivalent of the SpriteBanks array is `D_800A3B70`, and is
currently in the repo as extracted data, without using `sotn-assets`;
this PR makes it so the types in the overlays (which do use
`sotn-assets`) will match the types in DRA.
I did a find and replace for "unsigned short" to "u16" in the
sotn-assets `build.go` file and the same for "signed short" to "s16".
Then I went through and added common.h imports to fix each of the build
errors until it worked.
Using the numerical types is preferred due to being shorter (reducing
line wrapping), and to be future-proof for any systems where the "short"
type might not be 16 bits.
This adds this overlay to the project.
Importantly, we use a new tool (tools/auto_dedupe_new_overlay) to
process the new overlay and automatically split its C files according to
known duplicate files. As of now, this does not do any decompiling (the
whole overlay is kept as INCLUDE_ASM), but it automatically does all the
file splits and code copying needed. Hopefully this reduces some amount
of duplicated work.
To be clear, this new script is not in any build chain, but is meant to
be used after `make-config` to take the overlay from being a single
giant lump of `us.c` to being individual files, for the sake of easier
deduplicating. It will likely need some revision for future overlays,
but at least it should be a good start toward reducing tedious work.
I'll leave this overlay in place like this for about a week in case
there are any newcomers who would like to try de-duplicating some of
these files and decompiling the new functions; after that week I'll get
into doing things myself.
More research on how cutscenes work.
I normalized all the various C files as `cutscene.c`, marked all the
isolated function as `static` and renamed the main entity as
`{STAGE}_CutsceneExec` (e.g. `CEN_CutsceneExec`). I am using the
`OVL_EXPORT` to automate the names.
TO-DO:
- [x] Rename entity as `{STAGE}_EntityCutscene` for consistency
- [x] CEN
- [x] DRE
- [x] NO3
- [x] NZ0
- [x] ST0
- [x] MAR
~~SEL~~
The offset of the portrait data seems to be hardcoded. I have no idea
how to resolve these offsets at compilation time. The entire cutscene
script thing is very sketchy and horribly designed by the original
developers. What a nightmare to integrate into our project.
This is how a cutscene script gets decompiled:
```
LOAD_PORTRAIT(0x80188D8C, 0),
SET_PORTRAIT(1, 0),
SCRIPT_UNKNOWN_11(),
PLAY_SOUND(0x37B),
WAIT_FOR_SOUND(),
SET_SPEED(4),
'T','h','a','t',' ','v','o','i','c','e','!',' ',
SET_WAIT(16),
SET_SPEED(3),
'A','l','u','c','a','r','d',',',
LINE_BREAK(),
SET_WAIT(16),
SET_FLAG(2),
'i','t','\'','s',' ','y','o','u','!',
SET_WAIT(48),
NEXT_DIALOG(),
```
Had an idea, not sure if this is useful or not but I figured it might
help give an indication for newer people or those not in the Discord
loop if code is already duplicated when looking at the function report.
Here's an example output from the current state (minus the scratches
which didn't seem to load for me locally so I skipped them)
[functions.md](https://github.com/user-attachments/files/17162454/functions.md)
@bismurphy guessing this would be your area of expertise 😄
`CutsceneUnk2` uses 8 prims but only 7 are allocated. It looks like at
some point the game writes to the address `0x00000000` but I did not
confirm it with a PS1 debugger. This is a potential bug. `CutsceneUnk2`
signature was also wrong.
I had to get rid of `GfxBank` in `src/st/cen/header.c` and I probably
need to do the same for the other header files. The code reads chunks of
WORDS at the time and a structure would be too large to include
`GFX_TERMINATE` in it.
CEN is there but not linked because it conflicts with some WRP symbols.
I decided to take a much simpler approach compared to what I did with
WRP (which needs to be refactored later on).
There's a cutscene parser. The asset manager is exporting it to
`src/st/cen/cutscene_data.h` if you want to have a look. I know I am
still using `config/assets.us.weapon.yaml` and it probably needs to be
renamed as `config/assets.us.yaml` later on. I am still deciding.
Export the symbol list from the compiled elf and call `splat split
TARGET --disassemble-all`, so `symbols cross` can correctly
cross-reference the symbols.
Now there is also a warning when symbols cannot be cross-referenced
correctly.
cc. @hohle I noticed this issue on your stream
Before:
```
$ tools/make-config.py --version hd cen
✔ generating psx splat config
✔ splitting config/splat.hd.stcen.yaml
✔ adjusting files at src/st/cen
✔ disassembling matched functions
✔ finding duplicates across overlays
✔ adding cross-referenced function names
✔ renamed 99 functions, splitting again
✔ cross-referencing 0 functions
✔ adding cross-referenced symbol names
```
Now:
```
$ tools/make-config.py --version hd cen
✔ generating psx splat config
✔ splitting config/splat.hd.stcen.yaml
✔ adjusting files at src/st/cen
✔ disassembling matched functions
✔ finding duplicates across overlays
✔ adding cross-referenced function names
✔ renamed 99 functions, splitting again
✔ splitting config/splat.us.stcen.yaml 👈 NEW!
✔ cross-referencing 98 functions 👈 FIXED!
✔ adding cross-referenced symbol names
✔ renamed 56 data/bss symbols, splitting again 👈 56 instead of 0!
```
- Decompiled duplicate function func_us_801714F4
- Decompiled duplicate function func_us_80172904
- Decompiled duplicate function func_us_8017293C (renamed
Tt001UpdateAnim per tt_000 duplicate)
- Decompiled duplicate function func_us_80172B50
- Decompiled duplicate function func_us_80172B88
- Added EntitySearch D_80170EE4[] to servant.h to be used by
func_us_80172B88 similar to duplicate in tt_000
A lot of these are in the same order in tt_000, so I would assume they
could be moved to shared static functions at some point.
- `column` util was missing from the Docker image, added it and
organised the requirements file so it's more human friendly
- rearranged the USER argument in the Dockerfile to avoid the scenario
where you didnt specify the user and so lose the layers where apt
installed all the packages
- add the python deps at the end to ensure docker can reuse cached
layers if a python dep changes.
- update maspsx (I wanted to test latest maspsx against sotn, so this is
what led to the other changes!)
In analyze_calls, we need to analyze a function's calls, and be able to
track where they go. Given that a `jal` call only has a function name,
it's possible for it to go to a function in any overlay. Therefore, we
need to disambiguate across overlays.
We do this by identifying what overlay every function is in. This comes
from the filename of the assembly file.
Since the overlay might be in, for example, asm/us/dra (a "level 3
overlay" because it has 3 directory names) or asm/us/st/cen (a "level 4
overlay") we had special logic for testing for "st".
Now that we can also have level 4 overlays in "boss" (with Maria and BO3
coming up) we need to test for those too.
While we're at it, we also only run on the "us" version's assembly
files.
Fixed a tab matching issue with the `check` target. These were matching
'\' and 't' as separator characters rather than '↹' (horizontal tab)
Made `tools/make-config.py` executable.
Followed [the
guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay)
which seemed to work relatively painlessly for this overlay.
I hit an issue in `make-config.py` where no duplicates were found and
have added a separate commit which fixed that. Otherwise no issues with
guide.
```
josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001
✔ generating psx splat config
✔ splitting config/splat.us.tt_001.yaml
✔ adjusting files at src/servant/tt_001
✔ disassembling matched functions
✖ finding duplicates across overlays
Traceback (most recent call last):
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module>
raise e
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module>
make_config(args.input, args.version)
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config
if found > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
```
I was also able to remove the weird `goto block` on `RicMain`.
The HD version seems to be an older version. I am not entirely sure of
the US changes, but it seem to fix a series of bugs when Richter gets
grabbed by Gaibon? There's a new debug string which ints
`g_Player.unk7A` being named as `run_disable_f` but I did not yet rename
it.
Fixes a few issues found by @bismurphy where certain symbols that were
not supposed to be added, were added.
The major fix is when the second pass found a meaningful symbol name
like `MakeExplosion`, the third pass could add a new symbol with the
same offset but with the default splat name `func_8019055C`. With the
new `add_symbol_unique` the symbol table will stay unique. On top of
that when adding a new symbol `func_8019055C` can be replaced by
`MakeExplosion` but not the other way around. This will ensure only
meaningful names will be resolved.
Improve `make-config.py` to internally use the `symbols.py cross` and
cross-reference symbols from duplicate functions. This allows to add
even more symbols for new overlays with a third pass.
This was one of my biggest pain points when I was de-duplicating the
boss `MAR` overlay as I was not able to directly copy&paste duplicate
functions or include shared ones without having to individually fix the
symbols coming from `.data` and `.bss`. This is now automated.
```
$ python3 ./tools/make-config.py ric --version hd
✔ generating psx splat config
✔ splitting config/splat.hd.ric.yaml
✔ adjusting files at src/ric
✔ disassembling matched functions
✔ finding duplicates across overlays
✔ adding cross-referenced function names
✔ renamed 141 functions, splitting again
✔ cross-referencing 141 functions
✔ adding cross-referenced symbol names
✔ renamed 145 data/bss symbols, splitting again
```
I found `estimate_gnu_c_function_begin` might not be reliable all the
times. This happened with RIC HD. I found a way to circumnavigate the
issue for the time being.
Adds a spinner used as an indicator that the tool is working. I broke
down the duplicate function finder into multiple steps to give faster
feedback to the user.
I am now checking the duplicates only between overlays that make sense
to search to. Splitting `ric` will only cross-reference it with `ric
us`. Same for the servants, stages and the `sel` stage. On my machine
this change trimmed down the execution time from 1m25s to just 10s.
The spinner is very easy to use. Just use `spinner_start` instead of
`print`. Use `spinner_stop(True)` to terminate it and go to a new line,
but a new `spinner_start` will do that for you. Use
`spinner_stop(False)` when an operation fails.
This can be merged independently from #1643 and #1644.
I was experimenting with `python3 tools/symbols.py cross
asm/us/ric/matchings/1AC60/RicInit.s
asm/hd/ric/nonmatchings/hd/RicInit.s` and I found the tool was giving no
matches.
With this fix the output produced is now:
```
PLAYER_rotY = 0x800733F4;
PLAYER_rotX = 0x800733F2;
ric_anim_stand_relax = 0x80155480;
g_IsPrologueStage = 0x80154570;
D_80175958 = 0x801758AC;
D_801759D8 = 0x8017592C;
D_801530AC = 0x801530AC;
D_80153AA0 = 0x80153AA0;
D_80153D24 = 0x80153D24;
D_801541A8 = 0x801541A8;
```
Without this patch, the tool was looking at
`disks/hd/PSP_GAME/USRDIR/res/ps/hdbin/ric.bin`, which does not exist.
Now it is possible to successfully create configs for HD overlays.
Ubuntu/Debian and Python recommend using virtual environments for
project-specific Python dependencies and as of Ubuntu 24 this is lightly
enforced when installing packages via pip. This is due to pip and the
system package manager installing files to the same location which may
cause either's internal state to no longer reflect what is actually
installed.
This updates the project to use a Python `virtualenv` for project
dependencies and updates internal scripts to support both global and
virtualenvs, but favors virtualenvs for new workspaces.
All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the
first `python3` in the path. For those with a virtualenv configured,
this will be the Python managed there. For everyone else, this should be
the system Python or whatever other scheme they may have used previously
(assuming `python3` already existed in their `PATH`).
The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and
use `python3` from there if it exists and has been configured. It also
has a new `python-dependencies` target which will configure the venv and
install all python dependnecies there.
The `Dockerfile` has been updated to create an external `.venv` outside
of the project directory. Python's `virtualenv`s are not relocatable and
may hardcode paths which are mounted differently in the container and
host. To deal with differences in paths between the container (which
mounts the host's project directory to `/sotn`) host which may be at an
arbitarary directory the `VENV_PATH` environment variable is used to
override paths in the `Makefile`.
GitHub workflows have been updated to use `.venv`.
PSX: https://decomp.me/scratch/RL97B
PSP: https://decomp.me/scratch/3WKmX
`func_us_8018B4A0` needed to be decompiled before importing all the
assets due to GCC compiling `PfnEntityUpdates[entity->entityId - 1]`
into a hardcoded offset landing to the `header.c` data as a fake symbol.
The decompiled function seem to be some kind of object spawner for the
cutscenes controller. I have yet to confirm that, so I did not rename
any symbols yet.
EDIT: `func_us_8018B4A0` turned out to be a shared function I renamed as
`CutsceneRun`. DRE and CEN use slightly different variants and they will
not use the new `cutscene.h`. I plan on putting more stuff into
`cutscene.h` in a separate PR and maybe rename all the C files that
handle cutscenes accordingly.
As promised, a new tool to automatically generate a splat config for PSP
and PSX overlays for multiple game versions. This first version will
perform the following tasks:
* Create the splat config with data, rodata, text and bss sections to
their right offsets
* Generate a symbol list by cross-referencing duplicate symbols
* Split the overlay by creating the correspondent C file
This tool is not currently:
* Splitting C files
* Importing shared code already decompiled
There is a lot of tech involved, especially to detect the various
sections for the PSX overlays. But I will leave it to you to discover
how the implementation work if you're interested. Hopefully the code is
self-documented enough.
Usage examples:
* `python3 tools/make-config.py mar`
* `VERSION=jp12 python3 tools/make-config.py no2`
* `python3 tools/make-config.py tt_001 --version pspko`
With this tool I aim to add in the project multiple game versions of
TT_001, MAR and RBO3 from #557 and NO0.
`sotn-lint` will now convert `flags` and `drawFlags` fields from
integers to enum values.
`sotn-lint` will no longer overwrite files that haven't changed, so
running `make format build` only has to rebuild files that are modified.
The logic from `DrawModeTransformer` has been pulled out into
`BitFlagLineTransformer` which will handle any type of bitmasked enum
type.
A few updates that didn't make it into the first round:
* masks are now identified and expanded
* AND-EQUALS and OR-EQUALS are supported
* `sotn-lint` is run before `clang-format` to avoid having to run the
`format-src` target twice to catch line breaks
* `.h` files are included in `sotn-lint` since they are being used for
shared code
* `sotn` lint ignores the `mednafen` directory
Adds a transformer for `drawMode` fields which converts hex or integer
values to an OR'd bitmask.
Like `RelicTransformer`, `DrawModeTransformer` reproduces the values and
names in the linter.
From the [public
announcement](https://discord.com/channels/1079389589950705684/1079395108501331990/1276310695964835910)
on Discord:
> We've been progressing very far with the project and I started
wondering what would be the best course of action to protect the work of
existing and future contributors. I personally consider contributions to
a decompilation project as creative work. Therefore it is my desire to
protect this creative work from being re-claimed by other people. This
is especially true in the light of the ability of modding the existing
decompiled code and create derivative work.
>
> I talked privately with the current most active contributors. And me,
@sozud , @bismurphy and @joshlory so far agreed on pushing through the
AGPLv3 license for all the decompiled code in `sotn-decomp/src`,
excluding any third party code and decompiled code from the PSX SDK. But
I've been wanting to ask all of you if you are fine your code to be
licensed and protected under the AGPLv3 terms.
>
>
https://raw.githubusercontent.com/Xeeynamo/sotn-decomp/89686514916cabd0dd88ae0387f749a889c19e05/src/LICENSE
>
> I put and all the contributors, which can be accessed to the page
https://github.com/Xeeynamo/sotn-decomp/graphs/contributors . Again,
this will work for existing and future contributors. In the license I
put my full name as I will be responsible of any legality. And I
understand not every contributor wants to disclose their identity.
Please give an approval to my latest PR if you do not have any
objection. If not, please write here publicly.
>
> PS. I understand past contributors will either not see the message
above or decide to ignore it. If I do not hear from anyone of the
contributors for more than one week between those who did not interact
with me or the server, I will still go ahead and apply the license. It
is not in my intention to ignore past contributors. But I do not want
contributors who have been inactive for months or years to be a blocker.
Please do not mistake this as an act of malevolence.
---------
Co-authored-by: Joey Murphy <tjmurphy@mit.edu>
Previous system directly replicated the game's behavior, I realized it
could be simplified way down and not need to generate the full CLUT
table for each fetch. Now we translate between clut number and its
location in VRAM to fetch just the given CLUT. Comments describe the
logic in a way that is hopefully easy to follow.
