`CutsceneUnk2` uses 8 prims but only 7 are allocated. It looks like at
some point the game writes to the address `0x00000000` but I did not
confirm it with a PS1 debugger. This is a potential bug. `CutsceneUnk2`
signature was also wrong.
I had to get rid of `GfxBank` in `src/st/cen/header.c` and I probably
need to do the same for the other header files. The code reads chunks of
WORDS at the time and a structure would be too large to include
`GFX_TERMINATE` in it.
CEN is there but not linked because it conflicts with some WRP symbols.
I decided to take a much simpler approach compared to what I did with
WRP (which needs to be refactored later on).
There's a cutscene parser. The asset manager is exporting it to
`src/st/cen/cutscene_data.h` if you want to have a look. I know I am
still using `config/assets.us.weapon.yaml` and it probably needs to be
renamed as `config/assets.us.yaml` later on. I am still deciding.
Export the symbol list from the compiled elf and call `splat split
TARGET --disassemble-all`, so `symbols cross` can correctly
cross-reference the symbols.
Now there is also a warning when symbols cannot be cross-referenced
correctly.
cc. @hohle I noticed this issue on your stream
Before:
```
$ tools/make-config.py --version hd cen
✔ generating psx splat config
✔ splitting config/splat.hd.stcen.yaml
✔ adjusting files at src/st/cen
✔ disassembling matched functions
✔ finding duplicates across overlays
✔ adding cross-referenced function names
✔ renamed 99 functions, splitting again
✔ cross-referencing 0 functions
✔ adding cross-referenced symbol names
```
Now:
```
$ tools/make-config.py --version hd cen
✔ generating psx splat config
✔ splitting config/splat.hd.stcen.yaml
✔ adjusting files at src/st/cen
✔ disassembling matched functions
✔ finding duplicates across overlays
✔ adding cross-referenced function names
✔ renamed 99 functions, splitting again
✔ splitting config/splat.us.stcen.yaml 👈 NEW!
✔ cross-referencing 98 functions 👈 FIXED!
✔ adding cross-referenced symbol names
✔ renamed 56 data/bss symbols, splitting again 👈 56 instead of 0!
```
- Decompiled duplicate function func_us_801714F4
- Decompiled duplicate function func_us_80172904
- Decompiled duplicate function func_us_8017293C (renamed
Tt001UpdateAnim per tt_000 duplicate)
- Decompiled duplicate function func_us_80172B50
- Decompiled duplicate function func_us_80172B88
- Added EntitySearch D_80170EE4[] to servant.h to be used by
func_us_80172B88 similar to duplicate in tt_000
A lot of these are in the same order in tt_000, so I would assume they
could be moved to shared static functions at some point.
- `column` util was missing from the Docker image, added it and
organised the requirements file so it's more human friendly
- rearranged the USER argument in the Dockerfile to avoid the scenario
where you didnt specify the user and so lose the layers where apt
installed all the packages
- add the python deps at the end to ensure docker can reuse cached
layers if a python dep changes.
- update maspsx (I wanted to test latest maspsx against sotn, so this is
what led to the other changes!)
In analyze_calls, we need to analyze a function's calls, and be able to
track where they go. Given that a `jal` call only has a function name,
it's possible for it to go to a function in any overlay. Therefore, we
need to disambiguate across overlays.
We do this by identifying what overlay every function is in. This comes
from the filename of the assembly file.
Since the overlay might be in, for example, asm/us/dra (a "level 3
overlay" because it has 3 directory names) or asm/us/st/cen (a "level 4
overlay") we had special logic for testing for "st".
Now that we can also have level 4 overlays in "boss" (with Maria and BO3
coming up) we need to test for those too.
While we're at it, we also only run on the "us" version's assembly
files.
Fixed a tab matching issue with the `check` target. These were matching
'\' and 't' as separator characters rather than '↹' (horizontal tab)
Made `tools/make-config.py` executable.
Followed [the
guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay)
which seemed to work relatively painlessly for this overlay.
I hit an issue in `make-config.py` where no duplicates were found and
have added a separate commit which fixed that. Otherwise no issues with
guide.
```
josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001
✔ generating psx splat config
✔ splitting config/splat.us.tt_001.yaml
✔ adjusting files at src/servant/tt_001
✔ disassembling matched functions
✖ finding duplicates across overlays
Traceback (most recent call last):
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module>
raise e
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module>
make_config(args.input, args.version)
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config
if found > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
```
I was also able to remove the weird `goto block` on `RicMain`.
The HD version seems to be an older version. I am not entirely sure of
the US changes, but it seem to fix a series of bugs when Richter gets
grabbed by Gaibon? There's a new debug string which ints
`g_Player.unk7A` being named as `run_disable_f` but I did not yet rename
it.
Fixes a few issues found by @bismurphy where certain symbols that were
not supposed to be added, were added.
The major fix is when the second pass found a meaningful symbol name
like `MakeExplosion`, the third pass could add a new symbol with the
same offset but with the default splat name `func_8019055C`. With the
new `add_symbol_unique` the symbol table will stay unique. On top of
that when adding a new symbol `func_8019055C` can be replaced by
`MakeExplosion` but not the other way around. This will ensure only
meaningful names will be resolved.
Improve `make-config.py` to internally use the `symbols.py cross` and
cross-reference symbols from duplicate functions. This allows to add
even more symbols for new overlays with a third pass.
This was one of my biggest pain points when I was de-duplicating the
boss `MAR` overlay as I was not able to directly copy&paste duplicate
functions or include shared ones without having to individually fix the
symbols coming from `.data` and `.bss`. This is now automated.
```
$ python3 ./tools/make-config.py ric --version hd
✔ generating psx splat config
✔ splitting config/splat.hd.ric.yaml
✔ adjusting files at src/ric
✔ disassembling matched functions
✔ finding duplicates across overlays
✔ adding cross-referenced function names
✔ renamed 141 functions, splitting again
✔ cross-referencing 141 functions
✔ adding cross-referenced symbol names
✔ renamed 145 data/bss symbols, splitting again
```
I found `estimate_gnu_c_function_begin` might not be reliable all the
times. This happened with RIC HD. I found a way to circumnavigate the
issue for the time being.
Adds a spinner used as an indicator that the tool is working. I broke
down the duplicate function finder into multiple steps to give faster
feedback to the user.
I am now checking the duplicates only between overlays that make sense
to search to. Splitting `ric` will only cross-reference it with `ric
us`. Same for the servants, stages and the `sel` stage. On my machine
this change trimmed down the execution time from 1m25s to just 10s.
The spinner is very easy to use. Just use `spinner_start` instead of
`print`. Use `spinner_stop(True)` to terminate it and go to a new line,
but a new `spinner_start` will do that for you. Use
`spinner_stop(False)` when an operation fails.
This can be merged independently from #1643 and #1644.
I was experimenting with `python3 tools/symbols.py cross
asm/us/ric/matchings/1AC60/RicInit.s
asm/hd/ric/nonmatchings/hd/RicInit.s` and I found the tool was giving no
matches.
With this fix the output produced is now:
```
PLAYER_rotY = 0x800733F4;
PLAYER_rotX = 0x800733F2;
ric_anim_stand_relax = 0x80155480;
g_IsPrologueStage = 0x80154570;
D_80175958 = 0x801758AC;
D_801759D8 = 0x8017592C;
D_801530AC = 0x801530AC;
D_80153AA0 = 0x80153AA0;
D_80153D24 = 0x80153D24;
D_801541A8 = 0x801541A8;
```
Without this patch, the tool was looking at
`disks/hd/PSP_GAME/USRDIR/res/ps/hdbin/ric.bin`, which does not exist.
Now it is possible to successfully create configs for HD overlays.
Ubuntu/Debian and Python recommend using virtual environments for
project-specific Python dependencies and as of Ubuntu 24 this is lightly
enforced when installing packages via pip. This is due to pip and the
system package manager installing files to the same location which may
cause either's internal state to no longer reflect what is actually
installed.
This updates the project to use a Python `virtualenv` for project
dependencies and updates internal scripts to support both global and
virtualenvs, but favors virtualenvs for new workspaces.
All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the
first `python3` in the path. For those with a virtualenv configured,
this will be the Python managed there. For everyone else, this should be
the system Python or whatever other scheme they may have used previously
(assuming `python3` already existed in their `PATH`).
The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and
use `python3` from there if it exists and has been configured. It also
has a new `python-dependencies` target which will configure the venv and
install all python dependnecies there.
The `Dockerfile` has been updated to create an external `.venv` outside
of the project directory. Python's `virtualenv`s are not relocatable and
may hardcode paths which are mounted differently in the container and
host. To deal with differences in paths between the container (which
mounts the host's project directory to `/sotn`) host which may be at an
arbitarary directory the `VENV_PATH` environment variable is used to
override paths in the `Makefile`.
GitHub workflows have been updated to use `.venv`.
PSX: https://decomp.me/scratch/RL97B
PSP: https://decomp.me/scratch/3WKmX
`func_us_8018B4A0` needed to be decompiled before importing all the
assets due to GCC compiling `PfnEntityUpdates[entity->entityId - 1]`
into a hardcoded offset landing to the `header.c` data as a fake symbol.
The decompiled function seem to be some kind of object spawner for the
cutscenes controller. I have yet to confirm that, so I did not rename
any symbols yet.
EDIT: `func_us_8018B4A0` turned out to be a shared function I renamed as
`CutsceneRun`. DRE and CEN use slightly different variants and they will
not use the new `cutscene.h`. I plan on putting more stuff into
`cutscene.h` in a separate PR and maybe rename all the C files that
handle cutscenes accordingly.
As promised, a new tool to automatically generate a splat config for PSP
and PSX overlays for multiple game versions. This first version will
perform the following tasks:
* Create the splat config with data, rodata, text and bss sections to
their right offsets
* Generate a symbol list by cross-referencing duplicate symbols
* Split the overlay by creating the correspondent C file
This tool is not currently:
* Splitting C files
* Importing shared code already decompiled
There is a lot of tech involved, especially to detect the various
sections for the PSX overlays. But I will leave it to you to discover
how the implementation work if you're interested. Hopefully the code is
self-documented enough.
Usage examples:
* `python3 tools/make-config.py mar`
* `VERSION=jp12 python3 tools/make-config.py no2`
* `python3 tools/make-config.py tt_001 --version pspko`
With this tool I aim to add in the project multiple game versions of
TT_001, MAR and RBO3 from #557 and NO0.
`sotn-lint` will now convert `flags` and `drawFlags` fields from
integers to enum values.
`sotn-lint` will no longer overwrite files that haven't changed, so
running `make format build` only has to rebuild files that are modified.
The logic from `DrawModeTransformer` has been pulled out into
`BitFlagLineTransformer` which will handle any type of bitmasked enum
type.
A few updates that didn't make it into the first round:
* masks are now identified and expanded
* AND-EQUALS and OR-EQUALS are supported
* `sotn-lint` is run before `clang-format` to avoid having to run the
`format-src` target twice to catch line breaks
* `.h` files are included in `sotn-lint` since they are being used for
shared code
* `sotn` lint ignores the `mednafen` directory
Adds a transformer for `drawMode` fields which converts hex or integer
values to an OR'd bitmask.
Like `RelicTransformer`, `DrawModeTransformer` reproduces the values and
names in the linter.
From the [public
announcement](https://discord.com/channels/1079389589950705684/1079395108501331990/1276310695964835910)
on Discord:
> We've been progressing very far with the project and I started
wondering what would be the best course of action to protect the work of
existing and future contributors. I personally consider contributions to
a decompilation project as creative work. Therefore it is my desire to
protect this creative work from being re-claimed by other people. This
is especially true in the light of the ability of modding the existing
decompiled code and create derivative work.
>
> I talked privately with the current most active contributors. And me,
@sozud , @bismurphy and @joshlory so far agreed on pushing through the
AGPLv3 license for all the decompiled code in `sotn-decomp/src`,
excluding any third party code and decompiled code from the PSX SDK. But
I've been wanting to ask all of you if you are fine your code to be
licensed and protected under the AGPLv3 terms.
>
>
https://raw.githubusercontent.com/Xeeynamo/sotn-decomp/89686514916cabd0dd88ae0387f749a889c19e05/src/LICENSE
>
> I put and all the contributors, which can be accessed to the page
https://github.com/Xeeynamo/sotn-decomp/graphs/contributors . Again,
this will work for existing and future contributors. In the license I
put my full name as I will be responsible of any legality. And I
understand not every contributor wants to disclose their identity.
Please give an approval to my latest PR if you do not have any
objection. If not, please write here publicly.
>
> PS. I understand past contributors will either not see the message
above or decide to ignore it. If I do not hear from anyone of the
contributors for more than one week between those who did not interact
with me or the server, I will still go ahead and apply the license. It
is not in my intention to ignore past contributors. But I do not want
contributors who have been inactive for months or years to be a blocker.
Please do not mistake this as an act of malevolence.
---------
Co-authored-by: Joey Murphy <tjmurphy@mit.edu>
Previous system directly replicated the game's behavior, I realized it
could be simplified way down and not need to generate the full CLUT
table for each fetch. Now we translate between clut number and its
location in VRAM to fetch just the given CLUT. Comments describe the
logic in a way that is hopefully easy to follow.
Should have thought of this to begin with!
This makes the script fetch the VRAM while PCSX is running. That way you
don't have to deal with keeping files around on your hard drive if you
don't want to.
A tool I made that allows you to identify some entities based on the
primitives that they create. Usage is documented at top of the file.
Allows extraction of images, which has already been useful in a few
places.
Happy to provide any assistance with using this; it's come in handy for
me and hopefully will be beneficial for others too.
In addition to decompiling the function, two big changes were needed:
- Decompiling this function uncovered a maspsx bug with generating `nop`
padding instructions; that is now fixed and this PR includes a maspsx
update.
- We also discovered (thanks @mkst !) that g_Tilemap.bg was not actually
a member of g_Tilemap. I renamed it to g_BgLayers, and did a
find-and-replace across the repo to change all references. This appears
to have no impact on any existing function, but leads to the correct
register loading on this function.
It's a weird one, and was tricky to get matching (actually, this was a
super old decomp.me browser tab I discovered was still open, which is
why I came back to it), but these tricky ones are great for discovering
where we have mistakes in our structure of the game's data.
Allow to compile weapon overlays as a single C files. The benefit is to
disallow symbols to be exported by statically compile them all but the
header. We would no longer need to use `OVL_EXPORT` as long as the
symbols are marked as `static`. The frames data is also generated as an
array of data that can be just included into their respective C files,
generated by JSON files on the fly.
I changed `animset` to `frameset` as the data does not represent a set
of animations but rather a set of frames, where every frame has multiple
sprites. The information on how the frames are played (e.g. animation)
is found elsewhere.
Following the full list of changes to achieve this project:
* The `animset` from splat is now dummied out
* The `frameset` JSON is now generated by the asset manager
* The `frameset` JSON is now compiled into `src/weapon/w_0xx_y.h`
* The `header.c` has been moved into `shared.h`
* For simplicity I added a `#define g_Animset w_000_1`. I did not want
to hack the asset manager too much to force the name to be `g_Animset`
and I did not want to change all the symbols yet.
* Simplified `weapon_pc.c` as we now only need the exported header
* Always try to build the most up-to-date asset manager: this should
avoid weird errors whenever people pull new changes
As the asset manager now can accept those `config` files, now the CLI
supports the following commands:
* `stage extract`
* `stage build`
* `config extract` (NEW)
* `config build` (NEW)
The config structure is heavily inspired to Splat to maintain
consistency and continuity.
Currently function_finder misses functions since we don't continue to
fetch if there's another page of results. However just increasing the
page size makes the script super slow.
This adds caches to the zip file and result fetching to try and get this
running at a reasonable speed for `weapon`. We also fetch until `next`
is null and increase the page size so all the results are fetched. This
runs about 12 minutes on my system now. Results look like this
https://gist.github.com/sozud/69aeafcc671d6354da474db952e8afef
The duplicates report has several false negatives when a function has
the same name as another function which has not been decompiled. This
affects overlays which share many of the same function names (e.g.
`EntityWeaponAttack`).
The duplicates tool now parses the `INCLUDE_ASM` macro to extract the
path and ASM file. It then uses that when determining whether or not
each ASM file is actually included or not. Previously, only the function
name was checked, so if any `INCLUDE_ASM` file had the target function
name, it would be marked as not decompiled.
Before:
% | Decomp? | Name | Asm Path
-----|-------|-------------------------------|--------------------------------
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_008/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_009/EntityWeaponAttack.s
| 0.91 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_010/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_011/EntityWeaponAttack.s
| 0.98 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_025/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_026/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_027/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_028/EntityWeaponAttack.s
| 0.97 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_058/EntityWeaponAttack.s
After:
% | Decomp? | Name | Asm Path
-----|-------|-------------------------------|--------------------------------
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_008/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_009/EntityWeaponAttack.s
0.91 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_010/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_011/EntityWeaponAttack.s
0.98 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_025/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_026/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_027/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_028/EntityWeaponAttack.s
0.97 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_058/EntityWeaponAttack.s
(note: `w_008`, `w_009`, and `w_011` are decompiled in my workspace, but
not in GH)
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
Now that @sozud has added function similarity, I wonder if it's worth
including prefix matches in the results of function_finder. This would
let us name a scratch ex: `EntityWeaponAttack_w_053` and have it still
show up in functions.md. Thoughts?
This changes the psx function finder to compare the local and remote asm
to find a match. Right now there's a lot of false positives due to name
collisions. The new output looks like this:
https://gist.github.com/sozud/4eaa80c2ebb475551986b4f55d42d036
This should be off by default for the other platforms but I can't check
saturn locally right now.
Iterating on this problem in master.
Now that wrp/rwrp are sharing so much code, analyze_calls is having
trouble telling what functions call each other in which overlay.
I think this will resolve all the issues we have for now, but I'm sure
more will appear over time.
This will decompile all the functions but the new `EntityRWarpRoom` from
the RWRP overlay.
Note that as `e_misc.c` has a function with a different castle flag, I
had to use a shared header. Everything else was identical.
---------
Co-authored-by: bismurphy <tjmurphy@mit.edu>
Mostly identical to MAD and PSP. There are four very different functions
from US that needs to be decompiled to be 100% complete:
* [x] `e_misc.c` `EntityMessageBox`
* [x] `e_misc.c` `EntityRelicOrb`
* [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist
* [x] `e_stage_name.c` `EntityStageNamePopup`
These four very different functions are the only ones that use the JP
text instead of the US one. This overlay also lacks of `BlitChar`.
`EntityStageNamePopup` is very similar to the PSP counterpart. I used
@joshlory scratch to match the HD part.
`EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD
overlay). I found some code that crossed between BETA and HD, so I just
put an `#else` after `VERSION_US`. We are starting to reconstruct how
the source code originally evolved across different game builds.
Now with the maspsx changes, this can also match by using
SPRITESHEET_PTR.
Tried to document all the variables, use flags when possible, etc, so
hopefully this is mostly complete. Should be exciting to have
RenderEntities and RenderPrimitives both working now!
This aims to deprecate all the Splat tools in `tools/splat_ext` in
favour of a more centralised asset manager. This brings the following
advantages:
* Much faster extraction
* Faster build
* Automatically define `static` symbols or unique names whenever
`static` is not possible
* Allow to embed assets into the output binary
* Drastically simplify `Makefile` by removing all the asset build rules
* Avoid situations where it is not possible to extract and build assets
that is not 4-byte aligned
This is achieved by having the splat YAML targeting a normal C file as
data and have an external tool to take care of the following:
1. Extract asset files straight from the overlay binary file into human
readable file in `assets/st/STAGE_NAME`
2. Build assets as header files that go into `src/st/STAGE_NAME` to just
include them from any C file
This requires each stage header to have the following new format: please
see `src/st/nz0/header.c`
Built assets in `src/st` are ignored by Git.
As for now, for simplicity sake, the steps `make extract_assets` and
`make build_assets` are just executed within `make extract` exclusively
for the US version.
I plan to auto-generate files such as `src/st/nz0/tile_data.c`.
For a first iteration I am aiming to handle the following:
* [X] Extract rooms: `assets/st/*/rooms.json`
* [X] Extract room layers: `assets/st/*/entity_layouts.json`
* [X] Extract tilemap data: `assets/st/*/tilemap_*.bin`
* [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json`
* [X] Extract sprites: `assets/st/*/sprites.json`
* [x] Extract entity layouts
* [X] Build rooms: `src/st/*/rooms.h`
* [X] Build room layers: `src/st/*/layers.h`
* [X] Build tilemap data: `src/st/*/tilemap_*.h`
* [X] Build tilemap definitions: `src/st/*/tiledef_*.h`
* [x] Build sprites (aka `g_SpriteBanks`)
* [x] Build entity layouts
* [x] Allow the tool to suggest how to adjust the Splat config for each
overlay
I want the tool to cover the following stages:
* [x] CEN
* [x] DRE
* ~MAD~ I do not think this can be done, it is way too different from
the other overlays
* [x] NO3
* [x] NP3
* [X] NZ0
* [x] ST0
* [X] WRP
* [x] RWRP
* ~WRP (PSP)~ Maybe in a follow-up PR
For a later iteration I plan on extracting and build:
* Entity GFX thingie
* The CLUT thingie in the header
* Uncompressed GFX data
* Cutscene data
* Blueprints
* The `src/config_us.h` thingie
---------
Co-authored-by: Josh Lory <josh.lory@outlook.com>
`sotn_str` can now handle escaped quotes, right parenthesis, and more in
in `_S` strings. This allows all of the strings in `dra/menu.c` to be
encoded as static C-strings.
This also converts the spell button sequences to UTF-8.
---------
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
We have now fully decompiled every function which had a DECOMP_ME_WIP
comment, and we do not expect to ever have a need to add one in the
future.
Therefore, the function finder can be simplified by removing the process
of searching for these.
With no DECOMP_ME_WIP strings, `get_c_files` will never append anything
to `files`, so it always returns an empty list. We can therefore remove
the function.
With `c_files` always being an empty list, iterating over it will do
nothing, so we can remove `c_files` and the entire block of `for c_file
in c_files`.
Those are all the changes being made here, pretty simple.
This took me a lot of manual work. But I think I confirmed a pattern
that will help me to automate all of this for all the next stage
overlays that will be imported in the repo.
I noticed some stages with only one room having more than two layers or
more than two tile definitions, it might be either debugging or unused
content? I did not have time to explore any of that.
I modified the `tiledef` splat extension to greatly minimise the set-up
and noise.
Still work in progress. I removed splat as a submodule and started using
it as a pip package instead. Everything is matching but the memory card
icons part in both DRA and SEL. I still have no idea what the issue is.
Once this PR is good to be merged, we can get rid of the splat fork too.
I had to cherry-pick some existing changes from another branch, but that
part is tested and working.
For the blueprint stuff, I changed the fields from the strings `"TRUE"`
and `"FALSE"` to native JSON boolean values `true` and `false`. That
made things much easier to parse.
The parsing mechanic is straightforward. I'd like to have @bismurphy
review on the blueprint parser and asset changes specifically.
Secondary function for the Skull Shield. Gets spawned by the ShieldSpell
that just got merged.
I also noticed too late that my previous PR reverted the update to
mwccgap since I hadn't updated that; this PR includes a fix to that
accidental revert.
This one was a beast, but nice to get it working.
This seemed to match the same entity extension as weapon 29. Weapons 28
and 29 are both shields, so I decided to rename the Weapon29 struct to
be Shield instead. We will see if it continues to match for the future
shields.
This had a ton of FIX(), enums, #defines, etc, so hopefully I got them
all :)
Made a small tweak to mwccgap so that we don't compile a .c file twice
if we don't need to... compiling psp used to take 2.7s and now takes
2.0s on my laptop - so not super noticeable at this point, but makes
sense not to compile things twice!
Also added sha1sum for the wrp.bin to avoid people copying in the wrong
file and having a bad time (dont know why I didnt catch that pspeu means
psp eu.. not psp us!)
When using the `dec` script, it is possible to select a function where
there are multiple functions sharing a name. The test case I had for
this was `EntityEquipItemDrop`. If you run `dec EntityEquipItemDrop`, it
will tell you that there are 4 occurrences of functions with that name,
and list the four, telling you to re-run with -n or -f to specify which.
If you choose to specify option #0, with `dec EntityEquipItemDrop -n 0`,
then the number_occurence variable will be set to zero. But the
if-statement `if number_occurrence` fails, because the value is zero,
which is parsed as False. Decompilation therefore fails, as if you had
not passed `-n`.
This change explicitly tests for None, so that zero is a valid
selection.
Thanks to @sozud for the hint and @bismurphy for the refactoring idea.
The function `PreventEntityFromRespawning` on PSP hints that the struct
starts 8 bytes earlier. Also there's a missing `nop` at
`PreventEntityFromRespawning`, suggesting the function had to be moved
to the previous function.
I noticed that this struct had some overlap with other values in memory,
so I have pulled all those values into this struct.
The boundaries of this struct are uncertain and are a matter of ongoing
research.
OK. Here goes. Version 0.0.1 alpha of the MWCC Global Assembly Processor
(mwccgap). It's currently very simple/limited, but it appears to work
for `src/servant/tt_000/10E8.c`.
There is lot more that can be done to improve mwccgap - i.e. supporting
.rodata migration would be a good addition, but let's see how far we can
get with it in it's current state.
Note that the Makefile could do with some improvements - we don't nede
to use mwccgap for any C file that *dont* have INCLUDE_ASM macros (it's
a waste of time) so these could be ignored, i.e. for SSSV I do something
like this to find the files that need fixing up:
```
GLOBAL_ASM_C_FILES := $(shell $(GREP) GLOBAL_ASM $(SRC_DIR) </dev/null 2>/dev/null)
```
.. although this is perhaps too simple given that SOTN has a mix of PSP
and PSX functions (and therefore there may be INCLUDE_ASM for a PSX
function but none for PSP functions...
In CUE files `FILE` is a stateful, global declaration that applies to
all following `TRACK`s. `sotn-disk` was treating `FILE` declarations as
`TRACK` delimiters which would result in incorrect parsing of CUE files
with a single `FILE`, but multiple tracks.
Now when reading a `FILE` declaration, the path is stored and processing
continues. A `TRACK` declaration will use the previously defined path,
and if a previous track had been started, append that previous track to
the list.
This also builds `sotn-disk` using the local repository instead of
pulling the latest commit from GitHub. The target depends on `sotn-disk`
sources and will rebuild as necessary (or with `make
~/go/bin/sotn-disk`).
As an aside,
[pull/232](https://github.com/Xeeynamo/sotn-decomp/pull/232) ran into
this same error, but fixes in a slightly different way that leaves side
effects that may make supporting things like `INDEX` (for extracting the
placeholder audio, for example) more error prone in the future.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
Having `platform: psx` will not correctly disassemble PSP functions with
opcodes that are not part of the R3000 (the PSX CPU) architecture. These
functions would be disassembled by Splat as a bunch of `.word
0xBLAHBLAH`.
Changing the platform to `psp` introduces all sort of new challenges.
Function prototypes needs to be declared earlier. But also the MWCC
compiler will not accept the `%lo` and `%hi` dialect from GNU AS. There
were some patches on `mwcpp.py` to use `la SYMBOL_NAME` that would
expand into a `lui / addiu` combo. But even though symbols needs to be
declared like function prototypes at the top of the file. This is simply
not feasible on bigger overlays.
As a solution to the problem above, I replaced the existing patches by
converting instructions into `.word`. The overlay cannot longer be
relocated with this approach, but it is not an issue as the final goal
is to decompile these functions any way.
The labels in the jump table has the same problem, which forced me to
change the segment type from `rodata` to `data.
This is just another single step to create the conditions to start
including bigger re-compilable PSP overlays. I am sure the `mwcpp.py`
solution will be thrown into the bin at some point, but this PR improves
our current situation.
TT_000 is the first overlay from PlayStation 1 that we are now able to
compile from the source and produce a 1:1 binary. This lead me to start
exploring the same overlay from the game Castlevania: Dracula X
Chronicles, which contains a PSP re-build of Symphony of the Night.
This PR adds all the infrastructure to add the same flow for a PSP
matching decomp. Use `export VERSION=pspeu` and then the usual `sotn`
command to splat the overlay, build it and check if it matches. Running
`make extract_disk` should not be necessary as the same ISO used from
`VERSION=hd` is also used for `pspeu`, so you will probably have it
extracted already.
Some important notes about the PSP build:
* The whole PSP build seems to be compiled with `-O0`, which makes it
much easier to decompile
* Having ŧhe PSX, PSP and Saturn builds will allow to easily
cross-reference the code and reduce fake matches
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin` is the HD PSX
build
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PSPBIN/tt_000.bin` has the same
code from the HD build, but for PSP
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PACK_E/TP00.BIN` is the same as
above, but it packs both overlay and graphics. This is the file the PSP
game seems to actually use
* The PSP build uses the Metrowerk CodeWarrior's C compiler, which is
very different from the GCC one used on PSX.
* Thanks to @mkst lead, we found a way to still use the GNU assembler
and linker
* MWCC uses [wibo](https://github.com/decompals/WiBo/), a think
compatibility layer to run Windows CLI tools on Linux. It is much more
lightweight than Wine.
* MWCC does not support the `INCLUDE_ASM` dialect, so the custom
pre-processor `tools/mwcpp` had to be created
* The exact MWCC compiler version is unknown, but I suspect it is `build
147`
* I am not yet sure of any implications for using GNU AS and GNU LD
instead of the MW correspondent tools
* Not all the functions can be correctly disassembled, spimdisasm will
just produce a bunch of `.word 0x________` due to the in-progress effort
of adding the Allegrex-specific opcodes support
---
TO-DO list before marking the PR as ready:
- [X] Add PSP build to the CI
- [x] Add progress reporting to the PSP build
- [x] Integrate source file from `src/servant/tt_000_psp` to
`src/servant/tt_000` to promote the psp build as first-citizen
---
TO-DO in a follow-up PR:
* Figure out what `header` is: can we extract it as assembly code? or
maybe as custom re-compilable asset via splat? Is it a MW stuff or a
Castlevania-specific file?
* Get rid of the last line in `Makefile.psp.mk`
I don't have the frogress secrets etc. to fully test this but it seemed
to work on a dry run:
```
{'timestamp': 1707176381, 'git_hash': '5f3c92dc729020b7e8a6d7a98ac7549fc697089c', 'categories': {'code': {'main': 19812, 'main/total': 108268}, 'functions': {'main': 201, 'main/total': 534}}}
```
Another disappointing non-match, even closer than #1008 . There is only
one registry swap in one single line. I've been permuting for a day
without success. Putting this here if anyone wants to give it a try:
https://decomp.me/scratch/AuSvv
A side-project I've been working on that helped me understanding some
structures that are now a bit better documented in #1034 . Richter
sprites are extracted as usual. Doing `python3
tools/splat_ext/spritesheet.py decode disks/us/BIN/ARC_F.BIN assets/arc
alucard disks/us/BIN/F_GAME.BIN 1` can extract Alucard sprites too.
There is no repacking yet as I want to avoid using splat for this one.
This is a proof-of-concept for Japanese sotn_strs. The macro `_SJ` is
used to designate them. I only converted one string in config_jp to
serve as a test for the pipeline. I moved sotn_str to a folder and added
some tests. I renamed it since python doesn't like modules with - in the
name. The code could be cleaner and the conversion table could use some
more work but I think this is a reasonable start.
Mostly a proof of concept to extract resources from `BIN/F_GAME.BIN`.
There are tons of hacks to make this work:
* Tweaked png2s to decode binaries into PNGs
* Add a `config/gfx.game.json` that describes how to extract different
sprites out of a binary file
* Coded the palette location by reading the `clut` parameter in the
source code. A clut value of `0x196` corresponds to `"palette": "0x96"`
or `"palette": 150`.
* Coded the bits per pixel for a couple of 8-bit images
* Coded the palette location of those graphics that expects the palette
from DRA.BIN instead of F_GAME.BIN
* For all the undiscovered palette, I called the files `unk` and
defaulted their palette to grey
The expectation is that sprites will slowly be documented and added into
`config/gfx.game.json` by modders and enthusiasts. This does not yet
pack back these PNGs into the binary file. The way these assets are
extracted is probably not final or perfect, but we have to start from
somewhere.
Once decomp.me is updated I will update the SOTN preset to include
`--expand-li`. Let me know if you run into any issues but this builds an
🆗 binary for me.
As per title. I also added a new tool called `dirt_patch`. As I
mentioned in our Discord server there is some left-over data from
previous dev builds in DRA.BIN and potentially elsewhere too. The tool
uses the new file `config/dirt.us.json` which stores a list of patches
to avoid crazy hacks and `#ifdef` for the sake of getting a match. I
hope this tool will not be abused.
Extract the memory card icons out of DRA.BIN and SEL.BIN.
I also took the opportunity to extract `g_MemcardPortMask` into its own
source file `save_mgr_pre.c`. Unfortunately this is required to keep the
original data order. This suggests that most likely the icon palette and
bitmap were baked into the original `save_mgr.c` as byte arrays. I
decided to take a different approach and extracted them as PNGs for
better moddability.
I had to spin-off Splat into a new fork due to some breaking changes on
0.18.0.
After finding a copy of GetFreeDraEntity in RIC, I decided to rename the
function (in both DRA and RIC) to GetFreeEntity. Similarly, the function
right after it is GetFreeEntityReverse (since its logic runs in
reverse).
I also found CreateEntFactoryFromEntity in RIC, so I renamed that to
match the version in DRA. We will need to go through and use the FACTORY
macro for its uses, but I'm not going to tackle that quite yet.
Seems there is a lot of work to be done making RIC catch up to DRA,
we'll see if I keep working on that moving forward, it's neat to find
the ways the two match.
Another function, not sure what it really does besides messing with a
bunch of primitives.
Some clues in the rodata (particularly 00 00 00 00 bytes) indicated that
we should merge these C files, so I did.
An issue with maspsx was discovered in the process of decompiling this
one. I submitted a PR to maspsx which was accepted, and therefore I also
updated maspsx in this PR in order to capture the fix.
For some reason this was included as:
```
#if defined(VERSION_US)
INCLUDE_ASM("asm/us/dra/nonmatchings/627C4", func_801028AC);
#elif defined(VERSION_HD)
INCLUDE_ASM("asm/hd/dra/nonmatchings/627C4", func_801028AC);
#endif
```
But that wasn't actually needed, the functions are identical. Not sure
why it was that way but it's fixed now.
In Italian we say "Abbiamo fatto 30, facciamo 31". It literally means
"We come all the way up to 30, it wouldn't cost nothing to push a little
further for 31".
I moved `log.h` into `include/` to easily logging stuff elsewhere and
without weird `#ifdef VERSION_*` by adding `-DNO_LOGS` in the main
Makefile.
This is a bit hacky, but it does the trick. I had to expand `D_80138784`
otherwise I would get a segfault.
Both `SoundInit` and `func_801361F8` are called in the main, so I
removed them. To restore the previous behaviour and isolate the sound
engine, just replace `MainGame()` in the `main.c` with a `SoundInit()`
and then a loop that calls `UpdateGame()`.
Last, but not least, SDL2 is in.
Occasional maintenance:
* asm-differ: latest commit
* m2c: latest commit
* maspsx: latest commit
* spimdisasm: 0.18.0 latest
* splat: 0.17.3
The latest version of splat is 0.19.1 but a breaking change in
[0.18.0](https://github.com/ethteck/splat/pull/294) is preventing me to
upgrade further ([discussion
here](https://discord.com/channels/710646040331681844/813939516385525790/1172978921000669274))
Some YAML were malformed and since splat 0.17.0 there are additional
checks to ensure they are compliant. There are also new checks that
prevents a malformed symbol list including duplicates, which I fixed
too.
This has been one of the most weird functions I ever decompiled. I will
share a few tricks I learnt. In short, I tried inlining as much as
possible by removing all the temps.
---
```c
switch (self->step) {
...
case 2:
isEntityAlive = 0;
if (self->step != 2) {
return;
}
```
This was the output from M2C. As there is no way that `self->step` is
different than `2`, deleting had no effect on the matching.
---
```c
var_v0 = self->step;
switch (var_v0) {
case 1:
...
if (statement) {
self->step++;
}
...
D_SOME_VARIABLE = var_v0;
}
```
This was another very weird one. I couldn't understand why
`D_SOME_VARIABLE` was assigned that way much further down the `case 1`.
The way I fixed it is that `var_v0` was always `1` due to `case 1:`. By
doing `D_SOME_VARIABLE = 1` I got a match.
---
```c
temp_a0_2 = D_80174C2C + 1;
...
D_80174C2C = temp_a0_2 & -(temp_a0_2 < 0x10);
```
To understand this madness I used a random C compiler I found online and
tested in a `for` loop what's the output for all the given `temp_a0_2`.
It seemed the value never changed but over `16` the value was always 0.
I logically re-written that statement into something that made logically
more sense for me and it matched, even if it looks very different from
the original:
```c
D_80174C2C++;
D_80174C2C = D_80174C2C >= 16 ? 0 : D_80174C2C;
```
Fairly straightforward one here I think.
Unfortunately the subweapon name does not appear in the definition
(actually... are subweapon names in the game at all? I don't recall them
being in there, I think it's all just in the manual?) so it's hard to
map these definitions to the subweapons. Might be good to think about
whether we could insert comments somehow. Anyway, this works for now and
might help with collecting the subweapons in one place to understand the
remaining members of the definition.
