Previous system directly replicated the game's behavior, I realized it
could be simplified way down and not need to generate the full CLUT
table for each fetch. Now we translate between clut number and its
location in VRAM to fetch just the given CLUT. Comments describe the
logic in a way that is hopefully easy to follow.
Should have thought of this to begin with!
This makes the script fetch the VRAM while PCSX is running. That way you
don't have to deal with keeping files around on your hard drive if you
don't want to.
A tool I made that allows you to identify some entities based on the
primitives that they create. Usage is documented at top of the file.
Allows extraction of images, which has already been useful in a few
places.
Happy to provide any assistance with using this; it's come in handy for
me and hopefully will be beneficial for others too.
In addition to decompiling the function, two big changes were needed:
- Decompiling this function uncovered a maspsx bug with generating `nop`
padding instructions; that is now fixed and this PR includes a maspsx
update.
- We also discovered (thanks @mkst !) that g_Tilemap.bg was not actually
a member of g_Tilemap. I renamed it to g_BgLayers, and did a
find-and-replace across the repo to change all references. This appears
to have no impact on any existing function, but leads to the correct
register loading on this function.
It's a weird one, and was tricky to get matching (actually, this was a
super old decomp.me browser tab I discovered was still open, which is
why I came back to it), but these tricky ones are great for discovering
where we have mistakes in our structure of the game's data.
Allow to compile weapon overlays as a single C files. The benefit is to
disallow symbols to be exported by statically compile them all but the
header. We would no longer need to use `OVL_EXPORT` as long as the
symbols are marked as `static`. The frames data is also generated as an
array of data that can be just included into their respective C files,
generated by JSON files on the fly.
I changed `animset` to `frameset` as the data does not represent a set
of animations but rather a set of frames, where every frame has multiple
sprites. The information on how the frames are played (e.g. animation)
is found elsewhere.
Following the full list of changes to achieve this project:
* The `animset` from splat is now dummied out
* The `frameset` JSON is now generated by the asset manager
* The `frameset` JSON is now compiled into `src/weapon/w_0xx_y.h`
* The `header.c` has been moved into `shared.h`
* For simplicity I added a `#define g_Animset w_000_1`. I did not want
to hack the asset manager too much to force the name to be `g_Animset`
and I did not want to change all the symbols yet.
* Simplified `weapon_pc.c` as we now only need the exported header
* Always try to build the most up-to-date asset manager: this should
avoid weird errors whenever people pull new changes
As the asset manager now can accept those `config` files, now the CLI
supports the following commands:
* `stage extract`
* `stage build`
* `config extract` (NEW)
* `config build` (NEW)
The config structure is heavily inspired to Splat to maintain
consistency and continuity.
Currently function_finder misses functions since we don't continue to
fetch if there's another page of results. However just increasing the
page size makes the script super slow.
This adds caches to the zip file and result fetching to try and get this
running at a reasonable speed for `weapon`. We also fetch until `next`
is null and increase the page size so all the results are fetched. This
runs about 12 minutes on my system now. Results look like this
https://gist.github.com/sozud/69aeafcc671d6354da474db952e8afef
The duplicates report has several false negatives when a function has
the same name as another function which has not been decompiled. This
affects overlays which share many of the same function names (e.g.
`EntityWeaponAttack`).
The duplicates tool now parses the `INCLUDE_ASM` macro to extract the
path and ASM file. It then uses that when determining whether or not
each ASM file is actually included or not. Previously, only the function
name was checked, so if any `INCLUDE_ASM` file had the target function
name, it would be marked as not decompiled.
Before:
% | Decomp? | Name | Asm Path
-----|-------|-------------------------------|--------------------------------
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_008/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_009/EntityWeaponAttack.s
| 0.91 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_010/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_011/EntityWeaponAttack.s
| 0.98 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_025/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_026/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_027/EntityWeaponAttack.s
| 1.00 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_028/EntityWeaponAttack.s
| 0.97 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_058/EntityWeaponAttack.s
After:
% | Decomp? | Name | Asm Path
-----|-------|-------------------------------|--------------------------------
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_008/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_009/EntityWeaponAttack.s
0.91 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_010/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_011/EntityWeaponAttack.s
0.98 | false | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_025/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_026/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_027/EntityWeaponAttack.s
1.00 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_028/EntityWeaponAttack.s
0.97 | true | EntityWeaponAttack |
../../asm/us/weapon/nonmatchings/w_058/EntityWeaponAttack.s
(note: `w_008`, `w_009`, and `w_011` are decompiled in my workspace, but
not in GH)
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
Now that @sozud has added function similarity, I wonder if it's worth
including prefix matches in the results of function_finder. This would
let us name a scratch ex: `EntityWeaponAttack_w_053` and have it still
show up in functions.md. Thoughts?
This changes the psx function finder to compare the local and remote asm
to find a match. Right now there's a lot of false positives due to name
collisions. The new output looks like this:
https://gist.github.com/sozud/4eaa80c2ebb475551986b4f55d42d036
This should be off by default for the other platforms but I can't check
saturn locally right now.
Iterating on this problem in master.
Now that wrp/rwrp are sharing so much code, analyze_calls is having
trouble telling what functions call each other in which overlay.
I think this will resolve all the issues we have for now, but I'm sure
more will appear over time.
This will decompile all the functions but the new `EntityRWarpRoom` from
the RWRP overlay.
Note that as `e_misc.c` has a function with a different castle flag, I
had to use a shared header. Everything else was identical.
---------
Co-authored-by: bismurphy <tjmurphy@mit.edu>
Mostly identical to MAD and PSP. There are four very different functions
from US that needs to be decompiled to be 100% complete:
* [x] `e_misc.c` `EntityMessageBox`
* [x] `e_misc.c` `EntityRelicOrb`
* [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist
* [x] `e_stage_name.c` `EntityStageNamePopup`
These four very different functions are the only ones that use the JP
text instead of the US one. This overlay also lacks of `BlitChar`.
`EntityStageNamePopup` is very similar to the PSP counterpart. I used
@joshlory scratch to match the HD part.
`EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD
overlay). I found some code that crossed between BETA and HD, so I just
put an `#else` after `VERSION_US`. We are starting to reconstruct how
the source code originally evolved across different game builds.
Now with the maspsx changes, this can also match by using
SPRITESHEET_PTR.
Tried to document all the variables, use flags when possible, etc, so
hopefully this is mostly complete. Should be exciting to have
RenderEntities and RenderPrimitives both working now!
This aims to deprecate all the Splat tools in `tools/splat_ext` in
favour of a more centralised asset manager. This brings the following
advantages:
* Much faster extraction
* Faster build
* Automatically define `static` symbols or unique names whenever
`static` is not possible
* Allow to embed assets into the output binary
* Drastically simplify `Makefile` by removing all the asset build rules
* Avoid situations where it is not possible to extract and build assets
that is not 4-byte aligned
This is achieved by having the splat YAML targeting a normal C file as
data and have an external tool to take care of the following:
1. Extract asset files straight from the overlay binary file into human
readable file in `assets/st/STAGE_NAME`
2. Build assets as header files that go into `src/st/STAGE_NAME` to just
include them from any C file
This requires each stage header to have the following new format: please
see `src/st/nz0/header.c`
Built assets in `src/st` are ignored by Git.
As for now, for simplicity sake, the steps `make extract_assets` and
`make build_assets` are just executed within `make extract` exclusively
for the US version.
I plan to auto-generate files such as `src/st/nz0/tile_data.c`.
For a first iteration I am aiming to handle the following:
* [X] Extract rooms: `assets/st/*/rooms.json`
* [X] Extract room layers: `assets/st/*/entity_layouts.json`
* [X] Extract tilemap data: `assets/st/*/tilemap_*.bin`
* [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json`
* [X] Extract sprites: `assets/st/*/sprites.json`
* [x] Extract entity layouts
* [X] Build rooms: `src/st/*/rooms.h`
* [X] Build room layers: `src/st/*/layers.h`
* [X] Build tilemap data: `src/st/*/tilemap_*.h`
* [X] Build tilemap definitions: `src/st/*/tiledef_*.h`
* [x] Build sprites (aka `g_SpriteBanks`)
* [x] Build entity layouts
* [x] Allow the tool to suggest how to adjust the Splat config for each
overlay
I want the tool to cover the following stages:
* [x] CEN
* [x] DRE
* ~MAD~ I do not think this can be done, it is way too different from
the other overlays
* [x] NO3
* [x] NP3
* [X] NZ0
* [x] ST0
* [X] WRP
* [x] RWRP
* ~WRP (PSP)~ Maybe in a follow-up PR
For a later iteration I plan on extracting and build:
* Entity GFX thingie
* The CLUT thingie in the header
* Uncompressed GFX data
* Cutscene data
* Blueprints
* The `src/config_us.h` thingie
---------
Co-authored-by: Josh Lory <josh.lory@outlook.com>
`sotn_str` can now handle escaped quotes, right parenthesis, and more in
in `_S` strings. This allows all of the strings in `dra/menu.c` to be
encoded as static C-strings.
This also converts the spell button sequences to UTF-8.
---------
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
We have now fully decompiled every function which had a DECOMP_ME_WIP
comment, and we do not expect to ever have a need to add one in the
future.
Therefore, the function finder can be simplified by removing the process
of searching for these.
With no DECOMP_ME_WIP strings, `get_c_files` will never append anything
to `files`, so it always returns an empty list. We can therefore remove
the function.
With `c_files` always being an empty list, iterating over it will do
nothing, so we can remove `c_files` and the entire block of `for c_file
in c_files`.
Those are all the changes being made here, pretty simple.
This took me a lot of manual work. But I think I confirmed a pattern
that will help me to automate all of this for all the next stage
overlays that will be imported in the repo.
I noticed some stages with only one room having more than two layers or
more than two tile definitions, it might be either debugging or unused
content? I did not have time to explore any of that.
I modified the `tiledef` splat extension to greatly minimise the set-up
and noise.
Still work in progress. I removed splat as a submodule and started using
it as a pip package instead. Everything is matching but the memory card
icons part in both DRA and SEL. I still have no idea what the issue is.
Once this PR is good to be merged, we can get rid of the splat fork too.
I had to cherry-pick some existing changes from another branch, but that
part is tested and working.
For the blueprint stuff, I changed the fields from the strings `"TRUE"`
and `"FALSE"` to native JSON boolean values `true` and `false`. That
made things much easier to parse.
The parsing mechanic is straightforward. I'd like to have @bismurphy
review on the blueprint parser and asset changes specifically.
Secondary function for the Skull Shield. Gets spawned by the ShieldSpell
that just got merged.
I also noticed too late that my previous PR reverted the update to
mwccgap since I hadn't updated that; this PR includes a fix to that
accidental revert.
This one was a beast, but nice to get it working.
This seemed to match the same entity extension as weapon 29. Weapons 28
and 29 are both shields, so I decided to rename the Weapon29 struct to
be Shield instead. We will see if it continues to match for the future
shields.
This had a ton of FIX(), enums, #defines, etc, so hopefully I got them
all :)
Made a small tweak to mwccgap so that we don't compile a .c file twice
if we don't need to... compiling psp used to take 2.7s and now takes
2.0s on my laptop - so not super noticeable at this point, but makes
sense not to compile things twice!
Also added sha1sum for the wrp.bin to avoid people copying in the wrong
file and having a bad time (dont know why I didnt catch that pspeu means
psp eu.. not psp us!)
When using the `dec` script, it is possible to select a function where
there are multiple functions sharing a name. The test case I had for
this was `EntityEquipItemDrop`. If you run `dec EntityEquipItemDrop`, it
will tell you that there are 4 occurrences of functions with that name,
and list the four, telling you to re-run with -n or -f to specify which.
If you choose to specify option #0, with `dec EntityEquipItemDrop -n 0`,
then the number_occurence variable will be set to zero. But the
if-statement `if number_occurrence` fails, because the value is zero,
which is parsed as False. Decompilation therefore fails, as if you had
not passed `-n`.
This change explicitly tests for None, so that zero is a valid
selection.
Thanks to @sozud for the hint and @bismurphy for the refactoring idea.
The function `PreventEntityFromRespawning` on PSP hints that the struct
starts 8 bytes earlier. Also there's a missing `nop` at
`PreventEntityFromRespawning`, suggesting the function had to be moved
to the previous function.
I noticed that this struct had some overlap with other values in memory,
so I have pulled all those values into this struct.
The boundaries of this struct are uncertain and are a matter of ongoing
research.
OK. Here goes. Version 0.0.1 alpha of the MWCC Global Assembly Processor
(mwccgap). It's currently very simple/limited, but it appears to work
for `src/servant/tt_000/10E8.c`.
There is lot more that can be done to improve mwccgap - i.e. supporting
.rodata migration would be a good addition, but let's see how far we can
get with it in it's current state.
Note that the Makefile could do with some improvements - we don't nede
to use mwccgap for any C file that *dont* have INCLUDE_ASM macros (it's
a waste of time) so these could be ignored, i.e. for SSSV I do something
like this to find the files that need fixing up:
```
GLOBAL_ASM_C_FILES := $(shell $(GREP) GLOBAL_ASM $(SRC_DIR) </dev/null 2>/dev/null)
```
.. although this is perhaps too simple given that SOTN has a mix of PSP
and PSX functions (and therefore there may be INCLUDE_ASM for a PSX
function but none for PSP functions...
In CUE files `FILE` is a stateful, global declaration that applies to
all following `TRACK`s. `sotn-disk` was treating `FILE` declarations as
`TRACK` delimiters which would result in incorrect parsing of CUE files
with a single `FILE`, but multiple tracks.
Now when reading a `FILE` declaration, the path is stored and processing
continues. A `TRACK` declaration will use the previously defined path,
and if a previous track had been started, append that previous track to
the list.
This also builds `sotn-disk` using the local repository instead of
pulling the latest commit from GitHub. The target depends on `sotn-disk`
sources and will rebuild as necessary (or with `make
~/go/bin/sotn-disk`).
As an aside,
[pull/232](https://github.com/Xeeynamo/sotn-decomp/pull/232) ran into
this same error, but fixes in a slightly different way that leaves side
effects that may make supporting things like `INDEX` (for extracting the
placeholder audio, for example) more error prone in the future.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
Having `platform: psx` will not correctly disassemble PSP functions with
opcodes that are not part of the R3000 (the PSX CPU) architecture. These
functions would be disassembled by Splat as a bunch of `.word
0xBLAHBLAH`.
Changing the platform to `psp` introduces all sort of new challenges.
Function prototypes needs to be declared earlier. But also the MWCC
compiler will not accept the `%lo` and `%hi` dialect from GNU AS. There
were some patches on `mwcpp.py` to use `la SYMBOL_NAME` that would
expand into a `lui / addiu` combo. But even though symbols needs to be
declared like function prototypes at the top of the file. This is simply
not feasible on bigger overlays.
As a solution to the problem above, I replaced the existing patches by
converting instructions into `.word`. The overlay cannot longer be
relocated with this approach, but it is not an issue as the final goal
is to decompile these functions any way.
The labels in the jump table has the same problem, which forced me to
change the segment type from `rodata` to `data.
This is just another single step to create the conditions to start
including bigger re-compilable PSP overlays. I am sure the `mwcpp.py`
solution will be thrown into the bin at some point, but this PR improves
our current situation.
TT_000 is the first overlay from PlayStation 1 that we are now able to
compile from the source and produce a 1:1 binary. This lead me to start
exploring the same overlay from the game Castlevania: Dracula X
Chronicles, which contains a PSP re-build of Symphony of the Night.
This PR adds all the infrastructure to add the same flow for a PSP
matching decomp. Use `export VERSION=pspeu` and then the usual `sotn`
command to splat the overlay, build it and check if it matches. Running
`make extract_disk` should not be necessary as the same ISO used from
`VERSION=hd` is also used for `pspeu`, so you will probably have it
extracted already.
Some important notes about the PSP build:
* The whole PSP build seems to be compiled with `-O0`, which makes it
much easier to decompile
* Having ŧhe PSX, PSP and Saturn builds will allow to easily
cross-reference the code and reduce fake matches
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin` is the HD PSX
build
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PSPBIN/tt_000.bin` has the same
code from the HD build, but for PSP
* `disks/pspeu/PSP_GAME/USRDIR/res/ps/PACK_E/TP00.BIN` is the same as
above, but it packs both overlay and graphics. This is the file the PSP
game seems to actually use
* The PSP build uses the Metrowerk CodeWarrior's C compiler, which is
very different from the GCC one used on PSX.
* Thanks to @mkst lead, we found a way to still use the GNU assembler
and linker
* MWCC uses [wibo](https://github.com/decompals/WiBo/), a think
compatibility layer to run Windows CLI tools on Linux. It is much more
lightweight than Wine.
* MWCC does not support the `INCLUDE_ASM` dialect, so the custom
pre-processor `tools/mwcpp` had to be created
* The exact MWCC compiler version is unknown, but I suspect it is `build
147`
* I am not yet sure of any implications for using GNU AS and GNU LD
instead of the MW correspondent tools
* Not all the functions can be correctly disassembled, spimdisasm will
just produce a bunch of `.word 0x________` due to the in-progress effort
of adding the Allegrex-specific opcodes support
---
TO-DO list before marking the PR as ready:
- [X] Add PSP build to the CI
- [x] Add progress reporting to the PSP build
- [x] Integrate source file from `src/servant/tt_000_psp` to
`src/servant/tt_000` to promote the psp build as first-citizen
---
TO-DO in a follow-up PR:
* Figure out what `header` is: can we extract it as assembly code? or
maybe as custom re-compilable asset via splat? Is it a MW stuff or a
Castlevania-specific file?
* Get rid of the last line in `Makefile.psp.mk`
