Minor follow-up to #1686 . I couldn't de-fake the g_Dialogue symbols in
ST0 like I did with `src/st/dre/bss.c` due some functions that are not
yet fully decompiled.
Various small cleanups on this function.
In the past it was EntityStrongWarg, but I found that this is actually
the Fire Warg.
It is unused in NO3, but is used in RNO3.
Also removed a Multi that was being used that applied across two
different entities.
Various comments, cleanups, renamings, etc
More research on how cutscenes work.
I normalized all the various C files as `cutscene.c`, marked all the
isolated function as `static` and renamed the main entity as
`{STAGE}_CutsceneExec` (e.g. `CEN_CutsceneExec`). I am using the
`OVL_EXPORT` to automate the names.
TO-DO:
- [x] Rename entity as `{STAGE}_EntityCutscene` for consistency
- [x] CEN
- [x] DRE
- [x] NO3
- [x] NZ0
- [x] ST0
- [x] MAR
~~SEL~~
The offset of the portrait data seems to be hardcoded. I have no idea
how to resolve these offsets at compilation time. The entire cutscene
script thing is very sketchy and horribly designed by the original
developers. What a nightmare to integrate into our project.
This is how a cutscene script gets decompiled:
```
LOAD_PORTRAIT(0x80188D8C, 0),
SET_PORTRAIT(1, 0),
SCRIPT_UNKNOWN_11(),
PLAY_SOUND(0x37B),
WAIT_FOR_SOUND(),
SET_SPEED(4),
'T','h','a','t',' ','v','o','i','c','e','!',' ',
SET_WAIT(16),
SET_SPEED(3),
'A','l','u','c','a','r','d',',',
LINE_BREAK(),
SET_WAIT(16),
SET_FLAG(2),
'i','t','\'','s',' ','y','o','u','!',
SET_WAIT(48),
NEXT_DIALOG(),
```
Several quality of life changes for building:
* The Makefile has a `help` target which will show common targets.
Targets with comments starting with `##@` will be included in the help
output.
* The `check` target now has colored output.
* Added `check_disk` target which will check hashes of extracted disk
contents (useful for those who dump their own discs)
* Added `dump_disk` target which will create a bin/cue pair from an
original disc.
Good amount of deduplication here.
The only issue I ran into is that NZ0 has slightly different data for
one array (it's missing two zeros at the end). I tried doing `#undef
STAGE #define STAGE STAGE_NZ0` in `nz0.h` and testing `#if STAGE ==
STAGE_NZ0`, but for some reason that was failing. I went with `#define
STAGE_IS_NZ0` and `#if !defined(STAGE_IS_NZ0)` for now; not sure what I
was doing wrong.
I noticed that there was large overlap between these, but it was
incomplete as different stages had different levels of de-duplication
completed.
I went through and completely de-duplicated this file, so it is now
st_common.h everywhere.
There were a few leftover functions in MAD and ST0, so I decompiled
those in my previous PRs, and now in this one, I'm doing ifdef to make
it all match for all stages.
This is pretty cool and represents a large amount of code reduction.
The original C file which was used as the basis for st_common.h was the
WRP one, which has PSP support, so this may be a great step to making
many overlays match on PSP too.
NO3 and NP3 had a weird mix of Bone Scimitar functions where each was
partially complete and missing different parts.
Now both are complete and modernized. Could use another pass to make it
PSP-matching, but for now I'll leave this one at this point where the
code seems nice.
Copied the `EnttityMerman` and `EntityMerman2` implementations from NP3
to NO3. These are identical, however, rely on too many things which
require renaming to share at this point.
NO3's `EntityMerman2` was previously `EntityMerman3`, but is identical
to NP3's Merman2, so it was renamed to match.
A small amount of cleanup was done to the implementations to bring them
closer to current standards.
This is a simple synchronization between methods already decompiled in
NP3 to their counterparts in NO3. Player water effect has been split to
match, but no data has been mapped in this pass.
Decompiles methods based on the NP3 implementation:
* `EntityBackgroundBushes`
* `func_801B94F0`
* `func_801D0A2C`
* `EntityAlucardWaterEffect`
* `EntityMermanSpawner`
* `EntityMermanExplosion`
Splits `e_bone_scimitar`, and `player_water_effect` into a separates
file to match `NP3`.
Maps `create_entity` and `e_collect` data, text, and bss for all stages.
Adds macros for defining padding for various sections by size.
Tables for section sizes have been added to common includes to make
calculating offsets easier when segments.
I did not do it for `weapon`. This is the script I used:
`python3 a.py asm/us/st/dre/data/23264.sbss.s > src/st/dre/bss.c`
```python
import sys
with open(sys.argv[1], "r") as f:
lines = f.readlines()
print('#include "common.h"')
print("")
for line in lines:
if line == "\n":
continue
elif line.startswith(".include"):
continue
elif line.startswith(".section"):
continue
elif line.startswith("glabel"):
label = line[7:].replace("\n", "")
len = 0
elif ".word" in line:
if len > 0 and n != 4:
print(f"WARN: {label}", file=sys.stderr)
n = 4
len += 1
elif ".short" in line:
if len > 0 and n != 2:
print(f"WARN: {label}", file=sys.stderr)
n = 2
len += 1
elif ".byte" in line:
if len > 0 and n != 1:
print(f"WARN: {label}", file=sys.stderr)
n = 1
len += 1
elif line.startswith(".size"):
if len == 1:
if n == 1:
print(f"u8 {label};")
elif n == 2:
print(f"u16 {label};")
elif n == 4:
print(f"u32 {label};")
else:
if n == 1:
print(f"u8 {label}[{len}];")
elif n == 2:
print(f"u16 {label}[{len}];")
elif n == 4:
print(f"u32 {label}[{len}];")
```
the script is a bit dumb. It does not account of the header. Some types
are wrong compared to their prototype. The memory layout matches though,
so we can keep iterating on top of this.
Start to perform a series of code refactoring to align the various
overlays on how NZ0, WRP and RWRP are currently organised, aiming to
share as much code as possible. This is a good stop gap that does not
change too much. There are virtually no major changes other than the
removal of the fake symbol `D_8003BEEC`.
I found it is very hard to share every single line of code due to some
minor differences in the castle flags. I couldn't yet spot a reliable
pattern on how castle flags are manager across stages.
My vision is to share as much code as possible, possibly not as header
files but as C files in a similar fashion of #1425 . I am aiming to
model NP3 so we can merge it with NO3 given the fact they share a good
chunk of the code base.
I am waiting to split the rest of the entities into their own individual
files. My idea is to have a file per entity or per entity group with
their own set of data and helper functions, possibly all marked as
`static`. We agreed it is probably a good idea to wait exploring the PSP
build of NZ0, NO3 or NP3 before performing this step.
This aims to deprecate all the Splat tools in `tools/splat_ext` in
favour of a more centralised asset manager. This brings the following
advantages:
* Much faster extraction
* Faster build
* Automatically define `static` symbols or unique names whenever
`static` is not possible
* Allow to embed assets into the output binary
* Drastically simplify `Makefile` by removing all the asset build rules
* Avoid situations where it is not possible to extract and build assets
that is not 4-byte aligned
This is achieved by having the splat YAML targeting a normal C file as
data and have an external tool to take care of the following:
1. Extract asset files straight from the overlay binary file into human
readable file in `assets/st/STAGE_NAME`
2. Build assets as header files that go into `src/st/STAGE_NAME` to just
include them from any C file
This requires each stage header to have the following new format: please
see `src/st/nz0/header.c`
Built assets in `src/st` are ignored by Git.
As for now, for simplicity sake, the steps `make extract_assets` and
`make build_assets` are just executed within `make extract` exclusively
for the US version.
I plan to auto-generate files such as `src/st/nz0/tile_data.c`.
For a first iteration I am aiming to handle the following:
* [X] Extract rooms: `assets/st/*/rooms.json`
* [X] Extract room layers: `assets/st/*/entity_layouts.json`
* [X] Extract tilemap data: `assets/st/*/tilemap_*.bin`
* [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json`
* [X] Extract sprites: `assets/st/*/sprites.json`
* [x] Extract entity layouts
* [X] Build rooms: `src/st/*/rooms.h`
* [X] Build room layers: `src/st/*/layers.h`
* [X] Build tilemap data: `src/st/*/tilemap_*.h`
* [X] Build tilemap definitions: `src/st/*/tiledef_*.h`
* [x] Build sprites (aka `g_SpriteBanks`)
* [x] Build entity layouts
* [x] Allow the tool to suggest how to adjust the Splat config for each
overlay
I want the tool to cover the following stages:
* [x] CEN
* [x] DRE
* ~MAD~ I do not think this can be done, it is way too different from
the other overlays
* [x] NO3
* [x] NP3
* [X] NZ0
* [x] ST0
* [X] WRP
* [x] RWRP
* ~WRP (PSP)~ Maybe in a follow-up PR
For a later iteration I plan on extracting and build:
* Entity GFX thingie
* The CLUT thingie in the header
* Uncompressed GFX data
* Cutscene data
* Blueprints
* The `src/config_us.h` thingie
---------
Co-authored-by: Josh Lory <josh.lory@outlook.com>
Decompiles stage header data for all in progress stages.
Of note is that NO3 and NP3 (and likely other future stages) use an
`AbbreviatedHeader` because their `SpriteBanks` start immediately after
`UpdateStageEntities`. There are several areas which use
`sizeof(Overlay)` to copy this data over, in those cases sprite bank
data is copied into the tail fields of the overlay but are never used.
This took me a lot of manual work. But I think I confirmed a pattern
that will help me to automate all of this for all the next stage
overlays that will be imported in the repo.
I noticed some stages with only one room having more than two layers or
more than two tile definitions, it might be either debugging or unused
content? I did not have time to explore any of that.
I modified the `tiledef` splat extension to greatly minimise the set-up
and noise.
Still work in progress. I removed splat as a submodule and started using
it as a pip package instead. Everything is matching but the memory card
icons part in both DRA and SEL. I still have no idea what the issue is.
Once this PR is good to be merged, we can get rid of the splat fork too.
As far as the duplicate-finder can tell, this is the last of the
cutscenes, but who knows - we may find more. Luckily, the pattern is
pretty recognizable.
Named it due to what we see at the very end in `case 7`, with the
TimeAttackController getting triggered.
More data imported into the C files. I reached this stopgap due to
`st_debug.c`, where I am failing to import the data from PSP. Strangely
enough, PSP suggests `st_debug`, `e_breakable` and even `warp` being in
the same file?!
The INIT section seems to follow the `PfnEntityUpdates` on both PSP and
PSX. But on PSX this is very close to the top while on PSP is right in
the middle.
Overall the PSX C and DATA order seem to align perfectly. PSP is far
more unpredictable, but the DATA order seem to also follow the C files.
The advantage of PSP is that almost everything is out of order compared
to PSX, which easily suggests file splits.
Another key difference between PSP and PSX is that arrays filled with
`0x00` are found in DATA on PSX and in BSS on PSP.
Given the fact that sharing compiled C objects is not exactly possible
(code heavily copy&pasted maybe? this initiative is now abandoned 👉53a566f075)
I decided to keep pressing forward with shared headers. Thanks a lot to
@hohle for making our life much easier by cross-referencing symbols.
The file split on WRP seems to be the closest file split we might have
compared to the original source code (still speculating here). I think
it would be a good idea to start splitting other overlays too with the
same approach.
My idea is to have a file split like the following:
```
st/
cen/
e_particles.c
e_misc.c
st_common.c
nz0/
e_particles.c
e_misc.c
st_common.c
wrp/
e_particles.c
e_misc.c
st_common.c
e_particles.h
e_misc.h
st_common.h
```
each of those C files will just be a one-line `#include
"../the_shared_code.h"` as usual. Right now we create individual headers
for single functions, sometimes for more than one function when we think
grouping makes sense. But I think we can start merging some of those
headers and consolidate the code. This can be done gradually. For
example `src/st/e_particles.h` is still importing function headers under
the hood. That is okay for now, but later on I wish to import those
headers functions into their respective parent headers.
Another important aspect to consider that will validate a correct file
split is to start importing the data inside these new C files. Right now
we have floating data such as `src/st/wrp_psp/wrp_data_EA00.c` or
monstrosities such as `src/st/wrp/6FD0.c`. An example of a (possibly)
correct migrated data is what this PR does with WRP PSP and NZ0 PSX,
with data pointing in `src/st/*/e_particles.c`.
Decompile and de-duplicate EntityStageNamePopup for all the overlays the
function is in.
I also detected a helper function I renamed as `PrimDecreaseBrightness`
that was originally accepting a completely wrong parameter type. This
helped me to get rid of `Unkstruct_80128BBC` too.
Follow-up to #797 by decompiling the MAD counterpart and share all the
functions within the same file. I renamed the function as
`UnkPrimHelper` as I do not know what it does.
Not sure what this one does, but it's in all the overlays.
I couldn't get `mad` to work, since it gives errors when I reference
functions like `RotMatrixZ` (which isn't even one of the new `gte_`
defines). I figured we can leave that for later since `mad` is its own
weird thing.
I also found an issue with the splat for `no3` where some boundaries
were being drawn in the wrong places; this is now corrected which allows
the function to work in that location.
The function takes a primitive as its argument and does some stuff with
it. I don't know what it actually does. It appears that the different
overlays use it in different places (and some overlays don't call it at
all). The function loads an external SVECTOR from a variable, which I
added to the `.h` file for each overlay. The vector is in a weird
location in some overlays, especially in no3 where it appears to be the
last few bytes of the rodata.
Anyway, happy to take naming ideas for either the function or the
SVECTOR that it loads. And I imagine we'll end up renaming those sp
variables.
We have a lot of casts and especially LOH calls in this one. Maybe we're
interpreting the primitive wrong. It doesn't match up with the FakePrim
we've been using lately, so I don't know what to make of this.
I think those are the main highlights! There aren't many functions that
use the GTE and it's nice that there is so much duplication so we can
deal with a lot of them all at once. I expect a lot of changes on this
one so please don't hold back :)
Here's the scratch for one instance of the function, in case you'd like
to play around with anything: https://decomp.me/scratch/N5RR5
Occasional maintenance:
* asm-differ: latest commit
* m2c: latest commit
* maspsx: latest commit
* spimdisasm: 0.18.0 latest
* splat: 0.17.3
The latest version of splat is 0.19.1 but a breaking change in
[0.18.0](https://github.com/ethteck/splat/pull/294) is preventing me to
upgrade further ([discussion
here](https://discord.com/channels/710646040331681844/813939516385525790/1172978921000669274))
Some YAML were malformed and since splat 0.17.0 there are additional
checks to ensure they are compliant. There are also new checks that
prevents a malformed symbol list including duplicates, which I fixed
too.
Occasional maintenance:
* asm-differ: latest commit
* m2c: latest commit
* maspsx: latest commit
* spimdisasm: 0.18.0 latest
* splat: 0.17.3
The latest version of splat is 0.19.1 but a breaking change in
[0.18.0](https://github.com/ethteck/splat/pull/294) is preventing me to
upgrade further ([discussion
here](https://discord.com/channels/710646040331681844/813939516385525790/1172978921000669274))
Some YAML were malformed and since splat 0.17.0 there are additional
checks to ensure they are compliant. There are also new checks that
prevents a malformed symbol list including duplicates, which I fixed
too.
Previously if either the YAML or the symbol list were changed, we were
forced to either manually delete the `asm` folder of the specific
overlay or to invoke `make clean`, where the latter forced us to
re-extract all the overlays via `make -j extract`.
That changes with this PR. How `make` works is `target: dep1 dep2` where
if one of the dependencies has the last modified date greater than the
target, the rule is triggered again. Previously we were extracting
overlays doing `make extract_stcen`. But since every overlay extraction
generates a linker script I now changed the rules to do `make
build/us/stcen.ld` to extract the same overlay. As I explained above, if
either the YAML or one of the related symbols changes, the linker
modified date will be older and the extraction for that overlay to be
trigger again.
The above allowed me to stop polluting the repo root with the linker
scripts as they are now moved into `build/us` or `build/hd` depending of
what you're trying to build. If you do `make extract` twice in a row you
will be welcomed with a `make: Nothing to be done for 'extract'.`.
There are still some instances where you **might** need `make clean`
beforehand, especially when modifying the `segment` section in the YAML
or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I
want to see if the current solution will be enough.
### Please try removing `make clean` from your workflow, once this is
merged, to quickly detect possible problems with this new approach.
I am also planning to make more substantial changes on our build-chain
like this PR or #660 in the future. I am aiming to small incremental
changes over time in case I break someone's flow or detect design flaws.
I am also considering the breaking changes introduced in make 4.4, which
will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a
deadline to finish all the new build-chain work.
---------
Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
Following up #454 , #453 and related from @sozud , I started to convert
some previously defined headers into proper shared C code. The advantage
from the header is the C file is completely independent, it will have
its own `.data` and `.rodata` and it will be compiled once.
I also took the opportunity to move `include/st/AnimateEntity.h` to
`src/st/animate_entity.h` for consistency. I think the change is small
enough to be included as part of this PR.
As I finished importing the data in WRP, I came with a possible new
pattern when de-duplicating functions. I kept using a header file
(easily indexable from VS Code) in `src/st`. But this time it gets
included by `entity_relic_orb.c`. All the sections `.data`, `.rodata`
and `.text` are migrated into `entity_relic_orb.c` but the file itself
just includes `src/st/entity_relic_orb.h`. Thinking forward, once we
will be able to fix the problem present in #464 it will just be a matter
of renaming `entity_relic_orb.h` into `entity_relic_orb.c` and remove
each individual `entity_relic_orb.c` that currently acts as a proxy.
~~This PR is a draft. If you agree this is a good pattern I will proceed
to do the same with the remaining overlays after #528 is good to be
merged.~~
This splits on TestCollisions, giving a block of the following 3
functions in a separate file and that are fully decompiled with the
exception of MAD.
```
#include "../random.h"
#include "../update.h"
#include "../update_stage_entities.h"
```
It should now be possible to change the build system to make that a C
file that is shared among the overlays.
The point of this PR is splitting overlays at the `Random()` function in
preparation for making it a C file eventually. After this I will do a
split at `Update()`. I have a script to partially automate this that I
will include in a separate PR.
I'm still not super confident decompiling jump table/rodata functions,
but I think I did this one right. Took a while fighting the splat to get
things to match.
Two things of note with this function. Thing 1: In my initial
decompilation, I had the local variables `newEntity`, `i`, and
`primIndex`. I found that `i` and `primIndex` used the same register,
and in my code I was seeing them on different registers. I ended up
lumping them into one variable. Since it serves two totally unrelated
purposes, I don't have a good name for it, so I just called it localVar.
Thing 2: The function starts by preloading pointers for `g_Entities` and
`g_CurrentRoomTileLayout`. I couldn't get the compiler to do this by
itself, so I ended up creating two local variables that hold these
locations. It's ugly, but I guess it works.
This ends up matching. Note that this calls my other recently-decompiled
function func_800FF7B8/InitStatsAndGear (which is how I wound up
decompiling this one in the first place), so these two PR's will create
a conflict since this one still calls that function by its old name.
---------
Co-authored-by: bismurphy <bismurphy@users.noreply.github.com>
Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
NZ0:
EntityAxeKnight (I'm proud to present the first enemy decompiled in the
project perhaps?)
Here is the scratch in case somebody wants to tune it and provide
changes:
https://decomp.me/scratch/LKkYQ
EntitySubWeaponContainer
EntityBloodSplatter
func_801B69E8
func_801C0D08
func_801C9E98
NP3:
EntityBloodSplatter
EntityZombie
func_801B8CC0
