This updates `m2ctx` to be aware of the `VERSION` env var and sets that
env var in the Makefile prior to running `m2ctx` to eliminate the:
warning: #warning "Version not specified. Falling back to the US
version." [-Wcpp]
error caused by `common.h`.
This also removes `#define __STDC_HOSTED__ 0` from the generated
context. That macro causes redefinition warnings in decomp.me.
Instead of outputting a `D_80123456.dec` from an extracted compressed
asset, now a PNG is written from the decompressed data from the
specified YAML entry. The asset tool will synchronize with splat to
retrieve the symbol names for graphics and palettes. The new splat
extension `cmpgfx` will also be able to detect unused graphics by
failing with a hint on how to tweak the YAML. That way I found some
unused placeholders under the name `stage_placeholder` and some kind of
beta graphics from the prologue stage.
I also deduped the common shared servant functions except
ProcessSfxState. That one has a jumptable and I didn't want to mess with
multiple files/.rodata while just adding a new overlay.
This comes with a few fake symbols fixed on a few functions, types being
corrected when needed and `LEN` being used where I could spot a pattern.
Data in `bossfight` is used in a mix between `bossfight.c`, `slogra.c`
and `gaibon.c`, which I believe they were originally one file.
`0x3CB8` is not extracted as an asset. It is an unused blob of data.
Using Tile Molester doesn't suggest it is a clear 4bpp image and the
decompression tool doesn't work on it. I left it as it is until we know
more.
I had to make a few file splits to get a match. I am yet to understand
the pattern on how files were originally organised. In some instances it
suggests files were split with a C file for every entity. But in other
instances it does not seem to be the case. I hope NZ0 will come with a
different function order so we will get more hints.
I will probably make a short tool to import the GFX part into a
`header.c` for the remaining overlays as it is quite tome consuming to
do it manually.
Adds the CEN overaly to the HD version of the game.
Resolves#1638.
This uses most, but not all, matching functions. Some were excluded
because due to symbols mapping that has not bee completed yet.
Not all data has been imported, and mapping symbols, and data should
make many of the remaining functions easy to import.
This adds this overlay to the project.
Importantly, we use a new tool (tools/auto_dedupe_new_overlay) to
process the new overlay and automatically split its C files according to
known duplicate files. As of now, this does not do any decompiling (the
whole overlay is kept as INCLUDE_ASM), but it automatically does all the
file splits and code copying needed. Hopefully this reduces some amount
of duplicated work.
To be clear, this new script is not in any build chain, but is meant to
be used after `make-config` to take the overlay from being a single
giant lump of `us.c` to being individual files, for the sake of easier
deduplicating. It will likely need some revision for future overlays,
but at least it should be a good start toward reducing tedious work.
I'll leave this overlay in place like this for about a week in case
there are any newcomers who would like to try de-duplicating some of
these files and decompiling the new functions; after that week I'll get
into doing things myself.
More research on how cutscenes work.
I normalized all the various C files as `cutscene.c`, marked all the
isolated function as `static` and renamed the main entity as
`{STAGE}_CutsceneExec` (e.g. `CEN_CutsceneExec`). I am using the
`OVL_EXPORT` to automate the names.
TO-DO:
- [x] Rename entity as `{STAGE}_EntityCutscene` for consistency
- [x] CEN
- [x] DRE
- [x] NO3
- [x] NZ0
- [x] ST0
- [x] MAR
~~SEL~~
The offset of the portrait data seems to be hardcoded. I have no idea
how to resolve these offsets at compilation time. The entire cutscene
script thing is very sketchy and horribly designed by the original
developers. What a nightmare to integrate into our project.
This is how a cutscene script gets decompiled:
```
LOAD_PORTRAIT(0x80188D8C, 0),
SET_PORTRAIT(1, 0),
SCRIPT_UNKNOWN_11(),
PLAY_SOUND(0x37B),
WAIT_FOR_SOUND(),
SET_SPEED(4),
'T','h','a','t',' ','v','o','i','c','e','!',' ',
SET_WAIT(16),
SET_SPEED(3),
'A','l','u','c','a','r','d',',',
LINE_BREAK(),
SET_WAIT(16),
SET_FLAG(2),
'i','t','\'','s',' ','y','o','u','!',
SET_WAIT(48),
NEXT_DIALOG(),
```
Fixed a tab matching issue with the `check` target. These were matching
'\' and 't' as separator characters rather than '↹' (horizontal tab)
Made `tools/make-config.py` executable.
Followed [the
guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay)
which seemed to work relatively painlessly for this overlay.
I hit an issue in `make-config.py` where no duplicates were found and
have added a separate commit which fixed that. Otherwise no issues with
guide.
```
josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001
✔ generating psx splat config
✔ splitting config/splat.us.tt_001.yaml
✔ adjusting files at src/servant/tt_001
✔ disassembling matched functions
✖ finding duplicates across overlays
Traceback (most recent call last):
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module>
raise e
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module>
make_config(args.input, args.version)
File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config
if found > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
```
I was also able to remove the weird `goto block` on `RicMain`.
The HD version seems to be an older version. I am not entirely sure of
the US changes, but it seem to fix a series of bugs when Richter gets
grabbed by Gaibon? There's a new debug string which ints
`g_Player.unk7A` being named as `run_disable_f` but I did not yet rename
it.
Improve `make-config.py` to internally use the `symbols.py cross` and
cross-reference symbols from duplicate functions. This allows to add
even more symbols for new overlays with a third pass.
This was one of my biggest pain points when I was de-duplicating the
boss `MAR` overlay as I was not able to directly copy&paste duplicate
functions or include shared ones without having to individually fix the
symbols coming from `.data` and `.bss`. This is now automated.
```
$ python3 ./tools/make-config.py ric --version hd
✔ generating psx splat config
✔ splitting config/splat.hd.ric.yaml
✔ adjusting files at src/ric
✔ disassembling matched functions
✔ finding duplicates across overlays
✔ adding cross-referenced function names
✔ renamed 141 functions, splitting again
✔ cross-referencing 141 functions
✔ adding cross-referenced symbol names
✔ renamed 145 data/bss symbols, splitting again
```
Several quality of life changes for building:
* The Makefile has a `help` target which will show common targets.
Targets with comments starting with `##@` will be included in the help
output.
* The `check` target now has colored output.
* Added `check_disk` target which will check hashes of extracted disk
contents (useful for those who dump their own discs)
* Added `dump_disk` target which will create a bin/cue pair from an
original disc.
Ubuntu/Debian and Python recommend using virtual environments for
project-specific Python dependencies and as of Ubuntu 24 this is lightly
enforced when installing packages via pip. This is due to pip and the
system package manager installing files to the same location which may
cause either's internal state to no longer reflect what is actually
installed.
This updates the project to use a Python `virtualenv` for project
dependencies and updates internal scripts to support both global and
virtualenvs, but favors virtualenvs for new workspaces.
All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the
first `python3` in the path. For those with a virtualenv configured,
this will be the Python managed there. For everyone else, this should be
the system Python or whatever other scheme they may have used previously
(assuming `python3` already existed in their `PATH`).
The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and
use `python3` from there if it exists and has been configured. It also
has a new `python-dependencies` target which will configure the venv and
install all python dependnecies there.
The `Dockerfile` has been updated to create an external `.venv` outside
of the project directory. Python's `virtualenv`s are not relocatable and
may hardcode paths which are mounted differently in the container and
host. To deal with differences in paths between the container (which
mounts the host's project directory to `/sotn`) host which may be at an
arbitarary directory the `VENV_PATH` environment variable is used to
override paths in the `Makefile`.
GitHub workflows have been updated to use `.venv`.
As promised, a new tool to automatically generate a splat config for PSP
and PSX overlays for multiple game versions. This first version will
perform the following tasks:
* Create the splat config with data, rodata, text and bss sections to
their right offsets
* Generate a symbol list by cross-referencing duplicate symbols
* Split the overlay by creating the correspondent C file
This tool is not currently:
* Splitting C files
* Importing shared code already decompiled
There is a lot of tech involved, especially to detect the various
sections for the PSX overlays. But I will leave it to you to discover
how the implementation work if you're interested. Hopefully the code is
self-documented enough.
Usage examples:
* `python3 tools/make-config.py mar`
* `VERSION=jp12 python3 tools/make-config.py no2`
* `python3 tools/make-config.py tt_001 --version pspko`
With this tool I aim to add in the project multiple game versions of
TT_001, MAR and RBO3 from #557 and NO0.
A few updates that didn't make it into the first round:
* masks are now identified and expanded
* AND-EQUALS and OR-EQUALS are supported
* `sotn-lint` is run before `clang-format` to avoid having to run the
`format-src` target twice to catch line breaks
* `.h` files are included in `sotn-lint` since they are being used for
shared code
* `sotn` lint ignores the `mednafen` directory
From the [public
announcement](https://discord.com/channels/1079389589950705684/1079395108501331990/1276310695964835910)
on Discord:
> We've been progressing very far with the project and I started
wondering what would be the best course of action to protect the work of
existing and future contributors. I personally consider contributions to
a decompilation project as creative work. Therefore it is my desire to
protect this creative work from being re-claimed by other people. This
is especially true in the light of the ability of modding the existing
decompiled code and create derivative work.
>
> I talked privately with the current most active contributors. And me,
@sozud , @bismurphy and @joshlory so far agreed on pushing through the
AGPLv3 license for all the decompiled code in `sotn-decomp/src`,
excluding any third party code and decompiled code from the PSX SDK. But
I've been wanting to ask all of you if you are fine your code to be
licensed and protected under the AGPLv3 terms.
>
>
https://raw.githubusercontent.com/Xeeynamo/sotn-decomp/89686514916cabd0dd88ae0387f749a889c19e05/src/LICENSE
>
> I put and all the contributors, which can be accessed to the page
https://github.com/Xeeynamo/sotn-decomp/graphs/contributors . Again,
this will work for existing and future contributors. In the license I
put my full name as I will be responsible of any legality. And I
understand not every contributor wants to disclose their identity.
Please give an approval to my latest PR if you do not have any
objection. If not, please write here publicly.
>
> PS. I understand past contributors will either not see the message
above or decide to ignore it. If I do not hear from anyone of the
contributors for more than one week between those who did not interact
with me or the server, I will still go ahead and apply the license. It
is not in my intention to ignore past contributors. But I do not want
contributors who have been inactive for months or years to be a blocker.
Please do not mistake this as an act of malevolence.
---------
Co-authored-by: Joey Murphy <tjmurphy@mit.edu>
Allow to compile weapon overlays as a single C files. The benefit is to
disallow symbols to be exported by statically compile them all but the
header. We would no longer need to use `OVL_EXPORT` as long as the
symbols are marked as `static`. The frames data is also generated as an
array of data that can be just included into their respective C files,
generated by JSON files on the fly.
I changed `animset` to `frameset` as the data does not represent a set
of animations but rather a set of frames, where every frame has multiple
sprites. The information on how the frames are played (e.g. animation)
is found elsewhere.
Following the full list of changes to achieve this project:
* The `animset` from splat is now dummied out
* The `frameset` JSON is now generated by the asset manager
* The `frameset` JSON is now compiled into `src/weapon/w_0xx_y.h`
* The `header.c` has been moved into `shared.h`
* For simplicity I added a `#define g_Animset w_000_1`. I did not want
to hack the asset manager too much to force the name to be `g_Animset`
and I did not want to change all the symbols yet.
* Simplified `weapon_pc.c` as we now only need the exported header
* Always try to build the most up-to-date asset manager: this should
avoid weird errors whenever people pull new changes
As the asset manager now can accept those `config` files, now the CLI
supports the following commands:
* `stage extract`
* `stage build`
* `config extract` (NEW)
* `config build` (NEW)
The config structure is heavily inspired to Splat to maintain
consistency and continuity.
I moved *some* of the PSX-specific stuff out of the common `Makefile`
and into the `Makefile.psx.mk` -- there is still more that could be done
here, but this should be enough to have the correct rules for PSX and
PSP with the desired assembler.
Also threw in a bonus feature where I've removed the `move` macro from
`macro.inc` because `maspsx` does this automagically for you now.
Mostly identical to MAD and PSP. There are four very different functions
from US that needs to be decompiled to be 100% complete:
* [x] `e_misc.c` `EntityMessageBox`
* [x] `e_misc.c` `EntityRelicOrb`
* [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist
* [x] `e_stage_name.c` `EntityStageNamePopup`
These four very different functions are the only ones that use the JP
text instead of the US one. This overlay also lacks of `BlitChar`.
`EntityStageNamePopup` is very similar to the PSP counterpart. I used
@joshlory scratch to match the HD part.
`EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD
overlay). I found some code that crossed between BETA and HD, so I just
put an `#else` after `VERSION_US`. We are starting to reconstruct how
the source code originally evolved across different game builds.
This aims to deprecate all the Splat tools in `tools/splat_ext` in
favour of a more centralised asset manager. This brings the following
advantages:
* Much faster extraction
* Faster build
* Automatically define `static` symbols or unique names whenever
`static` is not possible
* Allow to embed assets into the output binary
* Drastically simplify `Makefile` by removing all the asset build rules
* Avoid situations where it is not possible to extract and build assets
that is not 4-byte aligned
This is achieved by having the splat YAML targeting a normal C file as
data and have an external tool to take care of the following:
1. Extract asset files straight from the overlay binary file into human
readable file in `assets/st/STAGE_NAME`
2. Build assets as header files that go into `src/st/STAGE_NAME` to just
include them from any C file
This requires each stage header to have the following new format: please
see `src/st/nz0/header.c`
Built assets in `src/st` are ignored by Git.
As for now, for simplicity sake, the steps `make extract_assets` and
`make build_assets` are just executed within `make extract` exclusively
for the US version.
I plan to auto-generate files such as `src/st/nz0/tile_data.c`.
For a first iteration I am aiming to handle the following:
* [X] Extract rooms: `assets/st/*/rooms.json`
* [X] Extract room layers: `assets/st/*/entity_layouts.json`
* [X] Extract tilemap data: `assets/st/*/tilemap_*.bin`
* [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json`
* [X] Extract sprites: `assets/st/*/sprites.json`
* [x] Extract entity layouts
* [X] Build rooms: `src/st/*/rooms.h`
* [X] Build room layers: `src/st/*/layers.h`
* [X] Build tilemap data: `src/st/*/tilemap_*.h`
* [X] Build tilemap definitions: `src/st/*/tiledef_*.h`
* [x] Build sprites (aka `g_SpriteBanks`)
* [x] Build entity layouts
* [x] Allow the tool to suggest how to adjust the Splat config for each
overlay
I want the tool to cover the following stages:
* [x] CEN
* [x] DRE
* ~MAD~ I do not think this can be done, it is way too different from
the other overlays
* [x] NO3
* [x] NP3
* [X] NZ0
* [x] ST0
* [X] WRP
* [x] RWRP
* ~WRP (PSP)~ Maybe in a follow-up PR
For a later iteration I plan on extracting and build:
* Entity GFX thingie
* The CLUT thingie in the header
* Uncompressed GFX data
* Cutscene data
* Blueprints
* The `src/config_us.h` thingie
---------
Co-authored-by: Josh Lory <josh.lory@outlook.com>
After building for all supported architectures `make format` was taking
around 30s. This commit includes various changes which get that down to
around 3½s on the same machine.
This changes the `format` target in a few ways:
* filters out files generated by CMake
* runs `clang-format` in parallel
* breaks `format` into 3 separate targets which can be run in parallel
When building the PC version cmake creates a file to test compiler
features. This file is picked up by the pattern which matches files for
`clang-format`. This file took around 20s, filtering it out brought
total format time to around 10s.
Instead of passing a list created by a `find` command directly as the
args to `clang-format` the list is now passed to `xargs` which runs
several processes in parallel, each looking at 10 files (determined by
experimentation). This has the added advantage of ensuring as the source
list grows `ARG_MAX` won't be a concern. This brought total format time
to around 6s.
`format` was broken up into three targets `-src`, `-tools`, and
`-symbols`. These targets can be run in parallel, bringing the total
time down to 3½s.
---------
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
This updates the Makefile so that targets which depend on phony targets
(like `*_dirs`) have direct dependencies on those targets rather than
relying on transient dependencies from dependent targets. Other targets
explicitly create parent directories for their files.
This resolves#739, making it possible to using make-4.4 to build in
parallel.
Still work in progress. I removed splat as a submodule and started using
it as a pip package instead. Everything is matching but the memory card
icons part in both DRA and SEL. I still have no idea what the issue is.
Once this PR is good to be merged, we can get rid of the splat fork too.
This adds some additional context for func_801B9744 and renames globals
to fit their assumed purpose.
Migrated various stream metadata to C definitions.
The makefile changes add a dependency on `sel.h` to all the objects that
get built from the `st/sel` directory.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
The `check` recipe now has dependencies on all of the files it checks
allowing them to be rebuilt if necessary.
Consistent use of `dd` and suppressing `dd`'s output which noise for the
replacement and padding it's doing.
Run 7z in non-interactive mode to avoid confirming overwrite of files
during extraction (equivalent behavior to `VERSION=us` `extract_disk`).
`patch` is a phony target.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
Extract ST/WRP out of #1119 . All the function symbols should have been
cross-referenced. There as some PSX functions missing from PSP and some
new functions from PSP that are not present on PSX (e.g.
`st_init_wrp.c`).
The files `st_debug.c` and `e_breakable.c` are shared between WRP PSX
and WRP PSP. Everything else from PSP is isolated into its own folder. I
had to do some tricks on the YAML config to allow shared code.
`ST_WRP_MERGE = st_debug e_breakable` in the `Makefile` is a bit
annoying as MWCC complains about every single minute detail from the C
source that has been already decompiled for the PSX US build.
`EntityWarpSmallRocks` is matching on PSP but I couldn't extract the
rodata without having a bunch of linker errors. This might be a Splat
issue. I need to investigate further.
`func_psp_09244760` is soooo interesting. The values from `0x11` to
`0x17` matches the Entity IDs that are unique to the WRP overlay. This
aligns to what we have in `typedef enum EntityIDs`.
Overall I am very excited to the recent discoveries from the PSP build!
OK. Here goes. Version 0.0.1 alpha of the MWCC Global Assembly Processor
(mwccgap). It's currently very simple/limited, but it appears to work
for `src/servant/tt_000/10E8.c`.
There is lot more that can be done to improve mwccgap - i.e. supporting
.rodata migration would be a good addition, but let's see how far we can
get with it in it's current state.
Note that the Makefile could do with some improvements - we don't nede
to use mwccgap for any C file that *dont* have INCLUDE_ASM macros (it's
a waste of time) so these could be ignored, i.e. for SSSV I do something
like this to find the files that need fixing up:
```
GLOBAL_ASM_C_FILES := $(shell $(GREP) GLOBAL_ASM $(SRC_DIR) </dev/null 2>/dev/null)
```
.. although this is perhaps too simple given that SOTN has a mix of PSP
and PSX functions (and therefore there may be INCLUDE_ASM for a PSX
function but none for PSP functions...
This replaces several dozen BIOS trampolines from the extracted ASM to
"decompiled" source. These are modeled on the `INCLUDE_ASM` macro, but
generate the instructions necessary for each trampoline directly instead
of importing an extracted source file.
Because these trampolines never return, and GCC 2.6 doesnt appear to
have builtins for leaving off the return jump postamble, these will
likely need to remain assembly.
This also changes the `main.elf` target to depend on `main.ld`, and
undefined symbols files, allowing `make build` to regenerate those files
if necessary.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
In CUE files `FILE` is a stateful, global declaration that applies to
all following `TRACK`s. `sotn-disk` was treating `FILE` declarations as
`TRACK` delimiters which would result in incorrect parsing of CUE files
with a single `FILE`, but multiple tracks.
Now when reading a `FILE` declaration, the path is stored and processing
continues. A `TRACK` declaration will use the previously defined path,
and if a previous track had been started, append that previous track to
the list.
This also builds `sotn-disk` using the local repository instead of
pulling the latest commit from GitHub. The target depends on `sotn-disk`
sources and will rebuild as necessary (or with `make
~/go/bin/sotn-disk`).
As an aside,
[pull/232](https://github.com/Xeeynamo/sotn-decomp/pull/232) ran into
this same error, but fixes in a slightly different way that leaves side
effects that may make supporting things like `INDEX` (for extracting the
placeholder audio, for example) more error prone in the future.
Co-authored-by: Jonathan Hohle <jon@ttkb.co>
100% decompiled.
`US` adds some padding to have the file exactly 40KB long. The HD
version does not add any padding, hence why I ended up producing
`tt_000_raw.bin`. I would have used just `tt_000.bin` as a name as found
in `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin`, but on
Windows-based file system it would collide with `TT_000.BIN` due to the
OS having case insensitive names.
I modified `make clean` as I found annoying that `VERSION=hd make clean`
would wipe out `us` build stuff.
The only different function in HD is `ProcessEvent`, which has a weaker
check. Another hint suggesting HD being older than US.
`s32 _unused[26];` added enough padding in the bss section to get an
🆗 . I am pretty sure it is unused data because the final binary is
not aligned by any power of 2.
A side-project I've been working on that helped me understanding some
structures that are now a bit better documented in #1034 . Richter
sprites are extracted as usual. Doing `python3
tools/splat_ext/spritesheet.py decode disks/us/BIN/ARC_F.BIN assets/arc
alucard disks/us/BIN/F_GAME.BIN 1` can extract Alucard sprites too.
There is no repacking yet as I want to avoid using splat for this one.
This is a proof-of-concept for Japanese sotn_strs. The macro `_SJ` is
used to designate them. I only converted one string in config_jp to
serve as a test for the pipeline. I moved sotn_str to a folder and added
some tests. I renamed it since python doesn't like modules with - in the
name. The code could be cleaner and the conversion table could use some
more work but I think this is a reasonable start.
Once decomp.me is updated I will update the SOTN preset to include
`--expand-li`. Let me know if you run into any issues but this builds an
🆗 binary for me.
Needed a break from all the asm. I realize that Makefile changes aren't
always appreciated as they can break things in unexpected ways,
especially if they come from newbies to the project (ahem), so feel free
to close this if it's not needed.
One comment about removing `/**/*.s` and `/**/*.c`: as far as I could
see there were no subdirectories, so didn't seem needed.
Uses a small JSON parsing library called cJSON (MIT license) to parse
the assets `equipment.json` and `accessory.json` and load them in memory
rather than hard coding it or pre-building it.
As per title. I also added a new tool called `dirt_patch`. As I
mentioned in our Discord server there is some left-over data from
previous dev builds in DRA.BIN and potentially elsewhere too. The tool
uses the new file `config/dirt.us.json` which stores a list of patches
to avoid crazy hacks and `#ifdef` for the sake of getting a match. I
hope this tool will not be abused.
Extract the memory card icons out of DRA.BIN and SEL.BIN.
I also took the opportunity to extract `g_MemcardPortMask` into its own
source file `save_mgr_pre.c`. Unfortunately this is required to keep the
original data order. This suggests that most likely the icon palette and
bitmap were baked into the original `save_mgr.c` as byte arrays. I
decided to take a different approach and extracted them as PNGs for
better moddability.
I had to spin-off Splat into a new fork due to some breaking changes on
0.18.0.
