Game-Decompilation/sotn-decomp
1
0
Fork 0
mirror of https://github.com/Xeeynamo/sotn-decomp.git synced 2024-11-23 21:19:55 +00:00
Code Issues Actions 5 Packages Projects Releases Wiki Activity
e357fcaeeb
sotn-decomp/Makefile

619 lines
25 KiB
Makefile
Raw Normal View History

Simplify Makefile
2022-02-23 18:17:18 +00:00
.SECONDEXPANSION:
Do not delete intermediary files
2022-03-02 00:38:54 +00:00
.SECONDARY:
Simplify Makefile
2022-02-23 18:17:18 +00:00
Upload disassembly
2022-01-01 17:45:41 +00:00
# Binaries
Switch ASPATCH to MASPSX (#254) This PR replaces the ASPATCH command with my custom script named MASPSX. It's very new, and so is likely to have bugs and/or missing functionality. That said it's producing an :ok: build across all overlays, and is matching a few functions that required psyq3.5/aspsx. I have not attempted to match any of the functions that require rodata, nor any of the functions that refused to compile / had diffs. Note that I have added `-gcoff` to the `CC_FLAGS`, this will give you line numbers in asm-differ. ---------------------- Thank you in advance for contributing to sotn-decomp. Before submitting this PR, please make sure: - [x] The PR is small and focuses to address a single task - [x] `make all` reports all `OK` - [x] `make format` reports no files changed - [x] Have Actions enabled in your fork to run the auto-formatter
2023-06-16 18:51:16 +00:00
VERSION ?= us
Upload disassembly
2022-01-01 17:45:41 +00:00
MAIN := main
DRA := dra
# Compilers
Switch ASPATCH to MASPSX (#254) This PR replaces the ASPATCH command with my custom script named MASPSX. It's very new, and so is likely to have bugs and/or missing functionality. That said it's producing an :ok: build across all overlays, and is matching a few functions that required psyq3.5/aspsx. I have not attempted to match any of the functions that require rodata, nor any of the functions that refused to compile / had diffs. Note that I have added `-gcoff` to the `CC_FLAGS`, this will give you line numbers in asm-differ. ---------------------- Thank you in advance for contributing to sotn-decomp. Before submitting this PR, please make sure: - [x] The PR is small and focuses to address a single task - [x] `make all` reports all `OK` - [x] `make format` reports no files changed - [x] Have Actions enabled in your fork to run the auto-formatter
2023-06-16 18:51:16 +00:00
CC1PSX := ./bin/cc1-psx-26
Upload disassembly
2022-01-01 17:45:41 +00:00
CROSS := mipsel-linux-gnu-
AS := $(CROSS)as
Update compiler to use little endian
2023-04-05 18:08:14 +00:00
CC := $(CC1PSX)
Upload disassembly
2022-01-01 17:45:41 +00:00
LD := $(CROSS)ld
Switch ASPATCH to MASPSX (#254) This PR replaces the ASPATCH command with my custom script named MASPSX. It's very new, and so is likely to have bugs and/or missing functionality. That said it's producing an :ok: build across all overlays, and is matching a few functions that required psyq3.5/aspsx. I have not attempted to match any of the functions that require rodata, nor any of the functions that refused to compile / had diffs. Note that I have added `-gcoff` to the `CC_FLAGS`, this will give you line numbers in asm-differ. ---------------------- Thank you in advance for contributing to sotn-decomp. Before submitting this PR, please make sure: - [x] The PR is small and focuses to address a single task - [x] `make all` reports all `OK` - [x] `make format` reports no files changed - [x] Have Actions enabled in your fork to run the auto-formatter
2023-06-16 18:51:16 +00:00
CPP := $(CROSS)cpp
Upload disassembly
2022-01-01 17:45:41 +00:00
OBJCOPY := $(CROSS)objcopy
Use allegrex assembler for PSP (#1424) I moved *some* of the PSX-specific stuff out of the common `Makefile` and into the `Makefile.psx.mk` -- there is still more that could be done here, but this should be enough to have the correct rules for PSX and PSP with the desired assembler. Also threw in a bonus feature where I've removed the `move` macro from `macro.inc` because `maspsx` does this automagically for you now.
2024-07-22 19:36:28 +00:00
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
CC_FLAGS += -G0 -w -O2 -funsigned-char -fpeephole -ffunction-cse -fpcc-struct-return -fcommon -fverbose-asm -msoft-float -g
Decompile some PSX SDK functions (#1002)
2024-01-21 17:58:32 +00:00
CPP_FLAGS += -Iinclude -Iinclude/psxsdk -undef -Wall -fno-builtin
Sound test: add 42398.c (#799) In Italian we say "Abbiamo fatto 30, facciamo 31". It literally means "We come all the way up to 30, it wouldn't cost nothing to push a little further for 31". I moved `log.h` into `include/` to easily logging stuff elsewhere and without weird `#ifdef VERSION_*` by adding `-DNO_LOGS` in the main Makefile. This is a bit hacky, but it does the trick. I had to expand `D_80138784` otherwise I would get a segfault. Both `SoundInit` and `func_801361F8` are called in the main, so I removed them. To restore the previous behaviour and isolate the sound engine, just replace `MainGame()` in the `main.c` with a `SoundInit()` and then a loop that calls `UpdateGame()`. Last, but not least, SDL2 is in.
2023-11-21 02:17:51 +00:00
CPP_FLAGS += -Dmips -D__GNUC__=2 -D__OPTIMIZE__ -D__mips__ -D__mips -Dpsx -D__psx__ -D__psx -D_PSYQ -D__EXTENSIONS__ -D_MIPSEL -D_LANGUAGE_C -DLANGUAGE_C -DNO_LOGS -DHACKS -DUSE_INCLUDE_ASM
SEL import data/rodata (part 1) (#663) I started importing the data in SEL also as a proof of concept we can also import the SOTN-specific formatted strings. For that I integrated the new tool `tools/sotn-str.py` into the build chain. It works like the following: * `./tools/sotn-str.py parse disks/us/ST/SEL/SEL.BIN 27598`: read the binary content in a specific offset and parse it as a readable Unicode string. * `cat src/st/sel/2C048.c | ./tools/sotn-str.py process` converts the Unicode string into a byte sequence that can be understood by the SOTN engine. There is a dirty hack I did to make this work: ```c #ifndef SOTN_STR #define _S(x) (x) // Strings processed by tools/sotn-str.py #endif ``` When invoking `mipsel-linux-gnu-cpp` I am passing -DSOTN_STR, effectively ignoring `_S` in the pre-processor. Then I am passing it into `sotn-str` before `iconv` to resolve any string surrounded by `_S`. I am a bit of afraid this might be annoying when using the permuter. I tried doing `$(SOTN_STR) -f $< | $(CPP)` but apparently `cpp` does not work well with content passed via stdin.
2023-10-10 20:29:30 +00:00
CPP_FLAGS += -D_internal_version_$(VERSION) -DSOTN_STR
Add mwccgap (#1131) OK. Here goes. Version 0.0.1 alpha of the MWCC Global Assembly Processor (mwccgap). It's currently very simple/limited, but it appears to work for `src/servant/tt_000/10E8.c`. There is lot more that can be done to improve mwccgap - i.e. supporting .rodata migration would be a good addition, but let's see how far we can get with it in it's current state. Note that the Makefile could do with some improvements - we don't nede to use mwccgap for any C file that *dont* have INCLUDE_ASM macros (it's a waste of time) so these could be ignored, i.e. for SSSV I do something like this to find the files that need fixing up: ``` GLOBAL_ASM_C_FILES := $(shell $(GREP) GLOBAL_ASM $(SRC_DIR) </dev/null 2>/dev/null) ``` .. although this is perhaps too simple given that SOTN has a mix of PSP and PSX functions (and therefore there may be INCLUDE_ASM for a PSX function but none for PSP functions...
2024-05-18 10:41:00 +00:00
LD_FLAGS := -nostdlib --no-check-sections
Upload disassembly
2022-01-01 17:45:41 +00:00
# Directories
Change assembly location from asm/ to asm/us/
2023-02-21 19:11:05 +00:00
ASM_DIR := asm/$(VERSION)
Use allegrex assembler for PSP (#1424) I moved *some* of the PSX-specific stuff out of the common `Makefile` and into the `Makefile.psx.mk` -- there is still more that could be done here, but this should be enough to have the correct rules for PSX and PSP with the desired assembler. Also threw in a bonus feature where I've removed the `move` macro from `macro.inc` because `maspsx` does this automagically for you now.
2024-07-22 19:36:28 +00:00
BIN_DIR := bin
Upload disassembly
2022-01-01 17:45:41 +00:00
SRC_DIR := src
Asset extraction and build experimentation
2023-02-03 15:38:48 +00:00
ASSETS_DIR := assets
Automatically format code to the LLVM format
2022-08-26 18:16:27 +00:00
INCLUDE_DIR := include
Writes built binaries into build/us/
2023-02-21 20:06:27 +00:00
BUILD_DIR := build/$(VERSION)
Change how game assets are managed
2023-02-21 23:25:24 +00:00
DISK_DIR := $(BUILD_DIR)/${VERSION}/disk
Upload disassembly
2022-01-01 17:45:41 +00:00
CONFIG_DIR := config
TOOLS_DIR := tools
Makefile refactoring (#866) Needed a break from all the asm. I realize that Makefile changes aren't always appreciated as they can break things in unexpected ways, especially if they come from newbies to the project (ahem), so feel free to close this if it's not needed. One comment about removing `/**/*.s` and `/**/*.c`: as far as I could see there were no subdirectories, so didn't seem needed.
2023-12-18 21:22:42 +00:00
PSXLIBS := $(addprefix lib, c c2 api etc card gpu gs gte cd snd spu)
Upload disassembly
2022-01-01 17:45:41 +00:00
# Files
Makefile refactoring (#866) Needed a break from all the asm. I realize that Makefile changes aren't always appreciated as they can break things in unexpected ways, especially if they come from newbies to the project (ahem), so feel free to close this if it's not needed. One comment about removing `/**/*.s` and `/**/*.c`: as far as I could see there were no subdirectories, so didn't seem needed.
2023-12-18 21:22:42 +00:00
PSXLIB_DIRS := $(addprefix psxsdk/, . $(PSXLIBS))
Migrate .rodata to functions for main.exe (#1078) I feel like getting all the rodata migrated will make it easier to update splat (also makes it easier to match functions that use rodata).
2024-02-24 11:35:01 +00:00
PSXLIB_DATA_DIRS := $(addprefix data/, . $(PSXLIB_DIRS))
MAIN_ASM_DIRS := $(addprefix $(ASM_DIR)/$(MAIN)/,. $(PSXLIB_DIRS) data $(PSXLIB_DATA_DIRS))
Makefile refactoring (#866) Needed a break from all the asm. I realize that Makefile changes aren't always appreciated as they can break things in unexpected ways, especially if they come from newbies to the project (ahem), so feel free to close this if it's not needed. One comment about removing `/**/*.s` and `/**/*.c`: as far as I could see there were no subdirectories, so didn't seem needed.
2023-12-18 21:22:42 +00:00
MAIN_SRC_DIRS := $(addprefix $(SRC_DIR)/$(MAIN)/,. $(PSXLIB_DIRS))
MAIN_S_FILES := $(wildcard $(addsuffix /*.s, $(MAIN_ASM_DIRS)))
MAIN_C_FILES := $(wildcard $(addsuffix /*.c, $(MAIN_SRC_DIRS)))
MAIN_O_FILES := $(patsubst %.s,%.s.o,$(MAIN_S_FILES))
MAIN_O_FILES += $(patsubst %.c,%.c.o,$(MAIN_C_FILES))
MAIN_O_FILES := $(addprefix $(BUILD_DIR)/,$(MAIN_O_FILES))
Upload disassembly
2022-01-01 17:45:41 +00:00
MAIN_TARGET := $(BUILD_DIR)/$(MAIN)
Python virtualenv Support (#1620) Ubuntu/Debian and Python recommend using virtual environments for project-specific Python dependencies and as of Ubuntu 24 this is lightly enforced when installing packages via pip. This is due to pip and the system package manager installing files to the same location which may cause either's internal state to no longer reflect what is actually installed. This updates the project to use a Python `virtualenv` for project dependencies and updates internal scripts to support both global and virtualenvs, but favors virtualenvs for new workspaces. All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the first `python3` in the path. For those with a virtualenv configured, this will be the Python managed there. For everyone else, this should be the system Python or whatever other scheme they may have used previously (assuming `python3` already existed in their `PATH`). The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and use `python3` from there if it exists and has been configured. It also has a new `python-dependencies` target which will configure the venv and install all python dependnecies there. The `Dockerfile` has been updated to create an external `.venv` outside of the project directory. Python's `virtualenv`s are not relocatable and may hardcode paths which are mounted differently in the container and host. To deal with differences in paths between the container (which mounts the host's project directory to `/sotn`) host which may be at an arbitarary directory the `VENV_PATH` environment variable is used to override paths in the `Makefile`. GitHub workflows have been updated to use `.venv`.
2024-09-18 06:19:20 +00:00
VENV_PATH ?= .venv
export PATH := $(VENV_PATH)/bin:$(PATH)
Upload disassembly
2022-01-01 17:45:41 +00:00
# Tooling
PYTHON := python3
Update splat to 0.24.4 (#1112) Still work in progress. I removed splat as a submodule and started using it as a pip package instead. Everything is matching but the memory card icons part in both DRA and SEL. I still have no idea what the issue is. Once this PR is good to be merged, we can get rid of the splat fork too.
2024-06-12 17:50:32 +00:00
SPLAT := splat split
Ensure asm-differ is installed
2022-10-14 20:19:34 +00:00
ASMDIFFER_DIR := $(TOOLS_DIR)/asm-differ
ASMDIFFER_APP := $(ASMDIFFER_DIR)/diff.py
Create context with Makefile
2022-10-15 10:18:32 +00:00
M2CTX_APP := $(TOOLS_DIR)/m2ctx.py
M2CTX := $(PYTHON) $(M2CTX_APP)
Decompile code with make
2022-10-15 10:45:20 +00:00
M2C_DIR := $(TOOLS_DIR)/m2c
M2C_APP := $(M2C_DIR)/m2c.py
M2C := $(PYTHON) $(M2C_APP)
Switch ASPATCH to MASPSX (#254) This PR replaces the ASPATCH command with my custom script named MASPSX. It's very new, and so is likely to have bugs and/or missing functionality. That said it's producing an :ok: build across all overlays, and is matching a few functions that required psyq3.5/aspsx. I have not attempted to match any of the functions that require rodata, nor any of the functions that refused to compile / had diffs. Note that I have added `-gcoff` to the `CC_FLAGS`, this will give you line numbers in asm-differ. ---------------------- Thank you in advance for contributing to sotn-decomp. Before submitting this PR, please make sure: - [x] The PR is small and focuses to address a single task - [x] `make all` reports all `OK` - [x] `make format` reports no files changed - [x] Have Actions enabled in your fork to run the auto-formatter
2023-06-16 18:51:16 +00:00
M2C_ARGS := -P 4
Add mwccgap (#1131) OK. Here goes. Version 0.0.1 alpha of the MWCC Global Assembly Processor (mwccgap). It's currently very simple/limited, but it appears to work for `src/servant/tt_000/10E8.c`. There is lot more that can be done to improve mwccgap - i.e. supporting .rodata migration would be a good addition, but let's see how far we can get with it in it's current state. Note that the Makefile could do with some improvements - we don't nede to use mwccgap for any C file that *dont* have INCLUDE_ASM macros (it's a waste of time) so these could be ignored, i.e. for SSSV I do something like this to find the files that need fixing up: ``` GLOBAL_ASM_C_FILES := $(shell $(GREP) GLOBAL_ASM $(SRC_DIR) </dev/null 2>/dev/null) ``` .. although this is perhaps too simple given that SOTN has a mix of PSP and PSX functions (and therefore there may be INCLUDE_ASM for a PSX function but none for PSP functions...
2024-05-18 10:41:00 +00:00
SOTNSTR := $(PYTHON) $(TOOLS_DIR)/sotn_str/sotn_str.py process
Fix CI pipeline
2023-06-17 11:29:42 +00:00
MASPSX_DIR := $(TOOLS_DIR)/maspsx
MASPSX_APP := $(MASPSX_DIR)/maspsx.py
Update maspsx (#1100) Also tweaked the Dockerfile since this is what I use to build the project.
2024-03-15 21:04:57 +00:00
MASPSX := $(PYTHON) $(MASPSX_APP) --expand-div --aspsx-version=2.34
More libgpu, maspsx update (#1391) This file seems to use an older aspsx. Thanks to @mkst for the fixes
2024-07-08 07:19:21 +00:00
MASPSX_21 := $(PYTHON) $(MASPSX_APP) --expand-div --aspsx-version=2.21
Switch ASPATCH to MASPSX (#254) This PR replaces the ASPATCH command with my custom script named MASPSX. It's very new, and so is likely to have bugs and/or missing functionality. That said it's producing an :ok: build across all overlays, and is matching a few functions that required psyq3.5/aspsx. I have not attempted to match any of the functions that require rodata, nor any of the functions that refused to compile / had diffs. Note that I have added `-gcoff` to the `CC_FLAGS`, this will give you line numbers in asm-differ. ---------------------- Thank you in advance for contributing to sotn-decomp. Before submitting this PR, please make sure: - [x] The PR is small and focuses to address a single task - [x] `make all` reports all `OK` - [x] `make format` reports no files changed - [x] Have Actions enabled in your fork to run the auto-formatter
2023-06-16 18:51:16 +00:00
GO := $(HOME)/go/bin/go
GOPATH := $(HOME)/go
SOTNDISK := $(GOPATH)/bin/sotn-disk
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
SOTNASSETS := $(GOPATH)/bin/sotn-assets
Switch ASPATCH to MASPSX (#254) This PR replaces the ASPATCH command with my custom script named MASPSX. It's very new, and so is likely to have bugs and/or missing functionality. That said it's producing an :ok: build across all overlays, and is matching a few functions that required psyq3.5/aspsx. I have not attempted to match any of the functions that require rodata, nor any of the functions that refused to compile / had diffs. Note that I have added `-gcoff` to the `CC_FLAGS`, this will give you line numbers in asm-differ. ---------------------- Thank you in advance for contributing to sotn-decomp. Before submitting this PR, please make sure: - [x] The PR is small and focuses to address a single task - [x] `make all` reports all `OK` - [x] `make format` reports no files changed - [x] Have Actions enabled in your fork to run the auto-formatter
2023-06-16 18:51:16 +00:00
GFXSTAGE := $(PYTHON) $(TOOLS_DIR)/gfxstage.py
Extract memory card icons (#166) Extract the memory card icons out of DRA.BIN and SEL.BIN. ![image](https://user-images.githubusercontent.com/6128729/227794798-4dd071f9-512b-4c72-8f5d-fcbf7a615519.png) I also took the opportunity to extract `g_MemcardPortMask` into its own source file `save_mgr_pre.c`. Unfortunately this is required to keep the original data order. This suggests that most likely the icon palette and bitmap were baked into the original `save_mgr.c` as byte arrays. I decided to take a different approach and extracted them as PNGs for better moddability. I had to spin-off Splat into a new fork due to some breaking changes on 0.18.0.
2023-12-12 07:12:13 +00:00
PNG2S := $(PYTHON) $(TOOLS_DIR)/png2s.py
Decompile StoreSaveData (#571) This took me a monumental effort. The PR does: * Decompile `StoreSaveData` for DRA (US and HD) * Decompile `StoreSaveData` for SEL * Match `MemcardFormat` in SEL * Cross-reference symbols between DRA and SEL * Bring common code between DRA and SEL to parity * Discover `save_mgr.c`, `save_mgr2.c` and `save_mgr3.c` have the same layout in both DRA and SEL * Deduplicate `save_mgr3.c` * Import `.data` and `.rodata` from both DRA and SEL * Convert UTF-8 Japanese characters into Shift-JIS while building * Minor refactor with less hardcoded magic numbers I made the following compromises: * In DRA US `MakeMemcardPath` there is some dirty data at the end of the string. I had to create a byte array called `HACK_SaveName`. This does not occur on DRA HD or SEL * I did not de-duplciate `save_mgr` as there are a good amount of functions and symbols I have no knowledge about * I did not decompile `save_mgr2` and it will come as a separate PR.
2023-09-09 20:09:46 +00:00
ICONV := iconv --from-code=UTF-8 --to-code=Shift-JIS
Import 4A538, demo and 5087C data (#835) As per title. I also added a new tool called `dirt_patch`. As I mentioned in our Discord server there is some left-over data from previous dev builds in DRA.BIN and potentially elsewhere too. The tool uses the new file `config/dirt.us.json` which stores a list of patches to avoid crazy hacks and `#ifdef` for the sake of getting a match. I hope this tool will not be abused.
2023-12-12 18:36:34 +00:00
DIRT_PATCHER := $(PYTHON) $(TOOLS_DIR)/dirt_patcher.py
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
SHASUM := shasum
Upload disassembly
2022-01-01 17:45:41 +00:00
Simplify Makefile
2022-02-23 18:17:18 +00:00
define list_src_files
$(foreach dir,$(ASM_DIR)/$(1),$(wildcard $(dir)/**.s))
$(foreach dir,$(ASM_DIR)/$(1)/data,$(wildcard $(dir)/**.s))
$(foreach dir,$(ASM_DIR)/$(1)/psxsdk,$(wildcard $(dir)/**.s))
Mark SEL handwritten functions as asm (#977)
2024-01-14 19:27:34 +00:00
$(foreach dir,$(ASM_DIR)/$(1)/handwritten,$(wildcard $(dir)/**.s))
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(foreach dir,$(SRC_DIR)/$(1),$(wildcard $(dir)/**.c))
$(foreach dir,$(SRC_DIR)/$(1)/psxsdk,$(wildcard $(dir)/**.c))
WRP split some data and add new split tools (#109) The objective is to separate the data that is usually written into `asm/*/data` into separate files. Those will initially be binary files. There is also an utility that allows to decompress compressed data. The decompressed data is currently not read as we are missing a compression algorithm that matches the original one. I tried writing one in the past: it worked, it re-created similar files but sadly not the same ones. The decompression algorithm is there to allow to do some data mining and understand what the data is and how it is used.
2023-03-09 13:32:00 +00:00
$(foreach dir,$(ASSETS_DIR)/$(1),$(wildcard $(dir)/**))
Simplify Makefile
2022-02-23 18:17:18 +00:00
endef
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
define list_st_src_files
$(foreach dir,$(ASM_DIR)/$(1),$(wildcard $(dir)/**.s))
$(foreach dir,$(ASM_DIR)/$(1)/data,$(wildcard $(dir)/**.s))
$(foreach dir,$(SRC_DIR)/$(1),$(wildcard $(dir)/**.c))
$(foreach dir,$(ASSETS_DIR)/$(1),$(wildcard $(dir)/D_8018*.bin))
endef
Simplify Makefile
2022-02-23 18:17:18 +00:00
define list_o_files
$(foreach file,$(call list_src_files,$(1)),$(BUILD_DIR)/$(file).o)
endef
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
define list_st_o_files
$(foreach file,$(call list_st_src_files,$(1)),$(BUILD_DIR)/$(file).o)
endef
Start sharing some common stage code (#464) Following up #454 , #453 and related from @sozud , I started to convert some previously defined headers into proper shared C code. The advantage from the header is the C file is completely independent, it will have its own `.data` and `.rodata` and it will be compiled once. I also took the opportunity to move `include/st/AnimateEntity.h` to `src/st/animate_entity.h` for consistency. I think the change is small enough to be included as part of this PR.
2023-09-03 06:55:48 +00:00
define list_shared_src_files
$(foreach dir,$(SRC_DIR)/$(1),$(wildcard $(dir)/*.c))
endef
define list_shared_o_files
$(foreach file,$(call list_shared_src_files,$(1)),$(BUILD_DIR)/$(file).o)
endef
Simplify Makefile
2022-02-23 18:17:18 +00:00
define link
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
$(LD) $(LD_FLAGS) -o $(2) \
Simplify Makefile
2022-02-23 18:17:18 +00:00
-Map $(BUILD_DIR)/$(1).map \
Enhance make extract on PSX (#662) Previously if either the YAML or the symbol list were changed, we were forced to either manually delete the `asm` folder of the specific overlay or to invoke `make clean`, where the latter forced us to re-extract all the overlays via `make -j extract`. That changes with this PR. How `make` works is `target: dep1 dep2` where if one of the dependencies has the last modified date greater than the target, the rule is triggered again. Previously we were extracting overlays doing `make extract_stcen`. But since every overlay extraction generates a linker script I now changed the rules to do `make build/us/stcen.ld` to extract the same overlay. As I explained above, if either the YAML or one of the related symbols changes, the linker modified date will be older and the extraction for that overlay to be trigger again. The above allowed me to stop polluting the repo root with the linker scripts as they are now moved into `build/us` or `build/hd` depending of what you're trying to build. If you do `make extract` twice in a row you will be welcomed with a `make: Nothing to be done for 'extract'.`. There are still some instances where you **might** need `make clean` beforehand, especially when modifying the `segment` section in the YAML or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I want to see if the current solution will be enough. ### Please try removing `make clean` from your workflow, once this is merged, to quickly detect possible problems with this new approach. I am also planning to make more substantial changes on our build-chain like this PR or #660 in the future. I am aiming to small incremental changes over time in case I break someone's flow or detect design flaws. I am also considering the breaking changes introduced in make 4.4, which will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a deadline to finish all the new build-chain work. --------- Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
2023-10-05 08:26:29 +00:00
-T $(BUILD_DIR)/$(1).ld \
Pass undefined_syms files to linker (#1065) Creating a PR for this, will look at updating splat in a separate PR
2024-02-18 14:57:03 +00:00
-T $(CONFIG_DIR)/undefined_syms.$(VERSION).txt \
Rename config folder to use game version
2023-02-21 01:42:33 +00:00
-T $(CONFIG_DIR)/undefined_syms_auto.$(VERSION).$(1).txt \
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
-T $(CONFIG_DIR)/undefined_funcs_auto.$(VERSION).$(1).txt
Simplify Makefile
2022-02-23 18:17:18 +00:00
endef
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
SOTNDISK_SOURCES := $(shell find tools/sotn-disk -name '*.go')
SOTNASSETS_SOURCES := $(shell find tools/sotn-assets -name '*.go')
Support Single BIN CUE Files (#1127) In CUE files `FILE` is a stateful, global declaration that applies to all following `TRACK`s. `sotn-disk` was treating `FILE` declarations as `TRACK` delimiters which would result in incorrect parsing of CUE files with a single `FILE`, but multiple tracks. Now when reading a `FILE` declaration, the path is stored and processing continues. A `TRACK` declaration will use the previously defined path, and if a previous track had been started, append that previous track to the list. This also builds `sotn-disk` using the local repository instead of pulling the latest commit from GitHub. The target depends on `sotn-disk` sources and will rebuild as necessary (or with `make ~/go/bin/sotn-disk`). As an aside, [pull/232](https://github.com/Xeeynamo/sotn-decomp/pull/232) ran into this same error, but fixes in a slightly different way that leaves side effects that may make supporting things like `INDEX` (for extracting the placeholder audio, for example) more error prone in the future. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-05-14 18:59:56 +00:00
Makefile Quality of Life Changes (#1275) The `check` recipe now has dependencies on all of the files it checks allowing them to be rebuilt if necessary. Consistent use of `dd` and suppressing `dd`'s output which noise for the replacement and padding it's doing. Run 7z in non-interactive mode to avoid confirming overwrite of files during extraction (equivalent behavior to `VERSION=us` `extract_disk`). `patch` is a phony target. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-08 17:13:03 +00:00
CHECK_FILES := $(shell cut -d' ' -f3 config/check.$(VERSION).sha)
Enhance make extract on PSX (#662) Previously if either the YAML or the symbol list were changed, we were forced to either manually delete the `asm` folder of the specific overlay or to invoke `make clean`, where the latter forced us to re-extract all the overlays via `make -j extract`. That changes with this PR. How `make` works is `target: dep1 dep2` where if one of the dependencies has the last modified date greater than the target, the rule is triggered again. Previously we were extracting overlays doing `make extract_stcen`. But since every overlay extraction generates a linker script I now changed the rules to do `make build/us/stcen.ld` to extract the same overlay. As I explained above, if either the YAML or one of the related symbols changes, the linker modified date will be older and the extraction for that overlay to be trigger again. The above allowed me to stop polluting the repo root with the linker scripts as they are now moved into `build/us` or `build/hd` depending of what you're trying to build. If you do `make extract` twice in a row you will be welcomed with a `make: Nothing to be done for 'extract'.`. There are still some instances where you **might** need `make clean` beforehand, especially when modifying the `segment` section in the YAML or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I want to see if the current solution will be enough. ### Please try removing `make clean` from your workflow, once this is merged, to quickly detect possible problems with this new approach. I am also planning to make more substantial changes on our build-chain like this PR or #660 in the future. I am aiming to small incremental changes over time in case I break someone's flow or detect design flaws. I am also considering the breaking changes introduced in make 4.4, which will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a deadline to finish all the new build-chain work. --------- Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
2023-10-05 08:26:29 +00:00
.PHONY: build
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
##@ Variables
##@
##@ VERSION the game version to build (us, hd, pspeu, saturn, pc) (Default: us)
##@
##@ Primary Targets
##@
all: ##@ (Default) build and check
Patch assembly code
2022-10-26 06:41:13 +00:00
all: build check
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
extract: ##@ split game files into assets and assembly
extract: extract_$(VERSION)
build: ##@ build game files
Integrate HD build to the CI (#375) I can only hope it will work at first try. The trick lies in `extract: extract_$(VERSION)` and `build: build_$(VERSION)` to customise different versions/ports to do different things and keep the flow the same. You can now do: ``` export VERSION=hd make build ``` or: ``` VERSION=hd make build ``` or: ``` make build_hd ``` If we are happy with this pattern we might want to do the same with the Saturn version too.
2023-07-19 17:28:50 +00:00
build: build_$(VERSION)
#1637: Add servant/tt_001.bin (#1659) Followed [the guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay) which seemed to work relatively painlessly for this overlay. I hit an issue in `make-config.py` where no duplicates were found and have added a separate commit which fixed that. Otherwise no issues with guide. ``` josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001 ✔ generating psx splat config ✔ splitting config/splat.us.tt_001.yaml ✔ adjusting files at src/servant/tt_001 ✔ disassembling matched functions ✖ finding duplicates across overlays Traceback (most recent call last): File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module> raise e File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module> make_config(args.input, args.version) File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config if found > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int' ```
2024-09-24 01:39:35 +00:00
build_us: main dra weapon ric cen dre mad no3 np3 nz0 sel st0 wrp rwrp mar tt_000 tt_001
HD WRP matching (#1421) Mostly identical to MAD and PSP. There are four very different functions from US that needs to be decompiled to be 100% complete: * [x] `e_misc.c` `EntityMessageBox` * [x] `e_misc.c` `EntityRelicOrb` * [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist * [x] `e_stage_name.c` `EntityStageNamePopup` These four very different functions are the only ones that use the JP text instead of the US one. This overlay also lacks of `BlitChar`. `EntityStageNamePopup` is very similar to the PSP counterpart. I used @joshlory scratch to match the HD part. `EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD overlay). I found some code that crossed between BETA and HD, so I just put an `#else` after `VERSION_US`. We are starting to reconstruct how the source code originally evolved across different game builds.
2024-07-20 00:05:26 +00:00
build_hd: dra $(BUILD_DIR)/WRP.BIN tt_000
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
clean: ##@ clean extracted files, assets, and build artifacts
Rename config folder to use game version
2023-02-21 01:42:33 +00:00
git clean -fdx assets/
Add TT_000 from the HD version (#1111) 100% decompiled. `US` adds some padding to have the file exactly 40KB long. The HD version does not add any padding, hence why I ended up producing `tt_000_raw.bin`. I would have used just `tt_000.bin` as a name as found in `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin`, but on Windows-based file system it would collide with `TT_000.BIN` due to the OS having case insensitive names. I modified `make clean` as I found annoying that `VERSION=hd make clean` would wipe out `us` build stuff. The only different function in HD is `ProcessEvent`, which has a weaker check. Another hint suggesting HD being older than US. `s32 _unused[26];` added enough padding in the bss section to get an :ok: . I am pretty sure it is unused data because the final binary is not aligned by any power of 2.
2024-04-04 21:28:57 +00:00
git clean -fdx asm/$(VERSION)/
git clean -fdx build/$(VERSION)/
Weapon assets and statically linked (#1470) Allow to compile weapon overlays as a single C files. The benefit is to disallow symbols to be exported by statically compile them all but the header. We would no longer need to use `OVL_EXPORT` as long as the symbols are marked as `static`. The frames data is also generated as an array of data that can be just included into their respective C files, generated by JSON files on the fly. I changed `animset` to `frameset` as the data does not represent a set of animations but rather a set of frames, where every frame has multiple sprites. The information on how the frames are played (e.g. animation) is found elsewhere. Following the full list of changes to achieve this project: * The `animset` from splat is now dummied out * The `frameset` JSON is now generated by the asset manager * The `frameset` JSON is now compiled into `src/weapon/w_0xx_y.h` * The `header.c` has been moved into `shared.h` * For simplicity I added a `#define g_Animset w_000_1`. I did not want to hack the asset manager too much to force the name to be `g_Animset` and I did not want to change all the symbols yet. * Simplified `weapon_pc.c` as we now only need the exported header * Always try to build the most up-to-date asset manager: this should avoid weird errors whenever people pull new changes As the asset manager now can accept those `config` files, now the CLI supports the following commands: * `stage extract` * `stage build` * `config extract` (NEW) * `config build` (NEW) The config structure is heavily inspired to Splat to maintain consistency and continuity.
2024-08-08 23:57:26 +00:00
git clean -fdx $(SRC_DIR)/weapon
Add HD version of the RIC overlay (#1660) I was also able to remove the weird `goto block` on `RicMain`. The HD version seems to be an older version. I am not entirely sure of the US changes, but it seem to fix a series of bugs when Richter gets grabbed by Gaibon? There's a new debug string which ints `g_Player.unk7A` being named as `run_disable_f` but I did not yet rename it.
2024-09-23 22:39:39 +00:00
git clean -fdx config/*$(VERSION)*
Function Analyzer CI integration (#417) Following the incredible work of @bismurphy in #396, this will integrate the tool in the CI. The markdown file `gh-duplicates/functions.md` will have clickable links to navigate right to the SVG call tree of the correspondent functions. --------- Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
2023-07-31 19:36:35 +00:00
git clean -fdx function_calls/
git clean -fdx sotn_calltree.txt
Speed Up `make format` (#1359) After building for all supported architectures `make format` was taking around 30s. This commit includes various changes which get that down to around 3½s on the same machine. This changes the `format` target in a few ways: * filters out files generated by CMake * runs `clang-format` in parallel * breaks `format` into 3 separate targets which can be run in parallel When building the PC version cmake creates a file to test compiler features. This file is picked up by the pattern which matches files for `clang-format`. This file took around 20s, filtering it out brought total format time to around 10s. Instead of passing a list created by a `find` command directly as the args to `clang-format` the list is now passed to `xargs` which runs several processes in parallel, each looking at 10 files (determined by experimentation). This has the added advantage of ensuring as the source list grows `ARG_MAX` won't be a concern. This brought total format time to around 6s. `format` was broken up into three targets `-src`, `-tools`, and `-symbols`. These targets can be run in parallel, bringing the total time down to 3½s. --------- Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-30 08:26:53 +00:00
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
##@
##@ Misc Targets
##@
# this help target will find targets which are followed by a comment beging with '#' '#' '@' and
# print them in a summary form. Any comments on a line by themselves with start with `#' '#' '@'
# will act as section dividers.
.PHONY: help
help: ##@ Print listing of key targets with their descriptions
@printf "\nUsage: make [VERSION=version] <target> …\n"
@grep -F -h "##@" $(MAKEFILE_LIST) | grep -F -v grep -F | sed -e 's/\\$$//' | awk 'BEGIN {FS = ":*[[:space:]]*##@[[:space:]]?"}; \
{ \
if($$2 == "") \
printf ""; \
else if($$0 ~ /^#/) \
printf "\n%s\n", $$2; \
else if($$1 == "") \
printf " %-20s%s\n", "", $$2; \
else { \
system("tput setaf 4; printf \"\n %-20s \" " $$1 "; tput sgr0"); \
printf "%s\n", $$2; \
}; \
}'
format: ##@ Format source code, clean symbols, other linting
Add AGPL license (#1565) From the [public announcement](https://discord.com/channels/1079389589950705684/1079395108501331990/1276310695964835910) on Discord: > We've been progressing very far with the project and I started wondering what would be the best course of action to protect the work of existing and future contributors. I personally consider contributions to a decompilation project as creative work. Therefore it is my desire to protect this creative work from being re-claimed by other people. This is especially true in the light of the ability of modding the existing decompiled code and create derivative work. > > I talked privately with the current most active contributors. And me, @sozud , @bismurphy and @joshlory so far agreed on pushing through the AGPLv3 license for all the decompiled code in `sotn-decomp/src`, excluding any third party code and decompiled code from the PSX SDK. But I've been wanting to ask all of you if you are fine your code to be licensed and protected under the AGPLv3 terms. > > https://raw.githubusercontent.com/Xeeynamo/sotn-decomp/89686514916cabd0dd88ae0387f749a889c19e05/src/LICENSE > > I put and all the contributors, which can be accessed to the page https://github.com/Xeeynamo/sotn-decomp/graphs/contributors . Again, this will work for existing and future contributors. In the license I put my full name as I will be responsible of any legality. And I understand not every contributor wants to disclose their identity. Please give an approval to my latest PR if you do not have any objection. If not, please write here publicly. > > PS. I understand past contributors will either not see the message above or decide to ignore it. If I do not hear from anyone of the contributors for more than one week between those who did not interact with me or the server, I will still go ahead and apply the license. It is not in my intention to ignore past contributors. But I do not want contributors who have been inactive for months or years to be a blocker. Please do not mistake this as an act of malevolence. --------- Co-authored-by: Joey Murphy <tjmurphy@mit.edu>
2024-09-06 20:14:55 +00:00
format: format-src format-tools format-symbols format-license
Speed Up `make format` (#1359) After building for all supported architectures `make format` was taking around 30s. This commit includes various changes which get that down to around 3½s on the same machine. This changes the `format` target in a few ways: * filters out files generated by CMake * runs `clang-format` in parallel * breaks `format` into 3 separate targets which can be run in parallel When building the PC version cmake creates a file to test compiler features. This file is picked up by the pattern which matches files for `clang-format`. This file took around 20s, filtering it out brought total format time to around 10s. Instead of passing a list created by a `find` command directly as the args to `clang-format` the list is now passed to `xargs` which runs several processes in parallel, each looking at 10 files (determined by experimentation). This has the added advantage of ensuring as the source list grows `ARG_MAX` won't be a concern. This brought total format time to around 6s. `format` was broken up into three targets `-src`, `-tools`, and `-symbols`. These targets can be run in parallel, bringing the total time down to 3½s. --------- Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-30 08:26:53 +00:00
format-src: bin/clang-format
Enchanced Draw Mode Lint (#1589) A few updates that didn't make it into the first round: * masks are now identified and expanded * AND-EQUALS and OR-EQUALS are supported * `sotn-lint` is run before `clang-format` to avoid having to run the `format-src` target twice to catch line breaks * `.h` files are included in `sotn-lint` since they are being used for shared code * `sotn` lint ignores the `mednafen` directory
2024-09-07 20:00:44 +00:00
cargo run --release --manifest-path ./tools/lints/sotn-lint/Cargo.toml ./src
Speed Up `make format` (#1359) After building for all supported architectures `make format` was taking around 30s. This commit includes various changes which get that down to around 3½s on the same machine. This changes the `format` target in a few ways: * filters out files generated by CMake * runs `clang-format` in parallel * breaks `format` into 3 separate targets which can be run in parallel When building the PC version cmake creates a file to test compiler features. This file is picked up by the pattern which matches files for `clang-format`. This file took around 20s, filtering it out brought total format time to around 10s. Instead of passing a list created by a `find` command directly as the args to `clang-format` the list is now passed to `xargs` which runs several processes in parallel, each looking at 10 files (determined by experimentation). This has the added advantage of ensuring as the source list grows `ARG_MAX` won't be a concern. This brought total format time to around 6s. `format` was broken up into three targets `-src`, `-tools`, and `-symbols`. These targets can be run in parallel, bringing the total time down to 3½s. --------- Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-30 08:26:53 +00:00
@# find explainer:
@# find $(SRC_DIR) $(INCLUDE_DIR) : look in src and include
@# -type d \( -name 3rd -o -name CMakeFiles \) -prune : if an entry is both a directory and 3rd or CMakeFiles
@# ignore it and don't traverse it
@# -o \( -type f \( -name '*.h' -o -name '*.c' \) \) : or if an entry is a file and named *.h or *.c, include it
@# -print0 : print only the matching entries, delimit by NULL to
@# ensure files with characters xargs uses as delimiters are
@# properly handled
find $(SRC_DIR) $(INCLUDE_DIR) \
-type d \( -name 3rd -o -name CMakeFiles \) -prune \
-o \( -type f \( -name '*.c' -o -name '*.h' \) \) \
-print0 \
| xargs -0 -n10 -P$$(nproc) bin/clang-format -i
format-tools:
Auto format python code
2023-07-25 17:31:49 +00:00
black tools/*.py
Change makefile to something the CI likes
2023-07-25 17:37:44 +00:00
black tools/splat_ext/*.py
black tools/split_jpt_yaml/*.py
Speed Up `make format` (#1359) After building for all supported architectures `make format` was taking around 30s. This commit includes various changes which get that down to around 3½s on the same machine. This changes the `format` target in a few ways: * filters out files generated by CMake * runs `clang-format` in parallel * breaks `format` into 3 separate targets which can be run in parallel When building the PC version cmake creates a file to test compiler features. This file is picked up by the pattern which matches files for `clang-format`. This file took around 20s, filtering it out brought total format time to around 10s. Instead of passing a list created by a `find` command directly as the args to `clang-format` the list is now passed to `xargs` which runs several processes in parallel, each looking at 10 files (determined by experimentation). This has the added advantage of ensuring as the source list grows `ARG_MAX` won't be a concern. This brought total format time to around 6s. `format` was broken up into three targets `-src`, `-tools`, and `-symbols`. These targets can be run in parallel, bringing the total time down to 3½s. --------- Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-30 08:26:53 +00:00
format-symbols:
Enhance formatter
2023-07-02 23:17:21 +00:00
VERSION=us $(PYTHON) ./tools/symbols.py sort
VERSION=hd $(PYTHON) ./tools/symbols.py sort
More TT_000 PSP functions matching (#1115)
2024-04-23 19:06:05 +00:00
VERSION=pspeu $(PYTHON) ./tools/symbols.py sort
Remove orphan symbols (#595) Allows to not hard code the location in-memory of decompiled functions and imported data if not required. This allows to relocate the referenced symbols when editing the original code or game data. If you were getting the offset of those symbols from the symbol list in `config/` I suggest to use the built `build/us/*.map` file instead.
2023-09-16 09:40:40 +00:00
./tools/symbols.py remove-orphans config/splat.us.dra.yaml
./tools/symbols.py remove-orphans config/splat.hd.dra.yaml
./tools/symbols.py remove-orphans config/splat.us.ric.yaml
Add HD version of the RIC overlay (#1660) I was also able to remove the weird `goto block` on `RicMain`. The HD version seems to be an older version. I am not entirely sure of the US changes, but it seem to fix a series of bugs when Richter gets grabbed by Gaibon? There's a new debug string which ints `g_Player.unk7A` being named as `run_disable_f` but I did not yet rename it.
2024-09-23 22:39:39 +00:00
./tools/symbols.py remove-orphans config/splat.hd.ric.yaml
Remove orphan symbols (#595) Allows to not hard code the location in-memory of decompiled functions and imported data if not required. This allows to relocate the referenced symbols when editing the original code or game data. If you were getting the offset of those symbols from the symbol list in `config/` I suggest to use the built `build/us/*.map` file instead.
2023-09-16 09:40:40 +00:00
./tools/symbols.py remove-orphans config/splat.us.stcen.yaml
./tools/symbols.py remove-orphans config/splat.us.stdre.yaml
./tools/symbols.py remove-orphans config/splat.us.stno3.yaml
./tools/symbols.py remove-orphans config/splat.us.stnp3.yaml
./tools/symbols.py remove-orphans config/splat.us.stnz0.yaml
./tools/symbols.py remove-orphans config/splat.us.stsel.yaml
./tools/symbols.py remove-orphans config/splat.us.stst0.yaml
./tools/symbols.py remove-orphans config/splat.us.stwrp.yaml
HD WRP matching (#1421) Mostly identical to MAD and PSP. There are four very different functions from US that needs to be decompiled to be 100% complete: * [x] `e_misc.c` `EntityMessageBox` * [x] `e_misc.c` `EntityRelicOrb` * [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist * [x] `e_stage_name.c` `EntityStageNamePopup` These four very different functions are the only ones that use the JP text instead of the US one. This overlay also lacks of `BlitChar`. `EntityStageNamePopup` is very similar to the PSP counterpart. I used @joshlory scratch to match the HD part. `EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD overlay). I found some code that crossed between BETA and HD, so I just put an `#else` after `VERSION_US`. We are starting to reconstruct how the source code originally evolved across different game builds.
2024-07-20 00:05:26 +00:00
./tools/symbols.py remove-orphans config/splat.hd.stwrp.yaml
Remove orphan symbols (#595) Allows to not hard code the location in-memory of decompiled functions and imported data if not required. This allows to relocate the referenced symbols when editing the original code or game data. If you were getting the offset of those symbols from the symbol list in `config/` I suggest to use the built `build/us/*.map` file instead.
2023-09-16 09:40:40 +00:00
./tools/symbols.py remove-orphans config/splat.us.strwrp.yaml
Add BOSS/MAR.BIN (#1608) Small PR to enable #557 and to let be used as an example for [our wiki](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay). This was done with the new `make-config.py` tool.
2024-09-15 20:49:56 +00:00
./tools/symbols.py remove-orphans config/splat.us.bomar.yaml
Remove orphan symbols (#595) Allows to not hard code the location in-memory of decompiled functions and imported data if not required. This allows to relocate the referenced symbols when editing the original code or game data. If you were getting the offset of those symbols from the symbol list in `config/` I suggest to use the built `build/us/*.map` file instead.
2023-09-16 09:40:40 +00:00
./tools/symbols.py remove-orphans config/splat.us.tt_000.yaml
HD WRP matching (#1421) Mostly identical to MAD and PSP. There are four very different functions from US that needs to be decompiled to be 100% complete: * [x] `e_misc.c` `EntityMessageBox` * [x] `e_misc.c` `EntityRelicOrb` * [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist * [x] `e_stage_name.c` `EntityStageNamePopup` These four very different functions are the only ones that use the JP text instead of the US one. This overlay also lacks of `BlitChar`. `EntityStageNamePopup` is very similar to the PSP counterpart. I used @joshlory scratch to match the HD part. `EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD overlay). I found some code that crossed between BETA and HD, so I just put an `#else` after `VERSION_US`. We are starting to reconstruct how the source code originally evolved across different game builds.
2024-07-20 00:05:26 +00:00
./tools/symbols.py remove-orphans config/splat.hd.tt_000.yaml
#1637: Add servant/tt_001.bin (#1659) Followed [the guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay) which seemed to work relatively painlessly for this overlay. I hit an issue in `make-config.py` where no duplicates were found and have added a separate commit which fixed that. Otherwise no issues with guide. ``` josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001 ✔ generating psx splat config ✔ splitting config/splat.us.tt_001.yaml ✔ adjusting files at src/servant/tt_001 ✔ disassembling matched functions ✖ finding duplicates across overlays Traceback (most recent call last): File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module> raise e File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module> make_config(args.input, args.version) File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config if found > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int' ```
2024-09-24 01:39:35 +00:00
./tools/symbols.py remove-orphans config/splat.us.tt_001.yaml
Remove orphan symbols (#595) Allows to not hard code the location in-memory of decompiled functions and imported data if not required. This allows to relocate the referenced symbols when editing the original code or game data. If you were getting the offset of those symbols from the symbol list in `config/` I suggest to use the built `build/us/*.map` file instead.
2023-09-16 09:40:40 +00:00
./tools/symbols.py remove-orphans config/splat.us.stmad.yaml
Add AGPL license (#1565) From the [public announcement](https://discord.com/channels/1079389589950705684/1079395108501331990/1276310695964835910) on Discord: > We've been progressing very far with the project and I started wondering what would be the best course of action to protect the work of existing and future contributors. I personally consider contributions to a decompilation project as creative work. Therefore it is my desire to protect this creative work from being re-claimed by other people. This is especially true in the light of the ability of modding the existing decompiled code and create derivative work. > > I talked privately with the current most active contributors. And me, @sozud , @bismurphy and @joshlory so far agreed on pushing through the AGPLv3 license for all the decompiled code in `sotn-decomp/src`, excluding any third party code and decompiled code from the PSX SDK. But I've been wanting to ask all of you if you are fine your code to be licensed and protected under the AGPLv3 terms. > > https://raw.githubusercontent.com/Xeeynamo/sotn-decomp/89686514916cabd0dd88ae0387f749a889c19e05/src/LICENSE > > I put and all the contributors, which can be accessed to the page https://github.com/Xeeynamo/sotn-decomp/graphs/contributors . Again, this will work for existing and future contributors. In the license I put my full name as I will be responsible of any legality. And I understand not every contributor wants to disclose their identity. Please give an approval to my latest PR if you do not have any objection. If not, please write here publicly. > > PS. I understand past contributors will either not see the message above or decide to ignore it. If I do not hear from anyone of the contributors for more than one week between those who did not interact with me or the server, I will still go ahead and apply the license. It is not in my intention to ignore past contributors. But I do not want contributors who have been inactive for months or years to be a blocker. Please do not mistake this as an act of malevolence. --------- Co-authored-by: Joey Murphy <tjmurphy@mit.edu>
2024-09-06 20:14:55 +00:00
format-license:
find src/ | grep -E '\.c$$|\.h$$' | grep -vE 'PsyCross|mednafen|psxsdk|3rd|saturn/lib' | python3 ./tools/lint-license.py - AGPL-3.0-or-later
python3 ./tools/lint-license.py include/game.h AGPL-3.0-or-later
python3 ./tools/lint-license.py include/entity.h AGPL-3.0-or-later
python3 ./tools/lint-license.py include/items.h AGPL-3.0-or-later
python3 ./tools/lint-license.py include/lba.h AGPL-3.0-or-later
python3 ./tools/lint-license.py include/memcard.h AGPL-3.0-or-later
Speed Up `make format` (#1359) After building for all supported architectures `make format` was taking around 30s. This commit includes various changes which get that down to around 3½s on the same machine. This changes the `format` target in a few ways: * filters out files generated by CMake * runs `clang-format` in parallel * breaks `format` into 3 separate targets which can be run in parallel When building the PC version cmake creates a file to test compiler features. This file is picked up by the pattern which matches files for `clang-format`. This file took around 20s, filtering it out brought total format time to around 10s. Instead of passing a list created by a `find` command directly as the args to `clang-format` the list is now passed to `xargs` which runs several processes in parallel, each looking at 10 files (determined by experimentation). This has the added advantage of ensuring as the source list grows `ARG_MAX` won't be a concern. This brought total format time to around 6s. `format` was broken up into three targets `-src`, `-tools`, and `-symbols`. These targets can be run in parallel, bringing the total time down to 3½s. --------- Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-30 08:26:53 +00:00
# fast-format
ff: MAKEFLAGS += --jobs
ff:
$(MAKE) format
Import 4A538, demo and 5087C data (#835) As per title. I also added a new tool called `dirt_patch`. As I mentioned in our Discord server there is some left-over data from previous dev builds in DRA.BIN and potentially elsewhere too. The tool uses the new file `config/dirt.us.json` which stores a list of patches to avoid crazy hacks and `#ifdef` for the sake of getting a match. I hope this tool will not be abused.
2023-12-12 18:36:34 +00:00
patch:
$(DIRT_PATCHER) config/dirt.$(VERSION).json
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
check: ##@ compare built files to original game files
Makefile Quality of Life Changes (#1275) The `check` recipe now has dependencies on all of the files it checks allowing them to be rebuilt if necessary. Consistent use of `dd` and suppressing `dd`'s output which noise for the replacement and padding it's doing. Run 7z in non-interactive mode to avoid confirming overwrite of files during extraction (equivalent behavior to `VERSION=us` `extract_disk`). `patch` is a phony target. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-08 17:13:03 +00:00
check: config/check.$(VERSION).sha patch $(CHECK_FILES)
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
@$(SHASUM) --check $< | awk 'BEGIN{ FS=": " }; { \
printf "%s\t[ ", $$1; \
if ($$2 == "OK") \
color = 28; \
else \
color = 196; \
system("tput setaf " color "; printf " $$2 "; tput sgr0"); \
printf " ]\n"; \
Fix Check Output, Executable make-config.py (#1668) Fixed a tab matching issue with the `check` target. These were matching '\' and 't' as separator characters rather than '↹' (horizontal tab) Made `tools/make-config.py` executable.
2024-09-24 18:10:38 +00:00
}' | column --separator $$'\t' --table
Add rule to create the 'expected' folder for asm-differ
2022-10-23 16:55:36 +00:00
expected: check
Integrate HD build to the CI (#375) I can only hope it will work at first try. The trick lies in `extract: extract_$(VERSION)` and `build: build_$(VERSION)` to customise different versions/ports to do different things and keep the flow the same. You can now do: ``` export VERSION=hd make build ``` or: ``` VERSION=hd make build ``` or: ``` make build_hd ``` If we are happy with this pattern we might want to do the same with the Saturn version too.
2023-07-19 17:28:50 +00:00
mkdir -p expected/build
rm -rf expected/build/$(VERSION)
cp -r build/$(VERSION) expected/build/
Upload disassembly
2022-01-01 17:45:41 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
main: $(MAIN_TARGET).exe
Upload disassembly
2022-01-01 17:45:41 +00:00
main_dirs:
$(foreach dir,$(MAIN_ASM_DIRS) $(MAIN_SRC_DIRS),$(shell mkdir -p $(BUILD_DIR)/$(dir)))
$(MAIN_TARGET).exe: $(MAIN_TARGET).elf
Update splat right before 0.11.0
2022-10-16 09:31:54 +00:00
$(OBJCOPY) -O binary $< $@
Replace BIOS Trampolines (#1133) This replaces several dozen BIOS trampolines from the extracted ASM to "decompiled" source. These are modeled on the `INCLUDE_ASM` macro, but generate the instructions necessary for each trampoline directly instead of importing an extracted source file. Because these trampolines never return, and GCC 2.6 doesnt appear to have builtins for leaving off the return jump postamble, these will likely need to remain assembly. This also changes the `main.elf` target to depend on `main.ld`, and undefined symbols files, allowing `make build` to regenerate those files if necessary. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-05-17 07:45:26 +00:00
$(MAIN_TARGET).elf: $(MAIN_O_FILES) $(BUILD_DIR)/main.ld $(CONFIG_DIR)/undefined_syms.$(VERSION).txt $(CONFIG_DIR)/undefined_syms_auto.$(VERSION).$(MAIN).txt
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
$(LD) $(LD_FLAGS) -o $@ \
Upload disassembly
2022-01-01 17:45:41 +00:00
-Map $(MAIN_TARGET).map \
Enhance make extract on PSX (#662) Previously if either the YAML or the symbol list were changed, we were forced to either manually delete the `asm` folder of the specific overlay or to invoke `make clean`, where the latter forced us to re-extract all the overlays via `make -j extract`. That changes with this PR. How `make` works is `target: dep1 dep2` where if one of the dependencies has the last modified date greater than the target, the rule is triggered again. Previously we were extracting overlays doing `make extract_stcen`. But since every overlay extraction generates a linker script I now changed the rules to do `make build/us/stcen.ld` to extract the same overlay. As I explained above, if either the YAML or one of the related symbols changes, the linker modified date will be older and the extraction for that overlay to be trigger again. The above allowed me to stop polluting the repo root with the linker scripts as they are now moved into `build/us` or `build/hd` depending of what you're trying to build. If you do `make extract` twice in a row you will be welcomed with a `make: Nothing to be done for 'extract'.`. There are still some instances where you **might** need `make clean` beforehand, especially when modifying the `segment` section in the YAML or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I want to see if the current solution will be enough. ### Please try removing `make clean` from your workflow, once this is merged, to quickly detect possible problems with this new approach. I am also planning to make more substantial changes on our build-chain like this PR or #660 in the future. I am aiming to small incremental changes over time in case I break someone's flow or detect design flaws. I am also considering the breaking changes introduced in make 4.4, which will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a deadline to finish all the new build-chain work. --------- Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
2023-10-05 08:26:29 +00:00
-T $(BUILD_DIR)/main.ld \
Update splat (#247)
2023-06-12 05:04:25 +00:00
-T $(CONFIG_DIR)/undefined_syms.$(VERSION).txt \
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
-T $(CONFIG_DIR)/undefined_syms_auto.$(VERSION).$(MAIN).txt
Upload disassembly
2022-01-01 17:45:41 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
dra: $(BUILD_DIR)/DRA.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/DRA.BIN: $(BUILD_DIR)/$(DRA).elf
Upload disassembly
2022-01-01 17:45:41 +00:00
$(OBJCOPY) -O binary $< $@
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/$(DRA).elf: $(call list_o_files,dra)
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
echo $(call list_o_files,dra)
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(call link,dra,$@)
Upload disassembly
2022-01-01 17:45:41 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
ric: $(BUILD_DIR)/RIC.BIN
Add RIC.BIN
2022-03-07 22:46:43 +00:00
$(BUILD_DIR)/RIC.BIN: $(BUILD_DIR)/ric.elf
$(OBJCOPY) -O binary $< $@
$(BUILD_DIR)/ric.elf: $(call list_o_files,ric)
$(call link,ric,$@)
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
cen: $(BUILD_DIR)/CEN.BIN $(BUILD_DIR)/F_CEN.BIN
CEN overlay extracted (#19)
2022-10-28 22:46:47 +00:00
$(BUILD_DIR)/CEN.BIN: $(BUILD_DIR)/stcen.elf
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_CEN.BIN:
$(GFXSTAGE) e assets/st/cen $@
CEN overlay extracted (#19)
2022-10-28 22:46:47 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
dre: $(BUILD_DIR)/DRE.BIN $(BUILD_DIR)/F_DRE.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/DRE.BIN: $(BUILD_DIR)/stdre.elf
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_DRE.BIN:
$(GFXSTAGE) e assets/st/dre $@
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
mad: $(BUILD_DIR)/MAD.BIN $(BUILD_DIR)/F_MAD.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/MAD.BIN: $(BUILD_DIR)/stmad.elf
Add two map files
2022-01-20 22:02:05 +00:00
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_MAD.BIN:
$(GFXSTAGE) e assets/st/mad $@
Add two map files
2022-01-20 22:02:05 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
no3: $(BUILD_DIR)/NO3.BIN $(BUILD_DIR)/F_NO3.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/NO3.BIN: $(BUILD_DIR)/stno3.elf
Add two map files
2022-01-20 22:02:05 +00:00
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_NO3.BIN:
$(GFXSTAGE) e assets/st/no3 $@
Add two map files
2022-01-20 22:02:05 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
np3: $(BUILD_DIR)/NP3.BIN $(BUILD_DIR)/F_NP3.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/NP3.BIN: $(BUILD_DIR)/stnp3.elf
Add NP3.BIN
2022-02-05 17:16:17 +00:00
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_NP3.BIN:
$(GFXSTAGE) e assets/st/np3 $@
Add NP3.BIN
2022-02-05 17:16:17 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
nz0: $(BUILD_DIR)/NZ0.BIN $(BUILD_DIR)/F_NZ0.BIN
NZ0 Extracted + File descriptions (#20)
2022-10-29 17:58:02 +00:00
$(BUILD_DIR)/NZ0.BIN: $(BUILD_DIR)/stnz0.elf
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_NZ0.BIN:
$(GFXSTAGE) e assets/st/nz0 $@
NZ0 Extracted + File descriptions (#20)
2022-10-29 17:58:02 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
sel: $(BUILD_DIR)/SEL.BIN
Add SEL overlay
2022-12-21 16:33:15 +00:00
$(BUILD_DIR)/SEL.BIN: $(BUILD_DIR)/stsel.elf
$(OBJCOPY) -O binary $< $@
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
st0: $(BUILD_DIR)/ST0.BIN $(BUILD_DIR)/F_ST0.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/ST0.BIN: $(BUILD_DIR)/stst0.elf
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_ST0.BIN:
$(GFXSTAGE) e assets/st/st0 $@
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
wrp: $(BUILD_DIR)/WRP.BIN $(BUILD_DIR)/F_WRP.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/WRP.BIN: $(BUILD_DIR)/stwrp.elf
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
$(OBJCOPY) -O binary $< $@
Extract WRP stage graphics
2023-03-10 03:59:33 +00:00
$(BUILD_DIR)/F_WRP.BIN:
$(GFXSTAGE) e assets/st/wrp $@
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
rwrp: $(BUILD_DIR)/RWRP.BIN $(BUILD_DIR)/F_RWRP.BIN
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(BUILD_DIR)/RWRP.BIN: $(BUILD_DIR)/strwrp.elf
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
$(OBJCOPY) -O binary $< $@
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
$(BUILD_DIR)/F_RWRP.BIN:
$(GFXSTAGE) e assets/st/rwrp $@
Disassemble DRE, ST0, WRP and RWRP
2022-02-09 12:49:33 +00:00
Add BOSS/MAR.BIN (#1608) Small PR to enable #557 and to let be used as an example for [our wiki](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay). This was done with the new `make-config.py` tool.
2024-09-15 20:49:56 +00:00
mar: $(BUILD_DIR)/MAR.BIN $(BUILD_DIR)/F_MAR.BIN
$(BUILD_DIR)/MAR.BIN: $(BUILD_DIR)/bomar.elf
$(OBJCOPY) -O binary $< $@
$(BUILD_DIR)/F_MAR.BIN:
$(GFXSTAGE) e assets/boss/mar $@
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
tt_000: $(BUILD_DIR)/TT_000.BIN
Add TT_000 from the HD version (#1111) 100% decompiled. `US` adds some padding to have the file exactly 40KB long. The HD version does not add any padding, hence why I ended up producing `tt_000_raw.bin`. I would have used just `tt_000.bin` as a name as found in `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin`, but on Windows-based file system it would collide with `TT_000.BIN` due to the OS having case insensitive names. I modified `make clean` as I found annoying that `VERSION=hd make clean` would wipe out `us` build stuff. The only different function in HD is `ProcessEvent`, which has a weaker check. Another hint suggesting HD being older than US. `s32 _unused[26];` added enough padding in the bss section to get an :ok: . I am pretty sure it is unused data because the final binary is not aligned by any power of 2.
2024-04-04 21:28:57 +00:00
$(BUILD_DIR)/tt_000_raw.bin: $(BUILD_DIR)/tt_000.elf
Add SERVANT/TT_000.BIN overlay
2023-02-08 14:03:44 +00:00
$(OBJCOPY) -O binary $< $@
Add TT_000 from the HD version (#1111) 100% decompiled. `US` adds some padding to have the file exactly 40KB long. The HD version does not add any padding, hence why I ended up producing `tt_000_raw.bin`. I would have used just `tt_000.bin` as a name as found in `disks/pspeu/PSP_GAME/USRDIR/res/ps/hdbin/tt_000.bin`, but on Windows-based file system it would collide with `TT_000.BIN` due to the OS having case insensitive names. I modified `make clean` as I found annoying that `VERSION=hd make clean` would wipe out `us` build stuff. The only different function in HD is `ProcessEvent`, which has a weaker check. Another hint suggesting HD being older than US. `s32 _unused[26];` added enough padding in the bss section to get an :ok: . I am pretty sure it is unused data because the final binary is not aligned by any power of 2.
2024-04-04 21:28:57 +00:00
$(BUILD_DIR)/TT_000.BIN: $(BUILD_DIR)/tt_000_raw.bin
cp $< $@
Makefile Quality of Life Changes (#1275) The `check` recipe now has dependencies on all of the files it checks allowing them to be rebuilt if necessary. Consistent use of `dd` and suppressing `dd`'s output which noise for the replacement and padding it's doing. Run 7z in non-interactive mode to avoid confirming overwrite of files during extraction (equivalent behavior to `VERSION=us` `extract_disk`). `patch` is a phony target. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-08 17:13:03 +00:00
dd status=none if=/dev/zero bs=1 count=$$((40960 - $$(stat -c %s $<))) >> $@
Add SERVANT/TT_000.BIN overlay
2023-02-08 14:03:44 +00:00
#1637: Add servant/tt_001.bin (#1659) Followed [the guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay) which seemed to work relatively painlessly for this overlay. I hit an issue in `make-config.py` where no duplicates were found and have added a separate commit which fixed that. Otherwise no issues with guide. ``` josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001 ✔ generating psx splat config ✔ splitting config/splat.us.tt_001.yaml ✔ adjusting files at src/servant/tt_001 ✔ disassembling matched functions ✖ finding duplicates across overlays Traceback (most recent call last): File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module> raise e File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module> make_config(args.input, args.version) File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config if found > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int' ```
2024-09-24 01:39:35 +00:00
tt_001: $(BUILD_DIR)/TT_001.BIN
$(BUILD_DIR)/tt_001_raw.bin: $(BUILD_DIR)/tt_001.elf
$(OBJCOPY) -O binary $< $@
$(BUILD_DIR)/TT_001.BIN: $(BUILD_DIR)/tt_001_raw.bin
cp $< $@
dd status=none if=/dev/zero bs=1 count=$$((40960 - $$(stat -c %s $<))) >> $@
Start sharing some common stage code (#464) Following up #454 , #453 and related from @sozud , I started to convert some previously defined headers into proper shared C code. The advantage from the header is the C file is completely independent, it will have its own `.data` and `.rodata` and it will be compiled once. I also took the opportunity to move `include/st/AnimateEntity.h` to `src/st/animate_entity.h` for consistency. I think the change is small enough to be included as part of this PR.
2023-09-03 06:55:48 +00:00
mad_fix: stmad_dirs $$(call list_o_files,st/mad) $$(call list_o_files,st)
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
$(LD) $(LD_FLAGS) -o $(BUILD_DIR)/stmad_fix.elf \
MAD reworked symbols and patches from the ground-up
2023-02-19 18:15:49 +00:00
-Map $(BUILD_DIR)/stmad_fix.map \
Enhance make extract on PSX (#662) Previously if either the YAML or the symbol list were changed, we were forced to either manually delete the `asm` folder of the specific overlay or to invoke `make clean`, where the latter forced us to re-extract all the overlays via `make -j extract`. That changes with this PR. How `make` works is `target: dep1 dep2` where if one of the dependencies has the last modified date greater than the target, the rule is triggered again. Previously we were extracting overlays doing `make extract_stcen`. But since every overlay extraction generates a linker script I now changed the rules to do `make build/us/stcen.ld` to extract the same overlay. As I explained above, if either the YAML or one of the related symbols changes, the linker modified date will be older and the extraction for that overlay to be trigger again. The above allowed me to stop polluting the repo root with the linker scripts as they are now moved into `build/us` or `build/hd` depending of what you're trying to build. If you do `make extract` twice in a row you will be welcomed with a `make: Nothing to be done for 'extract'.`. There are still some instances where you **might** need `make clean` beforehand, especially when modifying the `segment` section in the YAML or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I want to see if the current solution will be enough. ### Please try removing `make clean` from your workflow, once this is merged, to quickly detect possible problems with this new approach. I am also planning to make more substantial changes on our build-chain like this PR or #660 in the future. I am aiming to small incremental changes over time in case I break someone's flow or detect design flaws. I am also considering the breaking changes introduced in make 4.4, which will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a deadline to finish all the new build-chain work. --------- Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
2023-10-05 08:26:29 +00:00
-T $(BUILD_DIR)/stmad.ld \
Update splat (#247)
2023-06-12 05:04:25 +00:00
-T $(CONFIG_DIR)/undefined_syms.$(VERSION).txt \
MAD reworked symbols and patches from the ground-up
2023-02-19 18:15:49 +00:00
-T $(CONFIG_DIR)/undefined_syms_auto.stmad.txt \
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
-T $(CONFIG_DIR)/undefined_funcs_auto.stmad.txt
MAD reworked symbols and patches from the ground-up
2023-02-19 18:15:49 +00:00
$(OBJCOPY) -O binary $(BUILD_DIR)/stmad_fix.elf $(BUILD_DIR)/MAD.BIN
Compile partially fixed debug room
2022-03-05 02:45:54 +00:00
Add SERVANT/TT_000.BIN overlay
2023-02-08 14:03:44 +00:00
tt_%_dirs:
$(foreach dir,$(ASM_DIR)/servant/tt_$* $(ASM_DIR)/servant/tt_$*/data $(SRC_DIR)/servant/tt_$* $(ASSETS_DIR)/servant/tt_$*,$(shell mkdir -p $(BUILD_DIR)/$(dir)))
Add BOSS/MAR.BIN (#1608) Small PR to enable #557 and to let be used as an example for [our wiki](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay). This was done with the new `make-config.py` tool.
2024-09-15 20:49:56 +00:00
bo%_dirs:
$(foreach dir,$(ASM_DIR)/boss/$* $(ASM_DIR)/boss/$*/data $(ASM_DIR)/boss/$*/handwritten $(SRC_DIR)/boss/$* $(ASSETS_DIR)/boss/$*,$(shell mkdir -p $(BUILD_DIR)/$(dir)))
Simplify Makefile
2022-02-23 18:17:18 +00:00
st%_dirs:
Mark SEL handwritten functions as asm (#977)
2024-01-14 19:27:34 +00:00
$(foreach dir,$(ASM_DIR)/st/$* $(ASM_DIR)/st/$*/data $(ASM_DIR)/st/$*/handwritten $(SRC_DIR)/st/$* $(ASSETS_DIR)/st/$*,$(shell mkdir -p $(BUILD_DIR)/$(dir)))
Simplify Makefile
2022-02-23 18:17:18 +00:00
%_dirs:
Asset extraction and build experimentation
2023-02-03 15:38:48 +00:00
$(foreach dir,$(ASM_DIR)/$* $(ASM_DIR)/$*/data $(SRC_DIR)/$* $(ASSETS_DIR)/$*,$(shell mkdir -p $(BUILD_DIR)/$(dir)))
Simplify Makefile
2022-02-23 18:17:18 +00:00
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
$(BUILD_DIR)/tt_%.elf: $(BUILD_DIR)/tt_%.ld $$(call list_o_files,servant/tt_$$*) | tt_%_dirs
Add SERVANT/TT_000.BIN overlay
2023-02-08 14:03:44 +00:00
$(call link,tt_$*,$@)
Start sharing some common stage code (#464) Following up #454 , #453 and related from @sozud , I started to convert some previously defined headers into proper shared C code. The advantage from the header is the C file is completely independent, it will have its own `.data` and `.rodata` and it will be compiled once. I also took the opportunity to move `include/st/AnimateEntity.h` to `src/st/animate_entity.h` for consistency. I think the change is small enough to be included as part of this PR.
2023-09-03 06:55:48 +00:00
$(BUILD_DIR)/stmad.elf: $$(call list_o_files,st/mad) $$(call list_shared_o_files,st)
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
$(LD) $(LD_FLAGS) -o $@ \
MAD reworked symbols and patches from the ground-up
2023-02-19 18:15:49 +00:00
-Map $(BUILD_DIR)/stmad.map \
Enhance make extract on PSX (#662) Previously if either the YAML or the symbol list were changed, we were forced to either manually delete the `asm` folder of the specific overlay or to invoke `make clean`, where the latter forced us to re-extract all the overlays via `make -j extract`. That changes with this PR. How `make` works is `target: dep1 dep2` where if one of the dependencies has the last modified date greater than the target, the rule is triggered again. Previously we were extracting overlays doing `make extract_stcen`. But since every overlay extraction generates a linker script I now changed the rules to do `make build/us/stcen.ld` to extract the same overlay. As I explained above, if either the YAML or one of the related symbols changes, the linker modified date will be older and the extraction for that overlay to be trigger again. The above allowed me to stop polluting the repo root with the linker scripts as they are now moved into `build/us` or `build/hd` depending of what you're trying to build. If you do `make extract` twice in a row you will be welcomed with a `make: Nothing to be done for 'extract'.`. There are still some instances where you **might** need `make clean` beforehand, especially when modifying the `segment` section in the YAML or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I want to see if the current solution will be enough. ### Please try removing `make clean` from your workflow, once this is merged, to quickly detect possible problems with this new approach. I am also planning to make more substantial changes on our build-chain like this PR or #660 in the future. I am aiming to small incremental changes over time in case I break someone's flow or detect design flaws. I am also considering the breaking changes introduced in make 4.4, which will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a deadline to finish all the new build-chain work. --------- Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
2023-10-05 08:26:29 +00:00
-T $(BUILD_DIR)/stmad.ld \
Update splat (#247)
2023-06-12 05:04:25 +00:00
-T $(CONFIG_DIR)/undefined_syms.beta.txt \
MAD reworked symbols and patches from the ground-up
2023-02-19 18:15:49 +00:00
-T $(CONFIG_DIR)/undefined_syms_auto.stmad.txt \
Do not strip debug symbols and start to support modern GCC (#376) A few tweaks to the Makefile: * `PSXCC_FLAGS` now contains all the GCC flags that only work with the old versions * Add debug symbols in the PSX ELFs by adding `-g` and by removing `-s` * Remove `--no-check-sections` as the reallocation would break the executable if we add too much data I did not remove `-s` from the Saturn version. We might want to do that but we also need to add the `-g` flag in `tools/saturn-splitter/scripts/compile_dosemu/compile_dosemu.sh` (cc. @sozud)
2023-07-19 18:45:25 +00:00
-T $(CONFIG_DIR)/undefined_funcs_auto.stmad.txt
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
$(BUILD_DIR)/stsel.elf: $$(call list_o_files,st/sel) $$(call list_shared_o_files,st)
$(call link,stsel,$@)
$(BUILD_DIR)/st%.elf: $$(call list_st_o_files,st/$$*) $$(call list_shared_o_files,st)
Simplify Makefile
2022-02-23 18:17:18 +00:00
$(call link,st$*,$@)
Add BOSS/MAR.BIN (#1608) Small PR to enable #557 and to let be used as an example for [our wiki](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay). This was done with the new `make-config.py` tool.
2024-09-15 20:49:56 +00:00
$(BUILD_DIR)/bo%.elf: $$(call list_st_o_files,boss/$$*) $$(call list_shared_o_files,boss)
$(call link,bo$*,$@)
Simplify Makefile
2022-02-23 18:17:18 +00:00
Enhance Makefile for WEAPON0.BIN (#466) ## What is this Required for #465 . I re-wrote how the build process works for the `BIN/WEAPON0.BIN` ~~overlay~~ file. ## A bit of a background On the PSP build I found `WEAPON0.BIN` is just a blob with `f0_000.bin`, `w0_000.bin`, `f0_001.bin` and so on concatenated as a single file. `f0` contains the graphics (or CHR) and `w0` contains the actual overlay (or PRG). The individual overlays share a great amount of functions I documented in `config/symbols.us.weapon.txt`. ## The problem Splat creates its own `weapon.ld` linker script, insisting to link all the overlays as a single binary file. While this is not technically correct, it creates a bigger problem: symbols cannot be shared. In 58 overlays there are 58 `DestroyEntity`. Other decomps are circumnavigating the issue by adding a prefix. Following the same pattern we could have `W000_DestroyEntity`, `W001_DestroyEntity` and so on. But this is not maintainable and it does not reflect our long-term vision of having shared code between overlays. ## The solution I created the dedicated linker script `weapon0.ld` to output the individual overlay `w0_000.bin` and similar. `f0_000.bin` is also created from the correspondent PNG. Once all the overlays and graphics are created I am simply concatenating them back to `WEAPON0.BIN`. This is much more efficient in terms of separation of concerns, maintainability, build speed and it also allows to use the same symbol names across overlays. Also very important, the generated individual overlays matches the structure of what is found in the PSP build. ## Side note I thought it was wise to prepare the Makefile and the linker script to also allow rebasing and compiling for WEAPON1.BIN. The generated file does not currently match and it will probably need a bit more research. I left the build infrastructure there. To compile the non-matching WEAPON1 just modify `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES)` to `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES) $(WEAPON1_FILES)`.
2023-08-14 01:03:54 +00:00
# Weapon overlays
WEAPON0_FILES := $(foreach num,$(shell seq -w 000 058),$(BUILD_DIR)/weapon/f0_$(num).bin $(BUILD_DIR)/weapon/w0_$(num).bin)
WEAPON1_FILES := $(foreach num,$(shell seq -w 000 058),$(BUILD_DIR)/weapon/f1_$(num).bin $(BUILD_DIR)/weapon/w1_$(num).bin)
WEAPON_DIRS := $(BUILD_DIR)/$(ASSETS_DIR)/weapon $(BUILD_DIR)/$(ASM_DIR)/weapon/data $(BUILD_DIR)/$(SRC_DIR)/weapon $(BUILD_DIR)/weapon
weapon: $(WEAPON_DIRS) $(BUILD_DIR)/WEAPON0.BIN
$(WEAPON_DIRS):
@mkdir -p $@
$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES)
cat $^ > $@
$(BUILD_DIR)/weapon/f%.bin: $(BUILD_DIR)/weapon/f%.elf
$(OBJCOPY) -O binary $< $@
$(BUILD_DIR)/weapon/w%.bin: $(BUILD_DIR)/weapon/w%.elf
$(OBJCOPY) -O binary $< $@
Makefile Quality of Life Changes (#1275) The `check` recipe now has dependencies on all of the files it checks allowing them to be rebuilt if necessary. Consistent use of `dd` and suppressing `dd`'s output which noise for the replacement and padding it's doing. Run 7z in non-interactive mode to avoid confirming overwrite of files during extraction (equivalent behavior to `VERSION=us` `extract_disk`). `patch` is a phony target. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-08 17:13:03 +00:00
dd status=none if=/dev/zero of=$@ bs=1 seek=12287 count=1 conv=notrunc
Completes w_051 (#1122) All the stuff marked as `static` does not have the `g_` prepended as by definition they are not global
2024-05-12 17:31:16 +00:00
$(ASM_DIR)/weapon/data/w_%.data.s: # create a fake empty file if all the data has been imported
touch $@
$(ASM_DIR)/weapon/data/w_%.sbss.s: # create a fake empty file if all the bss section has been imported
touch $@
Weapon assets and statically linked (#1470) Allow to compile weapon overlays as a single C files. The benefit is to disallow symbols to be exported by statically compile them all but the header. We would no longer need to use `OVL_EXPORT` as long as the symbols are marked as `static`. The frames data is also generated as an array of data that can be just included into their respective C files, generated by JSON files on the fly. I changed `animset` to `frameset` as the data does not represent a set of animations but rather a set of frames, where every frame has multiple sprites. The information on how the frames are played (e.g. animation) is found elsewhere. Following the full list of changes to achieve this project: * The `animset` from splat is now dummied out * The `frameset` JSON is now generated by the asset manager * The `frameset` JSON is now compiled into `src/weapon/w_0xx_y.h` * The `header.c` has been moved into `shared.h` * For simplicity I added a `#define g_Animset w_000_1`. I did not want to hack the asset manager too much to force the name to be `g_Animset` and I did not want to change all the symbols yet. * Simplified `weapon_pc.c` as we now only need the exported header * Always try to build the most up-to-date asset manager: this should avoid weird errors whenever people pull new changes As the asset manager now can accept those `config` files, now the CLI supports the following commands: * `stage extract` * `stage build` * `config extract` (NEW) * `config build` (NEW) The config structure is heavily inspired to Splat to maintain consistency and continuity.
2024-08-08 23:57:26 +00:00
$(BUILD_DIR)/weapon/w0_%.elf: $(BUILD_DIR)/$(SRC_DIR)/weapon/w_%.c.o $(BUILD_DIR)/$(ASM_DIR)/weapon/data/w_%.data.s.o $(BUILD_DIR)/$(ASM_DIR)/weapon/data/w_%.sbss.s.o
Enhance Makefile for WEAPON0.BIN (#466) ## What is this Required for #465 . I re-wrote how the build process works for the `BIN/WEAPON0.BIN` ~~overlay~~ file. ## A bit of a background On the PSP build I found `WEAPON0.BIN` is just a blob with `f0_000.bin`, `w0_000.bin`, `f0_001.bin` and so on concatenated as a single file. `f0` contains the graphics (or CHR) and `w0` contains the actual overlay (or PRG). The individual overlays share a great amount of functions I documented in `config/symbols.us.weapon.txt`. ## The problem Splat creates its own `weapon.ld` linker script, insisting to link all the overlays as a single binary file. While this is not technically correct, it creates a bigger problem: symbols cannot be shared. In 58 overlays there are 58 `DestroyEntity`. Other decomps are circumnavigating the issue by adding a prefix. Following the same pattern we could have `W000_DestroyEntity`, `W001_DestroyEntity` and so on. But this is not maintainable and it does not reflect our long-term vision of having shared code between overlays. ## The solution I created the dedicated linker script `weapon0.ld` to output the individual overlay `w0_000.bin` and similar. `f0_000.bin` is also created from the correspondent PNG. Once all the overlays and graphics are created I am simply concatenating them back to `WEAPON0.BIN`. This is much more efficient in terms of separation of concerns, maintainability, build speed and it also allows to use the same symbol names across overlays. Also very important, the generated individual overlays matches the structure of what is found in the PSP build. ## Side note I thought it was wise to prepare the Makefile and the linker script to also allow rebasing and compiling for WEAPON1.BIN. The generated file does not currently match and it will probably need a bit more research. I left the build infrastructure there. To compile the non-matching WEAPON1 just modify `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES)` to `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES) $(WEAPON1_FILES)`.
2023-08-14 01:03:54 +00:00
$(LD) $(LD_FLAGS) --no-check-sections -o $@ \
-Map $(BUILD_DIR)/weapon/w0_$*.map \
-T weapon0.ld \
-T $(CONFIG_DIR)/undefined_syms.$(VERSION).txt \
-T $(CONFIG_DIR)/undefined_syms_auto.$(VERSION).weapon.txt \
-T $(CONFIG_DIR)/undefined_funcs_auto.$(VERSION).weapon.txt \
$^
Weapon assets and statically linked (#1470) Allow to compile weapon overlays as a single C files. The benefit is to disallow symbols to be exported by statically compile them all but the header. We would no longer need to use `OVL_EXPORT` as long as the symbols are marked as `static`. The frames data is also generated as an array of data that can be just included into their respective C files, generated by JSON files on the fly. I changed `animset` to `frameset` as the data does not represent a set of animations but rather a set of frames, where every frame has multiple sprites. The information on how the frames are played (e.g. animation) is found elsewhere. Following the full list of changes to achieve this project: * The `animset` from splat is now dummied out * The `frameset` JSON is now generated by the asset manager * The `frameset` JSON is now compiled into `src/weapon/w_0xx_y.h` * The `header.c` has been moved into `shared.h` * For simplicity I added a `#define g_Animset w_000_1`. I did not want to hack the asset manager too much to force the name to be `g_Animset` and I did not want to change all the symbols yet. * Simplified `weapon_pc.c` as we now only need the exported header * Always try to build the most up-to-date asset manager: this should avoid weird errors whenever people pull new changes As the asset manager now can accept those `config` files, now the CLI supports the following commands: * `stage extract` * `stage build` * `config extract` (NEW) * `config build` (NEW) The config structure is heavily inspired to Splat to maintain consistency and continuity.
2024-08-08 23:57:26 +00:00
$(BUILD_DIR)/weapon/w1_%.elf: $(BUILD_DIR)/$(SRC_DIR)/weapon/w_%.c.o $(BUILD_DIR)/$(ASM_DIR)/weapon/data/w_%.data.s.o $(BUILD_DIR)/$(ASM_DIR)/weapon/data/w_%.sbss.s.o
Enhance Makefile for WEAPON0.BIN (#466) ## What is this Required for #465 . I re-wrote how the build process works for the `BIN/WEAPON0.BIN` ~~overlay~~ file. ## A bit of a background On the PSP build I found `WEAPON0.BIN` is just a blob with `f0_000.bin`, `w0_000.bin`, `f0_001.bin` and so on concatenated as a single file. `f0` contains the graphics (or CHR) and `w0` contains the actual overlay (or PRG). The individual overlays share a great amount of functions I documented in `config/symbols.us.weapon.txt`. ## The problem Splat creates its own `weapon.ld` linker script, insisting to link all the overlays as a single binary file. While this is not technically correct, it creates a bigger problem: symbols cannot be shared. In 58 overlays there are 58 `DestroyEntity`. Other decomps are circumnavigating the issue by adding a prefix. Following the same pattern we could have `W000_DestroyEntity`, `W001_DestroyEntity` and so on. But this is not maintainable and it does not reflect our long-term vision of having shared code between overlays. ## The solution I created the dedicated linker script `weapon0.ld` to output the individual overlay `w0_000.bin` and similar. `f0_000.bin` is also created from the correspondent PNG. Once all the overlays and graphics are created I am simply concatenating them back to `WEAPON0.BIN`. This is much more efficient in terms of separation of concerns, maintainability, build speed and it also allows to use the same symbol names across overlays. Also very important, the generated individual overlays matches the structure of what is found in the PSP build. ## Side note I thought it was wise to prepare the Makefile and the linker script to also allow rebasing and compiling for WEAPON1.BIN. The generated file does not currently match and it will probably need a bit more research. I left the build infrastructure there. To compile the non-matching WEAPON1 just modify `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES)` to `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES) $(WEAPON1_FILES)`.
2023-08-14 01:03:54 +00:00
$(LD) $(LD_FLAGS) --no-check-sections -o $@ \
-Map $(BUILD_DIR)/weapon/w1_$*.map \
-T weapon1.ld \
-T $(CONFIG_DIR)/undefined_syms.$(VERSION).txt \
-T $(CONFIG_DIR)/undefined_syms_auto.$(VERSION).weapon.txt \
-T $(CONFIG_DIR)/undefined_funcs_auto.$(VERSION).weapon.txt \
$^
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
$(BUILD_DIR)/$(SRC_DIR)/weapon/w_%.c.o: $(SRC_DIR)/weapon/w_%.c $(MASPSX_APP) $(CC1PSX) | weapon_dirs
SEL import data/rodata (part 1) (#663) I started importing the data in SEL also as a proof of concept we can also import the SOTN-specific formatted strings. For that I integrated the new tool `tools/sotn-str.py` into the build chain. It works like the following: * `./tools/sotn-str.py parse disks/us/ST/SEL/SEL.BIN 27598`: read the binary content in a specific offset and parse it as a readable Unicode string. * `cat src/st/sel/2C048.c | ./tools/sotn-str.py process` converts the Unicode string into a byte sequence that can be understood by the SOTN engine. There is a dirty hack I did to make this work: ```c #ifndef SOTN_STR #define _S(x) (x) // Strings processed by tools/sotn-str.py #endif ``` When invoking `mipsel-linux-gnu-cpp` I am passing -DSOTN_STR, effectively ignoring `_S` in the pre-processor. Then I am passing it into `sotn-str` before `iconv` to resolve any string surrounded by `_S`. I am a bit of afraid this might be annoying when using the permuter. I tried doing `$(SOTN_STR) -f $< | $(CPP)` but apparently `cpp` does not work well with content passed via stdin.
2023-10-10 20:29:30 +00:00
$(CPP) $(CPP_FLAGS) -lang-c -DW_$* $< | $(SOTNSTR) | $(ICONV) | $(CC) $(CC_FLAGS) $(PSXCC_FLAGS) | $(MASPSX) | $(AS) $(AS_FLAGS) -o $@
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
$(BUILD_DIR)/$(SRC_DIR)/weapon/w_029.c.o: $(SRC_DIR)/weapon/w_029.c $(MASPSX_APP) $(CC1PSX) | weapon_dirs
SEL import data/rodata (part 1) (#663) I started importing the data in SEL also as a proof of concept we can also import the SOTN-specific formatted strings. For that I integrated the new tool `tools/sotn-str.py` into the build chain. It works like the following: * `./tools/sotn-str.py parse disks/us/ST/SEL/SEL.BIN 27598`: read the binary content in a specific offset and parse it as a readable Unicode string. * `cat src/st/sel/2C048.c | ./tools/sotn-str.py process` converts the Unicode string into a byte sequence that can be understood by the SOTN engine. There is a dirty hack I did to make this work: ```c #ifndef SOTN_STR #define _S(x) (x) // Strings processed by tools/sotn-str.py #endif ``` When invoking `mipsel-linux-gnu-cpp` I am passing -DSOTN_STR, effectively ignoring `_S` in the pre-processor. Then I am passing it into `sotn-str` before `iconv` to resolve any string surrounded by `_S`. I am a bit of afraid this might be annoying when using the permuter. I tried doing `$(SOTN_STR) -f $< | $(CPP)` but apparently `cpp` does not work well with content passed via stdin.
2023-10-10 20:29:30 +00:00
$(CPP) $(CPP_FLAGS) -lang-c -DW_029 $< | $(SOTNSTR) | $(ICONV) | $(CC) $(CC_FLAGS) $(PSXCC_FLAGS) -O1 | $(MASPSX) | $(AS) $(AS_FLAGS) -o $@
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
$(BUILD_DIR)/weapon/f0_%.elf: $(BUILD_DIR)/$(ASSETS_DIR)/weapon/f_%.o | weapon_dirs
Enhance Makefile for WEAPON0.BIN (#466) ## What is this Required for #465 . I re-wrote how the build process works for the `BIN/WEAPON0.BIN` ~~overlay~~ file. ## A bit of a background On the PSP build I found `WEAPON0.BIN` is just a blob with `f0_000.bin`, `w0_000.bin`, `f0_001.bin` and so on concatenated as a single file. `f0` contains the graphics (or CHR) and `w0` contains the actual overlay (or PRG). The individual overlays share a great amount of functions I documented in `config/symbols.us.weapon.txt`. ## The problem Splat creates its own `weapon.ld` linker script, insisting to link all the overlays as a single binary file. While this is not technically correct, it creates a bigger problem: symbols cannot be shared. In 58 overlays there are 58 `DestroyEntity`. Other decomps are circumnavigating the issue by adding a prefix. Following the same pattern we could have `W000_DestroyEntity`, `W001_DestroyEntity` and so on. But this is not maintainable and it does not reflect our long-term vision of having shared code between overlays. ## The solution I created the dedicated linker script `weapon0.ld` to output the individual overlay `w0_000.bin` and similar. `f0_000.bin` is also created from the correspondent PNG. Once all the overlays and graphics are created I am simply concatenating them back to `WEAPON0.BIN`. This is much more efficient in terms of separation of concerns, maintainability, build speed and it also allows to use the same symbol names across overlays. Also very important, the generated individual overlays matches the structure of what is found in the PSP build. ## Side note I thought it was wise to prepare the Makefile and the linker script to also allow rebasing and compiling for WEAPON1.BIN. The generated file does not currently match and it will probably need a bit more research. I left the build infrastructure there. To compile the non-matching WEAPON1 just modify `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES)` to `$(BUILD_DIR)/WEAPON0.BIN: $(WEAPON0_FILES) $(WEAPON1_FILES)`.
2023-08-14 01:03:54 +00:00
$(LD) -r -b binary -o $@ $<
$(BUILD_DIR)/weapon/f1_%.elf: $(BUILD_DIR)/$(ASSETS_DIR)/weapon/f_%.o
$(LD) -r -b binary -o $@ $<
$(BUILD_DIR)/$(ASSETS_DIR)/weapon/%.o: $(ASSETS_DIR)/weapon/%.png
./tools/png2bin.py $< $@
Add ST/WRP from PSP (#1172) Extract ST/WRP out of #1119 . All the function symbols should have been cross-referenced. There as some PSX functions missing from PSP and some new functions from PSP that are not present on PSX (e.g. `st_init_wrp.c`). The files `st_debug.c` and `e_breakable.c` are shared between WRP PSX and WRP PSP. Everything else from PSP is isolated into its own folder. I had to do some tricks on the YAML config to allow shared code. `ST_WRP_MERGE = st_debug e_breakable` in the `Makefile` is a bit annoying as MWCC complains about every single minute detail from the C source that has been already decompiled for the PSX US build. `EntityWarpSmallRocks` is matching on PSP but I couldn't extract the rodata without having a bunch of linker errors. This might be a Splat issue. I need to investigate further. `func_psp_09244760` is soooo interesting. The values from `0x11` to `0x17` matches the Entity IDs that are unique to the WRP overlay. This aligns to what we have in `typedef enum EntityIDs`. Overall I am very excited to the recent discoveries from the PSP build!
2024-05-25 18:56:06 +00:00
ifneq (,$(filter psp%,$(VERSION)))
include Makefile.psp.mk
else
include Makefile.psx.mk
endif
include Makefile.saturn.mk
Extract Saturn adpcm music (#252) This adds a new tool that extracts the Saturn ADPCM music files to WAV. @sonicdcer pointed me to an old winamp plugin that had support for this format, so I reworked the relevant part into a standalone tool. This doesn't try to re-encode, since I would want to examine the original decompression code to make sure that it's 1:1 with this implementation first. Not all the files seem to be in this format, so I explicitly list the ones that are in the Makefile. I think the other PCM files are the voice tracks, which would make sense to use a voice-oriented codec. --------- Co-authored-by: sozud <sozud@users.noreply.github.com>
2023-06-16 18:39:45 +00:00
Allow to forcefully export all the assembly code
2023-05-29 17:06:28 +00:00
# Force to extract all the assembly code regardless if a function is already decompiled
force_extract:
mv src src_tmp
Enhance make extract on PSX (#662) Previously if either the YAML or the symbol list were changed, we were forced to either manually delete the `asm` folder of the specific overlay or to invoke `make clean`, where the latter forced us to re-extract all the overlays via `make -j extract`. That changes with this PR. How `make` works is `target: dep1 dep2` where if one of the dependencies has the last modified date greater than the target, the rule is triggered again. Previously we were extracting overlays doing `make extract_stcen`. But since every overlay extraction generates a linker script I now changed the rules to do `make build/us/stcen.ld` to extract the same overlay. As I explained above, if either the YAML or one of the related symbols changes, the linker modified date will be older and the extraction for that overlay to be trigger again. The above allowed me to stop polluting the repo root with the linker scripts as they are now moved into `build/us` or `build/hd` depending of what you're trying to build. If you do `make extract` twice in a row you will be welcomed with a `make: Nothing to be done for 'extract'.`. There are still some instances where you **might** need `make clean` beforehand, especially when modifying the `segment` section in the YAML or renaming the symbols. A `rm -rf asm/$(VERSION)/$*` can help but I want to see if the current solution will be enough. ### Please try removing `make clean` from your workflow, once this is merged, to quickly detect possible problems with this new approach. I am also planning to make more substantial changes on our build-chain like this PR or #660 in the future. I am aiming to small incremental changes over time in case I break someone's flow or detect design flaws. I am also considering the breaking changes introduced in make 4.4, which will most likely be included in Ubuntu 24.04 LTS. I set March 2024 as a deadline to finish all the new build-chain work. --------- Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
2023-10-05 08:26:29 +00:00
rm $(BUILD_DIR)/*.ld
Allow to forcefully export all the assembly code
2023-05-29 17:06:28 +00:00
make extract -j
rm -rf src/
mv src_tmp src
Allow make-config to cross-reference symbols from `data` and `bss` (#1650) Improve `make-config.py` to internally use the `symbols.py cross` and cross-reference symbols from duplicate functions. This allows to add even more symbols for new overlays with a third pass. This was one of my biggest pain points when I was de-duplicating the boss `MAR` overlay as I was not able to directly copy&paste duplicate functions or include shared ones without having to individually fix the symbols coming from `.data` and `.bss`. This is now automated. ``` $ python3 ./tools/make-config.py ric --version hd ✔ generating psx splat config ✔ splitting config/splat.hd.ric.yaml ✔ adjusting files at src/ric ✔ disassembling matched functions ✔ finding duplicates across overlays ✔ adding cross-referenced function names ✔ renamed 141 functions, splitting again ✔ cross-referencing 141 functions ✔ adding cross-referenced symbol names ✔ renamed 145 data/bss symbols, splitting again ```
2024-09-21 19:59:55 +00:00
force_symbols: ##@ Extract a full list of symbols from a successful build
$(PYTHON) ./tools/symbols.py elf build/us/dra.elf > config/symbols.us.dra.txt
$(PYTHON) ./tools/symbols.py elf build/us/ric.elf > config/symbols.us.ric.txt
$(PYTHON) ./tools/symbols.py elf build/us/stcen.elf > config/symbols.us.stcen.txt
$(PYTHON) ./tools/symbols.py elf build/us/stdre.elf > config/symbols.us.stdre.txt
$(PYTHON) ./tools/symbols.py elf build/us/stno3.elf > config/symbols.us.stno3.txt
$(PYTHON) ./tools/symbols.py elf build/us/stnp3.elf > config/symbols.us.stnp3.txt
$(PYTHON) ./tools/symbols.py elf build/us/stnz0.elf > config/symbols.us.stnz0.txt
$(PYTHON) ./tools/symbols.py elf build/us/stsel.elf > config/symbols.us.stsel.txt
$(PYTHON) ./tools/symbols.py elf build/us/stst0.elf > config/symbols.us.stst0.txt
$(PYTHON) ./tools/symbols.py elf build/us/stwrp.elf > config/symbols.us.stwrp.txt
$(PYTHON) ./tools/symbols.py elf build/us/strwrp.elf > config/symbols.us.strwrp.txt
$(PYTHON) ./tools/symbols.py elf build/us/bomar.elf > config/symbols.us.bomar.txt
$(PYTHON) ./tools/symbols.py elf build/us/tt_000.elf > config/symbols.us.tt_000.txt
#1637: Add servant/tt_001.bin (#1659) Followed [the guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay) which seemed to work relatively painlessly for this overlay. I hit an issue in `make-config.py` where no duplicates were found and have added a separate commit which fixed that. Otherwise no issues with guide. ``` josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001 ✔ generating psx splat config ✔ splitting config/splat.us.tt_001.yaml ✔ adjusting files at src/servant/tt_001 ✔ disassembling matched functions ✖ finding duplicates across overlays Traceback (most recent call last): File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module> raise e File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module> make_config(args.input, args.version) File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config if found > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int' ```
2024-09-24 01:39:35 +00:00
$(PYTHON) ./tools/symbols.py elf build/us/tt_001.elf > config/symbols.us.tt_001.txt
Regenerate orphaned symbols in the CI (#625) [As mentioned](https://discord.com/channels/1079389589950705684/1135205782703570994/1154948845638254672), since #595 some tools are not able to pick up some function names as the symbol names are gone. This PR adds a quick tool to re-generate that list from a map file and integrates it in the CI flow. EDIT: CI is failing because the running Linter is from `master`, where `mapfile-parser` is not installed.
2023-09-24 16:55:33 +00:00
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
context: ##@ create a context for decomp.me. Set the SOURCE variable prior to calling this target
DRA functions + PSXSDK docs (#30) Co-authored-by: @pixel-stuck, @Mr-Wiseguy, @Rainchus, @DavidSM64
2022-12-19 05:32:00 +00:00
$(M2CTX) $(SOURCE)
@echo ctx.c has been updated.
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
extract_disk: ##@ Extract game files from a disc image.
Add make extract_disk compatibility for PSP ISOs This allows to extract PSP versions of the game, including the infamous HDBIN version of the game.
2023-05-05 21:26:39 +00:00
extract_disk: extract_disk_$(VERSION)
Custom debugging module (#108) # What Adds a custom debug menu. It is written in C and it is meant to replace the Bat familar `SERVANT/TT_000.BIN`. Once loaded you can long-press SELECT+START to soft-reset the game and keep using the debug menu everywhere, including when playing with Richter or during the credits. One key requirement to run this is to have an emulator that emulates the 8MB of RAM. This is a key requirement to have the debug module surviving soft-resets or accessing to the in-game menu. I personally used [PCSX Redux](https://github.com/grumpycoders/pcsx-redux) to build this module. I am not sure about the compatibility with other emulators. This does **NOT** work on real hardware and it is a choice by design. The debug module is intended to test different areas of the game and help decompiling. It is not intended to be used in normal gameplay. # Build Simply invoke `make disk_debug` to create a disk image of the game in `build/` with the debug module replacing the Bat familiar. # Usage ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/15a040b6-6191-41c4-b2b8-a4a906ed59eb) On PCSX Redux go to Configuration, Emulation and tick the box `8MB`. ## Loading the module ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/65b7ccb3-800e-4b66-84f5-5703fc91babe) You need to enable the Bat Card from the menu. This will load the debug module from the disk. If you want to re-load the module you need to select another Familar Card, un-pause, pause again and select the Bat Card once more. ## The main screen ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/b528425a-ea6c-4c10-9c19-522612d0ad2a) You will know you have loaded the debug module when you see the blue rectangle on the top right. You can press R2 to cycle between the menus. Some menus will temporarily freeze the game, some not. To quickly return in-game you can either press TRIANGLE or START. To bring back the paused debug menu just press R2 once again. ## Debug Mode ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/12cac1ee-725b-484f-8564-89c03b6a755b) ### Stage Teleports the player to a different stage. It is not stable and it can crash often. ### Player Switches between Alucard and Richter. Currently switching from Richter to Alucard consistently crashes the game. ### No Clip Allows to freely move the player within the room and without the collisions on. Once the flag is enabled from the debug menu, pressing L2 while in-game will temporarily freeze the player movement and make it immune to collision checking. You can then press the directional buttons to slowly move the player or you can hold CROSS to move it faster. You can also use SQUARE or CIRCLE to cycle between the player frames. Press L2 again to deactivate the NoClip mode. ### Frame by frame Freezes the game outside the debug module. Press L1 to advance by 1 frame. Hold L2 to put the game in slow-motion. ### Show hitboxes ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/ea8dd918-cf37-4be1-bb87-541a66ac7f16) As shown in the image ### Show debug messages ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/5c9bc7b2-6bc7-4831-949a-bf73cdf910e8) When the debug menu is un-paused, prints on the top left all the debug messages from the game itself. ### Show collision layer ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/959ad1ab-dfe0-42fe-a8d0-65c619469280) This prints the internal collision value for every 16x16 tile on the screen. Look the CheckCollision function for more information on how each printed value is used. ### Show draw calls ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/a52970fc-bf30-430d-98e3-5fca33c9f6f7) Shows the maximum GPU resource usage since the game started. Currently only the `max` option works. The `current` option will not show anything. ### Show HBlank ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/81759019-6bf9-4cf2-b88c-aa60eaa5ddcb) Prints the current horizontal blank interrupt count. ## Entity Spawn ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/f0fdc024-4537-4c07-a80d-816d605a3583) Allows to immediately spawn new entities in the current map ### Mode There are three list of entities, each one with their own ID: DRA, Stage and RIC. The option RIC is hidden if Richter is not the current playing character. As the list of entities per stage is maintained manually, stage entities might not be available for all stages. The `Alloc` shows how many entities are reserved or actually used. Pressing the SQUARE button here will destroy all the entities within that range. ### ID Press Left or Right to cycle between the different IDs available. Some of them might crash the game immediately once spawned. Press CROSS to immediately spawn the entity. ### Params Each entity might have its own parameters. Sometimes the flag 0x8000 is used, which can be toggled with the SQUARE button. Press CROSS to immediately spawn the entity. ### Entity preview Shows the entity before spawning it. This is turned off by default as it can immediately crash by cycling through the available entity IDs. ### Place entity Pressing CROSS will allow to move the entity across the screen before placing it. Press CROSS again to place the entity and return to the previous screen. Press SQUARE to quickly place multiple entities of the same type. ## Sound player ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/9d8ad3a5-bc78-4674-abd0-94537bbfae98) There are three macro categories the sound player is split into. For what is currently known only the sounds within the Kind 3 changes based on the loaded stage. ### Stop all sounds This will also disable the SPU IRQ, effectively unlocking the frame rate. ### Load Stage Loads a different sound font than the current loaded stage. This can help to quickly preview and test SFXs from other stages without necessarily moving the player there. ### Load Servant Loads the sound font of a specific servant without necessarily equipping the Familiar Card. ## Castle flags Preview all the flags used to modify the behaviour of different parts of the two castles. ### Edit mode ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/b03ada2f-0fb4-4345-ad50-81209702032c) You can move the cursor with the directional buttons and flip the flag with CROSS. Press L1 or R1 to cycle between the pages. The cursor warps when reaching the border of the flag grid, allowing a faster navigation. ### View mode Allows to move between the flags more flexibly. ### Listen mode ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/4f9ed118-d09d-4658-ba25-5865dbebbf3c) Listens for the modified flags while playing. Every time a flag is modified the offset and its value is registered on the top left up to 4 rows. When all the rows are occupied, new values will just remove the oldest one. The last modified flag will always be displayed at the bottom. --------- Co-authored-by: Alejandro Asenjo Nitti <sonicdcer@users.noreply.github.com> Co-authored-by: sozud <sozud@users.noreply.github.com>
2023-08-10 17:55:24 +00:00
disk_prepare: build $(SOTNDISK)
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
mkdir -p $(DISK_DIR)
Change how game assets are managed
2023-02-21 23:25:24 +00:00
cp -r disks/${VERSION}/* $(DISK_DIR)
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/main.exe $(DISK_DIR)/SLUS_000.67
cp $(BUILD_DIR)/DRA.BIN $(DISK_DIR)/DRA.BIN
cp $(BUILD_DIR)/RIC.BIN $(DISK_DIR)/BIN/RIC.BIN
cp $(BUILD_DIR)/CEN.BIN $(DISK_DIR)/ST/CEN/CEN.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_CEN.BIN $(DISK_DIR)/ST/CEN/F_CEN.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/DRE.BIN $(DISK_DIR)/ST/DRE/DRE.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_DRE.BIN $(DISK_DIR)/ST/DRE/F_DRE.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/MAD.BIN $(DISK_DIR)/ST/MAD/MAD.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_MAD.BIN $(DISK_DIR)/ST/MAD/F_MAD.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/NO3.BIN $(DISK_DIR)/ST/NO3/NO3.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_NO3.BIN $(DISK_DIR)/ST/NO3/F_NO3.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/NP3.BIN $(DISK_DIR)/ST/NP3/NP3.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_NP3.BIN $(DISK_DIR)/ST/NP3/F_NP3.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/NZ0.BIN $(DISK_DIR)/ST/NZ0/NZ0.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_NZ0.BIN $(DISK_DIR)/ST/NZ0/F_NZ0.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/RWRP.BIN $(DISK_DIR)/ST/RWRP/RWRP.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_RWRP.BIN $(DISK_DIR)/ST/RWRP/F_RWRP.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/SEL.BIN $(DISK_DIR)/ST/SEL/SEL.BIN
cp $(BUILD_DIR)/ST0.BIN $(DISK_DIR)/ST/ST0/ST0.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_ST0.BIN $(DISK_DIR)/ST/ST0/F_ST0.BIN
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
cp $(BUILD_DIR)/WRP.BIN $(DISK_DIR)/ST/WRP/WRP.BIN
Extract and assemble stage graphics for all stages
2023-03-10 04:04:18 +00:00
cp $(BUILD_DIR)/F_WRP.BIN $(DISK_DIR)/ST/WRP/F_WRP.BIN
Add BOSS/MAR.BIN (#1608) Small PR to enable #557 and to let be used as an example for [our wiki](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay). This was done with the new `make-config.py` tool.
2024-09-15 20:49:56 +00:00
cp $(BUILD_DIR)/MAR.BIN $(DISK_DIR)/BOSS/MAR/MAR.BIN
cp $(BUILD_DIR)/F_MAR.BIN $(DISK_DIR)/BOSS/MAR/F_MAR.BIN
Add SERVANT/TT_000.BIN overlay
2023-02-08 14:03:44 +00:00
cp $(BUILD_DIR)/TT_000.BIN $(DISK_DIR)/SERVANT/TT_000.BIN
#1637: Add servant/tt_001.bin (#1659) Followed [the guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay) which seemed to work relatively painlessly for this overlay. I hit an issue in `make-config.py` where no duplicates were found and have added a separate commit which fixed that. Otherwise no issues with guide. ``` josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001 ✔ generating psx splat config ✔ splitting config/splat.us.tt_001.yaml ✔ adjusting files at src/servant/tt_001 ✔ disassembling matched functions ✖ finding duplicates across overlays Traceback (most recent call last): File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module> raise e File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module> make_config(args.input, args.version) File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config if found > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int' ```
2024-09-24 01:39:35 +00:00
cp $(BUILD_DIR)/TT_001.BIN $(DISK_DIR)/SERVANT/TT_001.BIN
Custom debugging module (#108) # What Adds a custom debug menu. It is written in C and it is meant to replace the Bat familar `SERVANT/TT_000.BIN`. Once loaded you can long-press SELECT+START to soft-reset the game and keep using the debug menu everywhere, including when playing with Richter or during the credits. One key requirement to run this is to have an emulator that emulates the 8MB of RAM. This is a key requirement to have the debug module surviving soft-resets or accessing to the in-game menu. I personally used [PCSX Redux](https://github.com/grumpycoders/pcsx-redux) to build this module. I am not sure about the compatibility with other emulators. This does **NOT** work on real hardware and it is a choice by design. The debug module is intended to test different areas of the game and help decompiling. It is not intended to be used in normal gameplay. # Build Simply invoke `make disk_debug` to create a disk image of the game in `build/` with the debug module replacing the Bat familiar. # Usage ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/15a040b6-6191-41c4-b2b8-a4a906ed59eb) On PCSX Redux go to Configuration, Emulation and tick the box `8MB`. ## Loading the module ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/65b7ccb3-800e-4b66-84f5-5703fc91babe) You need to enable the Bat Card from the menu. This will load the debug module from the disk. If you want to re-load the module you need to select another Familar Card, un-pause, pause again and select the Bat Card once more. ## The main screen ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/b528425a-ea6c-4c10-9c19-522612d0ad2a) You will know you have loaded the debug module when you see the blue rectangle on the top right. You can press R2 to cycle between the menus. Some menus will temporarily freeze the game, some not. To quickly return in-game you can either press TRIANGLE or START. To bring back the paused debug menu just press R2 once again. ## Debug Mode ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/12cac1ee-725b-484f-8564-89c03b6a755b) ### Stage Teleports the player to a different stage. It is not stable and it can crash often. ### Player Switches between Alucard and Richter. Currently switching from Richter to Alucard consistently crashes the game. ### No Clip Allows to freely move the player within the room and without the collisions on. Once the flag is enabled from the debug menu, pressing L2 while in-game will temporarily freeze the player movement and make it immune to collision checking. You can then press the directional buttons to slowly move the player or you can hold CROSS to move it faster. You can also use SQUARE or CIRCLE to cycle between the player frames. Press L2 again to deactivate the NoClip mode. ### Frame by frame Freezes the game outside the debug module. Press L1 to advance by 1 frame. Hold L2 to put the game in slow-motion. ### Show hitboxes ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/ea8dd918-cf37-4be1-bb87-541a66ac7f16) As shown in the image ### Show debug messages ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/5c9bc7b2-6bc7-4831-949a-bf73cdf910e8) When the debug menu is un-paused, prints on the top left all the debug messages from the game itself. ### Show collision layer ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/959ad1ab-dfe0-42fe-a8d0-65c619469280) This prints the internal collision value for every 16x16 tile on the screen. Look the CheckCollision function for more information on how each printed value is used. ### Show draw calls ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/a52970fc-bf30-430d-98e3-5fca33c9f6f7) Shows the maximum GPU resource usage since the game started. Currently only the `max` option works. The `current` option will not show anything. ### Show HBlank ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/81759019-6bf9-4cf2-b88c-aa60eaa5ddcb) Prints the current horizontal blank interrupt count. ## Entity Spawn ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/f0fdc024-4537-4c07-a80d-816d605a3583) Allows to immediately spawn new entities in the current map ### Mode There are three list of entities, each one with their own ID: DRA, Stage and RIC. The option RIC is hidden if Richter is not the current playing character. As the list of entities per stage is maintained manually, stage entities might not be available for all stages. The `Alloc` shows how many entities are reserved or actually used. Pressing the SQUARE button here will destroy all the entities within that range. ### ID Press Left or Right to cycle between the different IDs available. Some of them might crash the game immediately once spawned. Press CROSS to immediately spawn the entity. ### Params Each entity might have its own parameters. Sometimes the flag 0x8000 is used, which can be toggled with the SQUARE button. Press CROSS to immediately spawn the entity. ### Entity preview Shows the entity before spawning it. This is turned off by default as it can immediately crash by cycling through the available entity IDs. ### Place entity Pressing CROSS will allow to move the entity across the screen before placing it. Press CROSS again to place the entity and return to the previous screen. Press SQUARE to quickly place multiple entities of the same type. ## Sound player ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/9d8ad3a5-bc78-4674-abd0-94537bbfae98) There are three macro categories the sound player is split into. For what is currently known only the sounds within the Kind 3 changes based on the loaded stage. ### Stop all sounds This will also disable the SPU IRQ, effectively unlocking the frame rate. ### Load Stage Loads a different sound font than the current loaded stage. This can help to quickly preview and test SFXs from other stages without necessarily moving the player there. ### Load Servant Loads the sound font of a specific servant without necessarily equipping the Familiar Card. ## Castle flags Preview all the flags used to modify the behaviour of different parts of the two castles. ### Edit mode ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/b03ada2f-0fb4-4345-ad50-81209702032c) You can move the cursor with the directional buttons and flip the flag with CROSS. Press L1 or R1 to cycle between the pages. The cursor warps when reaching the border of the flag grid, allowing a faster navigation. ### View mode Allows to move between the flags more flexibly. ### Listen mode ![image](https://github.com/Xeeynamo/sotn-decomp/assets/6128729/4f9ed118-d09d-4658-ba25-5865dbebbf3c) Listens for the modified flags while playing. Every time a flag is modified the offset and its value is registered on the top left up to 4 rows. When all the rows are occupied, new values will just remove the oldest one. The last modified flag will always be displayed at the bottom. --------- Co-authored-by: Alejandro Asenjo Nitti <sonicdcer@users.noreply.github.com> Co-authored-by: sozud <sozud@users.noreply.github.com>
2023-08-10 17:55:24 +00:00
disk: disk_prepare
$(SOTNDISK) make build/sotn.$(VERSION).cue $(DISK_DIR) $(CONFIG_DIR)/disk.us.lba
disk_debug: disk_prepare
cd tools/sotn-debugmodule && make
cp $(BUILD_DIR)/../sotn-debugmodule.bin $(DISK_DIR)/SERVANT/TT_000.BIN
Writes built binaries into build/us/
2023-02-21 20:06:27 +00:00
$(SOTNDISK) make build/sotn.$(VERSION).cue $(DISK_DIR) $(CONFIG_DIR)/disk.us.lba
Creates a new game image based on built binaries (#34)
2023-01-01 20:49:12 +00:00
Add ST/WRP from PSP (#1172) Extract ST/WRP out of #1119 . All the function symbols should have been cross-referenced. There as some PSX functions missing from PSP and some new functions from PSP that are not present on PSX (e.g. `st_init_wrp.c`). The files `st_debug.c` and `e_breakable.c` are shared between WRP PSX and WRP PSP. Everything else from PSP is isolated into its own folder. I had to do some tricks on the YAML config to allow shared code. `ST_WRP_MERGE = st_debug e_breakable` in the `Makefile` is a bit annoying as MWCC complains about every single minute detail from the C source that has been already decompiled for the PSX US build. `EntityWarpSmallRocks` is matching on PSP but I couldn't extract the rodata without having a bunch of linker errors. This might be a Splat issue. I need to investigate further. `func_psp_09244760` is soooo interesting. The values from `0x11` to `0x17` matches the Entity IDs that are unique to the WRP overlay. This aligns to what we have in `typedef enum EntityIDs`. Overall I am very excited to the recent discoveries from the PSP build!
2024-05-25 18:56:06 +00:00
# put this here as both PSX HD and PSP use it
extract_disk_psp%:
mkdir -p disks/psp$*
Makefile Quality of Life Changes (#1275) The `check` recipe now has dependencies on all of the files it checks allowing them to be rebuilt if necessary. Consistent use of `dd` and suppressing `dd`'s output which noise for the replacement and padding it's doing. Run 7z in non-interactive mode to avoid confirming overwrite of files during extraction (equivalent behavior to `VERSION=us` `extract_disk`). `patch` is a phony target. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-08 17:13:03 +00:00
7z x -y disks/sotn.psp$*.iso -odisks/psp$*/
Add ST/WRP from PSP (#1172) Extract ST/WRP out of #1119 . All the function symbols should have been cross-referenced. There as some PSX functions missing from PSP and some new functions from PSP that are not present on PSX (e.g. `st_init_wrp.c`). The files `st_debug.c` and `e_breakable.c` are shared between WRP PSX and WRP PSP. Everything else from PSP is isolated into its own folder. I had to do some tricks on the YAML config to allow shared code. `ST_WRP_MERGE = st_debug e_breakable` in the `Makefile` is a bit annoying as MWCC complains about every single minute detail from the C source that has been already decompiled for the PSX US build. `EntityWarpSmallRocks` is matching on PSP but I couldn't extract the rodata without having a bunch of linker errors. This might be a Splat issue. I need to investigate further. `func_psp_09244760` is soooo interesting. The values from `0x11` to `0x17` matches the Entity IDs that are unique to the WRP overlay. This aligns to what we have in `typedef enum EntityIDs`. Overall I am very excited to the recent discoveries from the PSP build!
2024-05-25 18:56:06 +00:00
Python virtualenv Support (#1620) Ubuntu/Debian and Python recommend using virtual environments for project-specific Python dependencies and as of Ubuntu 24 this is lightly enforced when installing packages via pip. This is due to pip and the system package manager installing files to the same location which may cause either's internal state to no longer reflect what is actually installed. This updates the project to use a Python `virtualenv` for project dependencies and updates internal scripts to support both global and virtualenvs, but favors virtualenvs for new workspaces. All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the first `python3` in the path. For those with a virtualenv configured, this will be the Python managed there. For everyone else, this should be the system Python or whatever other scheme they may have used previously (assuming `python3` already existed in their `PATH`). The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and use `python3` from there if it exists and has been configured. It also has a new `python-dependencies` target which will configure the venv and install all python dependnecies there. The `Dockerfile` has been updated to create an external `.venv` outside of the project directory. Python's `virtualenv`s are not relocatable and may hardcode paths which are mounted differently in the container and host. To deal with differences in paths between the container (which mounts the host's project directory to `/sotn`) host which may be at an arbitarary directory the `VENV_PATH` environment variable is used to override paths in the `Makefile`. GitHub workflows have been updated to use `.venv`.
2024-09-18 06:19:20 +00:00
python-dependencies:
# the python setup cannot depend on the virtualenv
# because it may not be set up yet
[ -d $(VENV_PATH) ] || python3 -m venv $(VENV_PATH)
pip install -r $(TOOLS_DIR)/requirements-python.txt
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
update-dependencies: ##@ update tools and internal dependencies
Python virtualenv Support (#1620) Ubuntu/Debian and Python recommend using virtual environments for project-specific Python dependencies and as of Ubuntu 24 this is lightly enforced when installing packages via pip. This is due to pip and the system package manager installing files to the same location which may cause either's internal state to no longer reflect what is actually installed. This updates the project to use a Python `virtualenv` for project dependencies and updates internal scripts to support both global and virtualenvs, but favors virtualenvs for new workspaces. All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the first `python3` in the path. For those with a virtualenv configured, this will be the Python managed there. For everyone else, this should be the system Python or whatever other scheme they may have used previously (assuming `python3` already existed in their `PATH`). The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and use `python3` from there if it exists and has been configured. It also has a new `python-dependencies` target which will configure the venv and install all python dependnecies there. The `Dockerfile` has been updated to create an external `.venv` outside of the project directory. Python's `virtualenv`s are not relocatable and may hardcode paths which are mounted differently in the container and host. To deal with differences in paths between the container (which mounts the host's project directory to `/sotn`) host which may be at an arbitarary directory the `VENV_PATH` environment variable is used to override paths in the `Makefile`. GitHub workflows have been updated to use `.venv`.
2024-09-18 06:19:20 +00:00
update-dependencies: $(ASMDIFFER_APP) $(M2CTX_APP) $(M2C_APP) $(MASPSX_APP) $(SATURN_SPLITTER_APP) $(GO) $(ALLEGREX_AS) python-dependencies
update-dependencies updates Saturn toolsets
2023-06-17 13:23:44 +00:00
cd $(SATURN_SPLITTER_DIR)/rust-dis && cargo build --release
cd $(SATURN_SPLITTER_DIR)/adpcm-extract && cargo build --release
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
rm $(SOTNDISK) && make $(SOTNDISK) || true
rm $(SOTNASSETS) && make $(SOTNASSETS) || true
Update compiler to use little endian
2023-04-05 18:08:14 +00:00
git clean -fd bin/
Ensure asm-differ is installed
2022-10-14 20:19:34 +00:00
Rewrite Saturn make rules (#660) ## What Enhance the existing `Makefile` to build new rules for the Saturn side of the decomp. This should allow more flexibility when adding new overlays or when tuning existing rules. ## Changes I separated part of the Saturn build process in the separate file `Makefile.saturn.mk`. I realise that naming it `saturn.mk` would have been enough, but I pre-pended `Makefile.` so it can be found right below the main `Makefile` when listing files in an alphabetic order. I plan to do the same with the psx and psp toolchain, therefore you will find `include Makefile.*.mk` in the main `Makefile`. I deleted all the game building process done with Docker. Now that we have an established way to do it natively I think it is no longer required. We can always run the entire buildchain within a Docker container instead of having `_native` and `_docker`. Now all the `_native` references are removed. `build_saturn_native` is now `build_saturn`. `check_saturn` is no longer responsible of stripping the ELF into a binary. That is now part of `build_saturn`. I removed the references to `_li.o` (e.g. `alucard_li.o`) and used `.elf` instead, which is closer to how the PSX build chain works. If `_li.o` was a better preference, please let me know. I am no longer using `./compile_dosemu.sh`. Instead I am using the new `$(DOSEMU)` to directly invoke the tool within the Makefile. I have done that to reduce the amount of dependent files. I tried minimising duplication as much as possible. We now have a list of overlays found in `SATURN_OVL_TARGETS`. Each expected output triggers a series of dependencies so seamlessly build everything. `Makefile` is smart enough to call `$(SATURN_TOOLCHAIN)` only once. If the game was already built but just one source or symbol file changed, triggering a new `build_saturn` will only compile the modified overlay. The Saturn ADPCM files are now extracted in `assets/` instead of `build/`. I think `assets/` should contain all the uncompressed and uncooked files. The list of PCM file is not hardcoded. Instead I am now using `$(wildcard disks/saturn/SD/*.PCM)`. This now means the tool tries to convert PCMs from `SDD0.PCM` to `SDF0.PCM` with no success. As the tool is silently failing I decided to leave it as I wrote it. ## Problems I rewrote everything thinking about concurrency in mind. But `make -j build_saturn` gives some unexpected output on `stdout`. I did not dig too much into it. I suspect it might be dosemu. This is not a stopper as we were not using `-j` when building the game anyway. I also noticed doing `VERSION=saturn make build` calls `mipsel-linux-gnu-ld` for `stage_02`. I suspect it is calling the rule `$(MAIN_TARGET).elf: $(MAIN_O_FILES)` and simply moving `include Makefile.*.mk` above it should fix it. But then it would cause the same problem if I split the PSX rules into their own separate file. We never used `make build` by setting the env variable `VERSION`, so this is not either a breaking change or a stopper. ## Post thoughts I am happy with what I achieved so far. I used the knowledge I accumulated when maintaining the PSX counterpart. Since I now better understand how `make` works, I was able to make some better decisions in the Saturn counterpart. For example triggering a new build when the symbol list changes is something the PSX build chain lacks of. I think in the future it would be nice to trigger `make extract` when either the YAML or the symbol list changes.
2023-10-03 20:42:08 +00:00
bin/%: bin/%.tar.gz
sha256sum --check $<.sha256
cd bin && tar -xzf ../$<
rm $<
touch $@
bin/%.tar.gz: bin/%.tar.gz.sha256
wget -O $@ https://github.com/Xeeynamo/sotn-decomp/releases/download/cc1-psx-26/$*.tar.gz
Ensure asm-differ is installed
2022-10-14 20:19:34 +00:00
$(ASMDIFFER_APP):
Fix update-tools
2022-10-14 20:26:57 +00:00
git submodule init $(ASMDIFFER_DIR)
git submodule update $(ASMDIFFER_DIR)
Create context with Makefile
2022-10-15 10:18:32 +00:00
$(M2CTX_APP):
curl -o $@ https://raw.githubusercontent.com/ethteck/m2ctx/main/m2ctx.py
Decompile code with make
2022-10-15 10:45:20 +00:00
$(M2C_APP):
git submodule init $(M2C_DIR)
git submodule update $(M2C_DIR)
Fix Directory Creation Dependencies in Makefile (#1353) This updates the Makefile so that targets which depend on phony targets (like `*_dirs`) have direct dependencies on those targets rather than relying on transient dependencies from dependent targets. Other targets explicitly create parent directories for their files. This resolves #739, making it possible to using make-4.4 to build in parallel.
2024-06-28 13:23:41 +00:00
$(PYTHON) -m pip install --upgrade pycparser
Fix CI pipeline
2023-06-17 11:29:42 +00:00
$(MASPSX_APP):
git submodule init $(MASPSX_DIR)
git submodule update $(MASPSX_DIR)
Patch assembly code
2022-10-26 06:41:13 +00:00
$(GO):
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
curl -L -o go1.22.4.linux-amd64.tar.gz https://go.dev/dl/go1.22.4.linux-amd64.tar.gz
tar -C $(HOME) -xzf go1.22.4.linux-amd64.tar.gz
rm go1.22.4.linux-amd64.tar.gz
Support Single BIN CUE Files (#1127) In CUE files `FILE` is a stateful, global declaration that applies to all following `TRACK`s. `sotn-disk` was treating `FILE` declarations as `TRACK` delimiters which would result in incorrect parsing of CUE files with a single `FILE`, but multiple tracks. Now when reading a `FILE` declaration, the path is stored and processing continues. A `TRACK` declaration will use the previously defined path, and if a previous track had been started, append that previous track to the list. This also builds `sotn-disk` using the local repository instead of pulling the latest commit from GitHub. The target depends on `sotn-disk` sources and will rebuild as necessary (or with `make ~/go/bin/sotn-disk`). As an aside, [pull/232](https://github.com/Xeeynamo/sotn-decomp/pull/232) ran into this same error, but fixes in a slightly different way that leaves side effects that may make supporting things like `INDEX` (for extracting the placeholder audio, for example) more error prone in the future. Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-05-14 18:59:56 +00:00
$(SOTNDISK): $(GO) $(SOTNDISK_SOURCES)
cd tools/sotn-disk; $(GO) install
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
$(SOTNASSETS): $(GO) $(SOTNASSETS_SOURCES)
cd tools/sotn-assets; $(GO) install
Upload disassembly
2022-01-01 17:45:41 +00:00
WRP split some data and add new split tools (#109) The objective is to separate the data that is usually written into `asm/*/data` into separate files. Those will initially be binary files. There is also an utility that allows to decompress compressed data. The decompressed data is currently not read as we are missing a compression algorithm that matches the original one. I tried writing one in the past: it worked, it re-created similar files but sadly not the same ones. The decompression algorithm is there to allow to do some data mining and understand what the data is and how it is used.
2023-03-09 13:32:00 +00:00
# Handles assets
RIC extract spritesheet (#199) Thanks to @MottZilla research I was able to extract the spritesheets of Richter as PNGs in conjunction with the original palette from `F_GAME2.BIN`. When the binaries are built, the PNGs are recompiled back into binary. In case of modding, it is possible to modify the sprite size to accommodate bigger or smaller sprites. Although it is not possible to modify the existing palette as currently we have no way to extract and rebuild `F_GAME2.BIN`. You would have to modify the palette via hex yourself. The file `config/splat.us.ric.yaml` shows in which offset the palette is located. Just for reference, the colour format for the palette is RGBA5551. It is important that edited PNGs has 16 colours otherwise the build will fail. The `padding` field is required to reconstruct some left-over / junk data that is not part of the data sprite. For the sake of modding, the `padding` field can be whatever. I am not sure if `RIC.BIN` pointers can be relocated. If that is the case you might get a black screen when playing with Richter. It should be easily fixable though. ![image](https://user-images.githubusercontent.com/6128729/233791249-0155a6fb-c5fc-4142-b294-9f44b5524dde.png)
2023-04-22 20:03:55 +00:00
$(BUILD_DIR)/$(ASSETS_DIR)/%.spritesheet.json.o: $(ASSETS_DIR)/%.spritesheet.json
Allow spritesheet to be extracted from the CLI (#1035) A side-project I've been working on that helped me understanding some structures that are now a bit better documented in #1034 . Richter sprites are extracted as usual. Doing `python3 tools/splat_ext/spritesheet.py decode disks/us/BIN/ARC_F.BIN assets/arc alucard disks/us/BIN/F_GAME.BIN 1` can extract Alucard sprites too. There is no repacking yet as I want to avoid using splat for this one.
2024-02-05 21:57:48 +00:00
./tools/splat_ext/spritesheet.py encode $< $(BUILD_DIR)/$(ASSETS_DIR)/$*.s
RIC extract spritesheet (#199) Thanks to @MottZilla research I was able to extract the spritesheets of Richter as PNGs in conjunction with the original palette from `F_GAME2.BIN`. When the binaries are built, the PNGs are recompiled back into binary. In case of modding, it is possible to modify the sprite size to accommodate bigger or smaller sprites. Although it is not possible to modify the existing palette as currently we have no way to extract and rebuild `F_GAME2.BIN`. You would have to modify the palette via hex yourself. The file `config/splat.us.ric.yaml` shows in which offset the palette is located. Just for reference, the colour format for the palette is RGBA5551. It is important that edited PNGs has 16 colours otherwise the build will fail. The `padding` field is required to reconstruct some left-over / junk data that is not part of the data sprite. For the sake of modding, the `padding` field can be whatever. I am not sure if `RIC.BIN` pointers can be relocated. If that is the case you might get a black screen when playing with Richter. It should be easily fixable though. ![image](https://user-images.githubusercontent.com/6128729/233791249-0155a6fb-c5fc-4142-b294-9f44b5524dde.png)
2023-04-22 20:03:55 +00:00
$(AS) $(AS_FLAGS) -o $(BUILD_DIR)/$(ASSETS_DIR)/$*.o $(BUILD_DIR)/$(ASSETS_DIR)/$*.s
New asset manager (#1343) This aims to deprecate all the Splat tools in `tools/splat_ext` in favour of a more centralised asset manager. This brings the following advantages: * Much faster extraction * Faster build * Automatically define `static` symbols or unique names whenever `static` is not possible * Allow to embed assets into the output binary * Drastically simplify `Makefile` by removing all the asset build rules * Avoid situations where it is not possible to extract and build assets that is not 4-byte aligned This is achieved by having the splat YAML targeting a normal C file as data and have an external tool to take care of the following: 1. Extract asset files straight from the overlay binary file into human readable file in `assets/st/STAGE_NAME` 2. Build assets as header files that go into `src/st/STAGE_NAME` to just include them from any C file This requires each stage header to have the following new format: please see `src/st/nz0/header.c` Built assets in `src/st` are ignored by Git. As for now, for simplicity sake, the steps `make extract_assets` and `make build_assets` are just executed within `make extract` exclusively for the US version. I plan to auto-generate files such as `src/st/nz0/tile_data.c`. For a first iteration I am aiming to handle the following: * [X] Extract rooms: `assets/st/*/rooms.json` * [X] Extract room layers: `assets/st/*/entity_layouts.json` * [X] Extract tilemap data: `assets/st/*/tilemap_*.bin` * [X] Extract tilemap definitions: `assets/st/*/tiledef_*.json` * [X] Extract sprites: `assets/st/*/sprites.json` * [x] Extract entity layouts * [X] Build rooms: `src/st/*/rooms.h` * [X] Build room layers: `src/st/*/layers.h` * [X] Build tilemap data: `src/st/*/tilemap_*.h` * [X] Build tilemap definitions: `src/st/*/tiledef_*.h` * [x] Build sprites (aka `g_SpriteBanks`) * [x] Build entity layouts * [x] Allow the tool to suggest how to adjust the Splat config for each overlay I want the tool to cover the following stages: * [x] CEN * [x] DRE * ~MAD~ I do not think this can be done, it is way too different from the other overlays * [x] NO3 * [x] NP3 * [X] NZ0 * [x] ST0 * [X] WRP * [x] RWRP * ~WRP (PSP)~ Maybe in a follow-up PR For a later iteration I plan on extracting and build: * Entity GFX thingie * The CLUT thingie in the header * Uncompressed GFX data * Cutscene data * Blueprints * The `src/config_us.h` thingie --------- Co-authored-by: Josh Lory <josh.lory@outlook.com>
2024-07-02 20:38:36 +00:00
$(BUILD_DIR)/$(ASSETS_DIR)/dra/%.json.o: $(ASSETS_DIR)/dra/%.json
./tools/splat_ext/assets.py $< $(BUILD_DIR)/$(ASSETS_DIR)/dra/$*.s
$(AS) $(AS_FLAGS) -o $(BUILD_DIR)/$(ASSETS_DIR)/dra/$*.o $(BUILD_DIR)/$(ASSETS_DIR)/dra/$*.s
$(BUILD_DIR)/$(ASSETS_DIR)/ric/%.json.o: $(ASSETS_DIR)/ric/%.json
./tools/splat_ext/assets.py $< $(BUILD_DIR)/$(ASSETS_DIR)/ric/$*.s
$(AS) $(AS_FLAGS) -o $(BUILD_DIR)/$(ASSETS_DIR)/ric/$*.o $(BUILD_DIR)/$(ASSETS_DIR)/ric/$*.s
WRP split some data and add new split tools (#109) The objective is to separate the data that is usually written into `asm/*/data` into separate files. Those will initially be binary files. There is also an utility that allows to decompress compressed data. The decompressed data is currently not read as we are missing a compression algorithm that matches the original one. I tried writing one in the past: it worked, it re-created similar files but sadly not the same ones. The decompression algorithm is there to allow to do some data mining and understand what the data is and how it is used.
2023-03-09 13:32:00 +00:00
$(BUILD_DIR)/$(ASSETS_DIR)/%.bin.o: $(ASSETS_DIR)/%.bin
HD WRP matching (#1421) Mostly identical to MAD and PSP. There are four very different functions from US that needs to be decompiled to be 100% complete: * [x] `e_misc.c` `EntityMessageBox` * [x] `e_misc.c` `EntityRelicOrb` * [x] ~`e_stage_name.c` `StageNamePopupHelper`~ it does not exist * [x] `e_stage_name.c` `EntityStageNamePopup` These four very different functions are the only ones that use the JP text instead of the US one. This overlay also lacks of `BlitChar`. `EntityStageNamePopup` is very similar to the PSP counterpart. I used @joshlory scratch to match the HD part. `EntityRelicOrb` had some #ifdef between US and BETA (aka the MAD overlay). I found some code that crossed between BETA and HD, so I just put an `#else` after `VERSION_US`. We are starting to reconstruct how the source code originally evolved across different game builds.
2024-07-20 00:05:26 +00:00
mkdir -p $(dir $@)
WRP split some data and add new split tools (#109) The objective is to separate the data that is usually written into `asm/*/data` into separate files. Those will initially be binary files. There is also an utility that allows to decompress compressed data. The decompressed data is currently not read as we are missing a compression algorithm that matches the original one. I tried writing one in the past: it worked, it re-created similar files but sadly not the same ones. The decompression algorithm is there to allow to do some data mining and understand what the data is and how it is used.
2023-03-09 13:32:00 +00:00
$(LD) -r -b binary -o $(BUILD_DIR)/$(ASSETS_DIR)/$*.o $<
$(BUILD_DIR)/$(ASSETS_DIR)/%.dec.o: $(ASSETS_DIR)/%.dec
# for now '.dec' files are ignored
touch $@
RIC extract spritesheet (#199) Thanks to @MottZilla research I was able to extract the spritesheets of Richter as PNGs in conjunction with the original palette from `F_GAME2.BIN`. When the binaries are built, the PNGs are recompiled back into binary. In case of modding, it is possible to modify the sprite size to accommodate bigger or smaller sprites. Although it is not possible to modify the existing palette as currently we have no way to extract and rebuild `F_GAME2.BIN`. You would have to modify the palette via hex yourself. The file `config/splat.us.ric.yaml` shows in which offset the palette is located. Just for reference, the colour format for the palette is RGBA5551. It is important that edited PNGs has 16 colours otherwise the build will fail. The `padding` field is required to reconstruct some left-over / junk data that is not part of the data sprite. For the sake of modding, the `padding` field can be whatever. I am not sure if `RIC.BIN` pointers can be relocated. If that is the case you might get a black screen when playing with Richter. It should be easily fixable though. ![image](https://user-images.githubusercontent.com/6128729/233791249-0155a6fb-c5fc-4142-b294-9f44b5524dde.png)
2023-04-22 20:03:55 +00:00
$(BUILD_DIR)/$(ASSETS_DIR)/%.png.o: $(ASSETS_DIR)/%.png
touch $@
WRP split some data and add new split tools (#109) The objective is to separate the data that is usually written into `asm/*/data` into separate files. Those will initially be binary files. There is also an utility that allows to decompress compressed data. The decompressed data is currently not read as we are missing a compression algorithm that matches the original one. I tried writing one in the past: it worked, it re-created similar files but sadly not the same ones. The decompression algorithm is there to allow to do some data mining and understand what the data is and how it is used.
2023-03-09 13:32:00 +00:00
Upload disassembly
2022-01-01 17:45:41 +00:00
SHELL = /bin/bash -e -o pipefail
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
##@
##@ Disc Dumping Targets
##@
dump_disk: ##@ dump a physical game disk
dump_disk: dump_disk_$(VERSION)
dump_disk_eu: dump_disk_cd
dump_disk_hk: dump_disk_cd
dump_disk_jp10: dump_disk_cd
dump_disk_jp11: dump_disk_cd
dump_disk_jp: dump_disk_cd
dump_disk_saturn: dump_disk_cd
dump_disk_us: dump_disk_cd
dump_disk_usproto: dump_disk_cd
dump_disk_psp%: dump_disk_not_supported
dump_disk_xbla%: dump_disk_not_supported
dump_disk_cd: disks/sotn.$(VERSION).cue
dump_disk_not_supported:
@echo "Auatomated dumping of $(VERSION) is not supported" >&2 && exit 1
disks/sotn.%.bin disks/sotn.%.cue:
@( which -s cdrdao && which -s toc2cue ) || (echo "cdrdao(1) and toc2cue(1) must be installed" && exit 1 )
cd disks && \
DEVICE="$(shell cdrdao scanbus 2>&1 | grep -vi cdrdao | head -n1 | sed 's/ : [^:]*$$//g')" && \
cdrdao read-cd \
--read-raw \
--datafile sotn.$*.bin \
--device "$$DEVICE" \
--driver generic-mmc-raw \
sotn.$*.toc && \
toc2cue sotn.$*.toc sotn.$*.cue && \
rm sotn.$*.toc
Add tool to automatically perform symbols cross referencing (#413) I wanted to make this tool for a long time. Given the fact HD recently became more dense into DRA, people who are willing to contribute to that overlay should not have the burden of manually cross-referencing symbols as I have been doing for the past weeks. Instead of creating a tutorial I created a tool that cross-reference symbols between functions. The two functions can be either between two different revisions of the game or between two overlays. The latter is especially important to finally remove all the duplicate functions. I also took the opportunity to add some unit tests (finally!) in `tools/tests` instead of creating the dedicated folder `tools/symbols` with tools and test in it. Let me know if `tools/tests` is a pattern we want to keep following for any future tool that needs to be tested. I also added an assembly parser written in Python. As for today it lives in `tools/symbols.py`. It is barebone but it can parse the majority of assembly lines. I also added tests around it in case we want to expand it or extract the parser into its own utility. I've been thinking to ideally use it for a V2 of `find_duplicates.py` if we find the current tool to be limiting. ### Demo ``` sotn-decomp$ ./tools/symbols.py cross asm/us/dra/nonmatchings/71830/func_80112BB0.s asm/hd/dra/nonmatchings/71830/func_80112BB0.s D_80139824 = 0x801393F0; D_800ACF7C = 0x800AD040; D_800ACF7E = 0x800AD042; D_80138FC8 = 0x80138B94; D_80138FCA = 0x80138B96; ``` these symbols can be automatically copy&pasted into `symbols.hd.dra.txt` or the relevant symbol list.
2023-07-31 19:14:57 +00:00
include tools/tools.mk
Speed Up `make format` (#1359) After building for all supported architectures `make format` was taking around 30s. This commit includes various changes which get that down to around 3½s on the same machine. This changes the `format` target in a few ways: * filters out files generated by CMake * runs `clang-format` in parallel * breaks `format` into 3 separate targets which can be run in parallel When building the PC version cmake creates a file to test compiler features. This file is picked up by the pattern which matches files for `clang-format`. This file took around 20s, filtering it out brought total format time to around 10s. Instead of passing a list created by a `find` command directly as the args to `clang-format` the list is now passed to `xargs` which runs several processes in parallel, each looking at 10 files (determined by experimentation). This has the added advantage of ensuring as the source list grows `ARG_MAX` won't be a concern. This brought total format time to around 6s. `format` was broken up into three targets `-src`, `-tools`, and `-symbols`. These targets can be run in parallel, bringing the total time down to 3½s. --------- Co-authored-by: Jonathan Hohle <jon@ttkb.co>
2024-06-30 08:26:53 +00:00
.PHONY: all, clean, patch, check, build, expected
.PHONY: format, ff, format-src, format-tools, format-symbols
#1637: Add servant/tt_001.bin (#1659) Followed [the guide](https://github.com/Xeeynamo/sotn-decomp/wiki/Decompilation#add-new-overlay) which seemed to work relatively painlessly for this overlay. I hit an issue in `make-config.py` where no duplicates were found and have added a separate commit which fixed that. Otherwise no issues with guide. ``` josh@JoshsPC:/mnt/c/dev/sotn-decomp$ tools/make-config.py tt_001 ✔ generating psx splat config ✔ splitting config/splat.us.tt_001.yaml ✔ adjusting files at src/servant/tt_001 ✔ disassembling matched functions ✖ finding duplicates across overlays Traceback (most recent call last): File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1132, in <module> raise e File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1126, in <module> make_config(args.input, args.version) File "/mnt/c/dev/sotn-decomp/tools/make-config.py", line 1106, in make_config if found > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int' ```
2024-09-24 01:39:35 +00:00
.PHONY: main, dra, ric, cen, dre, mad, no3, np3, nz0, st0, wrp, rwrp, bomar, tt_000, tt_001
Simplify Makefile
2022-02-23 18:17:18 +00:00
.PHONY: %_dirs
.PHONY: extract, extract_%
Python virtualenv Support (#1620) Ubuntu/Debian and Python recommend using virtual environments for project-specific Python dependencies and as of Ubuntu 24 this is lightly enforced when installing packages via pip. This is due to pip and the system package manager installing files to the same location which may cause either's internal state to no longer reflect what is actually installed. This updates the project to use a Python `virtualenv` for project dependencies and updates internal scripts to support both global and virtualenvs, but favors virtualenvs for new workspaces. All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the first `python3` in the path. For those with a virtualenv configured, this will be the Python managed there. For everyone else, this should be the system Python or whatever other scheme they may have used previously (assuming `python3` already existed in their `PATH`). The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and use `python3` from there if it exists and has been configured. It also has a new `python-dependencies` target which will configure the venv and install all python dependnecies there. The `Dockerfile` has been updated to create an external `.venv` outside of the project directory. Python's `virtualenv`s are not relocatable and may hardcode paths which are mounted differently in the container and host. To deal with differences in paths between the container (which mounts the host's project directory to `/sotn`) host which may be at an arbitarary directory the `VENV_PATH` environment variable is used to override paths in the `Makefile`. GitHub workflows have been updated to use `.venv`.
2024-09-18 06:19:20 +00:00
.PHONY: update-dependencies python-dendencies
Build QoL (#1632) Several quality of life changes for building: * The Makefile has a `help` target which will show common targets. Targets with comments starting with `##@` will be included in the help output. * The `check` target now has colored output. * Added `check_disk` target which will check hashes of extracted disk contents (useful for those who dump their own discs) * Added `dump_disk` target which will create a bin/cue pair from an original disc.
2024-09-20 22:56:34 +00:00
.PHONY: dump_disk dump_disk_%
Reference in New Issue Copy Permalink