- Decompiled duplicate function func_us_801714F4
- Decompiled duplicate function func_us_80172904
- Decompiled duplicate function func_us_8017293C (renamed
Tt001UpdateAnim per tt_000 duplicate)
- Decompiled duplicate function func_us_80172B50
- Decompiled duplicate function func_us_80172B88
- Added EntitySearch D_80170EE4[] to servant.h to be used by
func_us_80172B88 similar to duplicate in tt_000
A lot of these are in the same order in tt_000, so I would assume they
could be moved to shared static functions at some point.
In analyze_calls, we need to analyze a function's calls, and be able to
track where they go. Given that a `jal` call only has a function name,
it's possible for it to go to a function in any overlay. Therefore, we
need to disambiguate across overlays.
We do this by identifying what overlay every function is in. This comes
from the filename of the assembly file.
Since the overlay might be in, for example, asm/us/dra (a "level 3
overlay" because it has 3 directory names) or asm/us/st/cen (a "level 4
overlay") we had special logic for testing for "st".
Now that we can also have level 4 overlays in "boss" (with Maria and BO3
coming up) we need to test for those too.
While we're at it, we also only run on the "us" version's assembly
files.
Ubuntu/Debian and Python recommend using virtual environments for
project-specific Python dependencies and as of Ubuntu 24 this is lightly
enforced when installing packages via pip. This is due to pip and the
system package manager installing files to the same location which may
cause either's internal state to no longer reflect what is actually
installed.
This updates the project to use a Python `virtualenv` for project
dependencies and updates internal scripts to support both global and
virtualenvs, but favors virtualenvs for new workspaces.
All tools that hardcode `/usr/bin/python3` now use `env(1)` to find the
first `python3` in the path. For those with a virtualenv configured,
this will be the Python managed there. For everyone else, this should be
the system Python or whatever other scheme they may have used previously
(assuming `python3` already existed in their `PATH`).
The `Makefile` has been updated to prepend `.venv/bin` to the `PATH` and
use `python3` from there if it exists and has been configured. It also
has a new `python-dependencies` target which will configure the venv and
install all python dependnecies there.
The `Dockerfile` has been updated to create an external `.venv` outside
of the project directory. Python's `virtualenv`s are not relocatable and
may hardcode paths which are mounted differently in the container and
host. To deal with differences in paths between the container (which
mounts the host's project directory to `/sotn`) host which may be at an
arbitarary directory the `VENV_PATH` environment variable is used to
override paths in the `Makefile`.
GitHub workflows have been updated to use `.venv`.
Iterating on this problem in master.
Now that wrp/rwrp are sharing so much code, analyze_calls is having
trouble telling what functions call each other in which overlay.
I think this will resolve all the issues we have for now, but I'm sure
more will appear over time.
This will decompile all the functions but the new `EntityRWarpRoom` from
the RWRP overlay.
Note that as `e_misc.c` has a function with a different castle flag, I
had to use a shared header. Everything else was identical.
---------
Co-authored-by: bismurphy <tjmurphy@mit.edu>
Another disappointing non-match, even closer than #1008 . There is only
one registry swap in one single line. I've been permuting for a day
without success. Putting this here if anyone wants to give it a try:
https://decomp.me/scratch/AuSvv
After finding a copy of GetFreeDraEntity in RIC, I decided to rename the
function (in both DRA and RIC) to GetFreeEntity. Similarly, the function
right after it is GetFreeEntityReverse (since its logic runs in
reverse).
I also found CreateEntFactoryFromEntity in RIC, so I renamed that to
match the version in DRA. We will need to go through and use the FACTORY
macro for its uses, but I'm not going to tackle that quite yet.
Seems there is a lot of work to be done making RIC catch up to DRA,
we'll see if I keep working on that moving forward, it's neat to find
the ways the two match.
I've been meaning to do this for a long time now.
The analyze_calls script running in the CI creates a .md file which acts
as a directory pointing to all the function graphs that were generated.
It also generates an HTML file. Because Github does not natively render
HTML files, but does render Markdown files, it is preferrable to have
this for the sake of easy viewing and linking within the gh-duplicates
branch.
We need to just move the generated file into the proper directory, which
will hopefully make it show up at
https://github.com/Xeeynamo/sotn-decomp/tree/gh-duplicates. I'm very
hopeful that this will work, and not be yet another "Fix CI" commit...
I also added a space to an output string. Doesn't really matter but
might as well make things look a tiny bit nicer.
`func_8011AC3C` appears to be a very important function when it comes to
spawning entities. From what I can tell, it seems to be responsible for
creating all of the entities listed in `asm/us/dra/data/CF4C.data.s`
from 800AD0C4 through 800AD1D0, which is the game's main entity updating
function lookup table.
When func_8011AC3C is called, it uses its `self.params` to index into a
table which contains spawning information for entities. For example, if
func_8011AC3C is called with `self.params` of 0x21, it will create a
child entity with entityID of 25, which is the entity for the Hellfire
spell (which I recently decompiled and will submit as a PR soon).
I think func_8011AC3C is going to be useful for understanding entity
spawning logic, so I have gone ahead and taken the data table that it
uses at D_800AD1D4, and created it as an asset which will now be
extracted by splat. This way, we can more directly view the data, rather
than just seeing it as a giant array of u8 values. I am hoping that
having this available as json will be helpful for understanding the
entity creation logic. Overall, this PR is meant as a stepping stone to
fully understanding func_8011AC3C and what it does to create the
entities that it creates.
Doing this has helped me understand one struct member's purpose, which
is to set the ID of the child entity, so I've added that to the struct,
both the entity extension for func_8011AC3C and the struct which this is
an array of, which for now is Unkstruct_800AD1D4.
This took me a while. I used a [throw-away
script](https://gist.github.com/Xeeynamo/58da1ff8f3831d0ba5d23da27cbca025)
to help me with the quest, but I still had to manually check every
single YAML subsegment.
I transported over what we were able to decompile from main. For a few
functions I added the signatures and documented existing DRA code,
especially on the sound department (got inspired by the recent @sozud
PRs).
All the `.text` part from `main.exe` is now completely extracted as
C/ASM where appropiate. Almost all the functions have their original
PSY-Q names but 7 of them: `func_80012DBC`, `func_80012F84`,
`func_80017008`, `func_80017078`, `func_8001929C`, `func_80021F0C` and
`func_800286E0`. I did not feel confident enough to rename them, so I
left them be. The rest of the functions I am 99% sure they are all
accurate.
I am now excluding the
[asm/](https://github.com/Xeeynamo/sotn-decomp/tree/master/asm/us/main/psxsdk)
folder from the repo. It was useless.
This research confirms me the game uses the PSY-Q 3.5 libraries, with
the exception of `LIBGPU.LIB`, which is from PSY-Q 3.0 for some unknown
reason.
EDIT: `make format` was not taking care of duplicated symbols in our
symbol list. To speed-up my work, all the duplicate symbols (duplicates
= name AND offset) are now removed.
Markdown file is currently not being created in the CI. This rearranges
how we manage that, and also adds a debugging line which might help. I
don't know the nature of the error, but this will attempt to solve it,
and if not, hopefully provide some kind of information to suggest why it
isn't working.
Small change here, just linking to the raw SVG so it's clickable. The
github preview makes the bubbles no longer operate as links.
While we're at it, would it be okay if I moved the function_graphs.md up
a directory, so that it sits alongside duplicates.md and functions.txt,
rather than being mixed in the function_calls folder with all the svgs?
In the interest of improving the diffs shown in "Update reports" commits
(example here:
50e507f0c0),
we recently changed the SVG generation code to be more deterministic,
such that any differences are only the result of actual changes to the
source code of the repo.
While perusing this latest diff, I noticed that the index.html is
showing as a "large diff", which is because the generated HTML code does
not contain any newlines. To be clear, HTML has its own tags which
control the lines in the final rendered page, but the actual text in the
HTML file does not care about whether the code is on multiple lines or
is all in one line.
For the sake of generating diffs, it is heavily preferred for the HTML
to run as multiple lines. With this change, the raw HTML file will be
more intelligible and diffs should be able to properly show individual
changes, rather than struggling to deal with changes that are
technically all within a single-line file.
As requested by Sozud, this will sort the callees and callers
alphabetically when generating the call graphs. Therefore, between two
runs of the tool, the result should be more deterministic.
This also has the nice side effect that, because sorting places capital
letters first, decompiled functions with names with capital letters
(such as **A**llocEntity) will be placed before those with func_, so
they are highlighted.
Of course, if a function's name changes, it will move around in the call
graphs, but hopefully this change will mean we only see movement in the
graphs between two CI runs in the places where relevant changes have
occurred.
Small bugfix, little overall impact. Should be able to merge, but if
you're curious what I changed here:
One of the important functions in analyze_calls is the one that resolves
ambiguous function names for duplicates. For example, when
UpdateEntityRichter calls UpdateAnim, it is unclear whether it is
calling the `dra` version of that function, or the `servant` version of
that function. `find_func_match` aims to resolve those ambiguities by
matching based on the file paths. Something which is in, for example,
`no3`, is almost certainly calling the `no3` version of a function. We
can resolve the vast majority of function naming ambiguities this way.
Due to an oversight in an `assert` statement (forgot to use an ==1), we
were not properly detecting failures in resolving those ambiguities.
Thus, if you look to
https://raw.githubusercontent.com/Xeeynamo/sotn-decomp/gh-duplicates/function_calls/ric.UpdateEntityRichter.svg,
you will see (4th from the far right) that we thought this RIC function
was calling `servant/UpdateAnim`. This is of course incorrect. It turns
out that, currently, UpdateAnim is the only function which has this type
of overlap. Notably, most functions that share names are because they
are identical duplicates, but these UpdateAnim functions are not
identical (though they are similar). For now, I've hard-coded this to
use the `dra` version of UpdateAnim.
I also fixed a problem with the __repr__ method of `sotn_function`,
which is only used in debugging.
I have also made some slight adjustments to the formatting of the
rendered graphs. The numbers next to the arrows, which indicate how many
times a given function is called, are now adjusted to be further from
the central node, so that it is more clear which number belongs to which
arrow. The graphs have also been stretched vertically, for the same
reason, to reduce how much the arrows bunch up. For large functions this
should be especially beneficial.
I have rewritten much of the overall logic flow of analyze_calls. This
should make it run faster. Hopefully the program logic is also easier to
understand.
The output of this script is mostly identical to the previous version.
It will now include calls through g_api to a given function, which is a
nice upgrade (for example, a call to g_api.func_800FE97C will now
properly show up on func_800FE97C's call graph).
Important change: I am now using the argparse library to parse
arguments. Running without arguments will generate all the call trees,
and save the files along with the HTML page that links to all of them.
There are two optional arguments:
`--dry` will render all call trees, but will not save them to files. The
bytes of the SVG are created, but are then simply forgotten.
`--ultradry` will perform the analysis of every function, what it calls,
and what calls it, but will stop there without generating any trees.
I believe that `--ultradry` will be sufficient to automatically run in
PRs, since the majority of problems are in the function analysis stage,
and not in the call tree generation stage. Therefore, for the CI, I
would recommend that we set PRs to run with `--ultradry`, and the main
repo to run with no arguments. In the event that we find a situation
where `--ultradry` misses an error that `--dry` catches, we can revert
to using `--dry` on PRs.
For some benchmarking: Normal runs and --dry runs take approximately 30
seconds on my computer. It turns out that writing to files is a
relatively small amount of the execution time. On the other hand,
`--ultradry` takes less than 1 second. This is the main reason I suggest
using `--ultradry` for PRs.
This will hopefully be applicable to the stalled
`weapon0-weapon1-makefile` branch.
Note: The previous run modes of this script (running with command line
interface, or running generating single flow charts one at a time) are
now removed. If anyone was relying on them, I can reimplement them, but
I assume the web interface is more convenient for everyone.
Splits `BIN/WEAPON0.BIN` in its own individual C files and assets. There
are 59 overlays 59 PNGs extracted as part of it. The `.data`, `.text`
and `.sbss` section are correctly split. I did not yet split the
`.rodata` but I assume it will be so small we can do it on-the-go.
This is fully integrated in the build process, it gives an 🆗 once
built back and it is fully integrated in the CI. We did not yet know how
weapons work in-game and we already have a few instances in the code
where we call `D_8017A000();` and similar. I think it is time to start
documenting them.
Thanks to @bismurphy <bismurphy@users.noreply.github.com> for addressing `analyze_calls.py` for this specific
PR.
Two major changes here:
Running `python3 analyze_calls.py ALL_DRY` will do a "dry run" of
generating all the call graphs. It will generate sotn_calltree.txt, and
then it will generate all the SVG files, but crucially they will not be
saved to .svg, they will just be generated and then forgotten. No HTML
file will be generated. Then, sotn_calltree.txt will be deleted.
All known failure states for the tool now use exit(#) with different
exit codes depending on the type of failure. This should make it easier
for the CI to detect problems.
Note that this PR only allows analyze_calls to do the dry run - it does
not include the actual workflow modifications that would be needed to
actually do the dry-run.
Should fix the issues now by adding the last few missed symbols.
These are placeholder names for now and should be changed to the actual
SDK call names. For now this is functional.
This works locally to fix things.
The new changes to analyze_calls relies on symbols.us.txt including a
symbol for every single function in main. This was not the case - many
functions which do not have names, for example, func_80020D3C did not
have an entry in symbols, because splat would just autogenerate a symbol
name with the numeric placeholder.
This PR adds explicit symbols for every function that was previously
throwing errors. I just did this by manually patching in each function
until analyze_calls would run successfully. Hopefully it works in the CI
just as it works on my own computer.
---------
Co-authored-by: bismurphy <bismurphy@users.noreply.github.com>
Following the incredible work of @bismurphy in #396, this will integrate
the tool in the CI. The markdown file `gh-duplicates/functions.md` will
have clickable links to navigate right to the SVG call tree of the
correspondent functions.
---------
Co-authored-by: sozud <122322823+sozud@users.noreply.github.com>
Very happy to have this working!
Now, when you run `python3 analyze_calls.py ALL`, it will generate the
call graphs for every function in the game, and save as an SVG file in
the directory `tools/generated_graphs`. It will then finish by creating
an `index.html` that links to every one of those graphs. Each graph's
bubbles are clickable to take you to the next graph.
I have deployed an example of the result of this, linked in the sotn
decomp Discord, but because it is hosted on my personal computer, I will
not be linking it here (accessible to the wider internet and crawlers,
etc). It would be great if this script could be worked into the CI to
create a publicly visible list of functions linking to graphs, which
would automatically update whenever the repo gets updated.
As always, be sure to run `make force_extract` before invoking this.
It will be very exciting once this is up and running!
The charts generated by analyze_calls will now show how many times a
given function is called in each instance. Here is an example output of
the new form, where the arrows are each labeled with a number:
https://imgur.com/a/F5HLjbE
Similarly, in command-line mode, the output for this looks like:
```
Analyzing func_80132A04; Decompiled: True
Functions called:
SsUtKeyOnV called 2 times; Decompiled: N/A
func_801327B4 called 1 times; Decompiled: False
Functions which call this:
Called by func_80135D8C 2 times; Decompiled: False
Called by func_80135624 1 times; Decompiled: True
Called by func_80134C60 1 times; Decompiled: True
Called by func_80134D14 1 times; Decompiled: True
```
As usual, this may have bugs but appears to work well; I very much
welcome anyone else's changes to this tool :)
Had a bug in the sequencing for analyze_calls, this fixes it. Ran into
this with 800FA4D0. There's a chance this introduces more bugs but if so
I'll deal with them as I find them.
This is the script demonstrated in Discord, with both graphical and
command line interfaces.
I won't say it's particularly good Python, but it gets the job done for
now :)
This traverses all functions in /asm (after doing `make force_extract`),
and logs all functions called by every function. This takes several
seconds, so the result of this traversing is saved to a text file called
sotn_calltree.txt and used for future runs. I've added this to the
gitignore. As mentioned in the comments at the top of the Python file,
this file can be deleted if the user would like to re-generate the call
tree (mainly if symbols have been renamed, which would change the
generated assembly files).
This is very much meant to be a first draft; it may have bugs and it
definitely has ugly, low-quality code (I had to do a whole bunch of
special cases for assembly that calls functions indirectly, for
example), but at least for now this could be a useful starting point.
Hopefully myself or others will make improvements into the future!
