Game-Decompilation/dinosaur-planet
1
0
Fork 0
mirror of https://github.com/zestydevy/dinosaur-planet.git synced 2024-11-23 13:29:46 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Documentation overhaul (#84)

Browse Source 
* Add Overview.md

* Add Recompilation.md

* Fix executable flag for some tools

* [WIP] Guide.md

* Finish decomp guide

* Update CONTRIBUTING.md + other doc tweaks/fixes
This commit is contained in:
Ethan Lafrenais 2022-04-13 22:38:53 -04:00 committed by GitHub
parent 05aaf1ba93
commit 8ebf8a5ee9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 585 additions and 87 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

97
CONTRIBUTING.md
View File

@ -14,105 +14,36 @@ If you're interested in contributing, please consider joining us on Discord over
- [How does decompilation work](#how-does-decompilation-work)
- [Things to know before contributing](#things-to-know-before-contributing)
- [How can I contribute?](#how-can-i-contribute)
- [Documenting code](#documenting-code)
- [Decompiling functions](#decompiling-functions)
- [Style guide](#style-guide)
- [Opening a pull request](#opening-a-pull-request)
- [Resources](#resources)
## How does decompilation work
The goal behind the decompilation is to extract and convert the game's code from MIPS machine code to human readable MIPS assembly and ideally C code. Once this code has been converted, the game can then be recompiled using the same compiler and assembler as the original game to produce an identical ROM. A identical recompiled ROM is important because it verifies that the converted code is, at least, functionally 100% the same as the original ROM.
The goal behind the decompilation is to extract and convert the game's code from MIPS machine code to human readable MIPS assembly and ideally C code. Once this code has been converted, the game can then be recompiled using the same compiler as the original game to produce an identical ROM. A identical recompiled ROM is important because it verifies that the converted code is, at least, functionally 100% the same as the original ROM.
At a high level, the process goes like this:
1. Disassemble the ROM's executable machine code into MIPS assembly code (found under `asm/`).
- This is done in part by using a tool called [splat](tools/splat/README.md), which splits the code into separate files. See `splat.yaml` for the configuration.
2. Import assembly code in C source files on a function-by-function basis using `#pragma GLOBAL_ASM(...)` (found under `src/`).
- How splat splits code into different files also affects this part. You'll notice that for each folder under `asm/nonmatchings/` there is a counterpart C source file.
1. Disassemble the ROM's executable machine code into MIPS assembly code.
2. Import assembly code in C source files on a function-by-function basis using `#pragma GLOBAL_ASM(...)`.
3. Replace each `#pragma GLOBAL_ASM(...)` by decompiling the included assembly code into a C function.
4. Recompile the ROM using IDO 5.3. If the recompiled ROM matches the original ROM, the function decompiled in step 3 is correct!
- Note: IDO (IRIS Development Option) is a set of compiler tools developed for the IRIX operating system and do not run natively on x86. This project uses a [recompiled version of each program](https://github.com/Emill/ido-static-recomp), which you can find under `tools/ido_recomp/`.
4. Recompile the ROM. If the recompiled ROM matches the original ROM, the function decompiled in step 3 is correct!
## Things to know before contributing
Before working on the decompilation, you should:
Before working on the decompilation, you should (ideally):
- Have a decent understanding of the C89 programming language.
- Be at least a little familiar with the MIPS instruction set and/or assembly programming.
- Understand the basics, purpose, and relationship of compilers and assemblers.
- Be at least a little familiar with the MIPS instruction set and/or assembly programming in general.
- Understand the basics of compilers and assemblers.
- Have some familiarity with compiler optimizations.
It's perfectly okay if you don't know all of this right away, but you will need to get familiar with these topics to create matching code.
Some documents to help:
- [MIPS IV Instruction Set](http://math-atlas.sourceforge.net/devel/assembly/mips-iv.pdf)
- [C89 Draft](https://port70.net/~nsz/c/c89/c89-draft.html)
### Compiler requirements
#### Line endings
If you plan on building the ROM on your machine, it's very important that the repository is cloned with Unix line endings as the compiler will not recognize Windows line endings.
#### Compiling on Windows
The compiler will not run natively on Windows and will need to be ran using something like [WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10).
It is perfectly okay if you don't know all of this right away! Decomp can be fairly involved and it's normal to need to learn things as you go.
## How can I contribute?
There's different ways you can contribute to the decompilation, some don't involve directly decompiling code at all!
The most important work that needs to be done currently is decompiling and understanding functions and data structures.
- You could help [document the codebase](#documenting-code) by giving useful names to variables and functions, defining data structures, and writing in-code documentation.
- If you're comfortable with or interested in the [decompilation process](#how-does-decompilation-work), you could help [decompile and match functions](#decompiling-functions).
### Documenting code
Code documentation is an ongoing effort and isn't necessarily complete even for matching code. Using an emulator with debugging support (such as [Project 64](https://www.pj64-emu.com/)) you can play around with functions and variables to determine how they work and what their purpose is.
There's two ways to find the addresses of functions and global variables:
- The `symbol_addrs.txt` and `undefined_syms.txt` files.
- The `dino.map` file produced by building a ROM.
#### Naming symbols
If you've found a good name for a function, global variable, or are looking to label a new memory address:
1. Make sure the name follows the project's [style guide](#style-guide).
2. If this is an existing symbol, be sure to change all references in C code to the new name.
3. Add the new name/address to `symbol_addrs.txt` for new symbols, otherwise rename the existing one. **IMPORTANT:** If the symbol is not referenced statically in assembly code, you will need to define the symbol in `undefined_syms.txt` instead.
4. For new global variables, if you're confident about the data type, consider defining it in `include/variables.h`.
5. For functions that haven't been decompiled yet, be sure to update its `GLOBAL_ASM` include.
6. Rebuild the ROM to verify nothing broke.
#### Defining structs
If you've discovered a new struct or more about an existing one:
- For new structs, be sure to update the global variable's type that uses it. Also consider defining the struct in `include/variables.h` if there isn't a more appropriate header file for it.
- For existing structs, update the existing definition. **IMPORTANT:** Be wary of updates to structs already in use. You must make sure your change doesn't break existing decompiled code, and if it does, you'll need to update that as well.
- Like always, be sure to follow the project's [style guide](#style-guide).
#### Writing in-code documentation
If you would like to document an explanation or bits of information for a function or variable:
1. Make sure the symbol of the function/variable is in code (see [Naming symbols](#naming-symbols)).
2. Add a C comment to the function/variable with any information you would like to add. **Be sure to follow the [style guide](#style-guide)**, this project prefers [doxygen](https://www.doxygen.nl/index.html) styled block comments.
### Decompiling functions
> Be sure to read the [How does decompilation work](#how-does-decompilation-work) section first to understand the process at a high level.
Decompiling MIPS assembly into C code is where most of the project's development time will be spent and where help is needed the most. There's many different methods for decompiling code that really comes down to preference. For an overview of how to do this:
1. Find a nonmatching function that you would like to decompile.
- This could be a function that is still just a `GLOBAL_ASM` include, or one that has already been decompiled but isn't producing matching code when recompiled.
2. Replace the `GLOBAL_ASM` include (if necessary) with a C function that works just like the original assembly code.
- For an in depth decompilation guide, please see [-O2 decompilation (for IDO 5.3 and 7.1)](https://hackmd.io/vPmcgdaFSlq4R2mfkq4bJg).
- There's many tools that can assist in this process like a decompiler such as [mips2c](https://simonsoftware.se/other/mips_to_c.py) or [Ghidra](https://ghidra-sre.org/) (Ghidra needs a [custom loader for the ROM](https://github.com/HugoPeters/N64LoaderWV-DinoPlanet)). However, a decompiler isn't required. If preferred, you could decompile functions by hand. Decompilers usually don't result in matching code and will require additional work. Sometimes decompiling by hand can be easier.
3. Tweak the C implementation until it produces a matching ROM or is at least functionally equivalent (ideally with minimal differences).
- Use `./dino.py diff` to compare the recompiled ROM to the original and see which instructions are different.
4. Make sure the code adheres to the project's [style guide](#style-guide).
- If you are new to decomp, please see [our full introductory decompilation guide](docs/Guide.md)!
- If you are already familiar with decomp, you can find project documentation under the [docs](./docs) directory.
## Style guide
@ -133,7 +64,9 @@ Before opening a pull request, you should make sure that:
## Resources
The following is a list of external resources that may be helpful:
- Tools
- [decomp.me](https://decomp.me/) - Online space for decompiling individual functions with a live diff.
- [mips2c (online)](https://simonsoftware.se/other/mips_to_c.py) - Online MIPS to C decompiler.
- [mips2c (offline)](https://github.com/matt-kempster/mips_to_c) - Source of mips2c.
- [Ghidra](https://ghidra-sre.org/) - Reverse engineering suite (including a decompiler).
@ -147,8 +80,12 @@ Before opening a pull request, you should make sure that:
- [-O2 decompilation (for IDO 5.3 and 7.1)](https://hackmd.io/vPmcgdaFSlq4R2mfkq4bJg) - Guide for decompiling code compiled with IDO 5.3/7.1 with the -O2 flag enabled (the majority of the code here).
- [N64 Developers' Manual, Version 5.2](https://jrra.zone/n64/doc/) - An online N64 developer manual (contains documentation for things like operating system functions and more).
- [N64 Gfx Decompilation (F3DEX & S2DEX)](https://hackmd.io/m3E7g2YaSxiwUv9QKBLYyA) - Guide for decompiling N64 graphics code.
- [Nintendo Ultra64 RSP Programmer’s Guide](https://ultra64.ca/files/documentation/silicon-graphics/SGI_Nintendo_64_RSP_Programmers_Guide.pdf) - Full documentation for the Nintendo 64 Reality Signal Processor.
- [IDO `cc` Manual](https://cocky-wescoff-177c47.netlify.app/cc_manual.html) - Manual for the IDO C compiler.
- [C89 Draft](https://port70.net/~nsz/c/c89/c89-draft.html) - Useful to understand the subset of C required by IDO 5.3 compared to modern C.
- [MIPS System V Application Binary Interface](https://refspecs.linuxfoundation.org/elf/mipsabi.pdf) - Full documentation of the the MIPS ABI.
- [MIPSpro™ Assembly Language Programmer's Guide](https://techpubs.jurassic.nl/library/manuals/2000/007-2418-001/pdf/007-2418-001.pdf) - IRIX Manual for writing MIPS assembly code.
- [MIPS Compiling and Performance Tuning Guide](https://techpubs.jurassic.nl/library/manuals/2000/007-2479-001/pdf/007-2479-001.pdf) - IRIX manual for understanding things like assembler flags and internals.
- Other
- [libreultra](https://github.com/n64decomp/libreultra/) - A matching decompilation of libultra.
- [N64 development tools](https://github.com/glankk/n64) - Collection of tools including a F3DEX2 display list disassembler/decompiler.

13
README.md
View File

@ -1,5 +1,7 @@
![Dinosaur Planet Decompilation](docs/banner.png)
[![](https://img.shields.io/badge/Discord-Dinosaur%20Planet%20Community-5865F2?logo=discord)](https://discord.gg/H6WGkznZBc)
A WIP decompilation of Dinosaur Planet for the Nintendo 64, as released by Forest of Illusion on Feb. 20, 2021.
**Note**: To use this repository, you must already have a ROM for the game.
@ -44,6 +46,13 @@ builds is not 'shiftable', so cannot be used yet as a source code base for gener
2. Rebuild the ROM:
- `./dino.py build`
### Documentation
Please see the [docs](./docs) directory for project documentation such as:
- [an overview of the project](./docs/Overview.md)
- [a break-down of the ROM extraction and build system](./docs/Recompilation.md)
- [an introductory guide to decomp](./docs/Guide.md)
- and more!
### Tools
The repository comes with a bunch of tools for decompilation and managing the repository:
- `dino.py` - An all in one script for working with the repository. Some of the common commands:
@ -73,6 +82,6 @@ docker run --rm -it -v $(pwd):/dino dpdecomp dino build
```
## Contributing
PRs are welcome. Please make sure that the ROM builds and matches successfully before submitting a non-draft PR; the CI system will also verify this.
Pull requests are welcome! Please see our [contribution guide](./CONTRIBUTING.md) for more information on how this project works and how to contribute.
Also please see our [contribution guide](./CONTRIBUTING.md) for more information on how this project works and the ways to contribute.
If you're interested in contributing, please also consider joining us on Discord over at the [Dinosaur Planet Community server](https://discord.gg/H6WGkznZBc) in the `#decompilation` channel!

2
dino.py Normal file → Executable file
View File

@ -192,7 +192,7 @@ class DinoCommandRunner:
subprocess.check_call(args)
def main():
parser = argparse.ArgumentParser(description="Quick commands for working on the Dinosuar Planet decompilation.")
parser = argparse.ArgumentParser(description="Quick commands for working on the Dinosaur Planet decompilation.")
parser.add_argument("-v", "--verbose", action="store_true", help="Show actual commands being ran.", default=False)
subparsers = parser.add_subparsers(dest="command", required=True)

378
docs/Guide.md Normal file
View File

@ -0,0 +1,378 @@
[[← back]](README.md)
# Decompilation Guide
This document is intended to be an introductory guide to decomp (tailored specifically for Dinosaur Planet). If this is your first time with decomp then please read on! If you're already familiar with decomp then you may still find some useful information here, as this document is a little specific to Dinosaur Planet.
## Contents
- [1. Introduction](#1-introduction)
- [2. Decompiling a Function](#2-decompiling-a-function)
- [2a. Assembly Extraction](#2a-assembly-extraction)
- [2b. Reversing](#2b-reversing)
- [3. Symbols](#3-symbols)
- [4. Data Sections](#4-data-sections)
- [5. Optimization Levels and Debug Symbols](#5-optimization-levels-and-debug-symbols)
- [6. Decompilation Tools](#6-decompilation-tools)
## 1. Introduction
It's important to first understand exactly what this project is attempting to accomplish. The goal is to create a matching decompilation. This means that when all reversed code is recompiled, the re-created ROM is byte-for-byte identical to the original.
This process is typically done on a per-function basis. The raw machine code for a function is extracted/disassembled from the original ROM into readable MIPS assembly code. Then, that function is decompiled into C code that when recompiled results in the exact same machine code that it started from.
Before continuing, please consider reading the [overview of the project](./Overview.md) to get an idea of where everything is located. Don't worry if not everything makes sense yet, this document will attempt to explain more in detail.
## 2. Decompiling a Function
Let's go over the full process of decompiling a single function. This section will be using the already decompiled function `vec3_normalize` as an example (you can find the final source in `src/vec3.c`).
### 2a. Assembly Extraction
Before we can begin decompiling a function, we need to extract its assembly code from the ROM. This is done automatically using a tool called [splat](https://github.com/ethteck/splat). Currently, splat is already configured to extract most functions. Splat's configuration for Dinosaur Planet can be found at `splat.yaml` in the repository root.
Splat works by mapping an address range of the ROM to individual files. For example, `vec3_normalize` is currently part of the `vec3.c` file. You can see this mapping in `splat.yaml`:
```yaml
# From 0x17180 until the next entry is mapped to src/vec3.c
- [0x17180, c, vec3]
```
With this configuration, splat will take the recompiled contents of `vec3.c` and stitch them into the recompiled ROM starting at that exact address.
The final step is to temporarily include the original assembly of each function that is part of the file. This is done using the special pragma `GLOBAL_ASM`:
```c
#pragma GLOBAL_ASM("asm/nonmatchings/vec3/vec3_normalize.s")
```
Splat extracts each function it finds into the `asm/nonmatchings` directory as individual `.s` files grouped by the name of the mapped file they are a part of. The `GLOBAL_ASM` pragma is used by another tool, [asm-processor](https://github.com/simonlindholm/asm-processor), which is responsible for actually including the referenced `.s` file at that position in the C source.
Great! Now, we can start to decompile this function.
### 2b. Reversing
First, let's create some space to start decompiling and reversing `vec3_normalize`. We'll use an `#if` directive to temporarily replace the `GLOBAL_ASM` pragma with an actual C function:
```c
#if 0
#pragma GLOBAL_ASM("asm/nonmatchings/vec3/vec3_normalize.s")
#else
// We don't know anything about the function signature yet, so we'll
// just make it void for now.
void vec3_normalize() {
}
#endif
```
Next, let's look at the full assembly code of the function:
```
glabel vec3_normalize
addiu $sp, $sp, -0x18
sw $ra, 0x14($sp)
lwc1 $f2, ($a0)
lwc1 $f14, 4($a0)
lwc1 $f0, 8($a0)
mul.s $f4, $f2, $f2
sw $a0, 0x18($sp)
mul.s $f6, $f14, $f14
add.s $f8, $f4, $f6
mul.s $f10, $f0, $f0
jal sqrtf
add.s $f12, $f10, $f8
mtc1 $zero, $f16
lw $a0, 0x18($sp)
mov.s $f12, $f0
c.eq.s $f0, $f16
lui $at, 0x3f80
bc1tl .L8001678C
mov.s $f0, $f12
mtc1 $at, $f18
lwc1 $f4, ($a0)
lwc1 $f10, 4($a0)
div.s $f2, $f18, $f0
lwc1 $f16, 8($a0)
mul.s $f6, $f4, $f2
nop
mul.s $f8, $f10, $f2
nop
mul.s $f18, $f16, $f2
swc1 $f6, ($a0)
swc1 $f8, 4($a0)
swc1 $f18, 8($a0)
mov.s $f0, $f12
.L8001678C:
lw $ra, 0x14($sp)
addiu $sp, $sp, 0x18
jr $ra
nop
```
At this point, it's very helpful to have a [MIPS assembly manual](https://www.cs.cornell.edu/courses/cs3410/2008fa/MIPS_Vol2.pdf) on hand so we can look up the meaning of each instruction. Each line in the above disassembly is a single instruction in the format `mnemonic [operand1,operand2,...]`. For example, the first instruction is the mnemonic 'addiu' with the operands 'sp,sp,-0x18'.
For the first step of decompilation, we'll convert each individual instruction into the equivalent C code:
```c
void vec3_normalize(f32 *a0) {
// addiu $sp, $sp, -0x18 ; We'll skip the first two instructions,
// sw $ra, 0x14($sp) ; which are just part of the stack setup
// ; First is some float value loads from $a0,
// lwc1 $f2, ($a0) ; the first parameter of the function
f32 f2 = *a0;
// lwc1 $f14, 4($a0)
f32 f14 = *(a0+4)
// lwc1 $f0, 8($a0)
f32 f0 = *(a0+8)
// mul.s $f4, $f2, $f2 ; Next, is some arithmetic
f32 f4 = f2 * f2;
// sw $a0, 0x18($sp) ; There's an upcoming function call that
// ; doesn't use the $a0 register, so the code
// ; must save it to the stack first
*(sp+0x18) = a0;
// mul.s $f6, $f14, $f14
f32 f6 = f14 * f14;
// add.s $f8, $f4, $f6
f32 f8 = f4 + f6;
// mul.s $f10, $f0, $f0
f32 f10 = f0 * f0;
// jal sqrtf ; Here we have a function call with a
// add.s $f12, $f10, $f8 ; branch delay slot. The delay slot is
// ; executed first in this case.
f32 f12 = f10 + f8;
// ; Note: Normally, function calls will use
// ; the a0-a3 registers but sqrtf is
// ; implemented in assembly. Looking at its
// ; code shows it takes $f12 as its input
// ; and outputs to $f0
f32 f0 = sqrtf(f12);
// mtc1 $zero, $f16
f32 f16 = 0;
// lw $a0, 0x18($sp) ; $a0 is restored from the stack
a0 = *(sp+0x18);
// mov.s $f12, $f0
f32 f12 = f0;
// c.eq.s $f0, $f16 ; Here we have a conditional branch with
// lui $at, 0x3f80 ; a 'likely' delay slot. The delay slot
// bc1tl .L8001678C ; in this case is only executed if the
// mov.s $f0, $f12 ; condition is true
int at = 0x3f800000;
if (f0 == f16) {
f0 = f12;
goto L8001678C;
}
// mtc1 $at, $f18 ; More arithmetic
f32 f18 = at;
// lwc1 $f4, ($a0)
f32 f4 = *a0;
// lwc1 $f10, 4($a0)
f32 f10 = *(a0+4);
// div.s $f2, $f18, $f0
f32 f2 = f18 / f0;
// lwc1 $f16, 8($a0)
f32 f16 = *(a0+8);
// mul.s $f6, $f4, $f2
f32 f6 = f4 * f2;
// nop
// mul.s $f8, $f10, $f2
f32 f8 = f10 * f2;
// nop
// mul.s $f18, $f16, $f2
f32 f18 = f16 * f2;
// swc1 $f6, ($a0)
*a0 = f6;
// swc1 $f8, 4($a0)
*(a0+4) = f8;
// swc1 $f18, 8($a0)
*(a0+8) = f18;
// mov.s $f0, $f12
f32 f0 = f12;
L8001678C:
// lw $ra, 0x14($sp) ; Finally, the stack is restored from
// addiu $sp, $sp, 0x18 ; the previous frame
// jr $ra ; Note: Returning a float uses the $f0
// ; register instead of $v0 like normal.
// ; We can infer that a float is returned
// ; since $f0 is set at the end of the
// ; function and not used.
return f0;
// nop
}
```
> Whew! That was a bit of code to go through and annotate. It's worth mentioning at this point that auto-decompilers can be used instead of manually translating each instruction, such as [mips2c](https://simonsoftware.se/other/mips_to_c.py) and [Ghidra](https://ghidra-sre.org/).
>
> It's unlikely that a tool will decompile assembly into a perfectly matching function, but it can still save a lot of time.
Now that we have each instruction translated, we can start analyzing the function and start reducing code into readable C (currently the code won't even compile).
The first interesting code we can see is the use of `$a0`:
```c
f32 f2 = *a0;
f32 f14 = *(a0+4);
f32 f0 = *(a0+8);
// ...
*a0 = f6;
*(a0+4) = f8;
*(a0+8) = f18;
```
This looks a lot like the access of struct fields. From this, we could assume that `$a0` holds a pointer to a structure with three four-byte wide fields. Since this function is known to be working with three-dimensional vectors, let's use the following definition:
```c
typedef struct {
f32 x, y, z;
} Vec3f;
```
Additionally, we were able to infer that this function returns a single float. Now that the return value and parameters of the function are known, let's update the signature:
```c
f32 vec3_normalize(Vec3f *a0)
```
Next up, let's start reducing and cleaning up the function implementation. Many of the local variables derived from `$f` registers aren't individual variables but are part of larger expressions.
```c
f32 vec3_normalize(Vec3f *a0) {
f32 f18, f2;
f32 f12 = sqrtf((a0->z * a0->z) + ((a0->x * a0->x) + (a0->y * a0->y)));
if (f12 == 0.0f) {
goto L8001678C;
}
f18 = 0x3f800000;
f2 = f18 / f12;
a0->x = a0->x * f2;
a0->y = a0->y * f2;
a0->z = a0->z * f2;
L8001678C:
return f12;
}
```
This looks significantly better already (and compiles)! Let's check our progress so far by diffing our implementation against the original ROM. We can do this by building the ROM (`./dino.py build`) and then running diff on the function's symbol (`./dino.py diff vec3_normalize`). Looks like everything matches except for one instruction: `lui at,0x3f80`, which from our implementation is currently `lui at,0x4e7e`. If we trace this instruction back to the C code, we can see it's from the line `f18 = 0x3f800000;` Let's fix it.
Looking back at the assembly, we can see these two instructions:
```
lui at,0x3f80
mtc1 at,$f18
```
These two instructions are actually 'bitwise' casting the integer `0x3f800000` into a float, rather than simply assigning the float 1,065,353,216.0 to `f18`. Let's use [a floating point converter](https://www.h-schmidt.net/FloatConverter/IEEE754.html) to see what `0x3f800000` is bit-for-bit as a float. Turns out it's actually `1.0`! If we replace the statement with `f18 = 1.0f;`, re-build, and re-run the diff, we can now see that the function matches!
Before we call this function done however, let's clean it up by getting rid of the `goto` and using better variable names based on what we know about vector normalization:
> Note: Most functions will not use `goto` at all and usually won't match with them still in place. Instead, branches should be translated into `if` and sometimes simple `switch` statements.
```c
f32 vec3_normalize(Vec3f *v) {
f32 length, lengthInv;
length = sqrtf((v->z * v->z) + ((v->x * v->x) + (v->y * v->y)));
if (length != 0.0f) {
lengthInv = 1.0f / length;
v->x = v->x * lengthInv;
v->y = v->y * lengthInv;
v->z = v->z * lengthInv;
}
return length;
}
```
Congratulations 🎉, you've gone through the full decompilation process of a function!
There is still so much more to reversing assembly, but this is the general process of breaking it down one step at a time. For much more information on decompiling assembly from IDO, please see the ever-evolving [-O2 decompilation (for IDO 5.3 and 7.1)](https://hackmd.io/vPmcgdaFSlq4R2mfkq4bJg).
## 3. Symbols
An important part of decomp is giving names to memory addresses, known as symbols. This includes global/local static variables and functions.
Part of splat's job is to automatically generate symbols for addresses that it sees referenced. The full list of these generated symbols can be found (after running `./dino.py extract`) in the `undefined_funcs_auto.txt` and `undefined_syms_auto.txt` files in the repository root. Initially, splat will also generate a name for each symbol, such as `func_<address>` and `D_<address>` for functions and variables respectively. These names can be overwritten using the `symbol_addrs.txt` file (also in the repository root).
The format of each line in `symbol_addrs.txt` is as follows:
```
<symbol name> = <VRAM address in hex>; // type:<data or func> size:<byte size in hex>
```
For example:
```
gActorCount = 0x800b2934; // type:data size:0x2
texture_load = 0x8003cda8; // type:func
```
The `size` attribute is optional.
Sometimes, there will be an address that should have a symbol that splat was unable to detect. This usually happens when the address is not explicitly referenced in the final assembly code. In this case, the symbol cannot be added to `symbol_addrs.txt` and instead must be added to `undefined_funcs.txt` or `undefined_syms.txt` for functions and variables respectively (found in the repository root). These files are fed directly into the linker and as such should not contain the `type` or `size` attributes (just the format `<name> = <VRAM address in hex>;`).
> Note: When modifying `symbol_addrs.txt`, you will need to re-run `./dino.py extract` to re-generate the linker the script with the new symbols. If you just modify `undefined_funcs.txt` or `undefined_syms.txt`, then this is not required.
## 4. Data Sections
Each compilation unit (`.c` files, `.s`, etc.) contributes code and data to various sections of its resulting object file (and ultimately the final linked ELF file). The common sections are:
- `.text` - Executable machine code, each compiled function ends up here.
- `.data` - *Initialized* global/local static variables.
- `.rodata` - *Initialized* global/local static constants (also includes things like jump tables and string literals).
- `.bss` - *Uninitialized* global/local static variables and constants.
The `.data`, `.rodata`, and `.bss` sections are known as data sections (or data segments). Over the course of decomp, these sections need to be defined in the `splat.yaml` configuration for each file. Data sections are defined very similarly to code in splat, for example:
```yaml
# From 0x92370 until the next entry is the .data section (in the ROM)
# for the resulting object file of src/crash.c
- [0x92370, .data, crash]
```
The preceding example will tell splat to generate the linker script such that the `.data` section of `crash.c`'s object file (`build/src/crash.o`) will be inserted at the address `0x92370` in the final ROM.
> Note: Modifying anything in `splat.yaml` will require you to re-run `./dino.py extract` to re-extract code/data and, more importantly, to re-generate the linker script.
### Example: A `switch` jump table
As an example, let's take a look at the function `func_8005D3A4`, which happens to be the first function in `video.c` with a `switch` statement that compiles into a jump table.
If we look through the first few lines of assembly, we can see that the function references a jump table at the VRAM address `0x8009ACC0`. In order to get this function to match, the `.rodata` section for the file (`video.c`) needs to marked in `splat.yaml` such that the `switch` statement, when compiled, places its jump table at `0x8009ACC0`. To do this, we'll first need to convert the VRAM address of the jump table to its ROM address.
To convert the address, we need to first look at the current splat config. The configured segment for code contains:
```yaml
start: 0x1000 # ROM offset
vram: 0x80000400 # VRAM offset
```
With this, we can convert the VRAM address to the ROM address as follows:
```
(0x8009ACC0 - 0x80000400) + 0x1000 = 0x9B8C0
```
Finally, we tell splat about the `.rodata` section for `video.c`:
```yaml
- [0x9B8C0, .rodata, video]
```
> Note: This works in `video.c`'s case because this jump table is the **first** piece of data contributed to `.rodata` for the file. If there was, for example, another jump table further up in the file, then splat would need the ROM address of that jump table.
## 5. Optimization Levels and Debug Symbols
Just about all game code in Dinosaur Planet was originally compiled with the same compiler flags. However, the game statically links other libraries that were compiled differently. The two [IDO `cc` flags](https://cocky-wescoff-177c47.netlify.app/cc_manual.html) that primarily influence assembly generation are:
- `-O[0-3]` - Defines the optimization level to compile with (`-O0` is off and `-O3` is all optimizations).
- `-g[0-3]` - Defines the symbol tables to be generated for debugging (`-g0` is off and `-g3` is for full debugging).
The flag combinations currently known for the decomp are:
- `-O2 -g3` - The default. Used for almost everything.
- `-O2 -g0` - Used for compiling *modified* libultra code.
- `-O1 -g0` - Used for compiling *unmodified* libultra code.
Sometimes, to get a function to match you will need to change the optimization and debugging levels that the containing file is compiled with. This can be done currently by placing the file under a directory named `O1` (for `-O1 -g0`) or `g0` (for `-O2 -g0`).
> Note: The direct parent directory must be `O1` or `g0` for the flags to be changed. For example, `src/O1/stuff/file.c` will **not** work, but `src/stuff/O1/file.c` will.
## 6. Decompilation Tools
There are many tools out there to assist with decomp. You can find [a full list of tools and other resources in the contribution guide](../CONTRIBUTING.md#resources).
Here are some recommended tools:
### decomp.me
[decomp.me](https://decomp.me/) is an online collaborative space for decomp. Here, you can create a 'scratch' for a function and get a live diff of your current C implementation and the expected assembly code. You can also share a link to your function for others to play around with. The site has a preset for the Dinosaur Planet decomp that will automatically set up the correct compiler version and default flags.
> Note: To use decomp.me, you'll need a 'context' file for the function you're working on. This is essentially just a C header file that contains all of the declarations of types, functions, and macros that are used by your function. You can generate this file by running `./dino.py context <path to your file.c>` (you may need to remove function implementations at the bottom of the generated file).
### mips2c
mips2c can be used to automatically decompile a function from assembly into C. There is an [online](https://simonsoftware.se/other/mips_to_c.py) and [offline](https://github.com/matt-kempster/mips_to_c) version of the tool. This tool can save a lot of time by producing an initial C implementation to work with that you can then tweak until the function matches.
### Ghidra
[Ghidra](https://ghidra-sre.org/) is a full reverse engineering tool suite. It can be used for many things such as exploring the full assembly of the ROM, getting instant C decompilations of each function, tracing references between addresses, and testing function and structure definitions. Ghidra can be a bit overwhelming at first, but once learned it can be extremely valuable for analyzing and understanding functions and data structures.
> Note: To use Ghidra with the Dinosaur Planet ROM, you will need [the custom loader plugin N64LoaderWV-DinoPlanet](https://github.com/HugoPeters/N64LoaderWV-DinoPlanet). This plugin allows Ghidra to read the N64 style ROM and analyze Dinosaur Planet DLL code.

81
docs/Overview.md Normal file
View File

@ -0,0 +1,81 @@
[[← back]](README.md)
# Overview
The following is a general overview of the project's structure.
## Contents
- [Decompiled Code](#decompiled-code)
- [Extracted Code](#extracted-code)
- [Tools](#tools)
## Decompiled Code
Decompiled C and assembly code (finished and unfinished) can be found under the `src`, `include`, and `asm` directories.
### src
The `src` directory contains all of the decompiled C implementation files (`.c` files). Each file (excluding DLL code) is mapped from a specific address range in the base ROM via splat. These files are generally stubbed out by splat when a new address range is marked as C code and contain `GLOBAL_ASM` pragmas that include the original assembly code (in order) for each function in the file. When a function is fully decompiled and matching, the `GLOBAL_ASM` pragma is removed.
#### DLLs
The `src/dlls` directory is special. Dinosaur Planet uses a unique DLL system. Each directory under `dlls` is named after the DLL number it is for. Inside each DLL directory is a single C file for the full implementation of that DLL and an `exports.s` file which defines the exports table for that DLL ([see the full DLL documentation for more information](./DLLs.md)).
Additionally, the `dlls` directory currently contains a special linker script for just DLL code.
### include
The `include` directory contains all of the C header files (`.h`) for the project as well as shared assembly files (`.inc`) that are meant to be included in a `.s` file.
#### libultra
Headers related to libultra are currently split into three directories: `PR`, `libultra`, and `libc`.
- `PR` (project reality) contains the typical headers used when developing an N64 title and only contains public variables, functions, and macros.
- `libultra` contains headers for internal libultra code that has been decompiled (typically these are taken/formatted like the existing libultra decompilation [libreultra](https://github.com/n64decomp/libreultra), however Dinosaur Planet uses a fairly modified version of libultra so not everything is the same).
- `libc` contains headers for... libc!
Additionally, the root of `include` contains the header `ultra64.h` which includes most of the `PR` headers all at once.
#### macro.inc
The `macro.inc` file is included at the top of every extracted assembly file and just contains a prelude of macros needed to reassemble those files.
#### Game code
TODO: document how headers for game systems and general code should be organized
## Extracted Code
Code and data extracted directly from the base ROM are located under the `asm` and `bin` directories respectively.
### asm
With the exception of the `nonmatchings` directory, everything under `asm` are segments of the ROM marked as assembly code that (usually) can't be decompiled into C code. The `nonmatchings` directory contains a single `.s` file for every function included from a `.c` file via `GLOBAL_ASM`. Everything under `asm` is managed directly by splat and shouldn't be modified directly.
### bin
The `bin` directory contains a `.bin` file for every ROM segment marked as generic binary data. This data could be anything from an audio file to the `.rodata` section of a file that hasn't been marked in splat yet.
The single exception to `bin` is the `bin/assets/dlls` directory, which contains individual `.dll` files for every DLL in the game unpacked directly from `DLLS.bin`.
### Symbols and Linker Scripts
The root of the repository contains various files that define symbol addresses, the configuration for splat, and the linker script used for the re-built ROM.
- `dino.ld` - The linker script generated by splat. Should not be modified directly.
- `splat.yaml` - The project configuration for splat. Here is where ROM segments are marked for extraction.
- `symbol_addrs.txt` - Defines addresses for known symbols (only those are referenced directly in assembly code).
- `undefined_funcs.txt` - Defines addresses for functions that are known but aren't referenced directly in assembly code.
- `undefined_syms.txt` - Defines addresses for symbols that are known but aren't referenced directly in assembly code.
- `undefined_funcs_auto.txt` - Addresses of functions that splat found automatically. This file should not be modified directly.
- `undefined_syms_auto.txt` - Addresses of symbols that splat found automatically. This file should not be modified directly.
## Tools
The root of the repository and the `tools` directory contain many scripts and programs used as part of the build system and as utilities for decomp.
- `dino.py` - A general script for common tasks such as extracting and building the ROM as well as diffing.
- `tools/asm_differ` - Diffs assembly between the base and re-built ROM. The project configuration for asm differ can be found in `diff_settings.py` in the repository root.
- `tools/asm_processor` - Processes `GLOBAL_ASM` pragmas when compiling C code by patching in the included assembly into the resulting object file.
- `tools/build_clang_commands` - Generates a `compile_commands.json` file for `clangd`.
- `tools/ido_recomp` - Contains recompiled IDO executables to run on a modern system.
- `tools/splat` - Extracts assembly and data from the base ROM and generates the linker script for the re-built ROM.
- `tools/configure.py` - Configures the Ninja build script file (`build.ninja`) for the project.
- `tools/dino_dll.py` - Packs and unpacks the `DLLS.bin` and `DLLS_tab.bin` files.
- `tools/dlldump.py` - Displays the header, relocation tables, and executable assembly for a given Dinosaur Planet `.dll` file.
- `tools/elf2dll` - Converts a standard ELF file to the unique Dinosaur Planet DLL format.
- `tools/first-diff.py` - Find the first N differences between the base and re-built ROM.
- `tools/first_bin_diff.py` - Find the first N differences between two binary files.
- `tools/m2ctx.py` - Creates a context file for mips2c/decomp.me.
- `tools/progress.py` - Calculates the current progress of the decomp.

8
docs/README.md Normal file
View File

@ -0,0 +1,8 @@
# Dinosaur Planet Decompilation Documentation
Welcome to the project documentation!
## Contents
- [Overview](./Overview.md) - Gives a general overview and explanation of the overall project structure.
- [Decompilation Guide](./Guide.md) - A full introductory guide to decomp for Dinosaur Planet.
- [Recompilation](./Recompilation.md) - Explains the full ROM extraction and build process.

85
docs/Recompilation.md Normal file
View File

@ -0,0 +1,85 @@
[[← back]](README.md)
# Recompilation
The goal of this document is the explain the full process of extracting code and data from the original ROM and then recompiling it back into a new ROM.
## Contents
- [Overview](#overview)
- [ROM Extraction](#rom-extraction)
- [Building](#building)
## Overview
A high level overview of the extraction and build process is as follows:
1. "Split" out readable assembly code and binary data from the original ROM using tools such as [splat](https://github.com/ethteck/splat).
2. Generate a linker script capable of stitching all code and assets back together into a new ROM at the exact same addresses they were extracted from.
3. Reverse extracted assembly code into C.
4. Recompile C code using IDO 5.3 (IRIS Development Option, the original compiler used for the game), reassemble leftover assembly, and convert binary data all into individual object files.
5. Link all resulting object files into a new ROM.
Through this process, code can be incrementally decompiled into C and then constantly recompiled into a full ROM to be compared against the original ROM. If everything was decompiled accurately, the ROMs will be byte-for-byte identical.
## ROM Extraction
### splat
Code and data are extracted from the original ROM primarily using a tool called splat. At the root of the repository is the file `splat.yaml`, which tells splat exactly what address ranges should be extracted and to which files.
With the exception of DLLs, each C file under `src` and binary file under `bin` are mapped to an address range in `splat.yaml`. Additionally, each file has a 'type' that tells splat what kind of data it is (ex. assembly code, C code, binary data, .data, .rodata, etc.).
**Note:** Running `./dino.py extract` will re-run splat, fully rewriting the `asm` and `bin` directories and creating new `.c` stubs under `src` if necessary.
Please see [splat's full documentation for more information](https://github.com/ethteck/splat).
### DLLs
Dinosaur Planet DLLs are in a special format that splat isn't able to simply extract. Instead, splat is configured (by `splat.yaml`) to extract all DLLs to a single `DLLS.bin` file (found under `bin/assets`). This file is then unpacked by the tool `dino_dll.py` (found under `tools`). Each unpacked DLL can be found under `bin/assets/dlls`.
### Linker Script
In addition to extracting code and data, splat is also responsible for generating a linker script that is capable of re-linking all extracted files back into a ROM in the exact same order as the original. After running `./dino.py extract`, this can be found at `dino.ld` in the repository root.
The linker script is vital for creating matching ROMs because it places data from each object file (`.o`) at its original address in the new ROM.
### Symbols
While extracting assembly code, splat will also generate files containing symbols for global variables and functions that it comes across. These symbols will initially have auto-generated names like `func_<address>` and `D_<address>`. Automatically detected variables are placed in `undefined_syms_auto.txt` and functions in `undefined_funcs_auto.txt` (found in the repository root after splat is ran).
To give these symbols actual names, the file `symbol_addrs.txt` is used. Splat reads this file while extracting and will use the names given here for addresses it thinks are variables or functions.
Sometimes, splat won't be able to detect an address as a proper symbol. This usually happens when the address isn't referenced explicitly in assembly. For addresses that should have a symbol, the `undefined_syms.txt` and `undefined_funcs.txt` files can be used to define things like variables and functions respectively.
## Building
Rebuilding all of the extracted and decompiled code/data back into a ROM is done generally in two steps:
1. Compiling all source and binary files into object files (`.o`).
2. Linking all object files into a final ROM.
### Build System
[Ninja](https://ninja-build.org/) is the build system currently used by the project. It's responsible for executing every command necessary to build the full ROM. Instead of maintaining a complex build script for the project, one is generated by the custom script `tools/configure.py`. This script scans every directory for source files, binary files, and linker scripts and decides each step needed to convert them into a single ROM. These steps are then written to a Ninja build script (`build.ninja`), which can be found in the repository root after running `./dino.py configure` (or `build` for the first time).
### C Files
C files are compiled in a three-part process:
1. For partially decompiled files, [asm-processor](https://github.com/simonlindholm/asm-processor) pre-processes the file looking for `GLOBAL_ASM` pragmas. Temporary stubs are created for each block of included assembly.
2. The `.c` file is compiled using IDO into an object file (`.o`).
- Note: IDO (IRIS Development Option) is a set of compiler tools developed for the IRIX operating system and do not run natively on x86. This project uses a [recompiled version of each program](https://github.com/Emill/ido-static-recomp), which can be found under `tools/ido_recomp`.
3. Finally, asm-processor post-processes the resulting object file and patches in the actual included assembly code.
### DLLs
DLLs are a special exception. Instead of being linked directly into the final ROM, each DLL must be linked individually, converted to Dinosaur Planet's DLL format (`.dll`), and finally packed into `DLLS.bin` (before `DLLS.bin` is converted to an object file). The process is as follows:
1. Compile the C source and `exports.s` file into object files.
2. Using a special linker script (found at `src/dlls/dll.ld`), link each object file into an ELF file.
3. Using `elf2dll` (found at `tools/elf2dll`), convert the ELF file to Dinosaur Planet's DLL format (`.dll`).
4. Finally, after all decompiled DLLs have been recompiled, repack all DLLs into a new `DLLS.bin` file using `dino_dll.py` (found at `tools/dino_dll.py`).
After all that, `DLLS.bin` is linked into the file ROM like a normal binary file.
### Assembly Files
All remaining assembly files (`.s`, those not found under `asm/nonmatchings`) are simply assembled using a modern assembler into individual object files.
### Binary Files
All binary files are simply linked into individual object files for the sake of being able to link them into the final ROM.
### Linking
The very final step of building the ROM is linking every object file together into a resulting ELF (the linker script can be found at `dino.ld` in the repository root). The ELF file is then converted to a generic binary file (removing ELF headers), which is the final ROM.

0
tools/configure.py Normal file → Executable file
View File

4
tools/dll_extract.py
View File

@ -120,8 +120,8 @@ def extract(dll: DLL, data: bytearray, src_path: Path, asm_path: Path):
create_c_stub(c_file_path, asm_path, dll.functions)
def main():
parser = argparse.ArgumentParser(description="Extract assembly from a Dinosuar Planet DLL and set up a directory for it that is ready for recompilation.")
parser.add_argument("dll", type=argparse.FileType("rb"), help="The Dinosuar Planet .dll file to read.")
parser = argparse.ArgumentParser(description="Extract assembly from a Dinosaur Planet DLL and set up a directory for it that is ready for recompilation.")
parser.add_argument("dll", type=argparse.FileType("rb"), help="The Dinosaur Planet .dll file to read.")
parser.add_argument("--src", type=str, help="A directory to create source file stubs in (e.g. ./src/dlls/12).", required=True)
parser.add_argument("--asm", type=str, help="A directory extract assembly code into (e.g. ./asm/nonmatchings/dlls/12).", required=True)

4
tools/dlldump.py
View File

@ -103,8 +103,8 @@ def dump_text_disassembly(dll: DLL,
print("(no matching symbols found)")
def main():
parser = argparse.ArgumentParser(description="Display information from Dinosuar Planet DLLs.")
parser.add_argument("dll", type=argparse.FileType("rb"), help="The Dinosuar Planet .dll file to read.")
parser = argparse.ArgumentParser(description="Display information from Dinosaur Planet DLLs.")
parser.add_argument("dll", type=argparse.FileType("rb"), help="The Dinosaur Planet .dll file to read.")
parser.add_argument("-x", "--header", action="store_true", help="Display the contents of the header.")
parser.add_argument("-r", "--reloc", action="store_true", help="Display the contents of the relocation table.")
parser.add_argument("-d", "--disassemble", action="store_true", help="Display assembler contents of the executable section.")

0
tools/m2ctx.py Normal file → Executable file
View File