mirror of
https://github.com/open-goal/jak-project.git
synced 2025-03-07 05:19:44 +00:00
Update documentation and clean up (#129)
* cleanup * clean up file layout
This commit is contained in:
water111
GitHub
parent
55c2330e02
commit
660ef41136
@ -39,7 +39,7 @@ IF (WIN32)
|
||||
ENDIF ()
|
||||
|
||||
option(CODE_COVERAGE "Enable Code Coverage Compiler Flags" OFF)
|
||||
set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake/modules/)
|
||||
set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/third-party/cmake/modules/)
|
||||
|
||||
if(CMAKE_COMPILER_IS_GNUCXX AND CODE_COVERAGE)
|
||||
include(CodeCoverage)
|
||||
@ -62,9 +62,6 @@ set(SPDLOG_BUILD_SHARED ON CACHE BOOL "a" FORCE)
|
||||
# this makes the spdlog includes not use the header only version, making compiling faster
|
||||
add_definitions(-DSPDLOG_COMPILED_LIB)
|
||||
|
||||
# build asset packer/unpacker
|
||||
add_subdirectory(asset_tool)
|
||||
|
||||
# build goos
|
||||
add_subdirectory(common/goos)
|
||||
|
||||
|
||||
286
README.md
286
README.md
@ -8,30 +8,29 @@
|
||||
|
||||
<!-- toc -->
|
||||
|
||||
- [Project Description](#project-description)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Requirements](#requirements)
|
||||
- [Getting Started - Linux (Ubuntu)](#getting-started---linux-ubuntu)
|
||||
- [Getting Started - Windows](#getting-started---windows)
|
||||
- [Project Description](#project-description)
|
||||
- [Project Layout](#project-layout)
|
||||
- [Design](#design)
|
||||
- [Coding Guidelines](#coding-guidelines)
|
||||
- [TODOs](#todos)
|
||||
- [GOAL Decompiler](#goal-decompiler)
|
||||
- [GOAL Runtime](#goal-runtime)
|
||||
- [GOAL Compiler](#goal-compiler)
|
||||
- [Asset Extraction Tool](#asset-extraction-tool)
|
||||
- [Asset Packing Tool](#asset-packing-tool)
|
||||
|
||||
<!-- tocstop -->
|
||||
## Project Description
|
||||
|
||||
## Requirements
|
||||
This project is to port Jak 1 (NTSC, "black label" version) to PC. Over 99% of this game is written in GOAL, a custom Lisp language developed by Naughty Dog. Our strategy is:
|
||||
- decompile the original game code into human-readable GOAL code
|
||||
- develop our own compiler for GOAL and recompile game code for x86-64
|
||||
- create a tool to extract game assets into formats that can be easily viewed or modified
|
||||
- create tools to repack game assets into a format that our port uses.
|
||||
|
||||
Our objectives are:
|
||||
- make the port a "native application" on x86-64, with high performance. It shouldn't emulated, interpreted, or transpiled.
|
||||
- Our GOAL compiler's performance should be around the same as unoptimized C.
|
||||
- try to match things from the original game and development as possible. For example, the original GOAL compiler supported live modification of code while the game is running, so we do the same, even though it's not required for just porting the game.
|
||||
- support modifications. It should be possible to make edits to the code without everything else breaking.
|
||||
-
|
||||
|
||||
We support both Linux and Windows on x86-64.
|
||||
|
||||
- `cmake` for build system
|
||||
- `clang-format` for formatting code (there is already a `.clang-format` provided)
|
||||
- `gtest` for testing. (Run `git submodule update --init --recursive` to check out the repository)
|
||||
- `nasm` for assembling x86
|
||||
- Third party libraries (`nlohmann/json`, `minilzo`, and `linenoise`) are provided in the `third-party` folder
|
||||
|
||||
## Getting Started - Linux (Ubuntu)
|
||||
|
||||
@ -84,190 +83,99 @@ You may also wish to view the files that pertain to each CMake target, rather th
|
||||
|
||||
|
||||
|
||||
TODO
|
||||
|
||||
- more steps to follow as we actually figure it out!
|
||||
- running tests
|
||||
- etc
|
||||
|
||||
## Project Description
|
||||
|
||||
This project is to port Jak 1 (NTSC, "black label" version) to PC. The strategy is to:
|
||||
|
||||
- recompile for x86 to get much better performance than emulation
|
||||
- create human-reabable GOAL source code that can be modified
|
||||
- create a GOAL compiler for x86-64 which supports live patching of code like the original
|
||||
- attempt to match the original game as close as possible (for no reason other than it's fun)
|
||||
- unpack assets in a format that can be modified
|
||||
|
||||
There are 6 components to this project
|
||||
|
||||
- GOAL decompiler. The result will be manually cleaned up for running on a PC.
|
||||
- GOAL compiler for x86-64.
|
||||
- Game source code, made from cleaning up the result of the GOAL decompiler.
|
||||
- GOAL runtime. This will replace the parts of the game written in C++
|
||||
- Asset extraction tool to extract the models/textures/large data from the game
|
||||
- Asset packing tool.
|
||||
|
||||
The process to build the port will be
|
||||
|
||||
- Build data extraction tool, GOAL compiler, and GOAL runtime library (all written in C++)
|
||||
- Run the GOAL compiler on the game source code to build the game engine
|
||||
- Run asset extraction on the game disc to get level data, textures, geometry data, music...
|
||||
- Run the asset packing tool to combine the unpacked assets with the compiled game engine to create the game!
|
||||
|
||||
Some statistics:
|
||||
|
||||
- Estimated ~500k lines of GOAL code
|
||||
- 10410 functions
|
||||
- 5451 functions with no control flow (no branching, loops, if/else, short-circuiting boolean operators, gotos, etc)
|
||||
|
||||
The rough timeline is to finish sometime in 2022. If it looks like this is impossible, the project will be abandoned. But I have already spent about 4 months preparing to start this and seems doable. I also have some background in compilers, and familiarity with PS2 (worked on DobieStation PS2 emulator) / MIPS in general (wrote a PS1 emulator). I think the trick will be making good automated tools - the approach taken for SM64 and other N64 decompilations is way too labor-intensive to work.
|
||||
|
||||
## Project Layout
|
||||
There are four main components to the project.
|
||||
|
||||
- `goalc` is the GOAL compiler
|
||||
- `decompiler` is the decompiler
|
||||
- `decompiler_out` is the decompiler output
|
||||
- `data` will contain big assets and the output of the GOAL compiler (not checked in to git)
|
||||
- `out` will contain the finished game (not checked into git)
|
||||
- `resources` will contain data which is checked into git
|
||||
- `game` will contain the game source code (C/C++)
|
||||
- `goal_src` will contain GOAL source code for the game
|
||||
- `common` will contain all data/type shared between different applications.
|
||||
- `doc` will contain documentation
|
||||
- `iso_data` is where the files from the DVD go
|
||||
- `third-party` will contain code we didn't write. Google Test is a git submodule in this folder.
|
||||
- `tests` will contain all tests
|
||||
- `asset_tool` will contain the asset packer/unpacker
|
||||
The first is `goalc`, which is a GOAL compiler for x86-64. Our implementation of GOAL is called OpenGOAL. All of the compiler source code is in `goalc`. To run the compiler on Linux, there is a script `gc.sh`. The compiler is controlled through a prompt which can be used to enter commands to compile, connect to a running GOAL program for interaction, run the OpenGOAL debugger, or, if you are connected to a running GOAL program, can be used as a REPL to run code interactively. In addition to compiling code files, the compiler has features to pack and build data files.
|
||||
|
||||
## Design
|
||||
The second component to the project is the decompiler. You must have a copy of the PS2 game and place all files from the DVD into the `iso_data` folder. Then run `decomp.sh` to run the decompiler. The decompile will extract assets to the `assets` folder. These assets will be used by the compiler when building the port. The decompiler will output code and other data intended to be inspected by humans in the `decompiler_out` folder. Stuff in this folder will not be used by the compiler.
|
||||
|
||||
(if anybody has better ideas, feel free to suggest improvements! This is just a rough plan for now)
|
||||
The third is the game source code, written in OpenGOAL. This is located in `goal_src`. All GOAL and GOOS code should be in this folder. Right now most of this is placeholders, but you can take a look at `kernel/gcommon.gc` or `goal-lib.gc` to see some in-progress source code.
|
||||
|
||||
- All C++ code should build from the top-level `cmake`.
|
||||
- All C++ applications (GOAL compiler, asset extractor, asset packer, runtime, test) should have a script in the top level which launches them.
|
||||
- All file paths should be relative to the `jak` folder.
|
||||
- The planned workflow for building a game:
|
||||
- `git submodule update --init --recursive` : check out gtest
|
||||
- `mkdir build; cd build` : create build folder for C++
|
||||
- `cmake ..; make -j` : build C++ code
|
||||
- `cd ..`
|
||||
- `./test.sh` : run gtests
|
||||
- `./asset_extractor.sh ./iso_data` : extract assets from game
|
||||
- `./build_engine.sh` : run GOAL compiler to build all game code
|
||||
- `./build_game.sh` : run the asset packer to build the game
|
||||
- `./run_game.sh` : run the game
|
||||
- Workflow for development:
|
||||
- `./gc.sh` : run the compiler in interactive mode
|
||||
- `./gs.sh` : run a goos interpreter in interactive mode
|
||||
- `./decomp.sh : run the decompiler
|
||||
The final component is the "runtime", located in `game`. This is the part of the game that's written in C++. In the port, that includes:
|
||||
- The "C Kernel", which contains the GOAL linker and some low-level GOAL language features. GOAL has a completely custom dynamically linked object file format so in order to load the first GOAL code, you need a linker written in C++. Some low-level functions for memory allocation, communicating with the I/O Processor, symbol table, strings, and the type system are also implemented in C, as these are required for the linker. It also listens for incoming messages from the compiler and passes them to the running game. This also initializes the game, by initializing the PS2 hardware, allocating the GOAL heaps, loading the GOAL kernel off of the DVD, and executing the kernel dispatcher function. This is in the `game/kernel` folder. This should be as close as possible to the game, and all differences should be noted with a comment.
|
||||
|
||||
## Coding Guidelines
|
||||
- Implementation of Sony's standard library. GOAL code can call C library functions, and Naughty Dog used some Sony library functions to access files, memory cards, controllers, and communicate with the separate I/O Processor. The library functions are in `game/sce`. Implementations of library features specific to the PC port are located in `game/system`.
|
||||
|
||||
- Avoid warnings
|
||||
- Use asserts over throwing exceptions in game code (throwing exceptions from C code called by GOAL code is sketchy)
|
||||
- The I/O Processor driver, Overlord. The PS2 had a separate CPU called the I/O Processor (IOP) that was directly connected to the DVD drive hardware and the sound hardware. Naughty Dog created a custom driver for the IOP that handled streaming data off of the DVD. It is much more complicated than I first expected. It's located in `game/overlord`. Like the C kernel, we try to keep this as close as possible to the actual game.
|
||||
|
||||
## TODOs
|
||||
- Sound Code. Naughty Dog used a third party library for sound. We have not started on this yet.
|
||||
|
||||
- Clean up header guard names (or just use `#pragma once`?)
|
||||
- Investigate a better config format
|
||||
- The current JSON library seems to have issues with comments, which I really like
|
||||
- Clean up use of namespaces
|
||||
- Clean up the print message when `gk` starts.
|
||||
- Finish commenting runtime stuff
|
||||
- Runtime document
|
||||
- GOOS document
|
||||
- Listener protocol document
|
||||
- Gtest setup for checking decompiler results against hand-decompiled stuff
|
||||
- Clean up decompiler print spam, finish up the CFG stuff
|
||||
- Decompiler document
|
||||
- PC specific graphics stuff. We have not started on this yet.
|
||||
|
||||
### GOAL Decompiler
|
||||
## Directory Layout
|
||||
|
||||
The decompiler is in progress, at
|
||||
https://github.com/water111/jak-disassembler
|
||||
- `.github`: GitHub actions CI setup
|
||||
- `.vs`: Visual Studio project configurations
|
||||
- `assets`: extracted assets (textures, translated game text) generated by the decompiler. Not included in the repository. To be used when building the PC port.
|
||||
- `build`: C++ CMake build folder
|
||||
- `common`: common C++ code shared between the compiler, decompiler, and game.
|
||||
- `cross_os_debug`: platform-independent library for implementing the OpenGOAL debugger. Linux-only currently
|
||||
- `cross_sockets`: platform-indpendent library for sockets. Used to connect the compiler to a running game. Linux and Windows.
|
||||
- `goos`: the compiler-time macro language and parser for OpenGOAL.
|
||||
- `type_system`: the OpenGOAL type system
|
||||
- `util`: Random utility functions for accessing files, timers, etc.
|
||||
- `decompiler`: Source code for the decompiler
|
||||
- `config`: JSON config files for the decompiler and type definition file.
|
||||
- `data`: utilities to extract assets from the game
|
||||
- `Disasm`: MIPS disassembler
|
||||
- `Function`: Tools for analyzing GOAL functions
|
||||
- `gui`: an early prototype of a Python GUI for reading the output of the decompiler
|
||||
- `IR`: the "Intermediate Representation" for GOAL functions
|
||||
- `ObjectFile`: Utilities for processing the GOAL object file format.
|
||||
- `scripts`: Useful scripts for setting up the decompilation
|
||||
- `util`: random utilities
|
||||
- `decompiler_out`: output of the decompiler that's not automaically used by the compiler. This is for humans to read and use. Not included in the repository.
|
||||
- `doc`: more documentation!
|
||||
- `game`: the source code for the game executable
|
||||
- `common`: shared stuff between the `kernel` and `overlord`
|
||||
- `kernel`: the part of the GOAL kernel written in C. The entry point for the game is in `kboot.cpp`.
|
||||
- `overlord`: the I/O processor driver used to get data off of the DVD
|
||||
- `sce`: the Sony library implementation
|
||||
- `system`: PC-port specific stuff
|
||||
- `goal_src`: The GOAL code for the game. It's mostly empty now.
|
||||
- `build`: info related to the GOAL build system.
|
||||
- `engine`: the game engine
|
||||
- `kernel`: The GOAL kernel
|
||||
- `levels`: Level specific code.
|
||||
- `goalc`: The OpenGOAL compiler
|
||||
- `compiler`: The implementation of the OpenGOAL language
|
||||
- `data_compiler`: Tools for packing data
|
||||
- `debugger`: The OpenGOAL debugger (part of the compiler)
|
||||
- `emitter`: x86-64 emitter and object file generator
|
||||
- `listener`: The OpenGOAL listener, which connects the compiler to a running GOAL program for the interactive REPL
|
||||
- `regalloc`: Register alocator
|
||||
- `iso_data`:
|
||||
- `out`: Outputs from the build process. Only the `iso` subfolder should contain assets used by the game.
|
||||
- `iso`: Final outputs that are used by the game.
|
||||
- `obj`: Object files generated by the compiler.
|
||||
- `resources`: To be removed. Contains fake versions of some files required to get things booting.
|
||||
- `scripts`: Utility scripts.
|
||||
- `test`: Unit tests (run on GitHub Actions)
|
||||
- `third-party`: Third party libraries
|
||||
- CMake Code Coverage. For code coverage statistics on GitHub builds
|
||||
- `fmt`. String formatting library
|
||||
- `googletest`: Test framework
|
||||
- `inja`: templating library used for generating test code for compiler tests
|
||||
- `minilzo`: decompression code for Jak 2 and later DGOs
|
||||
- `mman`: Windows library used to emulate `mmap` on Linux
|
||||
- `run-clang-format`: Utility to check and enforce code formatting
|
||||
- `run-clang-tidy`
|
||||
- `spdlog`: Logging library
|
||||
- `zydis`: x86-64 disassembler used in the OpenGOAL debugger
|
||||
- `json`: A JSON library
|
||||
- `linenoise`: Used for the REPL input. Support history and useful editing shortcuts.
|
||||
- `svpng`: Save a PNG file
|
||||
|
||||
Here is the plan for writing the decompiler:
|
||||
|
||||
- [x] Decode the CGO/DGO format.
|
||||
- [x] Decode the linking data format.
|
||||
- [x] Identify all code and disassemble
|
||||
- [x] Recover references
|
||||
- [x] Split code into functions, and build a graph of basic blocks
|
||||
- [ ] Create a control flow graph for each function (currently succeeds for 9857/10410 functions)
|
||||
- [ ] Extract type/method information from debug data
|
||||
- [ ] Convert instructions to an intermediate representation (IR) and eliminate GOAL/MIPS idioms
|
||||
- [ ] Regsiter liveness analysis
|
||||
- [ ] Type propagation
|
||||
- [ ] Variable map and scoping
|
||||
- [ ] S-expression construction (expression stack)
|
||||
## More Documentation
|
||||
Check out these files for more documentation. Some of it is still in progress
|
||||
- `doc/goal_dbg_doc.md`: OpenGOAL debugger
|
||||
- `doc/goal_doc.md`: OpenGOAL language
|
||||
- `doc/reader.md`: OpenGOAL "reader" documentation (OpenGOAL syntax)
|
||||
- `doc/type_system.md`: OpenGOAL type system documentation
|
||||
- `doc/porting_to_x86.md`: Summary of changes we're making to port to x86-64
|
||||
- `doc/goos.md`: GOOS macro language
|
||||
|
||||
### GOAL Runtime
|
||||
|
||||
The "runtime" will be a replacement for all of the C/C++ code of the original game. There is C/C++ code that runs on the main processor (EE) and the separate I/O processor (IOP).
|
||||
|
||||
- The "C Kernel", which runs on the EE and contains
|
||||
- [ ] File I/O (for debugging, not used by retail game)
|
||||
- [x] Initialization to boostrap the GOAL Kernel and start the game engine
|
||||
- [x] Connection to compiler for debugging/live code patching
|
||||
- [x] Interface to OVERLORD (see next section) for DGO loading
|
||||
- [x] GOAL Linker
|
||||
- [ ] PS2-specific hardware initialization as required by Sony libraries
|
||||
- [x] GOAL "kheap" allocator
|
||||
- [ ] Memory card interface
|
||||
- [x] GOAL printf (called `format`) implementation
|
||||
- [x] GOAL hash/symbol table implementation
|
||||
- [x] Implementation of some built-in GOAL methods/functions
|
||||
- The "OVERLORD" IOP driver, which ran on the PS2's separate I/O Processor for loading things off the DVD and doing sound things
|
||||
- [x] DGO loader
|
||||
- [x] File system for loading from DVD or "fakeiso" mode for loading from a folder
|
||||
- [x] "ISOThread" queue system for prioritizing reads
|
||||
- [ ] Sound stuff
|
||||
- [ ] Streaming animation stuff
|
||||
- [ ] Ramdisk stuff (implemented but totally untested)
|
||||
- The "989_snd" sound driver (no progress has been made here, the rough plan is to do a high level emulation of the sound system)
|
||||
- Sony libraries
|
||||
- [x] SIF (interface between EE/IOP for sending data, receiving data, and making remote procedure calls)
|
||||
- [x] IOP Kernel (single-processor non-preemptive multitasking)
|
||||
- [x] stubs for stuff that doesn't really matter
|
||||
|
||||
The "Sony libraries" are a simple wrapper around my `system` library, which implements the threading/communication stuff.
|
||||
|
||||
Likely there will be sound/graphics code in here at some point, but this part is not fully planned yet.
|
||||
|
||||
### GOAL Compiler
|
||||
|
||||
The GOAL compiler will target x86-64. At first just Linux. There is a macro language called GOOS which is basically just Scheme but with a few bonus features.
|
||||
|
||||
The compiler will reuse a significant amount of code from my existing LISP compiler for x86-64. I have a very bad WIP combination which is capable of building a modified `gkernel.gc` for x86 as a proof of concept. It can create and run functions in threads.
|
||||
|
||||
An important part of the compiler is the test suite. Without tests the compiler will be full of bugs. So every feature should have a good set of tests.
|
||||
|
||||
### Asset Extraction Tool
|
||||
|
||||
Not started yet. The simplest version of this tool is just to use the decompiler logic to turn the level/art-group/texture/TXT files into GOAL source code, and copy all STR/sound/visibility files, as these don't need to be modified.
|
||||
|
||||
Eventually this should export to a more useful format.
|
||||
|
||||
File formats:
|
||||
|
||||
- [ ] Art group (a GOAL object format)
|
||||
- There may be more formats related to art groups.
|
||||
- [ ] Texture page (a GOAL object format)
|
||||
- [ ] Texture page directory (a GOAL object format)
|
||||
- [ ] Level (`vis-bt`) (a GOAL object format)
|
||||
- [ ] `TEXT/*.TXT` (text, a GOAL object format)
|
||||
- [ ] `MUS` (sequenced music)
|
||||
- [ ] `SBK` (sound bank)
|
||||
- [ ] `STR` (streaming animation)
|
||||
- [ ] `VAG` (ADPCM audio)
|
||||
- [ ] `VIS` (visibility data bitstream)
|
||||
- [ ] Loading screen image
|
||||
- [ ] save game icon (I do not care about this)
|
||||
|
||||
### Asset Packing Tool
|
||||
|
||||
Packs together all assets/compiled code/runtime into a format that can be played. The simplest version to go with the simplest extraction tool will just pass the level/art-group/texture/TXT files to the compiler, and copy STR/sound/visbility files into the fakeiso. Then pack in CGOs/DGOs.
|
||||
|
||||
It's important that the asset extraction/packing can be automated so we can avoid distributing the assets, which are large and probably not supposed to be distributed.
|
||||
## Tests
|
||||
3
assets/.gitignore
vendored
3
assets/.gitignore
vendored
@ -1,3 +1,2 @@
|
||||
*
|
||||
!.gitignore
|
||||
!textures/*
|
||||
!.gitignore
|
||||
2
assets/textures/.gitignore
vendored
2
assets/textures/.gitignore
vendored
@ -1,2 +0,0 @@
|
||||
*
|
||||
!.gitignore
|
||||
@ -1,3 +1,8 @@
|
||||
/*!
|
||||
* @file xsocket.cpp
|
||||
* Cross platform socket library used for the listener.
|
||||
*/
|
||||
|
||||
#ifdef __linux
|
||||
#include <sys/socket.h>
|
||||
#include <netinet/tcp.h>
|
||||
|
||||
@ -1,5 +1,10 @@
|
||||
#pragma once
|
||||
|
||||
/*!
|
||||
* @file xsocket.h
|
||||
* Cross platform socket library used for the listener.
|
||||
*/
|
||||
|
||||
#ifdef __linux
|
||||
#include <sys/socket.h>
|
||||
#include <netinet/tcp.h>
|
||||
|
||||
@ -1,7 +1,6 @@
|
||||
/*!
|
||||
* @file Interpreter.cpp
|
||||
* The GOOS Interpreter and implementation of special forms.
|
||||
* Additional "built-in" forms are implemented in InterpreterEval
|
||||
* The GOOS Interpreter and implementation of special and "built-in forms"
|
||||
*/
|
||||
|
||||
#include <utility>
|
||||
@ -101,10 +100,10 @@ void Interpreter::disable_printfs() {
|
||||
}
|
||||
|
||||
/*!
|
||||
* Load the goos library, by interpreting (load-file "goal/gs/goos-lib.gs") in the global env.
|
||||
* Load the goos library, by interpreting (load-file "goal_src/goos-lib.gs") in the global env.
|
||||
*/
|
||||
void Interpreter::load_goos_library() {
|
||||
auto cmd = "(load-file \"goalc/gs/goos-lib.gs\")";
|
||||
auto cmd = "(load-file \"goal_src/goos-lib.gs\")";
|
||||
eval_with_rewind(reader.read_from_string(cmd), global_environment.as_env());
|
||||
}
|
||||
|
||||
|
||||
@ -2,12 +2,9 @@
|
||||
|
||||
/*!
|
||||
* @file Interpreter.h
|
||||
* The GOOS Interpreter
|
||||
* The GOOS Interpreter and implementation of special and "built-in forms"
|
||||
*/
|
||||
|
||||
#ifndef JAK1_INTERPRETER_H
|
||||
#define JAK1_INTERPRETER_H
|
||||
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include "Object.h"
|
||||
@ -237,5 +234,3 @@ class Interpreter {
|
||||
std::unordered_map<std::string, ObjectType> string_to_type;
|
||||
};
|
||||
} // namespace goos
|
||||
|
||||
#endif // JAK1_INTERPRETER_H
|
||||
|
||||
@ -1,3 +1,43 @@
|
||||
/*!
|
||||
* @file Object.cpp
|
||||
* An "Object" represents a scheme object.
|
||||
* There are different types of objects, as represented by ObjectType.
|
||||
* An "Object" is an efficient wrapper around any of these types.
|
||||
* Some types are "heap allocated", and have reference semantics, and others are
|
||||
* "fixed" and have value semantics. Heap allocated objects implement reference counting with
|
||||
* std::shared_ptr.
|
||||
*
|
||||
* To create a new Object for a heap allocated type, use the make_new static method of the type of
|
||||
* object you want to make. This will return a correctly setup Object. For fixed objects, use
|
||||
* Object::make_<type>
|
||||
*
|
||||
* To convert an Object into a more specific object, use the as_<type> method of Object.
|
||||
* It will throw an exception is you get the type wrong.
|
||||
*
|
||||
* These are all the types:
|
||||
*
|
||||
* EMPTY_LIST - a special heap allocated object. There is only one EMPTY_LIST allocated, and
|
||||
* EmptyListObject::make_new() will always return an Object which references that one.
|
||||
*
|
||||
* INTEGER - a fixed type. Use Object::make_integer() to create one. Internally uses int64_t
|
||||
* FLOAT - a fixed type. Use Object::make_float() to create one. Internally uses double
|
||||
* CHAR - a fixed type. Use Object::make_char() to create one. Internally uses char
|
||||
*
|
||||
* SYMBOL - a special heap allocated object. SymbolObject::make_new requires a SymbolTable to
|
||||
* store the newly allocated symbol in, and will return an existing symbol if there already is one.
|
||||
*
|
||||
* STRING - a heap allocated object. Create with StringObject::make_new. Uses std::string internally
|
||||
*
|
||||
* PAIR - a heap allocated object containing two Objects.
|
||||
*
|
||||
* ARRAY - a heap allocated object containing a std::vector<Object>
|
||||
*
|
||||
* LAMBDA - a heap allocated object representing a GOOS lambda
|
||||
* MACRO - a heap allocated object representing a GOOS macro
|
||||
* ENVIRONMENT - a heap allocated object representing a GOOS environment
|
||||
*
|
||||
*/
|
||||
|
||||
#include "Object.h"
|
||||
#include "common/util/FileUtil.h"
|
||||
|
||||
|
||||
@ -40,9 +40,6 @@
|
||||
*
|
||||
*/
|
||||
|
||||
#ifndef JAK1_OBJECT_H
|
||||
#define JAK1_OBJECT_H
|
||||
|
||||
#include <string>
|
||||
#include <cassert>
|
||||
#include <memory>
|
||||
@ -640,5 +637,3 @@ class ArrayObject : public HeapObject {
|
||||
Object build_list(const std::vector<Object>& objects);
|
||||
|
||||
} // namespace goos
|
||||
|
||||
#endif // JAK1_OBJECT_H
|
||||
|
||||
@ -1,3 +1,9 @@
|
||||
/*!
|
||||
* @file PrettyPrinter.cpp
|
||||
* A Pretty Printer for GOOS.
|
||||
* It is not very good, but significantly better than putting everything on one line
|
||||
*/
|
||||
|
||||
#include <cassert>
|
||||
#include <stdexcept>
|
||||
#include <utility>
|
||||
|
||||
@ -1,9 +1,11 @@
|
||||
#pragma once
|
||||
|
||||
/*!
|
||||
* @file PrettyPrinter.h
|
||||
* A Pretty Printer for GOOS.
|
||||
* It is not very good, but significantly better than putting everything on one line
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include "common/goos/Object.h"
|
||||
|
||||
@ -11,9 +11,6 @@
|
||||
* launching the compiler or the compiler test.
|
||||
*/
|
||||
|
||||
#ifndef JAK1_READER_H
|
||||
#define JAK1_READER_H
|
||||
|
||||
#include <memory>
|
||||
#include <cassert>
|
||||
#include <utility>
|
||||
@ -104,5 +101,3 @@ class Reader {
|
||||
|
||||
std::string get_readable_string(const char* in);
|
||||
} // namespace goos
|
||||
|
||||
#endif // JAK1_READER_H
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
/*!
|
||||
* @file TextDB.h
|
||||
* @file TextDB.cpp
|
||||
* The Text Database for storing source code text.
|
||||
* This allows us to ask for things like "where did this form come from?"
|
||||
* and be able to print the file name and offset into the file, as well as the line.
|
||||
|
||||
@ -13,9 +13,6 @@
|
||||
* (+ 1 (+ a b)) ; compute the sum
|
||||
*/
|
||||
|
||||
#ifndef JAK1_TEXTDB_H
|
||||
#define JAK1_TEXTDB_H
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <stdexcept>
|
||||
@ -98,5 +95,3 @@ class TextDb {
|
||||
std::unordered_map<std::shared_ptr<goos::HeapObject>, TextRef> map;
|
||||
};
|
||||
} // namespace goos
|
||||
|
||||
#endif // JAK1_TEXTDB_H
|
||||
|
||||
@ -1,8 +1,12 @@
|
||||
/*!
|
||||
* @file Type.cpp
|
||||
* Representation of a GOAL type in the type system.
|
||||
*/
|
||||
|
||||
#include <stdexcept>
|
||||
#include <cassert>
|
||||
#include <third-party/fmt/core.h>
|
||||
#include "Type.h"
|
||||
#include "type_util.h"
|
||||
|
||||
namespace {
|
||||
std::string reg_kind_to_string(RegKind kind) {
|
||||
|
||||
@ -1,5 +1,10 @@
|
||||
#pragma once
|
||||
|
||||
/*!
|
||||
* @file Type.h
|
||||
* Representation of a GOAL type in the type system.
|
||||
*/
|
||||
|
||||
#ifndef JAK_TYPE_H
|
||||
#define JAK_TYPE_H
|
||||
|
||||
|
||||
@ -1,3 +1,8 @@
|
||||
/*!
|
||||
* @file TypeSpec.cpp
|
||||
* A GOAL TypeSpec is a reference to a type or compound type.
|
||||
*/
|
||||
|
||||
#include "TypeSpec.h"
|
||||
#include "Type.h"
|
||||
|
||||
|
||||
@ -2,6 +2,7 @@
|
||||
|
||||
/*!
|
||||
* @file TypeSpec.h
|
||||
* A GOAL TypeSpec is a reference to a type or compound type.
|
||||
*/
|
||||
|
||||
#ifndef JAK_TYPESPEC_H
|
||||
|
||||
@ -1,9 +1,15 @@
|
||||
#include <cassert>
|
||||
#include <third-party/fmt/core.h>
|
||||
#include "TypeSystem.h"
|
||||
#include "type_util.h"
|
||||
/*!
|
||||
* @file TypeSystem.cpp
|
||||
* The GOAL Type System.
|
||||
* Stores types, symbol types, methods, etc, and does typechecking, lowest-common-ancestor, field
|
||||
* access types, and reverse type lookups.
|
||||
*/
|
||||
|
||||
#include <cassert>
|
||||
#include <stdexcept>
|
||||
#include <third-party/fmt/core.h>
|
||||
#include "TypeSystem.h"
|
||||
#include "common/util/math_util.h"
|
||||
|
||||
TypeSystem::TypeSystem() {
|
||||
// the "none" and "_type_" types are included by default.
|
||||
|
||||
@ -1,7 +1,11 @@
|
||||
#pragma once
|
||||
|
||||
#ifndef JAK_TYPESYSTEM_H
|
||||
#define JAK_TYPESYSTEM_H
|
||||
/*!
|
||||
* @file TypeSystem.h
|
||||
* The GOAL Type System.
|
||||
* Stores types, symbol types, methods, etc, and does typechecking, lowest-common-ancestor, field
|
||||
* access types, and reverse type lookups.
|
||||
*/
|
||||
|
||||
#include <unordered_map>
|
||||
#include <unordered_set>
|
||||
@ -165,5 +169,3 @@ class TypeSystem {
|
||||
};
|
||||
|
||||
TypeSpec coerce_to_reg_type(const TypeSpec& in);
|
||||
|
||||
#endif // JAK_TYPESYSTEM_H
|
||||
|
||||
@ -1,3 +1,9 @@
|
||||
/*!
|
||||
* @file deftype.cpp
|
||||
* Parser for the GOAL "deftype" form.
|
||||
* This is used both in the compiler and in the decompiler for the type definition file.
|
||||
*/
|
||||
|
||||
#include "deftype.h"
|
||||
#include "third-party/fmt/core.h"
|
||||
|
||||
|
||||
@ -1,5 +1,11 @@
|
||||
#pragma once
|
||||
|
||||
/*!
|
||||
* @file deftype.h
|
||||
* Parser for the GOAL "deftype" form.
|
||||
* This is used both in the compiler and in the decompiler for the type definition file.
|
||||
*/
|
||||
|
||||
#include "TypeSystem.h"
|
||||
#include "common/goos/Object.h"
|
||||
|
||||
|
||||
@ -1,7 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#ifndef JAK_V2_BINARYREADER_H
|
||||
#define JAK_V2_BINARYREADER_H
|
||||
/*!
|
||||
* @file BinaryReader.h
|
||||
* Read raw data like a stream.
|
||||
*/
|
||||
|
||||
#include <cstdint>
|
||||
#include <cassert>
|
||||
@ -38,5 +40,3 @@ class BinaryReader {
|
||||
uint32_t size;
|
||||
uint32_t seek = 0;
|
||||
};
|
||||
|
||||
#endif // JAK_V2_BINARYREADER_H
|
||||
|
||||
@ -1,7 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#ifndef JAK_BINARYWRITER_H
|
||||
#define JAK_BINARYWRITER_H
|
||||
/*!
|
||||
* @file BinaryWriter.h
|
||||
* Write raw data like a stream.
|
||||
*/
|
||||
|
||||
#include <cassert>
|
||||
#include <stdexcept>
|
||||
@ -72,5 +74,3 @@ class BinaryWriter {
|
||||
private:
|
||||
std::vector<uint8_t> data;
|
||||
};
|
||||
|
||||
#endif // JAK_BINARYWRITER_H
|
||||
|
||||
@ -1,3 +1,8 @@
|
||||
/*!
|
||||
* @file DgoWriter.cpp
|
||||
* Create a DGO from existing files.
|
||||
*/
|
||||
|
||||
#include "BinaryWriter.h"
|
||||
#include "FileUtil.h"
|
||||
#include "DgoWriter.h"
|
||||
@ -9,7 +14,8 @@ void build_dgo(const DgoDescription& description) {
|
||||
writer.add_cstr_len(description.dgo_name.c_str(), 60);
|
||||
|
||||
for (auto& obj : description.entries) {
|
||||
auto obj_data = file_util::read_binary_file(file_util::get_file_path({"data", obj.file_name}));
|
||||
auto obj_data =
|
||||
file_util::read_binary_file(file_util::get_file_path({"out", "obj", obj.file_name}));
|
||||
// size
|
||||
writer.add<uint32_t>(obj_data.size());
|
||||
// name
|
||||
@ -22,7 +28,6 @@ void build_dgo(const DgoDescription& description) {
|
||||
}
|
||||
}
|
||||
|
||||
printf("DGO: %15s %.3f MB\n", description.dgo_name.c_str(),
|
||||
(float)(writer.get_size()) / (1 << 20));
|
||||
writer.write_to_file(file_util::get_file_path({"out", description.dgo_name}));
|
||||
file_util::create_dir_if_needed(file_util::get_file_path({"out", "iso"}));
|
||||
writer.write_to_file(file_util::get_file_path({"out", "iso", description.dgo_name}));
|
||||
}
|
||||
|
||||
@ -1,7 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#ifndef JAK_DGOWRITER_H
|
||||
#define JAK_DGOWRITER_H
|
||||
/*!
|
||||
* @file DgoWriter.h
|
||||
* Create a DGO from existing files.
|
||||
*/
|
||||
|
||||
#include <vector>
|
||||
|
||||
@ -15,5 +17,3 @@ struct DgoDescription {
|
||||
};
|
||||
|
||||
void build_dgo(const DgoDescription& description);
|
||||
|
||||
#endif // JAK_DGOWRITER_H
|
||||
|
||||
@ -1,6 +1,12 @@
|
||||
/*!
|
||||
* @file FileUtil.cpp
|
||||
* Utility functions for reading and writing files.
|
||||
*/
|
||||
|
||||
#include "FileUtil.h"
|
||||
#include <iostream>
|
||||
#include <stdio.h> /* defines FILENAME_MAX */
|
||||
#include <filesystem>
|
||||
#include <cstdio> /* defines FILENAME_MAX */
|
||||
#include <fstream>
|
||||
#include <sstream>
|
||||
#include <cassert>
|
||||
@ -55,6 +61,14 @@ std::string file_util::get_file_path(const std::vector<std::string>& input) {
|
||||
return filePath;
|
||||
}
|
||||
|
||||
bool file_util::create_dir_if_needed(const std::string& path) {
|
||||
if (!std::filesystem::is_directory(path)) {
|
||||
std::filesystem::create_directories(path);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void file_util::write_binary_file(const std::string& name, void* data, size_t size) {
|
||||
FILE* fp = fopen(name.c_str(), "wb");
|
||||
if (!fp) {
|
||||
|
||||
@ -1,10 +1,17 @@
|
||||
#pragma once
|
||||
|
||||
/*!
|
||||
* @file FileUtil.h
|
||||
* Utility functions for reading and writing files.
|
||||
*/
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace file_util {
|
||||
std::string get_project_path();
|
||||
std::string get_file_path(const std::vector<std::string>& input);
|
||||
bool create_dir_if_needed(const std::string& path);
|
||||
void write_binary_file(const std::string& name, void* data, size_t size);
|
||||
void write_rgba_png(const std::string& name, void* data, int w, int h);
|
||||
void write_text_file(const std::string& file_name, const std::string& text);
|
||||
|
||||
@ -1,14 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#ifndef JAK_TYPE_UTIL_H
|
||||
#define JAK_TYPE_UTIL_H
|
||||
|
||||
template <typename T>
|
||||
T align(T current, T alignment, T offset = 0) {
|
||||
while ((current % alignment) != 0) {
|
||||
current++;
|
||||
}
|
||||
return current + offset;
|
||||
}
|
||||
|
||||
#endif // JAK_TYPE_UTIL_H
|
||||
}
|
||||
2
data/.gitignore
vendored
2
data/.gitignore
vendored
@ -1,2 +0,0 @@
|
||||
*
|
||||
!.gitignore
|
||||
@ -726,8 +726,11 @@ TPageResultStats process_tpage(ObjectFileData& data) {
|
||||
}
|
||||
|
||||
// write texture to a PNG.
|
||||
file_util::create_dir_if_needed(
|
||||
file_util::get_file_path({"assets", "textures", texture_page.name}));
|
||||
file_util::write_rgba_png(
|
||||
fmt::format(file_util::get_file_path({"assets", "textures", "{}-{}-{}-{}.png"}),
|
||||
fmt::format(file_util::get_file_path(
|
||||
{"assets", "textures", texture_page.name, "{}-{}-{}-{}.png"}),
|
||||
data.name_in_dgo, tex.name, tex.w, tex.h),
|
||||
out.data(), tex.w, tex.h);
|
||||
stats.successful_textures++;
|
||||
@ -769,8 +772,11 @@ TPageResultStats process_tpage(ObjectFileData& data) {
|
||||
}
|
||||
|
||||
// write texture to a PNG.
|
||||
file_util::create_dir_if_needed(
|
||||
file_util::get_file_path({"assets", "textures", texture_page.name}));
|
||||
file_util::write_rgba_png(
|
||||
fmt::format(file_util::get_file_path({"assets", "textures", "{}-{}-{}-{}.png"}),
|
||||
fmt::format(file_util::get_file_path(
|
||||
{"assets", "textures", texture_page.name, "{}-{}-{}-{}.png"}),
|
||||
data.name_in_dgo, tex.name, tex.w, tex.h),
|
||||
out.data(), tex.w, tex.h);
|
||||
stats.successful_textures++;
|
||||
@ -794,8 +800,11 @@ TPageResultStats process_tpage(ObjectFileData& data) {
|
||||
}
|
||||
|
||||
// write texture to a PNG.
|
||||
file_util::create_dir_if_needed(
|
||||
file_util::get_file_path({"assets", "textures", texture_page.name}));
|
||||
file_util::write_rgba_png(
|
||||
fmt::format(file_util::get_file_path({"assets", "textures", "{}-{}-{}-{}.png"}),
|
||||
fmt::format(file_util::get_file_path(
|
||||
{"assets", "textures", texture_page.name, "{}-{}-{}-{}.png"}),
|
||||
data.name_in_dgo, tex.name, tex.w, tex.h),
|
||||
out.data(), tex.w, tex.h);
|
||||
stats.successful_textures++;
|
||||
@ -835,8 +844,11 @@ TPageResultStats process_tpage(ObjectFileData& data) {
|
||||
}
|
||||
|
||||
// write texture to a PNG.
|
||||
file_util::create_dir_if_needed(
|
||||
file_util::get_file_path({"assets", "textures", texture_page.name}));
|
||||
file_util::write_rgba_png(
|
||||
fmt::format(file_util::get_file_path({"assets", "textures", "{}-{}-{}-{}.png"}),
|
||||
fmt::format(file_util::get_file_path(
|
||||
{"assets", "textures", texture_page.name, "{}-{}-{}-{}.png"}),
|
||||
data.name_in_dgo, tex.name, tex.w, tex.h),
|
||||
out.data(), tex.w, tex.h);
|
||||
stats.successful_textures++;
|
||||
@ -876,8 +888,11 @@ TPageResultStats process_tpage(ObjectFileData& data) {
|
||||
}
|
||||
|
||||
// write texture to a PNG.
|
||||
file_util::create_dir_if_needed(
|
||||
file_util::get_file_path({"assets", "textures", texture_page.name}));
|
||||
file_util::write_rgba_png(
|
||||
fmt::format(file_util::get_file_path({"assets", "textures", "{}-{}-{}-{}.png"}),
|
||||
fmt::format(file_util::get_file_path(
|
||||
{"assets", "textures", texture_page.name, "{}-{}-{}-{}.png"}),
|
||||
data.name_in_dgo, tex.name, tex.w, tex.h),
|
||||
out.data(), tex.w, tex.h);
|
||||
stats.successful_textures++;
|
||||
|
||||
@ -1,77 +0,0 @@
|
||||
# Build System
|
||||
Game Build System. The build system should allow us to
|
||||
- automatically extract large asset files from the game so we don't have to keep them in this repository
|
||||
- support modifications (adding new files, modifying files, etc)
|
||||
- have checks that the resulting data matches the original if expected, assuming no modifications are applied
|
||||
|
||||
Intermediate files should go somewhere in `data` and final files should go in `out`.
|
||||
|
||||
## 1. C++ Build
|
||||
All C++ code should be built through `cmake`, including the compiler, decompiler, runtime, and any tools. It's expected to be built in the `build` folder.
|
||||
|
||||
## 2. Unpack Assets
|
||||
This step extracts data from asset files from the game. An asset file is any file that's not a CGO/DGO. All files on the DVD will be in `iso_data/`, and all unpacked data should go in `data/`
|
||||
|
||||
## 3. Unpack DGOs
|
||||
This step extracts data from CGO/DGO files and unpacks it. All unpacked data should go in `data/`
|
||||
|
||||
## 4. Pack Assets
|
||||
This step creates final asset files from data in `data/`. The final files should do in `out`.
|
||||
|
||||
## 5. Compile GOAL Code / Data
|
||||
This step creates all the data for the CGOs/DGOs. The compiler will have a big list of files which it will compile in order. Each `.gc` file contains GOAL code, which is compiled to a GOAL code object (`.o` file) stored in `data/obj`. Each `.gd` file is a "GOAL data description file" which contains a short script to launch the appropriate tool to produce the data object. The objects will also be stored in `data/obj`, but will have a `.go` extension.
|
||||
|
||||
## 6. Create CGO/DGOs
|
||||
This step takes the output from step 5, (stored in `data/obj`) and creates all the DGO/CGO files, which go in `out`. The list of what goes in the DGOs is in `goal_src/build/dgos.txt`.
|
||||
|
||||
## 7. Asset Check
|
||||
There might be some asset files which we expect to be unchanged between the version in `iso_data` and in `out`. At this point we should check if these are actually the same.
|
||||
|
||||
## 8. CGO/DGO check
|
||||
The data objects contained in CGO/DGO files should be the same between the original version and the ported version. We should go through each CGO/DGO file, look at each object, and if it's a data object, make sure it matches.
|
||||
|
||||
|
||||
# What can be worked on now?
|
||||
## C++ Build
|
||||
Getting more things working on Windows.
|
||||
|
||||
### Compiler
|
||||
The compiler right now only supports GOOS, the macro language. It may be useful to play around with this. There may be missing features or bugs in the language.
|
||||
|
||||
|
||||
## Unpack Assets
|
||||
I imagine that each type of asset file will have its own tool to unpack things. We'll need a master "unpacker" tool which runs the correct tool on each file.
|
||||
|
||||
This unpacker could take a config file, maybe JSON?, that says what to do for each file:
|
||||
```
|
||||
[
|
||||
{"filename":"MUS/SNOW.MUS", "unpack_tool":"music_unpacker", "destination":"data/music/snow/"... arguments to music_unpacker},
|
||||
...
|
||||
]
|
||||
```
|
||||
and then would run the correct tool on each file. For starters, we could make a simple unpacking tool named `copy` that just directly copies the asset file into the `data` folder.
|
||||
|
||||
We could also look into particular files and start writing unpacking tools.
|
||||
|
||||
## Unpack DGOs
|
||||
Again, there will be a main tool which runs through each DGO/CGO file, finds data objects, then runs an appropriate unpacking tools based on the type of data object (level, texture, art-group,...). I imagine there would a config file (json maybe?) that looks something like this:
|
||||
```
|
||||
{"dgo":"MAI.DGO", "objects":[{"tpage1234", "texture-unpacker", ...}, {"spider", "art-group-unpacker", ...}]}
|
||||
```
|
||||
|
||||
My plan for duplicate data objects:
|
||||
- When there are duplicates, only one will actually be unpacked
|
||||
- Ones that aren't unpacked will be checked if they are the same as the unpacked one to make sure we don't make a mistake
|
||||
- The list of which objects should be unpacked vs. checked, and what name the unpacking should get will be determined by the decompiler.
|
||||
- If there is an object not in the config file, or a missing object, then there is an error
|
||||
|
||||
The actual unpacking tools probably can't be developed until I package up the GOAL disassembler into a usable library, and we probably won't be able to understand the data format until later. We could write a very simple `copy` unpacker which just copies the object data directly into `data` and doesn't unpack it at all.
|
||||
|
||||
## Pack Assets
|
||||
Again, there will be a main tool which runs the appropriate packer for each file. We could write this main tool, and a `copy` packer which just copies the exact file in `data/` to `out/`.
|
||||
|
||||
## Asset Check
|
||||
Just a tool with a list of asset files to check, does a comparison and reports an error if there is a difference. We might expect some files to change slightly in the PC version, and these will be excluded from the check.
|
||||
|
||||
## CGO/DGO Check
|
||||
Iterates through objects in the original / rebuilt CGO/DGOs. Code objects are not checked, but data objects are checked. Should also check that the order of objects is the same, and all are present.
|
||||
212
doc/porting_to_x86.md
Normal file
212
doc/porting_to_x86.md
Normal file
@ -0,0 +1,212 @@
|
||||
# Porting to x86
|
||||
This document will keep track of stuff that needs to be ported or modified significantly for x86. Anything that uses PS2-specific hardware or relies on stuff in the C Kernel will need to be ported.
|
||||
|
||||
## Basic Info
|
||||
Most of the game is written in GOAL. All this source lives in `goal_src`.
|
||||
|
||||
The "runtime" is all the support code written in C++. It's located in `game/`. Sometimes the "runtime" + GOAL code together is called the "target".
|
||||
|
||||
Most of the code in "runtime" is reverse engineered code from the real game, with small tweaks to make it work on x86 and with OpenGOAL.
|
||||
|
||||
The code in `game/system` is **not** from the game and is an implementation of system functions that are implemented by Sony in the PS2 game. It's stuff like threading, I/O, etc.
|
||||
|
||||
The code in `game/sce` is my implementation of the Sony libraries. When possible, I tried to keep exactly the same names/functions as the real Sony libraries. This way our reverse engineered game code can look very similar to the original, which is satisfying and fun.
|
||||
|
||||
The PS2's main CPU is called the EE. It runs GOAL code and the C Kernel, which is Naughty Dog's C++ code. The C Kernel is responsible for bootstrapping GOAL's kernel and exposing Sony library functions to GOAL code. The C Kernel is in `game/kernel`. In OpenGOAL the "EE Thread" runs code than ran on the EE on the PS2. This includes the C Kernel and all GOAL code. There is a single EE thread - GOAL implements its own threading, but this all runs in the same Linux/Windows thread.
|
||||
|
||||
The PS2 has a separate I/O Processor called the IOP. It runs the OVERLORD driver written by Naughty Dog in C. OpenGOAL uses C++ for its implementation of OVERLORD. Like with the EE, there are Sony libraries for the IOP. These are in `game/sce/iop` to distiguish them from EE code. The IOP can run independently from the EE. Unlike the EE, the IOP itself has multiple threads (7 threads) that each have their own OS thread. But only one IOP thread runs at a time, as the IOP was a single-core CPU.
|
||||
|
||||
To give an idea of the size of these (counted by `wc -l`):
|
||||
- OVERLORD is 3700 lines, but still has a lot to implement,
|
||||
- C Kernel is 7432 lines, and is mostly done
|
||||
- SCE is 973 lines, and still has some to implement
|
||||
- System is 1294 lines, and still has some to implement
|
||||
|
||||
## Math Libraries
|
||||
I think most of the math libraries can be decompiled, but there are a few that will need manual work. These are also a great place to do tests as the math functions have very few dependencies and we know what the right answer should be for many of them.
|
||||
|
||||
- `bounding-box` (only some stuff)
|
||||
- `math` (only `rand-uint31-gen`)
|
||||
- `matrix` (only some stuff)
|
||||
- `geometry` (only some stuff)
|
||||
- `trigonometry` (only some stuff)
|
||||
|
||||
At some point we may want to re-implement some of these to be more efficient.
|
||||
|
||||
## The IOP (I/O Processor) Framework
|
||||
This is already implemented.
|
||||
|
||||
The IOP was a separate I/O Processor on the PS2. It runs a cooperative multi-tasking kernel developed by Sony. In OpenGOAL it is implemented in `game/system/IOP_Kernel.h`. The IOP Kernel is managed by the IOP runtime thread (`game/system/iop_thread.h`).
|
||||
|
||||
The library in `game/sce/iop.h` wraps the `IOP_Kernel` in an interface that looks like the Sony libraries used by the game so the IOP code can be ported directly.
|
||||
|
||||
There are a few stub functions that are hardcoded to return the correct values for stuff like CD drive initialization. The main features currently supported are:
|
||||
- Threads (create, wakup, start, sleep, delay, get ID)
|
||||
- Messageboxes to pass data between threads (send, poll)
|
||||
- SIF RPC, a library to receive remote procedure calls from the EE. See `game/sce/sif_ee.h` for the wrapper of the EE side library, and `game/kernel/kdgo.cpp` and `ksound.cpp` for the wrapper around the EE library that's exposed to GOAL.
|
||||
- DMA to the EE for sending data from the IOP to GOAL.
|
||||
|
||||
All this stuff is currently used for loading DGOs, which is tested and working.
|
||||
|
||||
## OVERLORD Framework
|
||||
This is already implemented.
|
||||
|
||||
The OVERLORD is the code written by Naughty Dog that runs on the IOP. It is responsible for sound and loading data. It's surprisingly complicated and some parts of it are extremely poorly written, especially the thread synchronization stuff. My implementation of OVERLORD is in `game/overlord`. It's not complete yet, but the basics are there and it does enough to load DGOs.
|
||||
|
||||
The framework for OVERLORD is already implemented. The C Kernel calls a Sony library function to load OVERLORD. This library function is `sceSifLoadModule`, implemented in `sif_ee.cpp`, which tells the IOP Kernel code to start. This starts up the OVERLORD thread which eventually calls `start_overlord` in `game/overlord/overlord.cpp`. This `start_overlord` function is the entry point to Naughty Dog's OVERLORD library and starts a bunch more threads (see `InitISOFS`), using the Sony IOP library. In total there are 7 threads.
|
||||
|
||||
Once `start_overlord` returns, the initial call to `sceSifLoadModule` returns and the runtime keeps initializing.
|
||||
|
||||
## OVERLORD ISO Thread
|
||||
This is partially implemented.
|
||||
|
||||
This thread is responsible for controlling the DVD drive and the small DVD data buffers used in the IOP. It has a big loop in `ISOThread()` in `iso.cpp` that looks for pending reads, executes them, waits for data to be read, then calls a callback. This code is unbelievably confusing.
|
||||
|
||||
It receives commands from other OVERLORD threads (using a MessageBox) and uses the priority queue implemented in `iso_queue.cpp` to decide which read gets to go first.
|
||||
|
||||
To interact with the DVD drive, it uses an `IsoFS` abstraction, which is a struct containing function pointers to control the drive. The version of OVERLORD in the retail game has only one implemented, called `iso_cd` which uses the actual drive in the PS2. There's also a reference to `fakeiso`, but this is empty in the game. Instead of "emulating" the CD drive functions, I implemented my own version of `fakeiso` mode in `fake_iso.cpp`. This just reads files from your hard drive and uses the `fakeiso.txt` file to map files in the `jak-project` folder to OVERLORD file names (it has it's own system for naming files).
|
||||
|
||||
It also has some sound stuff in it for managing VAG audio streams, but this isn't implemented yet.
|
||||
|
||||
The other threads in OVERLORD are "RPC" threads. They sit in a loop waiting for the main runtime thread (EE thread) to send a remote procedure call (RPC). Then they do something (like maybe sending a message to the ISO thread), maybe wait for something to happen, and then return.
|
||||
|
||||
From the GOAL/EE side of things, RPC calls can be blocking or non-blocking. They can be issued from GOAL (with `rpc-call`) or from the C Kernel (`RpcCall`). Per "channel" (corresponds to an IOP thread), there can only be one RPC call happening at a time. The `rpc-busy?` command can be used to check if an RPC is complete.
|
||||
|
||||
## IOP PLAY (6)
|
||||
This is unimplemented.
|
||||
|
||||
The `PLAY` RPC appears to be relatively simple and plays/stops/pauses/queues a VAG audio stream. It can either use the "AnimationName" system or another system to get the name of the audio stream. I don't know what sound effects in the game are streamed, but I believe there are some.
|
||||
|
||||
I suspect the GOAL side code for this is in `gsound` and `gsound-h`.
|
||||
|
||||
## IOP STR (5)
|
||||
This is unimplemented.
|
||||
|
||||
This is an RPC for streaming data back to the EE. I think this is used to control animation streaming.
|
||||
|
||||
## IOP DGO (4)
|
||||
This is implemented.
|
||||
|
||||
This is the RPC for loading DGO files. The DGO loading is super complicated, but the basic idea is that loading / linking are double buffered. In order to allow linking files to allocate memory, the currently loading file goes in a temporary buffer on the top of the heap. (There are actually two temp buffers that rotate, one for loading out of and one for linking, as the "copy to heap" step is done as part of linking, not loading)
|
||||
|
||||
The final chunk is not double buffered. This is so it can be loaded directly into its final location in the heap. This has three advantages: you don't need to copy it out of a temporary buffer, you can have a file larger than the temp buffer and you can also entirely fill the heap this way (the temp buffers are freed so you don't have to worry about that).
|
||||
|
||||
The IOP side state machine for this is in `iso.cpp`, implemented inside of the DGO load buffer complete callback and is somewhat complicated because DGO info may be split between multiple buffers, and you have to deal with getting partial info. The EE side is in `kdgo.cpp`.
|
||||
|
||||
The DGO syncronization is pretty confusing but I believe I have it working. It may be worth documenting it more (I thought I did already, but where did I put it?).
|
||||
|
||||
## IOP Server/Ramdisk (3)
|
||||
This is implemented, but so far unused and untested.
|
||||
|
||||
This RPC is used to store files in RAM on the IOP. There's a buffer of around 800 kB. I believe it's used for a few different things, in particular the level visibility data. The EE requests data to be loaded from a file on DVD into the "ramdisk" (just a buffer on the IOP), then can request chunks of this file. Of course it is not as fast as storing the file in the EE RAM, but it is much faster than reading from the DVD again.
|
||||
|
||||
This is what Andy Gavin refers to when they said they did "things they weren't supposed to" with the "one megabyte of memory that wasn't being used".
|
||||
|
||||
|
||||
## IOP Loader (2)
|
||||
This is unimplemented.
|
||||
|
||||
This is used to control the loading of music and soundbanks. I haven't touched it yet. Music and soundbanks are loaded into IOP memory when you switch levels.
|
||||
|
||||
## IOP Player (1)
|
||||
This is unimplemented.
|
||||
|
||||
This is used to control the playing of sound, and goes with Loader. Like PLAY it can play VAG audio streams. I'm not sure which one is actually used for streaming audio, maybe both?
|
||||
|
||||
## IOP VBlank Handler
|
||||
This is unimplemented.
|
||||
|
||||
The IOP's kernel will call `VBlank_Handler` on each vblank. This is once per frame, and I don't know where it is, or if its tied to the actual HW vblank or framebuffer swap, if it happens at 30/60 fps (or even/odd frames if 30 fps). I suspect it's the real vblank at 60 fps but I don't know.
|
||||
|
||||
This does some music fade calculations and sends some data to the EE. In GOAL this is called the `sound-iop-info`.
|
||||
|
||||
The EE first has to do some set up to tell the IOP where to copy the data, which I believe is done in another sound RPC from GOAL.
|
||||
|
||||
We'll also need to add some stuff to `system` and `sce/iop` to set this up, which will have to work with frame timing stuff so it happens at the right part of the frame.
|
||||
|
||||
|
||||
## Sound Library
|
||||
This is a pretty big one. To actually make sounds, OVERLORD code uses a third-party sound library called 989SND. Internally 989SND uses the SPU2 (Sound Processor) to actually do the "sound math" to decode ADPCM, do ADSR for the sampled sounds, and do reverb/mixing.
|
||||
|
||||
I think the lowest effort sound implementation is to try to reimplement 989SND + the SPU as a single library. This could be tested and developed in isolation from everything else.
|
||||
|
||||
We'll also need to pick a library for doing audio on PC and a design for how to keep the audio in sync. My gut feeling is to let the IOP side audio stuff just run totally independent from everything else, like the real game does. Let the audio sampling be driven by the sound device so you never have any crackling/interpolation artifacts. This is why the audio keeps going even after the game crashes on PS2.
|
||||
|
||||
## GOAL Kernel
|
||||
The GOAL kernel needs some modification to work on x86. It implements userspace threading and needs to know the details of how to back up the current CPU state and restore it. It also needs to work with the compiler to make sure that the kernel and compiler agree on what registers may not be preserved across a thread suspend There are also some CPU specific details on how to do dynamic throw/catches, unwinding stack frames, and passing initial arguments to a thread.
|
||||
|
||||
In OpenGOAL, the `rsp` is a "real" pointer and all other pointers are "GOAL pointer"s (offset from base of GOAL memory), so there are some details needed to correctly save/restore stacks.
|
||||
|
||||
A final detail is we will probably want/need the ability to increase the default size of stack that can be backed up on suspend. The default is 256 bytes so if our compiler does worse than the original and we use more stack space, we could run out. There's a check for this so it shouldn't be hard to detect.
|
||||
|
||||
## Jak Graphics Basics
|
||||
The PS2 has some special hardware that's used for graphics. These are the DMAC, the VU1, and the GS.
|
||||
|
||||
The DMAC is a sophisticated DMA controller. It runs separately from the EE and can copy data from one place to another at pretty high speed. If it is not stalled for any reason it can reach 2.4 GB/sec. The main RAM is only good for around 1.2 GB/sec so in practice "big" things don't move around any faster than 1.2 GB/sec on average. It's used to send graphics data from main memory to the other components. It can be configured, but it's not programmable. It can do simple transfers, like "copy this block of data from here to there", and more complicated things, like following linked lists.
|
||||
|
||||
The VU1 takes the role of vertex shaders. It can be programmed, but only in assembly, and it is extremely challenging and confusing. It has an extremely small memory (16 kB), but this memory is extremely fast. It's role is usually to do vertex transformations and lighting, then generate a list of commands to send to the GS. The `XGKICK` instruction on VU1 is used to send data from the VU1 memory to the GS.
|
||||
|
||||
The GS is the actual GPU. It has VRAM and receives commands from a few different places, including:
|
||||
- VU1 `XGICK`s stuff to it directly, bypassing the main bus used by DMAC/CPU memory access. This is called PATH 1 and is most commonly used in Jak 1.
|
||||
- When DMAing stuff to VU1, it first goes through a thing called VIF1 which can "unpack" data. There is a special command that you can give to VIF1 which tells it to "send data directly to the GS".
|
||||
- DMA sends data directly from EE main memory to GS (Path 3), unused by Jak 1
|
||||
|
||||
The GS is like pixel shaders but it's very simple - it's not programmable and only can do a few fixed things. The GS also has the VRAM, which can contain frame buffers, z buffers, textures, and scratch area for effects.
|
||||
|
||||
My understanding is that during a frame, the EE generates a long list of things to draw. These are a DMA "chain" - basically a complicated linked-list like data structure that the PS2's DMA knows how to handle. I believe some graphics calculations are done on the EE - particularly the environment mapping.
|
||||
|
||||
## DMA
|
||||
|
||||
## Display
|
||||
|
||||
## Texture
|
||||
|
||||
## Collision System
|
||||
|
||||
## Joint
|
||||
|
||||
## BSP
|
||||
|
||||
## Merc Blend Shape
|
||||
|
||||
## Ripple
|
||||
|
||||
## Bones
|
||||
|
||||
## Generic Merc
|
||||
|
||||
## Generic TIE
|
||||
|
||||
## Shadow
|
||||
|
||||
## Font
|
||||
|
||||
## Decompression
|
||||
|
||||
## Background
|
||||
|
||||
## Draw Node Culling
|
||||
|
||||
## Shrubbery
|
||||
|
||||
## TFRAG
|
||||
|
||||
## TIE
|
||||
|
||||
## Particle
|
||||
|
||||
## Time of Day
|
||||
|
||||
## Sky
|
||||
|
||||
## Load boundary
|
||||
|
||||
## Sound
|
||||
|
||||
## Controllers
|
||||
|
||||
## IOP Streaming
|
||||
|
||||
## Ocean
|
||||
|
||||
## Navigate
|
||||
@ -2,8 +2,8 @@
|
||||
; Each entry should consist of an ISO name, followed by a file name
|
||||
; note that tweakval, vagdir, screen1 have dummy data for now.
|
||||
|
||||
KERNEL.CGO out/KERNEL.CGO
|
||||
GAME.CGO out/GAME.CGO
|
||||
KERNEL.CGO out/iso/KERNEL.CGO
|
||||
GAME.CGO out/iso/GAME.CGO
|
||||
TEST.CGO resources/TEST.CGO
|
||||
TWEAKVAL.MUS resources/TWEAKVAL.MUS
|
||||
VAGDIR.AYB resources/VAGDIR.AYB
|
||||
|
||||
@ -255,8 +255,8 @@ char* DecodeFileName(const char* name) {
|
||||
result = MakeFileName(CODE_FILE_TYPE, name + 6, 0);
|
||||
} else if (!strncmp(name, "$RES/", 5)) {
|
||||
result = MakeFileName(RES_FILE_TYPE, name + 5, 0);
|
||||
} else if (!strncmp(name, "$JAK-PROJECT/", 13)) {
|
||||
result = MakeFileName(JAK_PROJECT_FILE_TYPE, name + 13, 0);
|
||||
} else if (!strncmp(name, "$ISO/", 5)) {
|
||||
result = MakeFileName(ISO_FILE_TYPE, name + 5, 0);
|
||||
} else {
|
||||
printf("[ERROR] DecodeFileName: UNKNOWN FILE NAME %s\n", name);
|
||||
result = nullptr;
|
||||
@ -380,8 +380,8 @@ char* MakeFileName(int type, const char* name, int new_string) {
|
||||
// REFPLANT? no idea
|
||||
static char nextDir[] = "/";
|
||||
sprintf(buf, "%sconfig_data/refplant/%s", nextDir, name);
|
||||
} else if (type == JAK_PROJECT_FILE_TYPE) {
|
||||
sprintf(buffer_633, "/%s", name);
|
||||
} else if (type == ISO_FILE_TYPE) {
|
||||
sprintf(buffer_633, "/out/iso/%s", name);
|
||||
} else {
|
||||
printf("UNKNOWN FILE TYPE %d\n", type);
|
||||
}
|
||||
|
||||
@ -42,8 +42,8 @@ enum GoalFileType {
|
||||
CNT_FILE_TYPE = 0x3a,
|
||||
RES_FILE_TYPE = 0x3b,
|
||||
REFPLANT_FILE_TYPE = 0x301,
|
||||
// added this, allows access directly to jak-project/ from within the game.
|
||||
JAK_PROJECT_FILE_TYPE = 0x302
|
||||
// added this, allows access directly to out/iso from fileio.
|
||||
ISO_FILE_TYPE = 0x302
|
||||
};
|
||||
|
||||
constexpr char FOLDER_PREFIX[] = "";
|
||||
|
||||
@ -1,12 +0,0 @@
|
||||
;; test the use of #cond to evaluate goos expressions at compile time
|
||||
|
||||
(#cond
|
||||
((> 2 (+ 2 1))
|
||||
1
|
||||
(invalid-code)
|
||||
)
|
||||
|
||||
((< 2 (+ 1 2))
|
||||
3
|
||||
)
|
||||
)
|
||||
@ -143,8 +143,9 @@ Val* Compiler::compile_asm_file(const goos::Object& form, const goos::Object& re
|
||||
|
||||
// save file
|
||||
if (write) {
|
||||
auto output_name = m_goos.reader.get_source_dir() + "/data/" + obj_file_name + ".o";
|
||||
file_util::write_binary_file(output_name, (void*)data.data(), data.size());
|
||||
file_util::create_dir_if_needed(file_util::get_file_path({"out", "obj"}));
|
||||
file_util::write_binary_file(file_util::get_file_path({"out", "obj", obj_file_name + ".o"}),
|
||||
(void*)data.data(), data.size());
|
||||
}
|
||||
} else {
|
||||
if (load) {
|
||||
|
||||
@ -200,8 +200,10 @@ void compile(const std::vector<std::unordered_map<int, std::string>>& text,
|
||||
}
|
||||
auto data = gen.generate_v2();
|
||||
|
||||
file_util::create_dir_if_needed(file_util::get_file_path({"out", "iso"}));
|
||||
file_util::write_binary_file(
|
||||
file_util::get_file_path({"out", fmt::format("{}{}.TXT", lang, uppercase(group_name))}),
|
||||
file_util::get_file_path(
|
||||
{"out", "iso", fmt::format("{}{}.TXT", lang, uppercase(group_name))}),
|
||||
data.data(), data.size());
|
||||
}
|
||||
}
|
||||
|
||||
14
out/hash.md5
14
out/hash.md5
@ -1,7 +1,7 @@
|
||||
b06369c2dd9197cb17aae6286246caf9 0COMMON.TXT
|
||||
da0d6012181f13803e8b3267a4099b97 1COMMON.TXT
|
||||
4a9a27beb4ce7e75fa4c70811d2c0013 2COMMON.TXT
|
||||
abcc25e5d7469dd6a572dc53dbb9671c 3COMMON.TXT
|
||||
82eabdb7159f2059fbdbd18bb6fc06aa 4COMMON.TXT
|
||||
5d62de2c78b4cf102b9a78f3aa96c8c9 5COMMON.TXT
|
||||
9495f80955e6782513fe12f6539fc8e7 6COMMON.TXT
|
||||
b06369c2dd9197cb17aae6286246caf9 iso/0COMMON.TXT
|
||||
da0d6012181f13803e8b3267a4099b97 iso/1COMMON.TXT
|
||||
4a9a27beb4ce7e75fa4c70811d2c0013 iso/2COMMON.TXT
|
||||
abcc25e5d7469dd6a572dc53dbb9671c iso/3COMMON.TXT
|
||||
82eabdb7159f2059fbdbd18bb6fc06aa iso/4COMMON.TXT
|
||||
5d62de2c78b4cf102b9a78f3aa96c8c9 iso/5COMMON.TXT
|
||||
9495f80955e6782513fe12f6539fc8e7 iso/6COMMON.TXT
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
|
||||
(start-test "game-text")
|
||||
|
||||
(let ((text (the game-text-info (load "$JAK-PROJECT/out/0TEST.TXT" *common-text-heap*))))
|
||||
(let ((text (the game-text-info (load "$ISO/0TEST.TXT" *common-text-heap*))))
|
||||
(format 0 "~I~%" text)
|
||||
(expect-true (= #x123 (-> text data 0 id)))
|
||||
(expect-true (= #x456 (-> text data 1 id)))
|
||||
|
||||
@ -56,11 +56,6 @@ GoalTest::CompilerTestRunner ControlStatementTests::runner;
|
||||
|
||||
TEST_F(ControlStatementTests, ConditionalCompilation) {
|
||||
runner.run_static_test(env, testCategory, "conditional-compilation.static.gc", {"3\n"});
|
||||
// TODO - test-conditional-compilation-2.gc
|
||||
// these numbers match the game's memory layout for where the symbol table lives.
|
||||
// it's probably not 100% needed to get this exactly, but it's a good sign that the global
|
||||
// heap lives in the right spot because there should be no divergence in memory layout when its
|
||||
// built. This also checks that #t, #f get "hashed" to the correct spot.
|
||||
}
|
||||
|
||||
TEST_F(ControlStatementTests, Blocks) {
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user