This should avoid the stuttering due to slow unloading on some drivers.
I also turned up the amount of stuff we load per frame since nobody has
been complaining about stutters there, but there has been a few cases of
levels loading in too slowly.
(this only changes graphics, not actual GOAL level load times).
Makes the glow sprite renderer flush when full capacity is reached,
instead of at the end. Also allows us to reduce the textures used for it
(finally). Worst case scenario there's 4-5 flushes per frame.
Fixes incessant flickering in the dig.
Switches the slime look up table to be a texture, since I guess intel
drivers are terrible and putting the array in the shader makes it
extremely slow.
Also, a few minor changes:
- removed art-groups from the test-zone levels since this causes the
compiler to re-decompile the game, and makes the launcher slower. (left
it in commented out)
- Switched `decompile_code` to false by default in jak 2, in case people
run the decompiler and don't want to wait forever
- Fixed build warnings
Major change to how `deftype` shows up in our code:
- the decompiler will no longer emit the `offset-assert`,
`method-count-assert`, `size-assert` and `flag-assert` parameters. There
are extremely few cases where having this in the decompiled code is
helpful, as the types there come from `all-types` which already has
those parameters. This also doesn't break type consistency because:
- the asserts aren't compared.
- the first step of the test uses `all-types`, which has the asserts,
which will throw an error if they're bad.
- the decompiler won't emit the `heap-base` parameter unless necessary
now.
- the decompiler will try its hardest to turn a fixed-offset field into
an `overlay-at` field. It falls back to the old offset if all else
fails.
- `overlay-at` now supports field "dereferencing" to specify the offset
that's within a field that's a structure, e.g.:
```lisp
(deftype foobar (structure)
((vec vector :inline)
(flags int32 :overlay-at (-> vec w))
)
)
```
in this structure, the offset of `flags` will be 12 because that is the
final offset of `vec`'s `w` field within this structure.
- **removed ID from all method declarations.** IDs are only ever
automatically assigned now. Fixes#3068.
- added an `:overlay` parameter to method declarations, in order to
declare a new method that goes on top of a previously-defined method.
Syntax is `:overlay <method-name>`. Please do not ever use this.
- added `state-methods` list parameter. This lets you quickly specify a
list of states to be put in the method table. Same syntax as the
`states` list parameter. The decompiler will try to put as many states
in this as it can without messing with the method ID order.
Also changes `defmethod` to make the first type definition (before the
arguments) optional. The type can now be inferred from the first
argument. Fixes#3093.
---------
Co-authored-by: Hat Kid <[email protected]>
Some files were in the `banned_objects` list and were thus excluded from
the `all_objs` file.
Also implements the `pexcw` instruction which is only used in `hfrag`
code.
Fixes missing collision geometry reported in
https://github.com/open-goal/jak-project/issues/3011
The issue happens when there are 256 polygons. In this case `num-polys`
is 0 (it's a u8). There are actual cases where there are 0 polygons, so
we have to do a more complicated check to get the real count. I should
have done this in the first place, but it seemed to work...
Fixes#2994 the underport and consite icons being green (same color as
the goal, likely copy-pasta).
Fixes the hundredths timer in the race timer actually being a 60ths
timer.
Started at 349,880,038 allocations and 42s
- Switched to making `Symbol` in GOOS be a "fixed type", just a wrapper
around a `const char*` pointing to the string in the symbol table. This
is a step toward making a lot of things better, but by itself not a huge
improvement. Some things may be worse due to more temp `std::string`
allocations, but one day all these can be removed. On linux it saved
allocations (347,685,429), and saved a second or two (41 s).
- cache `#t` and `#f` in interpreter, better lookup for special
forms/builtins (hashtable of pointers instead of strings, vector for the
small special form list). Dropped time to 38s.
- special-case in quasiquote when splicing is the last thing in a list.
Allocation dropped to 340,603,082
- custom hash table for environment lookups (lexical vars). Dropped to
36s and 314,637,194
- less allocation in `read_list` 311,613,616. Time about the same.
- `let` and `let*` in Interpreter.cpp 191,988,083, time down to 28s.
This sets up the extractor for jak 2. I was expecting that I'd have to
make some more significant changes to the decompiler/compiler path
stuff, but this was not the case!
The only real change is that you can now provide multiple ISO hashes for
an entry in `ISOMetadata`. This is needed for the two different NTSC
versions, which have the same configs, serials, and ELF hashes, but
slightly different contents.
I also didn't add the korean version because I don't have the info for
it.
---------
Co-authored-by: ManDude <[email protected]>
This change adds a few new features:
- Decompiler automatically knows the type of `find-parent-method` use in
jak 1 and jak2 when used in a method or virtual state handler.
- Decompiler inserts a call to `call-parent-method` or
`find-parent-state`
- Removed most casts related to these functions
There are still a few minor issues around this:
- There are still some casts needed when using `post` methods, as `post`
is just a `function`, and needs a cast to `(function none)` or similar.
It didn't seem easy to change the type of `post`, so I'm not going to
worry about it for this PR. It only shows up in like 3 places in jak 2.
(and 0 in jak 1)
- If "call the handler if it's not #f" logic should probably be another
macro.
Fixes#805
Adds controller LED features to Jak 2:
- progressive flickering denoting health
- copies tomb simon says puzzle colors
- unique colors for each gun
- orange color for being indax
- yellow color for being in mech
- purple color for being darkjak
- blue color for being in board
- red flash when wanted.
May add more features later?
Also did some minor clean-up on some types.
This renames the method object in `defmethod`s to `this` and adds
detection for the `set-time!` and `time-elapsed?` macros.
Definitely my biggest PR yet...
- state handlers that are not inlined lambdas have smarter type
checking, getting rid of 99.9% of the casts emitted (they were not
useful)
- art groups were not being properly linked to their "master" groups.
- `max` in `ja` in Jak 2 was not being detected.
Another huge PR...
Previously, `object` and `none` were both top-level types. This made
decompilation rather messy as they have no LCA and resulted in a lot of
variables coming out as type `none` which is very very wrong and
additionally there were plenty of casts to `object`. This changes it so
`none` becomes a child of `object` (it is still represented by
`NullType` which remains unusable in compilation).
This change makes `object` the sole top-level type, and the type that
can represent *any* GOAL object. I believe this matches the original
GOAL built-in type structure. A function that has a return type of
`object` can now return an integer or a `none` at the same time.
However, keep in mind that the return value of `(none)` is still
undefined, just as before. This also makes a cast to `object`
meaningless in 90% of the situations it showed up in (as every single
thing is already an `object`) and the decompiler will no longer emit
them. Casts to `none` are also reduced. Yay!
Additionally, state handlers also don't get the final `(none)` printed
out anymore. The return type of a state handler is completely
meaningless outside the event handler (which is return type `object`
anyway) so there are no limitations on what the last form needs to be. I
did this instead of making them return `object` to trick the decompiler
into not trying to output a variable to be used as a return value
(internally, in the decompiler they still have return type `none`, but
they have `object` elsewhere).
Fixes#1703Fixes#830Fixes#928
This PR adds detection of the `launch-particles` and `seconds-per-frame`
macros to the decompiler, removing a lot of bloat and hiding many
process register uses.
I also added `og:preserve-this` comments to as many manual patches and
comments as I could, which will soon be used in conjunction with CI to
hopefully catch any regressions in future big decomp update PRs.
I have some concerns about the `launch-particles` macro (more details in
`sparticle-launcher.gc`) , but thus far, I have not seen anything break
yet.
---------
Co-authored-by: water <[email protected]>
This will create a folder like `decompiler_out/jak1/entities` and save a
JSON file per level with all the actors.
Also, it should hopefully make custom level building a little faster.
I think this is very likely to fix
https://github.com/open-goal/jak-project/issues/2970
We had somebody report a stacktrace from the debugger, and it was
immediately after calling `gen-perms`.
I found that `gen-perms` writes past the end of a stack array during
this mission, and at the same time as the reported freezes.
I was unable to recreate the original freeze after making this change.
This PR adds a frame rate option to the graphics menu for some of the
most common refresh rates.
Jak 2 has much better support for variable frame rates than Jak 1 out of
the box, but there are still some edge cases, most prominently the fact
that sprites are still limited to the 300 tick system, which is most
noticeable on glow sprites. For this, I abused the glow boost debug
setting to scale the glow based on the frame rate.
While testing, I noticed two other cases that I have also patched,
there's likely to be many more that are yet to be found, but aside from
that, the game is playable as normal.
https://github.com/open-goal/jak-project/assets/6624576/ad4db24f-cd27-4237-a155-0db7008160f3
There are potentially still some minor issues with the resulting files.
Some of them appear to have minor artifacts that playing through the
actual game do not -- but this is a much better starting point for
someone to iterate from if they are interested in improving things.
Export models with "wind". The levels with wind models are:
firecanyon (9), beach (5), village1 (7), lavatube (2).
Sometimes a single object is made up of multiple models - for example
the tree in sandover is actually several meshes.
This tries to match the original behavior of the sprite allocation
"randomness", which should reduce the number of very empty sprite blocks
sent to C++.
- fix speaker names and time frames being uneditable in Jak 1
- added toggle to auto-selected a newly created scene as current
- changed the subtitle summary format slightly.
- current scene's name now appears in the UI
- fixes flava and mode not being correct after a new music starts
playback by setting them every frame (there is at most 1 frame where
it's wrong).
- fixes city pursuit theme not playing if the city alarm was triggered
while the music wasn't playing yet.
Some general improvements for the texture animator:
- Clouds are special cased, saving about 1 ms per frame
- Adjusting the amount of clouds now actually works.
- Fixed an issue with the brightness of clouds, and the way that they
fade out around the edges.
Fixes#2167
Reduces test flakiness if ran on multiple threads and gets rid of a few
hundred files from the source tree
I believe this also makes #1434 irrelevant and it can be closed.
---------
Co-authored-by: ManDude <[email protected]>
Fix an issue where the commit sha would not use the right blending mode
if `draw-raw-image` is running at the same time.
Fix an issue where japanese subtitles would accidentally overwrite other
textures, leading to random textures missing. (in particular, glows
would disappear after watching a cutscene with the subtitles on)
- fix deci2 hang when closing the game in retail mode.
- change bigmap to always filter because the pixels look really ugly.
- don't start the game in fullscreen by default if we're debugging.
Fix the bug described in
https://github.com/open-goal/jak-project/issues/2882 where some shrubs
are transparent when they shouldn't be. The problem was that we never
carefully looked at the settings in `gs-prim`, which has a bit to
enable/disable alpha blending entirely. Now it should be correct for
both jak 1 and jak 2. To see this change, you'll need to re-extract.
Also adds a setting to disable saving texture .pngs, to speed up
decompilation. I left it on for jak 1 (to avoid confusion for texture
swapping(, but off for jak 2 for now.
Rotates the log files with a timestamp instead of copying all files and
incrementing an integer. Increases the amount of info you have when
looking at user's log files (ie. when looking at all the files, the file
creation dates are accurate).
Also simplifies the API for setting the log file, and `gk` logs are now
game specific with `jak1` or `jak2`. Which should be useful going
forward.
Lastly, added a flag to all CLIs to disable ansi colors for people that
want to do so. Though at the same time, there is finally a workaround in
jenkins to fix ANSI colors in the truncated log view -- so I'm not sure
why anyone would want to get rid of the color information. You can even
setup text editors to display the color info making log parsing much
easier. Fixes#1917
---------
Co-authored-by: ManDude <[email protected]>
The way we got/stored background matrices is a bit weird and full of
leftovers from the first attempts at porting renderers. This doesn't
work well with the Jak 2 "other camera" system where some stuff is
rendered with a different camera matrix.
This cleans most of it up. The exception is that the collide mesh
renderer and the additional sprite culling I added still need to peek at
some cached camera matrices.
This fixes the problem where etie uses the wrong matrices for "other
camera" levels. Now the "hole covers" go in the holes in the background
of the throne room.
This changes how `BlitDisplays.cpp` works so it looks at the current
render buffer, rather than the back buffer.
This approach is a bit faster because we avoid copying the back buffer
on every single frame.
It also removes the black frames when the transition starts/stops.
The remaining issues are:
- there's still a single frame of weirdness with the sprite glow
renderer.
- when changing resolutions, it doesn't work super well.
When drawing the spinning palace, there's a terrible hack that lets some
stuff be drawn with different camera matrices.
The ocean is drawn with camera-other (which spins), but was being culled
with camera (doesn't spin). This changes ocean to use the right camera
matrix.
Note that the ocean is special when it comes to camera-other - it always
uses camera-other if there is one. Other uses are set per-level and
should already be handled correctly.
This should fix a crash and animation bug in snowy. The way to trigger
the bug:
- go on ice
- move forward slowly
- stop moving forward. Reach zero speed when the frame number isn't
between 30 and 35 of the ice walking animation
- Due to this bug, the animation gets stuck at frame 60
- Take damage (due to normal lurker or ice lurker)
- Sidekick eye animation crashes because a frame number is NaN.
There was a single static path buffer being shared between multiple file
i/o threads. So sometimes you would end up using the wrong path for the
file, and getting size/data for the wrong file.
I think the original reason to have this buffer was just me being lazy
when we changed how project paths works a long time ago. It was a bad
idea in the first place.
Fixes skull gems using a low resolution texture.
Fixes issue where jak in cutscenes is dark after watching oracle-level-1
(and likely other bugs with texture animations getting stuck)
---------
Co-authored-by: ManDude <[email protected]>
The progress menu loads its icon textures from a .STR file that we were
previously ignoring.
This change:
- updates the decompiler so it can process a .STR file containing a
texture
- adds a feature to force an entire page to always be loaded in the PC
renderer by putting all textures in the GAME.FR3 file.
- regenerates the texture offset map file for jak 2 with these new
textures
For now, I've just put the icon textures in GAME.FR3. The downside is
that these will always stay on the GPU, using up VRAM even when they
aren't needed. But the entire GAME.FR3 file is under 3 MB so I think
it's ok.
- Add security wall animation
- Add waterfall animations
- Add lava animations
- Update layer values from the game to fix the security wall
- Remove leftover debug in `level.gc` that would break level-specific
animations on the second time you visited the level
- Optionally load animated slot textures to the pool so generic can use
them (fixes skull gems in UI)
This fixes the crash reported in
https://github.com/open-goal/jak-project/issues/2833
There was a memory bug here for a long time where our array of `VagCmd`
in `iso_queue.cpp` was too small. This caused GetVagCommand to return
bogus pointers, and sound code would write over other parts of memory.
Added framework to do texture animations entirely in C++. Currently only
works on relatively simple ones, and doesn't handle updating all
parameters - only the speeds.
Connected texture animations to merc and tfrag for skull gems, dark
bomb, and scrolling conveyors.
Cleaned up Tfragment/Tfrag3, which used to be two classes. This was one
of the first C++ renderers, so it had a weird design.
I havn't tested it yet, but I can almost guarantee that atleast `goalc`
will not work in the slightest!
But the project is atleast fully compiling. My hope is to start
translating some AVX to NEON next / get `goalc` working...eventually.
The main thing that was done here was to slightly modify the new
subtitle-v2 JSON schema to be more similar to the existing one so that
it can properly be used in Crowdin.
Draft while I double-check the diff myself
Along the way the following was also done (among other things):
- got rid of as much duplication as was feasible in the serialization
and editor code
- separated the text serialization code from the subtitle code for
better organization
- simplified "base language" in the editor. The new subtitle format has
built-in support for defining a base language so the editor doesn't have
to be used as a crutch. Also, cutscenes only defined in the base come
first in the list now as that is generally the order you'd work from
(what you havn't done first)
- got rid of the GOAL subtitle format code completely
- switched jak 2 text translations to the JSON format as well
- found a few mistakes in the jak 1 subtitle metadata files
- added a couple minor features to the editor
- consolidate and removed complexity, ie. recently all jak 1 hints were
forced to the `named` type, so I got rid of the two types as there isn't
a need anymore.
- removed subtitle editor groups for jak 1, the only reason they existed
was so when the GOAL file was manually written out they were somewhat
organized, the editor has a decent filter control, there's no need for
them.
- removed the GOAL -> JSON python script helper, it's been a month or so
and no one has come forward with existing translations that they need
help with migrating. If they do need it, the script will be in the git
history.
I did some reasonably through testing in Jak1/Jak 2 and everything
seemed to work. But more testing is always a good idea.
---------
Co-authored-by: ManDude <[email protected]>
## Problem
OpenGOAL uses OpenGL 4.3.
Apple stopped upgrading OpenGL after 4.1, so the way OpenGOAL currently
works will never be playable on Macs using OpenGL.
## Solution
Luckily, downgrading to OpenGL 4.1 is not a huge change (at least it
doesn't seem like it to my untrained eyes).
## Changes
* set hints for OpenGL 4.1 instead of 4.3 for __APPLE__
* skip the OpenGL debugging callback setup for macOS (requires 4.3)
* bump down the version string for all shaders
* stop using the `binding` layout qualifier in shader code
* move the `flat` qualifier first (not sure if this is a 4.1 thing or
just Macs being more strict)
* don't mix signed and unsigned ints in shaders (not sure if this is a
4.1 thing or just Macs being more strict)
* add some hacky CPP to the Shader constructor for binding texture units
and bones buffers based on variable names in the shader code
## Results
The current event-based approach is very difficult to get right, and it
depends on no events ever being missed. This changes the keyboard/mouse
handling code to a polling-based approach.
Other fixes:
- an issue where modifier keys were not able to be successfully bound
(like Left Shift to `X`)
- improves cursor hiding (except when you use the start menu, this seems
like an SDL issue, see comment)
- Better discarding of kb/mouse inputs when imgui intercepts input
- properly swap bindings when an already set key is assigned, even if it
crosses the distinction of an analog vs normal button
Fixes#2800
Changes the DGO build order so that the city gets compiled first, and a
random guess at an "order" of which levels people might edit more often.
Most of the data-only borrow files are moved to the end as well.
Also moves around files in the `goal_src` tree to a structure that makes
a bit more sense, some files were either in the completely wrong place,
their folders had strange names, were too deep for no reason or were
just too far away from other relevant files. This structure should make
it easier to guess a file's location.
Removes a leftover case for falling to back to generic-merc in jak 1.
This should fix the bug where daxter's face wasn't animated in the intro
after floating-point blerc was added.
Fixes empty boxed arrays of strings breaking some decomp
(`ctywide-speech` and `race-info`).
Adds `decomp-as` tag to decompiler types so that the static data
decompiler can use macros like `meters` and `seconds` on fields that
aren't of type `meters` or `time-frame`.
Adds `override` tag to decompiler types which overrides the type of
field with that name. The type must be a child type of the original
field's type (or the same type, but why would you do this?).
Fixes the camera being offset for `drillmtn` after loading `palout`
once.
This is a huge refactor sadly.
Fixes orb softlocks during races and other side missions.
The side mission tasks will now not count as completed until the
precursor orb has been picked up.
Races will not let you advance (or even pause the game) until the
precursor orb has been picked up.
Avoids blocking other IOP threads on IO.
Idk if it'll really help anything, but at least theoretically it might
stop some pathologically slow IO case from blocking the VAG buffer
switching.
This is a major deviation from the original game, which did not have any
way to access the nest after beating Metal Kor as the air train gets
hidden when he is beaten. This was mostly annoying because there are
precursor orbs in that level that you might miss.
This makes it so the air train can once again be used to enter the nest
even after beating Metal Kor. The rest of the level remains mostly the
same, except the Rift Ring does not spawn and an invisible wall is added
to the Metal Kor arena to prevent you from entering it as you are
normally unable to leave it anyway.
Trying to make up for some of the startup speed lost in the SDL
transition. This saves about 1s from start (from ~3s), and about 500 MB
of RAM.
- Faster TIE unpack by merging matrix groups, more efficient vertex
transforms, and skipping normal transforms on groups with no normals.
- Refactor generic merc and merc to use a single renderer with multiple
interfaces, rather than many renderers. Removed "LightningRenderer" as a
special thing, but Warp is still special
- Add more profiling stuff to startup and the loader.
- Remove `SDL_INIT_HAPTIC` - this turned out to be needed for
force-feedback steering wheels, and not needed for controller vibration
- Switched `vag-player` to use quicksort instead of the default GOAL
sort (very slow)
```lisp
(defenum collide-action
:type uint32
:bitfield #t
(solid 0) ;; used for solid things
(rider-plat-sticky 1) ;; used for platforms in rider/platform interactions
(rider-target 2) ;; used for target in rider/platform interactions
(edgegrab-active 3) ;; set/cleared when entering/exiting edgegrab states
(rider-plat 4) ;; used for platforms in rider/platform interactions
(unused 5) ;; totally unused?
(edgegrab-possible 6) ;; used when edge grab checks should be done
(edgegrab-cam 7) ;; set/cleared when entering/exiting edgegrab states
(swingpole-active 8) ;; set/cleared when entering/exiting swingpole states
(racer 9) ;; set/cleared when entering/exiting racer states
(attackable 10) ;; used for something to do with attacking/damaging
(attackable-unused 11) ;; seems to relate to attacking - set in several places but never tested for?
(snowball 12) ;; set/cleared when entering/exiting snowball states
(tube 13) ;; set/cleared when entering/exiting tube states
(flut 14) ;; set/cleared when entering/exiting flutflut states
(racer-grounded 15) ;; set/cleared when entering/exiting certain racer states w/ extra conditions
(racer-unused 16) ;; seems to relate to racer - never set, only cleared in one place?
)
```
This moves the blerc math from mips2c to the Merc2 renderer, and uses
floats instead.
We could potentially do this on the GPU, which would be even faster, but
this isn't that slow in the first place.
Adds support for adding custom subtitles to Jak 2 audio. Comes with a
new editor for the new system and format. Compared to the Jak 1 system,
this is much simpler to make an editor for.
Comes with a few subtitles already made as an example.
Cutscenes are not officially supported but you can technically subtitle
those with editor, so please don't right now.
This new system supports multiple subtitles playing at once (even from a
single source!) and will smartly push the subtitles up if there's a
message already playing:
Unlike in Jak 1, it will not hide the bottom HUD when subtitles are
active:
Sadly this leaves us with not much space for the subtitle region (and
the subtitles are shrunk when the minimap is enabled) but when you have
guards and citizens talking all the time, hiding the HUD every time
anyone spoke would get really frustrating.
The subtitle speaker is also color-coded now, because I thought that
would be fun to do.
TODO:
- [x] proper cutscene support.
- [x] merge mode for cutscenes so we don't have to rewrite the script?
---------
Co-authored-by: Hat Kid <[email protected]>
Should fix https://github.com/open-goal/jak-project/issues/2679
Here's a test program that will trigger the bug when near these guards:
```lisp
(define *cquery* (new 'global 'collide-query))
(defun test-bad-collide ()
(let ((lower (new 'static 'vector :x 1681893.8750 :y 61314.2031 :z 345208.6562 :w 1.))
(upper (new 'static 'vector :x 1701603.8750 :y 67624.0625 :z 357881.0312 :w 1.))
;(cquery (new 'stack-no-clear 'collide-query))
)
(set! (-> *cquery* collide-with) (the-as collide-spec 1))
(set! (-> *cquery* ignore-process0) #f)
(set! (-> *cquery* ignore-process1) #f)
(set! (-> *cquery* ignore-pat) (new 'static 'pat-surface :noentity #x1 :nojak #x1 :probe #x1 :noendlessfall #x1))
(set! (-> *cquery* action-mask) (collide-action solid))
(set! (-> *cquery* bbox min quad) (-> lower quad))
(set! (-> *cquery* bbox max quad) (-> upper quad))
(format 0 "doing collide...~%")
(fill-using-bounding-box *collide-cache* *cquery*)
(format 0 "have ~d and ~d~%" (-> *collide-cache* num-tris) (-> *collide-cache* num-prims))
)
(none)
)
```
As far as I can tell, there's a totally invalid collide-hash with an
inside `axis-scale.z`. On the PS2, it gets ignore because of how
float->int works for floats that are too big. On PC, it ends up using a
negative value and loop forever.
Gives proper names to almost every color. It is very apparent that some
colors are context-sensitive/made for a specific purpose, so those
colors were named after that purpose instead of a generic color name.
Normally, when they allocate a VagCmd, they do a bunch of stuff to clear
all the status bits and reset things
in particular the InitVAGCmd function does a lot
but for the stereo command, they do a lot less:
Which means that the new_stereo_command can just have random status bits
left over from whatever the last user had.
we seem to end up in a state where byte21 is set, and this causes
everything else to be wrong and off-by-one dma transfer. My guess is
that the original game avoided this bug due to lucky timing that I don't
understand.
I think the fix of just clearing byte21 is ok because there's no way
that the old value of the byte is useful after the command is
repurposed.
Also fixed an original game bug in `loader.gc` on a method that's called
quite often, though I have no clue what erroneous behavior it could have
even caused.
- elec gates now always render at max quality if you have PS2 lods
disabled. the original render distances are so low that the one in
caspad is impossible to see in normal gameplay.
- `fort-entry-gate-11` and `com-airlock-outer-13` are specifically
banned from the all actors hack because they are placed in a bad spot
and Naughty Dog did not program the airlocks very well.
- fixed NPC talk distance being bad for 1 frame.
- fix `sew-scare-grunt` erroneously keeping its spool anim active if you
killed the enemy.
I sorted the list of sources on my SDL PR to reduce future merge
conflicts, but in the meantime everytime something gets added to it I
have a pretty rough set of conflicts to resolve. Committing this early
to preserve my sanity
Saves 16 bits and lets us align the `color_index` field properly.
This shouldn't improve or decrease performance by any noticeable amount
except maybe in really low end systems.
Adds sprite distort, fixes buggy sprite rendering in progress, adds
scissoring support (used in various scrolling menus) and a very basic
implementation of `blit-displays`. This is enough to make the fade
effect in the progress menu work, along with all the menus working
properly without needing to use the REPL. This does not make screen
flipping and the filter when failing a mission work.
Added support in the decompiler for detecting `dma-buffer-add-gs-set`
and `dma-buffer-add-gs-set-flusha` and updated all of the Jak 2 code to
use it. Readability improved!
Fixes decompiler issue with `with-dma-buffer-add-bucket` not inlining
forms which broke syntax. Fixes store error warnings showing up for
non-existent stores, there is now a dedicated pass for this at the end.
I started work on making `BITBLTBUF` stuff work in the DirectRenderer,
but stopped for now because it wasn't strictly necessary. It will still
assert like before.
I'm not sure what has changed here...but rebooting in debug mode now
hangs.
- It seems to exit the GFX loop prematurely because `MasterExit` is
still set to `RESTART_IN_DEBUG`
https://github.com/open-goal/jak-project/blob/master/game/runtime.cpp#L431
- And then it gets stuck waiting for the DECI thread to close
This works, but im not sure if it's the right fix / what has changed to
require this.
This fixes a long time issue with `lambda`. The `lambda` is a bit
overloaded in OpenGOAL: it's used in the implementation of `let`, and
also to define local anonymous functions.
```
(defmacro let (bindings &rest body)
`((lambda :inline #t ,(apply first bindings) ,@body)
,@(apply second bindings)))
```
```
(defmacro defun (name bindings &rest body)
(let ((docstring ""))
(when (and (> (length body) 1) (string? (first body)))
(set! docstring (first body))
(set! body (cdr body)))
`(define ,name ,docstring (lambda :name ,name ,bindings ,@body))))
```
In the first case of a `let`, a `return` from inside the `let` should
return from the functioning containing the `let`, not the scope of the
`lambda`. In the second case, we should return from the lambda. The way
we told the different between these cases was if the `lambda` was used
"immeidately", in the head of an expression (like it would be for the
`let` macro). But, this falsely triggers when an anonymous function is
used immediately: eg
```
((lambda () (return #f)))
```
should generate and call a real x86 function that returns immediately.
This should fix some death/mission failed stuff in jak 2.
Somehow this was only causing issues (afaik) with
`draw-decoration-load-save` getting corrupted, perhaps because other
processes either use the shared dram stack or the gigantic spr stack.
Slight change to float divide operations (again). Now it only turns into
inverse multiplication if the float is a power of 2 (positive or
negative). Non-zero immediate divisors will be compiled as regular float
divisions but will forgo the extra branches and checks for divide by
zero.
Also fixes#2584
Allows for removing a line in the subtitle editor finally, also fixed
the following:
- The path the editor used didn't include the game name
- Loading the subtitle project is too slow to do on startup now
(4-5seconds in a debug build), do it on demand now
Updates the decompiler for the new format and there's new macros. This
new format should be easier to read/parse.
Also rewrote `sp-init-fields!` (both jak 1 and 2) from assembly to GOAL.
Hopefully I did not miss any regressions in Jak 1/2 while updating the
files, it's a lot.
- [x] compare NTSC-K
- [x] compare NTSC-J
- [x] compare PAL
- [x] figure out version order
- [x] ~~write delta patch for spanish text~~ no need for now
Fixes#2530
Fixes issue where warp effect looks wrong near the edge of the screen -
there was an unhandled `REGION_CLAMP` texture setting.
Fixes a potential bug where "warp page" things wouldn't be drawn at all
because there is no PC warp bucket. Unclear if anything actually fits
this category, but it doesn't hurt.
Turn on PC-format texture uploads for the water page so the precursor
guy uses the right texture. It has to use generic because it abuses the
generic death query thing to spawn particles.
Workaround for some issues with rebuilding level files after changing
engine files. Not a perfect solution, but probably good enough.
Doesn't actually do anything in Jak 2 because the collide mesh isn't
extracted, but the functionality is all there. Also updated the renderer
a bit to keep the colors more readable.
Fixes crashes when killing the big spider, killing predator metal heads,
and watching the end cutscene. The presursor guy is the wrong color, but
I don't have time to look into this today.
Fixes citizens that disappear (permanently) when being pushed into a
"wall" which is the border of a nav mesh that is exactly aligned with
coordinate axes.
This bug feels very similar to punch glitch, where code worked on PS2
only because some value didn't reach exactly 1.0 or 0.0 with ps2-style
floating rounding. It might be better to track down the source of the
"only 1.0 on PC" value, rather than patching downstream code to handle
it, but I can't easily find it in this case - there's a lot of code that
touches this heading value. (it's also possible this bug happens on ps2,
but the result is the guard appears to face the wrong direction, rather
than disappearing).
Detailed debug notes:
Ran a process that just calls this function:
```
(defun print-guard-info ()
(let ((proc (process-by-name "crimson-guard-level-42" *active-pool*)))
(when proc
(format *stdcon* "PROC: ~A~%" (-> proc state))
(let* ((pd (the process-drawable proc))
(css (-> pd node-list))
(cs (-> css data 15))
)
(format *stdcon* "joint: ~A~%" (-> cs joint))
(format *stdcon* "pos: ~`vector`P~%" (-> cs bone position))
)
)
)
)
```
it prints out the state, and the bone position for some bone that
happens to be on the upper body. It goes to NaN when the upper half
disappears, in the state `tazer-hostile`.
Modified the code in this state to call this function in a bunch of
places:
```
(defun guard-nan-debug ((guard process-drawable) (info string))
(when (string= (-> guard name) "crimson-guard-level-42")
(format 0 "[guard-nan] ~S : ~`vector`P~%" info (-> guard node-list data 15 bone position))
)
)
```
which prints the bone position to stdout.
This shows that the problem happens after `post`, but before `trans`:
```
[guard-nan] post-end : #<vector 2723306.0000 269921.0312 388825.2187 1.0000 @ #x1f1350>
[guard-nan] trans-start : #<vector NaN NaN NaN NaN @ #x1f1350>
```
this is probably as part of the bone math.
To check, I added some prints to `execute-math-engine`, before and after
the call to `do-joint-math`:
```
[guard-nan-math] pre-math : #<vector 2722236.5000 268609.5312 385339.9062 1.0000 @ #x1f1350>
[guard-nan-math] post-math : #<vector NaN NaN NaN NaN @ #x1f1350>
```
The first part of `do-joint-math` is calling the
`generate-frame-function`, which does animation blending to compute a
bunch of joint transforms. I dumped these:
```
(let ((jaf (the-as joint-anim-frame (+ 2400 (scratchpad-object int)))))
(format 0 "generate frame:~%")
(format 0 "~`matrix`I~%" (-> jaf matrices 0))
(format 0 "~`matrix`I~%" (-> jaf matrices 1))
(dotimes (i (-> obj mgeo num-joints))
(format 0 "~`transformq`P~%" (-> jaf data i))
)
)
```
and confirmed that they always look good.
The next part is "prebind", which allows something to modify the
`joint-anim-frame`. Nothing does this on the guard.
The next part is what I've named the "root bind", which computes the
transform of the root bone based on the process-drawable root's
position.
```
(when (the-as (function cspace transformq none) t9-3)
(when *djm-debug* (format 0 "djm: first bind func ptrs #x~X #x~X~%" cspace<-transformq! t9-3))
(when *djm-debug* (format 0 "djm: first bind func input: ~`transformq`P~%" (-> v1-20 param1)))
((the-as (function object object object none) t9-3) v1-20 (-> v1-20 param1) (-> v1-20 param2))
(when *djm-debug* (format 0 "djm: first bind func result:~%~`matrix`I~%" (-> v1-20 bone transform)))
)
```
in this case, the first print confirms that we're using
`cspace<-transformq!` as the root binding function.
When the guard entirely disappears, it is caused by the root of the
process drawable having NaNs as its `quat`:
```
djm: first bind func input: #<transformq @ #x1f04f0
trans:2724817.7500 262144.0000 388891.1875 1.0000
quat: NaN NaN NaN NaN
scale: 1.0000 1.0000 1.0000 1.0000>
djm: first bind func result:
[001f0e70] matrix
[ NaN] [ NaN] [ NaN] [ NaN]
[ NaN] [ NaN] [ NaN] [ NaN]
[ NaN] [ NaN] [ NaN] [ NaN]
[2724817.7500] [ 262144.0000] [ 388891.1875] [ 1.0000]
```
After this, it's possible to get the lower half of the guard to return,
but I believe the real problem is this first `quat` being NaN. Then the
nans probably spread everywhere.
So now we now to look for the `(-> guard root quat)` becoming NaN:
```
(defun guard-nan-debug ((guard process-drawable) (info string))
(when (string= (-> guard name) "crimson-guard-level-42")
(format 0 "[guard-nan] ~S : ~`vector`P~%" info (-> guard root quat))
)
)
```
and we see this happens in post somewhere
```
[guard-nan] post-start : #<vector 0.0000 -1.0000 0.0000 0.0000 @ #x1f0500>
[guard-nan] post-end : #<vector NaN NaN NaN NaN @ #x1f0500>
```
Looking at individual methods
```
[guard-nan] before-142 : #<vector 0.0000 -1.0000 0.0000 0.0000 @ #x1f0500>
heading: #<vector 0.0000 0.0000 -1.0000 0.0000 @ #x3432670>
[guard-nan] before-143 : #<vector NaN NaN NaN NaN @ #x1f0500>
heading: #<vector 0.0000 0.0000 -1.0000 0.0000 @ #x3432670>
```
It looks like the problem is converting this heading to a quaternion.
Adds a decent way to customize the folders the project file expects the
iso data and decompiler data to be in. When you run any version other
than the default, for example Jak 1 PAL, it uses the `gameName`
decompiler config to consume and output it's results.
However the project file will assume `jak1` unless you hard-code it
differently -- basically, it needs to be explicitly told just the
decompiler is told what version to use.
We now have a per-user REPL Config json file, so that can be used to
override the default `jak1` behaviour.
Fixes#1993
Cleans up every `dummy-*` and `TODO-RENAME-*` method up with either
proper names or by renaming them to `[type-name]-method-[method-id]`
similar to Jak 2's `all-types`.
Also fixes the bad format string in `collide-cache` and adds the event
handler hack to Jak 1.
The game boots and runs fine, but I might have missed a PAL patch or
other manual patches here and there, please double-check if possible.
I saw that some new speedrunning-related text was added to the English
language, so I decided to translate them. Meanwhile I also made some
minor adjustments/fixes to existing translations.
BUT I have a question...
Do we really need all this text?
They are displayed in a menu that is called 'NEW INDIVIDUAL LEVEL RUN'.
So adding IL after the level names is kind of redundant in my opinion -
the existing level name text entries could be used instead. And the
reason why I care so much is because it is hard to come up with a
"translation" for IL. But if I am forced to, I will come up with
something.
I will keep this PR a draft until I know what the main jak-project
people think about this "IL" issue. And I will adjust my branch
accordingly.
The last of the missions that had a missing file.
I manually fixed some casting related to a `handle->process`, since this
is the last file...whatever not worth stressing about. But probably an
issue that will crop up in the future.
Co-authored-by: water <[email protected]>
- decompile `neon-baron-part`, which also has the hideout door for some
reason
- improve a few error messages in static data decompilation
- fix bug with disabling fog in merc
Disables the fog hack for Jak 2, where it's not useful and kind of
breaks in most levels which rely on dark vertices that aren't underwater
(e.g. city windows).
Add the vortex renderer. The vortex texture isn't there yet (it uses the
same texture as clouds), so it uses a checkerboard. But the
colors/vertices seem right.
This way a user can take multiple data samples from one/multiple play
sessions quickly. Creates a directory called profile_data that is added
to gitignore to place the data in, and checks to see if a file is there
and if so it creates prof1.json prof2.json prof3.json and so on...
Unsure what hacks or type casts might eventually be needed for the PAL
version, but these changes are enough to get things to extract and run
the game (though the extract phase requires iso_data to be in a jak2_pal
folder, but the build still looks for jak2 in iso_data and
decompiler_out).
- better handling of the `disable-fog` settings for merc, should fix the
spotlights. There's a setting in the merc effect, and also a runtime
flag for the draw-control. I'm not actually sure what reads these, but
the draw-control one is definitely used to disable fog on the
spotlights.
- increase merc draw limit to try to fix the issue about partially drawn
citizens in the city
- remove useless debug prints (it's okay to die in init, and the medium
load buffer size mode is understood now)
Fixes some mistakes with merc draw modes. The glass in the palace level
no longer writes to the depth buffer (it's "water"):
Also fixes the one-frame flickers when level draw orders change. We
might be able to make this more efficient in the future, but this will
at least fix the frame with nothing drawn.
Actors that use types from a level that got unloaded should now get
killed immediately instead of continuing to exist and eventually crash
the game.
NOTE: Will spam the console whenever a bad actor tries to spawn. I would
like to document all instances of where this happens!
This should fix a bunch of texture-related issues by generating a table
of overlapping textures and just... adjusting them slightly so they
don't overlap. It's not the most elegant solution in the world, but I
think it's no worse than the existing hard-coded tpage dir stuff.
Definitely needs a clean up pass, but I think the functionality is very
close.
There's a few "hacks" still:
- I am using the emerc logic for environment mapping, which doesn't care
about the length of the normals. I can't figure out how the normal
scaling worked in etie. I want to do a little bit more experimentation
with this before merging.
- There is some part about adgifs for TIE and ETIE that I don't
understand. The clearly correct behavior of TIE/ETIE is that the alpha
settings from the adgif shader are overwritten by the settings from the
renderer. But I can't figure out how this happens in all cases.
- Fade out is completely disabled. I think this is fine because the
performance difference isn't bad. But if you are comparing screenshots
with PCSX2, it will make things look a tiny bit different.
There are *a lot* of file changes and while I have carefully gone
through every gsrc change to fix up manual patches, there might still be
spots that I missed.
My mistake -- testing focused too much on preserving the existing
behaviour I clearly forgot to make sure the new stuff worked properly.
Just had to early out and not modify the args if they were in the new
format.
This fixes the issue where elevators leave you behind (stack structure
related)
At the same time I cleaned up some things I found along the way, most
notably changing `script-context`'s `basic` field to `object` as
non-basics use this method as well.
Fixes#2305
An attempt to cleanup the last CLI interface we have left to cleanup.
- `gk` args now follow the typical convention ie. `--proj-path` instead
of `-proj-path`.
- args that are passed through to the rest of the application / the
game's runtime use the typical convention of following a `--`
- I'm thinking some args shouldn't be handled at this level ie
(`-nodisplay`, `-vm`, `-novm` or `-jak2`) These could be better
documented as legitimate flags and passed in via a nice struct. They
don't seem to be used in `InitParams` but I'll triple check.
There's a temporary shim here so there is no coupled release with the
launcher (right now it executes `gk` with a few args). So I just change
the old args into the new format. After one release cycle, I can change
it in the launcher and delete it here.
I am unsure if this will break the bash shellscript usages -- not sure
which args were usually passed into `$@`
Three main changes:
- Adds support for the texture scrolling effect used on conveyor belts,
and turn it on for jak 2.
- Use merc instead of generic in jak 1 for ripple/water/texscroll stuff
(non-ocean water, lava, dark eco, etc). This is a pretty big speedup in
a lot of places.
- Fix a really old bug with blending mode used to draw environment maps.
The effect is that envmaps were half as bright as they should have been.
As usual, there's a flag to go back to the old behavior on jak 1. Set
these to `#t` to use generic like we used to.
```
*texscroll-force-generic*
*ripple-force-generic*
```
The format has changed, and everything must be rebuilt (C++, FR3's, GOAL
code)
Reasons for doing so include:
1. This should stop the confusion around editing the wrong config file's
flags -- when for example, extracting a level. Common settings can be in
one central place, with bespoke overrides being provided for each
version
2. Less verbose way of supporting multiple game versions. You don't have
to duplicate the entire `type_casts` file for example, just add or
override the json objects required.
3. Makes the folder structure consistent, Jak 1's `all-types` is now in
a `jak1` folder, etc.
Support rendering eyes with merc for both jak 1 and jak 2.
For jak 1, everything should look the same, but merc will be used to
draw eyes. This means that jak is now fully drawn with merc!
For jak 2, eyes should draw, but there are still a few issues:
- the tbp/clut ptr trick is used a lot for these eye textures, so
there's a lot that use the wrong texture
- I had to enable a bunch more "texture uploads" (basically emulating
the ps2 texture system) in order to get the eyes to upload. It would be
much better if the eye renderer could somehow grab the texture from the
merc model info, skipping the vram addressing stuff entirely. I plan to
return to this.
- I disabled some sky draws in `sky-tng`. After turning on pris2
uploads, the sky flashed in a really annoying way.
This gets rid of the `auto-lt` and `auto-dbg` CLI args to `goalc` that
have been marked deprecated for a month or so at this point. The
behaviour can be replicated via the `startup.gc` file.
I also removed `decompiler_out/` and `log/` from the repo, as our C++
code is smart enough to create these directories if they are missing now
so they are superfluous
Work in progress minimap. Known issues:
- "path finding" doesn't appear to work - it gets stuck forever in many
cases
- some nasty patches around timer-based code
- jak arrow blending issues
- would be nice to make it higher resolution
if the search is forced to terminate due to iteration/time limits, the
icon is not in the right place
I didn't actually visually notice much of a difference with these hacks
unlike in Jak 1, but I also avoided checking the missions thoroughly
since the game crashes very often right now.
Some backtraces are quite large, an option is to increase your terminal
buffer -- but dumping to a file is also useful if you want to share the
crash.
I'm not crazy about the way I hacked this in, but it felt like the least
invasive way for now and I don't want to cause a regression with the
debugger. It's also nice that it dumps with ansi colors as then you can
view the backtrace with the original coloring:
Usage:
```clj
(:di "./stacktrace.log")
```
Fix crash when the game tries to spawn `jellyfish` in Underport and
temporarily disable the rendering of the warp effect until it's properly
implemented.
- Disable depth test for sprite glow (VU1 programs only `sq.xy` and
leaves `z` alone, which has Z_MAX from the template)
- Fix tfrag/tie/shrub issues when `use-camera-other` is set.
Where applicable, of course.
My system language is set to English so I actually can't test this. If
anyone has their Windows language (NOT LOCALE) set to Spanish, German,
French, Italian or Japanese please test this.
Fixes#1734
Also fixes the opengoal debugger on Windows and fixes the decomp for
`menu` which was causing some crashes related to input handling.
`*alert-level-settings*` was actually an inline array and we were
indexing past the end of it.
I think this will fix some of the strange on-foot guard behavior too.
Increase level heaps and borrow heaps. The level heap increase was
likely not needed, but better safe than sorry. We allocate the 128 MB
main heap anyway so there's no harm.
Also fix the crash when using `-boot`. As I thought it was just a
one-line typo in the kernel.
Fixes the aspect ratio of the font renderer (Jak 2 does not need the
same hacks as Jak 1), the custom letterbox/pillarbox sizes and fixes the
heap display.
---------
Co-authored-by: water <[email protected]>
Adds the `pckernel` system to Jak 2, allowing you to do the PC-specific
things that Jak 1 lets you do like change game resolution, etc.
In other to reduce the amount of code duplication for something that
we're gonna be changing a lot over time, I split it into a few more code
files. In this new system, `pckernel-h.gc`, `pckernel-common.gc`
(previously `pckernel.gc`) and `pc-debug-common.gc` are the files that
should be shared across all games (I hacked the Jak 2 project to pull
these files from the Jak 1 folder), while `pckernel-impl.gc`,
`pckernel.gc` and `pc-debug-methods.gc` are their respective
game-specific counterparts that should be loaded after. I'm not fully
happy with this, I think it's slightly messy, but it cleanly separates
code that should be game-specific and not accidentally copied around and
code that should be the same for all games anyway.
Stops the REPL crashing with `device or resource busy` when running
`(reload)`
However I think this is indicative of a bigger problem where either the
`Compiler` or prompt is not ready to handle input immediately after
creation and setting the status to `OK`.
Ports some of the custom PC port debugging code from Jak 1, such as the
entity debugger, Anim Tester X and part tester, to Jak 2 and gives the
PC menu code the same treatment Jak 1 received recently in #2216.
Some adjustments have been made to make things work in Jak 2 and some
stuff is still incomplete.
The majority of the default PC menu code has been left commented out in
preparation for @ManDude's plan to port the rest of the Jak 1 `pckernel`
features to Jak 2.
Also adds the code for `palace-scenes` to gsrc as that seems to have
been missed.
Fixes a rare crash that can sometimes happen when civilians are picking
a direction to flee in, where a branch count may be 0, causing a divide
by 0 MATH_EXCEPTION.
This automatically generates documentation from goal_src docstrings,
think doxygen/java-docs/rust docs/etc. It mostly supports everything
already, but here are the following things that aren't yet complete:
- file descriptions
- high-level documentation to go along with this (think pure markdown
docs describing overall systems that would be co-located in goal_src for
organizational purposes)
- enums
- states
- std-lib functions (all have empty strings right now for docs anyway)
The job of the new `gen-docs` function is solely to generate a bunch of
JSON data which should give you everything you need to generate some
decent documentation (outputting markdown/html/pdf/etc). It is not it's
responsibility to do that nice formatting -- this is by design to
intentionally delegate that responsibility elsewhere. Side-note, this is
about 12-15MB of minified json for jak 2 so far :)
In our normal "goal_src has changed" action -- we will generate this
data, and the website can download it -- use the information to generate
the documentation at build time -- and it will be included in the site.
Likewise, if we wanted to include docs along with releases for offline
viewing, we could do so in a similar fashion (just write a formatting
script to generate said documentation).
Lastly this work somewhat paves the way for doing more interesting
things in the LSP like:
- whats the docstring for this symbol?
- autocompleting function arguments
- type checking function arguments
- where is this symbol defined?
- etc
Fixes#2215
Some people have complained about the button combo for entering the
speedrun menu, saying it's awkward to press Start.
This change lets you press Select instead (while holding L1+R1+X). The
old combo (pressing Start) still works as well.
Moves PC-specific entity and debug menu things to `entity-debug.gc` and
`default-menu-pc.gc` respectively and makes `(declare-file (debug))`
work as it should (no need to wrap the entire file in `(when
*debug-segment*` now!).
Also changes the DGO descriptor format so that it's less verbose. It
might break custom levels, but the format change is very simple so it
should not be difficult for anyone to update to the new format. Sadly,
you lose the completely useless ability to use DGO object names that
don't match the source file name. The horror!
I've also gone ahead and expanded the force envmap option to also force
the ripple effect to be active. I did not notice any performance or
visual drawbacks from this. Gets rid of some distracting LOD and some
water pools appearing super flat (and pitch back for dark eco).
Fixes#1424
Manual patches:
`(trans idle krew-boss)`: `collide-shape` array created on the stack
changed to `(pointer collide-shape)`
---------
Co-authored-by: water <[email protected]>
Add a debug window for the loader to show levels and fix an issue with
jak 2 level unloading. I never really thought about how this works for >
2 levels, and there is a chance that you could unload the wrong level in
some cases.
The problem is some combination of merc-only levels not counting toward
the "in use" detection, and the unloader ignoring what the game wants to
load.
I don't remember why using merc models doesn't contribute to "in use"
but I'm afraid to change this for jak 1. Instead, we can have the
unloader avoid unloading levels that the game is telling us are loaded.
This is what we should have done originally, but there was a time when
Jak 1 didn't tell the loader anything, and we had this stupid detection
thing.
I think this is the first step toward just getting rid of the "in use"
detection and trusting the game for everything.
Implements the jak 2 lightning renderer as an alternate path through
Generic2. Also set up some generic stuff in the goal code.
There is a problem with the texture pool, which doesn't support the case
where two textures have the same tbp, but different cluts. So lightning
is often the wrong color (usually red).
The direct commit option doesn't work as there is no way to bypass
required status checks -- this makes the merge queue less useful. Switch
to a PR instead.
I did a pass through all missions, fixing issues as they came up. Also
got `seal-at-waterslums` working -- the best mission in the game 🥳
Co-authored-by: water <[email protected]>
Manual patches:
- `drill-turret`: The static data for `*turret-13-path*`,
`*turret-14-path*` and `*turret-15-path*` was decompiled by hand and the
integers in the `set-speed-mult` events have been replaced with boxed
integer arrays that contain only that integer in order to make the
compiler happy. To that effect, the event handler in `target-turret` was
changed to access that array instead of just accessing the int.
- `hover-nav-control`: In `hover-nav-control::10`, `arg2` is usually a
`vector`, but there are some places where it is called with `#t` as
`arg2` and, subsequently, crashes the game because it tries to access
the `quad` of `arg2` if `arg2` is truthy. To mitigate this, the
condition `arg2` has been replaced with `(and (!= arg2 #t) arg2)` (in
this case, it would jump to the `else` that just resets the `dest-vel`
and `transv` `quad`s)
- `drill-baron`: The static data for `*drill-ship-turret-speed-event*`
has been decompiled by hand.
TODOs:
- Jellyfish crash the game
- Destroying the metalhead eggs that are on the breakable wall crashes
the game (already happened with the Peacemaker before)
- Figure out why static data of type `turret-path-event` doesn't
decompile
The docs for all the hover-nav and nav-network code could use some love
in the future, I'm not smart enough to figure out what any of that code
actually means, but it seems to work...
Also threw in the fix for the ▲ that was accidentally left commented
out.
Noticed this has been failing for a while for reasons. I think I fixed
them but can't re-run the current job (out of date from master). This
this gives me a way to manually re-run it.
No more ghost town!
Manual patches:
- `hal3-course`: In `(anon-function 17 hal3-course)`, the decompiler is
doing something strange with `s3-0` that fails to compile, so I just
removed the `set!` and inlined the condition in the `when`.
- `guard`: Rewrote `(.mula.s)` stuff
This is a WIP while I'm learning the ins and outs of decompilation, but
putting up what I have for 2 reasons:
- Hoping someone can double check I'm on the right path (all functions
have signatures, all reasonably safe guesses for types have been put in,
using "object" for type where they're not)
- Might be blocked by not being able to run the offline-tests as a PAL
scrub
I'm going to look at what might be involved in making tests work for
PAL, but wouldn't be surprised if I have to wait to get a black label
version and come back to this after :(
---------
Co-authored-by: Tyler Wilding <[email protected]>
This will probably take a while, since we also have to translate all the
text of the base game - Naughty Dog never translated this game to
Hungarian. This PR will stay a draft until it is complete.
We realized that every letter in our alphabet was already working, apart
from two: Ő and Ű (they are unique sounds, so leaving their marks
wouldn't be okay).
Since I did not find that double accent thing in the jak font, I decided
to use ~ (see my change in FontUtils.cpp). It is good enough, and my
memory tells me that I already saw this exact same "solution"
(workaround) somewhere in the past. If anyone knows a better solution,
please let us know.
We chose ID 14 for the Hungarian language, as it was the lowest free ID.
**Progress tracker**
A tick here means that everything was translated. It does not mean that
everything is perfect yet. We will review everything multiple times, to
have the best translations possible.
Game text:
- [x] base game text
- [x] pc port text
- [ ] credits text ?
[Sziloyoo](https://github.com/Sziloyoo) helped with reviewing my
changes, and gave advice/suggestions for some complicated translations.
Subtitles will be done in the future, not in this PR.
This PR adds a feature to merc2 to update vertices. This will be needed
to efficient do effects like blerc/ripple/texture scroll. It's enabled
for blerc in jak 1 and jak 2, but with a few disclaimers:
- currently we still use the mips2c blerc implementation, which is slow
and has some "jittering" because of integer precision. When porting to
PC, there was an additional synchronization problem because blerc
overwrites the merc data as its being read by the renderers. I _think_
this wasn't an issue on PS2 because the blerc dma is higher priority
than the VIF1 DMA, but I'm not certain. Either way, I had to add a mutex
for this on PC to avoid very slight flickering/gaps. This isn't ideal
for performance, but still beats generic by a significant amount in
every place I tested. If you see merc taking 2ms to draw, it is likely
because it is stuck waiting on blerc to finish. This will go away once
blerc itself is ported to C++.
- in jak 1, we end up using generic in some cases where we could use
merc. In particular maia in village3 hut. This will be fixed later once
we can use merc in more places. I don't want to mess with the
merc/generic selection logic when we're hopefully going to get rid of it
soon.
- There is no support for ripple or texture scroll. These use generic on
jak 1, and remain broken on jak 2.
- Like with `emerc`, jak 1 has a toggle to go back to the old behavior
`*blerc-hack*`.
- In most cases, toggling this causes no visual differences. One
exception is Gol's teeth. I believe this is caused by texture coordinate
rounding issues, where generic has an additional float -> int -> float
compared to PC merc. It is very hard to notice so I'm not going to worry
about it.
When I cleaned up the `game.gp` some DGOs were no longer referenced
because my first dependency script omitted them -- thinking they weren't
required. From the perspective of the source files they indeed weren't
required but we still have to produce the DGO file.
also works around #2177
This adds environment mapping support to `Merc2`, and turns it on for
Jak 1 and Jak 2.
- The performance is much better
- Jak 1 can be toggled back to the old behavior with `(set! *emerc-hack*
#f)`. The new environment mapping is identical to the old one everywhere
I checked.
- Jak 1 still falls back to generic for ripple/texscroll/blerc/eyes -
there's still no dynamic texture or vertex updating support. The eye
detection stuff will sometimes flag stuff as eyes which is not eyes,
which is fine, but means that generic will be used in some places where
emerc could be used. For example, the shiny plates on jak's arm will be
drawn with generic because jak has eyes.
- Jak 2 hasn't been checked super carefully against PCSX2 yet.
- Jak 2 still isn't technically using emerc, but instead putting emerc
models in the merc bucket.
- The interface to merc is a lot different now and totally custom
OpenGOAL DMA code. The original merc drawing asm doesn't run anymore.
- The FR3 format changed
- Something funky going on with foreground lighting in escape, but
doesn't seem to be related to this change?
Performance comparison, jak 1, in likely the most generic-merc heavy
spot:
Some more progress on vehicle code, the following files still remain:
- vehicle-guard (annoying stack types)
- traffic-engine (many issues, some which already have issues made for
them, other entirely new confusing things!)
Initial implementation of the `ocean-mid`, `ocean-far` and `ocean-near`
renderers for Jak 2.
There's still a few things to sort out, mainly:
- [x] ~Backwards compatibility with Jak 1. The only thing that currently
stands in the way of that is figuring out a clean way to "un-hardcode"
the texture base pointer in C++ without creating a completely separate
`OceanTexture` class for Jak 2. One thing I thought of would be
modifying `BucketRenderer`'s virtual `init_textures` method to also pass
the `GameVersion`, but I'm not sure if there's a better way.~
- [x] ~The sudden transition from `ocean-near` to `ocean-mid`. Not sure
why it's happening or how to fix it.~
- [x] The ocean has two new methods in Jak 2, `ocean::89` and
`ocean::79`, one of which seems to be related to time of day sky colors.
~Even without them implemented, the end result looks quite close, so we
may be able to skip them?~ `ocean::89` generates `ocean-mid` envmap
textures, so it will likely be required, but will not be handled right
now.
Reverted the VU prologue removals because it made the tests fail.
Some side missions require cars, they don't work yet. Also the
ring-races and collection ones do not grant orbs. The hoaming beacon
collection one causes a `hud` crash
draft because using the array is a little weird still, don't feel like
dealing with window's slow debugging builds today.
I get the following weird error:
```clj
(define test-array (new 'static 'boxed-array :type type vector))
gr> (-> test-array 0)
1538004 #x1777d4 0.0000 vector
gr> (type? (-> test-array 0) type)
1342757 #x147d25 0.0000 #t
gr> (new 'static (-> test-array 0))
-- Compilation Error! --
Got 3 arguments, but expected 2
Form:
(-> test-array 0)
Location:
Program string:1
(new 'static (-> test-array 0))
^
Code:
(new 'static (-> test-array 0))
```
Maybe this is expected though and the `new` method wants a symbol, not a
type?
Fixes#2060
Co-authored-by: water <[email protected]>
Adding support for better child-type method docstrings. This is a
problem unique to methods.
Usually, a child-type will have the same signature and a common name
will apply, but the implementation is different. This means, you
probably want a different docstring to describe what is happening.
Currently this is possible to do via `:replace`. The problem with
replace is two fold:
- a replaced method ends up in the generated `deftype`...because you
usually change the signature!
- we don't put docstrings in the `deftype` in normal GOAL, this is just
something we do for the `all-types` file (they go in the `defmethod`
instead)
- more importantly, this means anytime you now want to change the
parent's name/args/return type -- you have to apply that change
everywhere.
So this is a better design you can now just declare the method like so:
```clj
(:override-doc "my new docstring" <method_id>)
```
And internally a pseudo-replaced method will be added, but it will
inherit everything from the parent (except the docstring of course)
Unrelated - I also made all the keyword args for declaring methods not
depend on ordering
This also adds support for documenting virtual and non-virtual state
handlers. For example:
```clj
(:states
(part-tester-idle (:event "test") symbol))
```
or
```clj
(idle () _type_ :state (:event "test") 20)
```
I will probably add the ability to give some sort of over-view docstring
at a later date.
Co-authored-by: water <[email protected]>
- started documenting the files I glossed over, some are totally done,
others are just partially done
- I changed the decompiler to automatically initialize the
art-group-info from the json file. This makes updating gsrc, even a
single file at a time, have consistent naming
- Though I disabled this functionality for jak 1, as I have no idea if
using the ntsc art groups will cause a regression for different versions
- fix indentation for docstrings -- it still doesn't look great, but
this is now a formatting concern, rather than the docstring having a
bunch of happen-stance leading whitespace.
- make sure bsp is processed on `l` levels in extraction (caused missing
remaps)
- clean up a few prints in extraction
- handle the <15 byte differences in art group files automatically (no
more errors about file naming)
- fix potential exception thrown by merc2 in a few ways: fixed bad data
in FR3's, check texture index just in case, and handle exceptions a
little bit better (still a crash, but at least you get a print)
- fix mips2 ocean stuff causing ocean far crashes
`whack` is missing a single function (`birth-func-whack-score`) that I
couldn't figure out the right args for.
I was not able to test any of this code because I cannot get `atoll` to
extract.
Manual patches in:
- `tomb-boulder`: a few `ppointer->handle`s (I think) were not being
decompiled properly (all used in combination with `clone-anim-once`)
- `ruins-obs`, `pillar-collapse`: `art-joint-anim` casts
- `tomb-beetle`: commented out a call to `shadow-control-method-14` that
was crashing the game when spawning the beetles
- `grunt-mech`: commented out `(.mula.s)` instruction
Notes:
- `enemy-info`'s `idle-anim-script` is most likely a `(pointer
idle-control-frame)`, however, some `nav-enemy-info` labels set it to
`#f` (first encountered in `tomb-beetle`, but also present in `hal`,
`roboguard` and `metalkor-setup`), which crashes the decompiler. This
may become a problem in the future when we eventually get to these
files. For this PR, I made `tomb-beetle` decompile with
`idle-anim-script` set to `#f` and have not noticed any issues/crashes
with that.
- `tomb-boulder` compiles and doesn't crash, but when trying to play the
Daxter chase sequence, the boulder sometimes either spawns at the origin
or spawns in the correct place, but doesn't move.
Co-authored-by: water <[email protected]>
- worked around audio code for gungame, the tutorials now function
- auto-exit cutscenes upon entry. This doesn't always work nicely (i
didnt want to cause any side-effects due to messing with load states)
but it atleast stops jak from being stuck forever waiting for the
cutscene to load
Effects the following files:
- [x] vehicle-rider
- [x] vehicle-control
- [x] vehicle-effects
- [x] vehicle
~~- [ ] vehicle-util~~
- [x] vehicle-physics
- [x] vehicle-states
~~- [ ] vehicle-guard~~
~~- [ ] traffic-engine~~
~~- [ ] traffic-manager~~
With the exception of traffic-engine, most of these files are either
done or have 1-3 stubborn functions remaining. Draft while I try to
resolve as many as possible / cleanup names and such.
Co-authored-by: water <[email protected]>
Adds the following files:
- `amphibian`
- `centurion`
- `ginsu`
- `grenadier`
- `hopper`
- `metalmonk`
- `monster-frog`
- `predator-graph`
- `predator-h`
- `predator`
- `rapid-gunner`
- `rhino`
- `rhino-wall`
- `tomb-baby-spider`
Also adds the DGOs for the following levels:
- Mar's Tomb
- Mountain Temple
- Drill Platform
- Sacred Site (Sage Hut)
Manual patches:
- The decompiler emits `(b! #t cfg-17)` in `(trans hostile hopper)`
without putting a `(label cfg-17)` anywhere
- Added cast to `art-joint-anim` in `(code broken rhino-wall)` and
`(code hit rhino-wall)`
Other notes:
- `amphibian` seems to occasionally crash when using its tongue attack.
Haven't investigated this yet
- `ginsu` crashes after being killed somewhere in the `deactivate`
method, possibly because of its `part-spawner`
- Predators aren't spawning in the forest hunt mission, not sure if
`forest-obs` might be needed for that or if it's something else
- The `rhino-wall` STR animation seems to load (albeit a bit broken due
to missing VAG stream playback), but causes the viewport to shrink as
soon as it's played
- I added `pegasus::74` to the `event_handler_hack` bool in
`variable_naming.cpp` because I got a `none` cast after changing the
return value of `enemy::74`
- lets you split up your `startup.gc` file into two sections
- one that runs on initial startup / reloads
- the other that runs when you listen to a target
- allows for customization of the keybinds added a month or so ago
- removes a useless flag (--startup-cmd) and marks others for
deprecation.
- added another help prompt that lists all the keybinds and what they do
Co-authored-by: water <[email protected]>
Notable things:
- This assert is hit when trying to save the pc-settings file, NYI
https://github.com/open-goal/jak-project/blob/e630b506903d7f7028dff89702be7f586c2120e7/game/kernel/common/Symbol4.h#L14
so right now settings aren't persisted. But RPC defaults to on
- The existing functions can probably be made generic based off the game
version, but I didn't spend time refactoring them yet as they aren't
really ready to be used in jak 2 yet (we have no screenshots for the
levels for example)
The things that aren't working right now:
- returning the board (maybe cutscene related?)
- draft until i see if this can be sorted out
- seems to need streaming audio as well, the `scene-player` gets
properly initialized from what i can tell
- had to hack around the streaming audio parts.
Everything relating to IK in `target-mech` was commented out as I
couldn't get a lambda to decompile properly, but the mech still seems to
work fine without it.
Co-authored-by: water <[email protected]>
Two main problems fixed here:
- offline tests will fail on a comparison failure (a mistake from the
re-write)
- art-group-info is committed to the repo and shared with every thread
(running the tests with 1 thread, for example on the CI, and locally
were producing different results)
art files are still not provided to the jak2 offline tests:
- `*-ag` files are not being output
- `art-elts.gc` is not complete, as a handful of files claim to be
missing stuff
lastly, in jak1's offline tests we were also running `tpage` and `*-vis`
files through the decompiler. This omits that (they came from the
`all_objs.json` file) -- is this an issue?
Fixes:
- a very old bug in depth in DirectRenderer, probably from the original
tfrag stuff. Looked at PCSX2 source code to see how 32/24 bit depths are
handled. This fixes hud sprites being drawn behind level geometry.
- saturate `vftoi4` like the ps2 does when the float is too large,
fixing hidden text in `hud`. For now it's only using this in the font
code because this saturation is actually kinda slow and hasn't been a
problem in other places.
- fix crazy particle spawning issue with blue gun and dripping stuff.
This would happen if particles kill themselves while being processed
(through a callback)
This adds a new ImGUI menu to help filter out the clutter on screen.
https://user-images.githubusercontent.com/13153231/210192912-b1c28319-bacb-449c-ad7f-e7308fb75f50.mp4
This also:
- moves the imgui display bool into a game specific config file (you can
hide it in jak1, and not in jak2)
- the config file also persists the settings from this menu (except the
filters for now, future TODO)
- there is a new `ignore_imgui_hide_keybind` in this file to ignore
hiding it when you press Alt
It used to automatically assign actor IDs, but reset in between custom
levels, so it was possible for actors to share the same actor id with a
actor in another level. With this the user can now specify a base_id to
make sure that the custom levels never match! (or fix it if they do!)
CreateChildSound did not check for nullptr return from
soundbank::make_handler, which would happen for sounds with no grains.
Prevent further problems with this by switching to optional instead of
passing nullptr.
I did some manual modifications in a few places to work around some
truly mysterious control flow, and some unsupported stack array stuff.
Also fixes a bug in decompiling static improper lists.
As far as I can tell, guns work, other than some graphical issues and
the crazy particle spawning issue, but I strongly suspect these are
problems with the sparticle/graphics side.
A common thing that can be forgotten about / confusing to new people is
that:
1. if you add an object to the `allowed_objs` list but it's also in the
`banned_objs` list -- the ban still takes precedence with no failure
2. if you add an object to the `allowed_objs` list but have not allowed
the DGO in `inputs.jsonc` it will also silently log a failure and
continue.
This PR turns both situations into an explicit error with advice/a
reminder on what to do to fix it.
- Split up DGOs between threads in the multithreaded offline test
- fix some random warnings
- make the sig paths decompile a bit nicer to make some files smaller
When working with mips2c recently, I noticed adding the
`defmethod-mips2c` or `def-mips2c` code was a manual step. This is a bit
tedious to have to go and do yourself, but more importantly you have to
manually go and find the right spot in the source file else you might be
declaring it too early or too late.
This will automatically output the declaration for methods, and a
half-finished comment for the functions. I wasn't able to fully output
the function one because it seems the signature info from `all-types`
doesn't make it all the way through -- but maybe I'm wrong or this is an
easy fix?
Running reference tests/decompiler should now be possible on macos
(arm). Most of the changes were just cleaning up places where we were
sloppy with ifdefs, but there were two interesting ones:
- `Printer.cpp` was updated to not use a recursive function for printing
lists, to avoid stack overflow
- I replaced xxhash with another version of the same library that
supports arm (the one that comes in zstd). The interface is C instead of
C++ but it's not bad to use. I confirmed that the extractor succeeds on
jak 1 iso so it looks like this gives us the same results as the old
library.
- Stub `target-board-handler` to prevent jetboard crash
- Increase stack size for `target` to fix dark bomb crash
- Uncomment some code in `target-darkjak` to make Dark Jak work (texture
morph still crashes, so it's commented out)
- Increase stack size for a `process-spawn-function` in
`target-send-attack` to fix a crash with giant Dark Jak
A big one...
I figure even if we would like to change the way the particle/scene code
is output -- it'd be easier to find patterns with it all decompiled.
I've updated my script so it can easily be used to mass update these
files:
```bash
task update-gsrc-glob GLOB="**/*-part*.gc"
```
> for example will update gsrc files with `part` in their name -- if
they are in ref tests (so uncompleted ones aren't touched)
I found a few issues along the way that I'll have to make issues for
soon.
This solves two main problems:
- the looming threat of running out of memory since every thread would
consume duplicate (and probably not needed) resources
- though I will point out, jak 2's offline tests seem to hardly use any
memory even with 400+ files, duplicated across many threads. Where as
jak 1 does indeed use tons more memory. So I think there is something
going on besides just the source files
- condense the output so it's much easier to see what is happening / how
close the test is to completing.
- one annoying thing about the multiple thread change was errors were
typically buried far in the middle of the output, this fixes that
- refactors the offline test code in general to be a lot more modular
The pretty printing is not enabled by default, run with `-p` or
`--pretty-print` if you want to use it
https://user-images.githubusercontent.com/13153231/205513212-a65c20d4-ce36-44f6-826a-cd475505dbf9.mp4
Supports most of the grain types now while maintaining compatibility
with the old stuff (at least the subset of things jak1 uses)
Would benefit from some testing in Jak 1 to make sure I didn't break
anything.
Sorry the git history is a mess, I'll do something about it later.
- decompile `lights.gc`
- decompile remaining `target-death` function, you can die now
- fix float issue with `matrix-from-two-vectors-smooth!` making jak face
the wrong way in the slide thing
Mostly easy / particle def files. I did a bit in `ruins` but CFG
failures and missing `drawable` functions make it untestable for now so
I've put that on pause.
This allows you to not have to define the entire file path to a source
file to re-compile and load it. Technically a stop-gap until editor
tools are developed around writing OpenGOAL.
As opposed to `(ml "goal_src/jak2/engine/game/main.gc")` (which still
works)
This is accomplished via the following config (connection attempts is
irrelevant):
```json
{
"numConnectToTargetAttempts": 1,
"jak2": {
"asmFileSearchDirs": [
"goal_src/jak2"
]
}
}
```
This also provides a way to make game-specific configurations for the
REPL fairly easily.
- You can define a `startup.gc` in your user folder, each line will be
executed on startup (deprecates the usefulness of some cli flags)
- You can define a `repl-config.json` file to override REPL settings.
Long-term this is a better approach than a bunch of CLI flags as well
- Via this, you can override the amount of time the repl will attempt to
listen for the target
- At the same time, I think i may have found why on Windows it can
sometimes take forever to timeout when the game dies, will dig into this
later
- Added some keybinds for common operations, shown here
https://user-images.githubusercontent.com/13153231/202890278-1ff2bb06-dddf-4bde-9178-aa0883799167.mp4
> builds the game, connects to it, attaches a debugger and continues,
launches it, gets the backtrace, stops the target -- all with only
keybinds.
If you want these keybinds to work inside VSCode's integrated terminal,
you need to add the following to your settings file
```json
"terminal.integrated.commandsToSkipShell": [
"-workbench.action.quickOpen",
"-workbench.action.quickOpenView"
]
```
- target-part
- ;; ERROR: Failed to convert to atomic ops: Variable could not be
constructed from register r0 in `process-drawable-shock-wall-effect`
- gun-part
- missing sparticle decompiling case it seems related to `L155`
This is pretty rough but...im excited to see it working :)
The VU programs for the ocean renderer have changed a bit and
`ocean-texture` has a bunch of new stuff, otherwise things are
relatively similar to Jak 1.
This is the first time I used mips2c and I'm not sure I did it 100%
right, so that should be double-checked.
The only interesting one is `collide-hash`, which is untested.
The other two are very likely unused. I skipped the annoying code in
`collide-probe` because it's not used and the same as jak 1.
My changes for allowing an explicit tag have now been merged to the
original action, so no need to use my fork.
Additionally, buildcache itself reverted to using ubuntu 20.04 to fix
this issue, but we'll leave the explicit tag there anyway to avoid
sudden breakage in the future.
Offline tests ignore comments in their comparison, but there's no reason
to strip them from the file that goes into the reference test folder
when doing the typical update routine.
This just generates superfluous diffs for all the files already done
prior to this change and is meaningless (the lines are dropped anyway)
Couldn't finish any of the enemy/nav-enemy related files for one reason
or another, but quite a bit of work that will be easier to merge and
iterate on instead of keeping track of the branch.
enemy/idle-control has some very weird focus related code.
nav-mesh/nav-control still has a bunch of CFG resolution problems that
need to be manually resolved.
Co-authored-by: water <[email protected]>
- fix issue described in
https://github.com/open-goal/jak-project/issues/1939
- fix `text`, which was manually patched with the wrong offset (was
reading the symbol value off by one byte)
- clean up some random useless prints
- make the offline tests keep trying if there's a comparison error,
clean up the output a bit so the diffs are all at the end.
The offline-tests are going to end up taking too long for jak 2, I did
some rough math and by the end of it we'll be spending almost 2 minutes
for a full offline test on my machine.
These changes allow us to throw hardware at the problem
Still some work to do to make the output nicer, but seems to be fairly
reliable. By default it still uses 1 thread, use `num_threads` to change
this.
Almost done:
- `target-handler` (`(none)` event handler casts and CFG error)
- `target2` (`(none)` event handler casts)
- `powerups` (`cloud-track` does some weird stuff with `handle`s)
- `gun-states` (CFG error)
Some progress in:
- `water-flow`
Additionally:
- Clean up the two year old Jak 3 config file and add a config skeleton
(disassembling seems to not have worked, but I was able to dump obj
files and the `all_scripts` file)
- Fix automatic skelgroup detection and `defskelgroup` macro for Jak 2
(closes#1950)
- When a function decompiles without any major errors, a warning is
generated with the op id for each unresolved load and store that will
likely fail to compile (closes#1933)
Generates the `game.gp` code for all DGOs, code added to the project
file has been commented out for now.
In my opinion, this is way too much in a single file -- it would be nice
if this code would live directly inside the `.gd` files themselves, then
everything is nicely organized. But this approach might have issues I'm
not aware of.
I think this got removed awhile back, not sure if it was intentional,
but thought it was nice to have. Feel free to close if it's not
supported for a good reason 👍
The type makes no sense when compared to the actual data there. This
modifies the type slightly, which I think is better than spending a
bunch of effort to special case this in the compiler/decompiler.
If we try this, and it turns out it doesn't work, we can always revisit
it. But I think this will work.
This PR does a few main things:
- finish decompiling the progress related code
- implemented changes necessary to load the text files end-to-end
- japanese/korean character encodings were not added
- finish more camera code, which is required to spawn the progress menu
/ init the default language settings needed for text
- initialized the camera as well
Still havn't opened the menu as there are a lot of checks around
`*target*` which I havn't yet gone through and attempted to comment out.
For the level builder, I use .bak files whenever I modify a core file in
the project. These let you undo changes, but they don't need to be
pushed. I also couldn't find a nice way to do this, but we should also
ignore all contents of custom_levels besides test-zone.
A few issues:
- lwidea's fr3 is getting loaded and unloaded all the time
- the debug line drawing clipping is wrong (doesn't seem wrong in pcsx2,
so I think this is on us)
- nothing actually using vis data yet
- at a large distance, our view frustum culling seems slightly too
aggressive (might be that viewport scissoring is wrong)
- in the city, things seem darker as you move away. unclear how this is
happening (fog?)
Favors the `lg` namespace over `fmt` directly, as this will output the
logs to a file / has log levels.
I also made assertion errors go to a file, this unfortunately means
importing `lg` and hence `fmt` which was attempted to be avoided before.
But I'm not sure how else to do this aspect without re-inventing the
file logging.
We have a lot of commented out prints as well that we should probably
cleanup at some point / switch them to trace level and default to `info`
level.
I noticed the pattern of disabling debug logs behind some boolean,
something to consider cleaning up in the future -- if our logs were more
structured (knowing where they are coming from) then a lot this
boilerplate could be eliminated.
Closes#1358
And everything else needed for them!
A couple functions are bad currently.
- fixes#1929 - untested on linux
- fixes#1924 - now you need to type `,` before a lambda you want to put
in a pair.
- fix debugger symbol table in jak 2
- made the decompiler output `(meters 2)` instead of `(meters 2.0)`
- fixed a bug with the bitfield enum special -1 case
- made bad game text decomp not exit the decompiler
- added `editable-player` and `script`
eg
```
(set! (-> obj sound-pre-open) (static-sound-spec "airlock-slider"))
```
it is the same as jak 1, but the `fo-curve` option is set to 1
sometimes.
Just missing `target-board-handler` because it errored with `Bad delay
slot in clean_up_cond_no_else_final` and I wasn't able to figure out
where the asm branch was.
Commented out `target-board-clone-anim` because it crashes even after
adding `clone-anim` and `clone-anim-once`.
A small quality of life increase that is more impactful since jak 2 has
double the file count.
I often use the offline-tests to find compiler errors / automatically
resolve them when i am finalizing a file. As more and more files are
completed this becomes increasingly more inefficient. When I know that
only 1 file needs to be decompiled / compared / compiled, I'd prefer to
have a feature like this.
Fixes#1886 which was introduced in #1756
Changed to only read the dpad inputs for hud toggle if L2/R2 were
considered held (they're not read during a pause buffer)
Some notes:
- `draw-prototype-inline-array-shrub` can probably be decompiled but I'm
not comfortable doing this function myself. It's got a ton of inline
assembly (or at least I think it's inline assembly) and is very long. I
left this marked as asm for now but I am confident in the function
definition.
- The VU program is identical to Jak 1 (yay)
- There's some new `dma-test` stuff at the end of the file, not really
sure what it's doing but it runs in the top-level.
- decompile `subdivide`, `wind-work`, `tie-work`, `bsp`, `focus`
- support `ppacb` in compiler
- don't assert when bitfield stuff fails due to constant propgataion
weirdness
- finish up history
- div/mod unsigned assert fix in decompiler
- empty assert fix in decompiler for failed `add` type prop
- make jak 1 performance counters "work" (just measure time)
- fix cast/typos on pcgtb/vftoi15
Fixes a crash when there's a `(new 'static 'something :field <some
anonymous function with a particular set of errors in the decompler>)`.
The output will now be
```
(set! *duck-mods* (new 'static 'surface
:name 'duck
...
:mult-hook <lambda at L615>
:flags (surface-flag duck)
)
```
fixes https://github.com/open-goal/jak-project/issues/1882
Found the reason why this wasn't happening -- the path included by the
decompiler isn't a proper URL encoded URI -- so I have to convert it
appropriately so when vscode sends it's "this file with this URI has
changed" event it actually matches the tracked all-types file.
This will make jump-to-defs not drift as the file changes, and new
uncommented symbols will actually get picked up
Has boxed array accessing that prevents me from adding anything to ref
tests (the entire file is lambdas so the access pattern that i would
like to ignore happens at the top-level, can't ignore it.
This code actually already has quite a bit of original docstrings so
it's not too bad in that regard considering a `script-context` can have
16 arbitrary objects. It seems they rarely put more than a single object
in the context and the types are usually obvious / are actually type
checked!
It was written:
> If you the build directory you'll need…
I tried to fix it, hopefully I found the correct wording. Otherwise
please let me know what was missing.
The merc VU1 program is identical to Jak 1. Jak 2's merc is doing some
new texture login stuff and the DMA buffer setup is a bit different, but
overall very similar.
Fixes https://github.com/open-goal/jak-project/issues/1821 by adding a
special case for `new` method calls where the argument with type
`symbol` is actually an address to uninitialized structure on the stack.
Fixes https://github.com/open-goal/jak-project/issues/1849 (or at least
the cause of the issue Vaser gave in chat, and one random one I found in
`debug-sphere`)
Fixes https://github.com/open-goal/jak-project/issues/1853
Fixes https://github.com/open-goal/jak-project/issues/1857 by moving the
cast into the cond if the body is a single form and the destination type
is a bitfield/enum which is likely to work well. Seems to work on the
examples we could find in jak 1 and jak 2.
Also fixes an issue with casts on the result of `handle->process` (a
common place to use casts)
the output of process->handle is a plain process. Most of the time, you
end up casting this to a more specific. If you add a cast on every use
of the variable, the decompiler will decide to change the type of that
variable to the more specific type, and this breaks the handle cast.
so previously it was impossible to get code like
```
(let* ((s2-0 (the-as swingpole (handle->process (-> self control hack))))
(gp-0 (-> s2-0 dir))
)
```
But now it will work
Rewrites specific kinds of lets where the return value of the `set!`
itself is meant to be used to just be `set!`s. Implemented at the let
rewrite level. Seems to work for Jak 2 so far.
Fixes#1854 .
This shows the current settings needed to help provide information when
verifying a run, also it removes the two calls to show the version in
main.gc and instead does it within the speedrun/auto splitting scope.
- Shows Version
- Shows if Speedrunner mode is on (technically pointless since the text
only displays when it is enabled this more just serves to show that
there IS a speedrunner mode to viewers and stuff (josh) So hopefully it
may cut down on submissions with the mode off)
- Shows if cutscene skips is enabled/disabled.
Tested on 16x9 4x3 the really weird one and borderless/fullscreen
[](https://www.youtube.com/watch?v=qDC-beEg5Es)
Co-authored-by: Tyler Wilding <[email protected]>
I wasn't able to 100% complete `debug` due to a bunch of level boundary
debug stuff I couldn't figure out. But, I added a ref test and
documented/copied over everything else into goal_src.
Most of the functions that existed in Jak 1 are identical, others got an
extra param or 2 and some now make copies of arguments instead of
modifying them. There's a bunch of new functions, including all of the
debug functions that used to be in `level-boundary` are now in `debug`.
At the very end of `debug` there's also some weird asm functions
checking for some EE memory controller bug (not sure what's up with
that)?
* sparticle-launcher
* d/jak2: large amount of `sparticle-launcher` done
* d/jak2: finish the majority of `sparticle`
* decomp: improve format code ignoring
* d/jak2: make bits unique in `sp-cpuinfo-flag`
* d/jak1: revert config change
Still comes up from the bottom, but its probably the "correct" way to do it?
The other alternative would be killing Jak when the cutscene is skipped instead of `(start 'play (get-continue-by-name *game-info* "citadel-elevator"))` This would make the elevator at the top instantly, however it might mess with hp values and stored levels.
Credit to Barg/dallmeyer
* jak1: put common speedrunning code into it's own file
* jak1: enforce `60` fps while in speedrunning mode
* jak1: when speedrunning, display the version until you get the first powercell
* jak1: add an explicit option for skipping cutscenes
* jak1: extend `game-option` to allow any menu option to be disabled
* tests/jak1: allow whitelisting types to be redefined to satisfy typeconsistency checks
* jak1: add file headers
* jak1: cleanup bool checking
* test: delete the es substitle file
* test: add it back
* jak1: missed one cleanup spot related to bool comparisons
* Update default-menu.gc
* make fog hack slightly less aggressive
* add `reload-text` back but kinda fixed
* more unlocks for non-guaranteed musics
* smarter language save & load
* also save pc settings on autosave
* more special music log cases
* one more
* First pass at rigid-body-h
* More rigid-body stuff
* More rigid-body
* A little bit of rigid-body-plat
* Method types for known rigid-body-control proxy methods
* Add back note
* Move rigid body label types to a better spot
* d/jak2: finish `cylinder`
* d/jak2: mostly finish `history`
* d/jak2: finish a bit of `ripple` and `sync-info`
* d/jak2: finish `font-data` and chip away at some other files
* d/jak2: most of `text` done, a bit of `script` - symbol arr issue in `level-info`
* d/jak2: finish `mech-h`
* lint: format
* tests: fix typeconsistency
* d/jak2: address feedback
* Reverted default keyboard camera controls to IJKL and fixed bug where use mouse would always be set as false
* Fixed inverse axis on Right Y axis analog keyboard control
* Fixed newpad unit test, fixed variable typo, and fixed MapAnalog typo in newpad.h
Co-authored-by: animalstyletaco <[email protected]>
* wip
* getting stuff set up so we can actually run test cases
* better handle block entry stuff
* types2 working on gstring
* comments
* math ref working
* up to first stack stuff
* stack fixes
* bounding box
* math stuff is working
* float fixes
* temp debug for (method 9 profile-array)
* stupid stupid bug
* debugging
* everything is broken
* some amount of type stuff works
* bitfield
* texture bitfields not working
* temp
* types
* more stuff
* type check
* temp
* float related fixes for light and res problems
* revisit broken files, fix bugs
* more types
* vector debug
* bug fixes for decompiler crashes in harder functions
* update goal_src
* Added screenshot hotkey and added a toggle to debug GUI to disable screenshot hotkey (#1765)
* Renamed gfx_dumps folder to screenshots and screenshots taken with hotkey now use game render options not debug gui screenshot render options (#1765)
* Updated get current timestamp to use strftime
* fixed clang formating errors for screenshot hotkey
Co-authored-by: sardap <[email protected]>
* decomp: add `docstring` support to relevant places in `all-types`
* decomp: output method docstring into the `defmethod` instead
* goalc: handle docstrings in `define[-extern]` (gracefully ignore for now)
* decomp: output docstrings for bitfield deftypes too
* goalc: fix `defenum` parsing when coming from the compiler (no symbol metadata to store to)
* lsp/tests: fix ups
* lint: formatting
* goalc: handle edge-case of defining a string constant
* cleanup leftovers, fix codacy issues, rename struct
* scripts: remove references to non-existent script
* game: fix transition from fullscreen (not borderless) to windowed
* game: ensure window is position away from the corner of the display when switching from fullscreen
* game: add auto-splitter structs
* game/jak1: wire up auto-splitter updating to game when in SR mode
* game/jak1: add info to split on picking up white eco
* game/jak1: add another two tasks for fish and finalboss
* game/jak1: autosplitting code entirely in GOAL
* split some subtitles
* allow cutscene skipping in retail and fix softlock
* formatting
* make intro skipping work?
* citadel sagecage skipping
* finalboss sage skipping
* quicker quit to title
* uk text fixes
* fisher cutscene skipping
* fix test
* game: restore windowed mode settings properly
* game: use the game version for saving settings/saves
* game: prevent windowed mode from being auto-centered on initial init
* game: save and restore window coordinates
* lint: formatting
* redo cheat encodings
* fix error
* add no texture cheat
* tiny cleanup + add sidekick stats button
* crappy implementation of big/small head mode
* more correct bone scaling
* redo bone manip code a bit
* jp text fixes
* improved matrix math!
* add big fist cheat, minor type cleanup, add some debug toggles
* move all this mess to a new file
* slightly rework joint scaling function
* add big head npc cheat
* subtitles typo
* WIP mirror mode
* fix mirrored hud sprites
* fix mirror mode sound pan
* add cheats to menu!
* split some subtitles
* hud - show total counters on dpad up, level counters on dpad down
* make views toggle-able, extend hud timeout on toggle, decompilation cleanup
* review feedback round 1
* refactor draw-hud methods
* scaled version of draw-string-xy working, need to adjust hud labels still
* adjust scale/position of hud counters
* remove unnecessary adjustments
* update comments
* scripts: cleanup scripts directory some more
* stash, getting there
* stash again
* closer!
* it works! decently....cleanup time and add some extra features
* minor cleanup
* Added First working instance of controller/keyboard re-mapper
* Fixed clang formatting issues
* Updated newpad.cpp to inverse analog y-axis to make json labelling consistent
* Added mouse sensitivity options for X and Y axis in json, removed scroll mouse support, and other changes requested in feedback
* Added option to have ImGui debug menu appear on start up and remove hard coded set_imgui_visible calls
* Added newpad unit tests and updated function names to better describe intended functionalities
* Fixed formatting issue in newpad unit test
* Removed rumble unit test new pad
* Fixed codacy static analysis issues
* Fixed Linux build issues
* Implemented github feedback
* Implemented updated github feedback
* Fixed formatting errors
* Updated Pad::CheckPadIdx
* Implemented changes based on latest github feedback
* Implemented changes based on github feedback
Co-authored-by: animalstyletaco <[email protected]>
* dont crash on inline static value array
* [goalc] add support for non-static inline arrays of values
* add a kernel group to jak1
* move and cleanup pc debug code
* random cleanup in `hud-classes`
* pc port fix for gondola spools
* format
* address feedback (wow it was actually just changing 1 line all along...)
* minor decomp cleanup
* ref test
* fix invincibility cheat not working properly and fixed requirement
* typo
* added YMD for japan version of the game
* remove language cheat
* use language checks instead of territory checks in title logo
* forgot this call
* use `game-text-id->string` function here
* small cleanup to pc progress code
* better way to handle "locked" texts
* this is better
* fix potential incompatibilities with merc & ocean renderers
* show cheat requirements in menu + change requirements
* increase size of money starburst
* split some more subtitles
* potentially fix a vsync bug?
* change territory encoding logic
* pass game territory to compiler
* ugh LOL
* all-types: improve all-types generation
* all-types: re-generate all-types
* tests: remove the test reporting feature
the format indeed doesn't work, and all current actions require too many permissions for forked PRs.
I'll make my own eventually that works properly (use the new markdown feature)
* all-types: put the states in the method table instead
* all-types: replace all `*time*...uint64` fields with `time-frame` type
* all-types: address feedback
* tests: add a Jak2 `FormRegressionTest` fixture
* tests/ci: minimize test log output and produce a readable report
* improve naming of report due to underlying GitHub issue
* hopefully fix windows handling of this argument
* test...
* final test
* test linux coverage
* reduce max supersampling options in progress menu
* cleanup knuth rng file
* save & load cheats
* allow keyboard controls when cpad is connected
* ignore key presses when imgui is being used
* save settings when quitting game
* Automatically switch to dedicated GPU for runtime
Some machines with dual-GPUs cannot properly detect OpenGOAL as a high-performance application and properly utilize the dedicated GPU. This sets a flag for both Radeon and NVIDIA GPUs to always use the high performance GPU when available. I do not have a dual-GPU machine to test this on, but this should just work out of the box.
* clang-format
One day I will understand clang
* Add in-game option to switch fullscreen monitor
* mmm undefined memory :)
* Fix type consistency
* Optimize get_monitor and get_monitor_count since they're called often
* Address PR feedback
* Track fullscreen mode and minimized state to reduce GLFW calls per frame
* subtitles: add german subtitles for new progress menu options
* subtitles: add german subtitles for village1, beach
* missed some strings
* jungle, firecanyon, finalboss cutscenes
* add framrate options to progress menu
* fix samos line
* add citadel cutscenes
* add text version to subtitles
* add text version to text file
* `sidekick` hints
* fix up `mayor-introduction`
* oops
* `training` hints
* `village1` hints
* `lavatube` hints
* `snowy` and `spidercave` hints
* `finalboss` hints
* `firecanyon` hints and cutscenes
* `beach` hints
* update game text file
* finish up subtitles
* add german to progress menu sub setting
* split all bad lines and fix a few small errors
* remove test block
* Use sleepthread in RPC loop
* Keep a pointer to current IOP thread
* Implement IOP thread scheduling based on priority
And implement DelayThread as an actual delay.
* Run IOP flat out
* Use information from scheduler in wait_run_iop
* Lock sif mutex in set_rpc_queue
* always use kernel dispatch with wait_run
* Loop in dispatch until no thread is ready
* Use timestamp for next wakeup
instead of duration
* Wrap IOP thread entrypoints for safety
Libco threads are not supposed to return from their entrypoint
* Use a queue for IOP thread wakeups from EE thread
* IopThreadRecord -> IopThread
* add libco
* Use cooperative threading for IOP threads
* Ugly solution for overlord start
Needs to run in a thread
* Clean out thread shutdown logic
* Update comments
* Fix rat game camera speed > 60fps
* (Mostly) fix swamp balance platforms at high fps
I think there's other physics stuff factoring into this, but these changes should cover what's specific to the platforms.
* forgot notes
* Fix baby spider wiggling too fast on high fps
* Rat game camera now takes into account lag frames
* raise particles cap to 10x
* change a text line
* allow sound bank load during vag
* fix default settings and fps setting
* keep "auto save disabled" option in menu
* cheats menu!
* Update game_text_ja.gs
* fix battle hud crash
* add cheats menu to title
* update settings version
* Update all-types.gc
* disable envmap fade out with hires actor lod setting
* disable merc fadeout if not using ps2 actor lods
* dont update aspect ratio if game size is bad
* set vsync later
* increase memory for generic, sprite and dma.
* fix issues with aspect ratio not saving
* split forcing envmap to a separate toggle
* too slow, dont expose to users yet
* lsp: json-rpc example is working, a decent place to start...
* lsp: vendor library
* lsp: cleanup and time to get started
* lsp: commit what i got so far
* lsp: example `initialize` payload
* lsp: switch to `stdio`
* stash
* modularize the lsp implementation
* lsp: implement first actual LSP feature - function names in outline
* lsp: produce document diagnostics
* lsp: remove unused third-party lib
* lsp: support hovering MIPS instructions in IR files
* lsp: basic go-to all-types definition
* stash
* lsp: cleanup code, just need to add it to the release artifacts
* fix some project configuration
* fix linux build
* lsp: add lsp to PR artifacts and release assets
* lsp: address feedback
* add secrets menu, music player and some more pc settings
* remove todos
* rename string funcs
* replace cases
* add `game-text-id->string` function, refactor player menu generation
* remove process check, `cons` -> `dcons`, `strncpy`
* [graphics] a few optimizations
* fullscreen crap
* working, other than aspect ratio thing
* same behavior as before
* fix blackout bug, add more error messages
* fix error when 0 size buffer
* rm warning
* one last 0 size issue
* Enable L2 'boost if cheatmode is on and PC PORT
Nice QOL
* Enable daxter intro CS skip in retail mode on PC_PORT
* Add empty jak1 folder in ISO_DATA
We can revert this if we want but yeah I just think its more convenient by a tiny tiny tiny amount
* Fix typo in comment to be consistant
This probably isn't a typo but like a EU/US thing or something however I changed it just to be consistent with the other 382 instances in the port.
* Add gitignore back to root ISO_Data folder
* Remove early mention of Jak 2
* Update .gitignore
!**/.gitignore
* Fix instanced sprite distort renderer not always using the most recent sine table
Sometimes the game won't update the sine table on the same frame the aspect ratio changes (for example if the player is in a cutscene), but the renderer assumed that it was always up-to-date on the next frame. This caused it to occasionally use outdated vertex/tex coord data that usually made the distort sprites a little too small/big.
* Same fix, but better
* split rendering framebuffer
* fix blackout
* unhardcode c++ stuff
* optimization
* implement goal side of all this
* fix msaa
* sprite distort fixes
* fix resolution menu options
* save & load msaa
* linear filtering on the game screen
* fix bad texture units
* fun!
* unused
* make screenshot button capture the framebuffer and not the window buffer
* panik
* screenshot settings
* fix black screen on first frame of new framebuffer
* default fullscreen resolution to the screen size
* hide funny resolutions in windowed since it makes no sense
* only show subtitle notices during a subtitle context
* move language & subtitles (back) to game options
* remove "accessibility" menu and put its option in "misc"
* new subtitle language selection
* properly split audio & text language
* Update game_text_en.gs
* misc setting size fixes
* better fix for subtitle notice
* final touches
* subtitle fixes & uk-english subtitles
* more english fixes
* Fix floating rigidbodies at high FPS
- Caused the 'player impact' sound to play many times
- Simulation speed was also sped up in general (platforms would snap back to the neutral angle/position much faster)
* Fix lightning moles being a pain to chase on higher framerates
* Add high fps note
* Fix swamp rats wiggling too fast on high fps
* Fix 'kermit' moving super fast while airborne on high fps
* Fix low health animation for high fps
* Fix low eco animation for high fps
* Fix launcher cam rotating too fast with the analog stick at high fps
* Tweak high fps comments
* Fix zoomer heat HUD flashing too quickly on high fps
* Forgot a comment
* fix utf-8 handling around env-vars
* fix file opening errors related to unicode
* add uncaught exception handler in `gk` to ensure something is logged
* gracefully fail if window icon cant be loaded and work with unicode
* linux fix and add changes to vendor file
* [WIP] initial sprite distort implementation
* Clean up
* More clean up + document sprite distort VU program
* Format code
* Address CI issues
* Adjust hacks in distort fragment shader
* oops
* Optimize sprite distort rendering down to one draw call
~2x speed up
* Format file
* Clean up distort rendering and add separate profile scopes
* Fix glVertexAttribPointer
* Fix sprite distort getting messed up when the viewable area doesn't fit the window perfectly
* Add debug option to disable sprite distort
* One evil space to fail CI...
* oops
* Increase sprite-aux-list size when PC_BIG_MEMORY is true
* Address lints
* git: ignore vs build dir
* cmake: ditch `clang-cl` on windows in favor of actual `clang`
* build: suppress a significant number of warnings
* build: adjust workflows and vendor nasm
* docs: update docs to remove `clang-cl` mentions
* tests: move jak1 reference tests into their own folder
* tests: update offline tests to support multiple games
* tests: some additional fixes and multi-game handling
* tests: update reference tests
* add final cutscene subtitle and fix some editor bugs
* `title` and `finalboss` hints
* messages and notices for subtitle toggling
* fix colors in currently selected cutscene menu
* subtitle positioning fixes
* Update game_subtitle_en.gd
* extractor: refactor and cleanup for multi-game support
* deps: switch to `ghc::filesystem` as it is utf-8 everywhere by default
* extractor: finally working with unicode
* unicode: fix unicode cli args on windows in all `main` functions
* build: ensure `zydis` statically links when we want it to
* test
* test-static
* revert the static test commit
* build: don't generate cmake twice on release builds
* git: ignore vs build dir
* cmake: ditch `clang-cl` on windows in favor of actual `clang`
* build: suppress a significant number of warnings
* build: adjust workflows and vendor nasm
* docs: update docs to remove `clang-cl` mentions
* build: fix some copy-paste mistakes in the linux build
* build: remove C++20 compat warnings as that is useful if we want to upgrade
* put some duplicated code in a func
* make jak 2 text "work"
* group up all subtitles c++ code into one folder
* compact single-line subtitles
* fix a couple compiler crashes
* Update game_subtitle_en.gd
* `rolling` and `sunken`
* `swamp`
* `ogre`
* `village3`
* `maincave`
* `snow`
* `lavatube`
* `citadel`
* Update .gitignore
* clang
* fix encoding and decoding for quote
* properly fix quotes
* subtitle deserialize: sort by kind, ID and name
* sub editor: fix line speaker not being converted
* cleanup game text ids 1
* update text ids 2
* update source
* update refs
* fix repl-history saving
* subtitles: fix `\"` edge-case
* subtitles: add almost every remaining (current) cutscene
* merge: master
* subtitles: fix some crashes and disable `Remove Line` as it is broken
* Remove assets folder, use more std::filesystem
* windows fix
* another one for windows
* another one
* better system for different folders
* rm debugging stuff
* let extractor override everything
* dont revert jak1 change
* [extractor] validate files when extracted as folder
* jp text fixes
* move game text version to the text file and fix subtitle editor escape chars
* make bad subtitles not crash the game
* fix texscroll in lag
* fix mood, fix decomp of other versions, fix text decomp
* clang
* fix tests
* oops dammit
* new fixes
* shut up codacy
* fix nonexistant subtitles crashing the game
* fix text hacks and extractor re-use on folders
* Fix "last-hit" animations not playing for robotboss
This was causing destroyed parts of the robot to float in the air not moving for a moment.
* Add comments
* [build_level] drawable actor stuff
* clean up
* improved bsphere
* windows include nonsense
* final tuning
* m_pi
* what
* going insane
* merge conflict fix
* windows
* in-game vsync option, automatically framelimit correctly, fix minimize/alttab bugs
* fix jp text errors
* bump version number as panic
* this makes more sense i think
* clang
* oops delete debug print
* jak ambient wind, first person cam speed, cleanup
* fix seagulls!
* fix intro speed, mood, projectile collision, wind
* fix flying lurker and add comments
* oops didn't convert this back
* more comments
* Revert "more comments"
This reverts commit 9a211c594c.
* more comments
* yep
* missed 1 line
* more than 1 lol
* paren
* revert ripple for now, casting broke 60fps and didn't seem right
* Replace assert with user-friendly error if ISO not found
When the extractor runs and can't detect a game folder, it will assume there is an ISO file in the directory instead. If there isn't an ISO file, or it's a different extension, it will trigger an assert. This adds an additional check to make sure the file extension is of type `.iso` and returns a more clear message to the user.
Once I get home I'll iterate upon this a bit to add file size checks and reading header data as well as making sure the error code is able to be reported to the launcher.
* Fix `extension` call & formatting
more de-rust lol
* fix extension call (again)
* fix ONE extra parenthesis
* argument incompatible with fmt print
will fix print later
* add ONE missing parenthesis
never taking intellisense for granted again
* Add ISO size checks and rework type check
* actually print `data_dir_path` :p
* Normalize extension case before verifying file type
+ clang-format
* clang-format
Co-authored-by: doctashay <[email protected]>
* Fix one frame of ghost yellow eco in finalboss
* Fixed the eco duping in goggles
Whoops tried to fix an issue that wasn't there and paid the price for it.
* Add commented code
Describing the change and noting that is it different than the original game.
Could be useful in the future if anything odd happens to have our changes documented.
* Wrap change in a when PC_PORT block
* Move comment from inside block to outside
Having the comment inside the block seemed cringe
* remove str files from inputs
* removed unused verbose flag
* allow some kind of conditional game building for JP extras
* make `PrintBankInfo` do nothing
* clang
* Update gltf_mesh_extract.cpp
* fix `*jak1-full-game*` in extractor
* use json library to build json file
* screw it red label support as well
* PAL dumps files
* alrighty then
* make PAL warning slightly more specific
* PAL patches for `title-obs`
* Update all-types.gc
* PAL patch `beach-obs`
* `process-taskable` PAL patch
* `ambient` PAL patch
* `yakow` PAL patch
* `village-obs` PAL patch
* `sparticle-launcher` patch
* `swamp-obs` PAL patch
* `sequence-a-village1` PAL patch
* typo
* errors
* `powerups` PAL patch
* `ogreboss` PAL patch
* jak 1 v2 encoding
* `load-boundary` PAL patch
* `flying-lurker` PAL patch
* `mayor` PAL patch
* update game encoding to PAL (v2) encoding
* `cam-debug` and `cam-update` PAL patch
* `fisher` PAL patch
* `target` PAL patch
* `target2` PAL patch and fix text compiling
* `target-death` PAL patch
* `target-racer-h` PAL patch
* `logic-target` PAL patch
* `main` PAL patch
* `snow-flutflut-obs` PAL patch
* `rolling-obs` PAL patch
* `gsound` PAL patch
* update refs
* `progress` and `progress-draw` PAL patches
* clang
* wrong.
* complain
* clang
* fix test
* fix blurry jp text
* fix weird interrupt lag from setting window size
* patch more text lines, special handling for credits
* Update FontUtils.cpp
* Add xdelta3 and file patching interface
* add window lock toggle and update settings ver
* better particle hacks
* add PAL support to extractor
* Fix credits
* also NTSC-J support
* make xdelta3 a separate library
* address feedback
Co-authored-by: water <[email protected]>
* [cleanup] remove merc1, fix memory bugs, reduce memory usage
* change compiler log to see actual info from build_level
* save file
* editing text is hard
- Ubuntu: Added `libpulse-dev` as a dependency for audio support
- Arch: Removed `gcc`, `make` and `g++` as those are all already contained in the `base-devel` group, replaced `taskfile-git` with `go-task` as the former is outdated and added `libpulse` for audio
- Removed the section about the files in `goal_src` being placeholders as the game is mostly complete now.
- Added missing description for `iso_data` contents
- Added `discord-rpc` to the third-party library list
* wip
* learning about colors
* gltf node stuff working
* cleanup
* support textures
* bvh generation seems reasonable
* tree layout
* frag packer, untested and doesnt do real stripping yet
* temp
* working collide frags
* handle bad inputs better
* clean up
* format
* include
* another include
* reorganize for release build use
* docs for ee merc code
* 150fps support but it replaces 60fps
* oops switched wrong mode
* oops 50.0 not 50
* formatting
* fix cutscene speed
* oops
* Replace latest merc.md, not sure what happened
* Automatically switch between video modes (ntsc or 150fps) based on
refresh rate. Cleanup particle timing
* cleanup
* fix idle animation
* linter
* fix village2 crash
* can load all levels again
* update loader output and replace sparticle time with formula
* Add 100fps support, add some comments, fix build
* formatting
Co-authored-by: water <[email protected]>
* Fix outdated bucket id and change the bind display to not scroll off the screen
* Readability improvements to bind display (alignment)
* Provide script showing all font-color constants
* Rename display-all-colors.gc to keep with convention
* release: include error metadata file for launcher purposes
* release: put release assets in the top level of the 7z/tarball (no out/ dir)
* extractor: ensure critical directories are created
* extractor: handle weird Win32 path prefix `\\?\` by stripping it out
* release: avoid using `-C` with `tar`
* entity decomp and update battle hud code
* disable lag compensation in pc port
* delete unnecessary forward decls
* make battle hud track specific tasks
* fix mem leak + debug flying-lurker hud
* improve battle hud lights and fix bug
* tests
* stash
* temp
* tools: subtitle tool works! just gotta fill out the db / polish UX
* tools: added configuration for every subtitle we have so far
* tools: add some colors to the editor, time for repl controls and make it run the code!
* tools: continuing polish of tool, getting very close
* tools: finished UX polish, just need to write deserializers
* tools: added deserializer for subtitle data
* tools: exported subtitle files, all data appears intact
* tools: more UX polish and test all the cutscenes, majority work
* assets: update subtitle files
* lint: formatting and cleanup
* lint: codacy lints
* fix IOP getting stuck on music load
* fix regression? and clang
* fix a decomp
* fix another regression
* another
* fix "all actors"
* another regression!
* some jp support to fix some errors in the original game
* music fade toggle
* recognize `process-new` macros!!
* strip casts in this macro
* rename macro
* fix cast typecheck
* update source 1
* detect kernel stack case
* less boilerplate
* `manipy-spawn` special case
* pretty printer improvements
* revert dumb thing from earlier
* use shell detection on `send-event`
* fix some events
* remove unused argument
* detect `static-attack-info` and add `CondNoElse` to shell detect
* better `attack-info` detect
* support `process-spawn` in multi-lets
* detect `rand-float-gen` pt 1
* detect as return value
* detect in `countdown` and `dotimes`
* oops this wasnt working
* fancier `send-event`s
* clang
* update source!!
* fix tests
* fine jeez
* uh okay
* fix some accidental regressions
* fix more regressions
* regression fixes
* fix big bug...
* extra safety!
* music flava hack toggle
* fix hint subtitles not being considered offscreen
* music player works
* fixes + citadel, maincave, lavatube music
* add remaining tracks
* fix test
* fixes
* fix lib
* add "quit game" option to title
* reduce amount of particles in particle system
* increase merc2 memory
* enable extra particles by default
* fix tests
* various discord rpc additions
* fix formatting
* revert changes in game-task-h and task-control-h
* discord-rpc: refactor boss and racer/flut checks
now works via checking process states
* fix codacy
* tfrag3 data for merc2
* dma hooks for merc2
* start designing merc2 opengl, seems like the simple approach will be the best here
* before bone packing experiment
* fix up bones.gc
* use uniform buffer
* speedup, fix faces and eyes
* final fixes
* first pass at loader updates, tie is still bad
* temp
* improved loader
* run iop less often
* tfrag3 data for merc2
* dma hooks for merc2
* start designing merc2 opengl, seems like the simple approach will be the best here
* before bone packing experiment
* fix up bones.gc
* use uniform buffer
* speedup, fix faces and eyes
* final fixes
* update refs
* [decompiler] read and process art groups
* finish decompiler art group selection & detect in `ja-group?`
* make art stuff work on offline tests!
* [decompiler] detect `ja-group!` (primitive)
* corrections.
* more
* use new feature on skel groups!
* find `loop!` as well
* fully fledged `ja` macro & decomp + `loop` detect
* fancy fixed point printing!
* update source
* `:num! max` (i knew i should've done this)
* Update jak1_ntsc_black_label.jsonc
* hi imports
* make compiling the game work
* fix `defskelgroup`
* clang
* update refs
* fix chan
* fix seek and finalboss
* fix tests
* delete unused function
* track let rewrite stats
* reorder `rewrite_let`
* Update .gitattributes
* fix bug with `:num! max`
* Update robotboss-part.gc
* Update goal-lib.gc
* document `ja`
* get rid of pc fixes thing
* use std::abs
* docs for ee merc code
* wip
* more extraction stuff
* partial mat1 working
* mat1
* cleanup
* partial mat2 and mat3 support
* merc extraction seems to work
* Split cam inversion settings into first/third person settings. Connect settings to controls for main first/third person cameras
* inverted -> flipped everywhere
* cam-billy first-person camera impl
* respect camera inversion in cam-stick (debug option only), launcher cams
* refactor into common functions
* couple more cams
* tweak menu text and size, fix function ordering
* convert to macros, more descriptive names
* pc port comments
* one missed comment
* menu text/size/display tweaks
* respect flipped in main debug cam
* goalc: cleanup goalc's main method and add nrepl listener socket
* deps: add standalone ASIO for sockets
* lint: formatting
* common: make a common interface for creating a server socket
* goalc: setup new repl server
* deps: remove asio
* goalc: debug issues, nrepl is working again
* git: rename files
* attempt to fix linux function call
* test
* scripts: make the error message even more obvious....
* goalc: make suggested changes, still can't reconnect properly
* game: pull out single-client logic from XSocketServer
* nrepl: supports multiple clients and disconnection/reconnects
* goalc: some minor fixes for tests
* goalc: save repl history when the compiler reloads
* common: add include for linux networking
* a few small changes to fix tests
* is it the assert?
* change thread start order and add a print to an assert
Co-authored-by: water <[email protected]>
* [gtie] tie inst and proto functions
* first chain input to generic ee looks good
* m
* works
* ugh
* add tie generic
* rm debug print
* rm generic ties from fr3
* deps: add `xxhash` library
* extractor: hash ISO files as they are extracted
* extractor: report on game data validation errors
* lint: formatting
* extractor: automatically pick the right decompiler config
* detect `seek!` and `seekl!`
* fancy struct instead of pair mess
* fixes
* i think this was wrong?
* update refs
* update source
* More logical branching
* even better branching
* cpad1 keybinds and gamepad hooks in working state
* move pad info into arrays indexable by pad
* nice up the code some (mostly de-boilerplate) and improve pad detection
* last modification to remove unnecessarily duplicated code
* Fix formatting
* literally why did i add a newline after running clang-format
* Only append compiler flags in cmake
* add `RelWithDebInfo-clang` windows build
* bump serializer initial buffer size to 32mb
* extra flags for msvc
* proper anaphoric goal macros
* specify windows sdk version?
* fix "Object files are not named properly" fake error
* Update goal-lib.gc
* srpc: Implement part of VBlank_Handler
And call it on RPC message for lack of better ways
* ssound: Fix distance calculation
avoids negative array index
* srpc: fix sound id assignment
* srpc: bail out on missing sound id
* ssound: Fixes for angle and volume calculation
* srpc: Fix VAG filename stuff
* ssound: Fix CalculateAngle
* ssound: UpdateAutoVol fixes
* srpc: Fix up SET_PARAM command
* ci: fix windows releases (hopefully)
* scripts: fix Taskfile file references for linux
* asserts: add `ASSERT_MSG` macro and ensure `stdout` is flushed before `abort`ing
* asserts: refactor all `assert(false);` with a preceeding message instances
* lint: format
* temp...
* fix compiler errors
* assert: allow for string literals in `ASSERT_MSG`
* lint: formatting
* revert temp change for testing
* fix typo
* more typo
* shorten discord rpc text
* allow expanding enums after the fact (untested)
* make `game_text` work similar to subtitles
* update progress decomp
* update some types + `do-not-decompile` in bitfield
* fixes and fall back to original progress code
* update `progress` decomp with new enums
* update config files
* fix enums and debug menu
* always allocate (but not use) a lot of particles
* small rework to display mode options
* revert resolution/aspect-ratio symbol mess
* begin the override stuff
* make `progress-draw` more readable
* more fixes
* codacy good boy points
* first step overriding code
* finish progress overrides, game options menu fully functional!
* minor fixes
* Update game.gp
* Update sparticle-launcher.gc
* clang
* change camera controls text
* oops
* some cleanup
* derp
* nice job
* implement menu scrolling lol
* make scrollable menus less cramped, fix arrows
* make some carousell things i guess
* add msaa carousell to test
* oops
* Update progress-pc.gc
* make `pc-get-screen-size` (untested)
* resolution menu
* input fixes
* return when selecting resolution
* scroll fixes
* Update progress-pc.gc
* add "fit to screen" button
* bug
* complete resolutions menu
* aspect ratio menu
* subtitles language
* subtitle speaker
* final adjustments
* ref test
* fix tests
* fix ref!
* reduce redundancy a bit
* fix mem leaks?
* save settings on progress exit
* fix init reorder
* remove unused code
* rename goal project-like files to the project extension
* sha display toggle
* aspect ratio settings fixes
* dont store text db's in compiler
* properly save+load native aspect stuff
* [goalc] macro expansion in integer constants
* working
* didn't break it yet
* support conditional compilation
* fix up some more small bugs
* fix duplicate evaluation of bitfield definitions
* paranoid
description:Take a couple minutes to help our maintainers work faster.
options:
- label:I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Ajak1+sort%3Aupdated-desc+) for duplicate or closed bug reports
required:true
- label:I understand that I am supposed to provide my own legitimately obtained copy of the game
required:true
- type:textarea
attributes:
label:Describe the Bug
description:A clear and concise description of what the bug is. You may post screenshots or videos of the bug here.
validations:
required:true
- type:textarea
attributes:
label:How To Reproduce
description:Steps to reproduce the behavior. You can also post a video of it here.
validations:
required:true
- type:dropdown
attributes:
label:Does this problem occur on original hardware or PCSX2?
description:Some things that may seem like bugs are actually exactly how the original game behaved.
options:
- Yes,it's unique to OpenGOAL
- Didn't check
- Not needed, bug is obvious
validations:
required:true
- type:textarea
attributes:
label:Expected Behavior
description:A clear and concise description of the expected behavior.
placeholder:When I do X, Y should happen.
validations:
required:true
- type:textarea
attributes:
label:Environment Information
description:"You can upload the [Support Package](https://github.com/open-goal/launcher#asking-for-help) provided by the Launcher here, or you can provide the following information: CPU, GPU, OS Version, OpenGOAL Version (found in the window's title bar)"
validations:
required:true
- type:dropdown
attributes:
label:Game Version
options:
- NTSC 1.0 (black label)
- NTSC Greatest Hits version (red label)
- PAL
- JP
validations:
required:true
- type:dropdown
attributes:
label:Have you set the game to something other than `60fps`?
description:Take a couple minutes to help our maintainers work faster.
options:
- label:I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Ajak2+sort%3Aupdated-desc+) for duplicate or closed bug reports
required:true
- label:I understand that I am supposed to provide my own legitimately obtained copy of the game
required:true
- type:textarea
attributes:
label:Describe the Bug
description:A clear and concise description of what the bug is. You may post screenshots or videos of the bug here.
validations:
required:true
- type:textarea
attributes:
label:How To Reproduce
description:Steps to reproduce the behavior. You can also post a video of it here.
validations:
required:true
- type:dropdown
attributes:
label:Does this problem occur on original hardware or PCSX2?
description:Some things that may seem like bugs are actually exactly how the original game behaved.
options:
- Yes,it's unique to OpenGOAL
- Didn't check
- Not needed, bug is obvious
validations:
required:true
- type:textarea
attributes:
label:Expected Behavior
description:A clear and concise description of the expected behavior.
placeholder:When I do X, Y should happen.
validations:
required:true
- type:textarea
attributes:
label:Environment Information
description:"You can upload the [Support Package](https://github.com/open-goal/launcher#asking-for-help) provided by the Launcher here, or you can provide the following information: CPU, GPU, OS Version, OpenGOAL Version (found in the window's title bar)"
validations:
required:true
- type:dropdown
attributes:
label:Game Version
options:
- NTSC 1.0 (black label)
- NTSC Greatest Hits version (red label)
- PAL
- JP
validations:
required:true
- type:dropdown
attributes:
label:Have you set the game to something other than `60fps`?
"description": "Marks the file to be excluded from the LoC count"
},
"og:ignore-errors": {
"scope": "opengoal",
"prefix": ["og:ignore-errors"],
"body": [";; og:ignore-errors"],
"description": "Ignore lines beginning with ';; ERROR:' or ';; WARN:' when copying decompiled code into it"
},
"og:ignore-form": {
"scope": "opengoal",
"prefix": ["og:ignore-form"],
"body": [";; og:ignore-form:${1:substr}"],
"description": "If the provided substr is found in the starting line of a form, the entire form will be omitted when copying decompiled code into the file"
},
"og:preserve-this": {
"scope": "opengoal",
"prefix": ["og:preserve-this"],
"body": [";; og:preserve-this"],
"description": "When updating GOAL source files from decomp, CI will pick up if this comment is deleted in the diff. Use this to keep track of manually patched sections of code."
Naughty Dog created a custom programming language called [GOAL](https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp), along with a compiler for it. GOAL makes up 98% of all the code in Jak & Daxter, and the GOAL compiler does not have the same amount of optimizations as C or C++ compilers have, which means that the compiler often times left specific instructions that denote some specific kind of control flow, which lets us piece back the original structure of the code accurately for about 99% of the functions in the game. Nearly every type in GOAL also has an autogenerated debug `inspect` method, which prints all of a type's fields by default. These `inspect` methods can tell us what fields were in the type, at what offset, and what their type was.
### How finished is the project?
We estimate it is about 75% done. Most of the renderers are finished and there is no sound.
### When will the project be finished?
Hopefully just a few more months!
### Why is there no sound?
We have not started work on sound at the moment.
### Will you also decompile Jak II?
Likely yes!
### Will you also decompile Jak X?
We don't know yet.
### How can I run the game?
The project readme contains instructions on installing and running the game. You need to bring your own copy of the game, we provide none of its assets.
### I am having trouble running the game!
Keep in mind the project is currently in development and is not adapted for end users at the moment. If you are having trouble, there are some extra troubleshooting steps at the bottom of this page. If the common issues do not include your specific issue, feel free to join the discord server and ask for help in the help channel.
### Does the game have differences? I don't remember these options!
We have added a plethora of options to the game settings (and removed some that didn't make sense) so that you can have a more up to date experience, or a more PS2-like experience if you decide. It is up to you! There are also a bunch of extra goodies and added secrets to find out. We are aiming to keep the core gameplay (controls, physics, behaviors, etc.) identical however, so if you find any issues or differences with this then do not hesitate to tell us about it.
### Where can I find a list of differences?
[This page](https://github.com/open-goal/jak-project/issues/386) contains a list of ideas and suggestions we have for the game, but it does not cover all the deliberate changes we have made. We will probably compile a list at some point! In the meantime, feel free to ask us about this.
### Will there be bug fixes to the game?
The original game had a few issues of varying severity - from progress softlocks to crashing - that we are aiming to patch. The complexity of these issues is varying (if they were simple to reproduce and fix, Naughty Dog likely would have done it), but so far we have fixed 2 issues that could crash the game.
### Will there be mod support?
In the future yes! We intend to make it as easy as it can be to make your own code or import graphics for the game.
### What is this menu? Why am I not dying sometimes?
This might be the result of the game beeing booted in debugging mode. The project is still in development and we rely on the game's built-in debug features (along with some we made ourselves) for testing and development. The final product will have this debug mode disabled. In the meantime, have fun with all the options! Keep in mind some don't work very well.
### What are the minimum requirements?
These are not totally set in stone as the project is not complete. These are the minimum architecture requirements for the project:
- x86-64 CPU with AVX support. Most, if not all, 64-bit CPUs should work. OpenGOAL has a very small CPU footprint, so most modern CPUs will run the game fine.
- Graphics card that supports OpenGL 4.3. The rendering code is still very much in progress. NVIDIA graphics cards should have no issue running the game, but AMD cards might struggle unless you have a more powerful one.
- You might experience slower loads with a hard drive compared to an SSD.
### Will there be a Vulkan version of the game?
Probably yes, in the future!
### I'd like to contribute, but don't know where to start!
Great to hear! Any help is good help. If you haven't already, please read the project readme. It has information on what is done, what isn't done, what is being worked on, the project layout and how it all works. Then just join our development discord server and we can help you further.
### Will you also decompile other games that aren't Jak?
No.
### Is this project legal?
Yes.
## Troubleshooting
### None of the tools are working!
Because of its in-development state, we provide no pre-built binaries at the moment. The project readme has instructions on how to compile the tools.
### I am getting "illegal operation"/"illegal instruction" errors when running the game or tools.
This shouldn't happen, so please contact us in the support channel of the discord server and we can help, and tell us if your CPU supports AVX, AVX2 or AVX512.
### The game is really slow.
Your GPU is probably too weak and our renderers too slow. However, we plan on making significant changes to the renderers over time to improve performance. The merc renderer right now is very slow, if you would like to turn off then head to the debug menu by pressing R3, then go to `Render...` and turn `merc` off.
### The game crashed!!
Crap! Contact us in the help channel on the discord server and tell us about it! Post the log files as well. These are located in the `log` folder of the project.
### The game crashed when entering the final boss area!
This one specifically is known. We are investigating this at the moment, so don't worry.
> Our repositories on GitHub are primarily for development of the project and tracking active issues. Most of the information you will find here pertains to setting up the project for development purposes and is not relevant to the end-user.
For a setup guide on how to install and play the game there is the following video that you can check out: https://youtu.be/K84UUMnkJc4
For questions or additional information pertaining to the project, we have a Discord for discussion here: https://discord.gg/VZbXMHXzWv
Additionally, you can find further documentation and answers to **frequently asked questions** on the project's main website: https://opengoal.dev
> [!WARNING]
> **Do not use this decompilation project without providing your own legally purchased copy of the game.** We do not distribute any assets from the game - you must use your own legitimately obtained PS2 copy of the game. We support every retail PAL, NTSC, and NTSC-J build, including Greatest Hits copies.
- [Project Description](#project-description)
- [Current Status](#current-status)
- [What's Next](#whats-next)
- [Getting Started - Linux](#getting-started---linux)
- [Ubuntu (20.04)](#ubuntu-2004)
- [Arch](#arch)
- [With Nix](#with-nix)
- [Getting Started - Windows](#getting-started---windows)
- [Required Software](#required-software)
- [Setting up and Opening the Project](#setting-up-and-opening-the-project)
- [Building and Running the Game](#building-and-running-the-game)
- [Extract Assets](#extract-assets)
- [Build the Game](#build-the-game)
- [Run the Game](#run-the-game)
- [Connecting the REPL to the Game](#connecting-the-repl-to-the-game)
- [Running the Game Without Auto-Booting](#running-the-game-without-auto-booting)
- [Interacting with the Game](#interacting-with-the-game)
- [Project Layout](#project-layout)
- [Directory Layout](#directory-layout)
<!-- tocstop -->
- [Current Status](#current-status)
- [Methodology](#methodology)
- [Setting up a Development Environment](#setting-up-a-development-environment)
- [Docker](#docker)
- [Linux](#linux)
- [Ubuntu (20.04)](#ubuntu-2004)
- [Arch](#arch)
- [Fedora](#fedora)
- [Windows](#windows)
- [Required Software](#required-software)
- [Using Visual Studio](#using-visual-studio)
- [MacOS](#macos)
- [Intel Based](#intel-based)
- [Apple Silicon](#apple-silicon)
- [VSCode](#vscode)
- [Building and Debugging](#building-and-debugging)
- [Building and Running the Game](#building-and-running-the-game)
- [Extract Assets](#extract-assets)
- [Build the Game](#build-the-game)
- [Run the Game](#run-the-game)
- [Connecting the REPL to the Game](#connecting-the-repl-to-the-game)
- [Running the Game Without Auto-Booting](#running-the-game-without-auto-booting)
- [Interacting with the Game](#interacting-with-the-game)
- [Running the decompiler](#running-the-decompiler)
- [`goal_src/`](#goal_src)
- [`game` runtime](#game-runtime)
## Project Description
This project is to port Jak 1 (NTSC, "black label" version) to PC. Over 98% of this game is written in GOAL, a custom Lisp language developed by Naughty Dog. Our strategy is:
This project is to port the original Jak and Daxter and Jak II to PC. Over 98% of the games are written in GOAL, a custom Lisp language developed by Naughty Dog. Our strategy is:
- decompile the original game code into human-readable GOAL code
- develop our own compiler for GOAL and recompile game code for x86-64
- create a tool to extract game assets into formats that can be easily viewed or modified
- create tools to repack game assets into a format that our port uses.
Our objectives are:
- make the port a "native application" on x86-64, with high performance. It shouldn't emulated, interpreted, or transpiled.
- make the port a "native application" on x86-64, with high performance. It shouldn't be emulated, interpreted, or transpiled.
- Our GOAL compiler's performance should be around the same as unoptimized C.
- try to match things from the original game and development as possible. For example, the original GOAL compiler supported live modification of code while the game is running, so we do the same, even though it's not required for just porting the game.
- support modifications. It should be possible to make edits to the code without everything else breaking.
We support both Linux and Windows on x86-64.
We have a Discord server where we discuss development. https://discord.gg/VZbXMHXzWv
> We do not support, or plan to support the ARM architecture. This means that this will not run on devices such as an M1 Mac or a mobile device.
**[If you have any questions, please read the FAQ here beforehand.](FAQ.md)**
### Current Status
## Current Status
Jak 1 is largely playable from start to finish with a handful of bugs that are continually being ironed out. Jak 2 is in development.
So far, we've decompiled around 400,000 lines of GOAL code, out of an estimated 500,000 total lines. We have a working OpenGL renderer which renders most of the game world and foreground. Levels are fully playable, and you can finish the game with 100% completion! There is currently *no* audio.
To help with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where we believe the original code had type casts (or where they feel appropriate) until the decompiler succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in `goal_src`. Our decompiler is designed specifically for processing the output of the original GOAL compiler. As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is tested as part of our unit tests, and so far we have over 300,000 lines (460 files) that pass.
### Methodology
We don't save any assets from the game - you must bring your own copy of the game and use the decompiler to extract assets.
To help with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where we believe the original code had type casts (or where they feel appropriate) until the decompiler succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in `goal_src`.
## What's Next
Our decompiler is designed specifically for processing the output of the original GOAL compiler. As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is continually tested as part of our unit tests.
- Continue decompilation of GOAL code. We've made huge progress recently in decompiling gameplay code. We're finishing that up and also working on the some of the rendering code. Here's our decompilation progress over the past year: 
- Bug testing! The game can be beaten 100%, but it's possible a few things do not work correctly.
- Improve the decompiler. We are always finding new features and macros in the GOAL language.
- Investigate more complicated renderers. We have an in-progress port of the "merc" foreground renderer, shown in the screenshots above.
## Setting up a Development Environment
## Getting Started - Linux
The remainder of this README is catered towards people interested in building the project from source, typically with the intention on contributing as a developer.
### Ubuntu (20.04)
If this does not sound like you and you just want to play the game, refer to the above section [Quick Start](#quick-start)
### Docker
All three Linux systems are supported using Docker.
Pick your supported prefered flavour of linux and build your chosen image
```
docker build -f docker/(Arch|Fedora|Ubuntu)/Dockerfile -t jak .
```
This will create an image with all required dependencies and already built.
```
docker run -v "$(pwd)"/build:/home/jak/jak-project/build -it jak bash
```
Note: If you change the content of the `build/` directory you'll need to rerun the `build` command. Alternatively you can get the build via `docker cp`.
This will link your `build/` folder to the images so can validate your build or test it on an external device.
Docker images can be linked into your IDE (e.g. CLion) to help with codesniffing, static analysis, run tests and continuous build.
Unfortunately you'll still need task runner on your local machine to run the game or instead, manually run the game via the commands found in `Taskfile.yml`.
We primarily use Visual Studio on Windows for C++ development. Download the latest community edition from [here](https://visualstudio.microsoft.com/vs/)
### Windows
#### Required Software
We primarily use Visual Studio on Windows for C++ development. Download the latest community edition from [here](https://visualstudio.microsoft.com/vs/). At the time of writing this is Visual Studio 2022.
You will require the `Desktop development with C++` workload. This can be selected during the installation, or after via the `Visual Studio Installer`, modifying the Visual Studio Installation.
@@ -169,12 +213,10 @@ On Windows, it's recommended to use a package manager, we use Scoop. Follow the
Once Scoop is installed, run the following commands:
```sh
scoop install git llvm nasm python
scoop bucket add extras
scoop install task
scoop install git llvm nasm python task
```
### Setting up and Opening the Project
#### Using Visual Studio
Clone the repository by running the following command in your folder of choice.
@@ -186,12 +228,57 @@ This will create a `jak-project` folder, open the project as a CMake project via
Then build the entire project as `Windows Release (clang-cl)`. You can also press Ctrl+Shift+B as a hotkey for Build All. We currently prefer `clang-cl` on Windows as opposed to `msvc`, though it should work as well!
Then build the entire project as `Windows Release (clang)`. You can also press Ctrl+Shift+B as a hotkey for Build All. We currently prefer `clang` on Windows as opposed to `msvc`, though it should work as well!
## Building and Running the Game
### MacOS
> NOTE: At this time you can only run the game on macOS if you have an Intel processor.
Ensure that you have Xcode command line tools installed (this installs things like Apple Clang). If you don't, you can run the following command:
If you either don't want to or cannot use Visual Studio for working with the C++ project, VSCode is a good alternatively.
The `clangd` extension is [recommended](https://marketplace.visualstudio.com/items?itemName=llvm-vs-code-extensions.vscode-clangd) and requires `clangd` to be on your `$PATH`. If you can run `clangd` in a terminal successfully then you should be good to go.
Once you generate your CMake for the first time the clangd LSP should be able to index the project and give you intellisense.
#### Building and Debugging
TODO - Consider Contributing Documentation :)
### Building and Running the Game
Getting a running game involves 4 steps:
@@ -200,17 +287,35 @@ Getting a running game involves 4 steps:
3. Build the game
4. Run the game
### Extract Assets
#### Extract Assets
First, setup your settings so the following scripts know which game you are using, and which version. For the black label version of the game, run the following in a terminal:
```sh
task set-game-jak1
task set-decomp-ntscv1
```
For other versions of the game, you will need to use a different `-set-decomp-<VERSION>` command. An example for the PAL version:
```sh
task set-game-jak1
task set-decomp-pal
```
> Run `task --list` to see the other available options
> At the time of writing, only Jak 1 is expected to work end-to-end!
The first step is to extract your ISO file contents into the `iso_data/<game-name>` folder. In the case of Jak 1 this is `iso_data/jak1`.
Once this is done, open a terminal in the `jak-project` folder and run the following:
```sh
task extract-jak1
task extract
```
### Build the Game
#### Build the Game
The next step is to build the game itself. To do so, in the same terminal run the following:
@@ -221,16 +326,16 @@ task repl
You will be greeted with a prompt like so:
```sh
_____ _____ _____ _____ __
||___ ___ ___| __|| _ ||
||| . | -_||||||||__
_____ _____ _____ _____ __
||___ ___ ___| __|| _ ||
||| . | -_||||||||__
|_____| _|___|_|_|_____|_____|__|__|_____|
|_|
Welcome to OpenGOAL 0.8!
Run (repl-help)forhelp with common commands and REPL usage.
Run (lt) to connect to the local target.
g >
g >
```
Run the following to build the game:
@@ -239,7 +344,13 @@ Run the following to build the game:
g > (mi)
```
### Run the Game
> IMPORTANT NOTE! If you're not using the non-default version of the game, you may hit issues trying to run `(mi)` in this step. An example error might include something like:
>
> `Input file iso_data/jak1/MUS/TWEAKVAL.MUS does not exist.`
>
> This is because the decompiler inputs/outputs using the `gameName` JSON field in the decompiler config. For example if you are using Jak 1 PAL, it will assume `iso_data/jak1_pal` and `decompiler_out/jak1_pal`. Therefore, you can inform the REPL/compiler of this via the `gameVersionFolder` config field described [here](./goal_src/user/README.md)
#### Run the Game
Finally the game can be ran. Open a second terminal from the `jak-project` directory and run the following:
@@ -249,7 +360,7 @@ task boot-game
The game should boot automatically if everything was done correctly.
#### Connecting the REPL to the Game
##### Connecting the REPL to the Game
Connecting the REPL to the game allows you to inspect and modify code or data while the game is running.
@@ -271,7 +382,7 @@ For example, running the following will print out some basic information about J
gc> *target*
```
#### Running the Game Without Auto-Booting
##### Running the Game Without Auto-Booting
You can also start up the game without booting. To do so run the following in one terminal
@@ -297,103 +408,81 @@ gc> (test-play)
gc>
```
### Interacting with the Game
#### Interacting with the Game
In the graphics window, you can use the period key to bring up the debug menu. Controllers also work, using the same mapping as the original game.
Check out the `pc_debug`, `examples` and `engine/pc/` folders under `goal_src` for some examples of GOAL code we wrote. The debug files that are not loaded automatically by the engine have instructions for how to run them.
Check out the `pc_debug`, `examples` and `pc` folders under `goal_src` for some examples of GOAL code we wrote. The debug files that are not loaded automatically by the engine have instructions for how to run them.
## Project Layout
## Technical Project Overview
There are four main components to the project.
The first is `goalc`, which is a GOAL compiler for x86-64. Our implementation of GOAL is called OpenGOAL. All of the compiler source code is in `goalc`. To run the compiler on Linux, there is a script `gc.sh`. On Windows, there is a `gc.bat` scripts and a `gc-no-lt.bat` script, the latter of which will not attempt to automatically attach to a running target. The compiler is controlled through a prompt which can be used to enter commands to compile, connect to a running GOAL program for interaction, run the OpenGOAL debugger, or, if you are connected to a running GOAL program, can be used as a REPL to run code interactively. In addition to compiling code files, the compiler has features to pack and build data files.
1.`goalc` - the GOAL compiler for x86-64
2.`decompiler` - our decompiler
3.`goal_src/` - the folder containing all OpenGOAL / GOOS code
4.`game` - aka the runtime written in C++
The second component to the project is the decompiler. You must have a copy of the PS2 game and place all files from the DVD inside a folder corresponding to the game within `iso_data` folder (`jak1` for Jak 1 Black Label, etc.), as seen in this picture:
Let's break down each component.
### `goalc`
Our implementation of GOAL is called OpenGOAL.
All of the compiler source code is in `goalc/`. The compiler is controlled through a prompt which can be used to enter commands to compile, connect to a running GOAL program for interaction, run the OpenGOAL debugger, or, if you are connected to a running GOAL program, can be used as a REPL to run code interactively. In addition to compiling code files, the compiler has features to pack and build data files.
#### Running the compiler
**Environment Agnostic**
If you have installed `task` as recommended above, you can run the compiler with `task repl`
**Linux**
To run the compiler on Linux, there is a script `scripts/shell/gc.sh`.
**Windows**
On Windows, there is a `scripts/batch/gc.bat` scripts and a `scripts/batch/gc-no-lt.bat` script, the latter of which will not attempt to automatically attach to a running target.
### `decompiler`
The second component to the project is the decompiler.
The decompiler will output code and other data intended to be inspected by humans in the `decompiler_out` folder. Files in this folder will not be used by the compiler.
#### Running the decompiler
You must have a copy of the PS2 game and place all files from the DVD inside a folder corresponding to the game within `iso_data` folder (`jak1` for Jak 1 Black Label, etc.), as seen in this picture:
Then run `decomp.sh` (Linux) or `decomp-jak1.bat` (Windows) to run the decompiler. The decompiler will extract assets to the `assets` folder. These assets will be used by the compiler when building the port, and you may want to turn asset extraction off after running it once. The decompiler will output code and other data intended to be inspected by humans in the `decompiler_out` folder. Stuff in this folder will not be used by the compiler.
The decompiler will extract assets to the `assets` folder. These assets will be used by the compiler when building the port, and you may want to turn asset extraction off after running it once.
The third is the game source code, written in OpenGOAL. This is located in `goal_src`. All GOAL and GOOS code should be in this folder. Right now most of this is placeholders or incomplete, but you can take a look at `kernel/gcommon.gc` or `goal-lib.gc` to see some in-progress source code.
**Environment Agnostic**
The final component is the "runtime", located in `game`. This is the part of the game that's written in C++. In the port, that includes:
If you have installed `task` as recommended above, you can run the compiler with `task decomp`
**Linux**
To run, you can use `scripts/shell/decomp.sh` to run the decompiler
**Windows**
To run, you can use `scripts/shell/decomp-jak1.bat` to run the decompiler
### `goal_src/`
The game source code, written in OpenGOAL, is located in `goal_src`. All GOAL and GOOS code should be in this folder.
### `game` runtime
The final component is the "runtime", located in `game`. This is the part of the game that's written in C++.
In the port, that includes:
- The "C Kernel", which contains the GOAL linker and some low-level GOAL language features. GOAL has a completely custom dynamically linked object file format so in order to load the first GOAL code, you need a linker written in C++. Some low-level functions for memory allocation, communicating with the I/O Processor, symbol table, strings, and the type system are also implemented in C, as these are required for the linker. It also listens for incoming messages from the compiler and passes them to the running game. This also initializes the game, by initializing the PS2 hardware, allocating the GOAL heaps, loading the GOAL kernel off of the DVD, and executing the kernel dispatcher function. This is in the `game/kernel` folder. This should be as close as possible to the game, and all differences should be noted with a comment.
- Implementation of Sony's standard library. GOAL code can call C library functions, and Naughty Dog used some Sony library functions to access files, memory cards, controllers, and communicate with the separate I/O Processor. The library functions are in `game/sce`. Implementations of library features specific to the PC port are located in `game/system`.
- The I/O Processor driver, Overlord. The PS2 had a separate CPU called the I/O Processor (IOP) that was directly connected to the DVD drive hardware and the sound hardware. Naughty Dog created a custom driver for the IOP that handled streaming data off of the DVD. It is much more complicated than I first expected. It's located in `game/overlord`. Like the C kernel, we try to keep this as close as possible to the actual game.
- Sound Code. Naughty Dog used a third party library for sound. We have not started on this yet.
- PC specific graphics stuff. We have a functional OpenGL renderer and context that can create a game window and display graphics on it. The specific renderers used by the game however are mostly unimplemented. We have a debug, sprite and tfrag renderers, which is enough to render most of the game world, on-screen text and sprites like the progress menu. This is located in `game/graphics`. Many liberties will be taken to make this work, but the end result should closely match the actual game.
## Directory Layout
-`assets`: extracted assets (textures, translated game text, audio) generated by the decompiler. Not included in the repository. This data is used when building the PC port.
-`build`: C++ CMake build folder.
-`common`: common C++ code shared between the compiler, decompiler, and game.
-`audio`: tools for decoding the audio files.
-`cross_os_debug`: platform-independent library for implementing the OpenGOAL debugger. Linux and Windows.
-`cross_sockets`: platform-independent library for sockets. Used to connect the compiler to a running game. Linux and Windows.
-`goos`: the compiler-time macro language and parser for OpenGOAL.
-`type_system`: the OpenGOAL type system.
-`texture`: texture unpacking and format conversion.
-`dma`: utilities for handling the transfer of PS2 DMA packets.
-`util`, `math`, `log`: Random utility functions for accessing files, timers, etc.
-`decompiler`: Source code for the decompiler.
-`analysis`: analysis algorithms.
-`config`: JSON config files for the decompiler and type definition file.
-`data`: utilities to extract assets from the game.
-`Disasm`: MIPS disassembler.
-`Function`: Tools for analyzing GOAL functions.
-`gui`: an early prototype of a Python GUI for reading the output of the decompiler.
-`IR2`: the "Intermediate Representation" for GOAL functions and expressions.
-`ObjectFile`: Utilities for processing the GOAL object file format.
-`scripts`: Useful scripts for setting up the decompilation.
-`util`: random utilities.
-`VuDisasm`: disassembler for VU code.
-`decompiler_out`: output of the decompiler that's not automatically used by the compiler. This is for humans to read and use. Not included in the repository.
-`docs`: more documentation!
-`game`: the source code for the game executable.
-`common`: shared stuff between the `kernel` (EE) and `overlord` (IOP).
-`graphics`: PC Port graphics.
-`kernel`: the part of the GOAL kernel written in C. The entry point for the game is in `kboot.cpp`.
-`overlord`: the I/O processor driver used to get data off of the DVD.
-`sce`: the Sony library implementation.
-`system`: PC-port specific OS-level stuff, like file I/O, threads, controllers, debug network connection.
-`goal_src`: The GOAL code for the game.
-`build`: info related to the GOAL build system.
-`engine`: the game engine.
-`kernel`: The GOAL kernel.
-`levels`: Level specific code.
-`old`: Unused or deleted files from the original game.
-`dgos`: File lists for DGOs in the build system.
-`examples`: Example GOAL code we have written.
-`test`: Test code we have written.
-`pc_debug`: GOAL code for PC-specific debugging. Also written by us.
-`user`: User sandboxes. Written by you! And anyone else that wishes to share their user code.
-`goalc`: The OpenGOAL compiler.
-`compiler`: The implementation of the OpenGOAL language.
-`data_compiler`: Tools for packing data.
-`debugger`: The OpenGOAL debugger (part of the compiler).
-`emitter`: x86-64 emitter and object file generator.
-`listener`: The OpenGOAL listener, which connects the compiler to a running GOAL program for the interactive REPL.
-`make`: The OpenGOAL build system, builds both code and data files.
-`regalloc`: Register allocator.
-`iso_data`:
-`out`: Outputs from the build process. Only the `iso` subfolder should contain assets used by the game.
-`iso`: Final outputs that are used by the game.
-`obj`: Object files generated by the compiler.
-`resources`: To be removed. Contains fake versions of some files required to get things booting.
-`scripts`: Utility scripts. Windows-specific batch files are in a `batch` folder while Unix shell scripts are in a `shell` folder.
-`test`: Unit tests (run on GitHub Actions).
-`third-party`: Third party libraries.
- CMake Code Coverage. For code coverage statistics on GitHub builds.
-`fmt`. String formatting library.
-`googletest`: Test framework.
-`inja`: templating library used for generating test code for compiler tests.
-`lzokay`: decompression code for Jak 2 and later DGOs.
-`mman`: Windows library used to emulate `mmap` on Linux.
-`run-clang-format`: Utility to check and enforce code formatting.
-`run-clang-tidy`
-`zydis`: x86-64 disassembler used in the OpenGOAL debugger.
-`json`: A JSON library.
-`replxx`: Used for the REPL input. Supports history and useful editing shortcuts.
-`svpng`: Save a PNG file.
- The I/O Processor driver, OVERLORD. The PS2 had a separate CPU called the I/O Processor (IOP) that was directly connected to the DVD drive hardware and the sound hardware. Naughty Dog created a custom driver for the IOP that handled streaming data off of the DVD. It is much more complicated than I first expected. It's located in `game/overlord`. Like the C kernel, we try to keep this as close as possible to the actual game.
- Sound code. Naughty Dog used a third party library for sound called `989SND`. Code for the library and an interface for it is located in `game/sound`.
- PC specific graphics code. We have a functional OpenGL renderer and context that can create a game window and display graphics on it. The specific renderers used by the game however are mostly implemented. Aside from post-processing effects, everything in the game is rendered. This is located in `game/graphics`. While many liberties will be taken to make this work, the end result should very closely match the actual game.
- Extra assets used by the port in some fashion, located in `game/assets`. These include extra text files, icons, etc.
> This is still in a highly experimental state, adjust your expectations accordingly.
Despite LISP being very simple, and taking the stance of a highly opinionated formatter that doesn't concern itself with line-length in most situations writing a code formatter is still an incredibly annoying and difficult thing to do. This documentation serves as a place for me to collect my thoughts and explain how it works (or more likely, how it does not).
Also in general I found it _really_ hard to find modern and easy to understand articles on how to get started (with the exception of zprint's usage docs, but that doesn't really cover implementation ideas). This area is either dominated by 40 year old journal articles, or discussions around really difficult formatting projects. There isn't a whole lot in the vain of _here's how I made a decent, simple formatter_, so maybe it'll help someone else (even if it's teaching them potentially what not to do!)
## Architecture / Overview
Atleast for me, it helps to understand the path code takes as it flows through the formatting process. Originally I tried to do as much at once for sake of efficiency but this just makes things incredibly hard to reason about and fix bugs. So like any problem that is complex, break it down, the formatter goes through many distinct phases chipping away at the problem:
b1(Recursively Iterate each Node) --> b2{Form head has\na predefined config?}
b2 --> |yes|b3(Fetch it and set the config on the node)
b2 --> |no|b4(Apply a default configuration)
end
subgraph top3 [Apply Formatting]
direction TB
c1(For each node, recursively build up a\nlist of source code lines, no indentation) --> c2(Consolidate the first two lines if hanging\nor if the config demands it)
c2 --> c3(Append indentation to each line as appropriate\nand surround each form with relevant parens)
end
subgraph top4 [Finalize Lines]
direction TB
d1(Concatenate all lines with whatever\nfile ending the file started with) --> d2(If we're formatting a file append a trailing new-line\n otherwise we are done)
end
top1-->|FormatterTree|top2
top2-->|FormatterTree with Configurations|top3
top3-->|String List|top4
```
### Build a Formatting Tree
At the end of the day, formatting is about changing the location and amounts of white-space in a file. Meaning it would be ideal if we could start with a representation that is whitespace-agnostic. Some formatters (like cljfmt) choose to use the original whitespace as an input into the formatting process, essentially meaning that your code should already be somewhat-formatted. This is somewhat advantageous as it can simplify the formatter's implementation or be a bit more flexible as to what the original programmer intended (assuming they knew what they were doing). But of course it has one fatal flaw -- the code has to already be well formatted.
For our formatter I wanted it to be able to take in horribly, ugly, minified code and output an optimal result -- especially since so much of our code gets spat out by another tool. To do that, we need to be able to format the code without any existing whitespace helping us, adding in the whitespace from the ground up.
The easiest way to do this is to construct an AST of the language which has a lot of other benefits for building language features but for here all we care about is getting rid of the whitespace and having some basic interpretation of the source code structure. The easiest way now-a-days to make an AST for a language is to create a [treesitter grammar](https://github.com/open-goal/tree-sitter-opengoal).
#### Consolidate AST Nodes and Collect Metadata
One downside with using an AST, atleast with the way the grammar is currently implemented is it creates a somewhat difficult tree to work with from a formatting perspective. For example in a LISP, everything is a list. In our AST it does not look like how you may intuitively expect:
`(+ 1 2)`
```mermaid
flowchart TB
a("source\n(+ 1 2)") --> b("list_lit\n")
b --> c("(") --> ca("(")
b --> d("sym_lit\n+") --> da("+")
b --> e("sym_lit\n1") --> ea("1")
b --> f("sym_lit\n2") --> fa("2")
b --> g(")") --> ga(")")
```
vs what you would probably prefer:
```mermaid
flowchart TB
a("top-level") --> b("list\n- surrounded in ()")
b --> da("+")
b --> ea("1")
b --> fa("2")
```
As you can see from even this simple example, the AST representation can be very verbose and in a sense has duplication. Each node is aware of it's original location in the parsed source-code, but it's the leaf-node tokens that we care about for formatting and perhaps the form types they were a part of (ie. were they comments or in a list, etc).
So to keep things brief, the AST representation is a bit too cumbersome and superfluous to use for formatting directly. So the first step is recurse through it and optimize it, removing unneeded height and capturing crucial metadata that is actually relevant for formatting.
We build a `FormatterTree` which consists of `FormatterNode`s. Each `FormatterNode` has either a single token, or a list of `refs`. A `ref` is just another `FormatterNode`. Each node also has a bunch of metadata and configuration on how it should be formatted (just initialized for now).
Having a dead-simple representation of the code like this is really only possible because it's a LISP, but it's also future proofed for us. If OpenGOAL ever decides to support different brackets -- for example in Clojure a vector uses `[]` or a map uses `{}` it would be easy for update the formatter to wrap each list node in the right bracket type.
### Applying Formatting Configuration
Now that we have our simplified tree, we can get to actually formatting. The first step is to figure out _how_ we intend to format things. By separating this from the actual formatting it's a lot easier to reason about and find bugs. You can also do potentially interesting things, where you do a full traversal of your source-code tree to figure out an optimal decision without trying to build up the final formatted string at the same time.
It is lists, or `form`s that we care about figuring out how to format. Each `form` has a head element (unless it's empty) that we can use to specify how it would be formatted. For example we cannot specify the formatting for every single function name, but we can format a `defun` or a `let` in a special way. With this mechanism, the formatter is easily extensible and configurable. There is even the ability to specify formatting configuration for a list's children by index -- for example you may want to format the first element in a `let` (the binding list) differently than the rest of the code.
There are also some forms that don't have a head-item, that we still identify and set some default formatting rules:
- The top-level node - indentation is disabled
- Lists of only literals, called `constant lists` - These are indented with `1` space instead of the default `2` for a `flow`'d form.
- If these lists literals are paired with keywords we call these `constant pairs` - We strive to put the pairs on the same line, if they are too long we spill to the next with `2` space indentation
- TODO - cond/case pairs (similar to constant pairs)
This brings up two important terms that you will see in the formatter implementation a lot -- `flow` and `hang`. The difference is as such:
```clj
(println "this"
"form"
"is"
"hung / hanged")
```
```clj
(println
"this"
"form"
"is"
"flowed")
```
The difference between the two other than visually is that hanging always results in 1 less line spent _because we don't care about line-length_. This is great because it means we can always default to hanging unless overridden. Flowing trades off that 1 vertical line for horizontal space and looks great for block-style forms. Like the top-level of a `defun` or the content of a `let`.
### Applying Formatting
The final real-step in the process, actually using the configuration and generating the formatting code. This phase is actually broken up into 3 sub-phases as we recurse through the `FormatterTree`
#### Build Lines
First we build the source code lines without indentation, it's a lot easier to figure out the indentation after all tokens are on the correct lines instead of doing it in tandem. Most tokens are given their own line, but in the case of a hung form we combine the first two elements on the first line. Other forms are formatted in even more complicated ways, or comments that were originally inline should remain next to that token.
Secondly, once we have all our consolidated lines we can apply the indentation. This is most often based on the configuration of the list node itself. For example a `hung` form must be indented the width of a `(` and the length of the head-form, plus the whitespace separator -- conversely a `flowed` form only needs to be indented by `2` spaces, regardless of token length. Because all of these algorithms are recursive we can hyper-focus on getting the indentation and formatting correct from the bottom-up. In otherwords, we don't care if a form is 200 spaces indented into a function, once the lines are returned recursively, they will get another layer of indentation added on until we hit the top-level node.
### Finalize Lines
Finally, we concatenate our list of strings, respecting the original file's line-endings.
As it is best practice, if we are formatting an entire file we should terminate with a trailing new-line.
Both of these items are technically a TODO.
## Configuration
TODO
## Reference
This formatter is highly influenced by a lot of other existing formatters and writings. The most significant of which is Clojure's [zprint](https://github.com/kkinnear/zprint/tree/main/doc/options) which has incredible documentation around it's capabilities and why it chooses what to do.
The formatter I've built basically copies zprint in terms of basic functionality, but simplifies things by being even more opinionated (for example, doing far less in terms of trying to adhere to a line-length limit).
### Other references
- [The Hardest Program I’ve Ever Written - Dart Formatter](https://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/)
- [cljfmt](https://github.com/weavejester/cljfmt)
### Why not respect a line-length limit
The OpenGOAL formatter will respect line-length in only very specific situations:
- When determining if a form can be fit entirely on a single line
- Probably in the future for multi-line comments
Besides it adding a massive amount of complexity, even zprint explains that a line-length limit won't always be followed
> zprint will work very hard to fit the formatted output into this width, though there are limits to its effort. For instance, it will not reduce the minimum indents in order to satisfy a particular width requirement.
Let's illustrate with an example:
```clj
(defunsome-func
(+ 123
456
(+ 123
456
;; imagine this form exceeds our width, how can you "fix" this without
;; messing up the indentation levels of the entire expression
;; - you can't inline it (you're already past the width)
;; - you can't insert a line break because it'll still be at the same horizontal level
(+ 123
456
789))))
```
In LISP code, we don't have many ideal options to deal with this, all options are syntactically correct, but in terms of conventional readability they are bad because when reading LISP code you want to use the indentation as a guide (which is why new-line parens are considered ugly by basically all LISP style guides)
The other reason which is equally relevant is it is hard, it means the formatter would have to try to find an optimal solution by exploring different solutions until it finds the one that minimizes lines.
But with an opinionated formatting style you can avoid this complexity, if you want your code to look nice don't write code like the decompiler sometimes outputs when it encounters something truly cursed:
The event profiler is a tool to analyze timing of multiple frames. Unlike sampling-based profilers, this profile captures an exact timeline of what happens of what happens when.
## Capturing a profile
In the OpenGOAL window, click "Profiler" and check "Record" to start recording. The buffer has a fixed maximum size and it will automatically overwrite old data once it is full.
When something interesting happens, click the "dump to file" button to save the buffer (currently a few seconds) to `prof.json` in `jak-project`.
The idea is that you can leave this running as you play, and then when the game stutters or does something interesting, you can click the dump button and get the result.
## Viewing a profile
Open Google Chrome and go to `chrome://tracing`. Then click load and open the json file. Or, just drag and drop the file into chrome.
Press `1` for a box drawing tool. This lets you select a region of the flame chart and get a list of events inside the box.
Press `2` for panning.
Press `3` for zooming.
## Adding an event
The GOAL kernel automatically adds events for each process. If you want to add another event, you can use `(with-profiler "name-of-event" <body>)`. Do not call `suspend` or do a `return` inside of this. If you need more control over stopping/starting, there are functions in `gcommon.gc` to explicitly start/stop events. But you must match them up correctly yourself!
In C++, the graphics profiler automatically adds a profiler nodes as events. To add an event, you can use
```auto p = scoped_prof("name-of-event");```
The event is active from this call until the destruction of `p`.
## Multiple threads
The event profiler currently works on both the graphics and EE threads. Adding the events can safely be done from any thread, but enable/disable/dump should be done from a single thread at a time.
Each thread should periodically insert a `ROOT` instant event when there are no active range events. This is required to make the retroactive dump feature work properly as the event buffer does not capture the tree structure fully, and it must be able to find a point in time when no events are active.
Loaded 100 of 12801 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
water
