Currently PS2 Actor Vis and FPS are only enforced when starting a run -
this enforces them on every frame similar to cheats.
In the progress menu, FPS is already disabled in speedrunner mode - this
adds the same restriction for PS2 Actor Vis.
Jak 2 already does this properly, no change needed there
This PR does two main things:
1. Work through the main low-hanging fruit issues in the formatter
keeping it from feeling mature and usable
2. Iterate and prove that point by formatting all of the Jak 1 code
base. **This has removed around 100K lines in total.**
- The decompiler will now format it's results for jak 1 to keep things
from drifting back to where they were. This is controlled by a new
config flag `format_code`.
How am I confident this hasn't broken anything?:
- I compiled the entire project and stored it's `out/jak1/obj` files
separately
- I then recompiled the project after formatting and wrote a script that
md5's each file and compares it (`compare-compilation-outputs.py`
- The results (eventually) were the same:
> This proves that the only difference before and after is non-critical
whitespace for all code/macros that is actually in use.
I'm still aware of improvements that could be made to the formatter, as
well as general optimization of it's performance. But in general these
are for rare or non-critical situations in my opinion and I'll work
through them before doing Jak 2. The vast majority looks great and is
working properly at this point. Those known issues are the following if
you are curious:
People seem to be translating lines that aren't in the base english one,
such as `mtn-plat-buried-rocks-a`
This is fine, but Crowdin will continue to remove these every sync PR
because they aren't in the base english file. So some kind of
segregation needs to happen.
If we didn't want these scenes translated, then they should be banned
from being translated via the editor / etc in the first place (shouldn't
have been included in the metadata).
This adds support for generating collide meshes when importing custom
models. A couple of things to keep in mind:
- A single `collide-mesh` may only have up to 255 vertices.
- When exporting a GLTF file in Blender, a `collide-mesh` will be
generated for every mesh object that has collision properties applied
(ideally, you would set all visual meshes to `ignore` and your collision
meshes to `invisible` in the OpenGOAL plugin's custom properties).
- Ensure that your actor using the model properly allocates enough
`collide-shape-prim-mesh`es for each `collide-mesh` ([example from the
original game that uses multiple
meshes](https://github.com/open-goal/jak-project/blob/f6688659f2ef85f5ceaacea6271580c9f4d91ed1/goal_src/jak1/levels/finalboss/robotboss.gc#L2628-L2806)).
~One annoying problem that I haven't fully figured out yet (unrelated to
the actual functionality):
`collide-mesh`es are stored in art groups as an `(array collide-mesh)`
in the `art-joint-geo`'s `extra`, so I had to add a new `Res` type to
support this. The way that `array`s are stored in `res-lump`s is a bit
of a hack right now. The lump only stores a pointer to the array, so the
size of that is 4 bytes, but because we have to generate all the actual
array data too, the current `ResLump` code in C++ doesn't handle this
case well and would assert, so I decided to omit the asserts if an
`array` tag is present and "fake" the size so the object file is
generated more closely to how the game expects it until we figure out
something better.~
This was fixed by generating the array data beforehand and creating a
`ResRef` class that takes the pointer to the array data and adds it to
the lump.
This does a couple of things:
- The `custom_levels` folder was renamed to `custom_assets` and contains
`levels`, `models` and `texture_replacements` folders for Jak 1, 2 and 3
in order to keep everything regarding custom stuff in one place.
- With this, texture replacements now use separate folders for all games
- A build actor tool was added that generates art groups for custom
actors
- Custom levels can now specify what custom models from the `models`
folder they want to import, this will add them to the level's FR3.
- A `test-zone-obs.gc` file was added, containing a `test-actor` process
that uses a custom model as an example.
The build actor tool is still very WIP, the joints and the default
animation are hardcoded, but it allows for importing any GLB file as a
merc model.
For now, this just adds sky (clouds and fog), darkjak, and skull gem.
There are some unknown issues with drawing the skull gems still, but I
think it's unrelated to texture animations.
Also fixes https://github.com/open-goal/jak-project/issues/3523
Adds a quick perf report feature to `goalc` that lets you compare how
much faster / slower it takes to compile the projects, with some simple
features like filtering the files, adjusting for how large of a margin
of error in the speeds you care about, and which test iteration you want
to compare against.
This is something I plan to use as I work more in `goalc` as an easy way
to track / show the results.
Relates to #1353
This adds no new functionality or overhead to the compiler, yet. This is
the preliminary work that has:
- added code to the compiler in several spots to flag when something is
used without being properly required/imported/whatever (disabled by
default)
- that was used to generate project wide file dependencies (some
circulars were manually fixed)
- then that graph underwent a transitive reduction and the result was
written to all `jak1` source files.
The next step will be making this actually produce and use a dependency
graph. Some of the reasons why I'm working on this:
- eliminates more `game.gp` boilerplate. This includes the `.gd` files
to some extent (`*-ag` files and `tpage` files will still need to be
handled) this is the point of the new `bundles` form. This should make
it even easier to add a new file into the source tree.
- a build order that is actually informed from something real and
compiler warnings that tell you when you are using something that won't
be available at build time.
- narrows the search space for doing LSP actions -- like searching for
references. Since it would be way too much work to store in the compiler
every location where every symbol/function/etc is used, I have to do
ad-hoc searches. By having a dependency graph i can significantly reduce
that search space.
- opens the doors for common shared code with a legitimate pattern.
Right now jak 2 shares code from the jak 1 folder. This is basically a
hack -- but by having an explicit require syntax, it would be possible
to reference arbitrary file paths, such as a `common` folder.
Some stats:
- Jak 1 has about 2500 edges between files, including transitives
- With transitives reduced at the source code level, each file seems to
have a modest amount of explicit requirements.
Known issues:
- Tracking the location for where `defmacro`s and virtual state
definitions were defined (and therefore the file) is still problematic.
Because those forms are in a macro environment, the reader does not
track them. I'm wondering if a workaround could be to search the
reader's text_db by not just the `goos::Object` but by the text
position. But for the purposes of finishing this work, I just statically
analyzed and searched the code with throwaway python code.
This adds hfrag, but with a few remaining issues:
- The textures aren't animated. Instead, it just uses one texture.
- The texture filtering isn't as good as at it could be.
I also cleaned up a few issues with the background renderers:
- Cleaned up some stuff that is common to hfrag, tie, tfrag, shrub
- Moved time-of-day color packing stuff to FR3 creation, rather than at
level load. This appears to reduce the frame time spikes when a level is
first drawn by about 5 or 6 ms in big levels.
- Cleaned up the x86 specific stuff used in time of day. Now there's
only one place where we have an `ifdef`, rather than spreading it all
over the rendering code.
This fixes issues with certain Jak 3 levels not rendering because there
is a mismatch between the DGO name, nickname and real level name (bsp
name).
FR3s use a different filename, so you can delete the ones you have after
this is merged.
This affects custom levels, but I don't have that toolchain set up so
someone else will have to test that.
This is primarily driven for proper mod-support. Mods would like to
isolate their settings and saves (potentially) and that is currently
done by find-and-replacing code before building. Bad!
Additionally, this has the side-effect of allowing for portable
installations of the game so, win-win.
Testing in progress, i'll merge once it is ready.
Patching up the extractor while working on the launcher, fixes:
- makes it so you can compile successfully given a folder path
(currently assumes your project path contains `iso_data`)
- ignore `buildinfo.json` from validation code.
- fixes an edge-case that could recursively fill up your entire
hard-drive!
- allows overriding the decompilation configuration via flag
- adds a way to specify where the ISO should be extracted to
I noticed that jak 3's compilation was spending a lot of time accessing
the `unordered_map`s we use to store constants and symbol types.
I repurposed the `EnvironmentMap` originally made for GOOS for this. It
turns out that we were copying the entire constant map whenever we
encountered a `deftype`, and fixed that too.
This speeds up jak3 compiles from ~16 to 11 seconds for me.
This includes all the collision stuff needed to spawn `target`,
decompiles the sparticle code and adds some of the PC hacks needed for
merc to run (it doesn't work quite right and looks bad, likely due to a
combination of code copied from Jak 2 and the time of day hacks).
There are a bunch of temporary hacks (see commits) in place to prevent
the game from crashing quite as much, but it is still extremely prone to
doing so due to lots of missing functions/potentially bad decomp.
---------
Co-authored-by: water <[email protected]>
This was hiding options for people that only had 60hz monitors since the
two menus drifted -- it is not necessary for an entire separate menu,
just disable the option if applicable.
Don't merge this without testing it throughly
The `test-play` macro is back, though it doesn't call `play` yet. We can
at least load all of `game.cgo`, which involves loading a lot of the
code we've decompiled, loading/linking objects files compiled by
OpenGOAL (like dir-tpages), and loading/linking Jak's art-groups (for
jak 3 they are stored v5 format that I added to the linker).
There were no major issues - just a few forgotten mips2c entries and
minor bugs/functions that needed stubs. Most of the work was updating
the linker. Hopefully I'll never have to touch that code again - I think
it supports everything we need for jak 3!
- `fma-sphere`
- `prim-beam-h`
- `cam-start`
- `ragdoll`
- `light-trails-h`
- `light-trails`
- `menu`
- `water`
- `water-flow`
- `hud`
- `hud-classes`
- `progress`
- `progress-draw`
---
The `get-texture` macro replaces calls to `lookup-texture-by-id` and
`lookup-texture-by-id-fast`. The `defpart` macro detection was modified
to print a pair like `(texture-name tpage-name)` for the texture field
that gets turned into a `texture-id` constant. Only used in Jak 3 at the
moment, I'll probably go through the other games at a later point.
Small fix to decompiler when the original compiler uses `ra` register.
This seems to happen in "normal" code very rarely - perhaps they
manually specified this in an `rlet`.
Start figuring out buckets/textures for Jak 3.
The foreground code is not yet modified for PC - I want to wait until
the game is running, since it is too hard to do it now.
Reverse engineer the skinning matrix calculation and port to GOAL. This
is about 3x faster than the MIPS2c version.
As usual, there is a `*use-new-bones*` flag to go back to the old
version.
Fix for a bug in the compiler's `.div.vf` implementation (only happens
if src/dst are the same), and fix for a typo in the register allocator
that would sometimes cause it not to consider xmm8-xmm15.
This is not a fix, but it temporarily works around the potential crash
discussed here https://github.com/open-goal/jak-project/issues/2988
As long as this doesn't impact other things, this feels like a good
interim fix, as having a broken cutscene is better than a fatal crash
and it might be some time before this is properly identified and fixed.
The original buildcache action has been archived, and buildcache itself
has also moved to github.
I forked the action and was updating it to work, but then the latest
buildcache release once again breaks support for older runners (in our
case, macos-12 is the lowest, which is acceptable as 11 is EOL).
Either way, switching to sccache is not only a way cleaner solution but
it is also well maintained (and works!)
This updates `fmt` to the latest version and moves to just being a copy
of their repo to make updating easier (no editing their cmake / figuring
out which files to minimally include).
The motivation for this is now that we switched to C++ 20, there were a
ton of deprecated function usages that is going away in future compiler
versions. This gets rid of all those warnings.
- `speech`
- `ambient`
- `water-h`
- `vol-h`
- `generic-obs`
- `carry-h`
- `pilot-h`
- `board-h`
- `gun-h`
- `flut-h`
- `indax-h`
- `lightjak-h`
- `darkjak-h`
- `target-util`
- `history`
- `collide-reaction-target`
- `logic-target`
- `sidekick`
- `projectile`
- `voicebox`
- `ragdoll-edit`
- most of `ragdoll` (not added to gsrc yet)
- `curves`
- `find-nearest`
- `lightjak-wings`
- `target-handler`
- `target-anim`
- `target`
- `target2`
- `target-swim`
- `target-lightjak`
- `target-invisible`
- `target-death`
- `target-gun`
- `gun-util`
- `board-util`
- `target-board`
- `board-states`
- `mech-h`
- `vol`
- `vent`
- `viewer`
- `gem-pool`
- `collectables`
- `crates`
- `secrets-menu`
Additionally:
- Detection of non-virtual state inheritance
- Added a config file that allows overriding the process stack size set
by `stack-size-set!` calls
- Fix for integer multiplication with `r0`
- Fixed detection for the following macros:
- `static-attack-info`
- `defpart` and `defpartgroup` (probably still needs adjustments, uses
Jak 2 implementation at the moment)
- `sound-play` (Jak 3 seems to always call `sound-play-by-name` with a
`sound-group` of 0, so the macro has been temporarily defaulted to use
that)
One somewhat significant change made here that should be noted is that
the return type of `process::init-from-entity!` was changed to `object`.
I've been thinking about this for a while, since it looks a bit nicer
without the `(none)` at the end and I have recently encountered init
methods that early return `0`.
Removes trailing whitespace from goal_src files, eventually the
formatter will do this as well but it's not ready yet so this is a
decent interim solution.
A competent text editor will also do this / flag it for you.
During level extraction, the last 4 characters of the level name are
always removed, because it is assumed that '-vis' is there. But it isn't
always there. This is especially true in post-TPL games.
This causes multiple problems:
- There can be levels in the extraction that will miss their last 4
characters from their name, which is sad, and may make it harder to
identify them.
- If there are '-vis'-less levels whose names are identical apart from
the last 4 characters, the extractor will only get the last one (it
probably extracts all but overwrites everything but the last one). For
example 'ctyasha' and 'ctykora'.
This issue affects the glb extraction and the entities json extraction.
I personally think that just keeping the -vis in the name would be the
best solution, but I guess there was a reason why it was decided that it
should be removed from the name. So to adapt to this, my implementation
will still remove '-vis' from the name, but only if it is actually in
the name - otherwise it won't remove anything.
I hope my changes didn't break anything. Extraction seemed to run fine
after my changes, and I was able to see both ctyasha and ctykora json
files. And didn't see any '-vis', so it is still properly removed.
---------
Co-authored-by: Tyler Wilding <[email protected]>
For example, `AppData/OpenGOAL/jak2/features/speedrun-categories.json`
is defined as such:
```json
[
{
"cheats": 0,
"completed_task": 0,
"continue_point_name": "",
"features": 0,
"forbidden_features": 992,
"name": "Gunless",
"secrets": 0
},
{
"cheats": 1,
"completed_task": 29,
"continue_point_name": "ctypal-shaft",
"features": 1024,
"forbidden_features": 0,
"name": "Turbo Jetboard - After Praxis 1",
"secrets": 0
}
]
```
> These entries can be created using the in-game menu as well.
https://github.com/open-goal/jak-project/assets/13153231/9b17a116-4aa9-40ad-b9f5-02b04e0ad4f3
---------
Co-authored-by: dallmeyer <[email protected]>
The game stores the last 3 frames of input (50ms of time at 60fps) and
often checks if a button was pressed within those 3 saved frames as a
condition.
When transitioning states, it often checks if some input was received
during the previous state (within those 3 frames) in order to quickly
transition out. A good example of this is when transitioning to
standing, it checks if you can jump this frame and if you had pressed X
recently, and if so, transition immediately to jump. This allows
transitions between states to feel more smooth/forgiving by letting you
jump at a later time when you are transitioning from falling->standing
than if you were only falling.
At 165fps the last 3 frames is only 18ms of time so the input windows
for these smooth transitions are almost 3x shorter.
This PR saves 15 input frames (enough to cover 50ms of time at 300fps)
for each controller and adds a (recently-pressed?) macro that checks all
15 frames. However, it only updates the necessary frames in history
based on the current frame rate. This way, 60fps continues to only check
against 3 input frames, 165fps checks against 9, 240fps checks against
12, and 300fps checks all 15.
---------
Co-authored-by: Tyler Wilding <[email protected]>
It was narrowed down recently that a lot of people have issues with the
controller input because of Steam Input working as intended. Steam Input
can be configured to replicate controller inputs as keyboard inputs (for
example, pressing X on your controller presses Enter on the keyboard).
This results in the problem of "jumping pauses the game" and similar
issues. This is a consequence of the intended behaviour of the game
listening to all input sources at the same time.
Since the vast majority of players are using controllers over keyboards,
it makes sense to disable the keyboard input by default to solve this
problem. However that makes things awkward for users that want to use
the keyboard (how do they enable the setting). The solution is a new
imgui option in the settings menu:
**Known issue that I don't care about** -- in Jak 1's menu code, since
the flags are controlled by pointers to values instead of a lambda like
in jak 2, the menu won't update live with the imgui option. This has no
functional impact and I don't care enough to fix it.
I also made the pc-settings.gc file persist on first load if the file
wasn't found. Hopefully this helps diagnose the support issues related
to the black screen.
# Why not just ignore the keyboard inputs for a period of time?
This won't work, the keyboard is polled every frame. Therefore if you
hold down the X button on your controller, steam is continuously
signaling that `Enter` is held down on the keyboard.
Yes it would be possible to completely disable the keyboard while the
controller is being used, but this defeats the purpose of creating an
input system that allows multiple input sources at the same time.
With an explicit option, not only can the user decide the behaviour they
want (do they want the keyboard ignored or simultaneously listened to)
but we avoid breaking strange edge-cases in usage leading to never
ending complexity:
- ie. imagine steam input sends events to the mouse, well you can't
disable the mouse while using the keyboard because most times people are
using mouse and keyboard
- ie. a user that wants to hold a direction with the keyboard and press
buttons on the controller in tandem (something i frequently do while
TAS'ing, to move in a perfect straight line)
The bind carried forward from Jak 1 is annoying -- R1 shoots the gun.
Allow the user to use whatever button combination they want by modifying
it in the `pc-settings` file.
```clj
(controller-led-status? 1360729)
(speedrunner-mode-custom-bind 4098)
```
- Wired up the menu settings to change the settings in game, not just on
boot
- Removed all the duplication in the game options menu code
- Fixed the mouse code so that it properly brings the virtual analog
stick back to neutral when the mouse stops
- Extended the sensitivity min/max for those that want to ensure the
slightest movement maxes out virtual analog stick.
While trying to narrow down why sometimes SDL takes 20-40seconds to
initialize I built up some more profiling features.
TLDR - I still don't know why SDL is taking a long time but I've
narrowed it down to it initializing the `GAME_CONTROLLER` subsystem.
This isn't unprecedented, I found numerous github issues and articles
suggesting this is the problem:
I imagine it is hardware/OS related on some level, there are even some
recent commits in SDL that have made it worse on certain platforms. I've
had this problem myself so I will hope to get it again soon so i can
debug where in the SDL code the delay occurs and make a proper bug
report. Hopefully this helps but it's not yet confirmed -
https://github.com/open-goal/jak-project/pull/3384
This didn't really pan out as I hoped since files can have thousands of
these and there is a limit to how many diagnostics are shown (starting
from the top of the file). This also may have caused the recent
instability in the LSP, disabling this i wasn't able to reproduce an LSP
crash cycling through 50-100 files.
I finally read through all the joint code and wrote up some
documentation. I think this will be really helpful when we try to
understand all the functions in `process-drawable`, or if somebody ever
wants to import/export animations.
This switches all three games to using a new faster GOAL joint
decompressor. It is on by default, but you can go back to the old
version by setting `*use-new-decompressor*` to false.
Also fix the log-related crash, fix the clock speed used in timer math.
- `transformq`
- `wind-work`
- `progress-static`
- `merc-vu1`
- `emerc-vu1`
- `merc`
- `emerc`
- `cloth-h`
- Most of `cloth`, not added to gsrc yet
Art group dumps were incorrect for Jak 3 because the master art group
fields were at different offsets.
- `entity-h`
- `eye-h`
- `simple-sprite-h`
- `sprite-h`
- `cam-debug-h`
- `cam-update-h`
- `cam-interface-h`
- `hud-h`
- `actor-link-h`
- `camera-h`
- `progress-h`
Also fills out a lot of the `text-id` enum. `progress-static` is also
very close to ready, only being blocked by #3361.
Decomp aligner-h, joint-h, prim-h, debug-h, game-h, penetrate-h,
bones-h, foreground-h, scene-h, script-h.
Also, has a little bit of the joint-mod-h file, but it's not finished
yet.
- `pat-h`
- `engines`
- `res-h`
- `res`
- `fact-h`
- `game-info-h`
- `wind-h`
- `merc-h`
- `shadow-vu1-h`
- `shadow-cpu-h`
- `dynamics-h`
- `memcard-h`
- `surface-h`
- part of `gui-h`
- `ambient-h`
- `speech-h`
- `prototype-h`
- `smush-control-h`
- `generic-merc-h`
- `generic-work-h`
- `collide-func-h`
- `collide-mesh-h`
- `collide-shape-h` (only missing the `new` method for `collide-shape`
because we don't have `process-drawable` yet and that also needs joint
stuff etc.)
- `collide-touch-h`
- `collide-edge-grab-h`
- `lightning-h`
This also adds argument name remaps for the `relocate` and `mem-usage`
methods (Jak 1 and 2 ref tests were updated, but not the gsrc).
Added a new `rip_collision` flag to the decompiler config and wired it
up. Cleaned up the comments around these fields in the JSON too for
clarity
Also refactored a bunch of the related extract functions to read from
`config` object rather than pass down a bunch of booleans
types-h, vu1-macros, gsound-h, dma-h, video-h, vu1-user-h, profile-h
The `gsound-h` is very simple, but the rest have decent docs and all the
macros we had from jak 2.
So far, barely any differences! (there's a few in gsound-h)
Adds two new lump types: `cell-info` and `buzzer-info`.
`cell-info` takes a `game-task` enum value and defines an `eco-info`
lump for power cells.
`buzzer-info` takes a `game-task` enum value and an integer and
automatically calculates the required value for the buzzer task with the
formula `task + buzzer * (1 << 16)`.
Decompile `gcommon`. I adjusted the spacing of docstring comments, and
removed some spammy decompiler warning prints.
I also added some random notes I had on VU programs from jak1/jak2. They
are not polished, but I think it's still worth including since we'll
have to go through them again for jak 3.
- `vector-h`
- `gravity-h`
- `bounding-box-h`
- `matrix-h`
- `quaternion-h`
- `euler-h`
- `transform-h`
- `geometry-h`
- `trigonometry-h`
- `transformq-h`
- `bounding-box`
- `matrix`
- `matrix-compose`
- `transform`
- `quaternion`
- `euler`
- `trigonometry`
Not a whole lot of changes, just a couple of new functions and one new
file (`matrix-compose`).
In #3300, `per_object` in `ObjectFileDB::ir2_analyze_all_types()` was
changed to an `unordered_map`, which messed with the object file order
in `all-types`. This fixes the issue by going through each object file
in order and then looking up the type info from the map instead of
iterating through the map.
This also removes the method IDs from the old method comments and adds a
stub for the `new` method if there is an old method comment for it, in
order to stay consistent with the rest of the method list.
When giving entities custom actor IDs in your level JSON, it is possible
to break entity lookups by actor ID if the actors are not sorted by ID
because `entity-by-aid` expects them to be in order.
This sorts the actor list by ID before generating the level file and
also checks for any duplicates.
This sets up the C Kernel for Jak 3, and makes it possible to build and
load code built with `goalc --jak3`.
There's not too much interesting here, other than they switched to a
system where symbol IDs (unique numbers less than 2^14) are generated at
compile time, and those get included in the object file itself.
This is kind of annoying, since it means all tools that produce a GOAL
object file need to work together to assign unique symbol IDs. And since
the symbol IDs can't conflict, and are only a number between 0 and 2^14,
you can't just hash and hope for no collisions.
We work around this by ignoring the IDs and re-assigning our own. I
think this is very similar to what the C Kernel did on early builds of
Jak 3 which supported loading old format level files, which didn't have
the IDs included.
As far as I can tell, this shouldn't cause any problems. It defeats all
of their fancy tricks to save memory by not storing the symbol string,
but we don't care.
This sets out the bones of a Jak 3 build, many things are stubbed out,
guessed, or copied from Jak 2 but it should at least be good enough to:
run `task set-game-jak3`
launch the repl
run builds from the repl
build outputs themselves are untested but the build itself runs without
errors
---------
Co-authored-by: Tyler Wilding <[email protected]>
Updates Zydis to it's latest commit, this should fix building the
project on intel macs with a more recent version of macOS. This likely
needs some sanity checks that the debugger stuff still works as
expected.
This is the more correct way of doing what that code is trying to do.
Fixes#3296
Also fixed some type inconsistencies with related code, probably wasn't
causing issues though.
May also fix the "black screen on startup" issues people keep having,
but that would simply be a nice bonus and isn't the aim of this PR.
This adds support for using enums in lumps using the new lump types
`enum-int32` and `enum-uint32`. Also adds these other new lump types:
- `water-height` (takes 3 meter floats, an enum and another optional
meter float)
- `eco-info` (takes an enum and an int)
- `vector3m` (3 meter floats + `w` set to 1.0)
The logger used in `goalc` tries to print an already-formatted string
`message` using `fmt::print(message);` Usually this doesn't cause
problems, but if you try to print, for example, an exception that has
special characters (notably `{`) it will try to do another round of
formatting/replacements, despite not having any args to replace with,
which ends up throwing another exception. This is why errors when
parsing custom level JSON cause the REPL to exit.
I've hopefully identified all the various instances of this across the
codebase
Currently only tracks enemy kills, and how they were killed. There is
currently no menu for this, but I've already added most of the text for
it. Also did a bunch of misc decompilation fixes and renamed some
methods.
Fixes#3277Fixes#3278
This was just not implemented end to end. There are still two notable
issues, one that I can live with, one I need to narrow down eventually:
1. Rebinding confirm buttons with trigger (ie. X) behaviour is not 100%
as it should be. I fixed it enough that I can live with it but it's
still not proper. The difficulty is that unlike a button it will
re-trigger the pressed state on the journey back to neutral (aka
unpressed).
2. If you change the bind for the confirm button, then reset your
bindings, the next confirm input is eaten. This is unrelated to these
changes but I briefly looked into it and was unable to find the root
cause.
Adds the opengoal cheats to the secrets menu. Only cheats that are fully
functional and unlockable are there right now, which is eight cheats.
This update will reset most Jak 2 settings.
Also fixes#3274 .
Boards that have no runs yet look kinda awkward but i'd rather just wait
until those fill out (won't take long) rather than add another string to
be translated.
There are art groups that are present in multiple levels, and that means
that also their textures are present in multiple levels. With texture
replacement, currently we need to make replacements for all instances if
we want it replaced everywhere, but this is not ideal, especially when
you make changes to your replacement texture and now you have to put it
in each folder again.
I added a way to replace all instances of a texture, by letting
texture-replacer people put their replacements into an '_all' folder. I
set up the logic in such a way that if you have a replacement for the
texture in its corresponding folder, it will take priority over a
replacement that you placed into the '_all' folder.
I personally found this very useful for replacing guard textures. The
guards appear in a lot of levels. But ideally you want them to look the
same everywhere. And that is why I looked into this and made a PR.
Oh and I changed what is printed in the 'Replacing ' part because it was
printing the path to our replacement, which didn't look nicely when
several textures got replaced by the same replacement from the '_all'
folder. So now it will print the original texture's page and name, I
think this information is more useful anyway.
This should avoid the stuttering due to slow unloading on some drivers.
I also turned up the amount of stuff we load per frame since nobody has
been complaining about stutters there, but there has been a few cases of
levels loading in too slowly.
(this only changes graphics, not actual GOAL level load times).
Makes the glow sprite renderer flush when full capacity is reached,
instead of at the end. Also allows us to reduce the textures used for it
(finally). Worst case scenario there's 4-5 flushes per frame.
Fixes incessant flickering in the dig.
Switches the slime look up table to be a texture, since I guess intel
drivers are terrible and putting the array in the shader makes it
extremely slow.
Also, a few minor changes:
- removed art-groups from the test-zone levels since this causes the
compiler to re-decompile the game, and makes the launcher slower. (left
it in commented out)
- Switched `decompile_code` to false by default in jak 2, in case people
run the decompiler and don't want to wait forever
- Fixed build warnings
Major change to how `deftype` shows up in our code:
- the decompiler will no longer emit the `offset-assert`,
`method-count-assert`, `size-assert` and `flag-assert` parameters. There
are extremely few cases where having this in the decompiled code is
helpful, as the types there come from `all-types` which already has
those parameters. This also doesn't break type consistency because:
- the asserts aren't compared.
- the first step of the test uses `all-types`, which has the asserts,
which will throw an error if they're bad.
- the decompiler won't emit the `heap-base` parameter unless necessary
now.
- the decompiler will try its hardest to turn a fixed-offset field into
an `overlay-at` field. It falls back to the old offset if all else
fails.
- `overlay-at` now supports field "dereferencing" to specify the offset
that's within a field that's a structure, e.g.:
```lisp
(deftype foobar (structure)
((vec vector :inline)
(flags int32 :overlay-at (-> vec w))
)
)
```
in this structure, the offset of `flags` will be 12 because that is the
final offset of `vec`'s `w` field within this structure.
- **removed ID from all method declarations.** IDs are only ever
automatically assigned now. Fixes#3068.
- added an `:overlay` parameter to method declarations, in order to
declare a new method that goes on top of a previously-defined method.
Syntax is `:overlay <method-name>`. Please do not ever use this.
- added `state-methods` list parameter. This lets you quickly specify a
list of states to be put in the method table. Same syntax as the
`states` list parameter. The decompiler will try to put as many states
in this as it can without messing with the method ID order.
Also changes `defmethod` to make the first type definition (before the
arguments) optional. The type can now be inferred from the first
argument. Fixes#3093.
---------
Co-authored-by: Hat Kid <[email protected]>
Some files were in the `banned_objects` list and were thus excluded from
the `all_objs` file.
Also implements the `pexcw` instruction which is only used in `hfrag`
code.
Fixes missing collision geometry reported in
https://github.com/open-goal/jak-project/issues/3011
The issue happens when there are 256 polygons. In this case `num-polys`
is 0 (it's a u8). There are actual cases where there are 0 polygons, so
we have to do a more complicated check to get the real count. I should
have done this in the first place, but it seemed to work...
Fixes#2994 the underport and consite icons being green (same color as
the goal, likely copy-pasta).
Fixes the hundredths timer in the race timer actually being a 60ths
timer.
Started at 349,880,038 allocations and 42s
- Switched to making `Symbol` in GOOS be a "fixed type", just a wrapper
around a `const char*` pointing to the string in the symbol table. This
is a step toward making a lot of things better, but by itself not a huge
improvement. Some things may be worse due to more temp `std::string`
allocations, but one day all these can be removed. On linux it saved
allocations (347,685,429), and saved a second or two (41 s).
- cache `#t` and `#f` in interpreter, better lookup for special
forms/builtins (hashtable of pointers instead of strings, vector for the
small special form list). Dropped time to 38s.
- special-case in quasiquote when splicing is the last thing in a list.
Allocation dropped to 340,603,082
- custom hash table for environment lookups (lexical vars). Dropped to
36s and 314,637,194
- less allocation in `read_list` 311,613,616. Time about the same.
- `let` and `let*` in Interpreter.cpp 191,988,083, time down to 28s.
This sets up the extractor for jak 2. I was expecting that I'd have to
make some more significant changes to the decompiler/compiler path
stuff, but this was not the case!
The only real change is that you can now provide multiple ISO hashes for
an entry in `ISOMetadata`. This is needed for the two different NTSC
versions, which have the same configs, serials, and ELF hashes, but
slightly different contents.
I also didn't add the korean version because I don't have the info for
it.
---------
Co-authored-by: ManDude <[email protected]>
This change adds a few new features:
- Decompiler automatically knows the type of `find-parent-method` use in
jak 1 and jak2 when used in a method or virtual state handler.
- Decompiler inserts a call to `call-parent-method` or
`find-parent-state`
- Removed most casts related to these functions
There are still a few minor issues around this:
- There are still some casts needed when using `post` methods, as `post`
is just a `function`, and needs a cast to `(function none)` or similar.
It didn't seem easy to change the type of `post`, so I'm not going to
worry about it for this PR. It only shows up in like 3 places in jak 2.
(and 0 in jak 1)
- If "call the handler if it's not #f" logic should probably be another
macro.
Fixes#805
Adds controller LED features to Jak 2:
- progressive flickering denoting health
- copies tomb simon says puzzle colors
- unique colors for each gun
- orange color for being indax
- yellow color for being in mech
- purple color for being darkjak
- blue color for being in board
- red flash when wanted.
May add more features later?
Also did some minor clean-up on some types.
This renames the method object in `defmethod`s to `this` and adds
detection for the `set-time!` and `time-elapsed?` macros.
Definitely my biggest PR yet...
- state handlers that are not inlined lambdas have smarter type
checking, getting rid of 99.9% of the casts emitted (they were not
useful)
- art groups were not being properly linked to their "master" groups.
- `max` in `ja` in Jak 2 was not being detected.
Another huge PR...
Previously, `object` and `none` were both top-level types. This made
decompilation rather messy as they have no LCA and resulted in a lot of
variables coming out as type `none` which is very very wrong and
additionally there were plenty of casts to `object`. This changes it so
`none` becomes a child of `object` (it is still represented by
`NullType` which remains unusable in compilation).
This change makes `object` the sole top-level type, and the type that
can represent *any* GOAL object. I believe this matches the original
GOAL built-in type structure. A function that has a return type of
`object` can now return an integer or a `none` at the same time.
However, keep in mind that the return value of `(none)` is still
undefined, just as before. This also makes a cast to `object`
meaningless in 90% of the situations it showed up in (as every single
thing is already an `object`) and the decompiler will no longer emit
them. Casts to `none` are also reduced. Yay!
Additionally, state handlers also don't get the final `(none)` printed
out anymore. The return type of a state handler is completely
meaningless outside the event handler (which is return type `object`
anyway) so there are no limitations on what the last form needs to be. I
did this instead of making them return `object` to trick the decompiler
into not trying to output a variable to be used as a return value
(internally, in the decompiler they still have return type `none`, but
they have `object` elsewhere).
Fixes#1703Fixes#830Fixes#928
This PR adds detection of the `launch-particles` and `seconds-per-frame`
macros to the decompiler, removing a lot of bloat and hiding many
process register uses.
I also added `og:preserve-this` comments to as many manual patches and
comments as I could, which will soon be used in conjunction with CI to
hopefully catch any regressions in future big decomp update PRs.
I have some concerns about the `launch-particles` macro (more details in
`sparticle-launcher.gc`) , but thus far, I have not seen anything break
yet.
---------
Co-authored-by: water <[email protected]>
This will create a folder like `decompiler_out/jak1/entities` and save a
JSON file per level with all the actors.
Also, it should hopefully make custom level building a little faster.
I think this is very likely to fix
https://github.com/open-goal/jak-project/issues/2970
We had somebody report a stacktrace from the debugger, and it was
immediately after calling `gen-perms`.
I found that `gen-perms` writes past the end of a stack array during
this mission, and at the same time as the reported freezes.
I was unable to recreate the original freeze after making this change.
This PR adds a frame rate option to the graphics menu for some of the
most common refresh rates.
Jak 2 has much better support for variable frame rates than Jak 1 out of
the box, but there are still some edge cases, most prominently the fact
that sprites are still limited to the 300 tick system, which is most
noticeable on glow sprites. For this, I abused the glow boost debug
setting to scale the glow based on the frame rate.
While testing, I noticed two other cases that I have also patched,
there's likely to be many more that are yet to be found, but aside from
that, the game is playable as normal.
https://github.com/open-goal/jak-project/assets/6624576/ad4db24f-cd27-4237-a155-0db7008160f3
There are potentially still some minor issues with the resulting files.
Some of them appear to have minor artifacts that playing through the
actual game do not -- but this is a much better starting point for
someone to iterate from if they are interested in improving things.
Export models with "wind". The levels with wind models are:
firecanyon (9), beach (5), village1 (7), lavatube (2).
Sometimes a single object is made up of multiple models - for example
the tree in sandover is actually several meshes.
This tries to match the original behavior of the sprite allocation
"randomness", which should reduce the number of very empty sprite blocks
sent to C++.
- fix speaker names and time frames being uneditable in Jak 1
- added toggle to auto-selected a newly created scene as current
- changed the subtitle summary format slightly.
- current scene's name now appears in the UI
- fixes flava and mode not being correct after a new music starts
playback by setting them every frame (there is at most 1 frame where
it's wrong).
- fixes city pursuit theme not playing if the city alarm was triggered
while the music wasn't playing yet.
Some general improvements for the texture animator:
- Clouds are special cased, saving about 1 ms per frame
- Adjusting the amount of clouds now actually works.
- Fixed an issue with the brightness of clouds, and the way that they
fade out around the edges.
Fixes#2167
Reduces test flakiness if ran on multiple threads and gets rid of a few
hundred files from the source tree
I believe this also makes #1434 irrelevant and it can be closed.
---------
Co-authored-by: ManDude <[email protected]>
Fix an issue where the commit sha would not use the right blending mode
if `draw-raw-image` is running at the same time.
Fix an issue where japanese subtitles would accidentally overwrite other
textures, leading to random textures missing. (in particular, glows
would disappear after watching a cutscene with the subtitles on)
- fix deci2 hang when closing the game in retail mode.
- change bigmap to always filter because the pixels look really ugly.
- don't start the game in fullscreen by default if we're debugging.
Fix the bug described in
https://github.com/open-goal/jak-project/issues/2882 where some shrubs
are transparent when they shouldn't be. The problem was that we never
carefully looked at the settings in `gs-prim`, which has a bit to
enable/disable alpha blending entirely. Now it should be correct for
both jak 1 and jak 2. To see this change, you'll need to re-extract.
Also adds a setting to disable saving texture .pngs, to speed up
decompilation. I left it on for jak 1 (to avoid confusion for texture
swapping(, but off for jak 2 for now.
Rotates the log files with a timestamp instead of copying all files and
incrementing an integer. Increases the amount of info you have when
looking at user's log files (ie. when looking at all the files, the file
creation dates are accurate).
Also simplifies the API for setting the log file, and `gk` logs are now
game specific with `jak1` or `jak2`. Which should be useful going
forward.
Lastly, added a flag to all CLIs to disable ansi colors for people that
want to do so. Though at the same time, there is finally a workaround in
jenkins to fix ANSI colors in the truncated log view -- so I'm not sure
why anyone would want to get rid of the color information. You can even
setup text editors to display the color info making log parsing much
easier. Fixes#1917
---------
Co-authored-by: ManDude <[email protected]>
The way we got/stored background matrices is a bit weird and full of
leftovers from the first attempts at porting renderers. This doesn't
work well with the Jak 2 "other camera" system where some stuff is
rendered with a different camera matrix.
This cleans most of it up. The exception is that the collide mesh
renderer and the additional sprite culling I added still need to peek at
some cached camera matrices.
This fixes the problem where etie uses the wrong matrices for "other
camera" levels. Now the "hole covers" go in the holes in the background
of the throne room.
This changes how `BlitDisplays.cpp` works so it looks at the current
render buffer, rather than the back buffer.
This approach is a bit faster because we avoid copying the back buffer
on every single frame.
It also removes the black frames when the transition starts/stops.
The remaining issues are:
- there's still a single frame of weirdness with the sprite glow
renderer.
- when changing resolutions, it doesn't work super well.
When drawing the spinning palace, there's a terrible hack that lets some
stuff be drawn with different camera matrices.
The ocean is drawn with camera-other (which spins), but was being culled
with camera (doesn't spin). This changes ocean to use the right camera
matrix.
Note that the ocean is special when it comes to camera-other - it always
uses camera-other if there is one. Other uses are set per-level and
should already be handled correctly.
This should fix a crash and animation bug in snowy. The way to trigger
the bug:
- go on ice
- move forward slowly
- stop moving forward. Reach zero speed when the frame number isn't
between 30 and 35 of the ice walking animation
- Due to this bug, the animation gets stuck at frame 60
- Take damage (due to normal lurker or ice lurker)
- Sidekick eye animation crashes because a frame number is NaN.
There was a single static path buffer being shared between multiple file
i/o threads. So sometimes you would end up using the wrong path for the
file, and getting size/data for the wrong file.
I think the original reason to have this buffer was just me being lazy
when we changed how project paths works a long time ago. It was a bad
idea in the first place.
Fixes skull gems using a low resolution texture.
Fixes issue where jak in cutscenes is dark after watching oracle-level-1
(and likely other bugs with texture animations getting stuck)
---------
Co-authored-by: ManDude <[email protected]>
The progress menu loads its icon textures from a .STR file that we were
previously ignoring.
This change:
- updates the decompiler so it can process a .STR file containing a
texture
- adds a feature to force an entire page to always be loaded in the PC
renderer by putting all textures in the GAME.FR3 file.
- regenerates the texture offset map file for jak 2 with these new
textures
For now, I've just put the icon textures in GAME.FR3. The downside is
that these will always stay on the GPU, using up VRAM even when they
aren't needed. But the entire GAME.FR3 file is under 3 MB so I think
it's ok.
- Add security wall animation
- Add waterfall animations
- Add lava animations
- Update layer values from the game to fix the security wall
- Remove leftover debug in `level.gc` that would break level-specific
animations on the second time you visited the level
- Optionally load animated slot textures to the pool so generic can use
them (fixes skull gems in UI)
This fixes the crash reported in
https://github.com/open-goal/jak-project/issues/2833
There was a memory bug here for a long time where our array of `VagCmd`
in `iso_queue.cpp` was too small. This caused GetVagCommand to return
bogus pointers, and sound code would write over other parts of memory.
Added framework to do texture animations entirely in C++. Currently only
works on relatively simple ones, and doesn't handle updating all
parameters - only the speeds.
Connected texture animations to merc and tfrag for skull gems, dark
bomb, and scrolling conveyors.
Cleaned up Tfragment/Tfrag3, which used to be two classes. This was one
of the first C++ renderers, so it had a weird design.
I havn't tested it yet, but I can almost guarantee that atleast `goalc`
will not work in the slightest!
But the project is atleast fully compiling. My hope is to start
translating some AVX to NEON next / get `goalc` working...eventually.
The main thing that was done here was to slightly modify the new
subtitle-v2 JSON schema to be more similar to the existing one so that
it can properly be used in Crowdin.
Draft while I double-check the diff myself
Along the way the following was also done (among other things):
- got rid of as much duplication as was feasible in the serialization
and editor code
- separated the text serialization code from the subtitle code for
better organization
- simplified "base language" in the editor. The new subtitle format has
built-in support for defining a base language so the editor doesn't have
to be used as a crutch. Also, cutscenes only defined in the base come
first in the list now as that is generally the order you'd work from
(what you havn't done first)
- got rid of the GOAL subtitle format code completely
- switched jak 2 text translations to the JSON format as well
- found a few mistakes in the jak 1 subtitle metadata files
- added a couple minor features to the editor
- consolidate and removed complexity, ie. recently all jak 1 hints were
forced to the `named` type, so I got rid of the two types as there isn't
a need anymore.
- removed subtitle editor groups for jak 1, the only reason they existed
was so when the GOAL file was manually written out they were somewhat
organized, the editor has a decent filter control, there's no need for
them.
- removed the GOAL -> JSON python script helper, it's been a month or so
and no one has come forward with existing translations that they need
help with migrating. If they do need it, the script will be in the git
history.
I did some reasonably through testing in Jak1/Jak 2 and everything
seemed to work. But more testing is always a good idea.
---------
Co-authored-by: ManDude <[email protected]>
## Problem
OpenGOAL uses OpenGL 4.3.
Apple stopped upgrading OpenGL after 4.1, so the way OpenGOAL currently
works will never be playable on Macs using OpenGL.
## Solution
Luckily, downgrading to OpenGL 4.1 is not a huge change (at least it
doesn't seem like it to my untrained eyes).
## Changes
* set hints for OpenGL 4.1 instead of 4.3 for __APPLE__
* skip the OpenGL debugging callback setup for macOS (requires 4.3)
* bump down the version string for all shaders
* stop using the `binding` layout qualifier in shader code
* move the `flat` qualifier first (not sure if this is a 4.1 thing or
just Macs being more strict)
* don't mix signed and unsigned ints in shaders (not sure if this is a
4.1 thing or just Macs being more strict)
* add some hacky CPP to the Shader constructor for binding texture units
and bones buffers based on variable names in the shader code
## Results
The current event-based approach is very difficult to get right, and it
depends on no events ever being missed. This changes the keyboard/mouse
handling code to a polling-based approach.
Other fixes:
- an issue where modifier keys were not able to be successfully bound
(like Left Shift to `X`)
- improves cursor hiding (except when you use the start menu, this seems
like an SDL issue, see comment)
- Better discarding of kb/mouse inputs when imgui intercepts input
- properly swap bindings when an already set key is assigned, even if it
crosses the distinction of an analog vs normal button
Fixes#2800
Changes the DGO build order so that the city gets compiled first, and a
random guess at an "order" of which levels people might edit more often.
Most of the data-only borrow files are moved to the end as well.
Also moves around files in the `goal_src` tree to a structure that makes
a bit more sense, some files were either in the completely wrong place,
their folders had strange names, were too deep for no reason or were
just too far away from other relevant files. This structure should make
it easier to guess a file's location.
Removes a leftover case for falling to back to generic-merc in jak 1.
This should fix the bug where daxter's face wasn't animated in the intro
after floating-point blerc was added.
Fixes empty boxed arrays of strings breaking some decomp
(`ctywide-speech` and `race-info`).
Adds `decomp-as` tag to decompiler types so that the static data
decompiler can use macros like `meters` and `seconds` on fields that
aren't of type `meters` or `time-frame`.
Adds `override` tag to decompiler types which overrides the type of
field with that name. The type must be a child type of the original
field's type (or the same type, but why would you do this?).
Fixes the camera being offset for `drillmtn` after loading `palout`
once.
This is a huge refactor sadly.
Fixes orb softlocks during races and other side missions.
The side mission tasks will now not count as completed until the
precursor orb has been picked up.
Races will not let you advance (or even pause the game) until the
precursor orb has been picked up.
Avoids blocking other IOP threads on IO.
Idk if it'll really help anything, but at least theoretically it might
stop some pathologically slow IO case from blocking the VAG buffer
switching.
This is a major deviation from the original game, which did not have any
way to access the nest after beating Metal Kor as the air train gets
hidden when he is beaten. This was mostly annoying because there are
precursor orbs in that level that you might miss.
This makes it so the air train can once again be used to enter the nest
even after beating Metal Kor. The rest of the level remains mostly the
same, except the Rift Ring does not spawn and an invisible wall is added
to the Metal Kor arena to prevent you from entering it as you are
normally unable to leave it anyway.
Trying to make up for some of the startup speed lost in the SDL
transition. This saves about 1s from start (from ~3s), and about 500 MB
of RAM.
- Faster TIE unpack by merging matrix groups, more efficient vertex
transforms, and skipping normal transforms on groups with no normals.
- Refactor generic merc and merc to use a single renderer with multiple
interfaces, rather than many renderers. Removed "LightningRenderer" as a
special thing, but Warp is still special
- Add more profiling stuff to startup and the loader.
- Remove `SDL_INIT_HAPTIC` - this turned out to be needed for
force-feedback steering wheels, and not needed for controller vibration
- Switched `vag-player` to use quicksort instead of the default GOAL
sort (very slow)
```lisp
(defenum collide-action
:type uint32
:bitfield #t
(solid 0) ;; used for solid things
(rider-plat-sticky 1) ;; used for platforms in rider/platform interactions
(rider-target 2) ;; used for target in rider/platform interactions
(edgegrab-active 3) ;; set/cleared when entering/exiting edgegrab states
(rider-plat 4) ;; used for platforms in rider/platform interactions
(unused 5) ;; totally unused?
(edgegrab-possible 6) ;; used when edge grab checks should be done
(edgegrab-cam 7) ;; set/cleared when entering/exiting edgegrab states
(swingpole-active 8) ;; set/cleared when entering/exiting swingpole states
(racer 9) ;; set/cleared when entering/exiting racer states
(attackable 10) ;; used for something to do with attacking/damaging
(attackable-unused 11) ;; seems to relate to attacking - set in several places but never tested for?
(snowball 12) ;; set/cleared when entering/exiting snowball states
(tube 13) ;; set/cleared when entering/exiting tube states
(flut 14) ;; set/cleared when entering/exiting flutflut states
(racer-grounded 15) ;; set/cleared when entering/exiting certain racer states w/ extra conditions
(racer-unused 16) ;; seems to relate to racer - never set, only cleared in one place?
)
```
This moves the blerc math from mips2c to the Merc2 renderer, and uses
floats instead.
We could potentially do this on the GPU, which would be even faster, but
this isn't that slow in the first place.
Adds support for adding custom subtitles to Jak 2 audio. Comes with a
new editor for the new system and format. Compared to the Jak 1 system,
this is much simpler to make an editor for.
Comes with a few subtitles already made as an example.
Cutscenes are not officially supported but you can technically subtitle
those with editor, so please don't right now.
This new system supports multiple subtitles playing at once (even from a
single source!) and will smartly push the subtitles up if there's a
message already playing:
Unlike in Jak 1, it will not hide the bottom HUD when subtitles are
active:
Sadly this leaves us with not much space for the subtitle region (and
the subtitles are shrunk when the minimap is enabled) but when you have
guards and citizens talking all the time, hiding the HUD every time
anyone spoke would get really frustrating.
The subtitle speaker is also color-coded now, because I thought that
would be fun to do.
TODO:
- [x] proper cutscene support.
- [x] merge mode for cutscenes so we don't have to rewrite the script?
---------
Co-authored-by: Hat Kid <[email protected]>
Should fix https://github.com/open-goal/jak-project/issues/2679
Here's a test program that will trigger the bug when near these guards:
```lisp
(define *cquery* (new 'global 'collide-query))
(defun test-bad-collide ()
(let ((lower (new 'static 'vector :x 1681893.8750 :y 61314.2031 :z 345208.6562 :w 1.))
(upper (new 'static 'vector :x 1701603.8750 :y 67624.0625 :z 357881.0312 :w 1.))
;(cquery (new 'stack-no-clear 'collide-query))
)
(set! (-> *cquery* collide-with) (the-as collide-spec 1))
(set! (-> *cquery* ignore-process0) #f)
(set! (-> *cquery* ignore-process1) #f)
(set! (-> *cquery* ignore-pat) (new 'static 'pat-surface :noentity #x1 :nojak #x1 :probe #x1 :noendlessfall #x1))
(set! (-> *cquery* action-mask) (collide-action solid))
(set! (-> *cquery* bbox min quad) (-> lower quad))
(set! (-> *cquery* bbox max quad) (-> upper quad))
(format 0 "doing collide...~%")
(fill-using-bounding-box *collide-cache* *cquery*)
(format 0 "have ~d and ~d~%" (-> *collide-cache* num-tris) (-> *collide-cache* num-prims))
)
(none)
)
```
As far as I can tell, there's a totally invalid collide-hash with an
inside `axis-scale.z`. On the PS2, it gets ignore because of how
float->int works for floats that are too big. On PC, it ends up using a
negative value and loop forever.
Gives proper names to almost every color. It is very apparent that some
colors are context-sensitive/made for a specific purpose, so those
colors were named after that purpose instead of a generic color name.
Normally, when they allocate a VagCmd, they do a bunch of stuff to clear
all the status bits and reset things
in particular the InitVAGCmd function does a lot
but for the stereo command, they do a lot less:
Which means that the new_stereo_command can just have random status bits
left over from whatever the last user had.
we seem to end up in a state where byte21 is set, and this causes
everything else to be wrong and off-by-one dma transfer. My guess is
that the original game avoided this bug due to lucky timing that I don't
understand.
I think the fix of just clearing byte21 is ok because there's no way
that the old value of the byte is useful after the command is
repurposed.
Also fixed an original game bug in `loader.gc` on a method that's called
quite often, though I have no clue what erroneous behavior it could have
even caused.
- elec gates now always render at max quality if you have PS2 lods
disabled. the original render distances are so low that the one in
caspad is impossible to see in normal gameplay.
- `fort-entry-gate-11` and `com-airlock-outer-13` are specifically
banned from the all actors hack because they are placed in a bad spot
and Naughty Dog did not program the airlocks very well.
- fixed NPC talk distance being bad for 1 frame.
- fix `sew-scare-grunt` erroneously keeping its spool anim active if you
killed the enemy.
I sorted the list of sources on my SDL PR to reduce future merge
conflicts, but in the meantime everytime something gets added to it I
have a pretty rough set of conflicts to resolve. Committing this early
to preserve my sanity
Saves 16 bits and lets us align the `color_index` field properly.
This shouldn't improve or decrease performance by any noticeable amount
except maybe in really low end systems.
Adds sprite distort, fixes buggy sprite rendering in progress, adds
scissoring support (used in various scrolling menus) and a very basic
implementation of `blit-displays`. This is enough to make the fade
effect in the progress menu work, along with all the menus working
properly without needing to use the REPL. This does not make screen
flipping and the filter when failing a mission work.
Added support in the decompiler for detecting `dma-buffer-add-gs-set`
and `dma-buffer-add-gs-set-flusha` and updated all of the Jak 2 code to
use it. Readability improved!
Fixes decompiler issue with `with-dma-buffer-add-bucket` not inlining
forms which broke syntax. Fixes store error warnings showing up for
non-existent stores, there is now a dedicated pass for this at the end.
I started work on making `BITBLTBUF` stuff work in the DirectRenderer,
but stopped for now because it wasn't strictly necessary. It will still
assert like before.
I'm not sure what has changed here...but rebooting in debug mode now
hangs.
- It seems to exit the GFX loop prematurely because `MasterExit` is
still set to `RESTART_IN_DEBUG`
https://github.com/open-goal/jak-project/blob/master/game/runtime.cpp#L431
- And then it gets stuck waiting for the DECI thread to close
This works, but im not sure if it's the right fix / what has changed to
require this.
This fixes a long time issue with `lambda`. The `lambda` is a bit
overloaded in OpenGOAL: it's used in the implementation of `let`, and
also to define local anonymous functions.
```
(defmacro let (bindings &rest body)
`((lambda :inline #t ,(apply first bindings) ,@body)
,@(apply second bindings)))
```
```
(defmacro defun (name bindings &rest body)
(let ((docstring ""))
(when (and (> (length body) 1) (string? (first body)))
(set! docstring (first body))
(set! body (cdr body)))
`(define ,name ,docstring (lambda :name ,name ,bindings ,@body))))
```
In the first case of a `let`, a `return` from inside the `let` should
return from the functioning containing the `let`, not the scope of the
`lambda`. In the second case, we should return from the lambda. The way
we told the different between these cases was if the `lambda` was used
"immeidately", in the head of an expression (like it would be for the
`let` macro). But, this falsely triggers when an anonymous function is
used immediately: eg
```
((lambda () (return #f)))
```
should generate and call a real x86 function that returns immediately.
This should fix some death/mission failed stuff in jak 2.
Somehow this was only causing issues (afaik) with
`draw-decoration-load-save` getting corrupted, perhaps because other
processes either use the shared dram stack or the gigantic spr stack.
Slight change to float divide operations (again). Now it only turns into
inverse multiplication if the float is a power of 2 (positive or
negative). Non-zero immediate divisors will be compiled as regular float
divisions but will forgo the extra branches and checks for divide by
zero.
Also fixes#2584
Allows for removing a line in the subtitle editor finally, also fixed
the following:
- The path the editor used didn't include the game name
- Loading the subtitle project is too slow to do on startup now
(4-5seconds in a debug build), do it on demand now
Updates the decompiler for the new format and there's new macros. This
new format should be easier to read/parse.
Also rewrote `sp-init-fields!` (both jak 1 and 2) from assembly to GOAL.
Hopefully I did not miss any regressions in Jak 1/2 while updating the
files, it's a lot.
- [x] compare NTSC-K
- [x] compare NTSC-J
- [x] compare PAL
- [x] figure out version order
- [x] ~~write delta patch for spanish text~~ no need for now
Fixes#2530
Fixes issue where warp effect looks wrong near the edge of the screen -
there was an unhandled `REGION_CLAMP` texture setting.
Fixes a potential bug where "warp page" things wouldn't be drawn at all
because there is no PC warp bucket. Unclear if anything actually fits
this category, but it doesn't hurt.
Turn on PC-format texture uploads for the water page so the precursor
guy uses the right texture. It has to use generic because it abuses the
generic death query thing to spawn particles.
Workaround for some issues with rebuilding level files after changing
engine files. Not a perfect solution, but probably good enough.
Doesn't actually do anything in Jak 2 because the collide mesh isn't
extracted, but the functionality is all there. Also updated the renderer
a bit to keep the colors more readable.
Fixes crashes when killing the big spider, killing predator metal heads,
and watching the end cutscene. The presursor guy is the wrong color, but
I don't have time to look into this today.
Fixes citizens that disappear (permanently) when being pushed into a
"wall" which is the border of a nav mesh that is exactly aligned with
coordinate axes.
This bug feels very similar to punch glitch, where code worked on PS2
only because some value didn't reach exactly 1.0 or 0.0 with ps2-style
floating rounding. It might be better to track down the source of the
"only 1.0 on PC" value, rather than patching downstream code to handle
it, but I can't easily find it in this case - there's a lot of code that
touches this heading value. (it's also possible this bug happens on ps2,
but the result is the guard appears to face the wrong direction, rather
than disappearing).
Detailed debug notes:
Ran a process that just calls this function:
```
(defun print-guard-info ()
(let ((proc (process-by-name "crimson-guard-level-42" *active-pool*)))
(when proc
(format *stdcon* "PROC: ~A~%" (-> proc state))
(let* ((pd (the process-drawable proc))
(css (-> pd node-list))
(cs (-> css data 15))
)
(format *stdcon* "joint: ~A~%" (-> cs joint))
(format *stdcon* "pos: ~`vector`P~%" (-> cs bone position))
)
)
)
)
```
it prints out the state, and the bone position for some bone that
happens to be on the upper body. It goes to NaN when the upper half
disappears, in the state `tazer-hostile`.
Modified the code in this state to call this function in a bunch of
places:
```
(defun guard-nan-debug ((guard process-drawable) (info string))
(when (string= (-> guard name) "crimson-guard-level-42")
(format 0 "[guard-nan] ~S : ~`vector`P~%" info (-> guard node-list data 15 bone position))
)
)
```
which prints the bone position to stdout.
This shows that the problem happens after `post`, but before `trans`:
```
[guard-nan] post-end : #<vector 2723306.0000 269921.0312 388825.2187 1.0000 @ #x1f1350>
[guard-nan] trans-start : #<vector NaN NaN NaN NaN @ #x1f1350>
```
this is probably as part of the bone math.
To check, I added some prints to `execute-math-engine`, before and after
the call to `do-joint-math`:
```
[guard-nan-math] pre-math : #<vector 2722236.5000 268609.5312 385339.9062 1.0000 @ #x1f1350>
[guard-nan-math] post-math : #<vector NaN NaN NaN NaN @ #x1f1350>
```
The first part of `do-joint-math` is calling the
`generate-frame-function`, which does animation blending to compute a
bunch of joint transforms. I dumped these:
```
(let ((jaf (the-as joint-anim-frame (+ 2400 (scratchpad-object int)))))
(format 0 "generate frame:~%")
(format 0 "~`matrix`I~%" (-> jaf matrices 0))
(format 0 "~`matrix`I~%" (-> jaf matrices 1))
(dotimes (i (-> obj mgeo num-joints))
(format 0 "~`transformq`P~%" (-> jaf data i))
)
)
```
and confirmed that they always look good.
The next part is "prebind", which allows something to modify the
`joint-anim-frame`. Nothing does this on the guard.
The next part is what I've named the "root bind", which computes the
transform of the root bone based on the process-drawable root's
position.
```
(when (the-as (function cspace transformq none) t9-3)
(when *djm-debug* (format 0 "djm: first bind func ptrs #x~X #x~X~%" cspace<-transformq! t9-3))
(when *djm-debug* (format 0 "djm: first bind func input: ~`transformq`P~%" (-> v1-20 param1)))
((the-as (function object object object none) t9-3) v1-20 (-> v1-20 param1) (-> v1-20 param2))
(when *djm-debug* (format 0 "djm: first bind func result:~%~`matrix`I~%" (-> v1-20 bone transform)))
)
```
in this case, the first print confirms that we're using
`cspace<-transformq!` as the root binding function.
When the guard entirely disappears, it is caused by the root of the
process drawable having NaNs as its `quat`:
```
djm: first bind func input: #<transformq @ #x1f04f0
trans:2724817.7500 262144.0000 388891.1875 1.0000
quat: NaN NaN NaN NaN
scale: 1.0000 1.0000 1.0000 1.0000>
djm: first bind func result:
[001f0e70] matrix
[ NaN] [ NaN] [ NaN] [ NaN]
[ NaN] [ NaN] [ NaN] [ NaN]
[ NaN] [ NaN] [ NaN] [ NaN]
[2724817.7500] [ 262144.0000] [ 388891.1875] [ 1.0000]
```
After this, it's possible to get the lower half of the guard to return,
but I believe the real problem is this first `quat` being NaN. Then the
nans probably spread everywhere.
So now we now to look for the `(-> guard root quat)` becoming NaN:
```
(defun guard-nan-debug ((guard process-drawable) (info string))
(when (string= (-> guard name) "crimson-guard-level-42")
(format 0 "[guard-nan] ~S : ~`vector`P~%" info (-> guard root quat))
)
)
```
and we see this happens in post somewhere
```
[guard-nan] post-start : #<vector 0.0000 -1.0000 0.0000 0.0000 @ #x1f0500>
[guard-nan] post-end : #<vector NaN NaN NaN NaN @ #x1f0500>
```
Looking at individual methods
```
[guard-nan] before-142 : #<vector 0.0000 -1.0000 0.0000 0.0000 @ #x1f0500>
heading: #<vector 0.0000 0.0000 -1.0000 0.0000 @ #x3432670>
[guard-nan] before-143 : #<vector NaN NaN NaN NaN @ #x1f0500>
heading: #<vector 0.0000 0.0000 -1.0000 0.0000 @ #x3432670>
```
It looks like the problem is converting this heading to a quaternion.
Adds a decent way to customize the folders the project file expects the
iso data and decompiler data to be in. When you run any version other
than the default, for example Jak 1 PAL, it uses the `gameName`
decompiler config to consume and output it's results.
However the project file will assume `jak1` unless you hard-code it
differently -- basically, it needs to be explicitly told just the
decompiler is told what version to use.
We now have a per-user REPL Config json file, so that can be used to
override the default `jak1` behaviour.
Fixes#1993
Cleans up every `dummy-*` and `TODO-RENAME-*` method up with either
proper names or by renaming them to `[type-name]-method-[method-id]`
similar to Jak 2's `all-types`.
Also fixes the bad format string in `collide-cache` and adds the event
handler hack to Jak 1.
The game boots and runs fine, but I might have missed a PAL patch or
other manual patches here and there, please double-check if possible.
I saw that some new speedrunning-related text was added to the English
language, so I decided to translate them. Meanwhile I also made some
minor adjustments/fixes to existing translations.
BUT I have a question...
Do we really need all this text?
They are displayed in a menu that is called 'NEW INDIVIDUAL LEVEL RUN'.
So adding IL after the level names is kind of redundant in my opinion -
the existing level name text entries could be used instead. And the
reason why I care so much is because it is hard to come up with a
"translation" for IL. But if I am forced to, I will come up with
something.
I will keep this PR a draft until I know what the main jak-project
people think about this "IL" issue. And I will adjust my branch
accordingly.
The last of the missions that had a missing file.
I manually fixed some casting related to a `handle->process`, since this
is the last file...whatever not worth stressing about. But probably an
issue that will crop up in the future.
Co-authored-by: water <[email protected]>
- decompile `neon-baron-part`, which also has the hideout door for some
reason
- improve a few error messages in static data decompilation
- fix bug with disabling fog in merc
Disables the fog hack for Jak 2, where it's not useful and kind of
breaks in most levels which rely on dark vertices that aren't underwater
(e.g. city windows).
Add the vortex renderer. The vortex texture isn't there yet (it uses the
same texture as clouds), so it uses a checkerboard. But the
colors/vertices seem right.
This way a user can take multiple data samples from one/multiple play
sessions quickly. Creates a directory called profile_data that is added
to gitignore to place the data in, and checks to see if a file is there
and if so it creates prof1.json prof2.json prof3.json and so on...
Unsure what hacks or type casts might eventually be needed for the PAL
version, but these changes are enough to get things to extract and run
the game (though the extract phase requires iso_data to be in a jak2_pal
folder, but the build still looks for jak2 in iso_data and
decompiler_out).
- better handling of the `disable-fog` settings for merc, should fix the
spotlights. There's a setting in the merc effect, and also a runtime
flag for the draw-control. I'm not actually sure what reads these, but
the draw-control one is definitely used to disable fog on the
spotlights.
- increase merc draw limit to try to fix the issue about partially drawn
citizens in the city
- remove useless debug prints (it's okay to die in init, and the medium
load buffer size mode is understood now)
Fixes some mistakes with merc draw modes. The glass in the palace level
no longer writes to the depth buffer (it's "water"):
Also fixes the one-frame flickers when level draw orders change. We
might be able to make this more efficient in the future, but this will
at least fix the frame with nothing drawn.
Actors that use types from a level that got unloaded should now get
killed immediately instead of continuing to exist and eventually crash
the game.
NOTE: Will spam the console whenever a bad actor tries to spawn. I would
like to document all instances of where this happens!
This should fix a bunch of texture-related issues by generating a table
of overlapping textures and just... adjusting them slightly so they
don't overlap. It's not the most elegant solution in the world, but I
think it's no worse than the existing hard-coded tpage dir stuff.
Definitely needs a clean up pass, but I think the functionality is very
close.
There's a few "hacks" still:
- I am using the emerc logic for environment mapping, which doesn't care
about the length of the normals. I can't figure out how the normal
scaling worked in etie. I want to do a little bit more experimentation
with this before merging.
- There is some part about adgifs for TIE and ETIE that I don't
understand. The clearly correct behavior of TIE/ETIE is that the alpha
settings from the adgif shader are overwritten by the settings from the
renderer. But I can't figure out how this happens in all cases.
- Fade out is completely disabled. I think this is fine because the
performance difference isn't bad. But if you are comparing screenshots
with PCSX2, it will make things look a tiny bit different.
There are *a lot* of file changes and while I have carefully gone
through every gsrc change to fix up manual patches, there might still be
spots that I missed.
My mistake -- testing focused too much on preserving the existing
behaviour I clearly forgot to make sure the new stuff worked properly.
Just had to early out and not modify the args if they were in the new
format.
This fixes the issue where elevators leave you behind (stack structure
related)
At the same time I cleaned up some things I found along the way, most
notably changing `script-context`'s `basic` field to `object` as
non-basics use this method as well.
Fixes#2305
An attempt to cleanup the last CLI interface we have left to cleanup.
- `gk` args now follow the typical convention ie. `--proj-path` instead
of `-proj-path`.
- args that are passed through to the rest of the application / the
game's runtime use the typical convention of following a `--`
- I'm thinking some args shouldn't be handled at this level ie
(`-nodisplay`, `-vm`, `-novm` or `-jak2`) These could be better
documented as legitimate flags and passed in via a nice struct. They
don't seem to be used in `InitParams` but I'll triple check.
There's a temporary shim here so there is no coupled release with the
launcher (right now it executes `gk` with a few args). So I just change
the old args into the new format. After one release cycle, I can change
it in the launcher and delete it here.
I am unsure if this will break the bash shellscript usages -- not sure
which args were usually passed into `$@`
Three main changes:
- Adds support for the texture scrolling effect used on conveyor belts,
and turn it on for jak 2.
- Use merc instead of generic in jak 1 for ripple/water/texscroll stuff
(non-ocean water, lava, dark eco, etc). This is a pretty big speedup in
a lot of places.
- Fix a really old bug with blending mode used to draw environment maps.
The effect is that envmaps were half as bright as they should have been.
As usual, there's a flag to go back to the old behavior on jak 1. Set
these to `#t` to use generic like we used to.
```
*texscroll-force-generic*
*ripple-force-generic*
```
The format has changed, and everything must be rebuilt (C++, FR3's, GOAL
code)
Reasons for doing so include:
1. This should stop the confusion around editing the wrong config file's
flags -- when for example, extracting a level. Common settings can be in
one central place, with bespoke overrides being provided for each
version
2. Less verbose way of supporting multiple game versions. You don't have
to duplicate the entire `type_casts` file for example, just add or
override the json objects required.
3. Makes the folder structure consistent, Jak 1's `all-types` is now in
a `jak1` folder, etc.
Support rendering eyes with merc for both jak 1 and jak 2.
For jak 1, everything should look the same, but merc will be used to
draw eyes. This means that jak is now fully drawn with merc!
For jak 2, eyes should draw, but there are still a few issues:
- the tbp/clut ptr trick is used a lot for these eye textures, so
there's a lot that use the wrong texture
- I had to enable a bunch more "texture uploads" (basically emulating
the ps2 texture system) in order to get the eyes to upload. It would be
much better if the eye renderer could somehow grab the texture from the
merc model info, skipping the vram addressing stuff entirely. I plan to
return to this.
- I disabled some sky draws in `sky-tng`. After turning on pris2
uploads, the sky flashed in a really annoying way.
This gets rid of the `auto-lt` and `auto-dbg` CLI args to `goalc` that
have been marked deprecated for a month or so at this point. The
behaviour can be replicated via the `startup.gc` file.
I also removed `decompiler_out/` and `log/` from the repo, as our C++
code is smart enough to create these directories if they are missing now
so they are superfluous
Work in progress minimap. Known issues:
- "path finding" doesn't appear to work - it gets stuck forever in many
cases
- some nasty patches around timer-based code
- jak arrow blending issues
- would be nice to make it higher resolution
if the search is forced to terminate due to iteration/time limits, the
icon is not in the right place
I didn't actually visually notice much of a difference with these hacks
unlike in Jak 1, but I also avoided checking the missions thoroughly
since the game crashes very often right now.
Some backtraces are quite large, an option is to increase your terminal
buffer -- but dumping to a file is also useful if you want to share the
crash.
I'm not crazy about the way I hacked this in, but it felt like the least
invasive way for now and I don't want to cause a regression with the
debugger. It's also nice that it dumps with ansi colors as then you can
view the backtrace with the original coloring:
Usage:
```clj
(:di "./stacktrace.log")
```
Fix crash when the game tries to spawn `jellyfish` in Underport and
temporarily disable the rendering of the warp effect until it's properly
implemented.
- Disable depth test for sprite glow (VU1 programs only `sq.xy` and
leaves `z` alone, which has Z_MAX from the template)
- Fix tfrag/tie/shrub issues when `use-camera-other` is set.
Where applicable, of course.
My system language is set to English so I actually can't test this. If
anyone has their Windows language (NOT LOCALE) set to Spanish, German,
French, Italian or Japanese please test this.
Fixes#1734
Also fixes the opengoal debugger on Windows and fixes the decomp for
`menu` which was causing some crashes related to input handling.
`*alert-level-settings*` was actually an inline array and we were
indexing past the end of it.
I think this will fix some of the strange on-foot guard behavior too.
Increase level heaps and borrow heaps. The level heap increase was
likely not needed, but better safe than sorry. We allocate the 128 MB
main heap anyway so there's no harm.
Also fix the crash when using `-boot`. As I thought it was just a
one-line typo in the kernel.
Fixes the aspect ratio of the font renderer (Jak 2 does not need the
same hacks as Jak 1), the custom letterbox/pillarbox sizes and fixes the
heap display.
---------
Co-authored-by: water <[email protected]>
Adds the `pckernel` system to Jak 2, allowing you to do the PC-specific
things that Jak 1 lets you do like change game resolution, etc.
In other to reduce the amount of code duplication for something that
we're gonna be changing a lot over time, I split it into a few more code
files. In this new system, `pckernel-h.gc`, `pckernel-common.gc`
(previously `pckernel.gc`) and `pc-debug-common.gc` are the files that
should be shared across all games (I hacked the Jak 2 project to pull
these files from the Jak 1 folder), while `pckernel-impl.gc`,
`pckernel.gc` and `pc-debug-methods.gc` are their respective
game-specific counterparts that should be loaded after. I'm not fully
happy with this, I think it's slightly messy, but it cleanly separates
code that should be game-specific and not accidentally copied around and
code that should be the same for all games anyway.
Stops the REPL crashing with `device or resource busy` when running
`(reload)`
However I think this is indicative of a bigger problem where either the
`Compiler` or prompt is not ready to handle input immediately after
creation and setting the status to `OK`.
Ports some of the custom PC port debugging code from Jak 1, such as the
entity debugger, Anim Tester X and part tester, to Jak 2 and gives the
PC menu code the same treatment Jak 1 received recently in #2216.
Some adjustments have been made to make things work in Jak 2 and some
stuff is still incomplete.
The majority of the default PC menu code has been left commented out in
preparation for @ManDude's plan to port the rest of the Jak 1 `pckernel`
features to Jak 2.
Also adds the code for `palace-scenes` to gsrc as that seems to have
been missed.
Fixes a rare crash that can sometimes happen when civilians are picking
a direction to flee in, where a branch count may be 0, causing a divide
by 0 MATH_EXCEPTION.
This automatically generates documentation from goal_src docstrings,
think doxygen/java-docs/rust docs/etc. It mostly supports everything
already, but here are the following things that aren't yet complete:
- file descriptions
- high-level documentation to go along with this (think pure markdown
docs describing overall systems that would be co-located in goal_src for
organizational purposes)
- enums
- states
- std-lib functions (all have empty strings right now for docs anyway)
The job of the new `gen-docs` function is solely to generate a bunch of
JSON data which should give you everything you need to generate some
decent documentation (outputting markdown/html/pdf/etc). It is not it's
responsibility to do that nice formatting -- this is by design to
intentionally delegate that responsibility elsewhere. Side-note, this is
about 12-15MB of minified json for jak 2 so far :)
In our normal "goal_src has changed" action -- we will generate this
data, and the website can download it -- use the information to generate
the documentation at build time -- and it will be included in the site.
Likewise, if we wanted to include docs along with releases for offline
viewing, we could do so in a similar fashion (just write a formatting
script to generate said documentation).
Lastly this work somewhat paves the way for doing more interesting
things in the LSP like:
- whats the docstring for this symbol?
- autocompleting function arguments
- type checking function arguments
- where is this symbol defined?
- etc
Fixes#2215
Some people have complained about the button combo for entering the
speedrun menu, saying it's awkward to press Start.
This change lets you press Select instead (while holding L1+R1+X). The
old combo (pressing Start) still works as well.
Moves PC-specific entity and debug menu things to `entity-debug.gc` and
`default-menu-pc.gc` respectively and makes `(declare-file (debug))`
work as it should (no need to wrap the entire file in `(when
*debug-segment*` now!).
Also changes the DGO descriptor format so that it's less verbose. It
might break custom levels, but the format change is very simple so it
should not be difficult for anyone to update to the new format. Sadly,
you lose the completely useless ability to use DGO object names that
don't match the source file name. The horror!
I've also gone ahead and expanded the force envmap option to also force
the ripple effect to be active. I did not notice any performance or
visual drawbacks from this. Gets rid of some distracting LOD and some
water pools appearing super flat (and pitch back for dark eco).
Fixes#1424
Manual patches:
`(trans idle krew-boss)`: `collide-shape` array created on the stack
changed to `(pointer collide-shape)`
---------
Co-authored-by: water <[email protected]>
Add a debug window for the loader to show levels and fix an issue with
jak 2 level unloading. I never really thought about how this works for >
2 levels, and there is a chance that you could unload the wrong level in
some cases.
The problem is some combination of merc-only levels not counting toward
the "in use" detection, and the unloader ignoring what the game wants to
load.
I don't remember why using merc models doesn't contribute to "in use"
but I'm afraid to change this for jak 1. Instead, we can have the
unloader avoid unloading levels that the game is telling us are loaded.
This is what we should have done originally, but there was a time when
Jak 1 didn't tell the loader anything, and we had this stupid detection
thing.
I think this is the first step toward just getting rid of the "in use"
detection and trusting the game for everything.
Implements the jak 2 lightning renderer as an alternate path through
Generic2. Also set up some generic stuff in the goal code.
There is a problem with the texture pool, which doesn't support the case
where two textures have the same tbp, but different cluts. So lightning
is often the wrong color (usually red).
The direct commit option doesn't work as there is no way to bypass
required status checks -- this makes the merge queue less useful. Switch
to a PR instead.
I did a pass through all missions, fixing issues as they came up. Also
got `seal-at-waterslums` working -- the best mission in the game 🥳
Co-authored-by: water <[email protected]>
Manual patches:
- `drill-turret`: The static data for `*turret-13-path*`,
`*turret-14-path*` and `*turret-15-path*` was decompiled by hand and the
integers in the `set-speed-mult` events have been replaced with boxed
integer arrays that contain only that integer in order to make the
compiler happy. To that effect, the event handler in `target-turret` was
changed to access that array instead of just accessing the int.
- `hover-nav-control`: In `hover-nav-control::10`, `arg2` is usually a
`vector`, but there are some places where it is called with `#t` as
`arg2` and, subsequently, crashes the game because it tries to access
the `quad` of `arg2` if `arg2` is truthy. To mitigate this, the
condition `arg2` has been replaced with `(and (!= arg2 #t) arg2)` (in
this case, it would jump to the `else` that just resets the `dest-vel`
and `transv` `quad`s)
- `drill-baron`: The static data for `*drill-ship-turret-speed-event*`
has been decompiled by hand.
TODOs:
- Jellyfish crash the game
- Destroying the metalhead eggs that are on the breakable wall crashes
the game (already happened with the Peacemaker before)
- Figure out why static data of type `turret-path-event` doesn't
decompile
The docs for all the hover-nav and nav-network code could use some love
in the future, I'm not smart enough to figure out what any of that code
actually means, but it seems to work...
Also threw in the fix for the ▲ that was accidentally left commented
out.
Noticed this has been failing for a while for reasons. I think I fixed
them but can't re-run the current job (out of date from master). This
this gives me a way to manually re-run it.
No more ghost town!
Manual patches:
- `hal3-course`: In `(anon-function 17 hal3-course)`, the decompiler is
doing something strange with `s3-0` that fails to compile, so I just
removed the `set!` and inlined the condition in the `when`.
- `guard`: Rewrote `(.mula.s)` stuff
This is a WIP while I'm learning the ins and outs of decompilation, but
putting up what I have for 2 reasons:
- Hoping someone can double check I'm on the right path (all functions
have signatures, all reasonably safe guesses for types have been put in,
using "object" for type where they're not)
- Might be blocked by not being able to run the offline-tests as a PAL
scrub
I'm going to look at what might be involved in making tests work for
PAL, but wouldn't be surprised if I have to wait to get a black label
version and come back to this after :(
---------
Co-authored-by: Tyler Wilding <[email protected]>
This will probably take a while, since we also have to translate all the
text of the base game - Naughty Dog never translated this game to
Hungarian. This PR will stay a draft until it is complete.
We realized that every letter in our alphabet was already working, apart
from two: Ő and Ű (they are unique sounds, so leaving their marks
wouldn't be okay).
Since I did not find that double accent thing in the jak font, I decided
to use ~ (see my change in FontUtils.cpp). It is good enough, and my
memory tells me that I already saw this exact same "solution"
(workaround) somewhere in the past. If anyone knows a better solution,
please let us know.
We chose ID 14 for the Hungarian language, as it was the lowest free ID.
**Progress tracker**
A tick here means that everything was translated. It does not mean that
everything is perfect yet. We will review everything multiple times, to
have the best translations possible.
Game text:
- [x] base game text
- [x] pc port text
- [ ] credits text ?
[Sziloyoo](https://github.com/Sziloyoo) helped with reviewing my
changes, and gave advice/suggestions for some complicated translations.
Subtitles will be done in the future, not in this PR.
This PR adds a feature to merc2 to update vertices. This will be needed
to efficient do effects like blerc/ripple/texture scroll. It's enabled
for blerc in jak 1 and jak 2, but with a few disclaimers:
- currently we still use the mips2c blerc implementation, which is slow
and has some "jittering" because of integer precision. When porting to
PC, there was an additional synchronization problem because blerc
overwrites the merc data as its being read by the renderers. I _think_
this wasn't an issue on PS2 because the blerc dma is higher priority
than the VIF1 DMA, but I'm not certain. Either way, I had to add a mutex
for this on PC to avoid very slight flickering/gaps. This isn't ideal
for performance, but still beats generic by a significant amount in
every place I tested. If you see merc taking 2ms to draw, it is likely
because it is stuck waiting on blerc to finish. This will go away once
blerc itself is ported to C++.
- in jak 1, we end up using generic in some cases where we could use
merc. In particular maia in village3 hut. This will be fixed later once
we can use merc in more places. I don't want to mess with the
merc/generic selection logic when we're hopefully going to get rid of it
soon.
- There is no support for ripple or texture scroll. These use generic on
jak 1, and remain broken on jak 2.
- Like with `emerc`, jak 1 has a toggle to go back to the old behavior
`*blerc-hack*`.
- In most cases, toggling this causes no visual differences. One
exception is Gol's teeth. I believe this is caused by texture coordinate
rounding issues, where generic has an additional float -> int -> float
compared to PC merc. It is very hard to notice so I'm not going to worry
about it.
When I cleaned up the `game.gp` some DGOs were no longer referenced
because my first dependency script omitted them -- thinking they weren't
required. From the perspective of the source files they indeed weren't
required but we still have to produce the DGO file.
also works around #2177
This adds environment mapping support to `Merc2`, and turns it on for
Jak 1 and Jak 2.
- The performance is much better
- Jak 1 can be toggled back to the old behavior with `(set! *emerc-hack*
#f)`. The new environment mapping is identical to the old one everywhere
I checked.
- Jak 1 still falls back to generic for ripple/texscroll/blerc/eyes -
there's still no dynamic texture or vertex updating support. The eye
detection stuff will sometimes flag stuff as eyes which is not eyes,
which is fine, but means that generic will be used in some places where
emerc could be used. For example, the shiny plates on jak's arm will be
drawn with generic because jak has eyes.
- Jak 2 hasn't been checked super carefully against PCSX2 yet.
- Jak 2 still isn't technically using emerc, but instead putting emerc
models in the merc bucket.
- The interface to merc is a lot different now and totally custom
OpenGOAL DMA code. The original merc drawing asm doesn't run anymore.
- The FR3 format changed
- Something funky going on with foreground lighting in escape, but
doesn't seem to be related to this change?
Performance comparison, jak 1, in likely the most generic-merc heavy
spot:
Some more progress on vehicle code, the following files still remain:
- vehicle-guard (annoying stack types)
- traffic-engine (many issues, some which already have issues made for
them, other entirely new confusing things!)
Initial implementation of the `ocean-mid`, `ocean-far` and `ocean-near`
renderers for Jak 2.
There's still a few things to sort out, mainly:
- [x] ~Backwards compatibility with Jak 1. The only thing that currently
stands in the way of that is figuring out a clean way to "un-hardcode"
the texture base pointer in C++ without creating a completely separate
`OceanTexture` class for Jak 2. One thing I thought of would be
modifying `BucketRenderer`'s virtual `init_textures` method to also pass
the `GameVersion`, but I'm not sure if there's a better way.~
- [x] ~The sudden transition from `ocean-near` to `ocean-mid`. Not sure
why it's happening or how to fix it.~
- [x] The ocean has two new methods in Jak 2, `ocean::89` and
`ocean::79`, one of which seems to be related to time of day sky colors.
~Even without them implemented, the end result looks quite close, so we
may be able to skip them?~ `ocean::89` generates `ocean-mid` envmap
textures, so it will likely be required, but will not be handled right
now.
Reverted the VU prologue removals because it made the tests fail.
Some side missions require cars, they don't work yet. Also the
ring-races and collection ones do not grant orbs. The hoaming beacon
collection one causes a `hud` crash
draft because using the array is a little weird still, don't feel like
dealing with window's slow debugging builds today.
I get the following weird error:
```clj
(define test-array (new 'static 'boxed-array :type type vector))
gr> (-> test-array 0)
1538004 #x1777d4 0.0000 vector
gr> (type? (-> test-array 0) type)
1342757 #x147d25 0.0000 #t
gr> (new 'static (-> test-array 0))
-- Compilation Error! --
Got 3 arguments, but expected 2
Form:
(-> test-array 0)
Location:
Program string:1
(new 'static (-> test-array 0))
^
Code:
(new 'static (-> test-array 0))
```
Maybe this is expected though and the `new` method wants a symbol, not a
type?
Fixes#2060
Co-authored-by: water <[email protected]>
Adding support for better child-type method docstrings. This is a
problem unique to methods.
Usually, a child-type will have the same signature and a common name
will apply, but the implementation is different. This means, you
probably want a different docstring to describe what is happening.
Currently this is possible to do via `:replace`. The problem with
replace is two fold:
- a replaced method ends up in the generated `deftype`...because you
usually change the signature!
- we don't put docstrings in the `deftype` in normal GOAL, this is just
something we do for the `all-types` file (they go in the `defmethod`
instead)
- more importantly, this means anytime you now want to change the
parent's name/args/return type -- you have to apply that change
everywhere.
So this is a better design you can now just declare the method like so:
```clj
(:override-doc "my new docstring" <method_id>)
```
And internally a pseudo-replaced method will be added, but it will
inherit everything from the parent (except the docstring of course)
Unrelated - I also made all the keyword args for declaring methods not
depend on ordering
This also adds support for documenting virtual and non-virtual state
handlers. For example:
```clj
(:states
(part-tester-idle (:event "test") symbol))
```
or
```clj
(idle () _type_ :state (:event "test") 20)
```
I will probably add the ability to give some sort of over-view docstring
at a later date.
Co-authored-by: water <[email protected]>
- started documenting the files I glossed over, some are totally done,
others are just partially done
- I changed the decompiler to automatically initialize the
art-group-info from the json file. This makes updating gsrc, even a
single file at a time, have consistent naming
- Though I disabled this functionality for jak 1, as I have no idea if
using the ntsc art groups will cause a regression for different versions
- fix indentation for docstrings -- it still doesn't look great, but
this is now a formatting concern, rather than the docstring having a
bunch of happen-stance leading whitespace.
- make sure bsp is processed on `l` levels in extraction (caused missing
remaps)
- clean up a few prints in extraction
- handle the <15 byte differences in art group files automatically (no
more errors about file naming)
- fix potential exception thrown by merc2 in a few ways: fixed bad data
in FR3's, check texture index just in case, and handle exceptions a
little bit better (still a crash, but at least you get a print)
- fix mips2 ocean stuff causing ocean far crashes
`whack` is missing a single function (`birth-func-whack-score`) that I
couldn't figure out the right args for.
I was not able to test any of this code because I cannot get `atoll` to
extract.
Manual patches in:
- `tomb-boulder`: a few `ppointer->handle`s (I think) were not being
decompiled properly (all used in combination with `clone-anim-once`)
- `ruins-obs`, `pillar-collapse`: `art-joint-anim` casts
- `tomb-beetle`: commented out a call to `shadow-control-method-14` that
was crashing the game when spawning the beetles
- `grunt-mech`: commented out `(.mula.s)` instruction
Notes:
- `enemy-info`'s `idle-anim-script` is most likely a `(pointer
idle-control-frame)`, however, some `nav-enemy-info` labels set it to
`#f` (first encountered in `tomb-beetle`, but also present in `hal`,
`roboguard` and `metalkor-setup`), which crashes the decompiler. This
may become a problem in the future when we eventually get to these
files. For this PR, I made `tomb-beetle` decompile with
`idle-anim-script` set to `#f` and have not noticed any issues/crashes
with that.
- `tomb-boulder` compiles and doesn't crash, but when trying to play the
Daxter chase sequence, the boulder sometimes either spawns at the origin
or spawns in the correct place, but doesn't move.
Co-authored-by: water <[email protected]>
- worked around audio code for gungame, the tutorials now function
- auto-exit cutscenes upon entry. This doesn't always work nicely (i
didnt want to cause any side-effects due to messing with load states)
but it atleast stops jak from being stuck forever waiting for the
cutscene to load
Effects the following files:
- [x] vehicle-rider
- [x] vehicle-control
- [x] vehicle-effects
- [x] vehicle
~~- [ ] vehicle-util~~
- [x] vehicle-physics
- [x] vehicle-states
~~- [ ] vehicle-guard~~
~~- [ ] traffic-engine~~
~~- [ ] traffic-manager~~
With the exception of traffic-engine, most of these files are either
done or have 1-3 stubborn functions remaining. Draft while I try to
resolve as many as possible / cleanup names and such.
Co-authored-by: water <[email protected]>
Adds the following files:
- `amphibian`
- `centurion`
- `ginsu`
- `grenadier`
- `hopper`
- `metalmonk`
- `monster-frog`
- `predator-graph`
- `predator-h`
- `predator`
- `rapid-gunner`
- `rhino`
- `rhino-wall`
- `tomb-baby-spider`
Also adds the DGOs for the following levels:
- Mar's Tomb
- Mountain Temple
- Drill Platform
- Sacred Site (Sage Hut)
Manual patches:
- The decompiler emits `(b! #t cfg-17)` in `(trans hostile hopper)`
without putting a `(label cfg-17)` anywhere
- Added cast to `art-joint-anim` in `(code broken rhino-wall)` and
`(code hit rhino-wall)`
Other notes:
- `amphibian` seems to occasionally crash when using its tongue attack.
Haven't investigated this yet
- `ginsu` crashes after being killed somewhere in the `deactivate`
method, possibly because of its `part-spawner`
- Predators aren't spawning in the forest hunt mission, not sure if
`forest-obs` might be needed for that or if it's something else
- The `rhino-wall` STR animation seems to load (albeit a bit broken due
to missing VAG stream playback), but causes the viewport to shrink as
soon as it's played
- I added `pegasus::74` to the `event_handler_hack` bool in
`variable_naming.cpp` because I got a `none` cast after changing the
return value of `enemy::74`
- lets you split up your `startup.gc` file into two sections
- one that runs on initial startup / reloads
- the other that runs when you listen to a target
- allows for customization of the keybinds added a month or so ago
- removes a useless flag (--startup-cmd) and marks others for
deprecation.
- added another help prompt that lists all the keybinds and what they do
Co-authored-by: water <[email protected]>
Notable things:
- This assert is hit when trying to save the pc-settings file, NYI
https://github.com/open-goal/jak-project/blob/e630b506903d7f7028dff89702be7f586c2120e7/game/kernel/common/Symbol4.h#L14
so right now settings aren't persisted. But RPC defaults to on
- The existing functions can probably be made generic based off the game
version, but I didn't spend time refactoring them yet as they aren't
really ready to be used in jak 2 yet (we have no screenshots for the
levels for example)
The things that aren't working right now:
- returning the board (maybe cutscene related?)
- draft until i see if this can be sorted out
- seems to need streaming audio as well, the `scene-player` gets
properly initialized from what i can tell
- had to hack around the streaming audio parts.
Everything relating to IK in `target-mech` was commented out as I
couldn't get a lambda to decompile properly, but the mech still seems to
work fine without it.
Co-authored-by: water <[email protected]>
Two main problems fixed here:
- offline tests will fail on a comparison failure (a mistake from the
re-write)
- art-group-info is committed to the repo and shared with every thread
(running the tests with 1 thread, for example on the CI, and locally
were producing different results)
art files are still not provided to the jak2 offline tests:
- `*-ag` files are not being output
- `art-elts.gc` is not complete, as a handful of files claim to be
missing stuff
lastly, in jak1's offline tests we were also running `tpage` and `*-vis`
files through the decompiler. This omits that (they came from the
`all_objs.json` file) -- is this an issue?
Fixes:
- a very old bug in depth in DirectRenderer, probably from the original
tfrag stuff. Looked at PCSX2 source code to see how 32/24 bit depths are
handled. This fixes hud sprites being drawn behind level geometry.
- saturate `vftoi4` like the ps2 does when the float is too large,
fixing hidden text in `hud`. For now it's only using this in the font
code because this saturation is actually kinda slow and hasn't been a
problem in other places.
- fix crazy particle spawning issue with blue gun and dripping stuff.
This would happen if particles kill themselves while being processed
(through a callback)
This adds a new ImGUI menu to help filter out the clutter on screen.
https://user-images.githubusercontent.com/13153231/210192912-b1c28319-bacb-449c-ad7f-e7308fb75f50.mp4
This also:
- moves the imgui display bool into a game specific config file (you can
hide it in jak1, and not in jak2)
- the config file also persists the settings from this menu (except the
filters for now, future TODO)
- there is a new `ignore_imgui_hide_keybind` in this file to ignore
hiding it when you press Alt
It used to automatically assign actor IDs, but reset in between custom
levels, so it was possible for actors to share the same actor id with a
actor in another level. With this the user can now specify a base_id to
make sure that the custom levels never match! (or fix it if they do!)
CreateChildSound did not check for nullptr return from
soundbank::make_handler, which would happen for sounds with no grains.
Prevent further problems with this by switching to optional instead of
passing nullptr.
I did some manual modifications in a few places to work around some
truly mysterious control flow, and some unsupported stack array stuff.
Also fixes a bug in decompiling static improper lists.
As far as I can tell, guns work, other than some graphical issues and
the crazy particle spawning issue, but I strongly suspect these are
problems with the sparticle/graphics side.
A common thing that can be forgotten about / confusing to new people is
that:
1. if you add an object to the `allowed_objs` list but it's also in the
`banned_objs` list -- the ban still takes precedence with no failure
2. if you add an object to the `allowed_objs` list but have not allowed
the DGO in `inputs.jsonc` it will also silently log a failure and
continue.
This PR turns both situations into an explicit error with advice/a
reminder on what to do to fix it.
- Split up DGOs between threads in the multithreaded offline test
- fix some random warnings
- make the sig paths decompile a bit nicer to make some files smaller
When working with mips2c recently, I noticed adding the
`defmethod-mips2c` or `def-mips2c` code was a manual step. This is a bit
tedious to have to go and do yourself, but more importantly you have to
manually go and find the right spot in the source file else you might be
declaring it too early or too late.
This will automatically output the declaration for methods, and a
half-finished comment for the functions. I wasn't able to fully output
the function one because it seems the signature info from `all-types`
doesn't make it all the way through -- but maybe I'm wrong or this is an
easy fix?
Running reference tests/decompiler should now be possible on macos
(arm). Most of the changes were just cleaning up places where we were
sloppy with ifdefs, but there were two interesting ones:
- `Printer.cpp` was updated to not use a recursive function for printing
lists, to avoid stack overflow
- I replaced xxhash with another version of the same library that
supports arm (the one that comes in zstd). The interface is C instead of
C++ but it's not bad to use. I confirmed that the extractor succeeds on
jak 1 iso so it looks like this gives us the same results as the old
library.
- Stub `target-board-handler` to prevent jetboard crash
- Increase stack size for `target` to fix dark bomb crash
- Uncomment some code in `target-darkjak` to make Dark Jak work (texture
morph still crashes, so it's commented out)
- Increase stack size for a `process-spawn-function` in
`target-send-attack` to fix a crash with giant Dark Jak
A big one...
I figure even if we would like to change the way the particle/scene code
is output -- it'd be easier to find patterns with it all decompiled.
I've updated my script so it can easily be used to mass update these
files:
```bash
task update-gsrc-glob GLOB="**/*-part*.gc"
```
> for example will update gsrc files with `part` in their name -- if
they are in ref tests (so uncompleted ones aren't touched)
I found a few issues along the way that I'll have to make issues for
soon.
This solves two main problems:
- the looming threat of running out of memory since every thread would
consume duplicate (and probably not needed) resources
- though I will point out, jak 2's offline tests seem to hardly use any
memory even with 400+ files, duplicated across many threads. Where as
jak 1 does indeed use tons more memory. So I think there is something
going on besides just the source files
- condense the output so it's much easier to see what is happening / how
close the test is to completing.
- one annoying thing about the multiple thread change was errors were
typically buried far in the middle of the output, this fixes that
- refactors the offline test code in general to be a lot more modular
The pretty printing is not enabled by default, run with `-p` or
`--pretty-print` if you want to use it
https://user-images.githubusercontent.com/13153231/205513212-a65c20d4-ce36-44f6-826a-cd475505dbf9.mp4
Supports most of the grain types now while maintaining compatibility
with the old stuff (at least the subset of things jak1 uses)
Would benefit from some testing in Jak 1 to make sure I didn't break
anything.
Sorry the git history is a mess, I'll do something about it later.
- decompile `lights.gc`
- decompile remaining `target-death` function, you can die now
- fix float issue with `matrix-from-two-vectors-smooth!` making jak face
the wrong way in the slide thing
Mostly easy / particle def files. I did a bit in `ruins` but CFG
failures and missing `drawable` functions make it untestable for now so
I've put that on pause.
This allows you to not have to define the entire file path to a source
file to re-compile and load it. Technically a stop-gap until editor
tools are developed around writing OpenGOAL.
As opposed to `(ml "goal_src/jak2/engine/game/main.gc")` (which still
works)
This is accomplished via the following config (connection attempts is
irrelevant):
```json
{
"numConnectToTargetAttempts": 1,
"jak2": {
"asmFileSearchDirs": [
"goal_src/jak2"
]
}
}
```
This also provides a way to make game-specific configurations for the
REPL fairly easily.
- You can define a `startup.gc` in your user folder, each line will be
executed on startup (deprecates the usefulness of some cli flags)
- You can define a `repl-config.json` file to override REPL settings.
Long-term this is a better approach than a bunch of CLI flags as well
- Via this, you can override the amount of time the repl will attempt to
listen for the target
- At the same time, I think i may have found why on Windows it can
sometimes take forever to timeout when the game dies, will dig into this
later
- Added some keybinds for common operations, shown here
https://user-images.githubusercontent.com/13153231/202890278-1ff2bb06-dddf-4bde-9178-aa0883799167.mp4
> builds the game, connects to it, attaches a debugger and continues,
launches it, gets the backtrace, stops the target -- all with only
keybinds.
If you want these keybinds to work inside VSCode's integrated terminal,
you need to add the following to your settings file
```json
"terminal.integrated.commandsToSkipShell": [
"-workbench.action.quickOpen",
"-workbench.action.quickOpenView"
]
```
- target-part
- ;; ERROR: Failed to convert to atomic ops: Variable could not be
constructed from register r0 in `process-drawable-shock-wall-effect`
- gun-part
- missing sparticle decompiling case it seems related to `L155`
This is pretty rough but...im excited to see it working :)
The VU programs for the ocean renderer have changed a bit and
`ocean-texture` has a bunch of new stuff, otherwise things are
relatively similar to Jak 1.
This is the first time I used mips2c and I'm not sure I did it 100%
right, so that should be double-checked.
The only interesting one is `collide-hash`, which is untested.
The other two are very likely unused. I skipped the annoying code in
`collide-probe` because it's not used and the same as jak 1.
My changes for allowing an explicit tag have now been merged to the
original action, so no need to use my fork.
Additionally, buildcache itself reverted to using ubuntu 20.04 to fix
this issue, but we'll leave the explicit tag there anyway to avoid
sudden breakage in the future.
Offline tests ignore comments in their comparison, but there's no reason
to strip them from the file that goes into the reference test folder
when doing the typical update routine.
This just generates superfluous diffs for all the files already done
prior to this change and is meaningless (the lines are dropped anyway)
Couldn't finish any of the enemy/nav-enemy related files for one reason
or another, but quite a bit of work that will be easier to merge and
iterate on instead of keeping track of the branch.
enemy/idle-control has some very weird focus related code.
nav-mesh/nav-control still has a bunch of CFG resolution problems that
need to be manually resolved.
Co-authored-by: water <[email protected]>
- fix issue described in
https://github.com/open-goal/jak-project/issues/1939
- fix `text`, which was manually patched with the wrong offset (was
reading the symbol value off by one byte)
- clean up some random useless prints
- make the offline tests keep trying if there's a comparison error,
clean up the output a bit so the diffs are all at the end.
The offline-tests are going to end up taking too long for jak 2, I did
some rough math and by the end of it we'll be spending almost 2 minutes
for a full offline test on my machine.
These changes allow us to throw hardware at the problem
Still some work to do to make the output nicer, but seems to be fairly
reliable. By default it still uses 1 thread, use `num_threads` to change
this.
Almost done:
- `target-handler` (`(none)` event handler casts and CFG error)
- `target2` (`(none)` event handler casts)
- `powerups` (`cloud-track` does some weird stuff with `handle`s)
- `gun-states` (CFG error)
Some progress in:
- `water-flow`
Additionally:
- Clean up the two year old Jak 3 config file and add a config skeleton
(disassembling seems to not have worked, but I was able to dump obj
files and the `all_scripts` file)
- Fix automatic skelgroup detection and `defskelgroup` macro for Jak 2
(closes#1950)
- When a function decompiles without any major errors, a warning is
generated with the op id for each unresolved load and store that will
likely fail to compile (closes#1933)
Generates the `game.gp` code for all DGOs, code added to the project
file has been commented out for now.
In my opinion, this is way too much in a single file -- it would be nice
if this code would live directly inside the `.gd` files themselves, then
everything is nicely organized. But this approach might have issues I'm
not aware of.
I think this got removed awhile back, not sure if it was intentional,
but thought it was nice to have. Feel free to close if it's not
supported for a good reason 👍
The type makes no sense when compared to the actual data there. This
modifies the type slightly, which I think is better than spending a
bunch of effort to special case this in the compiler/decompiler.
If we try this, and it turns out it doesn't work, we can always revisit
it. But I think this will work.
This PR does a few main things:
- finish decompiling the progress related code
- implemented changes necessary to load the text files end-to-end
- japanese/korean character encodings were not added
- finish more camera code, which is required to spawn the progress menu
/ init the default language settings needed for text
- initialized the camera as well
Still havn't opened the menu as there are a lot of checks around
`*target*` which I havn't yet gone through and attempted to comment out.
For the level builder, I use .bak files whenever I modify a core file in
the project. These let you undo changes, but they don't need to be
pushed. I also couldn't find a nice way to do this, but we should also
ignore all contents of custom_levels besides test-zone.
A few issues:
- lwidea's fr3 is getting loaded and unloaded all the time
- the debug line drawing clipping is wrong (doesn't seem wrong in pcsx2,
so I think this is on us)
- nothing actually using vis data yet
- at a large distance, our view frustum culling seems slightly too
aggressive (might be that viewport scissoring is wrong)
- in the city, things seem darker as you move away. unclear how this is
happening (fog?)
Favors the `lg` namespace over `fmt` directly, as this will output the
logs to a file / has log levels.
I also made assertion errors go to a file, this unfortunately means
importing `lg` and hence `fmt` which was attempted to be avoided before.
But I'm not sure how else to do this aspect without re-inventing the
file logging.
We have a lot of commented out prints as well that we should probably
cleanup at some point / switch them to trace level and default to `info`
level.
I noticed the pattern of disabling debug logs behind some boolean,
something to consider cleaning up in the future -- if our logs were more
structured (knowing where they are coming from) then a lot this
boilerplate could be eliminated.
Closes#1358
And everything else needed for them!
A couple functions are bad currently.
- fixes#1929 - untested on linux
- fixes#1924 - now you need to type `,` before a lambda you want to put
in a pair.
- fix debugger symbol table in jak 2
- made the decompiler output `(meters 2)` instead of `(meters 2.0)`
- fixed a bug with the bitfield enum special -1 case
- made bad game text decomp not exit the decompiler
- added `editable-player` and `script`
eg
```
(set! (-> obj sound-pre-open) (static-sound-spec "airlock-slider"))
```
it is the same as jak 1, but the `fo-curve` option is set to 1
sometimes.
Just missing `target-board-handler` because it errored with `Bad delay
slot in clean_up_cond_no_else_final` and I wasn't able to figure out
where the asm branch was.
Commented out `target-board-clone-anim` because it crashes even after
adding `clone-anim` and `clone-anim-once`.
A small quality of life increase that is more impactful since jak 2 has
double the file count.
I often use the offline-tests to find compiler errors / automatically
resolve them when i am finalizing a file. As more and more files are
completed this becomes increasingly more inefficient. When I know that
only 1 file needs to be decompiled / compared / compiled, I'd prefer to
have a feature like this.
Fixes#1886 which was introduced in #1756
Changed to only read the dpad inputs for hud toggle if L2/R2 were
considered held (they're not read during a pause buffer)
Some notes:
- `draw-prototype-inline-array-shrub` can probably be decompiled but I'm
not comfortable doing this function myself. It's got a ton of inline
assembly (or at least I think it's inline assembly) and is very long. I
left this marked as asm for now but I am confident in the function
definition.
- The VU program is identical to Jak 1 (yay)
- There's some new `dma-test` stuff at the end of the file, not really
sure what it's doing but it runs in the top-level.
- decompile `subdivide`, `wind-work`, `tie-work`, `bsp`, `focus`
- support `ppacb` in compiler
- don't assert when bitfield stuff fails due to constant propgataion
weirdness
- finish up history
- div/mod unsigned assert fix in decompiler
- empty assert fix in decompiler for failed `add` type prop
- make jak 1 performance counters "work" (just measure time)
- fix cast/typos on pcgtb/vftoi15
Fixes a crash when there's a `(new 'static 'something :field <some
anonymous function with a particular set of errors in the decompler>)`.
The output will now be
```
(set! *duck-mods* (new 'static 'surface
:name 'duck
...
:mult-hook <lambda at L615>
:flags (surface-flag duck)
)
```
fixes https://github.com/open-goal/jak-project/issues/1882
Found the reason why this wasn't happening -- the path included by the
decompiler isn't a proper URL encoded URI -- so I have to convert it
appropriately so when vscode sends it's "this file with this URI has
changed" event it actually matches the tracked all-types file.
This will make jump-to-defs not drift as the file changes, and new
uncommented symbols will actually get picked up
Has boxed array accessing that prevents me from adding anything to ref
tests (the entire file is lambdas so the access pattern that i would
like to ignore happens at the top-level, can't ignore it.
This code actually already has quite a bit of original docstrings so
it's not too bad in that regard considering a `script-context` can have
16 arbitrary objects. It seems they rarely put more than a single object
in the context and the types are usually obvious / are actually type
checked!
It was written:
> If you the build directory you'll need…
I tried to fix it, hopefully I found the correct wording. Otherwise
please let me know what was missing.
The merc VU1 program is identical to Jak 1. Jak 2's merc is doing some
new texture login stuff and the DMA buffer setup is a bit different, but
overall very similar.
Fixes https://github.com/open-goal/jak-project/issues/1821 by adding a
special case for `new` method calls where the argument with type
`symbol` is actually an address to uninitialized structure on the stack.
Fixes https://github.com/open-goal/jak-project/issues/1849 (or at least
the cause of the issue Vaser gave in chat, and one random one I found in
`debug-sphere`)
Fixes https://github.com/open-goal/jak-project/issues/1853
Fixes https://github.com/open-goal/jak-project/issues/1857 by moving the
cast into the cond if the body is a single form and the destination type
is a bitfield/enum which is likely to work well. Seems to work on the
examples we could find in jak 1 and jak 2.
Also fixes an issue with casts on the result of `handle->process` (a
common place to use casts)
the output of process->handle is a plain process. Most of the time, you
end up casting this to a more specific. If you add a cast on every use
of the variable, the decompiler will decide to change the type of that
variable to the more specific type, and this breaks the handle cast.
so previously it was impossible to get code like
```
(let* ((s2-0 (the-as swingpole (handle->process (-> self control hack))))
(gp-0 (-> s2-0 dir))
)
```
But now it will work
Rewrites specific kinds of lets where the return value of the `set!`
itself is meant to be used to just be `set!`s. Implemented at the let
rewrite level. Seems to work for Jak 2 so far.
Fixes#1854 .
This shows the current settings needed to help provide information when
verifying a run, also it removes the two calls to show the version in
main.gc and instead does it within the speedrun/auto splitting scope.
- Shows Version
- Shows if Speedrunner mode is on (technically pointless since the text
only displays when it is enabled this more just serves to show that
there IS a speedrunner mode to viewers and stuff (josh) So hopefully it
may cut down on submissions with the mode off)
- Shows if cutscene skips is enabled/disabled.
Tested on 16x9 4x3 the really weird one and borderless/fullscreen
[](https://www.youtube.com/watch?v=qDC-beEg5Es)
Co-authored-by: Tyler Wilding <[email protected]>
I wasn't able to 100% complete `debug` due to a bunch of level boundary
debug stuff I couldn't figure out. But, I added a ref test and
documented/copied over everything else into goal_src.
Most of the functions that existed in Jak 1 are identical, others got an
extra param or 2 and some now make copies of arguments instead of
modifying them. There's a bunch of new functions, including all of the
debug functions that used to be in `level-boundary` are now in `debug`.
At the very end of `debug` there's also some weird asm functions
checking for some EE memory controller bug (not sure what's up with
that)?
* sparticle-launcher
* d/jak2: large amount of `sparticle-launcher` done
* d/jak2: finish the majority of `sparticle`
* decomp: improve format code ignoring
* d/jak2: make bits unique in `sp-cpuinfo-flag`
* d/jak1: revert config change
Still comes up from the bottom, but its probably the "correct" way to do it?
The other alternative would be killing Jak when the cutscene is skipped instead of `(start 'play (get-continue-by-name *game-info* "citadel-elevator"))` This would make the elevator at the top instantly, however it might mess with hp values and stored levels.
Credit to Barg/dallmeyer
* jak1: put common speedrunning code into it's own file
* jak1: enforce `60` fps while in speedrunning mode
* jak1: when speedrunning, display the version until you get the first powercell
* jak1: add an explicit option for skipping cutscenes
* jak1: extend `game-option` to allow any menu option to be disabled
* tests/jak1: allow whitelisting types to be redefined to satisfy typeconsistency checks
* jak1: add file headers
* jak1: cleanup bool checking
* test: delete the es substitle file
* test: add it back
* jak1: missed one cleanup spot related to bool comparisons
description:Take a couple minutes to help our maintainers work faster.
options:
- label:I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+sort%3Aupdated-desc) for duplicate or closed bug reports
required:true
- label:I understand that I am supposed to provide my own legitimately obtained copy of the game
required:true
- type:textarea
attributes:
label:Describe the Bug
description:A clear and concise description of what the bug is. You may post screenshots or videos of the bug here.
validations:
required:true
- type:textarea
attributes:
label:How To Reproduce
description:Steps to reproduce the behavior. You can also post a video of it here.
validations:
required:true
- type:dropdown
attributes:
label:Does this problem occur on original hardware or PCSX2?
description:Some things that may seem like bugs are actually exactly how the original game behaved.
options:
- Yes,it's unique to OpenGOAL
- Didn't check
- Not needed, bug is obvious
validations:
required:true
- type:textarea
attributes:
label:Expected Behavior
description:A clear and concise description of the expected behavior.
placeholder:When I do X, Y should happen.
validations:
required:true
- type:input
attributes:
label:Operating System
description:Windows version, Linux distribution, etc.
validations:
required:true
- type:input
attributes:
label:OpenGOAL Version
description:Either the release version (vX.X.X) or the commit sha if you've built from source
validations:
required:true
- type:dropdown
attributes:
label:Game Version
options:
- NTSC 1.0 (black label)
- NTSC Greatest Hits version (red label)
- PAL
- JP
validations:
required:true
- type:dropdown
attributes:
label:Are you using a high refresh-rate monitor (>60Hz)
description:Take a couple minutes to help our maintainers work faster.
options:
- label:I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Ajak1+sort%3Aupdated-desc+) for duplicate or closed bug reports
required:true
- label:I understand that I am supposed to provide my own legitimately obtained copy of the game
required:true
- type:textarea
attributes:
label:Describe the Bug
description:A clear and concise description of what the bug is. You may post screenshots or videos of the bug here.
validations:
required:true
- type:textarea
attributes:
label:How To Reproduce
description:Steps to reproduce the behavior. You can also post a video of it here.
validations:
required:true
- type:dropdown
attributes:
label:Does this problem occur on original hardware or PCSX2?
description:Some things that may seem like bugs are actually exactly how the original game behaved.
options:
- Yes,it's unique to OpenGOAL
- Didn't check
- Not needed, bug is obvious
validations:
required:true
- type:textarea
attributes:
label:Expected Behavior
description:A clear and concise description of the expected behavior.
placeholder:When I do X, Y should happen.
validations:
required:true
- type:textarea
attributes:
label:Environment Information
description:"You can upload the [Support Package](https://github.com/open-goal/launcher#asking-for-help) provided by the Launcher here, or you can provide the following information: CPU, GPU, OS Version, OpenGOAL Version (found in the window's title bar)"
validations:
required:true
- type:dropdown
attributes:
label:Game Version
options:
- NTSC 1.0 (black label)
- NTSC Greatest Hits version (red label)
- PAL
- JP
validations:
required:true
- type:dropdown
attributes:
label:Have you set the game to something other than `60fps`?
description:Take a couple minutes to help our maintainers work faster.
options:
- label:I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Ajak2+sort%3Aupdated-desc+) for duplicate or closed bug reports
required:true
- label:I understand that I am supposed to provide my own legitimately obtained copy of the game
required:true
- type:textarea
attributes:
label:Describe the Bug
description:A clear and concise description of what the bug is. You may post screenshots or videos of the bug here.
validations:
required:true
- type:textarea
attributes:
label:How To Reproduce
description:Steps to reproduce the behavior. You can also post a video of it here.
validations:
required:true
- type:dropdown
attributes:
label:Does this problem occur on original hardware or PCSX2?
description:Some things that may seem like bugs are actually exactly how the original game behaved.
options:
- Yes,it's unique to OpenGOAL
- Didn't check
- Not needed, bug is obvious
validations:
required:true
- type:textarea
attributes:
label:Expected Behavior
description:A clear and concise description of the expected behavior.
placeholder:When I do X, Y should happen.
validations:
required:true
- type:textarea
attributes:
label:Environment Information
description:"You can upload the [Support Package](https://github.com/open-goal/launcher#asking-for-help) provided by the Launcher here, or you can provide the following information: CPU, GPU, OS Version, OpenGOAL Version (found in the window's title bar)"
validations:
required:true
- type:dropdown
attributes:
label:Game Version
options:
- NTSC 1.0 (black label)
- NTSC Greatest Hits version (red label)
- PAL
- JP
validations:
required:true
- type:dropdown
attributes:
label:Have you set the game to something other than `60fps`?
"description": "Marks the file to be excluded from the LoC count"
},
"og:ignore-errors": {
"scope": "opengoal",
"prefix": ["og:ignore-errors"],
"body": [";; og:ignore-errors"],
"description": "Ignore lines beginning with ';; ERROR:' or ';; WARN:' when copying decompiled code into it"
},
"og:ignore-form": {
"scope": "opengoal",
"prefix": ["og:ignore-form"],
"body": [";; og:ignore-form:${1:substr}"],
"description": "If the provided substr is found in the starting line of a form, the entire form will be omitted when copying decompiled code into the file"
},
"og:preserve-this": {
"scope": "opengoal",
"prefix": ["og:preserve-this"],
"body": [";; og:preserve-this"],
"description": "When updating GOAL source files from decomp, CI will pick up if this comment is deleted in the diff. Use this to keep track of manually patched sections of code."
> Our repositories on GitHub are primarily for development of the project and tracking active issues. Most of the information you will find here pertains to setting up the project for development purposes and is not relevant to the end-user.
For a setup guide on how to install and play the game there is the following video that you can check out: https://youtu.be/K84UUMnkJc4
For questions or additional information pertaining to the project, we have a Discord for discussion here: https://discord.gg/VZbXMHXzWv
Additionally, you can find further documentation and answers to **frequently asked questions** on the project's main website: https://opengoal.dev
> [!WARNING]
> **Do not use this decompilation project without providing your own legally purchased copy of the game.** We do not distribute any assets from the game - you must use your own legitimately obtained PS2 copy of the game. We support every retail PAL, NTSC, and NTSC-J build, including Greatest Hits copies.
- [Project Description](#project-description)
- [Current Status](#current-status)
- [Methodology](#methodology)
@@ -25,6 +36,11 @@
- [Windows](#windows)
- [Required Software](#required-software)
- [Using Visual Studio](#using-visual-studio)
- [MacOS](#macos)
- [Intel Based](#intel-based)
- [Apple Silicon](#apple-silicon)
- [VSCode](#vscode)
- [Building and Debugging](#building-and-debugging)
- [Building and Running the Game](#building-and-running-the-game)
- [Extract Assets](#extract-assets)
- [Build the Game](#build-the-game)
@@ -40,25 +56,9 @@
- [`goal_src/`](#goal_src)
- [`game` runtime](#game-runtime)
## Please read first
Our repositories on GitHub are primarily for development of the project and tracking active issues. Most of the information you will find here pertains to setting up the project for development purposes and is not relevant to the end-user.
For questions or additional information pertaining to the project, we have a Discord for discussion here: https://discord.gg/VZbXMHXzWv
Additionally, you can find further documentation and answers to **frequently asked questions** on the project's main website: https://opengoal.dev
**Do not use this decompilation project without providing your own legally purchased copy of the game.**
### Quick Start
If you just want to play the game, you can follow the steps in this video: https://www.youtube.com/watch?v=p8I9NfuZOgE
We do not distribute any assets from the game - you must use your own legitimately obtained PS2 copy of the game. We support every retail PAL, NTSC, and NTSC-J build, including Greatest Hits copies.
## Project Description
This project is to port Jak 1 (NTSC, "black label" version) to PC. Over 98% of this game is written in GOAL, a custom Lisp language developed by Naughty Dog. Our strategy is:
This project is to port the original Jak and Daxter and Jak II to PC. Over 98% of the games are written in GOAL, a custom Lisp language developed by Naughty Dog. Our strategy is:
- decompile the original game code into human-readable GOAL code
- develop our own compiler for GOAL and recompile game code for x86-64
- create a tool to extract game assets into formats that can be easily viewed or modified
@@ -76,7 +76,7 @@ We support both Linux and Windows on x86-64.
### Current Status
Jak 1 is largely playable from start to finish with a handful of bugs that are continually being ironed out.
Jak 1 is largely playable from start to finish with a handful of bugs that are continually being ironed out. Jak 2 is in development.
@@ -112,13 +112,13 @@ This will create an image with all required dependencies and already built.
docker run -v "$(pwd)"/build:/home/jak/jak-project/build -it jak bash
```
Note: If you the build directory you'll need to rerun the build command. Alteratively you can get the build via `docker cp`
Note: If you change the content of the `build/` directory you'll need to rerun the `build` command. Alternatively you can get the build via `docker cp`.
This will link your build folder to the images so can validate your build or test it on an external device.
This will link your `build/` folder to the images so can validate your build or test it on an external device.
Docker images can be linked into your IDE (e.g. CLion) to help with codesniffing, static analysis, run tests and continuous build.
Unfortently you'll still need task runner on your local machine to run the game or instead, manually run the game via the commands found in `Taskfile.yml`
Unfortunately you'll still need task runner on your local machine to run the game or instead, manually run the game via the commands found in `Taskfile.yml`.
### Linux
@@ -127,7 +127,7 @@ Unfortently you'll still need task runner on your local machine to run the game
If you either don't want to or cannot use Visual Studio for working with the C++ project, VSCode is a good alternatively.
The `clangd` extension is [recommended](https://marketplace.visualstudio.com/items?itemName=llvm-vs-code-extensions.vscode-clangd) and requires `clangd` to be on your `$PATH`. If you can run `clangd` in a terminal successfully then you should be good to go.
Once you generate your CMake for the first time the clangd LSP should be able to index the project and give you intellisense.
#### Building and Debugging
TODO - Consider Contributing Documentation :)
### Building and Running the Game
Getting a running game involves 4 steps:
@@ -244,13 +289,20 @@ Getting a running game involves 4 steps:
#### Extract Assets
First, setup your settings so the following scripts know which game you are using, and which version. In a terminal, run the following:
First, setup your settings so the following scripts know which game you are using, and which version. For the black label version of the game, run the following in a terminal:
```sh
task set-game-jak1
task set-decomp-ntscv1
```
For other versions of the game, you will need to use a different `-set-decomp-<VERSION>` command. An example for the PAL version:
```sh
task set-game-jak1
task set-decomp-pal
```
> Run `task --list` to see the other available options
> At the time of writing, only Jak 1 is expected to work end-to-end!
@@ -292,6 +344,12 @@ Run the following to build the game:
g > (mi)
```
> IMPORTANT NOTE! If you're not using the non-default version of the game, you may hit issues trying to run `(mi)` in this step. An example error might include something like:
>
> `Input file iso_data/jak1/MUS/TWEAKVAL.MUS does not exist.`
>
> This is because the decompiler inputs/outputs using the `gameName` JSON field in the decompiler config. For example if you are using Jak 1 PAL, it will assume `iso_data/jak1_pal` and `decompiler_out/jak1_pal`. Therefore, you can inform the REPL/compiler of this via the `gameVersionFolder` config field described [here](./goal_src/user/README.md)
#### Run the Game
Finally the game can be ran. Open a second terminal from the `jak-project` directory and run the following:
> This is still in a highly experimental state, adjust your expectations accordingly.
Despite LISP being very simple, and taking the stance of a highly opinionated formatter that doesn't concern itself with line-length in most situations writing a code formatter is still an incredibly annoying and difficult thing to do. This documentation serves as a place for me to collect my thoughts and explain how it works (or more likely, how it does not).
Also in general I found it _really_ hard to find modern and easy to understand articles on how to get started (with the exception of zprint's usage docs, but that doesn't really cover implementation ideas). This area is either dominated by 40 year old journal articles, or discussions around really difficult formatting projects. There isn't a whole lot in the vain of _here's how I made a decent, simple formatter_, so maybe it'll help someone else (even if it's teaching them potentially what not to do!)
## Architecture / Overview
Atleast for me, it helps to understand the path code takes as it flows through the formatting process. Originally I tried to do as much at once for sake of efficiency but this just makes things incredibly hard to reason about and fix bugs. So like any problem that is complex, break it down, the formatter goes through many distinct phases chipping away at the problem:
```mermaid
flowchart TB
subgraph top1 [Build a Formatting Tree]
direction TB
a1(AST Parse) --> a2(Consolidate AST Nodes)
a1 --> a3(Collect Metadata)
end
subgraph top2 [Apply Formatting Configuration]
direction TB
b1(Recursively Iterate each Node) --> b2{Form head has\na predefined config?}
b2 --> |yes|b3(Fetch it and set the config on the node)
b2 --> |no|b4(Apply a default configuration)
end
subgraph top3 [Apply Formatting]
direction TB
c1(For each node, recursively build up a\nlist of source code lines, no indentation) --> c2(Consolidate the first two lines if hanging\nor if the config demands it)
c2 --> c3(Append indentation to each line as appropriate\nand surround each form with relevant parens)
end
subgraph top4 [Finalize Lines]
direction TB
d1(Concatenate all lines with whatever\nfile ending the file started with) --> d2(If we're formatting a file append a trailing new-line\n otherwise we are done)
end
top1-->|FormatterTree|top2
top2-->|FormatterTree with Configurations|top3
top3-->|String List|top4
```
### Build a Formatting Tree
At the end of the day, formatting is about changing the location and amounts of white-space in a file. Meaning it would be ideal if we could start with a representation that is whitespace-agnostic. Some formatters (like cljfmt) choose to use the original whitespace as an input into the formatting process, essentially meaning that your code should already be somewhat-formatted. This is somewhat advantageous as it can simplify the formatter's implementation or be a bit more flexible as to what the original programmer intended (assuming they knew what they were doing). But of course it has one fatal flaw -- the code has to already be well formatted.
For our formatter I wanted it to be able to take in horribly, ugly, minified code and output an optimal result -- especially since so much of our code gets spat out by another tool. To do that, we need to be able to format the code without any existing whitespace helping us, adding in the whitespace from the ground up.
The easiest way to do this is to construct an AST of the language which has a lot of other benefits for building language features but for here all we care about is getting rid of the whitespace and having some basic interpretation of the source code structure. The easiest way now-a-days to make an AST for a language is to create a [treesitter grammar](https://github.com/open-goal/tree-sitter-opengoal).
#### Consolidate AST Nodes and Collect Metadata
One downside with using an AST, atleast with the way the grammar is currently implemented is it creates a somewhat difficult tree to work with from a formatting perspective. For example in a LISP, everything is a list. In our AST it does not look like how you may intuitively expect:
`(+ 1 2)`
```mermaid
flowchart TB
a("source\n(+ 1 2)") --> b("list_lit\n")
b --> c("(") --> ca("(")
b --> d("sym_lit\n+") --> da("+")
b --> e("sym_lit\n1") --> ea("1")
b --> f("sym_lit\n2") --> fa("2")
b --> g(")") --> ga(")")
```
vs what you would probably prefer:
```mermaid
flowchart TB
a("top-level") --> b("list\n- surrounded in ()")
b --> da("+")
b --> ea("1")
b --> fa("2")
```
As you can see from even this simple example, the AST representation can be very verbose and in a sense has duplication. Each node is aware of it's original location in the parsed source-code, but it's the leaf-node tokens that we care about for formatting and perhaps the form types they were a part of (ie. were they comments or in a list, etc).
So to keep things brief, the AST representation is a bit too cumbersome and superfluous to use for formatting directly. So the first step is recurse through it and optimize it, removing unneeded height and capturing crucial metadata that is actually relevant for formatting.
We build a `FormatterTree` which consists of `FormatterNode`s. Each `FormatterNode` has either a single token, or a list of `refs`. A `ref` is just another `FormatterNode`. Each node also has a bunch of metadata and configuration on how it should be formatted (just initialized for now).
Having a dead-simple representation of the code like this is really only possible because it's a LISP, but it's also future proofed for us. If OpenGOAL ever decides to support different brackets -- for example in Clojure a vector uses `[]` or a map uses `{}` it would be easy for update the formatter to wrap each list node in the right bracket type.
### Applying Formatting Configuration
Now that we have our simplified tree, we can get to actually formatting. The first step is to figure out _how_ we intend to format things. By separating this from the actual formatting it's a lot easier to reason about and find bugs. You can also do potentially interesting things, where you do a full traversal of your source-code tree to figure out an optimal decision without trying to build up the final formatted string at the same time.
It is lists, or `form`s that we care about figuring out how to format. Each `form` has a head element (unless it's empty) that we can use to specify how it would be formatted. For example we cannot specify the formatting for every single function name, but we can format a `defun` or a `let` in a special way. With this mechanism, the formatter is easily extensible and configurable. There is even the ability to specify formatting configuration for a list's children by index -- for example you may want to format the first element in a `let` (the binding list) differently than the rest of the code.
There are also some forms that don't have a head-item, that we still identify and set some default formatting rules:
- The top-level node - indentation is disabled
- Lists of only literals, called `constant lists` - These are indented with `1` space instead of the default `2` for a `flow`'d form.
- If these lists literals are paired with keywords we call these `constant pairs` - We strive to put the pairs on the same line, if they are too long we spill to the next with `2` space indentation
- TODO - cond/case pairs (similar to constant pairs)
This brings up two important terms that you will see in the formatter implementation a lot -- `flow` and `hang`. The difference is as such:
```clj
(println "this"
"form"
"is"
"hung / hanged")
```
```clj
(println
"this"
"form"
"is"
"flowed")
```
The difference between the two other than visually is that hanging always results in 1 less line spent _because we don't care about line-length_. This is great because it means we can always default to hanging unless overridden. Flowing trades off that 1 vertical line for horizontal space and looks great for block-style forms. Like the top-level of a `defun` or the content of a `let`.
### Applying Formatting
The final real-step in the process, actually using the configuration and generating the formatting code. This phase is actually broken up into 3 sub-phases as we recurse through the `FormatterTree`
#### Build Lines
First we build the source code lines without indentation, it's a lot easier to figure out the indentation after all tokens are on the correct lines instead of doing it in tandem. Most tokens are given their own line, but in the case of a hung form we combine the first two elements on the first line. Other forms are formatted in even more complicated ways, or comments that were originally inline should remain next to that token.
Secondly, once we have all our consolidated lines we can apply the indentation. This is most often based on the configuration of the list node itself. For example a `hung` form must be indented the width of a `(` and the length of the head-form, plus the whitespace separator -- conversely a `flowed` form only needs to be indented by `2` spaces, regardless of token length. Because all of these algorithms are recursive we can hyper-focus on getting the indentation and formatting correct from the bottom-up. In otherwords, we don't care if a form is 200 spaces indented into a function, once the lines are returned recursively, they will get another layer of indentation added on until we hit the top-level node.
### Finalize Lines
Finally, we concatenate our list of strings, respecting the original file's line-endings.
As it is best practice, if we are formatting an entire file we should terminate with a trailing new-line.
Both of these items are technically a TODO.
## Configuration
TODO
## Reference
This formatter is highly influenced by a lot of other existing formatters and writings. The most significant of which is Clojure's [zprint](https://github.com/kkinnear/zprint/tree/main/doc/options) which has incredible documentation around it's capabilities and why it chooses what to do.
The formatter I've built basically copies zprint in terms of basic functionality, but simplifies things by being even more opinionated (for example, doing far less in terms of trying to adhere to a line-length limit).
### Other references
- [The Hardest Program I’ve Ever Written - Dart Formatter](https://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/)
- [cljfmt](https://github.com/weavejester/cljfmt)
### Why not respect a line-length limit
The OpenGOAL formatter will respect line-length in only very specific situations:
- When determining if a form can be fit entirely on a single line
- Probably in the future for multi-line comments
Besides it adding a massive amount of complexity, even zprint explains that a line-length limit won't always be followed
> zprint will work very hard to fit the formatted output into this width, though there are limits to its effort. For instance, it will not reduce the minimum indents in order to satisfy a particular width requirement.
Let's illustrate with an example:
```clj
(defunsome-func
(+ 123
456
(+ 123
456
;; imagine this form exceeds our width, how can you "fix" this without
;; messing up the indentation levels of the entire expression
;; - you can't inline it (you're already past the width)
;; - you can't insert a line break because it'll still be at the same horizontal level
(+ 123
456
789))))
```
In LISP code, we don't have many ideal options to deal with this, all options are syntactically correct, but in terms of conventional readability they are bad because when reading LISP code you want to use the indentation as a guide (which is why new-line parens are considered ugly by basically all LISP style guides)
The other reason which is equally relevant is it is hard, it means the formatter would have to try to find an optimal solution by exploring different solutions until it finds the one that minimizes lines.
But with an opinionated formatting style you can avoid this complexity, if you want your code to look nice don't write code like the decompiler sometimes outputs when it encounters something truly cursed:
Loaded 100 of 13829 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Tyler Wilding
