i18n: add finnish to menus

In the original game, they had no choice but to use the memory card file
as their method of persisting settings. We are not limited by such
things.

It's inconvenient to have to load your save-file when launching the game
to initialize these settings to your liking, it's also confusing
behaviour to even some players that have played the game heavily for
over a decade. We can do better by globally saving these settings to the
`pc-settings` file instead.

Originally I only migrated the volume settings, then i figured it would
be nice to also have play-hints and subtitles settings persisted. More
could debatably be moved (language is a big one...) but these were the
low hanging fruit.

I also reduced the default volumes as that is something else that has
come up a few times.

.clang-format

@@ -10,7 +10,7 @@ IncludeCategories:
     Priority: 2
   - Regex: '^(<|")common.*'
     Priority: 3
-  - Regex: '^"third-party.*'
+  - Regex: '^"(third-party|fmt|curl).*'
     Priority: 4
     SortPriority: 5
   - Regex: '.*'

.github/ISSUE_TEMPLATE/jak1-bug-report.yml

@@ -7,7 +7,7 @@ body:
       label: Acknowledgements
       description: Take a couple minutes to help our maintainers work faster.
       options:
-        - label: I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+sort%3Aupdated-desc) for duplicate or closed bug reports
+        - label: I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Ajak1+sort%3Aupdated-desc+) for duplicate or closed bug reports
           required: true
         - label: I understand that I am supposed to provide my own legitimately obtained copy of the game
           required: true

.github/ISSUE_TEMPLATE/jak2-bug-report.yml

@@ -0,0 +1,74 @@
+name: "\U0001F41B Jak 2 Bug Report"
+description: Create a bug report for Jak 2.
+labels: ["bug", "jak2"]
+body:
+  - type: checkboxes
+    attributes:
+      label: Acknowledgements
+      description: Take a couple minutes to help our maintainers work faster.
+      options:
+        - label: I have [searched](https://github.com/open-goal/jak-project/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Ajak2+sort%3Aupdated-desc+) for duplicate or closed bug reports
+          required: true
+        - label: I understand that I am supposed to provide my own legitimately obtained copy of the game
+          required: true
+
+  - type: textarea
+    attributes:
+      label: Describe the Bug
+      description: A clear and concise description of what the bug is. You may post screenshots or videos of the bug here.
+    validations:
+      required: true
+
+  - type: textarea
+    attributes:
+      label: How To Reproduce
+      description: Steps to reproduce the behavior. You can also post a video of it here.
+    validations:
+      required: true
+
+  - type: dropdown
+    attributes:
+      label: Does this problem occur on original hardware or PCSX2?
+      description: Some things that may seem like bugs are actually exactly how the original game behaved.
+      options:
+        - Yes, it's unique to OpenGOAL
+        - Didn't check
+        - Not needed, bug is obvious
+    validations:
+      required: true
+
+  - type: textarea
+    attributes:
+      label: Expected Behavior
+      description: A clear and concise description of the expected behavior.
+      placeholder: When I do X, Y should happen.
+    validations:
+      required: true
+
+  - type: textarea
+    attributes:
+      label: Environment Information
+      description: "You can upload the [Support Package](https://github.com/open-goal/launcher#asking-for-help) provided by the Launcher here, or you can provide the following information: CPU, GPU, OS Version, OpenGOAL Version (found in the window's title bar)"
+    validations:
+      required: true
+
+  - type: dropdown
+    attributes:
+      label: Game Version
+      options:
+        - NTSC 1.0 (black label)
+        - NTSC Greatest Hits version (red label)
+        - PAL
+        - JP
+    validations:
+      required: true
+
+  - type: dropdown
+    attributes:
+      label: Have you set the game to something other than `60fps`?
+      options:
+        - "No"
+        - "Yes"
+    validations:
+      required: true
+

.github/scripts/releases/extract_build_unix.sh

@@ -32,4 +32,4 @@ cp -r $SOURCE/decompiler/config $DEST/data/decompiler/
 cp -r $SOURCE/goal_src $DEST/data
 cp -r $SOURCE/game/assets $DEST/data/game/
 cp -r $SOURCE/game/graphics/opengl_renderer/shaders $DEST/data/game/graphics/opengl_renderer
-cp -r $SOURCE/custom_levels $DEST/data
+cp -r $SOURCE/custom_assets $DEST/data

.github/scripts/releases/extract_build_windows.sh

@@ -24,4 +24,4 @@ cp -r $SOURCE/decompiler/config $DEST/data/decompiler/
 cp -r $SOURCE/goal_src $DEST/data
 cp -r $SOURCE/game/assets $DEST/data/game/
 cp -r $SOURCE/game/graphics/opengl_renderer/shaders $DEST/data/game/graphics/opengl_renderer
-cp -r $SOURCE/custom_levels $DEST/data
+cp -r $SOURCE/custom_assets $DEST/data

.github/workflows/draft-new-release.yaml

@@ -23,7 +23,7 @@ jobs:
       - name: Bump Version and Push Tag
         if: github.repository == 'open-goal/jak-project'
         id: tag_version
-        uses: mathieudutour/github-tag-action@v6.1
+        uses: mathieudutour/github-tag-action@v6.2
         with:
           github_token: ${{ secrets.BOT_PAT }}
           tag_prefix: v

.github/workflows/inform-pages-repo.yaml

@@ -22,7 +22,7 @@ jobs:
       BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Install Package Dependencies
         run: |
@@ -58,12 +58,16 @@ jobs:
         run: |
           ./build/goalc/goalc --cmd "(begin (make-group \"all-code\") (gen-docs \"${PWD}\"))" --game jak2
 
+      - name: Generate Docs For Jak 3
+        run: |
+          ./build/goalc/goalc --cmd "(begin (make-group \"all-code\") (gen-docs \"${PWD}\"))" --game jak3
+
       - name: Append File Docstrings
         run: |
-          python ./scripts/docs/append-file-docs.py
+          python ./scripts/ci/docs/append-file-docs.py
 
       - name: Upload Docs Artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: opengoal-docs
           if-no-files-found: error
@@ -75,7 +79,7 @@ jobs:
         run: echo ${{ github.run_id }}
 
       - name: Send Dispatch
-        uses: peter-evans/repository-dispatch@v2
+        uses: peter-evans/repository-dispatch@v3
         with:
           token: ${{ secrets.BOT_PAT }}
           repository: 'open-goal/open-goal.github.io'

.github/workflows/linting.yaml

@@ -16,7 +16,7 @@ jobs:
     timeout-minutes: 10
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       # wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
       # sudo add-apt-repository 'deb http://apt.llvm.org/focal/ llvm-toolchain-focal-15 main'
@@ -39,7 +39,10 @@ jobs:
     timeout-minutes: 10
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
+
+      - name: Check for Trailing Whitespace in goal_src
+        run: python ./scripts/ci/lint-trailing-whitespace.py
 
       - name: Check for Unresolved Conflicts
         run: python ./scripts/gsrc/check-for-conflicts.py
@@ -56,7 +59,7 @@ jobs:
     timeout-minutes: 10
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Install Python Dependencies
         run: pip install unidiff colorama

.github/workflows/linux-build-clang.yaml

@@ -20,16 +20,9 @@ jobs:
     runs-on: ubuntu-20.04
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Install Package Dependencies
         run: |
@@ -37,13 +30,15 @@ jobs:
           sudo apt install build-essential cmake \
             clang gcc g++ lcov make nasm libxrandr-dev \
             libxinerama-dev libxcursor-dev libpulse-dev \
-            libxi-dev zip ninja-build libgl1-mesa-dev
+            libxi-dev zip ninja-build libgl1-mesa-dev libssl-dev
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.13
         with:
-          cache_key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -51,8 +46,8 @@ jobs:
           CXX: clang++
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`nproc`))
@@ -71,7 +66,7 @@ jobs:
           strip ./build/lsp/lsp
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         if: ${{ inputs.uploadArtifacts }}
         with:
           name: opengoal-linux-${{ inputs.cachePrefix }}

.github/workflows/linux-build-gcc.yaml

@@ -16,30 +16,30 @@ jobs:
     runs-on: ubuntu-20.04
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Install Package Dependencies
         run: |
           sudo apt update
           sudo apt install build-essential cmake \
-            clang gcc g++ lcov make nasm libxrandr-dev \
+            clang gcc-10 g++-10 lcov make nasm libxrandr-dev \
             libxinerama-dev libxcursor-dev libpulse-dev \
-            libxi-dev zip ninja-build libgl1-mesa-dev
+            libxi-dev zip ninja-build libgl1-mesa-dev libssl-dev
 
-      - name: Setup Buildcache
-        uses: mikehardy/[email protected]
+            sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 100
+            sudo update-alternatives --set gcc /usr/bin/gcc-10
+            sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 100
+            sudo update-alternatives --set g++ /usr/bin/g++-10
+
+      - name: Setup sccache
+        uses: hendrikmuhs/[email protected]
         with:
-          cache_key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -48,18 +48,11 @@ jobs:
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
             -DCODE_COVERAGE=OFF \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`nproc`)) -- -w dupbuild=warn
 
       - name: Run Tests
         run: ./test.sh
-
-      # - name: Submit Coverage Report to Codacy
-      #   uses: codacy/codacy-coverage-reporter-action@v1
-      #   continue-on-error: true
-      #   with:
-      #     project-token: ${{ secrets.CODACY_PROJECT_KEY }}
-      #     coverage-reports: ./build/goalc-test_coverage.info

.github/workflows/macos-build-arm.yaml

@@ -13,19 +13,12 @@ on:
 jobs:
   build:
     name: ARM
-    runs-on: macos-latest
+    runs-on: macos-12
     timeout-minutes: 120
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up ARM64 environment
         run: sudo softwareupdate --install-rosetta --agree-to-license
@@ -33,11 +26,13 @@ jobs:
       - name: Install Package Dependencies
         run: arch -arm64 brew install cmake ninja
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.13
         with:
-          cache_key: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -45,8 +40,8 @@ jobs:
           CXX: clang++
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`sysctl -n hw.logicalcpu`))
@@ -55,7 +50,7 @@ jobs:
         run: ./test.sh
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: opengoal-macos-${{ inputs.cachePrefix }}
           if-no-files-found: error

.github/workflows/macos-build.yaml

@@ -17,28 +17,23 @@ on:
 jobs:
   build:
     name: Intel
-    runs-on: macos-11
+    runs-on: macos-12
     timeout-minutes: 120
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Install Package Dependencies
         run: brew install cmake nasm ninja
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.13
         with:
-          cache_key: macos-11-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -46,8 +41,8 @@ jobs:
           CXX: clang++
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`sysctl -n hw.logicalcpu`))
@@ -64,7 +59,7 @@ jobs:
           strip ./build/lsp/lsp
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         if: ${{ inputs.uploadArtifacts }}
         with:
           name: opengoal-macos-${{ inputs.cachePrefix }}

.github/workflows/release-pipeline.yaml

@@ -46,12 +46,12 @@ jobs:
     name: "Upload Artifacts"
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Prepare Artifact Folder
         run: mkdir -p ./ci-artifacts/final
 
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         name: Download all Artifacts
         with:
           path: ./ci-artifacts/

.github/workflows/update-controller-db.yaml

@@ -10,7 +10,7 @@ jobs:
     if: github.repository == 'open-goal/jak-project'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           token: ${{ secrets.BOT_PAT }}
 
@@ -19,7 +19,7 @@ jobs:
           wget -O ./game/assets/sdl_controller_db.txt https://raw.githubusercontent.com/gabomdq/SDL_GameControllerDB/master/gamecontrollerdb.txt
 
       - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v5
+        uses: peter-evans/create-pull-request@v6
         with:
           token: ${{ secrets.BOT_PAT }}
           author: 'OpenGOALBot <[email protected]>'

.github/workflows/windows-build-clang.yaml

@@ -21,16 +21,9 @@ jobs:
     # Set some sort of timeout in the event of run-away builds.  We are limited on concurrent jobs so, get rid of them.
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Install NASM
         # TODO - Simplify this with just the first command once choco 2.0 rolls out everywhere
@@ -42,29 +35,38 @@ jobs:
             choco install ${{ github.workspace }}/third-party/nasm/old/nasm.2.15.05.nupkg
           }
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.13
         with:
-          cache_key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          variant: sccache
+          key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - uses: ilammy/msvc-dev-cmd@v1
 
+      # https://github.com/actions/runner-images/issues/5459
+      # Causes binaries to be built differently, get rid of this garbage
+      - name: Remove Strawberry Perl from PATH
+        run: |
+          $env:PATH = $env:PATH -replace "C:\\Strawberry\\c\\bin;", ""
+          "PATH=$env:PATH" | Out-File -FilePath $env:GITHUB_ENV -Append
+
       - name: CMake Generation
         shell: cmd
-        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe .
+        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache .
 
       - name: Build Project
         shell: cmd
         run: cmake --build build --parallel %NUMBER_OF_PROCESSORS%
 
       - name: Run Tests
-        timeout-minutes: 10
         env:
           GTEST_OUTPUT: "xml:opengoal-test-report.xml"
         run: ./build/bin/goalc-test.exe --gtest_color=yes --gtest_brief=0 --gtest_filter="-*MANUAL_TEST*"
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         if: ${{ inputs.uploadArtifacts }}
         with:
           name: opengoal-windows-${{ inputs.cachePrefix }}

.github/workflows/windows-build-msvc.yaml

@@ -16,16 +16,9 @@ jobs:
     runs-on: windows-2022
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Install NASM
         # TODO - Simplify this with just the first command once choco 2.0 rolls out everywhere
@@ -37,16 +30,25 @@ jobs:
             choco install ${{ github.workspace }}/third-party/nasm/old/nasm.2.15.05.nupkg
           }
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.13
         with:
-          cache_key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          variant: sccache
+          key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - uses: ilammy/msvc-dev-cmd@v1
 
+      # Fixes an issue with the image causing builds to fail - https://github.com/actions/runner-images/issues/8598
+      - name: Remove Strawberry Perl from PATH
+        run: |
+          $env:PATH = $env:PATH -replace "C:\\Strawberry\\c\\bin;", ""
+          "PATH=$env:PATH" | Out-File -FilePath $env:GITHUB_ENV -Append
+
       - name: CMake Generation - MSVC
         shell: cmd
-        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe .
+        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache .
 
       - name: Build Project
         shell: cmd

.gitignore

@@ -54,7 +54,14 @@ imgui.ini
 node_modules/
 
 # texture replacements
-texture_replacements/*
+custom_assets/jak1/texture_replacements/*
+custom_assets/jak2/texture_replacements/*
+custom_assets/jak3/texture_replacements/*
+
+# merc replacements
+custom_assets/jak1/merc_replacements/*
+custom_assets/jak2/merc_replacements/*
+custom_assets/jak3/merc_replacements/*
 
 # generated cmake files
 svnrev.h
@@ -76,3 +83,4 @@ __pycache__/
 /TODO.md
 unifont-15.0.03.ttf
 *.diff
+goalc-report.html

.prettierrc

@@ -1 +0,0 @@
-endOfLine: "auto"

.prettierrc.json

@@ -0,0 +1,11 @@
+{
+  "endOfLine": "auto",
+  "overrides": [
+    {
+      "files": ["**/*.jsonc"],
+      "options": {
+        "parser": "json"
+      }
+    }
+  ]
+}

.vs/launch.vs.json

@@ -7,42 +7,58 @@
       "project": "CMakeLists.txt",
       "projectTarget": "goalc-test.exe (bin\\goalc-test.exe)",
       "name": "Tests - Unit-Tests - Summary",
-      "args": ["--gtest_brief=1"]
+      "args": [
+        "--gtest_brief=1"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc-test.exe (bin\\goalc-test.exe)",
       "name": "Tests - Unit-Tests - Verbose",
-      "args": ["--gtest_brief=0"]
+      "args": [
+        "--gtest_brief=0"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc-test.exe (bin\\goalc-test.exe)",
       "name": "Tests - Draft Tests - Verbose",
-      "args": ["--gtest_brief=0", "--gtest_filter=\"*Draft*\""]
+      "args": [
+        "--gtest_brief=0",
+        "--gtest_filter=\"*Draft*\""
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc-test.exe (bin\\goalc-test.exe)",
       "name": "Tests - TypeConsistency - Verbose",
-      "args": ["--gtest_brief=0", "--gtest_filter=\"*TypeConsistency*\""]
+      "args": [
+        "--gtest_brief=0",
+        "--gtest_filter=\"*TypeConsistency*\""
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc-test.exe (bin\\goalc-test.exe)",
       "name": "Tests - TypeConsistency - Jak 2 - Verbose",
-      "args": ["--gtest_brief=0", "--gtest_filter=\"*Jak2TypeConsistency*\""]
+      "args": [
+        "--gtest_brief=0",
+        "--gtest_filter=\"*Jak2TypeConsistency*\""
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc-test.exe (bin\\goalc-test.exe)",
       "name": "Tests - WithGameTests - Verbose",
-      "args": ["--gtest_brief=0", "--gtest_filter=\"*WithGameTests*\""]
+      "args": [
+        "--gtest_brief=0",
+        "--gtest_filter=\"*WithGameTests*\""
+      ]
     },
     {
       "type": "default",
@@ -68,6 +84,18 @@
         "jak2"
       ]
     },
+    {
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "offline-test.exe (bin\\offline-test.exe)",
+      "name": "Tests - Offline Tests - Jak 3",
+      "args": [
+        "--iso_data_path",
+        "${workspaceRoot}/iso_data/jak3",
+        "--game",
+        "jak3"
+      ]
+    },
     {
       "type": "default",
       "project": "CMakeLists.txt",
@@ -101,56 +129,109 @@
       "project": "CMakeLists.txt",
       "projectTarget": "gk.exe (bin\\gk.exe)",
       "name": "Game - Jak 1 - Runtime",
-      "args": ["-v", "--game", "jak1", "--", "-fakeiso", "-debug"]
+      "args": [
+        "-v",
+        "--game",
+        "jak1",
+        "--",
+        "-fakeiso",
+        "-debug"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "gk.exe (bin\\gk.exe)",
       "name": "Game - Jak 1 - Runtime (boot)",
-      "args": ["-v", "--game", "jak1", "--", "-boot", "-fakeiso", "-debug"]
+      "args": [
+        "-v",
+        "--game",
+        "jak1",
+        "--",
+        "-boot",
+        "-fakeiso",
+        "-debug"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "gk.exe (bin\\gk.exe)",
       "name": "Game - Jak 2 - Runtime (no boot)",
-      "args": ["-v", "--game", "jak2", "--", "-fakeiso", "-debug"]
+      "args": [
+        "-v",
+        "--game",
+        "jak2",
+        "--",
+        "-fakeiso",
+        "-debug"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "gk.exe (bin\\gk.exe)",
       "name": "Game - Jak 2 - Runtime (boot)",
-      "args": ["-v", "--game", "jak2", "--", "-boot", "-fakeiso", "-debug"]
+      "args": [
+        "-v",
+        "--game",
+        "jak2",
+        "--",
+        "-boot",
+        "-fakeiso",
+        "-debug"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "gk.exe (bin\\gk.exe)",
       "name": "Game - Jak 2 - Runtime (release)",
-      "args": ["-v", "--game", "jak2", "--", "-boot", "-fakeiso"]
+      "args": [
+        "-v",
+        "--game",
+        "jak2",
+        "--",
+        "-boot",
+        "-fakeiso"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
-      "projectTarget": "goalc.exe (bin\\goalc.exe)",
-      "name": "REPL",
-      "args": ["--user-auto"]
+      "projectTarget": "gk.exe (bin\\gk.exe)",
+      "name": "Game - Jak 3 - Runtime (boot)",
+      "args": [
+        "-v",
+        "--game",
+        "jak3",
+        "--",
+        "-boot",
+        "-fakeiso",
+        "-debug"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc.exe (bin\\goalc.exe)",
       "name": "REPL - Jak 1",
-      "args": ["--user-auto", "--game", "jak1"]
+      "args": [
+        "--user-auto",
+        "--game",
+        "jak1"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc.exe (bin\\goalc.exe)",
       "name": "REPL - Jak 2",
-      "args": ["--user-auto", "--game", "jak2"]
+      "args": [
+        "--user-auto",
+        "--game",
+        "jak2"
+      ]
     },
     {
       "type": "default",
@@ -218,19 +299,58 @@
         "--config-override \"{\\\"decompile_code\\\": false, \\\"levels_extract\\\": true, \\\"allowed_objects\\\": []}\""
       ]
     },
+    {
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "decompiler.exe (bin\\decompiler.exe)",
+      "name": "Decompiler - Jak 3",
+      "args": [
+        "${workspaceRoot}/decompiler/config/jak3/jak3_config.jsonc",
+        "${workspaceRoot}/iso_data",
+        "${workspaceRoot}/decompiler_out",
+        "--version",
+        "ntsc_v1"
+      ]
+    },
+    {
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "decompiler.exe (bin\\decompiler.exe)",
+      "name": "Decompiler - Jak 3 - Extract",
+      "args": [
+        "${workspaceRoot}/decompiler/config/jak3/jak3_config.jsonc",
+        "${workspaceRoot}/iso_data",
+        "${workspaceRoot}/decompiler_out",
+        "--version",
+        "ntsc_v1",
+        "--config-override \"{\\\"decompile_code\\\": false, \\\"levels_extract\\\": true, \\\"allowed_objects\\\": []}\""
+      ]
+    },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "memory_dump_tool.exe (bin\\memory_dump_tool.exe)",
       "name": "Tools - EE Memory Analyze - Jak 1",
-      "args": ["\"${workspaceRoot}/eeMemory.bin\"", "--output-path", "\"${workspaceRoot}\"", "--game", "jak1"]
+      "args": [
+        "\"${workspaceRoot}/eeMemory.bin\"",
+        "--output-path",
+        "\"${workspaceRoot}\"",
+        "--game",
+        "jak1"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "memory_dump_tool.exe (bin\\memory_dump_tool.exe)",
       "name": "Tools - EE Memory Analyze - Jak 2",
-      "args": ["\"${workspaceRoot}/eeMemory.bin\"", "--output-path", "\"${workspaceRoot}\"", "--game", "jak2"]
+      "args": [
+        "\"${workspaceRoot}/eeMemory.bin\"",
+        "--output-path",
+        "\"${workspaceRoot}\"",
+        "--game",
+        "jak2"
+      ]
     },
     {
       "type": "default",
@@ -247,35 +367,53 @@
       "project": "CMakeLists.txt",
       "projectTarget": "extractor.exe (bin\\extractor.exe)",
       "name": "Tools - Extractor - Full",
-      "args": ["\"E:\\ISOs\\Jak\\Jak 1.iso\""]
+      "args": [
+        "\"E:\\ISOs\\Jak\\Jak 1.iso\""
+      ]
     },
     {
-      "type" : "default",
-      "project" : "CMakeLists.txt",
-      "projectTarget" : "lsp.exe (bin\\lsp.exe)",
-      "name" : "Tools - LSP",
-      "args" : []
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "lsp.exe (bin\\lsp.exe)",
+      "name": "Tools - LSP",
+      "args": []
     },
     {
-      "type" : "default",
-      "project" : "CMakeLists.txt",
-      "projectTarget" : "type_searcher.exe (bin\\type_searcher.exe)",
-      "name" : "Tools - Type Searcher",
-      "args" : ["--game", "jak2", "--output-path", "./search-results.json", "--size", 255, "--fields", "[{\\\"type\\\":\\\"quaternion\\\",\\\"offset\\\":48}]"]
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "type_searcher.exe (bin\\type_searcher.exe)",
+      "name": "Tools - Type Searcher",
+      "args": [
+        "--game",
+        "jak2",
+        "--output-path",
+        "./search-results.json",
+        "--size",
+        255,
+        "--fields",
+        "[{\\\"type\\\":\\\"quaternion\\\",\\\"offset\\\":48}]"
+      ]
     },
     {
-      "type" : "default",
-      "project" : "CMakeLists.txt",
-      "projectTarget" : "formatter.exe (bin\\formatter.exe)",
-      "name" : "Tools - Formatter",
-      "args" : ["--new", "--file", "C:\\Users\\xtvas\\Repos\\opengoal\\jak-project\\test-formatter.gc"]
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "formatter.exe (bin\\formatter.exe)",
+      "name": "Tools - Formatter - Inplace",
+      "args": [
+        "--write",
+        "--file",
+        "C:\\Users\\xtvas\\Repositories\\opengoal\\jak-project\\goal_src\\jak1\\engine\\camera\\cam-states.gc"
+      ]
     },
     {
       "type": "default",
       "project": "CMakeLists.txt",
       "projectTarget": "goalc-test.exe (bin\\goalc-test.exe)",
       "name": "Tests - Formatter",
-      "args": ["--gtest_brief=0", "--gtest_filter=\"*FormatterTests*\""]
+      "args": [
+        "--gtest_brief=0",
+        "--gtest_filter=\"*FormatterTests*\""
+      ]
     }
   ]
-}
+}

.vscode/launch.json

@@ -5,13 +5,13 @@
   "version": "0.2.0",
   "configurations": [
     {
-      "name": "Append File Docs",
+      "name": "run python script",
       "type": "python",
       "request": "launch",
-      "program": "${workspaceFolder}/scripts/ci/lint-characters.py",
+      "program": "${workspaceFolder}/scripts/gsrc/compare-compilation-outputs.py",
       "console": "integratedTerminal",
       "cwd": "${workspaceFolder}",
-      "args": ["--fix"]
+      "args": []
     },
   ]
 }

.vscode/settings.json

@@ -5,8 +5,8 @@
       "comments": true,
       "strings": true
     },
-    "editor.wordBasedSuggestions": true,
+    "editor.wordBasedSuggestions": "matchingDocuments",
     "editor.snippetSuggestions": "top"
   },
-  "python.formatting.provider": "black"
+  "cmake.configureOnOpen": false
 }

CMakeLists.txt

@@ -1,7 +1,7 @@
 # Top Level CMakeLists.txt
 cmake_minimum_required(VERSION 3.10)
 
-set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD 20)
 project(jak)
 include(CTest)
 
@@ -18,14 +18,16 @@ option(STATICALLY_LINK "Build for release purposes (statically link everything)"
 message(STATUS "Statically Link? ${STATICALLY_LINK}")
 if(STATICALLY_LINK)
     set(BUILD_SHARED_LIBS OFF)
+    set(BUILD_STATIC_LIBS ON)
 else()
     set(BUILD_SHARED_LIBS ON)
+    set(BUILD_STATIC_LIBS OFF)
 endif()
 
 # For clangd
 if (EXISTS "${CMAKE_CURRENT_BINARY_DIR}/compile_commands.json" )
     configure_file(
-        "${CMAKE_CURRENT_BINARY_DIR}/compile_commands.json" 
+        "${CMAKE_CURRENT_BINARY_DIR}/compile_commands.json"
         "${PROJECT_SOURCE_DIR}/build/compile_commands.json")
 endif()
 
@@ -37,7 +39,6 @@ if(CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
         -fcxx-exceptions \
         -fexceptions \
         -fdiagnostics-color=always \
-        -std=c++17 \
         -mavx \
         -Wall \
         -Wno-c++11-narrowing \
@@ -84,7 +85,6 @@ elseif(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
         -Wdisabled-optimization \
         -Wformat \
         -Wextra \
-        -Wmissing-include-dirs \
         -Woverloaded-virtual \
         -Wredundant-decls \
         -Wshadow \
@@ -101,7 +101,6 @@ elseif(CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
     message(STATUS "AppleClang detected - Setting Defaults")
     set(CMAKE_CXX_FLAGS
         "${CMAKE_CXX_FLAGS} \
-        -march=native \
         -Wall \
         -Winit-self \
         -ggdb \
@@ -111,13 +110,18 @@ elseif(CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
         -Wdisabled-optimization \
         -Wformat \
         -Wextra \
-        -Wmissing-include-dirs \
         -Woverloaded-virtual \
         -Wredundant-decls \
         -Wshadow \
         -Wsign-promo \
         -fdiagnostics-color=always"
         )
+    
+    # pin to AVX for macOS, hopefully all macOS runners have atleast this architecture
+    # technically speaking, SSE4 is the cutoff for Apple Silicon so...only a matter of time!
+    if(NOT CMAKE_CXX_COMPILER_TARGET STREQUAL "arm64-apple-darwin")
+        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mavx")
+    endif()
 
     # additional c++ flags for release mode for our projects
     if(CMAKE_BUILD_TYPE MATCHES "Release")
@@ -183,6 +187,28 @@ include_directories(./)
 # build templating engine library
 include_directories(SYSTEM third-party/inja)
 
+# libcurl for HTTP requests
+# Enable SSL Support, most URLs now-a-days use SSL!
+# TODO - probably integrate with ZSTD since we have it already
+set(CURL_USE_LIBSSH2 OFF)
+set(CURL_ZLIB OFF)
+if(WIN32)
+    set(CURL_USE_SCHANNEL ON) # native Windows SSL support
+elseif(APPLE)
+    set(CURL_USE_SECTRANSP ON) # native macOS SSL support
+else()
+    if(STATICALLY_LINK)
+        set(OPENSSL_USE_STATIC_LIBS TRUE)
+    endif()
+    set(CURL_USE_OPENSSL ON) # not native, but seems to be the best choice for linux
+endif()
+include_directories(third-party/curl/include)
+if(STATICALLY_LINK)
+    build_third_party_lib(curl libcurl_static)
+else()
+    build_third_party_lib(curl libcurl_shared)
+endif()
+
 # build repl library
 build_third_party_lib(replxx replxx)
 
@@ -195,12 +221,15 @@ build_third_party_lib(sqlite3 sqlite3)
 
 # build tree-sitter parser
 include_directories(third-party/tree-sitter/tree-sitter/lib/include)
-include_directories(third-party/tree-sitter/tree-sitter-opengoal/include)
 build_third_party_lib(tree-sitter tree-sitter)
 
 # native OS dialogs for error messages
 build_third_party_lib(libtinyfiledialogs libtinyfiledialogs)
 
+# build format library
+include_directories(third-party/fmt/include)
+build_third_party_lib(fmt fmt)
+
 # build common library
 add_subdirectory(common)
 
@@ -253,9 +282,6 @@ include(test/CMakeLists.txt)
 # build lzokay library
 build_third_party_lib(lzokay lzokay)
 
-# build format library
-build_third_party_lib(fmt fmt)
-
 build_third_party_lib(stb_image stb_image)
 build_third_party_lib(tiny_gltf tiny_gltf)
 build_third_party_lib(xdelta3 xdelta3)

CMakePresets.json

@@ -108,6 +108,16 @@
       "description": "Build with Clang with Debug Symbols",
       "inherits": ["base-windows-debug", "base-clang"]
     },
+    {
+      "name": "Debug-windows-clang-static",
+      "displayName": "Windows Static Debug (clang)",
+      "description": "Build with Clang as Debug but statically linked",
+      "inherits": ["base-windows-debug", "base-clang"],
+      "cacheVariables": {
+        "STATICALLY_LINK": "true",
+        "ZYDIS_BUILD_SHARED_LIB": "OFF"
+      }
+    },
     {
       "name": "RelWithDebInfo-windows-clang",
       "displayName": "Windows RelWithDebInfo (clang)",

README.md

@@ -6,14 +6,24 @@
   <a href="https://opengoal.dev/docs/intro" rel="nofollow"><img src="https://img.shields.io/badge/Documentation-Here-informational" alt="Documentation Badge" style="max-width:100%;"></a>
   <a title="Crowdin" target="_blank" href="https://crowdin.com/project/opengoal"><img src="https://badges.crowdin.net/opengoal/localized.svg"></a>
   <a target="_blank" rel="noopener noreferrer" href="https://github.com/open-goal/jak-project/actions/workflows/build-matrix.yaml"><img src="https://github.com/open-goal/jak-project/actions/workflows/build-matrix.yaml/badge.svg" alt="Linux and Windows Build" style="max-width:100%;"></a>
-  <a href="https://www.codacy.com/gh/open-goal/jak-project/dashboard?utm_source=github.com&utm_medium=referral&utm_content=open-goal/jak-project&utm_campaign=Badge_Coverage" rel="nofollow"><img src="https://app.codacy.com/project/badge/Coverage/29316d04a1644aa390c33be07289f3f5" alt="Codacy Badge" style="max-width:100%;"></a>
   <a href="https://www.codacy.com/gh/open-goal/jak-project/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=open-goal/jak-project&amp;utm_campaign=Badge_Grade" rel="nofollow"><img src="https://app.codacy.com/project/badge/Grade/29316d04a1644aa390c33be07289f3f5" alt="Codacy Badge" style="max-width:100%;"></a>
   <a href="https://discord.gg/VZbXMHXzWv"><img src="https://img.shields.io/discord/756287461377703987" alt="Discord"></a>
-  <a href="https://makeapullrequest.com"><img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square" alt=PRs Welcome></a>
 </p>
 
-- [Please read first](#please-read-first)
-  - [Quick Start](#quick-start)
+## Please read first <!-- omit from toc -->
+
+> [!IMPORTANT]
+> Our repositories on GitHub are primarily for development of the project and tracking active issues. Most of the information you will find here pertains to setting up the project for development purposes and is not relevant to the end-user.
+
+For a setup guide on how to install and play the game there is the following video that you can check out: https://youtu.be/K84UUMnkJc4
+
+For questions or additional information pertaining to the project, we have a Discord for discussion here: https://discord.gg/VZbXMHXzWv
+
+Additionally, you can find further documentation and answers to **frequently asked questions** on the project's main website: https://opengoal.dev
+
+> [!WARNING]
+> **Do not use this decompilation project without providing your own legally purchased copy of the game.** We do not distribute any assets from the game - you must use your own legitimately obtained PS2 copy of the game. We support every retail PAL, NTSC, and NTSC-J build, including Greatest Hits copies.
+
 - [Project Description](#project-description)
   - [Current Status](#current-status)
   - [Methodology](#methodology)
@@ -46,22 +56,6 @@
   - [`goal_src/`](#goal_src)
   - [`game` runtime](#game-runtime)
 
-## Please read first
-
-Our repositories on GitHub are primarily for development of the project and tracking active issues. Most of the information you will find here pertains to setting up the project for development purposes and is not relevant to the end-user.
-
-For questions or additional information pertaining to the project, we have a Discord for discussion here: https://discord.gg/VZbXMHXzWv
-
-Additionally, you can find further documentation and answers to **frequently asked questions** on the project's main website: https://opengoal.dev
-
-**Do not use this decompilation project without providing your own legally purchased copy of the game.**
-
-### Quick Start
-
-If you just want to play the game, you can follow the steps in this video: https://www.youtube.com/watch?v=p8I9NfuZOgE
-
-We do not distribute any assets from the game - you must use your own legitimately obtained PS2 copy of the game. We support every retail PAL, NTSC, and NTSC-J build, including Greatest Hits copies. 
-
 ## Project Description
 
 This project is to port the original Jak and Daxter and Jak II to PC. Over 98% of the games are written in GOAL, a custom Lisp language developed by Naughty Dog. Our strategy is:
@@ -133,7 +127,7 @@ Unfortunately you'll still need task runner on your local machine to run the gam
 Install packages and init repository:
 
 ```sh
-sudo apt install gcc make cmake build-essential g++ nasm clang-format libxrandr-dev libxinerama-dev libxcursor-dev libpulse-dev libxi-dev python libgl1-mesa-dev
+sudo apt install gcc make cmake build-essential g++ nasm clang-format libxrandr-dev libxinerama-dev libxcursor-dev libpulse-dev libxi-dev python libgl1-mesa-dev libssl-dev
 sudo sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b /usr/local/bin
 ```
 

Taskfile.yml

@@ -15,6 +15,8 @@ tasks:
     - 'python ./scripts/tasks/update-env.py --game jak1'
   set-game-jak2:
     - 'python ./scripts/tasks/update-env.py --game jak2'
+  set-game-jak3:
+    - 'python ./scripts/tasks/update-env.py --game jak3'
   set-decomp-ntscv1:
     desc: "aka black label"
     cmds:
@@ -72,24 +74,26 @@ tasks:
         msg: "Couldn't locate compiler executable in '{{.GOALC_BIN_RELEASE_DIR}}/goalc'"
     cmds:
       - "{{.GOALC_BIN_RELEASE_DIR}}/goalc --user-auto --game {{.GAME}}"
-  format:
-    desc: "Format code"
-    cmds:
-      - cmd: python ./scripts/cpp/format-includes.py
-      - cmd: python ./third-party/run-clang-format/run-clang-format.py -r common decompiler game goalc test tools lsp -i
-      # npm install -g prettier
-      - cmd: npx prettier --write ./decompiler/config/jak1/**/*.jsonc
-        ignore_error: true
-      - cmd: npx prettier --write ./decompiler/config/jak2/**/*.jsonc
-        ignore_error: true
   format-json:
     desc: "Format code"
     cmds:
       # npm install -g prettier
-      - cmd: npx prettier --write ./decompiler/config/jak1/**/*.jsonc
-        ignore_error: true
-      - cmd: npx prettier --write ./decompiler/config/jak2/**/*.jsonc
+      - cmd: npx prettier --write ./decompiler/config/jak*/**/*.jsonc
         ignore_error: true
+  format:
+    desc: "Format code"
+    cmds:
+      - cmd: python ./scripts/cpp/format-includes.py
+      - cmd: python ./scripts/ci/lint-trailing-whitespace.py --fix
+      - cmd: python ./third-party/run-clang-format/run-clang-format.py -r common decompiler game goalc test tools lsp -i
+      - task: format-json
+  format-gsrc:
+    desc: "Run formatter on gsrc file"
+    preconditions:
+      - sh: test -f {{.FORMATTER_BIN_RELEASE_DIR}}/formatter{{.EXE_FILE_EXTENSION}}
+        msg: "Couldn't locate formatter executable in '{{.FORMATTER_BIN_RELEASE_DIR}}/formatter'"
+    cmds:
+      - "{{.FORMATTER_BIN_RELEASE_DIR}}/formatter --write --file '{{.FILE}}'"
   # DECOMPILING
   decomp:
     cmds:
@@ -102,7 +106,7 @@ tasks:
       - python ./scripts/tasks/clean-decomp.py --game "{{.GAME}}"
   lint-gsrc-file:
     cmds:
-      - python ./scripts/gsrc/lint-gsrc-file.py --game {{.GAME}} --file {{.FILE}}
+      - python ./scripts/gsrc/lint-gsrc-file.py --game {{.GAME}} --file "{{.FILE}}"
   update-gsrc:
     cmds:
       - python ./scripts/gsrc/update-gsrc-via-refs.py --game "{{.GAME}}" --decompiler "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler" --decompiler_config {{.DECOMP_CONFIG}} --version {{.DECOMP_CONFIG_VERSION}}
@@ -112,8 +116,26 @@ tasks:
   update-gsrc-file:
     cmds:
       - task: decomp-file
-      - python ./scripts/gsrc/update-from-decomp.py --game "{{.GAME}}" --file {{.FILE}}
+      - python ./scripts/gsrc/update-from-decomp.py --game "{{.GAME}}" --file "{{.FILE}}"
       - task: lint-gsrc-file
+  copy-common-naming:
+    cmds:
+      - python ./scripts/gsrc/copy-common-naming.py --file "{{.FILE}}" --decompiler "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler"
+      - task: format-json
+  copy-common-naming-from-refs:
+    cmds:
+      - python ./scripts/gsrc/copy-common-naming.py --update-names-from-refs --decompiler "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler"
+      - task: format-json
+  # ASSET RIPPING
+  rip-textures:
+    cmds:
+      - "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler \"./decompiler/config/{{.DECOMP_CONFIG}}\" \"./iso_data\" \"./decompiler_out\" --version \"{{.DECOMP_CONFIG_VERSION}}\" --config-override '{\"decompile_code\": false, \"levels_extract\": true, \"save_texture_pngs\": true}'"
+  rip-levels:
+    cmds:
+      - "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler \"./decompiler/config/{{.DECOMP_CONFIG}}\" \"./iso_data\" \"./decompiler_out\" --version \"{{.DECOMP_CONFIG_VERSION}}\" --config-override '{\"decompile_code\": false, \"levels_extract\": true, \"rip_levels\": true}'"
+  rip-collision:
+    cmds:
+      - "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler \"./decompiler/config/{{.DECOMP_CONFIG}}\" \"./iso_data\" \"./decompiler_out\" --version \"{{.DECOMP_CONFIG_VERSION}}\" --config-override '{\"decompile_code\": false, \"levels_extract\": true, \"extract_collision\": true, \"rip_collision\": true}'"
   # TOOLS
   analyze-ee-memory:
     cmds:
@@ -139,6 +161,9 @@ tasks:
     desc: Finds and tries to fix invalid translation Characters
     cmds:
       - python ./scripts/ci/lint-characters.py --fix
+  lint:
+    cmds:
+      - python ./scripts/ci/lint-trailing-whitespace.py
   # TESTS
   offline-tests: # ran by jenkins
     cmds:
@@ -164,9 +189,6 @@ tasks:
         ignore_error: true
       - python ./scripts/update_decomp_reference.py ./failures ./test/decompiler/reference/ --game {{.GAME}}
       - task: offline-test-file
-  type-test-jak1:
-    cmds:
-      - cmd: '{{.GOALCTEST_BIN_RELEASE_DIR}}/goalc-test --gtest_brief=0 --gtest_filter="*Jak1TypeConsistency*" --gtest_break_on_failure'
   type-test:
     cmds:
-      - cmd: '{{.GOALCTEST_BIN_RELEASE_DIR}}/goalc-test --gtest_brief=0 --gtest_filter="*Jak2TypeConsistency*" --gtest_break_on_failure'
+      - cmd: '{{.GOALCTEST_BIN_RELEASE_DIR}}/goalc-test --gtest_brief=0 --gtest_filter="*{{.TYPE_CONSISTENCY_TEST_FILTER}}*" --gtest_break_on_failure'

common/CMakeLists.txt

@@ -51,7 +51,7 @@ add_library(common
         repl/config.cpp
         repl/nrepl/ReplClient.cpp
         repl/nrepl/ReplServer.cpp
-        repl/util.cpp
+        repl/repl_wrapper.cpp
         serialization/subtitles/subtitles_v1.cpp
         serialization/subtitles/subtitles_v2.cpp
         serialization/subtitles/subtitles.cpp
@@ -66,6 +66,7 @@ add_library(common
         type_system/TypeSpec.cpp
         type_system/TypeSystem.cpp
         util/Assert.cpp
+        util/ast_util.cpp
         util/BitUtils.cpp
         util/compress.cpp
         util/crc32.cpp
@@ -86,10 +87,11 @@ add_library(common
         util/term_util.cpp
         util/Timer.cpp
         util/unicode_util.cpp
+        util/gltf_util.cpp
         versions/versions.cpp
         )
 
-target_link_libraries(common fmt lzokay replxx libzstd_static tree-sitter sqlite3 libtinyfiledialogs)
+target_link_libraries(common fmt lzokay replxx libzstd_static tree-sitter sqlite3 libtinyfiledialogs tiny_gltf)
 
 if(WIN32)
     target_link_libraries(common wsock32 ws2_32 windowsapp)

common/audio/audio_formats.cpp

@@ -3,7 +3,7 @@
 #include "common/log/log.h"
 #include "common/util/BinaryWriter.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 /*!
  * Write a wave file from a vector of samples.
@@ -43,7 +43,7 @@ void write_wave_file(const std::vector<s16>& left_samples,
       writer.add(sample);
     }
   } else {
-    for (int i = 0; i < left_samples.size(); i++) {
+    for (size_t i = 0; i < left_samples.size(); i++) {
       writer.add(left_samples.at(i));
       if (i < right_samples.size()) {
         writer.add(right_samples.at(i));
@@ -56,11 +56,14 @@ void write_wave_file(const std::vector<s16>& left_samples,
   writer.write_to_file(name);
 }
 
-std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader, const bool mono) {
+std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader,
+                                                           const bool stereo) {
   std::vector<s16> left_samples;
   std::vector<s16> right_samples;
   s32 left_sample_prev[2] = {0, 0};
   s32 right_sample_prev[2] = {0, 0};
+  bool first_left = true;
+  bool first_right = true;
   constexpr s32 f1[5] = {0, 60, 115, 98, 122};
   constexpr s32 f2[5] = {0, 0, -52, -55, -60};
 
@@ -71,23 +74,38 @@ std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader,
   // instead they are partitioned into contiguous 8kb (thats 8192 bytes) chunks
   // alternating left/right
   bool processing_left_chunk = true;
+  // We need to skip the vag header for each channel
+  if (first_left) {
+    reader.ffwd(48);
+    bytes_read += 48;
+    first_left = false;
+  }
   while (true) {
     if (!reader.bytes_left()) {
       break;
     }
 
-    if (bytes_read == 0x2000) {
+    if (stereo && bytes_read == 0x2000) {
       // switch streams
       processing_left_chunk = !processing_left_chunk;
       bytes_read = 0;
+
+      // We need to skip the vag header for each channel
+      if (first_right) {
+        // skip the header
+        reader.ffwd(48);
+        bytes_read += 48;
+        first_right = false;
+      }
     }
 
     u8 shift_filter = reader.read<u8>();
     u8 shift = shift_filter & 0b1111;
     u8 filter = shift_filter >> 4;
     u8 flags = reader.read<u8>();
+    (void)flags;
 
-    // removed assertions here
+    // removed assertions here (and that's probably why the audio doesn't sound right)
 
     u8 input_buffer[14];
 
@@ -106,7 +124,7 @@ std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader,
       s32 sample = (s32)(s16)(nibble << 12);
       sample >>= shift;
 
-      if (mono || processing_left_chunk) {
+      if (!stereo || processing_left_chunk) {
         sample += (left_sample_prev[0] * f1[filter] + left_sample_prev[1] * f2[filter] + 32) / 64;
 
         if (sample > 0x7fff) {

common/cross_os_debug/xdbg.cpp

@@ -11,7 +11,7 @@
 #include "common/goal_constants.h"
 #include "common/util/Timer.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef __linux
 #include <fcntl.h>

common/cross_sockets/XSocket.cpp

@@ -19,7 +19,7 @@
 #include <stdio.h>
 #include <string.h>
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 #include "common/log/log.h"
 // clang-format on
 

common/cross_sockets/XSocketClient.cpp

@@ -5,6 +5,8 @@
 #include "common/cross_sockets/XSocket.h"
 #include "common/log/log.h"
 
+#include "fmt/core.h"
+
 // clang-format off
 #ifdef _WIN32
 #define NOMINMAX
@@ -13,9 +15,7 @@
 #include <WinSock2.h>
 #include <WS2tcpip.h>
 #endif
-#include "common/repl/nrepl/ReplServer.h"
 
-#include "third-party/fmt/core.h"
 // clang-format on
 
 XSocketClient::XSocketClient(int _tcp_port) {

common/cross_sockets/XSocketServer.cpp

@@ -4,7 +4,7 @@
 #include "common/cross_sockets/XSocket.h"
 #include "common/common_types.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef _WIN32
 #define NOMINMAX
@@ -35,7 +35,7 @@ void XSocketServer::shutdown_server() {
   close_server_socket();
 }
 
-bool XSocketServer::init_server() {
+bool XSocketServer::init_server(bool failure_may_occur) {
   listening_socket = open_socket(AF_INET, SOCK_STREAM, 0);
   if (listening_socket < 0) {
     listening_socket = -1;
@@ -76,19 +76,27 @@ bool XSocketServer::init_server() {
   addr.sin_port = htons(tcp_port);
 
   if (bind(listening_socket, (sockaddr*)&addr, sizeof(addr)) < 0) {
-    lg::error("[XSocketServer:{}] failed to bind", tcp_port);
+    if (failure_may_occur) {
+      lg::debug("[XSocketServer:{}] failed to bind", tcp_port);
+    } else {
+      lg::error("[XSocketServer:{}] failed to bind", tcp_port);
+    }
     close_server_socket();
     return false;
   }
 
   if (listen(listening_socket, 0) < 0) {
-    lg::error("[XSocketServer:{}] failed to listen", tcp_port);
+    if (failure_may_occur) {
+      lg::debug("[XSocketServer:{}] failed to listen", tcp_port);
+    } else {
+      lg::error("[XSocketServer:{}] failed to listen", tcp_port);
+    }
     close_server_socket();
     return false;
   }
 
   server_initialized = true;
-  lg::info("[XSocketServer:{}] initialized", tcp_port);
+  lg::debug("[XSocketServer:{}] initialized", tcp_port);
   post_init();
   return true;
 }

common/cross_sockets/XSocketServer.h

@@ -20,7 +20,7 @@ class XSocketServer {
   XSocketServer(const XSocketServer&) = delete;
   XSocketServer& operator=(const XSocketServer&) = delete;
 
-  bool init_server();
+  bool init_server(bool failure_may_occur = false);
   void shutdown_server();
   void close_server_socket();
 

common/custom_data/TFrag3Data.cpp

@@ -9,10 +9,16 @@
 #include "third-party/sse2neon/sse2neon.h"
 #endif
 
+#include "common/log/log.h"
 #include "common/util/Assert.h"
 
 namespace tfrag3 {
 
+void PackedTimeOfDay::serialize(Serializer& ser) {
+  ser.from_pod_vector(&data);
+  ser.from_ptr(&color_count);
+}
+
 void PackedTieVertices::serialize(Serializer& ser) {
   ser.from_pod_vector(&color_indices);
   ser.from_pod_vector(&matrices);
@@ -42,6 +48,7 @@ void ShrubDraw::serialize(Serializer& ser) {
   ser.from_ptr(&num_triangles);
   ser.from_ptr(&first_index_index);
   ser.from_ptr(&num_indices);
+  ser.from_ptr(&proto_idx);
 }
 
 void InstancedStripDraw::serialize(Serializer& ser) {
@@ -70,10 +77,9 @@ void TfragTree::serialize(Serializer& ser) {
     draw.serialize(ser);
   }
 
-  // ser.from_pod_vector(&vertices);
   ser.from_pod_vector(&packed_vertices.vertices);
   ser.from_pod_vector(&packed_vertices.cluster_origins);
-  ser.from_pod_vector(&colors);
+  colors.serialize(ser);
   bvh.serialize(ser);
   ser.from_ptr(&use_strips);
 }
@@ -397,7 +403,7 @@ void TieTree::serialize(Serializer& ser) {
   }
 
   packed_vertices.serialize(ser);
-  ser.from_pod_vector(&colors);
+  colors.serialize(ser);
   bvh.serialize(ser);
 
   ser.from_ptr(&has_per_proto_visibility_toggle);
@@ -405,7 +411,7 @@ void TieTree::serialize(Serializer& ser) {
 }
 
 void ShrubTree::serialize(Serializer& ser) {
-  ser.from_pod_vector(&time_of_day_colors);
+  time_of_day_colors.serialize(ser);
   ser.from_pod_vector(&indices);
   packed_vertices.serialize(ser);
   if (ser.is_saving()) {
@@ -416,6 +422,33 @@ void ShrubTree::serialize(Serializer& ser) {
   for (auto& draw : static_draws) {
     draw.serialize(ser);
   }
+
+  ser.from_ptr(&has_per_proto_visibility_toggle);
+  ser.from_string_vector(&proto_names);
+}
+
+void HfragmentBucket::serialize(Serializer& ser) {
+  ser.from_pod_vector(&corners);
+  ser.from_ptr(&montage_table);
+}
+
+void Hfragment::serialize(Serializer& ser) {
+  ser.from_pod_vector(&vertices);
+  ser.from_pod_vector(&indices);
+  ser.from_pod_vector(&corners);
+
+  if (ser.is_saving()) {
+    ser.save<size_t>(buckets.size());
+  } else {
+    buckets.resize(ser.load<size_t>());
+  }
+  for (auto& x : buckets) {
+    x.serialize(ser);
+  }
+  time_of_day_colors.serialize(ser);
+  ser.from_ptr(&wang_tree_tex_id);
+  ser.from_ptr(&draw_mode);
+  ser.from_ptr(&occlusion_offset);
 }
 
 void BVH::serialize(Serializer& ser) {
@@ -459,6 +492,7 @@ void MercDraw::serialize(Serializer& ser) {
   ser.from_ptr(&first_index);
   ser.from_ptr(&index_count);
   ser.from_ptr(&num_triangles);
+  ser.from_ptr(&no_strip);
 }
 
 void Blerc::serialize(Serializer& ser) {
@@ -600,6 +634,8 @@ void Level::serialize(Serializer& ser) {
     tree.serialize(ser);
   }
 
+  hfrag.serialize(ser);
+
   collision.serialize(ser);
   merc_data.serialize(ser);
 
@@ -650,8 +686,7 @@ void PackedShrubVertices::memory_usage(MemoryUsageTracker* tracker) const {
 }
 
 void ShrubTree::memory_usage(MemoryUsageTracker* tracker) const {
-  tracker->add(MemoryUsageCategory::SHRUB_TIME_OF_DAY,
-               sizeof(TimeOfDayColor) * time_of_day_colors.size());
+  tracker->add(MemoryUsageCategory::SHRUB_TIME_OF_DAY, sizeof(u8) * time_of_day_colors.data.size());
   packed_vertices.memory_usage(tracker);
   tracker->add(MemoryUsageCategory::SHRUB_DRAW, sizeof(ShrubDraw) * static_draws.size());
   tracker->add(MemoryUsageCategory::SHRUB_IND, sizeof(u32) * indices.size());
@@ -679,7 +714,7 @@ void TieTree::memory_usage(MemoryUsageTracker* tracker) const {
                  draw.vis_groups.size() * sizeof(StripDraw::VisGroup));
   }
   packed_vertices.memory_usage(tracker);
-  tracker->add(MemoryUsageCategory::TIE_TIME_OF_DAY, sizeof(TimeOfDayColor) * colors.size());
+  tracker->add(MemoryUsageCategory::TIE_TIME_OF_DAY, sizeof(u8) * colors.data.size());
 
   for (auto& draw : instanced_wind_draws) {
     draw.memory_usage(tracker);
@@ -703,7 +738,7 @@ void TfragTree::memory_usage(MemoryUsageTracker* tracker) const {
                  draw.vis_groups.size() * sizeof(StripDraw::VisGroup));
   }
   packed_vertices.memory_usage(tracker);
-  tracker->add(MemoryUsageCategory::TFRAG_TIME_OF_DAY, sizeof(TimeOfDayColor) * colors.size());
+  tracker->add(MemoryUsageCategory::TFRAG_TIME_OF_DAY, sizeof(u8) * colors.data.size());
   tracker->add(MemoryUsageCategory::TFRAG_BVH, sizeof(VisNode) * bvh.vis_nodes.size());
 }
 
@@ -716,6 +751,13 @@ void IndexTexture::memory_usage(MemoryUsageTracker* tracker) const {
   tracker->add(MemoryUsageCategory::SPECIAL_TEXTURE, 256 * 4);  // clut
 }
 
+void Hfragment::memory_usage(tfrag3::MemoryUsageTracker* tracker) const {
+  tracker->add(MemoryUsageCategory::HFRAG_VERTS, vertices.size() * sizeof(HfragmentVertex));
+  tracker->add(MemoryUsageCategory::HFRAG_INDEX, indices.size() * sizeof(u32));
+  tracker->add(MemoryUsageCategory::HFRAG_TIME_OF_DAY, time_of_day_colors.data.size() * sizeof(u8));
+  tracker->add(MemoryUsageCategory::HFRAG_CORNERS, corners.size() * sizeof(HfragmentCorner));
+}
+
 void Level::memory_usage(MemoryUsageTracker* tracker) const {
   for (const auto& texture : textures) {
     texture.memory_usage(tracker);
@@ -736,6 +778,7 @@ void Level::memory_usage(MemoryUsageTracker* tracker) const {
   for (const auto& tree : shrub_trees) {
     tree.memory_usage(tracker);
   }
+  hfrag.memory_usage(tracker);
   collision.memory_usage(tracker);
   merc_data.memory_usage(tracker);
 }
@@ -779,6 +822,11 @@ void print_memory_usage(const tfrag3::Level& lev, int uncompressed_data_size) {
       {"merc-mod-draw-1", mem_use.data[tfrag3::MemoryUsageCategory::MERC_MOD_DRAW_1]},
       {"merc-mod-draw-2", mem_use.data[tfrag3::MemoryUsageCategory::MERC_MOD_DRAW_2]},
       {"blerc", mem_use.data[tfrag3::MemoryUsageCategory::BLERC]},
+      {"hfrag-verts", mem_use.data[tfrag3::MemoryUsageCategory::HFRAG_VERTS]},
+      {"hfrag-index", mem_use.data[tfrag3::MemoryUsageCategory::HFRAG_INDEX]},
+      {"hfrag-time-of-day", mem_use.data[tfrag3::MemoryUsageCategory::HFRAG_TIME_OF_DAY]},
+      {"hfrag-corners", mem_use.data[tfrag3::MemoryUsageCategory::HFRAG_CORNERS]}
+
   };
   for (auto& known : known_categories) {
     total_accounted += known.second;
@@ -791,8 +839,8 @@ void print_memory_usage(const tfrag3::Level& lev, int uncompressed_data_size) {
 
   for (const auto& x : known_categories) {
     if (x.second) {
-      fmt::print("{:30s} : {:6d} kB {:3.1f}%\n", x.first, x.second / 1024,
-                 100.f * (float)x.second / uncompressed_data_size);
+      lg::print("{:30s} : {:6d} kB {:3.1f}%\n", x.first, x.second / 1024,
+                100.f * (float)x.second / uncompressed_data_size);
     }
   }
 }

common/custom_data/Tfrag3Data.h

@@ -18,7 +18,7 @@ namespace tfrag3 {
 // - if changing any large things (vertices, vis, bvh, colors, textures) update get_memory_usage
 // - if adding a new category to the memory usage, update extract_level to print it.
 
-constexpr int TFRAG3_VERSION = 38;
+constexpr int TFRAG3_VERSION = 41;
 
 enum MemoryUsageCategory {
   TEXTURE,
@@ -61,6 +61,11 @@ enum MemoryUsageCategory {
   MERC_MOD_TABLE,
   BLERC,
 
+  HFRAG_VERTS,
+  HFRAG_INDEX,
+  HFRAG_TIME_OF_DAY,
+  HFRAG_CORNERS,
+
   COLLISION,
 
   NUM_CATEGORIES
@@ -219,6 +224,8 @@ struct ShrubDraw {
 
   // for debug counting.
   u32 num_triangles = 0;
+
+  u16 proto_idx = 0;
   void serialize(Serializer& ser);
 };
 
@@ -267,18 +274,23 @@ struct BVH {
   void serialize(Serializer& ser);
 };
 
-// A time-of-day color. Each stores 8 colors. At a given "time of day", they are interpolated
-// to find a single color which goes into a color palette.
-struct TimeOfDayColor {
-  math::Vector<u8, 4> rgba[8];
+// This is split into groups of 4 colors.
+// The data in these groups is stored first by palette, then color, then channel.
+struct PackedTimeOfDay {
+  std::vector<u8> data;
+  u32 color_count = 0;
+  void serialize(Serializer& ser);
 
-  bool operator==(const TimeOfDayColor& other) const {
-    for (size_t i = 0; i < 8; i++) {
-      if (rgba[i] != other.rgba[i]) {
-        return false;
-      }
-    }
-    return true;
+  u8 read(int color, int palette, int channel) const {
+    const int color_quad = color / 4;
+    const int color_in_quad = color % 4;
+    return data[color_quad * 4 * 4 * 8 + palette * 4 * 4 + color_in_quad * 4 + channel];
+  }
+
+  u8& read(int color, int palette, int channel) {
+    const int color_quad = color / 4;
+    const int color_in_quad = color % 4;
+    return data[color_quad * 4 * 4 * 8 + palette * 4 * 4 + color_in_quad * 4 + channel];
   }
 };
 
@@ -317,8 +329,8 @@ struct TfragTree {
   TFragmentTreeKind kind;        // our tfrag kind
   std::vector<StripDraw> draws;  // the actual topology and settings
   PackedTfragVertices packed_vertices;
-  std::vector<TimeOfDayColor> colors;  // vertex colors (pre-interpolation)
-  BVH bvh;                             // the bvh for frustum culling
+  PackedTimeOfDay colors;  // vertex colors (pre-interpolation)
+  BVH bvh;                 // the bvh for frustum culling
   bool use_strips = true;
 
   struct {
@@ -399,7 +411,7 @@ struct TieTree {
   std::array<u32, kNumTieCategories + 1> category_draw_indices;
 
   PackedTieVertices packed_vertices;
-  std::vector<TimeOfDayColor> colors;  // vertex colors (pre-interpolation)
+  PackedTimeOfDay colors;  // vertex colors (pre-interpolation)
 
   std::vector<InstancedStripDraw> instanced_wind_draws;
   std::vector<TieWindInstance> wind_instance_info;
@@ -420,7 +432,7 @@ struct TieTree {
 
 struct ShrubTree {
   // todo some visibility structure
-  std::vector<TimeOfDayColor> time_of_day_colors;  // multiplier colors
+  PackedTimeOfDay time_of_day_colors;  // multiplier colors
 
   PackedShrubVertices packed_vertices;
   std::vector<ShrubDraw> static_draws;  // the actual topology and settings
@@ -430,11 +442,52 @@ struct ShrubTree {
     std::vector<ShrubGpuVertex> vertices;  // mesh vertices
   } unpacked;
 
+  // jak 2 and later can toggle on and off visibility per proto by name
+  bool has_per_proto_visibility_toggle = false;
+  std::vector<std::string> proto_names;
+
   void serialize(Serializer& ser);
   void memory_usage(MemoryUsageTracker* tracker) const;
   void unpack();
 };
 
+struct HfragmentVertex {
+  float height = 0;
+  u32 vi = 0;
+  u16 color_index = 0;
+  u8 u = 0, v = 0;
+  u32 pad = 0;
+};
+
+struct HfragmentCorner {
+  math::Vector<float, 4> bsphere;
+  u32 vis_id = 0;
+  u32 index_start = 0;
+  u32 index_length = 0;
+  u32 num_tris = 0;
+};
+
+struct HfragmentBucket {
+  std::vector<u32> corners;
+  std::array<u16, 16> montage_table;
+  void serialize(Serializer& ser);
+};
+
+struct Hfragment {
+  std::vector<HfragmentVertex> vertices;
+  std::vector<u32> indices;
+  std::vector<HfragmentCorner> corners;
+  std::vector<HfragmentBucket> buckets;
+  PackedTimeOfDay time_of_day_colors;
+
+  std::array<s32, 4> wang_tree_tex_id;
+  DrawMode draw_mode;
+  u32 occlusion_offset;
+
+  void serialize(Serializer& ser);
+  void memory_usage(MemoryUsageTracker* tracker) const;
+};
+
 struct CollisionMesh {
   struct Vertex {
     float x, y, z;
@@ -477,6 +530,8 @@ struct MercDraw {
   u32 first_index;
   u32 index_count;
   u32 num_triangles;
+  // no strip hack for custom models
+  bool no_strip = false;
   void serialize(Serializer& ser);
 };
 
@@ -560,6 +615,7 @@ struct Level {
   std::array<std::vector<TfragTree>, TFRAG_GEOS> tfrag_trees;
   std::array<std::vector<TieTree>, TIE_GEOS> tie_trees;
   std::vector<ShrubTree> shrub_trees;
+  Hfragment hfrag;
   CollisionMesh collision;
   MercModelGroup merc_data;
   u16 version2 = TFRAG3_VERSION;

common/dma/dma.cpp

@@ -2,9 +2,9 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
-std::string DmaTag::print() {
+std::string DmaTag::print() const {
   std::string result;
   const char* mode_names[8] = {"refe", "cnt", "next", "ref", "refs", "call", "ret", "end"};
   result += fmt::format("TAG: 0x{:08x} {:4s} qwc 0x{:04x}", addr, mode_names[(int)kind], qwc);
@@ -15,7 +15,7 @@ std::string DmaTag::print() {
   return result;
 }
 
-std::string VifCode::print() {
+std::string VifCode::print() const {
   std::string result;
 
   switch (kind) {

common/dma/dma.h

@@ -11,7 +11,7 @@
 #include "common/common_types.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 struct DmaStats {
   double sync_time_ms = 0;
@@ -52,7 +52,7 @@ struct DmaTag {
 
   bool operator!=(const DmaTag& other) const { return !((*this) == other); }
 
-  std::string print();
+  std::string print() const;
 };
 
 inline void emulate_dma(const void* source_base, void* dest_base, u32 tadr, u32 dadr) {
@@ -148,7 +148,7 @@ struct VifCode {
   u16 num;
   u16 immediate;
 
-  std::string print();
+  std::string print() const;
 };
 
 struct VifCodeStcycl {

common/dma/dma_copy.cpp

@@ -6,7 +6,7 @@
 #include "common/log/log.h"
 #include "common/util/Timer.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 /*!
  * Convert a DMA chain to an array of bytes that can be directly fed to VIF.

common/dma/gs.cpp

@@ -2,8 +2,8 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
-#include "third-party/fmt/format.h"
+#include "fmt/core.h"
+#include "fmt/format.h"
 
 std::string reg_descriptor_name(GifTag::RegisterDescriptor reg) {
   switch (reg) {

common/formatter/README.md

@@ -0,0 +1,185 @@
+# OpenGOAL Formatter Documentation
+
+> [!IMPORTANT]
+> This is still in a highly experimental state, adjust your expectations accordingly.
+
+Despite LISP being very simple, and taking the stance of a highly opinionated formatter that doesn't concern itself with line-length in most situations writing a code formatter is still an incredibly annoying and difficult thing to do.  This documentation serves as a place for me to collect my thoughts and explain how it works (or more likely, how it does not).
+
+Also in general I found it _really_ hard to find modern and easy to understand articles on how to get started (with the exception of zprint's usage docs, but that doesn't really cover implementation ideas).  This area is either dominated by 40 year old journal articles, or discussions around really difficult formatting projects.  There isn't a whole lot in the vain of _here's how I made a decent, simple formatter_, so maybe it'll help someone else (even if it's teaching them potentially what not to do!)
+
+## Architecture / Overview
+
+Atleast for me, it helps to understand the path code takes as it flows through the formatting process.  Originally I tried to do as much at once for sake of efficiency but this just makes things incredibly hard to reason about and fix bugs.  So like any problem that is complex, break it down, the formatter goes through many distinct phases chipping away at the problem:
+
+```mermaid
+flowchart TB
+  subgraph top1 [Build a Formatting Tree]
+    direction TB
+    a1(AST Parse) --> a2(Consolidate AST Nodes)
+    a1 --> a3(Collect Metadata)
+  end
+
+  subgraph top2 [Apply Formatting Configuration]
+    direction TB
+    b1(Recursively Iterate each Node) --> b2{Form head has\na predefined config?}
+    b2 --> |yes|b3(Fetch it and set the config on the node)
+    b2 --> |no|b4(Apply a default configuration)
+  end
+
+  subgraph top3 [Apply Formatting]
+    direction TB
+    c1(For each node, recursively build up a\nlist of source code lines, no indentation) --> c2(Consolidate the first two lines if hanging\nor if the config demands it)
+    c2 --> c3(Append indentation to each line as appropriate\nand surround each form with relevant parens)
+  end
+
+  subgraph top4 [Finalize Lines]
+    direction TB
+    d1(Concatenate all lines with whatever\nfile ending the file started with) --> d2(If we're formatting a file append a trailing new-line\n otherwise we are done)
+  end
+
+  top1-->|FormatterTree|top2
+  top2-->|FormatterTree with Configurations|top3
+  top3-->|String List|top4
+```
+
+### Build a Formatting Tree
+
+At the end of the day, formatting is about changing the location and amounts of white-space in a file.  Meaning it would be ideal if we could start with a representation that is whitespace-agnostic.  Some formatters (like cljfmt) choose to use the original whitespace as an input into the formatting process, essentially meaning that your code should already be somewhat-formatted.  This is somewhat advantageous as it can simplify the formatter's implementation or be a bit more flexible as to what the original programmer intended (assuming they knew what they were doing).  But of course it has one fatal flaw -- the code has to already be well formatted.
+
+For our formatter I wanted it to be able to take in horribly, ugly, minified code and output an optimal result -- especially since so much of our code gets spat out by another tool.  To do that, we need to be able to format the code without any existing whitespace helping us, adding in the whitespace from the ground up.
+
+The easiest way to do this is to construct an AST of the language which has a lot of other benefits for building language features but for here all we care about is getting rid of the whitespace and having some basic interpretation of the source code structure.  The easiest way now-a-days to make an AST for a language is to create a [treesitter grammar](https://github.com/open-goal/tree-sitter-opengoal).
+
+#### Consolidate AST Nodes and Collect Metadata
+
+One downside with using an AST, atleast with the way the grammar is currently implemented is it creates a somewhat difficult tree to work with from a formatting perspective.  For example in a LISP, everything is a list.  In our AST it does not look like how you may intuitively expect:
+
+`(+ 1 2)`
+
+```mermaid
+flowchart TB
+    a("source\n(+ 1 2)") --> b("list_lit\n")
+    b --> c("(") --> ca("(")
+    b --> d("sym_lit\n+") --> da("+")
+    b --> e("sym_lit\n1") --> ea("1")
+    b --> f("sym_lit\n2") --> fa("2")
+    b --> g(")") --> ga(")")
+```
+
+vs what you would probably prefer:
+
+```mermaid
+flowchart TB
+    a("top-level") --> b("list\n- surrounded in ()")
+    b --> da("+")
+    b --> ea("1")
+    b --> fa("2")
+```
+
+As you can see from even this simple example, the AST representation can be very verbose and in a sense has duplication.  Each node is aware of it's original location in the parsed source-code, but it's the leaf-node tokens that we care about for formatting and perhaps the form types they were a part of (ie. were they comments or in a list, etc).
+
+So to keep things brief, the AST representation is a bit too cumbersome and superfluous to use for formatting directly.  So the first step is recurse through it and optimize it, removing unneeded height and capturing crucial metadata that is actually relevant for formatting.
+
+We build a `FormatterTree` which consists of `FormatterNode`s.  Each `FormatterNode` has either a single token, or a list of `refs`.  A `ref` is just another `FormatterNode`.  Each node also has a bunch of metadata and configuration on how it should be formatted (just initialized for now).
+
+Having a dead-simple representation of the code like this is really only possible because it's a LISP, but it's also future proofed for us.  If OpenGOAL ever decides to support different brackets -- for example in Clojure a vector uses `[]` or a map uses `{}` it would be easy for update the formatter to wrap each list node in the right bracket type.
+
+### Applying Formatting Configuration
+
+Now that we have our simplified tree, we can get to actually formatting.  The first step is to figure out _how_ we intend to format things.  By separating this from the actual formatting it's a lot easier to reason about and find bugs.  You can also do potentially interesting things, where you do a full traversal of your source-code tree to figure out an optimal decision without trying to build up the final formatted string at the same time.
+
+It is lists, or `form`s that we care about figuring out how to format.  Each `form` has a head element (unless it's empty) that we can use to specify how it would be formatted.  For example we cannot specify the formatting for every single function name, but we can format a `defun` or a `let` in a special way.  With this mechanism, the formatter is easily extensible and configurable.  There is even the ability to specify formatting configuration for a list's children by index -- for example you may want to format the first element in a `let` (the binding list) differently than the rest of the code.
+
+There are also some forms that don't have a head-item, that we still identify and set some default formatting rules:
+- The top-level node - indentation is disabled
+- Lists of only literals, called `constant lists` - These are indented with `1` space instead of the default `2` for a `flow`'d form.
+  - If these lists literals are paired with keywords we call these `constant pairs` - We strive to put the pairs on the same line, if they are too long we spill to the next with `2` space indentation
+- TODO - cond/case pairs (similar to constant pairs)
+
+This brings up two important terms that you will see in the formatter implementation a lot -- `flow` and `hang`.  The difference is as such:
+
+```clj
+(println "this"
+         "form"
+         "is"
+         "hung / hanged")
+```
+
+```clj
+(println
+  "this"
+  "form"
+  "is"
+  "flowed")
+```
+
+The difference between the two other than visually is that hanging always results in 1 less line spent _because we don't care about line-length_.  This is great because it means we can always default to hanging unless overridden.  Flowing trades off that 1 vertical line for horizontal space and looks great for block-style forms.  Like the top-level of a `defun` or the content of a `let`.
+
+### Applying Formatting
+
+The final real-step in the process, actually using the configuration and generating the formatting code.  This phase is actually broken up into 3 sub-phases as we recurse through the `FormatterTree`
+
+#### Build Lines
+
+First we build the source code lines without indentation, it's a lot easier to figure out the indentation after all tokens are on the correct lines instead of doing it in tandem.  Most tokens are given their own line, but in the case of a hung form we combine the first two elements on the first line.  Other forms are formatted in even more complicated ways, or comments that were originally inline should remain next to that token.
+
+Secondly, once we have all our consolidated lines we can apply the indentation.  This is most often based on the configuration of the list node itself.  For example a `hung` form must be indented the width of a `(` and the length of the head-form, plus the whitespace separator -- conversely a `flowed` form only needs to be indented by `2` spaces, regardless of token length.  Because all of these algorithms are recursive we can hyper-focus on getting the indentation and formatting correct from the bottom-up.  In otherwords, we don't care if a form is 200 spaces indented into a function, once the lines are returned recursively, they will get another layer of indentation added on until we hit the top-level node.
+
+### Finalize Lines
+
+Finally, we concatenate our list of strings, respecting the original file's line-endings.
+
+As it is best practice, if we are formatting an entire file we should terminate with a trailing new-line.
+
+Both of these items are technically a TODO.
+
+## Configuration
+
+TODO
+
+## Reference
+
+This formatter is highly influenced by a lot of other existing formatters and writings.  The most significant of which is Clojure's [zprint](https://github.com/kkinnear/zprint/tree/main/doc/options) which has incredible documentation around it's capabilities and why it chooses what to do.
+
+The formatter I've built basically copies zprint in terms of basic functionality, but simplifies things by being even more opinionated (for example, doing far less in terms of trying to adhere to a line-length limit).
+
+### Other references
+
+- [The Hardest Program I’ve Ever Written - Dart Formatter](https://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/)
+- [cljfmt](https://github.com/weavejester/cljfmt)
+
+### Why not respect a line-length limit
+
+The OpenGOAL formatter will respect line-length in only very specific situations:
+- When determining if a form can be fit entirely on a single line
+- Probably in the future for multi-line comments
+
+Besides it adding a massive amount of complexity, even zprint explains that a line-length limit won't always be followed
+
+> zprint will work very hard to fit the formatted output into this width, though there are limits to its effort.  For instance, it will not reduce the minimum indents in order to satisfy a particular width requirement.
+
+Let's illustrate with an example:
+
+```clj
+(defun some-func
+  (+ 123
+     456
+     (+ 123
+        456
+        ;; imagine this form exceeds our width, how can you "fix" this without
+        ;; messing up the indentation levels of the entire expression
+        ;; - you can't inline it (you're already past the width)
+        ;; - you can't insert a line break because it'll still be at the same horizontal level
+        (+ 123
+           456
+           789))))
+```
+
+In LISP code, we don't have many ideal options to deal with this, all options are syntactically correct, but in terms of conventional readability they are bad because when reading LISP code you want to use the indentation as a guide (which is why new-line parens are considered ugly by basically all LISP style guides)
+
+The other reason which is equally relevant is it is hard, it means the formatter would have to try to find an optimal solution by exploring different solutions until it finds the one that minimizes lines.
+
+But with an opinionated formatting style you can avoid this complexity, if you want your code to look nice don't write code like the decompiler sometimes outputs when it encounters something truly cursed:
+
+![](./docs/triangle-of-death.png)
+> `game-save.gc` about 200+ nested `let`s!

common/formatter/formatter.cpp

@@ -2,13 +2,16 @@
 
 #include "formatter_tree.h"
 
+#include "common/formatter/rules/formatting_rules.h"
+#include "common/formatter/rules/rule_config.h"
 #include "common/log/log.h"
 #include "common/util/FileUtil.h"
+#include "common/util/ast_util.h"
 #include "common/util/string_util.h"
 
 #include "tree_sitter/api.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 // Declare the `tree_sitter_opengoal` function, which is
 // implemented by the `tree-sitter-opengoal` library.
@@ -16,131 +19,454 @@ extern "C" {
 extern const TSLanguage* tree_sitter_opengoal();
 }
 
-std::string apply_formatting(
-    const FormatterTreeNode& curr_node,
-    std::string output,
-    std::optional<formatter_rules::config::FormFormattingConfig> form_element_config) {
-  using namespace formatter_rules;
-  if (!curr_node.token && curr_node.refs.empty()) {
-    return output;
+int hang_indentation_width(const FormatterTreeNode& curr_node) {
+  if (curr_node.token || curr_node.refs.empty()) {
+    return 0;
   }
-  std::string curr_form = "";
-  // Print the token
-  if (curr_node.token) {
-    curr_form += curr_node.token.value();
-    return curr_form;
+  // Get the first element of the form
+  const auto& first_elt = curr_node.refs.at(0);
+  if (first_elt.token) {
+    return first_elt.token->length() +
+           2;  // +2 because the opening paren and then the following space
   }
-  if (!curr_node.metadata.is_top_level) {
-    curr_form += "(";
-  }
-  // Iterate the form
+  // Otherwise, continue nesting
+  return 1 + hang_indentation_width(first_elt);
+}
 
-  bool inline_form = false;
-  // Also check if the form should be constant-paired
-  const bool constant_pair_form = constant_pairs::form_should_be_constant_paired(curr_node);
-  if (!constant_pair_form) {
-    // Determine if the form should be inlined or hung/flowed
-    // TODO - this isn't entirely accurate, needs current cursor positioning (which is tricky
-    // because recursion!)
-    inline_form = indent::form_can_be_inlined(curr_form, curr_node);
+int get_total_form_inlined_width(const FormatterTreeNode& curr_node) {
+  if (curr_node.token) {
+    return curr_node.token->length();
   }
-  const bool flowing = indent::should_form_flow(curr_node, inline_form);
-  std::optional<formatter_rules::config::FormFormattingConfig> form_config;
-  if (!curr_node.refs.empty() && curr_node.refs.at(0).token) {
+  int width = 1;
+  for (int i = 0; i < curr_node.refs.size(); i++) {
+    width += get_total_form_inlined_width(curr_node.refs.at(i));
+    if (i != curr_node.refs.size() - 1) {
+      width += 1;  // add the space between elements
+    }
+  }
+  return width + 1;
+}
+
+// TODO - compute length of each node and store it AOT
+void apply_formatting_config(
+    FormatterTreeNode& curr_node,
+    std::optional<std::shared_ptr<formatter_rules::config::FormFormattingConfig>>
+        config_from_parent = {},
+    std::optional<std::shared_ptr<formatter_rules::config::FormFormattingConfig>>
+        config_merge_from_parent = {}) {
+  using namespace formatter_rules;
+  // node is empty, base-case
+  if (curr_node.token || curr_node.refs.empty()) {
+    return;
+  }
+  // first, check to see if this form already has a predefined formatting configuration
+  // if it does, that simplifies things because there is only 1 way of formatting the form
+  std::optional<formatter_rules::config::FormFormattingConfig> predefined_config;
+  if (!config_from_parent && !curr_node.refs.empty() && curr_node.refs.at(0).token) {
     const auto& form_head = curr_node.refs.at(0).token;
     if (form_head && config::opengoal_form_config.find(form_head.value()) !=
                          config::opengoal_form_config.end()) {
-      form_config = config::opengoal_form_config.at(form_head.value());
-    }
-  }
-  // TODO - might want to make some kind of per-form config struct, simplify the passing around of
-  // info below
-  for (int i = 0; i < curr_node.refs.size(); i++) {
-    const auto& ref = curr_node.refs.at(i);
-    // Figure out if the element should be inlined or not
-    bool inline_element = inline_form;
-    if (indent::inline_form_element(curr_node, i)) {
-      inline_element = indent::inline_form_element(curr_node, i).value();
-    }
-    // Append a newline if needed
-    // TODO - cleanup / move
-    bool is_binding_list = false;
-    bool force_newline = false;
-    bool override_force_flow = false;
-    if (form_config) {
-      force_newline = std::find(form_config->force_newline_at_indices.begin(),
-                                form_config->force_newline_at_indices.end(),
-                                i) != form_config->force_newline_at_indices.end();
-      // Check if it's a small enough binding list, if so we don't force a newline if the element
-      // can be inlined
-      if (inline_element && i > 0 && form_config->bindings_at_index == i - 1 &&
-          curr_node.refs.at(i - 1).refs.size() < form_config->allow_inlining_if_size_less_than) {
-        force_newline = false;
-        override_force_flow = true;
-      }
-      is_binding_list = form_config->bindings_at_index == i;
+      predefined_config = config::opengoal_form_config.at(form_head.value());
+      curr_node.formatting_config = predefined_config.value();
     }
 
-    if (!curr_node.metadata.is_top_level &&
-        (!inline_element || is_binding_list || force_newline ||
-         (form_element_config && form_element_config->force_flow))) {
-      indent::append_newline(curr_form, ref, curr_node, i, flowing, constant_pair_form,
-                             (form_element_config && form_element_config->force_flow));
+    if (config_merge_from_parent) {
+      const auto& merge_config = *config_merge_from_parent.value();
+      curr_node.formatting_config.parent_mutable_extra_indent +=
+          merge_config.parent_mutable_extra_indent;
+      curr_node.formatting_config.prevent_inlining = merge_config.prevent_inlining;
     }
-    // TODO - indent the line (or don't)
-    // Either print the element's token, or recursively format it as well
-    if (ref.token) {
-      // TODO depth hard-coded to 1, i think this can be removed, since
-      // forms are always done bottom-top recursively, they always act
-      // independently as if it was the shallowest depth
-      if (!inline_element || force_newline) {
-        indent::indent_line(curr_form, ref, curr_node, 1, i, flowing);
+  } else if (config_from_parent) {
+    // TODO - doesn't merge just replaces, a bit inflexible
+    predefined_config = *config_from_parent.value();
+    curr_node.formatting_config = predefined_config.value();
+  }
+  // In order to keep things simple, as well as because its ineffectual in lisp code (you can only
+  // enforce it so much without making things unreadable), line width will not matter for deciding
+  // whether or not to hang or flow the form
+  //
+  // This means that a hang would ALWAYS win, because it's 1 less line break.  Therefore this
+  // simplifies our approach there is no need to explore both braches to see which one would be
+  // preferred.
+  //
+  // Instead, we either use the predefined configuration (obviously) or we do some checks for some
+  // outlier conditions to see if things should be formatted differently
+  //
+  // Otherwise, we always default to a hang.
+  //
+  // NOTE - any modifications here to child elements could be superseeded later in the recursion!
+  // In order to maintain your sanity, only modify things here that _arent_ touched by default
+  // configurations.  These are explicitly prepended with `parent_mutable_`
+  if (!predefined_config && !curr_node.formatting_config.config_set) {
+    if (curr_node.metadata.is_top_level) {
+      curr_node.formatting_config.indentation_width = 0;
+      curr_node.formatting_config.hang_forms = false;
+    } else if (constant_list::is_constant_list(curr_node)) {
+      // - Check if the form is a constant list (ie. a list of numbers)
+      curr_node.formatting_config.indentation_width = 1;
+      curr_node.formatting_config.hang_forms = false;
+      curr_node.formatting_config.has_constant_pairs =
+          constant_pairs::form_should_be_constant_paired(curr_node);
+      // If applicable, iterate through the constant pairs, since we can potentially pair up
+      // non-constant second elements in a pair (like a function call), there is the potential that
+      // they need to spill to the next line and get indented in extra.  This is an exceptional
+      // circumstance, we do NOT do this sort of thing when formatting normal forms (cond/case pairs
+      // are another similar situation)
+      if (curr_node.formatting_config.has_constant_pairs) {
+        for (int i = 0; i < (int)curr_node.refs.size(); i++) {
+          auto& child_ref = curr_node.refs.at(i);
+          const auto& type = child_ref.metadata.node_type;
+          if (constant_types.find(type) == constant_types.end() &&
+              constant_pairs::is_element_second_in_constant_pair(curr_node, child_ref, i)) {
+            child_ref.formatting_config.parent_mutable_extra_indent = 2;
+          }
+        }
       }
-      if (ref.metadata.node_type == "comment" && ref.metadata.is_inline) {
-        curr_form += " " + ref.token.value();
-      } else if (ref.metadata.node_type == "block_comment") {
-        curr_form += comments::format_block_comment(ref.token.value());
+
+    } else if (curr_node.formatting_config.hang_forms && curr_node.refs.size() > 1 &&
+               curr_node.refs.at(1).metadata.is_comment) {
+      // - Check if the second argument is a comment, it looks better if we flow instead
+      curr_node.formatting_config.hang_forms = false;
+    }
+  }
+  // If we are hanging, lets determine the indentation width since it is based on the form itself
+  if (curr_node.formatting_config.hang_forms) {
+    // TODO - this isn't being calculated for a pre-defined config
+    // TODO - another idea is to do this during consolidation
+    curr_node.formatting_config.indentation_width = hang_indentation_width(curr_node);
+  }
+  // iterate through the refs
+  for (int i = 0; i < (int)curr_node.refs.size(); i++) {
+    auto& ref = curr_node.refs.at(i);
+    if (!ref.token) {
+      // If the child has a pre-defined configuration at that index, we pass it along
+      if (predefined_config && predefined_config->index_config_override.find(i) !=
+                                   predefined_config->index_config_override.end()) {
+        apply_formatting_config(ref, {}, predefined_config->index_config_override.at(i));
+      } else if (predefined_config && predefined_config->index_configs.find(i) !=
+                                          predefined_config->index_configs.end()) {
+        apply_formatting_config(ref, predefined_config->index_configs.at(i), {});
+      } else if (predefined_config && predefined_config->default_index_config) {
+        apply_formatting_config(ref, predefined_config->default_index_config, {});
       } else {
-        curr_form += ref.token.value();
+        apply_formatting_config(ref, {}, {});
       }
-      if (!curr_node.metadata.is_top_level) {
-        curr_form += " ";
+    }
+  }
+  // Precompute the column widths for things like deftype fields
+  if (curr_node.formatting_config.determine_column_widths_for_list_elements) {
+    // iterate through each ref and find the max length of each index (may be a token, may not be!)
+    // then store that info in each list element's `list_element_column_widths` to be used when
+    // printing out the tokens (pad width)
+    // Find the maximum number of columns
+    int max_columns = 0;
+    for (const auto& field : curr_node.refs) {
+      if ((int)field.refs.size() > max_columns) {
+        max_columns = field.refs.size();
+      }
+    }
+    // if only one field has a value in the max col position, it looks weird for it to be indented
+    bool ignore_final_column_width = true;
+    int fields_with_atleast_max_col = 0;
+    for (const auto& field : curr_node.refs) {
+      if ((int)field.refs.size() == max_columns) {
+        fields_with_atleast_max_col++;
+        if (fields_with_atleast_max_col > 1) {
+          ignore_final_column_width = false;
+          break;
+        }
+      }
+    }
+    // Now find the column max widths
+    std::vector<int> column_max_widths = {};
+    for (int col = 0; col < max_columns; col++) {
+      column_max_widths.push_back(0);
+      // -2 because its the indentation before the final column that we want to skip
+      if (col >= curr_node.formatting_config.num_columns_to_compute_widths ||
+          (ignore_final_column_width && col == max_columns - 2)) {
+        continue;
+      }
+      for (const auto& field : curr_node.refs) {
+        if ((int)field.refs.size() > col) {
+          const auto width = get_total_form_inlined_width(field.refs.at(col));
+          if (width > column_max_widths.at(col)) {
+            column_max_widths[col] = width;
+          }
+        }
+      }
+    }
+    // Apply column info to every list
+    for (auto& field : curr_node.refs) {
+      field.formatting_config.list_element_column_widths = column_max_widths;
+      field.formatting_config.config_set = true;
+    }
+  }
+}
+
+bool form_contains_comment(const FormatterTreeNode& curr_node) {
+  if (curr_node.metadata.is_comment) {
+    return true;
+  }
+  for (const auto& ref : curr_node.refs) {
+    const auto contains_comment = form_contains_comment(ref);
+    if (contains_comment) {
+      return true;
+    }
+  }
+  return false;
+}
+
+bool form_contains_node_that_prevents_inlining(const FormatterTreeNode& curr_node) {
+  if (curr_node.formatting_config.should_prevent_inlining(curr_node.formatting_config,
+                                                          curr_node.refs.size())) {
+    return true;
+  }
+  for (const auto& ref : curr_node.refs) {
+    const auto prevents_inlining = form_contains_node_that_prevents_inlining(ref);
+    if (prevents_inlining) {
+      return true;
+    }
+  }
+  return false;
+}
+
+bool can_node_be_inlined(const FormatterTreeNode& curr_node, int cursor_pos) {
+  using namespace formatter_rules;
+  if (curr_node.formatting_config.force_inline) {
+    // Ensure there are no comments, this still trumps this
+    if (form_contains_comment(curr_node)) {
+      return false;
+    }
+    return true;
+  }
+  // First off, we cannot inline the top level
+  if (curr_node.metadata.is_top_level) {
+    return false;
+  }
+  // If the config explicitly prevents inlining, or it contains a sub-node that prevents inlining
+  if (curr_node.formatting_config.prevent_inlining ||
+      form_contains_node_that_prevents_inlining(curr_node)) {
+    return false;
+  }
+  // nor can we inline something that contains a comment in the middle
+  if (form_contains_comment(curr_node)) {
+    return false;
+  }
+  // constant pairs are not inlined!
+  if (curr_node.formatting_config.has_constant_pairs) {
+    return false;
+  }
+  // If this is set in the config, then the form is intended to be partially inlined
+  if (curr_node.formatting_config.inline_until_index({})) {
+    return false;
+  }
+  // let's see if we can inline the form all on one line to do that, we recursively explore
+  // the form to find the total width
+  int line_width = cursor_pos + get_total_form_inlined_width(curr_node);
+  return line_width <= indent::line_width_target;  // TODO - comments
+}
+
+std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
+                                          std::vector<std::string> /*output*/ = {},
+                                          int cursor_pos = 0) {
+  using namespace formatter_rules;
+  if (!curr_node.token && curr_node.refs.empty()) {
+    // special case to handle an empty list
+    if (curr_node.node_prefix) {
+      return {fmt::format("{}()", curr_node.node_prefix.value())};
+    }
+    return {"()"};
+  }
+
+  // If its a token, just print the token and move on
+  if (curr_node.token) {
+    return {curr_node.token_str()};
+  }
+
+  bool inline_form = can_node_be_inlined(curr_node, cursor_pos);
+  // TODO - also if the form is inlinable, we can skip all the complication below and just...inline
+  // it!
+  // TODO - should figure out the inlining here as well, instead of the bool above
+
+  // Iterate the form, building up a list of the final lines but don't worry about indentation
+  // at this stage.  Once the lines are finalized, it's easy to add the indentation later
+  //
+  // This means we may combine elements onto the same line in this step.
+  std::vector<std::string> form_lines = {};
+
+  for (int i = 0; i < (int)curr_node.refs.size(); i++) {
+    const auto& ref = curr_node.refs.at(i);
+    // Add new line entry
+    if (ref.token) {
+      // Cleanup block-comments
+      if (ref.metadata.node_type == "block_comment") {
+        const auto comment_lines = comments::format_block_comment(ref.token_str());
+        for (const auto& line : comment_lines) {
+          form_lines.push_back(line);
+        }
+      } else {
+        form_lines.push_back(ref.token_str());
+      }
+
+      if (!curr_node.metadata.is_top_level && i == (int)curr_node.refs.size() - 1 &&
+          (ref.metadata.is_comment)) {
+        // if there's an inline comment at the end of a form, we have to force the paren to the next
+        // line and do a new-line paren this is ugly, but we have no choice!
+        form_lines.push_back("");
       }
     } else {
-      // See if the item at this position has specific formatting
-      std::optional<formatter_rules::config::FormFormattingConfig> config = {};
-      std::string formatted_form;
-      if (form_config && form_config->index_configs.find(i) != form_config->index_configs.end()) {
-        formatted_form = apply_formatting(ref, "", *form_config->index_configs.at(i));
-      } else {
-        formatted_form = apply_formatting(ref, "", {});
-      }
-      // TODO - align inner lines only
-      if (!curr_node.metadata.is_top_level) {
-        indent::align_lines(
-            formatted_form, ref, curr_node, constant_pair_form, flowing,
-            (!override_force_flow && form_config && i >= form_config->start_flow_at_index),
-            inline_element);
-      }
-      curr_form += formatted_form;
-      if (!curr_node.metadata.is_top_level) {
-        curr_form += " ";
+      // If it's not a token, we have to recursively build up the form
+      // TODO - add the cursor_pos here
+      const auto& lines = apply_formatting(ref, {}, cursor_pos);
+      for (int i = 0; i < (int)lines.size(); i++) {
+        const auto& line = lines.at(i);
+        form_lines.push_back(fmt::format(
+            "{}{}", str_util::repeat(ref.formatting_config.parent_mutable_extra_indent, " "),
+            line));
       }
     }
-    // Handle blank lines at the top level, skip if it's the final element
-    blank_lines::separate_by_newline(curr_form, curr_node, ref, i);
+    // If we are hanging forms, combine the first two forms onto the same line
+    if (i == (int)curr_node.refs.size() - 1 && form_lines.size() > 1 &&
+        (curr_node.formatting_config.hang_forms ||
+         curr_node.formatting_config.combine_first_two_lines)) {
+      form_lines.at(0) += fmt::format(" {}", str_util::ltrim(form_lines.at(1)));
+      form_lines.erase(form_lines.begin() + 1);
+    } else if ((i + 1) < (int)curr_node.refs.size()) {
+      const auto& next_ref = curr_node.refs.at(i + 1);
+      // combine the next inline comment or constant pair
+      if ((next_ref.metadata.node_type == "comment" && next_ref.metadata.is_inline) ||
+          (curr_node.formatting_config.has_constant_pairs &&
+           constant_pairs::is_element_second_in_constant_pair(curr_node, next_ref, i + 1)) ||
+          constant_pairs::is_element_second_in_constant_pair_new(curr_node.refs.at(i), next_ref)) {
+        // TODO
+        // has issues with not consolidating first lines, this should probably just be moved to
+        // outside this loop for simplicity, do it later
+        if (next_ref.token) {
+          form_lines.at(form_lines.size() - 1) += fmt::format(" {}", next_ref.token_str());
+          i++;
+          // We have to handle hang-consolidation here or else it will never be reached above!
+          if (i == (int)curr_node.refs.size() - 1 && form_lines.size() > 1 &&
+              (curr_node.formatting_config.hang_forms ||
+               curr_node.formatting_config.combine_first_two_lines)) {
+            form_lines.at(0) += fmt::format(" {}", form_lines.at(1));
+            form_lines.erase(form_lines.begin() + 1);
+          }
+        } else if (can_node_be_inlined(next_ref, cursor_pos)) {
+          const auto& lines = apply_formatting(next_ref, {}, cursor_pos);  // TODO - cursor pos
+          for (const auto& line : lines) {
+            form_lines.at(form_lines.size() - 1) += fmt::format(" {}", line);
+          }
+          i++;
+          // We have to handle hang-consolidation here or else it will never be reached above!
+          if (i == (int)curr_node.refs.size() - 1 && form_lines.size() > 1 &&
+              (curr_node.formatting_config.hang_forms ||
+               curr_node.formatting_config.combine_first_two_lines)) {
+            form_lines.at(0) += fmt::format(" {}", form_lines.at(1));
+            form_lines.erase(form_lines.begin() + 1);
+          }
+        }
+        if (!curr_node.metadata.is_top_level && next_ref.metadata.node_type == "comment" &&
+            (i + 1) == (int)curr_node.refs.size()) {
+          form_lines.push_back("");
+        }
+      }
+    }
+    // If we are at the top level, potential separate with a new line
+    if (blank_lines::should_insert_blank_line(curr_node, ref, i)) {
+      form_lines.at(form_lines.size() - 1) += "\n";
+    }
   }
+
+  // Consolidate any lines if the configuration requires it
+  // TODO there is a hack here so that multi-line forms that are consolidated still line up properly
+  // i have to make consolidate a more first-class feature of the config
+  // TODO - hacky, but prevents a bad situation, clean up
+  if (curr_node.formatting_config.inline_until_index(form_lines) &&
+      !str_util::contains(form_lines.at(0), ";")) {
+    std::vector<std::string> new_form_lines = {};
+    const auto original_form_head_width = str_util::split(form_lines.at(0), '\n').at(0).length();
+    bool consolidating_lines = true;
+    for (int i = 0; i < (int)form_lines.size(); i++) {
+      if (i < curr_node.formatting_config.inline_until_index(form_lines)) {
+        if (new_form_lines.empty()) {
+          new_form_lines.push_back(form_lines.at(i));
+        } else {
+          new_form_lines.at(0) += fmt::format(" {}", form_lines.at(i));
+        }
+      } else {
+        if (str_util::starts_with(form_lines.at(i), " ") && consolidating_lines) {
+          new_form_lines.push_back(fmt::format(
+              "{}{}", str_util::repeat(original_form_head_width, " "), form_lines.at(i)));
+        } else {
+          consolidating_lines = false;
+          new_form_lines.push_back(form_lines.at(i));
+        }
+      }
+    }
+    form_lines = new_form_lines;
+  }
+
+  // Add any column padding
+  if (!curr_node.formatting_config.list_element_column_widths.empty()) {
+    for (int i = 0; i < (int)form_lines.size(); i++) {
+      const auto& token = form_lines.at(i);
+      if (i < (int)form_lines.size() - 1) {
+        form_lines[i] = str_util::pad_right(
+            token, curr_node.formatting_config.list_element_column_widths.at(i), ' ');
+      }
+    }
+  }
+
+  // Apply necessary indentation to each line and add parens
   if (!curr_node.metadata.is_top_level) {
-    curr_form = str_util::rtrim(curr_form) + ")";
+    std::string form_surround_start = "(";
+    if (curr_node.node_prefix) {
+      form_surround_start = fmt::format("{}(", curr_node.node_prefix.value());
+    }
+    std::string form_surround_end = ")";
+    form_lines[0] = fmt::format("{}{}", form_surround_start, form_lines[0]);
+    form_lines[form_lines.size() - 1] =
+        fmt::format("{}{}", form_lines[form_lines.size() - 1], form_surround_end);
   }
-  return curr_form;
+  if (inline_form) {
+    // NOTE - not sure about this, if we are inlining a form, it always makes sense to eliminate
+    // trailing whitespace the only issue i can foresee is related to strings that span multiple
+    // lines.
+    std::vector<std::string> new_form_lines = {};
+    for (const auto& form_line : form_lines) {
+      new_form_lines.push_back(str_util::ltrim(form_line));
+    }
+    form_lines = {fmt::format("{}", fmt::join(new_form_lines, " "))};
+  } else {
+    bool currently_in_block_comment = false;
+    for (int i = 0; i < (int)form_lines.size(); i++) {
+      auto& line = form_lines.at(i);
+      if (str_util::contains(line, "|#")) {
+        currently_in_block_comment = false;
+      }
+      if (i > 0 && !currently_in_block_comment) {
+        line = fmt::format("{}{}",
+                           str_util::repeat(curr_node.formatting_config.indentation_width_for_index(
+                                                curr_node.formatting_config, i),
+                                            " "),
+                           line);
+      }
+      if (str_util::contains(line, "#|") && !str_util::contains(line, "|#")) {
+        currently_in_block_comment = true;
+      }
+    }
+  }
+  return form_lines;
+}
+
+std::string join_formatted_lines(const std::vector<std::string>& lines,
+                                 const std::string& line_ending) {
+  return fmt::format("{}", fmt::join(lines, line_ending));
 }
 
 std::optional<std::string> formatter::format_code(const std::string& source) {
   // Create a parser.
   std::shared_ptr<TSParser> parser(ts_parser_new(), TreeSitterParserDeleter());
-
-  // Set the parser's language (JSON in this case).
   ts_parser_set_language(parser.get(), tree_sitter_opengoal());
 
   // Build a syntax tree based on source code stored in a string.
@@ -150,14 +476,32 @@ std::optional<std::string> formatter::format_code(const std::string& source) {
 
   // Get the root node of the syntax tree.
   TSNode root_node = ts_tree_root_node(tree.get());
-  if (ts_node_is_null(root_node) || ts_node_has_error(root_node)) {
+  if (ts_node_is_null(root_node)) {
+    lg::error("null root node");
+    return std::nullopt;
+  } else if (ts_node_has_error(root_node)) {
+    lg::error("grammar parsing error, go figure it out!");
     return std::nullopt;
   }
 
   try {
-    const auto formatting_tree = FormatterTree(source, root_node);
-    std::string formatted_code = apply_formatting(formatting_tree.root, "", {});
-    return formatted_code;
+    // There are three phases of formatting
+    // 1. Simplify the AST down to something that is easier to work on from a formatting perspective
+    // this also gathers basic metadata that can be done at this stage, like if the token is a
+    // comment or if the form is on the top-level
+    auto formatting_tree = FormatterTree(source, root_node);
+    // 2. Recursively iterate through this simplified FormatterTree and figure out what rules
+    // need to be applied to produce an optimal result
+    apply_formatting_config(formatting_tree.root);
+    // 3. Use this updated FormatterTree to print out the final source-code, while doing so
+    // we may deviate from the optimal result to produce something even more optimal by inlining
+    // forms that can fit within the line width.
+    const auto formatted_lines = apply_formatting(formatting_tree.root);
+    // 4. Now we joint he lines together, it's easier when formatting to leave all lines independent
+    // so adding indentation is easier
+    const auto formatted_source =
+        join_formatted_lines(formatted_lines, file_util::get_majority_file_line_endings(source));
+    return formatted_source;
   } catch (std::exception& e) {
     lg::error("Unable to format code - {}", e.what());
   }

common/formatter/formatter.h

@@ -3,13 +3,8 @@
 #include <optional>
 #include <string>
 
-#include "common/formatter/rules/formatting_rules.h"
-#include "common/formatter/rules/rule_config.h"
-
 #include "tree_sitter/api.h"
 
-// TODO:
-// - Considering _eventually_ adding line-length heuristics
 namespace formatter {
 
 struct TreeSitterParserDeleter {

common/formatter/formatter_tree.cpp

@@ -2,7 +2,7 @@
 
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 std::string get_source_code(const std::string& source, const TSNode& node) {
   uint32_t start = ts_node_start_byte(node);
@@ -78,16 +78,24 @@ FormatterTree::FormatterTree(const std::string& source, const TSNode& root_node)
   construct_formatter_tree_recursive(source, root_node, root);
 }
 
+const std::unordered_map<std::string, std::vector<std::string>> node_type_ignorable_contents = {
+    {"list_lit", {"(", ")"}}};
+
 // TODO make an imperative version eventually
+// TODO - cleanup duplication
 void FormatterTree::construct_formatter_tree_recursive(const std::string& source,
                                                        TSNode curr_node,
-                                                       FormatterTreeNode& tree_node) {
+                                                       FormatterTreeNode& tree_node,
+                                                       std::optional<std::string> node_prefix) {
   if (ts_node_child_count(curr_node) == 0) {
-    tree_node.refs.push_back(FormatterTreeNode(source, curr_node));
+    auto new_node = FormatterTreeNode(source, curr_node);
+    new_node.node_prefix = node_prefix;
+    tree_node.refs.push_back(new_node);
     return;
   }
   const std::string curr_node_type = ts_node_type(curr_node);
   FormatterTreeNode list_node;
+  std::optional<std::string> next_node_prefix;
   if (curr_node_type == "list_lit") {
     list_node = FormatterTreeNode();
   } else if (curr_node_type == "str_lit") {
@@ -96,21 +104,74 @@ void FormatterTree::construct_formatter_tree_recursive(const std::string& source
     // formatting So for strings, we treat them as if they should be a single token
     tree_node.refs.push_back(FormatterTreeNode(source, curr_node));
     return;
+  } else if (curr_node_type == "quoting_lit") {
+    if (node_prefix) {
+      node_prefix.value() += "'";
+    } else {
+      node_prefix = "'";
+    }
+    construct_formatter_tree_recursive(source, ts_node_child(curr_node, 1), tree_node, node_prefix);
+    return;
+  } else if (curr_node_type == "unquoting_lit") {
+    if (node_prefix) {
+      node_prefix.value() += ",";
+    } else {
+      node_prefix = ",";
+    }
+    construct_formatter_tree_recursive(source, ts_node_child(curr_node, 1), tree_node, node_prefix);
+    return;
+  } else if (curr_node_type == "quasi_quoting_lit") {
+    if (node_prefix) {
+      node_prefix.value() += "`";
+    } else {
+      node_prefix = "`";
+    }
+    construct_formatter_tree_recursive(source, ts_node_child(curr_node, 1), tree_node, node_prefix);
+    return;
+  } else if (curr_node_type == "unquote_splicing_lit") {
+    if (node_prefix) {
+      node_prefix.value() += ",@";
+    } else {
+      node_prefix = ",@";
+    }
+    construct_formatter_tree_recursive(source, ts_node_child(curr_node, 1), tree_node, node_prefix);
+    return;
+  }
+  std::vector<std::string> skippable_nodes = {};
+  if (node_type_ignorable_contents.find(curr_node_type) != node_type_ignorable_contents.end()) {
+    skippable_nodes = node_type_ignorable_contents.at(curr_node_type);
   }
   for (size_t i = 0; i < ts_node_child_count(curr_node); i++) {
     const auto child_node = ts_node_child(curr_node, i);
-    // We skip parens
+    auto debug_child = ts_node_string(child_node);
     const auto contents = get_source_code(source, child_node);
-    if (contents == "(" || contents == ")") {
+    bool skip_node = false;
+    for (const auto& skippable_content : skippable_nodes) {
+      if (skippable_content == contents) {
+        skip_node = true;
+        break;
+      }
+    }
+    if (skip_node) {
       continue;
     }
     if (curr_node_type == "list_lit") {
-      construct_formatter_tree_recursive(source, child_node, list_node);
+      construct_formatter_tree_recursive(source, child_node, list_node, {});
+      if (node_prefix) {
+        list_node.node_prefix = node_prefix;
+      }
     } else {
-      construct_formatter_tree_recursive(source, child_node, tree_node);
+      construct_formatter_tree_recursive(source, child_node, tree_node, node_prefix);
+      if (node_prefix && !tree_node.refs.empty()) {
+        tree_node.refs.at(tree_node.refs.size() - 1).node_prefix = node_prefix;
+      }
     }
   }
   if (curr_node_type == "list_lit") {
+    // special case for empty lists
+    if (node_prefix && !list_node.node_prefix) {
+      list_node.node_prefix = node_prefix;
+    }
     tree_node.refs.push_back(list_node);
   }
 }

common/formatter/formatter_tree.h

@@ -5,6 +5,7 @@
 #include <string>
 #include <vector>
 
+#include "rules/rule_config.h"
 #include "tree_sitter/api.h"
 
 // Treesitter is fantastic for validating and parsing our code into a structured tree format without
@@ -17,8 +18,9 @@
 // we really care about is:
 // - getting all the text tokens for the source code
 // - having them in a proper, nested format
-// The treesitter format is complicated and highly nested, leading to some very hard to understand
-// code. So my solution is a 2-pass format.
+//
+// TLDR - The treesitter format is complicated and highly nested, leading to some very hard to
+// understand code. So my solution is atleast a 2-pass format.
 //
 // Pass 1 - convert the AST into a simplified FormatterTree
 // Pass 2 - use the simplified tree to output the final code
@@ -36,14 +38,26 @@ class FormatterTreeNode {
   std::vector<FormatterTreeNode> refs;
   Metadata metadata;
   // The token is optional because list nodes do not contain a token, they just contain a bunch of
-  // eventually token node refs
+  // eventually-containing token node refs
   std::optional<std::string> token;
+  std::optional<std::string> node_prefix;
+
+  formatter_rules::config::FormFormattingConfig formatting_config;
 
   FormatterTreeNode() = default;
   FormatterTreeNode(const std::string& source, const TSNode& node);
   FormatterTreeNode(const Metadata& _metadata) : metadata(_metadata){};
 
-  bool is_list() const { return token.has_value(); }
+  bool is_list() const { return !token.has_value(); }
+  std::string token_str() const {
+    if (node_prefix && token) {
+      return node_prefix.value() + token.value();
+    }
+    if (token) {
+      return token.value();
+    }
+    return "";
+  }
 };
 
 // A FormatterTree has a very simple and crude tree structure where:
@@ -59,5 +73,6 @@ class FormatterTree {
  private:
   void construct_formatter_tree_recursive(const std::string& source,
                                           TSNode curr_node,
-                                          FormatterTreeNode& tree_node);
+                                          FormatterTreeNode& tree_node,
+                                          std::optional<std::string> node_prefix = {});
 };

common/formatter/rules/formatting_rules.cpp

@@ -2,11 +2,9 @@
 
 #include <set>
 
-#include "rule_config.h"
-
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace formatter_rules {
 
@@ -14,31 +12,56 @@ namespace formatter_rules {
 // differentiate between a quoted symbol and a quoted form
 const std::set<std::string> constant_types = {"kwd_lit",  "num_lit",  "str_lit",
                                               "char_lit", "null_lit", "bool_lit"};
+const std::set<std::string> constant_type_forms = {"meters", "seconds", "degrees"};
+
+namespace constant_list {
+bool is_constant_list(const FormatterTreeNode& node) {
+  if (!node.is_list() || node.refs.empty()) {
+    return false;
+  }
+  if (!node.refs.at(0).token) {
+    return true;
+  }
+  const auto& type = node.refs.at(0).metadata.node_type;
+  return constant_types.find(type) != constant_types.end();
+}
+}  // namespace constant_list
+
 namespace blank_lines {
-void separate_by_newline(std::string& curr_text,
-                         const FormatterTreeNode& containing_node,
-                         const FormatterTreeNode& node,
-                         const int index) {
-  // We only are concerned with top level forms or elements
-  // Skip the last element, no trailing new-lines (let the editors handle this!)
-  // Also peek ahead to see if there was a comment on this line, if so don't separate things!
-  if (!containing_node.metadata.is_top_level || index >= containing_node.refs.size() - 1 ||
-      (containing_node.refs.at(index + 1).metadata.is_comment &&
-       containing_node.refs.at(index + 1).metadata.is_inline)) {
-    return;
+
+bool should_insert_blank_line(const FormatterTreeNode& containing_node,
+                              const FormatterTreeNode& node,
+                              const int index) {
+  // We only do this at the top level and don't leave a trailing new-line
+  if (!containing_node.metadata.is_top_level || index >= (int)containing_node.refs.size() - 1) {
+    return false;
   }
-  curr_text += "\n";
   // If it's a comment, but has no following blank lines, dont insert a blank line
   if (node.metadata.is_comment && node.metadata.num_blank_lines_following == 0) {
-    return;
+    return false;
   }
-  // Otherwise, add only 1 blank line
-  curr_text += "\n";
+  // If the next form is a comment and is inline, don't insert a new line
+  if ((index + 1) < (int)containing_node.refs.size() &&
+      containing_node.refs.at(index + 1).metadata.is_comment &&
+      containing_node.refs.at(index + 1).metadata.is_inline) {
+    return false;
+  }
+
+  if (node.formatting_config.elide_top_level_newline) {
+    if ((index + 1) < (int)containing_node.refs.size() &&
+        containing_node.refs.at(index + 1).metadata.is_comment) {
+      return true;
+    }
+    return false;
+  }
+
+  return true;
 }
+
 }  // namespace blank_lines
 
 namespace comments {
-std::string format_block_comment(const std::string& comment) {
+std::vector<std::string> format_block_comment(const std::string& comment) {
   // Normalize block comments, remove any trailing or leading whitespace
   // Only allow annotations on the first line, like #|@file
   // Don't mess with internal indentation as the user might intend it to be a certain way.
@@ -59,17 +82,28 @@ std::string format_block_comment(const std::string& comment) {
   // Remove trailing whitespace
   comment_contents = str_util::rtrim(comment_contents);
   // remove |#
-  // TODO - check suffix
-  comment_contents.pop_back();
-  comment_contents.pop_back();
+  if (str_util::ends_with(comment_contents, "|#")) {
+    comment_contents.pop_back();
+    comment_contents.pop_back();
+  }
   comment_contents = str_util::rtrim(comment_contents);
-  new_comment += fmt::format("\n{}\n|#", comment_contents);
-  return new_comment;
+  std::vector<std::string> lines = {new_comment};
+  const auto contents_as_lines = str_util::split_string(comment_contents, "\n");
+  if (contents_as_lines.size() > 1) {
+    for (const auto& line : contents_as_lines) {
+      lines.push_back(line);
+    }
+    lines.push_back("|#");
+  } else {
+    lines.at(0) = fmt::format("{} {} |#", new_comment, str_util::trim(contents_as_lines.at(0)));
+  }
+  return lines;
 }
 }  // namespace comments
 
 namespace constant_pairs {
 
+// TODO - remove index, not needed, could just pass in the previous node
 bool is_element_second_in_constant_pair(const FormatterTreeNode& containing_node,
                                         const FormatterTreeNode& node,
                                         const int index) {
@@ -85,9 +119,45 @@ bool is_element_second_in_constant_pair(const FormatterTreeNode& containing_node
     // not be paired
     return false;
   }
-  // Check the type of the element
-  if (constant_types.find(node.metadata.node_type) != constant_types.end()) {
-    return true;
+  return true;
+}
+
+// TODO - potentially remove the above
+bool is_element_second_in_constant_pair_new(const FormatterTreeNode& prev_node,
+                                            const FormatterTreeNode& curr_node) {
+  if (prev_node.metadata.node_type == "kwd_lit") {
+    // Handle standard constant types
+    // TODO - pair up sym_names as well
+    if (constant_types.find(curr_node.metadata.node_type) != constant_types.end()) {
+      if (curr_node.metadata.node_type != "kwd_lit") {
+        // NOTE - there is ambiugity here which cannot be totally solved (i think?)
+        // if the element itself is also a keyword, assume this is two adjacent keywords and they
+        // should not be paired
+        return true;
+      }
+    }
+    // Quoted symbols
+    if (curr_node.metadata.node_type == "sym_name" && curr_node.node_prefix &&
+        (curr_node.node_prefix.value() == "'" || curr_node.node_prefix.value() == ",")) {
+      return true;
+    }
+    if (!curr_node.refs.empty()) {
+      // Constant forms special cases (ie. meters)
+      if (constant_type_forms.find(curr_node.refs.at(0).token_str()) != constant_type_forms.end()) {
+        return true;
+      }
+      // If they are just a list of symbol names (enum or simple method call)
+      bool all_symbols = true;
+      for (const auto& ref : curr_node.refs) {
+        if (ref.metadata.node_type != "sym_name") {
+          all_symbols = false;
+          break;
+        }
+      }
+      if (all_symbols) {
+        return true;
+      }
+    }
   }
   return false;
 }
@@ -100,7 +170,7 @@ bool form_should_be_constant_paired(const FormatterTreeNode& node) {
     return false;
   }
   int num_pairs = 0;
-  for (int i = 0; i < node.refs.size() - 1; i++) {
+  for (int i = 0; i < (int)node.refs.size() - 1; i++) {
     const auto& ref = node.refs.at(i);
     const auto& next_ref = node.refs.at(i + 1);
     if (ref.token && next_ref.token) {
@@ -118,276 +188,4 @@ bool form_should_be_constant_paired(const FormatterTreeNode& node) {
 
 }  // namespace constant_pairs
 
-namespace indent {
-
-int cursor_pos(const std::string& curr_text) {
-  if (curr_text.empty()) {
-    return 0;
-  }
-  // Get the last line of the text (which is also the line we are on!)
-  int pos = 0;
-  for (int i = curr_text.size() - 1; i >= 0; i--) {
-    const auto& c = curr_text.at(i);
-    if (c == '\n') {
-      break;
-    }
-    pos++;
-  }
-  return pos;
-}
-
-int compute_form_width_after_index(const FormatterTreeNode& node,
-                                   const int index,
-                                   const int depth = 0) {
-  if (node.refs.empty()) {
-    if (node.token) {
-      return node.token->size();
-    } else {
-      return 0;
-    }
-  }
-  int form_width = 0;
-  for (int i = 0; i < node.refs.size(); i++) {
-    const auto& ref = node.refs.at(i);
-    if (depth == 0 && i < index) {
-      continue;
-    }
-    if (ref.token) {
-      form_width += ref.token->size() + 1;
-    } else {
-      form_width += compute_form_width_after_index(ref, index, depth + 1) + 1;
-    }
-  }
-  return form_width;
-}
-
-bool form_exceed_line_width(const std::string& curr_text,
-                            const FormatterTreeNode& containing_node,
-                            const int index) {
-  // Compute length from the current cursor position on the line as this check is done for every
-  // element of the form and not in advance
-  //
-  // This is for a good reason, intermediate nodes may override this styling and force to be
-  // formatted inline
-  //
-  // We early out as soon as we exceed the width
-  int curr_line_pos = cursor_pos(curr_text);
-  if (curr_line_pos >= line_width_target) {
-    return true;
-  }
-  int remaining_width_required = compute_form_width_after_index(containing_node, index);
-  if (curr_line_pos + remaining_width_required >= line_width_target) {
-    return true;
-  }
-  return false;
-}
-
-bool form_contains_comment(const FormatterTreeNode& node) {
-  if (node.metadata.is_comment) {
-    return true;
-  }
-  for (const auto& ref : node.refs) {
-    if (ref.metadata.is_comment) {
-      return true;
-    } else if (!node.refs.empty()) {
-      if (form_contains_comment(ref)) {
-        return true;
-      }
-    }
-  }
-  return false;
-}
-
-bool form_can_be_inlined(const std::string& curr_text, const FormatterTreeNode& list_node) {
-  // is the form too long to fit on a line TODO - increase accuracy here
-  if (form_exceed_line_width(curr_text, list_node, 0)) {
-    return false;
-  }
-  // are there any comments? (inlined or not, doesn't matter)
-  if (form_contains_comment(list_node)) {
-    return false;
-  }
-  return true;
-}
-
-bool should_form_flow(const FormatterTreeNode& list_node, const bool inlining_form) {
-  if (form_contains_comment(list_node)) {
-    return true;
-  }
-  // does the form begin with a constant (a list of content elements)
-  if (!inlining_form && !list_node.refs.empty() &&
-      constant_types.find(list_node.refs.at(0).metadata.node_type) != constant_types.end()) {
-    return true;
-  }
-
-  // TODO - make a function to make grabbing this metadata easier...
-  // TODO - honestly should just have an is_list metadata
-  if (!list_node.refs.empty() && !list_node.refs.at(0).token) {
-    // if the first element is a comment, force a flow
-    if (list_node.refs.size() > 1 && list_node.refs.at(1).metadata.is_comment) {
-      return true;
-    }
-    const auto& form_head = list_node.refs.at(0).token;
-    // See if we have any configuration for this form
-    if (form_head && config::opengoal_form_config.find(form_head.value()) !=
-                         config::opengoal_form_config.end()) {
-      const auto& form_config = config::opengoal_form_config.at(form_head.value());
-      return form_config.force_flow;
-    }
-  }
-
-  // TODO - cleanup, might be inside a let
-  /*if (!containing_form.refs.empty() && containing_form.refs.at(0).token) {
-    const auto& form_head = containing_form.refs.at(0).token;
-    if (form_head && config::opengoal_form_config.find(form_head.value()) !=
-                         config::opengoal_form_config.end()) {
-      const auto& form_config = config::opengoal_form_config.at(form_head.value());
-      if (form_config.force_flow) {
-        return true;
-      }
-    }
-  }*/
-
-  return false;
-}
-
-std::optional<bool> inline_form_element(const FormatterTreeNode& list_node, const int index) {
-  // TODO - honestly should just have an is_list metadata
-  if (list_node.refs.empty() || !list_node.refs.at(0).token) {
-    return std::nullopt;
-  }
-  const auto& form_head = list_node.refs.at(0).token;
-  // See if we have any configuration for this form
-  if (form_head &&
-      config::opengoal_form_config.find(form_head.value()) != config::opengoal_form_config.end()) {
-    const auto& form_config = config::opengoal_form_config.at(form_head.value());
-    if (form_config.inline_until_index != -1) {
-      return index < form_config.inline_until_index;
-    }
-  }
-  return std::nullopt;
-}
-
-void append_newline(std::string& curr_text,
-                    const FormatterTreeNode& node,
-                    const FormatterTreeNode& containing_node,
-                    const int index,
-                    const bool flowing,
-                    const bool constant_pair_form,
-                    const bool force_newline) {
-  if (force_newline && index >= 1 || (node.metadata.is_comment && !node.metadata.is_inline)) {
-    curr_text = str_util::rtrim(curr_text) + "\n";
-    return;
-  }
-  if (index <= 0 || containing_node.metadata.is_top_level ||
-      (node.metadata.is_comment && node.metadata.is_inline) || (!flowing && index <= 1)) {
-    return;
-  }
-  // Check if it's a constant pair
-  if (constant_pair_form &&
-      constant_pairs::is_element_second_in_constant_pair(containing_node, node, index)) {
-    return;
-  }
-  curr_text = str_util::rtrim(curr_text) + "\n";
-}
-
-void indent_line(std::string& curr_text,
-                 const FormatterTreeNode& node,
-                 const FormatterTreeNode& containing_node,
-                 const int depth,
-                 const int index,
-                 const bool flowing) {
-  if (node.metadata.is_top_level || (node.metadata.is_inline && node.metadata.is_comment)) {
-    return;
-  }
-  // If the element is the second element in a constant pair, that means we did not append a
-  // new-line before hand so we require no indentation (it's inline with the previous element)
-  if (constant_pairs::is_element_second_in_constant_pair(containing_node, node, index)) {
-    return;
-  }
-  // If the first element in the list is a constant, we only indent with 1 space instead
-  if (index > 0 &&
-      constant_types.find(containing_node.refs.at(0).metadata.node_type) != constant_types.end()) {
-    curr_text += str_util::repeat(depth, " ");
-  } else if (index > 0 && flowing) {
-    curr_text += str_util::repeat(depth, "  ");
-  } else if (index > 1 && !flowing) {
-    curr_text += str_util::repeat(containing_node.refs.at(0).token.value().length() + 2, " ");
-  }
-}
-
-// Recursively iterate through the node until we hit a token
-int length_to_hang(const FormatterTreeNode& node, int length) {
-  if (node.token || node.refs.at(0).token) {
-    return length;
-  }
-  return length_to_hang(node.refs.at(0), length + 1);
-}
-
-void align_lines(std::string& text,
-                 const FormatterTreeNode& node,
-                 const FormatterTreeNode& containing_node,
-                 const bool constant_pair_form,
-                 const bool flowing,
-                 const bool force_flow,
-                 const bool inline_element) {
-  const auto lines = str_util::split(text);
-  int start_index = 0;
-  if (inline_element) {
-    start_index = 1;
-  }
-  int alignment_width = 2;
-  if (force_flow) {
-    start_index = 0;
-  } else if (constant_pair_form &&
-             constant_types.find(containing_node.refs.at(0).metadata.node_type) !=
-                 constant_types.end()) {
-    start_index = 0;
-    alignment_width = 3;
-  } else if (!flowing) {
-    // If the form has a token (it's a normal list)
-    if (containing_node.refs.at(0).token) {
-      alignment_width = length_to_hang(containing_node.refs.at(1),
-                                       containing_node.refs.at(0).token.value().length()) +
-                        1;
-      if (!node.token) {
-        alignment_width++;
-      }
-    } else {
-      // otherwise, it's a list of lists
-      alignment_width = 1;
-    }
-  } else if (!node.token) {
-    // If it's a list of lists
-    alignment_width = 1;
-  }
-  std::string aligned_form = "";
-  for (int i = 0; i < lines.size(); i++) {
-    if (i >= start_index) {
-      aligned_form += str_util::repeat(alignment_width, " ");
-    }
-    aligned_form += lines.at(i);
-    if (i != lines.size() - 1) {
-      aligned_form += "\n";
-    }
-  }
-  if (!aligned_form.empty()) {
-    text = aligned_form;
-  }
-}
-}  // namespace indent
-namespace let {
-
-bool can_be_inlined(const FormatterTreeNode& form) {
-  // Check a variety of things specific to `let` style forms (ones with bindings)
-  // - does the binding list have more than one binding?
-  const auto& bindings = form.refs.at(1);  // TODO - assuming
-  if (bindings.refs.size() > 1) {
-    return false;
-  }
-  return true;
-}
-
-}  // namespace let
-
 }  // namespace formatter_rules

common/formatter/rules/formatting_rules.h

@@ -1,12 +1,19 @@
 #pragma once
 
+#include <set>
 #include <string>
 
 #include "common/formatter/formatter_tree.h"
 
 namespace formatter_rules {
+
+extern const std::set<std::string> constant_types;
+namespace constant_list {
+bool is_constant_list(const FormatterTreeNode& node);
+}
+
 // The formatter will try to collapse as much space as possible in the top-level, this means
-// separating forms by a single empty  blank line
+// separating forms by a single empty blank line
 //
 // The exception are comments, top level comments will retain their following blank lines from the
 // original source
@@ -18,11 +25,10 @@ namespace formatter_rules {
 //
 // Reference - https://github.com/kkinnear/zprint/blob/main/doc/options/blank.md
 namespace blank_lines {
-void separate_by_newline(std::string& curr_text,
-                         const FormatterTreeNode& containing_node,
-                         const FormatterTreeNode& node,
-                         const int index);
-}
+bool should_insert_blank_line(const FormatterTreeNode& containing_node,
+                              const FormatterTreeNode& node,
+                              const int index);
+}  // namespace blank_lines
 
 // TODO:
 // - align consecutive comment lines
@@ -30,7 +36,7 @@ void separate_by_newline(std::string& curr_text,
 //
 // Reference - https://github.com/kkinnear/zprint/blob/main/doc/options/comments.md
 namespace comments {
-std::string format_block_comment(const std::string& comment);
+std::vector<std::string> format_block_comment(const std::string& comment);
 }
 
 // Paired elements in a list will be kept in-line rather than the default new-line indentation
@@ -66,6 +72,8 @@ const static int min_pair_amount = 4;
 bool is_element_second_in_constant_pair(const FormatterTreeNode& containing_node,
                                         const FormatterTreeNode& node,
                                         const int index);
+bool is_element_second_in_constant_pair_new(const FormatterTreeNode& prev_node,
+                                            const FormatterTreeNode& curr_node);
 bool form_should_be_constant_paired(const FormatterTreeNode& node);
 }  // namespace constant_pairs
 
@@ -93,35 +101,6 @@ bool form_should_be_constant_paired(const FormatterTreeNode& node);
 namespace indent {
 const static int line_width_target = 120;
 
-bool form_can_be_inlined(const std::string& curr_text, const FormatterTreeNode& curr_node);
-
-// TODO - right now this is very primitive in that it only checks against our hard-coded config
-// eventually make this explore both routes and determine which is best
-// Also factor in distance from the gutter (theres some zprint rationale somewhere on this)
-bool should_form_flow(const FormatterTreeNode& list_node, const bool inlining_form);
-std::optional<bool> inline_form_element(const FormatterTreeNode& list_node, const int index);
-
-void append_newline(std::string& curr_text,
-                    const FormatterTreeNode& node,
-                    const FormatterTreeNode& containing_node,
-                    const int index,
-                    const bool flowing,
-                    const bool constant_pair_form,
-                    const bool force_newline);
-void indent_line(std::string& curr_text,
-                 const FormatterTreeNode& node,
-                 const FormatterTreeNode& containing_node,
-                 const int depth,
-                 const int index,
-                 const bool flowing);
-void align_lines(std::string& text,
-                 const FormatterTreeNode& node,
-                 const FormatterTreeNode& containing_node,
-                 const bool constant_pair_form,
-                 const bool flowing,
-                 const bool force_flow,
-                 const bool inline_element);
-
 }  // namespace indent
 
 // Let forms fall into two main categories
@@ -138,8 +117,5 @@ void align_lines(std::string& text,
 // - forms inside the let binding are flowed
 //
 // Reference - https://github.com/kkinnear/zprint/blob/main/doc/options/let.md
-namespace let {
-// TODO - like above, factor in current cursor position
-bool can_be_inlined(const FormatterTreeNode& form);
-}  // namespace let
+namespace let {}  // namespace let
 }  // namespace formatter_rules

common/formatter/rules/rule_config.cpp

@@ -1,34 +1,304 @@
 #include "rule_config.h"
 
+#include "common/util/string_util.h"
+
 namespace formatter_rules {
 namespace config {
 
-// TODO - populate these more
+static FormFormattingConfig new_inlinable_simple_flow_rule() {
+  return {.config_set = true, .hang_forms = false};
+}
 
-// TODO - this could be greatly simplified with C++20's designated initialization
-FormFormattingConfig new_flow_rule(int start_index) {
-  FormFormattingConfig cfg;
-  cfg.force_flow = true;
-  cfg.start_flow_at_index = start_index;
-  cfg.inline_until_index = start_index;
+static FormFormattingConfig new_permissive_flow_rule() {
+  return {.config_set = true, .hang_forms = false, .combine_first_two_lines = true};
+}
+
+static FormFormattingConfig new_flow_rule(int start_index, bool has_constant_pairs = false) {
+  return {.config_set = true,
+          .hang_forms = false,
+          .inline_until_index =
+              [start_index](const std::vector<std::string>& /*curr_lines*/) { return start_index; },
+          .has_constant_pairs = has_constant_pairs};
+}
+
+static FormFormattingConfig new_function_rule(int start_index, bool has_constant_pairs = false) {
+  FormFormattingConfig cfg = {
+      .config_set = true,
+      .hang_forms = false,
+      .inline_until_index =
+          [start_index](const std::vector<std::string>& /*curr_lines*/) { return start_index; },
+      .has_constant_pairs = has_constant_pairs};
+  auto arg_list_config = std::make_shared<FormFormattingConfig>();
+  arg_list_config->force_inline = true;
+  arg_list_config->hang_forms = false;
+  cfg.index_configs.emplace(2, arg_list_config);
   return cfg;
 }
 
-FormFormattingConfig new_binding_rule() {
+static FormFormattingConfig new_inlineable_flow_rule(int start_index,
+                                                     bool has_constant_pairs = false) {
+  return {.config_set = true,
+          .hang_forms = false,
+          .inline_until_index =
+              [start_index](const std::vector<std::string>& curr_lines) {
+                int total_width = 0;
+                for (const auto& line : curr_lines) {
+                  // Check for comments
+                  // TODO - this shows how this isn't really the best strategy but it holds up
+                  if (str_util::contains(line, ";")) {
+                    // Can't inline, there's a comment!
+                    return start_index;
+                  }
+                  total_width += line.length();
+                  // an empty line implies a new-line was forced, this is bleeding implementation
+                  // details, but fine for now
+                  if (line.empty()) {
+                    return start_index;
+                  }
+                }
+                if (total_width <= 120) {
+                  return (int)curr_lines.size();
+                }
+                return start_index;
+              },
+          .has_constant_pairs = has_constant_pairs};
+}
+
+static FormFormattingConfig new_defstate_rule(int start_index, bool has_constant_pairs = false) {
+  FormFormattingConfig cfg = {
+      .config_set = true,
+      .hang_forms = false,
+      .inline_until_index =
+          [start_index](const std::vector<std::string>& /*curr_lines*/) { return start_index; },
+      .has_constant_pairs = has_constant_pairs};
+  std::vector<int> state_handler_indexes = {4,  6, 8, 10,
+                                            12, 14};  // NOTE - not all of these have to be defined
+  for (const auto& index : state_handler_indexes) {
+    auto temp_config = std::make_shared<FormFormattingConfig>();
+    temp_config->prevent_inlining = true;
+    temp_config->parent_mutable_extra_indent = 2;
+    cfg.index_config_override.emplace(index, temp_config);
+  }
+  return cfg;
+}
+
+static FormFormattingConfig new_defmethod_rule(int start_index, bool has_constant_pairs = false) {
+  // TODO - might be nice to have a function that returns a config based on a given index, instead
+  // of hardcoding them!
+  // Right now this only works for non-`new` methods (else we may bleed into the body of a normal
+  // method)
+  auto arg_list_config = std::make_shared<FormFormattingConfig>();
+  arg_list_config->force_inline = true;
+  arg_list_config->hang_forms = false;
+  FormFormattingConfig cfg = {.config_set = true,
+                              .hang_forms = false,
+                              .inline_until_index =
+                                  [start_index](const std::vector<std::string>& curr_lines) {
+                                    if (curr_lines.size() >= 2 && curr_lines.at(1) == "new") {
+                                      // defmethod was changed to omit the type name for everything
+                                      // except the `new` method, so special case.
+                                      return start_index + 1;
+                                    }
+                                    return start_index;
+                                  },
+                              .has_constant_pairs = has_constant_pairs};
+  cfg.index_configs.emplace(2, arg_list_config);
+  return cfg;
+}
+
+static FormFormattingConfig new_lambda_rule(int start_index, bool has_constant_pairs = false) {
+  FormFormattingConfig cfg = {.config_set = true,
+                              .hang_forms = false,
+                              .inline_until_index =
+                                  [start_index](const std::vector<std::string>& curr_lines) {
+                                    if (curr_lines.size() >= 2 && curr_lines.at(1) == ":behavior") {
+                                      // defmethod was changed to omit the type name for everything
+                                      // except the `new` method, so special case.
+                                      return start_index + 2;
+                                    }
+                                    return start_index;
+                                  },
+                              .has_constant_pairs = has_constant_pairs};
+  return cfg;
+}
+
+static FormFormattingConfig new_defenum_rule() {
+  auto temp_list_config = std::make_shared<FormFormattingConfig>();
+  temp_list_config->force_inline = true;
+  temp_list_config->hang_forms = false;
+  return {
+      .config_set = true,
+      .hang_forms = false,
+      .inline_until_index = [](const std::vector<std::string>& /*curr_lines*/) { return 2; },
+      .has_constant_pairs = true,
+      .default_index_config = temp_list_config,
+  };
+}
+
+static FormFormattingConfig new_deftype_rule(
+    int start_index,
+    int num_columns_to_compute_widths,
+    const std::vector<int>& inlining_preventation_indices) {
   FormFormattingConfig cfg;
-  cfg.start_flow_at_index = 2;
-  cfg.bindings_at_index = 1;
-  cfg.force_flow = true;
-  cfg.force_newline_at_indices = {2};
-  cfg.allow_inlining_if_size_less_than = 2;
+  cfg.has_constant_pairs = true;
+  cfg.config_set = true;
+  cfg.hang_forms = false;
+  cfg.inline_until_index = [start_index](std::vector<std::string> curr_lines) {
+    if (curr_lines.size() >= 4 && curr_lines.at(3) == "()") {
+      return 4;
+    }
+    return start_index;
+  };
+  for (const auto& index : inlining_preventation_indices) {
+    auto temp_config = std::make_shared<FormFormattingConfig>();
+    temp_config->config_set = true;
+    temp_config->prevent_inlining = true;
+    temp_config->hang_forms = false;
+    temp_config->indentation_width = 1;
+    auto temp_list_config = std::make_shared<FormFormattingConfig>();
+    temp_list_config->force_inline = true;
+    temp_list_config->hang_forms = false;
+    temp_config->default_index_config = temp_list_config;
+    if (index == 3) {
+      temp_config->determine_column_widths_for_list_elements = true;
+      temp_config->num_columns_to_compute_widths = num_columns_to_compute_widths;
+    }
+    cfg.index_configs.emplace(index, temp_config);
+  }
+  return cfg;
+}
+
+static FormFormattingConfig new_binding_rule(int form_head_width) {
+  FormFormattingConfig cfg;
+  cfg.config_set = true;
+  cfg.hang_forms = false;
+  cfg.combine_first_two_lines = true;
   auto binding_list_config = std::make_shared<FormFormattingConfig>();
-  binding_list_config->force_flow = true;
+  binding_list_config->config_set = true;
+  binding_list_config->hang_forms = false;
+  binding_list_config->indentation_width = 1;
+  binding_list_config->indentation_width_for_index = [form_head_width](FormFormattingConfig /*cfg*/,
+                                                                       int index) {
+    if (index == 0) {
+      return 0;
+    }
+    return form_head_width;
+  };
+  binding_list_config->should_prevent_inlining = [](FormFormattingConfig /*config*/, int num_refs) {
+    // Only prevent inlining a binding list, if there are more than 1 bindings
+    if (num_refs > 1) {
+      return true;
+    }
+    return false;
+  };
+  binding_list_config->prevent_inlining =
+      true;  // TODO - we only want to prevent inlining if there are more than 2 elements
   cfg.index_configs.emplace(1, binding_list_config);
   return cfg;
 }
 
+static FormFormattingConfig new_inline_binding_rule(int form_head_width) {
+  FormFormattingConfig cfg;
+  cfg.config_set = true;
+  cfg.hang_forms = false;
+  cfg.combine_first_two_lines = true;
+  auto binding_list_config = std::make_shared<FormFormattingConfig>();
+  binding_list_config->config_set = true;
+  binding_list_config->hang_forms = false;
+  binding_list_config->indentation_width = 1;
+  binding_list_config->indentation_width_for_index = [form_head_width](FormFormattingConfig /*cfg*/,
+                                                                       int index) {
+    if (index == 0) {
+      return 0;
+    }
+    return form_head_width;
+  };
+  binding_list_config->should_prevent_inlining = [](FormFormattingConfig /*config*/, int num_refs) {
+    return false;
+  };
+  cfg.index_configs.emplace(1, binding_list_config);
+  return cfg;
+}
+
+static FormFormattingConfig new_pair_rule(bool combine_first_two_expr) {
+  FormFormattingConfig cfg;
+  cfg.config_set = true;
+  cfg.hang_forms = false;
+  cfg.prevent_inlining = true;
+  cfg.combine_first_two_lines = combine_first_two_expr;
+  auto pair_config = std::make_shared<FormFormattingConfig>();
+  pair_config->config_set = true;
+  pair_config->hang_forms = false;
+  pair_config->indentation_width = 1;
+  cfg.default_index_config = pair_config;
+  return cfg;
+}
+
+static FormFormattingConfig new_top_level_inline_form(bool elide_new_line) {
+  return {.force_inline = true, .elide_top_level_newline = elide_new_line};
+}
+
 const std::unordered_map<std::string, FormFormattingConfig> opengoal_form_config = {
+    {"case", new_pair_rule(true)},
+    {"case-str", new_pair_rule(true)},
+    {"cond", new_pair_rule(false)},
+    {"#cond", new_pair_rule(false)},
+    {"in-package", new_top_level_inline_form(true)},
+    {"bundles", new_top_level_inline_form(true)},
+    {"require", new_top_level_inline_form(true)},
+    {"def-art-elt", new_top_level_inline_form(true)},
+    {"def-joint-node", new_top_level_inline_form(true)},
+    {"declare-file", new_top_level_inline_form(false)},
+    {"defenum", new_defenum_rule()},
+    {"defmethod", new_defmethod_rule(3)},
+    {"lambda", new_lambda_rule(2)},
+    {"deftype", new_deftype_rule(3, 1, {3, 4, 5, 6})},
     {"defun", new_flow_rule(3)},
-    {"let", new_binding_rule()}};
+    {"defun-recursive", new_flow_rule(4)},
+    {"defun-debug-recursive", new_flow_rule(4)},
+    {"defun-debug", new_flow_rule(3)},
+    {"defbehavior", new_flow_rule(4)},
+    {"if", new_inlineable_flow_rule(2)},
+    {"#if", new_inlineable_flow_rule(2)},
+    {"define", new_permissive_flow_rule()},
+    {"def-mips2c", new_permissive_flow_rule()},
+    {"defconstant", new_permissive_flow_rule()},
+    {"defglobalconstant", new_permissive_flow_rule()},
+    {"defmethod-mips2c", new_permissive_flow_rule()},
+    {"define-extern", new_permissive_flow_rule()},
+    {"declare-type", new_permissive_flow_rule()},
+    {"defmacro", new_function_rule(3)},
+    {"desfun", new_function_rule(3)},
+    {"defskelgroup", new_flow_rule(2, true)},
+    {"defpartgroup", new_flow_rule(2, true)},
+    {"defpart", new_flow_rule(2, true)},
+    {"defstate", new_defstate_rule(3, true)},
+    {"behavior", new_flow_rule(2)},
+    {"dotimes", new_flow_rule(2)},
+    {"dolist", new_flow_rule(2)},
+    {"process-spawn-function", new_flow_rule(2)},
+    {"let", new_binding_rule(4)},
+    {"protect", new_binding_rule(4)},
+    {"let*", new_binding_rule(5)},
+    {"rlet", new_binding_rule(5)},
+    {"when", new_flow_rule(2)},
+    {"unless", new_flow_rule(2)},
+    {"with-profiler", new_flow_rule(2)},
+    {"with-pc", new_flow_rule(0)},
+    {"#unless", new_flow_rule(2)},
+    {"#when", new_flow_rule(2)},
+    {"countdown", new_flow_rule(2)},
+    {"until", new_flow_rule(2)},
+    {"loop", new_flow_rule(0)},
+    {"while", new_flow_rule(2)},
+    {"begin", new_flow_rule(0)},
+    {"seval", new_flow_rule(0)},
+    {"with-pp", new_flow_rule(0)},
+    {"with-gensyms", new_flow_rule(2)},
+    {"with-vf0", new_flow_rule(0)},
+    {"with-vf", new_inline_binding_rule(8)},
+    {"with-cnt-vif-block", new_inline_binding_rule(19)},
+    {"local-vars", new_inlinable_simple_flow_rule()},
+    {"with-dma-buffer-add-bucket", new_flow_rule(2)}};
 }  // namespace config
 }  // namespace formatter_rules

common/formatter/rules/rule_config.h

@@ -1,26 +1,45 @@
 #pragma once
 
+#include <functional>
 #include <memory>
+#include <optional>
 #include <string>
 #include <unordered_map>
 #include <vector>
 
-#include "common/formatter/rules/formatting_rules.h"
-
 namespace formatter_rules {
 namespace config {
 struct FormFormattingConfig {
-  bool force_hang = false;
-  bool force_flow = false;
-  std::optional<int> allow_inlining_if_size_less_than = {};
-  int start_hang_at_index = 0;
-  int start_flow_at_index = 0;
-  int inline_until_index = -1;
-  std::optional<int> bindings_at_index = {};
-  std::optional<int> skip_newlines_until_index = {};
-  std::vector<int> force_newline_at_indices = {};
-  bool bindings_force_newlines = false;
+  bool config_set = false;
+  bool force_inline = false;
+  bool hang_forms =
+      true;  // TODO - remove this eventually, it's only involved in setting the
+             // indentation width, which we can do via the new indentation_width function
+  int indentation_width =
+      2;  // 2 for a flow // TODO - also remove this, prefer storing the first node's width in the
+          // metadata on the first pass, that's basically all this does
+  std::function<int(FormFormattingConfig, int)> indentation_width_for_index =
+      [](FormFormattingConfig config, int /*index*/) { return config.indentation_width; };
+  bool combine_first_two_lines =
+      false;  // NOTE - basically hang, but will probably stick around after hang is gone, may be
+              // redundant (inline_until_index!)
+  std::function<std::optional<int>(const std::vector<std::string>& /*curr_lines*/)>
+      inline_until_index = [](std::vector<std::string> /*curr_lines*/) { return std::nullopt; };
+  bool has_constant_pairs = false;
+  bool prevent_inlining = false;  // TODO - duplicate of below
+  std::function<bool(FormFormattingConfig, int num_refs)> should_prevent_inlining =
+      [](FormFormattingConfig config, int /*num_refs*/) { return config.prevent_inlining; };
+  int parent_mutable_extra_indent = 0;
+  std::optional<std::shared_ptr<FormFormattingConfig>> default_index_config;
   std::unordered_map<int, std::shared_ptr<FormFormattingConfig>> index_configs = {};
+  // TODO / NOTe - not fully implemented, only doing `parent_mutable_extra_indent` right now
+  std::unordered_map<int, std::shared_ptr<FormFormattingConfig>> index_config_override = {};
+
+  bool determine_column_widths_for_list_elements = false;
+  int num_columns_to_compute_widths = 0;
+  std::vector<int> list_element_column_widths = {};
+
+  bool elide_top_level_newline = false;
 };
 
 extern const std::unordered_map<std::string, FormFormattingConfig> opengoal_form_config;

common/global_profiler/GlobalProfiler.cpp

@@ -9,7 +9,7 @@
 #include "common/util/FileUtil.h"
 #include "common/common_types.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 #include "third-party/json.hpp"
 
 #ifdef OS_POSIX
@@ -26,6 +26,8 @@ u64 get_current_tid() {
 }
 #endif
 #include "common/log/log.h"
+#include "common/util/compress.h"
+#include "common/util/string_util.h"
 // clang-format on
 
 u64 get_current_ts() {
@@ -34,16 +36,25 @@ u64 get_current_ts() {
 
 GlobalProfiler::GlobalProfiler() {
   m_t0 = get_current_ts();
-  set_max_events(65536);
+  m_nodes.resize(m_max_events);
 }
 
-void GlobalProfiler::set_max_events(size_t event_count) {
-  ASSERT(!m_enabled);
-  m_nodes.resize(event_count);
+void GlobalProfiler::update_event_buffer_size(size_t new_size) {
+  m_max_events = new_size;
+  m_nodes.resize(m_max_events);
+}
+
+void GlobalProfiler::set_waiting_for_event(const std::string& event_name) {
+  if (!event_name.empty()) {
+    m_waiting_for_event = event_name;
+  }
 }
 
 void GlobalProfiler::event(const char* name, ProfNode::Kind kind) {
-  if (!m_enabled) {
+  if (m_waiting_for_event && m_waiting_for_event.value() == name) {
+    m_ignore_events = true;
+  }
+  if (!m_enabled || m_ignore_events) {
     return;
   }
   size_t my_idx = (m_next_idx++ % m_nodes.size());
@@ -63,12 +74,16 @@ void GlobalProfiler::root_event() {
   instant_event("ROOT");
 }
 
+size_t GlobalProfiler::get_next_idx() {
+  return (m_next_idx % m_nodes.size());
+}
+
 void GlobalProfiler::begin_event(const char* name) {
   event(name, ProfNode::BEGIN);
 }
 
 void GlobalProfiler::end_event() {
-  if (!m_enabled) {
+  if (!m_enabled || m_ignore_events) {
     return;
   }
   size_t my_idx = (m_next_idx++ % m_nodes.size());
@@ -87,8 +102,10 @@ void GlobalProfiler::set_enable(bool en) {
   m_enabled = en;
 }
 
-void GlobalProfiler::dump_to_json(const std::string& path) {
-  ASSERT(!m_enabled);
+void GlobalProfiler::dump_to_json() {
+  if (m_enabled) {
+    set_enable(false);
+  }
 
   nlohmann::json json;
   auto& trace_events = json["traceEvents"];
@@ -166,7 +183,7 @@ void GlobalProfiler::dump_to_json(const std::string& path) {
       lg::debug("out of order: {} {} {} ms", event.ts / 1000.f, info.debug / 1000.f,
                 (info.debug - event.ts) / 1000000.f);
       lg::debug("  idx: {}, range {} {}", event_idx, info.lowest_at_target, info.highest_at_target);
-      lg::debug("  now: {}", m_next_idx);
+      lg::debug("  now: {}", m_next_idx.load());
     }
     info.debug = event.ts;
   }
@@ -176,7 +193,20 @@ void GlobalProfiler::dump_to_json(const std::string& path) {
               t.second.highest_at_target);
   }
 
-  file_util::write_text_file(path, json.dump());
+  if (m_enable_compression) {
+    const auto json_str = json.dump();
+    const auto compressed_data =
+        compression::compress_zstd_no_header(json_str.data(), json_str.size());
+    auto file_path = file_util::get_jak_project_dir() / "profile_data" /
+                     fmt::format("prof-{}.json.zst", str_util::current_local_timestamp_no_colons());
+    file_util::create_dir_if_needed_for_file(file_path);
+    file_util::write_binary_file(file_path, compressed_data.data(), compressed_data.size());
+  } else {
+    auto file_path = file_util::get_jak_project_dir() / "profile_data" /
+                     fmt::format("prof-{}.json", str_util::current_local_timestamp_no_colons());
+    file_util::create_dir_if_needed_for_file(file_path);
+    file_util::write_text_file(file_path, json.dump());
+  }
 }
 
 GlobalProfiler gprof;

common/global_profiler/GlobalProfiler.h

@@ -1,6 +1,7 @@
 #pragma once
 
 #include <atomic>
+#include <optional>
 #include <string>
 #include <vector>
 
@@ -16,24 +17,37 @@ struct ProfNode {
 class GlobalProfiler {
  public:
   GlobalProfiler();
-  void set_max_events(size_t event_count);
+  size_t get_max_events() { return m_max_events; }
+  void update_event_buffer_size(size_t new_size);
+  void set_waiting_for_event(const std::string& event_name);
   void instant_event(const char* name);
   void begin_event(const char* name);
   void event(const char* name, ProfNode::Kind kind);
   void end_event();
   void clear();
   void set_enable(bool en);
-  void dump_to_json(const std::string& path);
+  void dump_to_json();
   void root_event();
+  bool is_enabled() { return m_enabled; }
+  size_t get_next_idx();
+
+  bool m_enable_compression = false;
 
  private:
   std::atomic_bool m_enabled = false;
+  size_t m_max_events = 65536;
   u64 m_t0 = 0;
   std::atomic_size_t m_next_idx = 0;
   std::vector<ProfNode> m_nodes;
+  // this is very niche, but sometimes you want to capture up to a given event (ie. long startup)
+  // instead of having to make the user quit and record as fast as possible, we can instead just
+  // stop capturing events once we have received what we are looking for
+  std::optional<std::string> m_waiting_for_event = {};
+  bool m_ignore_events = false;
 };
 
 struct ScopedEvent {
+  ScopedEvent(GlobalProfiler* _prof) : prof(_prof){};
   ScopedEvent(const ScopedEvent&) = delete;
   ScopedEvent& operator=(const ScopedEvent&) = delete;
   GlobalProfiler* prof = nullptr;

common/goal_constants.h

@@ -62,12 +62,23 @@ constexpr int LEVEL_MAX = 6;
 constexpr int LEVEL_TOTAL = LEVEL_MAX + 1;
 }  // namespace jak2
 
+namespace jak3 {
+// for now, we don't have the ability to extend the size of the symbol table
+constexpr s32 GOAL_MAX_SYMBOLS = 0x4000;
+// amount of levels in level heap
+constexpr int LEVEL_MAX = 10;
+// total amount of levels, including ones outside level heap (default-level)
+constexpr int LEVEL_TOTAL = LEVEL_MAX + 1;
+}  // namespace jak3
+
 constexpr s32 max_symbols(GameVersion version) {
   switch (version) {
     case GameVersion::Jak1:
       return jak1::GOAL_MAX_SYMBOLS;
     case GameVersion::Jak2:
       return jak2::GOAL_MAX_SYMBOLS;
+    case GameVersion::Jak3:
+      return jak3::GOAL_MAX_SYMBOLS;
   }
 }
 

common/goos/Interpreter.cpp

@@ -12,13 +12,18 @@
 #include "common/goos/Printer.h"
 #include "common/log/log.h"
 #include "common/util/FileUtil.h"
+#include "common/util/crc32.h"
 #include "common/util/string_util.h"
 #include "common/util/unicode_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 Interpreter::Interpreter(const std::string& username) {
+  m_true_object = intern("#t");
+  m_false_object = intern("#f");
+  m_true_sym = m_true_object.as_symbol().name_ptr;
+  m_false_sym = m_false_object.as_symbol().name_ptr;
   // Interpreter startup:
   // create the GOOS global environment
   global_environment = EnvironmentObject::make_new("global");
@@ -33,15 +38,16 @@ Interpreter::Interpreter(const std::string& username) {
   define_var_in_env(goal_env, global_environment, "*global-env*");
 
   // set user profile name
-  auto user = SymbolObject::make_new(reader.symbolTable, username);
+  auto user = Object::make_symbol(&reader.symbolTable, username.c_str());
   define_var_in_env(global_environment, user, "*user*");
-  define_var_in_env(goal_env, user, "*user*");
 
   // setup maps
-  special_forms = {
+  init_special_forms({
       {"define", &Interpreter::eval_define},
       {"quote", &Interpreter::eval_quote},
       {"set!", &Interpreter::eval_set},
+      {"let", &Interpreter::eval_let},
+      {"let*", &Interpreter::eval_let_star},
       {"lambda", &Interpreter::eval_lambda},
       {"cond", &Interpreter::eval_cond},
       {"or", &Interpreter::eval_or},
@@ -49,53 +55,54 @@ Interpreter::Interpreter(const std::string& username) {
       {"macro", &Interpreter::eval_macro},
       {"quasiquote", &Interpreter::eval_quasiquote},
       {"while", &Interpreter::eval_while},
-  };
+  });
 
-  builtin_forms = {{"top-level", &Interpreter::eval_begin},
-                   {"begin", &Interpreter::eval_begin},
-                   {"exit", &Interpreter::eval_exit},
-                   {"read", &Interpreter::eval_read},
-                   {"read-data-file", &Interpreter::eval_read_data_file},
-                   {"read-file", &Interpreter::eval_read_file},
-                   {"print", &Interpreter::eval_print},
-                   {"inspect", &Interpreter::eval_inspect},
-                   {"load-file", &Interpreter::eval_load_file},
-                   {"try-load-file", &Interpreter::eval_try_load_file},
-                   {"eq?", &Interpreter::eval_equals},
-                   {"gensym", &Interpreter::eval_gensym},
-                   {"eval", &Interpreter::eval_eval},
-                   {"cons", &Interpreter::eval_cons},
-                   {"car", &Interpreter::eval_car},
-                   {"cdr", &Interpreter::eval_cdr},
-                   {"set-car!", &Interpreter::eval_set_car},
-                   {"set-cdr!", &Interpreter::eval_set_cdr},
-                   {"+", &Interpreter::eval_plus},
-                   {"-", &Interpreter::eval_minus},
-                   {"*", &Interpreter::eval_times},
-                   {"/", &Interpreter::eval_divide},
-                   {"=", &Interpreter::eval_numequals},
-                   {"<", &Interpreter::eval_lt},
-                   {">", &Interpreter::eval_gt},
-                   {"<=", &Interpreter::eval_leq},
-                   {">=", &Interpreter::eval_geq},
-                   {"null?", &Interpreter::eval_null},
-                   {"type?", &Interpreter::eval_type},
-                   {"fmt", &Interpreter::eval_format},
-                   {"error", &Interpreter::eval_error},
-                   {"string-ref", &Interpreter::eval_string_ref},
-                   {"string-length", &Interpreter::eval_string_length},
-                   {"string-append", &Interpreter::eval_string_append},
-                   {"string-starts-with?", &Interpreter::eval_string_starts_with},
-                   {"string-ends-with?", &Interpreter::eval_string_ends_with},
-                   {"string-split", &Interpreter::eval_string_split},
-                   {"string-substr", &Interpreter::eval_string_substr},
-                   {"ash", &Interpreter::eval_ash},
-                   {"symbol->string", &Interpreter::eval_symbol_to_string},
-                   {"string->symbol", &Interpreter::eval_string_to_symbol},
-                   {"get-environment-variable", &Interpreter::eval_get_env},
-                   {"make-string-hash-table", &Interpreter::eval_make_string_hash_table},
-                   {"hash-table-set!", &Interpreter::eval_hash_table_set},
-                   {"hash-table-try-ref", &Interpreter::eval_hash_table_try_ref}};
+  init_builtin_forms({{"top-level", &Interpreter::eval_begin},
+                      {"begin", &Interpreter::eval_begin},
+                      {"exit", &Interpreter::eval_exit},
+                      {"read", &Interpreter::eval_read},
+                      {"read-data-file", &Interpreter::eval_read_data_file},
+                      {"read-file", &Interpreter::eval_read_file},
+                      {"print", &Interpreter::eval_print},
+                      {"inspect", &Interpreter::eval_inspect},
+                      {"load-file", &Interpreter::eval_load_file},
+                      {"try-load-file", &Interpreter::eval_try_load_file},
+                      {"eq?", &Interpreter::eval_equals},
+                      {"gensym", &Interpreter::eval_gensym},
+                      {"eval", &Interpreter::eval_eval},
+                      {"cons", &Interpreter::eval_cons},
+                      {"car", &Interpreter::eval_car},
+                      {"cdr", &Interpreter::eval_cdr},
+                      {"set-car!", &Interpreter::eval_set_car},
+                      {"set-cdr!", &Interpreter::eval_set_cdr},
+                      {"+", &Interpreter::eval_plus},
+                      {"-", &Interpreter::eval_minus},
+                      {"*", &Interpreter::eval_times},
+                      {"/", &Interpreter::eval_divide},
+                      {"=", &Interpreter::eval_numequals},
+                      {"<", &Interpreter::eval_lt},
+                      {">", &Interpreter::eval_gt},
+                      {"<=", &Interpreter::eval_leq},
+                      {">=", &Interpreter::eval_geq},
+                      {"null?", &Interpreter::eval_null},
+                      {"type?", &Interpreter::eval_type},
+                      {"fmt", &Interpreter::eval_format},
+                      {"error", &Interpreter::eval_error},
+                      {"string-ref", &Interpreter::eval_string_ref},
+                      {"string-length", &Interpreter::eval_string_length},
+                      {"int->string", &Interpreter::eval_int_to_string},
+                      {"string-append", &Interpreter::eval_string_append},
+                      {"string-starts-with?", &Interpreter::eval_string_starts_with},
+                      {"string-ends-with?", &Interpreter::eval_string_ends_with},
+                      {"string-split", &Interpreter::eval_string_split},
+                      {"string-substr", &Interpreter::eval_string_substr},
+                      {"ash", &Interpreter::eval_ash},
+                      {"symbol->string", &Interpreter::eval_symbol_to_string},
+                      {"string->symbol", &Interpreter::eval_string_to_symbol},
+                      {"get-environment-variable", &Interpreter::eval_get_env},
+                      {"make-string-hash-table", &Interpreter::eval_make_string_hash_table},
+                      {"hash-table-set!", &Interpreter::eval_hash_table_set},
+                      {"hash-table-try-ref", &Interpreter::eval_hash_table_try_ref}});
 
   string_to_type = {{"empty-list", ObjectType::EMPTY_LIST},
                     {"integer", ObjectType::INTEGER},
@@ -113,6 +120,28 @@ Interpreter::Interpreter(const std::string& username) {
   load_goos_library();
 }
 
+void Interpreter::init_builtin_forms(
+    const std::unordered_map<std::string,
+                             Object (Interpreter::*)(const Object&,
+                                                     Arguments&,
+                                                     const std::shared_ptr<EnvironmentObject>&)>&
+        forms) {
+  for (const auto& [name, fn] : forms) {
+    builtin_forms[(void*)intern_ptr(name).name_ptr] = fn;
+  }
+}
+
+void Interpreter::init_special_forms(
+    const std::unordered_map<std::string,
+                             Object (Interpreter::*)(const Object&,
+                                                     const Object&,
+                                                     const std::shared_ptr<EnvironmentObject>&)>&
+        forms) {
+  for (const auto& [name, fn] : forms) {
+    special_forms.push_back(std::make_pair((void*)intern_ptr(name).name_ptr, fn));
+  }
+}
+
 /*!
  * Add a user defined special form. The given function will be called with unevaluated arguments.
  * Lookup from these forms occurs after special/builtin, but before any env lookups.
@@ -121,7 +150,7 @@ void Interpreter::register_form(
     const std::string& name,
     const std::function<
         Object(const Object&, Arguments&, const std::shared_ptr<EnvironmentObject>&)>& form) {
-  m_custom_forms[name] = form;
+  m_custom_forms.push_back(std::make_pair((void*)intern_ptr(name).name_ptr, form));
 }
 
 Interpreter::~Interpreter() {
@@ -149,19 +178,19 @@ void Interpreter::load_goos_library() {
 /*!
  * In env, set the variable named "name" to the value var.
  */
-void Interpreter::define_var_in_env(Object& env, Object& var, const std::string& name) {
-  env.as_env()->vars[intern_ptr(name)] = var;
+void Interpreter::define_var_in_env(Object& env, const Object& var, const std::string& name) {
+  env.as_env()->vars.set(InternedSymbolPtr{intern_ptr(name)}, var);
 }
 
 /*!
  * Get a symbol with the given name, creating one if none exist.
  */
 Object Interpreter::intern(const std::string& name) {
-  return SymbolObject::make_new(reader.symbolTable, name);
+  return Object::make_symbol(&reader.symbolTable, name.c_str());
 }
 
-HeapObject* Interpreter::intern_ptr(const std::string& name) {
-  return reader.symbolTable.intern_ptr(name);
+InternedSymbolPtr Interpreter::intern_ptr(const std::string& name) {
+  return reader.symbolTable.intern(name.c_str());
 }
 
 /*!
@@ -218,10 +247,9 @@ Object Interpreter::eval_with_rewind(const Object& obj,
  * Returns if the variable was found.
  */
 bool Interpreter::get_global_variable_by_name(const std::string& name, Object* dest) {
-  auto kv = global_environment.as_env()->vars.find(
-      SymbolObject::make_new(reader.symbolTable, name).as_symbol());
-  if (kv != global_environment.as_env()->vars.end()) {
-    *dest = kv->second;
+  auto* obj = global_environment.as_env()->find(name.c_str(), &reader.symbolTable);
+  if (obj) {
+    *dest = *obj;
     return true;
   }
   return false;
@@ -231,12 +259,11 @@ bool Interpreter::get_global_variable_by_name(const std::string& name, Object* d
  * Sets the variable to the value. Overwrites an existing value, or creates a new global.
  */
 void Interpreter::set_global_variable_by_name(const std::string& name, const Object& value) {
-  auto sym = SymbolObject::make_new(reader.symbolTable, name).as_symbol();
-  global_environment.as_env()->vars[sym] = value;
+  define_var_in_env(global_environment, value, name);
 }
 
 void Interpreter::set_global_variable_to_symbol(const std::string& name, const std::string& value) {
-  auto sym = SymbolObject::make_new(reader.symbolTable, value);
+  auto sym = Object::make_symbol(&reader.symbolTable, value.c_str());
   set_global_variable_by_name(name, sym);
 }
 
@@ -269,18 +296,18 @@ Arguments Interpreter::get_args(const Object& form, const Object& rest, const Ar
     const auto& arg = current->as_pair()->car;
 
     // did we get a ":keyword"
-    if (arg.is_symbol() && arg.as_symbol()->name.at(0) == ':') {
-      auto key_name = arg.as_symbol()->name.substr(1);
+    if (arg.is_symbol() && arg.as_symbol().name_ptr[0] == ':') {
+      auto key_name = arg.as_symbol().name_ptr + 1;
       const auto& kv = spec.named.find(key_name);
 
       // check for unknown key name
       if (!spec.varargs && kv == spec.named.end()) {
-        throw_eval_error(form, "Key argument " + key_name + " wasn't expected");
+        throw_eval_error(form, fmt::format("Key argument {} wasn't expected", key_name));
       }
 
       // check for multiple definition of key
       if (args.named.find(key_name) != args.named.end()) {
-        throw_eval_error(form, "Key argument " + key_name + " multiply defined");
+        throw_eval_error(form, fmt::format("Key argument {} multiply defined", key_name));
       }
 
       // check for well-formed :key value expression
@@ -409,7 +436,7 @@ ArgumentSpec Interpreter::parse_arg_spec(const Object& form, Object& rest) {
       throw_eval_error(form, "args must be symbols");
     }
 
-    if (arg.as_symbol()->name == "&rest") {
+    if (arg.as_symbol() == "&rest") {
       // special case for &rest
       current = current.as_pair()->cdr;
       if (!current.is_pair()) {
@@ -420,20 +447,20 @@ ArgumentSpec Interpreter::parse_arg_spec(const Object& form, Object& rest) {
         throw_eval_error(form, "rest name must be a symbol");
       }
 
-      spec.rest = rest_name.as_symbol()->name;
+      spec.rest = rest_name.as_symbol().name_ptr;
 
       if (!current.as_pair()->cdr.is_empty_list()) {
         throw_eval_error(form, "rest must be the last argument");
       }
-    } else if (arg.as_symbol()->name == "&key") {
+    } else if (arg.as_symbol() == "&key") {
       // special case for &key
       current = current.as_pair()->cdr;
       auto key_arg = current.as_pair()->car;
       if (key_arg.is_symbol()) {
         // form is &key name
-        auto key_arg_name = key_arg.as_symbol()->name;
+        auto key_arg_name = key_arg.as_symbol().name_ptr;
         if (spec.named.find(key_arg_name) != spec.named.end()) {
-          throw_eval_error(form, "key argument " + key_arg_name + " multiply defined");
+          throw_eval_error(form, fmt::format("key argument {} multiply defined", key_arg_name));
         }
         spec.named[key_arg_name] = NamedArg();
       } else if (key_arg.is_pair()) {
@@ -444,9 +471,9 @@ ArgumentSpec Interpreter::parse_arg_spec(const Object& form, Object& rest) {
         if (!kn.is_symbol()) {
           throw_eval_error(form, "key argument must have a symbol as a name");
         }
-        auto key_arg_name = kn.as_symbol()->name;
+        auto key_arg_name = kn.as_symbol().name_ptr;
         if (spec.named.find(key_arg_name) != spec.named.end()) {
-          throw_eval_error(form, "key argument " + key_arg_name + " multiply defined");
+          throw_eval_error(form, fmt::format("key argument {} multiply defined", key_arg_name));
         }
         NamedArg na;
 
@@ -466,7 +493,7 @@ ArgumentSpec Interpreter::parse_arg_spec(const Object& form, Object& rest) {
         throw_eval_error(form, "invalid key argument");
       }
     } else {
-      spec.unnamed.push_back(arg.as_symbol()->name);
+      spec.unnamed.push_back(arg.as_symbol().name_ptr);
     }
 
     current = current.as_pair()->cdr;
@@ -494,20 +521,22 @@ void Interpreter::vararg_check(
 /*!
  * Evaluate a list and return the result of the last evaluation.
  */
-Object Interpreter::eval_list_return_last(const Object& form,
+Object Interpreter::eval_list_return_last(const Object& /*form*/,
                                           Object rest,
                                           const std::shared_ptr<EnvironmentObject>& env) {
-  Object o = std::move(rest);
-  Object rv = Object::make_empty_list();
-  for (;;) {
-    if (o.is_pair()) {
-      auto op = o.as_pair();
-      rv = eval_with_rewind(op->car, env);
-      o = op->cdr;
-    } else if (o.is_empty_list()) {
-      return rv;
+  if (rest.is_empty_list()) {
+    return rest;
+  }
+
+  const Object* iter = &rest;
+  while (true) {
+    const Object* next = &iter->as_pair()->cdr;
+    const Object* item = &iter->as_pair()->car;
+    if (next->is_empty_list()) {
+      return eval_with_rewind(*item, env);
     } else {
-      throw_eval_error(form, "malformed body to evaluate");
+      eval(*item, env);
+      iter = next;
     }
   }
 }
@@ -552,7 +581,7 @@ bool try_symbol_lookup(const Object& sym,
                        const std::shared_ptr<EnvironmentObject>& env,
                        Object* dest) {
   // booleans are hard-coded here
-  if (sym.as_symbol()->name == "#t" || sym.as_symbol()->name == "#f") {
+  if (sym.as_symbol() == "#t" || sym.as_symbol() == "#f") {
     *dest = sym;
     return true;
   }
@@ -560,9 +589,9 @@ bool try_symbol_lookup(const Object& sym,
   // loop up envs until we find it.
   EnvironmentObject* search_env = env.get();
   for (;;) {
-    const auto& kv = search_env->vars.find(sym.as_symbol());
-    if (kv != search_env->vars.end()) {
-      *dest = kv->second;
+    auto* obj = search_env->vars.lookup(sym.as_symbol());
+    if (obj) {
+      *dest = *obj;
       return true;
     }
 
@@ -593,6 +622,59 @@ bool Interpreter::eval_symbol(const Object& sym,
   return try_symbol_lookup(sym, env, result);
 }
 
+Object Interpreter::eval_let_star(const goos::Object& form,
+                                  const goos::Object& rest,
+                                  const std::shared_ptr<EnvironmentObject>& env) {
+  return eval_let_common(form, rest, env, true);
+}
+
+Object Interpreter::eval_let(const goos::Object& form,
+                             const goos::Object& rest,
+                             const std::shared_ptr<EnvironmentObject>& env) {
+  return eval_let_common(form, rest, env, false);
+}
+
+Object Interpreter::eval_let_common(const goos::Object& form,
+                                    const goos::Object& rest,
+                                    const std::shared_ptr<EnvironmentObject>& env,
+                                    bool is_star) {
+  if (!rest.is_pair()) {
+    throw_eval_error(form, "first argument to let must be bindings");
+  }
+
+  const auto* bindings_iter = &rest.as_pair()->car;
+  const auto* body_iter = &rest.as_pair()->cdr;
+
+  if (!bindings_iter->is_pair()) {
+    throw_eval_error(form, "let cannot have empty bindings");
+  }
+
+  std::shared_ptr<EnvironmentObject> new_env = std::make_shared<EnvironmentObject>();
+  new_env->parent_env = env;
+
+  while (!bindings_iter->is_empty_list()) {
+    const auto* binding = &bindings_iter->as_pair()->car;
+    if (!binding->is_pair()) {
+      throw_eval_error(form, "let binding invalid");
+    }
+    const auto& name = binding->as_pair()->car;
+    if (!name.is_symbol()) {
+      throw_eval_error(form, "let binding invalid");
+    }
+
+    binding = &binding->as_pair()->cdr;
+    if (!binding->is_pair() || !binding->as_pair()->cdr.is_empty_list()) {
+      throw_eval_error(form, "let binding invalid");
+    }
+
+    new_env->vars.set(name.as_symbol(), eval(binding->as_pair()->car, is_star ? new_env : env));
+
+    bindings_iter = &bindings_iter->as_pair()->cdr;
+  }
+
+  return eval_list_return_last(*body_iter, *body_iter, new_env);
+}
+
 /*!
  * Evaluate a pair, either as special form, builtin form, macro application, or lambda application.
  */
@@ -606,13 +688,14 @@ Object Interpreter::eval_pair(const Object& obj, const std::shared_ptr<Environme
     const auto& head_sym = head.as_symbol();
 
     // try a special form first
-    const auto& kv_sf = special_forms.find(head_sym->name);
-    if (kv_sf != special_forms.end()) {
-      return ((*this).*(kv_sf->second))(obj, rest, env);
+    for (const auto& sf : special_forms) {
+      if (sf.first == head_sym.name_ptr) {
+        return ((*this).*(sf.second))(obj, rest, env);
+      }
     }
 
     // try builtins next
-    const auto& kv_b = builtin_forms.find(head_sym->name);
+    const auto& kv_b = builtin_forms.find((void*)head_sym.name_ptr);
     if (kv_b != builtin_forms.end()) {
       Arguments args = get_args(obj, rest, make_varargs());
       // all "built-in" forms expect arguments to be evaluated (that's why they aren't special)
@@ -621,10 +704,11 @@ Object Interpreter::eval_pair(const Object& obj, const std::shared_ptr<Environme
     }
 
     // try custom forms next
-    const auto& kv_u = m_custom_forms.find(head_sym->name);
-    if (kv_u != m_custom_forms.end()) {
-      Arguments args = get_args(obj, rest, make_varargs());
-      return (kv_u->second)(obj, args, env);
+    for (const auto& cf : m_custom_forms) {
+      if (cf.first == head_sym.name_ptr) {
+        Arguments args = get_args(obj, rest, make_varargs());
+        return (cf.second)(obj, args, env);
+      }
     }
 
     // try macros next
@@ -678,18 +762,18 @@ void Interpreter::set_args_in_env(const Object& form,
 
   // unnamed args
   for (size_t i = 0; i < arg_spec.unnamed.size(); i++) {
-    env->vars[intern(arg_spec.unnamed.at(i)).as_symbol()] = args.unnamed.at(i);
+    env->vars.set(intern_ptr(arg_spec.unnamed.at(i).c_str()), args.unnamed.at(i));
   }
 
   // named args
   for (const auto& kv : arg_spec.named) {
-    env->vars[intern(kv.first).as_symbol()] = args.named.at(kv.first);
+    env->vars.set(intern_ptr(kv.first), args.named.at(kv.first));
   }
 
   // rest args
   if (!arg_spec.rest.empty()) {
     // will correctly handle the '() case
-    env->vars[intern(arg_spec.rest).as_symbol()] = build_list(args.rest);
+    env->vars.set(intern_ptr(arg_spec.rest), build_list(args.rest));
   } else {
     if (!args.rest.empty()) {
       throw_eval_error(form, "got too many arguments");
@@ -714,7 +798,7 @@ Object Interpreter::eval_define(const Object& form,
   }
 
   Object value = eval_with_rewind(args.unnamed[1], env);
-  define_env->vars[args.unnamed[0].as_symbol()] = value;
+  define_env->vars.set(args.unnamed[0].as_symbol(), value);
   return value;
 }
 
@@ -732,10 +816,10 @@ Object Interpreter::eval_set(const Object& form,
 
   std::shared_ptr<EnvironmentObject> search_env = env;
   for (;;) {
-    auto kv = search_env->vars.find(to_define.as_symbol());
-    if (kv != search_env->vars.end()) {
-      kv->second = to_set;
-      return kv->second;
+    auto kv = search_env->vars.lookup(to_define.as_symbol());
+    if (kv) {
+      search_env->vars.set(to_define.as_symbol(), to_set);
+      return to_set;
     }
 
     auto pe = search_env->parent_env;
@@ -819,6 +903,32 @@ Object Interpreter::eval_quote(const Object& form,
   return args.unnamed.front();
 }
 
+Object build_list_with_spliced_tail(std::vector<Object>&& objects, const Object& tail) {
+  if (objects.empty()) {
+    return tail;
+  }
+
+  std::shared_ptr<PairObject> head = std::make_shared<PairObject>(objects.back(), tail);
+
+  s64 idx = ((s64)objects.size()) - 2;
+  while (idx >= 0) {
+    Object next;
+    next.type = ObjectType::PAIR;
+    next.heap_obj = std::move(head);
+
+    head = std::make_shared<PairObject>();
+    head->car = std::move(objects[idx]);
+    head->cdr = std::move(next);
+
+    idx--;
+  }
+
+  Object result;
+  result.type = ObjectType::PAIR;
+  result.heap_obj = std::move(head);
+  return result;
+}
+
 /*!
  * Recursive quasi-quote evaluation
  */
@@ -831,7 +941,7 @@ Object Interpreter::quasiquote_helper(const Object& form,
       const Object& item = lst_iter->as_pair()->car;
       if (item.type == ObjectType::PAIR) {
         if (item.as_pair()->car.type == ObjectType::SYMBOL &&
-            item.as_pair()->car.as_symbol()->name == "unquote") {
+            item.as_pair()->car.as_symbol() == "unquote") {
           const Object& unquote_arg = item.as_pair()->cdr;
           if (unquote_arg.type != ObjectType::PAIR ||
               unquote_arg.as_pair()->cdr.type != ObjectType::EMPTY_LIST) {
@@ -841,7 +951,7 @@ Object Interpreter::quasiquote_helper(const Object& form,
           lst_iter = &lst_iter->as_pair()->cdr;
           continue;
         } else if (item.as_pair()->car.type == ObjectType::SYMBOL &&
-                   item.as_pair()->car.as_symbol()->name == "unquote-splicing") {
+                   item.as_pair()->car.as_symbol() == "unquote-splicing") {
           const Object& unquote_arg = item.as_pair()->cdr;
           if (unquote_arg.type != ObjectType::PAIR ||
               unquote_arg.as_pair()->cdr.type != ObjectType::EMPTY_LIST) {
@@ -851,6 +961,11 @@ Object Interpreter::quasiquote_helper(const Object& form,
           // bypass normal addition:
           lst_iter = &lst_iter->as_pair()->cdr;
           Object splice_result = eval_with_rewind(unquote_arg.as_pair()->car, env);
+          if (lst_iter->type == ObjectType::EMPTY_LIST) {
+            // optimization!
+            return build_list_with_spliced_tail(std::move(result), splice_result);
+          }
+
           const Object* to_add = &splice_result;
           for (;;) {
             if (to_add->type == ObjectType::PAIR) {
@@ -864,8 +979,13 @@ Object Interpreter::quasiquote_helper(const Object& form,
           }
           continue;
         } else {
-          result.push_back(quasiquote_helper(item, env));
           lst_iter = &lst_iter->as_pair()->cdr;
+
+          if (item.is_pair()) {
+            result.push_back(quasiquote_helper(item, env));
+          } else {
+            result.push_back(item);
+          }
           continue;
         }
       }
@@ -891,7 +1011,7 @@ Object Interpreter::eval_quasiquote(const Object& form,
 }
 
 bool Interpreter::truthy(const Object& o) {
-  return !(o.is_symbol() && o.as_symbol()->name == "#f");
+  return !(o.is_symbol() && o.as_symbol().name_ptr == m_false_sym);
 }
 
 /*!
@@ -924,7 +1044,7 @@ Object Interpreter::eval_cond(const Object& form,
         lst = lst.as_pair()->cdr;
       }
     } else if (lst.type == ObjectType::EMPTY_LIST) {
-      return SymbolObject::make_new(reader.symbolTable, "#f");
+      return m_false_object;
     } else {
       throw_eval_error(form, "malformed cond");
     }
@@ -950,7 +1070,7 @@ Object Interpreter::eval_or(const Object& form,
       }
       lst = lst.as_pair()->cdr;
     } else if (lst.type == ObjectType::EMPTY_LIST) {
-      return SymbolObject::make_new(reader.symbolTable, "#f");
+      return m_false_object;
     } else {
       throw_eval_error(form, "invalid or form");
     }
@@ -973,7 +1093,7 @@ Object Interpreter::eval_and(const Object& form,
     if (lst.type == ObjectType::PAIR) {
       current = eval_with_rewind(lst.as_pair()->car, env);
       if (!truthy(current)) {
-        return SymbolObject::make_new(reader.symbolTable, "#f");
+        return m_false_object;
       }
       lst = lst.as_pair()->cdr;
     } else if (lst.type == ObjectType::EMPTY_LIST) {
@@ -1000,7 +1120,7 @@ Object Interpreter::eval_while(const Object& form,
     throw_eval_error(form, "while must have condition and body");
   }
 
-  Object rv = SymbolObject::make_new(reader.symbolTable, "#f");
+  Object rv = m_false_object;
   while (truthy(eval_with_rewind(condition, env))) {
     rv = eval_list_return_last(form, body, env);
   }
@@ -1126,7 +1246,7 @@ Object Interpreter::eval_try_load_file(const Object& form,
 
   auto path = {args.unnamed.at(0).as_string()->data};
   if (!fs::exists(file_util::get_file_path(path))) {
-    return SymbolObject::make_new(reader.symbolTable, "#f");
+    return m_false_object;
   }
 
   Object o;
@@ -1141,7 +1261,7 @@ Object Interpreter::eval_try_load_file(const Object& form,
   } catch (std::runtime_error& e) {
     throw_eval_error(form, std::string("eval error inside of try-load-file:\n") + e.what());
   }
-  return SymbolObject::make_new(reader.symbolTable, "#t");
+  return m_true_object;
 }
 
 /*!
@@ -1177,6 +1297,14 @@ Object Interpreter::eval_inspect(const Object& form,
   return Object::make_empty_list();
 }
 
+const Object& Interpreter::true_or_false(bool val) {
+  if (val) {
+    return m_true_object;
+  } else {
+    return m_false_object;
+  }
+}
+
 /*!
  * Fancy equality check (using Object::operator==)
  */
@@ -1185,8 +1313,7 @@ Object Interpreter::eval_equals(const Object& form,
                                 const std::shared_ptr<EnvironmentObject>& env) {
   (void)env;
   vararg_check(form, args, {{}, {}}, {});
-  return SymbolObject::make_new(reader.symbolTable,
-                                args.unnamed[0] == args.unnamed[1] ? "#t" : "#f");
+  return true_or_false(args.unnamed[0] == args.unnamed[1]);
 }
 
 /*!
@@ -1430,7 +1557,7 @@ Object Interpreter::eval_numequals(const Object& form,
       return Object::make_empty_list();
   }
 
-  return SymbolObject::make_new(reader.symbolTable, result ? "#t" : "#f");
+  return true_or_false(result);
 }
 
 template <typename T>
@@ -1441,7 +1568,7 @@ Object Interpreter::num_lt(const Object& form,
   (void)env;
   T a = number<T>(args.unnamed[0]);
   T b = number<T>(args.unnamed[1]);
-  return SymbolObject::make_new(reader.symbolTable, (a < b) ? "#t" : "#f");
+  return true_or_false(a < b);
 }
 
 Object Interpreter::eval_lt(const Object& form,
@@ -1469,7 +1596,7 @@ Object Interpreter::num_gt(const Object& form,
   (void)env;
   T a = number<T>(args.unnamed[0]);
   T b = number<T>(args.unnamed[1]);
-  return SymbolObject::make_new(reader.symbolTable, (a > b) ? "#t" : "#f");
+  return true_or_false(a > b);
 }
 
 Object Interpreter::eval_gt(const Object& form,
@@ -1497,7 +1624,7 @@ Object Interpreter::num_leq(const Object& form,
   (void)env;
   T a = number<T>(args.unnamed[0]);
   T b = number<T>(args.unnamed[1]);
-  return SymbolObject::make_new(reader.symbolTable, (a <= b) ? "#t" : "#f");
+  return true_or_false(a <= b);
 }
 
 Object Interpreter::eval_leq(const Object& form,
@@ -1525,7 +1652,7 @@ Object Interpreter::num_geq(const Object& form,
   (void)env;
   T a = number<T>(args.unnamed[0]);
   T b = number<T>(args.unnamed[1]);
-  return SymbolObject::make_new(reader.symbolTable, (a >= b) ? "#t" : "#f");
+  return true_or_false(a >= b);
 }
 
 Object Interpreter::eval_geq(const Object& form,
@@ -1591,7 +1718,7 @@ Object Interpreter::eval_gensym(const Object& form,
                                 const std::shared_ptr<EnvironmentObject>& env) {
   (void)env;
   vararg_check(form, args, {}, {});
-  return SymbolObject::make_new(reader.symbolTable, "gensym" + std::to_string(gensym_id++));
+  return Object::make_symbol(&reader.symbolTable, ("gensym" + std::to_string(gensym_id++)).c_str());
 }
 
 Object Interpreter::eval_cons(const Object& form,
@@ -1607,7 +1734,7 @@ Object Interpreter::eval_null(const Object& form,
                               const std::shared_ptr<EnvironmentObject>& env) {
   (void)env;
   vararg_check(form, args, {{}}, {});
-  return SymbolObject::make_new(reader.symbolTable, args.unnamed[0].is_empty_list() ? "#t" : "#f");
+  return true_or_false(args.unnamed[0].is_empty_list());
 }
 
 Object Interpreter::eval_type(const Object& form,
@@ -1616,15 +1743,15 @@ Object Interpreter::eval_type(const Object& form,
   (void)env;
   vararg_check(form, args, {{ObjectType::SYMBOL}, {}}, {});
 
-  auto kv = string_to_type.find(args.unnamed[0].as_symbol()->name);
+  auto kv = string_to_type.find(args.unnamed[0].as_symbol().name_ptr);
   if (kv == string_to_type.end()) {
     throw_eval_error(form, "invalid type given to type?");
   }
 
   if (args.unnamed[1].type == kv->second) {
-    return SymbolObject::make_new(reader.symbolTable, "#t");
+    return m_true_object;
   } else {
-    return SymbolObject::make_new(reader.symbolTable, "#f");
+    return m_false_object;
   }
 }
 
@@ -1663,7 +1790,7 @@ Object Interpreter::eval_format(const Object& form,
                    fmt::format_args(args2.data(), static_cast<unsigned>(args2.size())));
 
   if (truthy(dest)) {
-    lg::print(formatted.c_str());
+    lg::print("{}", formatted.c_str());
   }
 
   return StringObject::make_new(formatted);
@@ -1729,9 +1856,9 @@ Object Interpreter::eval_string_starts_with(const Object& form,
   auto& suffix = args.unnamed.at(1).as_string()->data;
 
   if (str_util::starts_with(str, suffix)) {
-    return SymbolObject::make_new(reader.symbolTable, "#t");
+    return m_true_object;
   }
-  return SymbolObject::make_new(reader.symbolTable, "#f");
+  return m_false_object;
 }
 
 Object Interpreter::eval_string_ends_with(const Object& form,
@@ -1743,9 +1870,9 @@ Object Interpreter::eval_string_ends_with(const Object& form,
   auto& suffix = args.unnamed.at(1).as_string()->data;
 
   if (str_util::ends_with(str, suffix)) {
-    return SymbolObject::make_new(reader.symbolTable, "#t");
+    return m_true_object;
   }
-  return SymbolObject::make_new(reader.symbolTable, "#f");
+  return m_false_object;
 }
 
 Object Interpreter::eval_string_split(const Object& form,
@@ -1792,14 +1919,21 @@ Object Interpreter::eval_symbol_to_string(const Object& form,
                                           Arguments& args,
                                           const std::shared_ptr<EnvironmentObject>&) {
   vararg_check(form, args, {ObjectType::SYMBOL}, {});
-  return StringObject::make_new(args.unnamed.at(0).as_symbol()->name);
+  return StringObject::make_new(args.unnamed.at(0).as_symbol().name_ptr);
 }
 
 Object Interpreter::eval_string_to_symbol(const Object& form,
                                           Arguments& args,
                                           const std::shared_ptr<EnvironmentObject>&) {
   vararg_check(form, args, {ObjectType::STRING}, {});
-  return SymbolObject::make_new(reader.symbolTable, args.unnamed.at(0).as_string()->data);
+  return Object::make_symbol(&reader.symbolTable, args.unnamed.at(0).as_string()->data.c_str());
+}
+
+Object Interpreter::eval_int_to_string(const Object& form,
+                                       Arguments& args,
+                                       const std::shared_ptr<EnvironmentObject>&) {
+  vararg_check(form, args, {ObjectType::INTEGER}, {});
+  return StringObject::make_new(std::to_string(args.unnamed.at(0).as_int()));
 }
 
 Object Interpreter::eval_get_env(const Object& form,
@@ -1839,7 +1973,7 @@ Object Interpreter::eval_hash_table_set(const Object& form,
   vararg_check(form, args, {ObjectType::STRING_HASH_TABLE, {}, {}}, {});
   const char* str = nullptr;
   if (args.unnamed.at(1).is_symbol()) {
-    str = args.unnamed.at(1).as_symbol()->name.c_str();
+    str = args.unnamed.at(1).as_symbol().name_ptr;
   } else if (args.unnamed.at(1).is_string()) {
     str = args.unnamed.at(1).as_string()->data.c_str();
   } else {
@@ -1861,7 +1995,7 @@ Object Interpreter::eval_hash_table_try_ref(const Object& form,
 
   const char* str = nullptr;
   if (args.unnamed.at(1).is_symbol()) {
-    str = args.unnamed.at(1).as_symbol()->name.c_str();
+    str = args.unnamed.at(1).as_symbol().name_ptr;
   } else if (args.unnamed.at(1).is_string()) {
     str = args.unnamed.at(1).as_string()->data.c_str();
   } else {
@@ -1870,10 +2004,9 @@ Object Interpreter::eval_hash_table_try_ref(const Object& form,
   const auto& it = table->data.find(str);
   if (it == table->data.end()) {
     // not in table
-    return PairObject::make_new(SymbolObject::make_new(reader.symbolTable, "#f"),
-                                Object::make_empty_list());
+    return PairObject::make_new(m_false_object, Object::make_empty_list());
   } else {
-    return PairObject::make_new(SymbolObject::make_new(reader.symbolTable, "#t"), it->second);
+    return PairObject::make_new(m_true_object, it->second);
   }
 }
 }  // namespace goos

common/goos/Interpreter.h

@@ -25,7 +25,7 @@ class Interpreter {
   void set_global_variable_to_int(const std::string& name, int value);
   Object eval(Object obj, const std::shared_ptr<EnvironmentObject>& env);
   Object intern(const std::string& name);
-  HeapObject* intern_ptr(const std::string& name);
+  InternedSymbolPtr intern_ptr(const std::string& name);
   void disable_printfs();
   Object eval_symbol(const Object& sym, const std::shared_ptr<EnvironmentObject>& env);
   bool eval_symbol(const Object& sym,
@@ -55,7 +55,7 @@ class Interpreter {
  private:
   friend class Goal;
   void load_goos_library();
-  void define_var_in_env(Object& env, Object& var, const std::string& name);
+  void define_var_in_env(Object& env, const Object& var, const std::string& name);
   void expect_env(const Object& form, const Object& o);
   void vararg_check(
       const Object& form,
@@ -226,6 +226,9 @@ class Interpreter {
   Object eval_string_to_symbol(const Object& form,
                                Arguments& args,
                                const std::shared_ptr<EnvironmentObject>& env);
+  Object eval_int_to_string(const Object& form,
+                            Arguments& args,
+                            const std::shared_ptr<EnvironmentObject>& env);
   Object eval_get_env(const Object& form,
                       Arguments& args,
                       const std::shared_ptr<EnvironmentObject>& env);
@@ -261,6 +264,16 @@ class Interpreter {
   Object eval_while(const Object& form,
                     const Object& rest,
                     const std::shared_ptr<EnvironmentObject>& env);
+  Object eval_let(const Object& form,
+                  const Object& rest,
+                  const std::shared_ptr<EnvironmentObject>& env);
+  Object eval_let_star(const Object& form,
+                       const Object& rest,
+                       const std::shared_ptr<EnvironmentObject>& env);
+  Object eval_let_common(const Object& form,
+                         const Object& rest,
+                         const std::shared_ptr<EnvironmentObject>& env,
+                         bool star);
 
   Object eval_make_string_hash_table(const Object& form,
                                      Arguments& args,
@@ -272,26 +285,46 @@ class Interpreter {
                              Arguments& args,
                              const std::shared_ptr<EnvironmentObject>& env);
 
+  const Object& true_or_false(bool val);
+
+  void init_special_forms(
+      const std::unordered_map<std::string,
+                               Object (Interpreter::*)(const Object&,
+                                                       const Object&,
+                                                       const std::shared_ptr<EnvironmentObject>&)>&
+          forms);
+
+  void init_builtin_forms(
+      const std::unordered_map<std::string,
+                               Object (Interpreter::*)(const Object&,
+                                                       Arguments&,
+                                                       const std::shared_ptr<EnvironmentObject>&)>&
+          forms);
+
   bool want_exit = false;
   bool disable_printing = false;
 
-  std::unordered_map<std::string,
-                     Object (Interpreter::*)(const Object& form,
-                                             Arguments& args,
-                                             const std::shared_ptr<EnvironmentObject>& env)>
+  std::unordered_map<
+      void*,
+      Object (Interpreter::*)(const Object&, Arguments&, const std::shared_ptr<EnvironmentObject>&)>
       builtin_forms;
 
-  std::unordered_map<
-      std::string,
-      std::function<Object(const Object&, Arguments&, const std::shared_ptr<EnvironmentObject>&)>>
+  std::vector<std::pair<
+      void*,
+      std::function<Object(const Object&, Arguments&, const std::shared_ptr<EnvironmentObject>&)>>>
       m_custom_forms;
-  std::unordered_map<std::string,
-                     Object (Interpreter::*)(const Object& form,
-                                             const Object& rest,
-                                             const std::shared_ptr<EnvironmentObject>& env)>
+
+  std::vector<std::pair<void*,
+                        Object (Interpreter::*)(const Object& form,
+                                                const Object& rest,
+                                                const std::shared_ptr<EnvironmentObject>& env)>>
       special_forms;
   int64_t gensym_id = 0;
 
   std::unordered_map<std::string, ObjectType> string_to_type;
+
+  const char* m_false_sym = nullptr;
+  const char* m_true_sym = nullptr;
+  Object m_false_object, m_true_object;
 };
 }  // namespace goos

common/goos/Object.cpp

@@ -44,12 +44,89 @@
 #include <cstring>
 
 #include "common/util/FileUtil.h"
+#include "common/util/crc32.h"
 #include "common/util/print_float.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 
+SymbolTable::SymbolTable() {
+  m_power_of_two_size = 1;  // 2 ^ 1 = 2
+  m_entries.resize(2);
+  m_used_entries = 0;
+  m_next_resize = (m_entries.size() * kMaxUsed);
+  m_mask = 0b1;
+}
+
+SymbolTable::~SymbolTable() {
+  for (auto& e : m_entries) {
+    delete[] e.name;
+  }
+}
+
+InternedSymbolPtr SymbolTable::intern(const char* str) {
+  size_t string_len = strlen(str);
+  u32 hash = crc32((const u8*)str, string_len);
+
+  // probe
+  for (u32 i = 0; i < m_entries.size(); i++) {
+    u32 slot_addr = (hash + i) & m_mask;
+    auto& slot = m_entries[slot_addr];
+    if (!slot.name) {
+      // not found, insert!
+      slot.hash = hash;
+      auto* name = new char[string_len + 1];
+      memcpy(name, str, string_len + 1);
+      slot.name = name;
+      m_used_entries++;
+
+      if (m_used_entries >= m_next_resize) {
+        resize();
+        return intern(str);
+      }
+      return {name};
+    } else {
+      if (slot.hash != hash) {
+        continue;  // bad hash
+      }
+      if (strcmp(slot.name, str) != 0) {
+        continue;  // bad name
+      }
+      return {slot.name};
+    }
+  }
+
+  // should be impossible to reach.
+  ASSERT_NOT_REACHED();
+}
+
+void SymbolTable::resize() {
+  m_power_of_two_size++;
+  m_mask = (1U << m_power_of_two_size) - 1;
+
+  std::vector<Entry> new_entries(m_entries.size() * 2);
+  for (const auto& old_entry : m_entries) {
+    if (old_entry.name) {
+      bool done = false;
+      for (u32 i = 0; i < new_entries.size(); i++) {
+        u32 slot_addr = (old_entry.hash + i) & m_mask;
+        auto& slot = new_entries[slot_addr];
+        if (!slot.name) {
+          slot.name = old_entry.name;
+          slot.hash = old_entry.hash;
+          done = true;
+          break;
+        }
+      }
+      ASSERT(done);
+    }
+  }
+
+  m_entries = std::move(new_entries);
+  m_next_resize = kMaxUsed * m_entries.size();
+}
+
 /*!
  * Convert type to string (name in brackets)
  */
@@ -137,14 +214,9 @@ std::string fixed_to_string(char x) {
   return {buff};
 }
 
-/*!
- * Create a new symbol object by interning
- */
-Object SymbolObject::make_new(SymbolTable& st, const std::string& name) {
-  Object obj;
-  obj.type = ObjectType::SYMBOL;
-  obj.heap_obj = st.intern(name);
-  return obj;
+template <>
+std::string fixed_to_string(InternedSymbolPtr x) {
+  return x.name_ptr;
 }
 
 /*!
@@ -238,8 +310,9 @@ bool Object::operator==(const Object& other) const {
       return float_obj == other.float_obj;
     case ObjectType::CHAR:
       return char_obj == other.char_obj;
-
     case ObjectType::SYMBOL:
+      return symbol_obj == other.symbol_obj;
+
     case ObjectType::ENVIRONMENT:
     case ObjectType::LAMBDA:
     case ObjectType::MACRO:
@@ -271,7 +344,7 @@ bool Object::operator==(const Object& other) const {
 }
 
 bool Object::is_symbol(const std::string& name) const {
-  return is_symbol() && as_symbol()->name == name;
+  return is_symbol() && name == as_symbol().name_ptr;
 }
 
 bool Object::is_string(const std::string& val) const {

common/goos/Object.h

@@ -43,6 +43,7 @@
  *
  */
 
+#include <cstring>
 #include <map>
 #include <memory>
 #include <stdexcept>
@@ -54,12 +55,10 @@
 
 #include "common/common_types.h"
 #include "common/util/Assert.h"
+#include "common/util/crc32.h"
 
 namespace goos {
 
-using FloatType = double;
-using IntType = s64;
-
 /*!
  * All objects have one of these kinds
  */
@@ -71,9 +70,9 @@ enum class ObjectType : u8 {
   INTEGER,
   FLOAT,
   CHAR,
+  SYMBOL,
 
   // allocated
-  SYMBOL,
   STRING,
   PAIR,
   ARRAY,
@@ -87,6 +86,29 @@ enum class ObjectType : u8 {
 std::string object_type_to_string(ObjectType kind);
 
 // Some objects are "fixed", meaning they are stored inline in the Object, and not heap allocated.
+// These use value semantics and are copied on object assignment.
+
+// For int/float, these are simply double/int64_t
+using FloatType = double;
+using IntType = s64;
+
+// For symbols, we simply store a pointer to the symbol's name string.
+// this is wrapped in a type just to avoid confusion with normal const char*.
+struct InternedSymbolPtr {
+  const char* name_ptr;
+
+  struct hash {
+    auto operator()(const InternedSymbolPtr& x) const {
+      return std::hash<const void*>()((const void*)x.name_ptr);
+    }
+  };
+  bool operator==(const char* msg) const { return strcmp(msg, name_ptr) == 0; }
+  bool operator!=(const char* msg) const { return strcmp(msg, name_ptr) != 0; }
+  bool operator==(const std::string& str) const { return str == name_ptr; }
+  bool operator!=(const std::string& str) const { return str != name_ptr; }
+  bool operator==(const InternedSymbolPtr& other) const { return other.name_ptr == name_ptr; }
+  bool operator!=(const InternedSymbolPtr& other) const { return other.name_ptr != name_ptr; }
+};
 
 /*!
  * By default, convert a fixed object data to string with std::to_string.
@@ -115,6 +137,31 @@ std::string fixed_to_string<char>(char);
 template <>
 std::string fixed_to_string<IntType>(IntType);
 
+template <>
+std::string fixed_to_string<InternedSymbolPtr>(InternedSymbolPtr);
+
+class SymbolTable {
+ public:
+  SymbolTable(const SymbolTable&) = delete;
+  SymbolTable& operator=(const SymbolTable&) = delete;
+  SymbolTable();
+  ~SymbolTable();
+  InternedSymbolPtr intern(const char* str);
+
+ private:
+  void resize();
+  int m_power_of_two_size = 0;
+  struct Entry {
+    u32 hash = 0;
+    const char* name = nullptr;
+  };
+  std::vector<Entry> m_entries;
+  int m_used_entries = 0;
+  int m_next_resize = 0;
+  u32 m_mask = 0;
+  static constexpr float kMaxUsed = 0.7;
+};
+
 /*!
  * Common implementation for a fixed object
  */
@@ -139,6 +186,8 @@ class FixedObject {
       return object_type_to_string(ObjectType::INTEGER);
     if (std::is_same<T, char>())
       return object_type_to_string(ObjectType::CHAR);
+    if (std::is_same<T, InternedSymbolPtr>())
+      return object_type_to_string(ObjectType::SYMBOL);
     throw std::runtime_error("Unsupported FixedObject type");
   }
 };
@@ -149,6 +198,7 @@ class FixedObject {
 using IntegerObject = FixedObject<int64_t>;
 using FloatObject = FixedObject<double>;
 using CharObject = FixedObject<char>;
+using SymbolObject = FixedObject<InternedSymbolPtr>;
 
 // Other objects are separate allocated on the heap. These objects should be HeapObjects.
 
@@ -162,7 +212,6 @@ class HeapObject {
 // forward declare all HeapObjects
 class PairObject;
 class EnvironmentObject;
-class SymbolObject;
 class StringObject;
 class LambdaObject;
 class MacroObject;
@@ -180,6 +229,7 @@ class Object {
     IntegerObject integer_obj;
     FloatObject float_obj;
     CharObject char_obj;
+    SymbolObject symbol_obj;
   };
 
   ObjectType type = ObjectType::INVALID;
@@ -192,6 +242,8 @@ class Object {
         return float_obj.print();
       case ObjectType::CHAR:
         return char_obj.print();
+      case ObjectType::SYMBOL:
+        return symbol_obj.print();
       case ObjectType::EMPTY_LIST:
         return "()";
       default:
@@ -207,6 +259,8 @@ class Object {
         return float_obj.inspect();
       case ObjectType::CHAR:
         return char_obj.inspect();
+      case ObjectType::SYMBOL:
+        return symbol_obj.inspect();
       case ObjectType::EMPTY_LIST:
         return "[empty list] ()\n";
       default:
@@ -244,16 +298,30 @@ class Object {
     return o;
   }
 
+  static Object make_symbol(SymbolTable* table, const char* name) {
+    Object o;
+    o.type = ObjectType::SYMBOL;
+    o.symbol_obj.value = table->intern(name);
+    return o;
+  }
+
   PairObject* as_pair() const;
   EnvironmentObject* as_env() const;
   std::shared_ptr<EnvironmentObject> as_env_ptr() const;
-  SymbolObject* as_symbol() const;
   StringObject* as_string() const;
   LambdaObject* as_lambda() const;
   MacroObject* as_macro() const;
   ArrayObject* as_array() const;
   StringHashTableObject* as_string_hash_table() const;
 
+  const InternedSymbolPtr& as_symbol() const {
+    if (type != ObjectType::SYMBOL) {
+      throw std::runtime_error("as_symbol called on a " + object_type_to_string(type) + " " +
+                               print());
+    }
+    return symbol_obj.value;
+  }
+
   IntType& as_int() {
     if (type != ObjectType::INTEGER) {
       throw std::runtime_error("as_int called on a " + object_type_to_string(type) + " " + print());
@@ -315,56 +383,6 @@ class Object {
   bool operator!=(const Object& other) const { return !((*this) == other); }
 };
 
-class SymbolTable;
-
-/*!
- * A Symbol Object, which is just a wrapper around a string.
- * The make_new function will correctly
- */
-class SymbolObject : public HeapObject {
- public:
-  std::string name;
-  explicit SymbolObject(std::string _name) : name(std::move(_name)) {}
-  static Object make_new(SymbolTable& st, const std::string& name);
-
-  std::string print() const override { return name; }
-
-  std::string inspect() const override { return "[symbol] " + name + "\n"; }
-
-  ~SymbolObject() = default;
-};
-
-/*!
- * A Symbol Table, which holds all symbols.
- */
-class SymbolTable {
- public:
-  std::shared_ptr<HeapObject> intern(const std::string& name) {
-    const auto& kv = table.find(name);
-    if (kv == table.end()) {
-      auto iter = table.insert({name, std::make_shared<SymbolObject>(name)});
-      return (*iter.first).second;
-    } else {
-      return kv->second;
-    }
-  }
-
-  HeapObject* intern_ptr(const std::string& name) {
-    const auto& kv = table.find(name);
-    if (kv == table.end()) {
-      auto iter = table.insert({name, std::make_shared<SymbolObject>(name)});
-      return (*iter.first).second.get();
-    } else {
-      return kv->second.get();
-    }
-  }
-
-  ~SymbolTable() = default;
-
- private:
-  std::unordered_map<std::string, std::shared_ptr<HeapObject>> table;
-};
-
 class StringObject : public HeapObject {
  public:
   std::string data;
@@ -439,14 +457,131 @@ class PairObject : public HeapObject {
   ~PairObject() = default;
 };
 
+template <typename T>
+class InternedPtrMap {
+ public:
+  InternedPtrMap(const InternedPtrMap&) = delete;
+  InternedPtrMap& operator=(const InternedPtrMap&) = delete;
+  InternedPtrMap() { clear(); }
+
+  T* lookup(InternedSymbolPtr str) {
+    if (m_entries.size() < 10) {
+      for (auto& e : m_entries) {
+        if (e.key == str.name_ptr) {
+          return &e.value;
+        }
+      }
+      return nullptr;
+    }
+    u32 hash = crc32((const u8*)&str.name_ptr, sizeof(const char*));
+
+    // probe
+    for (u32 i = 0; i < m_entries.size(); i++) {
+      u32 slot_addr = (hash + i) & m_mask;
+      auto& slot = m_entries[slot_addr];
+      if (!slot.key) {
+        return nullptr;
+      } else {
+        if (slot.key != str.name_ptr) {
+          continue;  // bad hash
+        }
+        return &slot.value;
+      }
+    }
+
+    // should be impossible to reach.
+    ASSERT_NOT_REACHED();
+  }
+  void set(InternedSymbolPtr ptr, const T& obj) {
+    u32 hash = crc32((const u8*)&ptr.name_ptr, sizeof(const char*));
+
+    // probe
+    for (u32 i = 0; i < m_entries.size(); i++) {
+      u32 slot_addr = (hash + i) & m_mask;
+      auto& slot = m_entries[slot_addr];
+      if (!slot.key) {
+        // not found, insert!
+        slot.key = ptr.name_ptr;
+        slot.value = obj;
+        m_used_entries++;
+
+        if (m_used_entries >= m_next_resize) {
+          resize();
+        }
+        return;
+      } else {
+        if (slot.key == ptr.name_ptr) {
+          slot.value = obj;
+          return;
+        }
+      }
+    }
+
+    // should be impossible to reach.
+    ASSERT_NOT_REACHED();
+  }
+  void clear() {
+    m_entries.clear();
+    m_power_of_two_size = 3;  // 2 ^ 3 = 8
+    m_entries.resize(8);
+    m_used_entries = 0;
+    m_next_resize = (m_entries.size() * kMaxUsed);
+    m_mask = 0b111;
+  }
+
+ private:
+  struct Entry {
+    const char* key = nullptr;
+    T value;
+  };
+  std::vector<Entry> m_entries;
+
+  void resize() {
+    m_power_of_two_size++;
+    m_mask = (1U << m_power_of_two_size) - 1;
+
+    std::vector<Entry> new_entries(m_entries.size() * 2);
+    for (const auto& old_entry : m_entries) {
+      if (old_entry.key) {
+        bool done = false;
+        u32 hash = crc32((const u8*)&old_entry.key, sizeof(const char*));
+        for (u32 i = 0; i < new_entries.size(); i++) {
+          u32 slot_addr = (hash + i) & m_mask;
+          auto& slot = new_entries[slot_addr];
+          if (!slot.key) {
+            slot.key = old_entry.key;
+            slot.value = std::move(old_entry.value);
+            done = true;
+            break;
+          }
+        }
+        ASSERT(done);
+      }
+    }
+
+    m_entries = std::move(new_entries);
+    m_next_resize = kMaxUsed * m_entries.size();
+  }
+  int m_power_of_two_size = 0;
+  int m_used_entries = 0;
+  int m_next_resize = 0;
+  u32 m_mask = 0;
+  static constexpr float kMaxUsed = 0.7;
+};
+
+using EnvironmentMap = InternedPtrMap<Object>;
+
 class EnvironmentObject : public HeapObject {
  public:
   std::string name;
   std::shared_ptr<EnvironmentObject> parent_env;
 
-  // the symbols will be stored in the symbol table and never removed, so we don't need shared
-  // pointers here.
-  std::unordered_map<HeapObject*, Object> vars;
+  // note: this is keyed on the address in the symbol table.
+  EnvironmentMap vars;
+
+  // this find work by any name string
+  Object* find(const char* n, SymbolTable* st) { return vars.lookup(st->intern(n)); }
+  Object* find(InternedSymbolPtr ptr) { return vars.lookup(ptr); }
 
   EnvironmentObject() = default;
 
@@ -478,11 +613,8 @@ class EnvironmentObject : public HeapObject {
 
   std::string inspect() const override {
     std::string result = "[environment]\n  name: " + name +
-                         "\n  parent: " + (parent_env ? parent_env->print() : "NONE") +
-                         "\n  vars:\n";
-    for (const auto& kv : vars) {
-      result += "    " + kv.first->print() + ": " + kv.second.print() + "\n";
-    }
+                         "\n  parent: " + (parent_env ? parent_env->print() : "NONE") + "\n";
+
     return result;
   }
 };
@@ -662,14 +794,6 @@ inline std::shared_ptr<EnvironmentObject> Object::as_env_ptr() const {
   return std::dynamic_pointer_cast<EnvironmentObject>(heap_obj);
 }
 
-inline SymbolObject* Object::as_symbol() const {
-  if (type != ObjectType::SYMBOL) {
-    throw std::runtime_error("as_symbol called on a " + object_type_to_string(type) + " " +
-                             print());
-  }
-  return static_cast<SymbolObject*>(heap_obj.get());
-}
-
 inline StringObject* Object::as_string() const {
   if (type != ObjectType::STRING) {
     throw std::runtime_error("as_string called on a " + object_type_to_string(type) + " " +

common/goos/ParseHelpers.cpp

@@ -1,6 +1,6 @@
 #include "ParseHelpers.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 
@@ -12,12 +12,12 @@ bool get_va(const goos::Object& rest, std::string* err_string, goos::Arguments*
     auto arg = current.as_pair()->car;
 
     // did we get a ":keyword"
-    if (arg.is_symbol() && arg.as_symbol()->name.at(0) == ':') {
-      auto key_name = arg.as_symbol()->name.substr(1);
+    if (arg.is_symbol() && arg.as_symbol().name_ptr[0] == ':') {
+      auto key_name = arg.as_symbol().name_ptr + 1;
 
       // check for multiple definition of key
       if (args.named.find(key_name) != args.named.end()) {
-        *err_string = "Key argument " + key_name + " multiply defined";
+        *err_string = fmt::format("Key argument {} multiply defined", key_name);
         return false;
       }
 

common/goos/PrettyPrinter.cpp

@@ -15,7 +15,7 @@
 #include "common/goos/PrettyPrinter2.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace pretty_print {
 
@@ -98,7 +98,7 @@ void add_to_token_list(const goos::Object& obj, std::vector<FormToken>* tokens)
       // look for a (quote x) to print as 'x
 
       auto& first = obj.as_pair()->car;
-      if (first.is_symbol() && first.as_symbol()->name == "quote") {
+      if (first.is_symbol() && first.as_symbol() == "quote") {
         auto& second = obj.as_pair()->cdr;
         if (second.is_pair() && second.as_pair()->cdr.is_empty_list()) {
           tokens->emplace_back(FormToken::TokenKind::QUOTE);

common/goos/PrettyPrinter2.cpp

@@ -3,8 +3,8 @@
 #include "common/common_types.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
-#include "third-party/fmt/format.h"
+#include "fmt/core.h"
+#include "fmt/format.h"
 
 namespace pretty_print {
 
@@ -288,7 +288,11 @@ void break_list(Node* node) {
       node->sub_elt_indent += name.size();
     } else if (name == "defmethod") {
       // things with 4 things in the top line: (defmethod <method> <type> <args>
-      node->top_line_count = 4;
+      // or just 3 things in the top line: (defmethod <method> <args>
+      node->top_line_count = 3;
+      if (node->child_nodes.size() >= 4 && node->child_nodes[2].kind == Node::Kind::ATOM) {
+        node->top_line_count = 4;
+      }
     } else if (name == "until" || name == "while" || name == "dotimes" || name == "countdown" ||
                name == "when" || name == "behavior" || name == "lambda" || name == "defpart" ||
                name == "define") {

common/goos/Printer.cpp

@@ -6,7 +6,7 @@
 #include "common/goos/Object.h"
 #include "common/util/print_float.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace pretty_print {
 namespace {
@@ -49,7 +49,7 @@ goos::Reader& get_pretty_printer_reader() {
 
 goos::Object to_symbol(const std::string& str) {
   std::lock_guard<std::mutex> guard(pretty_printer_reader_mutex);
-  return goos::SymbolObject::make_new(get_pretty_printer_reader().symbolTable, str);
+  return goos::Object::make_symbol(&get_pretty_printer_reader().symbolTable, str.c_str());
 }
 
 goos::Object new_string(const std::string& str) {

common/goos/Reader.cpp

@@ -12,11 +12,10 @@
 #include "Reader.h"
 
 #include "common/log/log.h"
-#include "common/repl/util.h"
 #include "common/util/FileUtil.h"
 #include "common/util/FontUtils.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 
@@ -238,9 +237,14 @@ Object Reader::read_from_string(const std::string& str,
  * Read a file
  */
 Object Reader::read_from_file(const std::vector<std::string>& file_path, bool check_encoding) {
-  std::string joined_path = fmt::format("{}", fmt::join(file_path, "/"));
+  std::string file_descriptor = fmt::format("{}", fmt::join(file_path, "/"));
+  const auto joined_file_path = file_util::get_file_path(file_path);
 
-  auto textFrag = std::make_shared<FileText>(file_util::get_file_path(file_path), joined_path);
+  if (!fs::exists(joined_file_path)) {
+    throw std::runtime_error(fmt::format("Cannot read {}, file doesn't exist", joined_file_path));
+  }
+
+  auto textFrag = std::make_shared<FileText>(joined_file_path, file_descriptor);
   db.insert(textFrag);
 
   auto result = internal_read(textFrag, check_encoding);
@@ -286,7 +290,7 @@ Object Reader::internal_read(std::shared_ptr<SourceText> text,
   try {
     auto objs = read_list(ts, false);
     if (add_top_level) {
-      return PairObject::make_new(SymbolObject::make_new(symbolTable, "top-level"), objs);
+      return PairObject::make_new(Object::make_symbol(&symbolTable, "top-level"), objs);
     } else {
       return objs;
     }
@@ -459,12 +463,51 @@ bool Reader::read_array(TextStream& stream, Object& o) {
   return true;
 }
 
+struct ListBuilder {
+  Object head;
+  std::shared_ptr<PairObject> prev_tail;
+  std::shared_ptr<PairObject> tail;
+  int size = 0;
+
+  ListBuilder() { head = Object::make_empty_list(); }
+
+  void push_back(Object&& o) {
+    size++;
+    if (!tail) {
+      tail = std::make_shared<PairObject>(o, Object{});
+      head.type = ObjectType::PAIR;
+      head.heap_obj = tail;
+    } else {
+      auto next = std::make_shared<PairObject>(o, Object{});
+      tail->cdr.type = ObjectType::PAIR;
+      tail->cdr.heap_obj = next;
+      prev_tail = std::move(tail);
+      tail = std::move(next);
+    }
+  }
+
+  Object pop_back() {
+    auto obj = tail->car;
+    tail = std::move(prev_tail);
+    return obj;
+  }
+
+  void finalize() {
+    if (tail) {
+      tail->cdr = Object::make_empty_list();
+    } else {
+      head = Object::make_empty_list();
+    }
+  }
+};
+
 /*!
  * Call this on the character after the open paren.
  */
 Object Reader::read_list(TextStream& ts, bool expect_close_paren) {
   ts.seek_past_whitespace_and_comments();
-  std::vector<Object> objects;
+  // std::vector<Object> objects;
+  ListBuilder list_builder;
 
   bool got_close_paren = false;      // does this list end?
   bool got_dot = false;              // did we get a . ?
@@ -506,22 +549,23 @@ Object Reader::read_list(TextStream& ts, bool expect_close_paren) {
     }
 
     // inserter function, used to properly insert a next object
-    auto insert_object = [&](const Object& o) {
+    auto insert_object = [&](Object&& o) {
       if (got_thing_after_dot) {
         throw_reader_error(ts, "A list cannot have multiple entries after the dot", -1);
       }
 
       if (reader_macro_string_stack.empty()) {
-        objects.push_back(o);
+        list_builder.push_back(std::move(o));
+        // objects.push_back(o);
       } else {
-        Object to_push_back = o;
+        Object to_push_back = std::move(o);
         while (!reader_macro_string_stack.empty()) {
-          to_push_back =
-              build_list({SymbolObject::make_new(symbolTable, reader_macro_string_stack.back()),
-                          to_push_back});
+          to_push_back = build_list(
+              {Object::make_symbol(&symbolTable, reader_macro_string_stack.back().c_str()),
+               to_push_back});
           reader_macro_string_stack.pop_back();
         }
-        objects.push_back(to_push_back);
+        list_builder.push_back(std::move(to_push_back));
       }
 
       // remember if we got an object after the dot
@@ -551,7 +595,7 @@ Object Reader::read_list(TextStream& ts, bool expect_close_paren) {
 
       if (read_object(tok, ts, obj)) {
         ts.seek_past_whitespace_and_comments();
-        insert_object(obj);
+        insert_object(std::move(obj));
       } else {
         throw_reader_error(ts, "invalid token encountered in reader: " + tok.text,
                            -int(tok.text.size()));
@@ -574,12 +618,12 @@ Object Reader::read_list(TextStream& ts, bool expect_close_paren) {
 
   // build up list or improper list, link it, and return!
   if (got_thing_after_dot) {
-    if (objects.size() < 2) {
+    if (list_builder.size < 2) {
       throw_reader_error(ts, "A list with a dot must have at least one thing before the dot", -1);
     }
-    auto back = objects.back();
-    objects.pop_back();
-    auto rv = build_list(objects);
+    auto back = list_builder.pop_back();
+    list_builder.finalize();
+    auto rv = list_builder.head;
 
     auto lst = rv;
     while (true) {
@@ -593,7 +637,8 @@ Object Reader::read_list(TextStream& ts, bool expect_close_paren) {
     db.link(rv, ts.text, start_offset);
     return rv;
   } else {
-    auto rv = build_list(std::move(objects));
+    list_builder.finalize();
+    auto rv = list_builder.head;
     db.link(rv, ts.text, start_offset);
     return rv;
   }
@@ -607,7 +652,7 @@ bool Reader::try_token_as_symbol(const Token& tok, Object& obj) {
   ASSERT(!tok.text.empty());
   char start = tok.text[0];
   if (m_valid_symbols_chars[(int)start]) {
-    obj = SymbolObject::make_new(symbolTable, tok.text);
+    obj = Object::make_symbol(&symbolTable, tok.text.c_str());
     return true;
   } else {
     return false;

common/goos/Reader.h

@@ -18,7 +18,7 @@
 
 #include "common/goos/Object.h"
 #include "common/goos/TextDB.h"
-#include "common/repl/util.h"
+#include "common/repl/repl_wrapper.h"
 #include "common/util/Assert.h"
 
 namespace goos {

common/goos/TextDB.cpp

@@ -15,7 +15,7 @@
 
 #include "common/util/FileUtil.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 
@@ -85,9 +85,9 @@ std::pair<int, int> SourceText::get_containing_line(int offset) {
 /*!
  * Read text from a file.
  */
-FileText::FileText(const std::string& filename, const std::string& description_name)
-    : m_filename(filename), m_desc_name(description_name) {
-  m_text = file_util::read_text_file(m_filename);
+FileText::FileText(const std::string& file_path, const std::string& description_name)
+    : m_filepath(file_path), m_desc_name(description_name) {
+  m_text = file_util::read_text_file(m_filepath);
   build_offsets();
 }
 

common/goos/TextDB.h

@@ -80,13 +80,13 @@ class ProgramString : public SourceText {
  */
 class FileText : public SourceText {
  public:
-  FileText(const std::string& filename, const std::string& description_name);
+  FileText(const std::string& file_path, const std::string& description_name);
 
   std::string get_description() { return m_desc_name; }
   ~FileText() = default;
 
  private:
-  std::string m_filename;
+  std::string m_filepath;
   std::string m_desc_name;
 };
 

common/link_types.h

@@ -54,6 +54,21 @@ struct LinkHeaderV4 {
   uint32_t code_size;  // length of object data before link data starts
 };
 
+struct LinkHeaderV5Core {
+  uint32_t length_to_get_to_code;  // 4 length.. of link data?
+  uint16_t version;                // 8
+  uint16_t unknown;                // 10
+  uint32_t length_to_get_to_link;  // 12
+  uint32_t link_length;            // 16
+  uint8_t n_segments;              // 20
+  char name[59];                   // 21 (really??)
+};
+
+struct LinkHeaderV5 {
+  uint32_t type_tag;  // 0 always 0?
+  LinkHeaderV5Core core;
+};
+
 // when a u32/s32 symbol link contains this value, (s7 + <val>) should be a 4-byte aligned address,
 // not including the 1 byte symbol offset. (no effect in jak 1).
 constexpr u32 LINK_SYM_NO_OFFSET_FLAG = 0xbadbeef;

common/log/log.cpp

@@ -1,10 +1,11 @@
 #include "log.h"
 
+#include <cstdarg>
 #include <cstdio>
 #include <cstdlib>
 #include <mutex>
 
-#include "third-party/fmt/color.h"
+#include "fmt/color.h"
 #ifdef _WIN32  // see lg::initialize
 #define NOMINMAX
 #define WIN32_LEAN_AND_MEAN
@@ -106,7 +107,28 @@ void log_print(const char* message) {
     }
 
     if (gLogger.stdout_log_level < lg::level::off_unless_die) {
-      fmt::print(message);
+      fmt::print("{}", message);
+      fflush(stdout);
+      fflush(stderr);
+    }
+  }
+}
+
+void log_vprintf(const char* format, va_list arg_list) {
+  {
+    // We always immediately flush prints because since it has no associated level
+    // it could be anything from a fatal error to a useless debug log.
+    std::lock_guard<std::mutex> lock(gLogger.mutex);
+    va_list arg_list_2;
+    va_copy(arg_list_2, arg_list);
+    if (gLogger.fp) {
+      // Log to File
+      vfprintf(gLogger.fp, format, arg_list);
+      fflush(gLogger.fp);
+    }
+
+    if (gLogger.stdout_log_level < lg::level::off_unless_die) {
+      vprintf(format, arg_list_2);
       fflush(stdout);
       fflush(stderr);
     }
@@ -114,7 +136,11 @@ void log_print(const char* message) {
 }
 }  // namespace internal
 
-// how many extra log files for a single program should be kept?
+void printstd(const char* format, va_list arg_list) {
+  internal::log_vprintf(format, arg_list);
+}
+
+// how many extra log files for a single program should be kept
 constexpr int LOG_ROTATE_MAX = 10;
 
 void set_file(const std::string& filename,
@@ -137,7 +163,7 @@ void set_file(const std::string& filename,
         file_util::find_files_in_dir(fs::path(complete_filename).parent_path(),
                                      std::regex(fmt::format("{}\\.(\\d\\.)?log", filename)));
     for (const auto& file : old_log_files) {
-      lg::info("removing {}", file.string());
+      lg::debug("removing {}", file.string());
       fs::remove(file);
     }
     // remove the oldest log file if there are more than LOG_ROTATE_MAX
@@ -146,14 +172,16 @@ void set_file(const std::string& filename,
     // sort the names and remove them
     existing_log_files = file_util::sort_filepaths(existing_log_files, true);
     if (existing_log_files.size() > (LOG_ROTATE_MAX - 1)) {
-      lg::info("removing {} log files", existing_log_files.size() - (LOG_ROTATE_MAX - 1));
-      for (int i = 0; i < existing_log_files.size() - (LOG_ROTATE_MAX - 1); i++) {
-        lg::info("removing {}", existing_log_files.at(i).string());
+      lg::debug("removing {} log files", existing_log_files.size() - (LOG_ROTATE_MAX - 1));
+      for (int i = 0; i < (int)existing_log_files.size() - (LOG_ROTATE_MAX - 1); i++) {
+        lg::debug("removing {}", existing_log_files.at(i).string());
         fs::remove(existing_log_files.at(i));
       }
     }
   } else {
-    complete_filename += ".log";
+    if (!str_util::ends_with(complete_filename, ".log")) {
+      complete_filename += ".log";
+    }
   }
 
   if (append) {

common/log/log.h

@@ -7,8 +7,8 @@
 #endif
 #include <string>
 
-#include "third-party/fmt/color.h"
-#include "third-party/fmt/core.h"
+#include "fmt/color.h"
+#include "fmt/core.h"
 
 namespace lg {
 
@@ -38,6 +38,7 @@ namespace internal {
 // log implementation stuff, not to be called by the user
 void log_message(level log_level, LogTime& now, const char* message);
 void log_print(const char* message);
+void log_vprintf(const char* format, va_list arg_list);
 }  // namespace internal
 
 void set_file(const std::string& filename,
@@ -60,13 +61,13 @@ void log(level log_level, const std::string& format, Args&&... args) {
 #else
   now.tim = time(nullptr);
 #endif
-  std::string formatted_message = fmt::format(format, std::forward<Args>(args)...);
+  std::string formatted_message = fmt::format(fmt::runtime(format), std::forward<Args>(args)...);
   internal::log_message(log_level, now, formatted_message.c_str());
 }
 
 template <typename... Args>
 void print(const std::string& format, Args&&... args) {
-  std::string formatted_message = fmt::format(format, std::forward<Args>(args)...);
+  std::string formatted_message = fmt::format(fmt::runtime(format), std::forward<Args>(args)...);
   internal::log_print(formatted_message.c_str());
 }
 template <typename... Args>
@@ -75,6 +76,9 @@ void print(const fmt::text_style& ts, const std::string& format, Args&&... args)
   internal::log_print(formatted_message.c_str());
 }
 
+// same as print but uses the C printf instead of fmt
+void printstd(const char* format, va_list arg_list);
+
 template <typename... Args>
 void trace(const std::string& format, Args&&... args) {
   log(level::trace, format, std::forward<Args>(args)...);

common/math/Vector.h

@@ -2,7 +2,7 @@
 
 #include <cmath>
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace math {
 
@@ -18,6 +18,12 @@ class Vector {
     return result;
   }
 
+  static Vector<T, Size> unit(int idx) {
+    Vector<T, Size> result = Vector<T, Size>::zero();
+    result[idx] = T(1);
+    return result;
+  }
+
   template <typename... Args>
   constexpr Vector(Args... args) : m_data{T(args)...} {
     static_assert(sizeof...(args) == Size, "Incorrect number of args");
@@ -226,6 +232,22 @@ class Vector {
     }
   }
 
+  Vector<T, Size> min(const Vector<T, Size>& other) const {
+    Vector<T, Size> result;
+    for (int i = 0; i < Size; i++) {
+      result[i] = std::min(m_data[i], other[i]);
+    }
+    return result;
+  }
+
+  Vector<T, Size> max(const Vector<T, Size>& other) const {
+    Vector<T, Size> result;
+    for (int i = 0; i < Size; i++) {
+      result[i] = std::max(m_data[i], other[i]);
+    }
+    return result;
+  }
+
   std::string to_string_aligned() const {
     std::string result = "[";
     for (auto x : m_data) {

common/math/geometry.h

@@ -50,4 +50,78 @@ math::Vector4f bsphere_of_triangle(const Vector3f* vertices);
 inline bool point_in_bsphere(const Vector4f& sphere, const Vector3f& pt) {
   return (sphere.xyz() - pt).squared_length() <= (sphere.w() * sphere.w());
 }
+
+template <typename T>
+math::Matrix<T, 4, 4> affine_inverse(const math::Matrix<T, 4, 4>& in) {
+  math::Matrix<T, 4, 4> result;
+
+  // transpose rotation
+  for (int i = 0; i < 3; i++) {
+    for (int j = 0; j < 3; j++) {
+      result(i, j) = in(j, i);
+    }
+  }
+
+  result(3, 0) = 0;
+  result(3, 1) = 0;
+  result(3, 2) = 0;
+  result(3, 3) = 1;
+  result(0, 3) = 0;
+  result(1, 3) = 0;
+  result(2, 3) = 0;
+  for (int rx = 0; rx < 3; rx++) {
+    for (int cx = 0; cx < 3; cx++) {
+      result(rx, 3) -= result(rx, cx) * in(cx, 3);
+    }
+  }
+
+  return result;
+}
+
+template <typename T>
+math::Matrix<T, 4, 4> inverse(const math::Matrix<T, 4, 4>& m) {
+  math::Matrix<T, 4, 4> im;
+  T A2323 = m(2, 2) * m(3, 3) - m(2, 3) * m(3, 2);
+  T A1323 = m(2, 1) * m(3, 3) - m(2, 3) * m(3, 1);
+  T A1223 = m(2, 1) * m(3, 2) - m(2, 2) * m(3, 1);
+  T A0323 = m(2, 0) * m(3, 3) - m(2, 3) * m(3, 0);
+  T A0223 = m(2, 0) * m(3, 2) - m(2, 2) * m(3, 0);
+  T A0123 = m(2, 0) * m(3, 1) - m(2, 1) * m(3, 0);
+  T A2313 = m(1, 2) * m(3, 3) - m(1, 3) * m(3, 2);
+  T A1313 = m(1, 1) * m(3, 3) - m(1, 3) * m(3, 1);
+  T A1213 = m(1, 1) * m(3, 2) - m(1, 2) * m(3, 1);
+  T A2312 = m(1, 2) * m(2, 3) - m(1, 3) * m(2, 2);
+  T A1312 = m(1, 1) * m(2, 3) - m(1, 3) * m(2, 1);
+  T A1212 = m(1, 1) * m(2, 2) - m(1, 2) * m(2, 1);
+  T A0313 = m(1, 0) * m(3, 3) - m(1, 3) * m(3, 0);
+  T A0213 = m(1, 0) * m(3, 2) - m(1, 2) * m(3, 0);
+  T A0312 = m(1, 0) * m(2, 3) - m(1, 3) * m(2, 0);
+  T A0212 = m(1, 0) * m(2, 2) - m(1, 2) * m(2, 0);
+  T A0113 = m(1, 0) * m(3, 1) - m(1, 1) * m(3, 0);
+  T A0112 = m(1, 0) * m(2, 1) - m(1, 1) * m(2, 0);
+
+  T det = m(0, 0) * (m(1, 1) * A2323 - m(1, 2) * A1323 + m(1, 3) * A1223) -
+          m(0, 1) * (m(1, 0) * A2323 - m(1, 2) * A0323 + m(1, 3) * A0223) +
+          m(0, 2) * (m(1, 0) * A1323 - m(1, 1) * A0323 + m(1, 3) * A0123) -
+          m(0, 3) * (m(1, 0) * A1223 - m(1, 1) * A0223 + m(1, 2) * A0123);
+  det = 1 / det;
+
+  im(0, 0) = det * (m(1, 1) * A2323 - m(1, 2) * A1323 + m(1, 3) * A1223);
+  im(0, 1) = det * -(m(0, 1) * A2323 - m(0, 2) * A1323 + m(0, 3) * A1223);
+  im(0, 2) = det * (m(0, 1) * A2313 - m(0, 2) * A1313 + m(0, 3) * A1213);
+  im(0, 3) = det * -(m(0, 1) * A2312 - m(0, 2) * A1312 + m(0, 3) * A1212);
+  im(1, 0) = det * -(m(1, 0) * A2323 - m(1, 2) * A0323 + m(1, 3) * A0223);
+  im(1, 1) = det * (m(0, 0) * A2323 - m(0, 2) * A0323 + m(0, 3) * A0223);
+  im(1, 2) = det * -(m(0, 0) * A2313 - m(0, 2) * A0313 + m(0, 3) * A0213);
+  im(1, 3) = det * (m(0, 0) * A2312 - m(0, 2) * A0312 + m(0, 3) * A0212);
+  im(2, 0) = det * (m(1, 0) * A1323 - m(1, 1) * A0323 + m(1, 3) * A0123);
+  im(2, 1) = det * -(m(0, 0) * A1323 - m(0, 1) * A0323 + m(0, 3) * A0123);
+  im(2, 2) = det * (m(0, 0) * A1313 - m(0, 1) * A0313 + m(0, 3) * A0113);
+  im(2, 3) = det * -(m(0, 0) * A1312 - m(0, 1) * A0312 + m(0, 3) * A0112);
+  im(3, 0) = det * -(m(1, 0) * A1223 - m(1, 1) * A0223 + m(1, 2) * A0123);
+  im(3, 1) = det * (m(0, 0) * A1223 - m(0, 1) * A0223 + m(0, 2) * A0123);
+  im(3, 2) = det * -(m(0, 0) * A1213 - m(0, 1) * A0213 + m(0, 2) * A0113);
+  im(3, 3) = det * (m(0, 0) * A1212 - m(0, 1) * A0212 + m(0, 2) * A0112);
+  return im;
+}
 }  // namespace math

common/repl/config.cpp

@@ -2,20 +2,26 @@
 
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace REPL {
 void to_json(json& j, const Config& obj) {
   j = json{
+      {"nreplPort", obj.nrepl_port},
       {"gameVersionFolder", obj.game_version_folder},
       {"numConnectToTargetAttempts", obj.target_connect_attempts},
       {"asmFileSearchDirs", obj.asm_file_search_dirs},
       {"keybinds", obj.keybinds},
       {"perGameHistory", obj.per_game_history},
+      {"permissiveRedefinitions", obj.permissive_redefinitions},
   };
 }
 
 void from_json(const json& j, Config& obj) {
+  // TODO - make a camelCase variant of json_serialize/deserialize macros
+  if (j.contains("nreplPort")) {
+    j.at("nreplPort").get_to(obj.nrepl_port);
+  }
   if (j.contains("gameVersionFolder")) {
     j.at("gameVersionFolder").get_to(obj.game_version_folder);
   }
@@ -55,6 +61,9 @@ void from_json(const json& j, Config& obj) {
   if (j.contains("perGameHistory")) {
     j.at("perGameHistory").get_to(obj.per_game_history);
   }
+  if (j.contains("permissiveRedefinitions")) {
+    j.at("permissiveRedefinitions").get_to(obj.permissive_redefinitions);
+  }
   // if there is game specific configuration, override any values we just set
   if (j.contains(version_to_game_name(obj.game_version))) {
     from_json(j.at(version_to_game_name(obj.game_version)), obj);

common/repl/config.h

@@ -26,11 +26,14 @@ struct KeyBind {
 void to_json(json& j, const KeyBind& obj);
 void from_json(const json& j, KeyBind& obj);
 
+// TODO - per-game config
 struct Config {
   GameVersion game_version;
   Config(GameVersion _game_version) : game_version(_game_version){};
 
   // this is the default REPL configuration
+  int nrepl_port = 8181;
+  int temp_nrepl_port = -1;
   std::string game_version_folder;
   int target_connect_attempts = 30;
   std::vector<std::string> asm_file_search_dirs = {};
@@ -45,6 +48,14 @@ struct Config {
       {KeyBind::Modifier::CTRL, "B", "Displays the most recently caught backtrace", "(:di)"},
       {KeyBind::Modifier::CTRL, "N", "Full build of the game", "(mi)"}};
   bool per_game_history = true;
+  bool permissive_redefinitions = false;
+
+  int get_nrepl_port() {
+    if (temp_nrepl_port != -1) {
+      return temp_nrepl_port;
+    }
+    return nrepl_port;
+  }
 };
 void to_json(json& j, const Config& obj);
 void from_json(const json& j, Config& obj);

common/repl/nrepl/ReplClient.cpp

@@ -4,7 +4,7 @@
 #include "common/cross_sockets/XSocket.h"
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef _WIN32
 #define NOMINMAX

common/repl/nrepl/ReplServer.cpp

@@ -4,7 +4,7 @@
 #include "common/cross_sockets/XSocket.h"
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef _WIN32
 #define NOMINMAX
@@ -16,10 +16,6 @@
 #include "common/log/log.h"
 // clang-format on
 
-// TODO - basically REPL to listen and inject commands into a running REPL
-// - we will need a C++ side client as well which will let us communicate with the repl via for
-// example, ImgUI
-//
 // TODO - The server also needs to eventually return the result of the evaluation
 
 ReplServer::~ReplServer() {
@@ -31,7 +27,18 @@ ReplServer::~ReplServer() {
 
 void ReplServer::post_init() {
   // Add the listening socket to our set of sockets
-  lg::info("[nREPL:{}:{}] awaiting connections", tcp_port, listening_socket);
+  lg::debug("[nREPL:{}:{}] awaiting connections", tcp_port, listening_socket);
+}
+
+void ReplServer::error_response(int socket, const std::string& error) {
+  std::string msg = fmt::format("[ERROR]: {}", error);
+  auto resp = write_to_socket(socket, msg.c_str(), msg.size());
+  if (resp == -1) {
+    lg::warn("[nREPL:{}] Client Disconnected: {}", tcp_port, address_to_string(addr),
+             ntohs(addr.sin_port), socket);
+    close_socket(socket);
+    client_sockets.erase(socket);
+  }
 }
 
 void ReplServer::ping_response(int socket) {
@@ -52,7 +59,6 @@ std::optional<std::string> ReplServer::get_msg() {
 
   // Add the server's main listening socket (where we accept clients from)
   FD_SET(listening_socket, &read_sockets);
-
   int max_sd = listening_socket;
   for (const int& sock : client_sockets) {
     if (sock > max_sd) {
@@ -64,12 +70,11 @@ std::optional<std::string> ReplServer::get_msg() {
   }
 
   // Wait for activity on _something_, with a timeout so we don't get stuck here on exit.
-  struct timeval timeout;
-  timeout.tv_sec = 0;
-  timeout.tv_usec = 100000;
+  struct timeval timeout = {0, 100000};
   auto activity = select(max_sd + 1, &read_sockets, NULL, NULL, &timeout);
-
-  if (activity < 0) {  // TODO - || error!
+  if (activity < 0 && errno != EINTR) {
+    lg::error("[nREPL:{}] select error, returned: {}, errno: {}", tcp_port, activity,
+              strerror(errno));
     return std::nullopt;
   }
 
@@ -78,44 +83,46 @@ std::optional<std::string> ReplServer::get_msg() {
     socklen_t addr_len = sizeof(addr);
     auto new_socket = accept_socket(listening_socket, (sockaddr*)&addr, &addr_len);
     if (new_socket < 0) {
-      // TODO - handle error
+      if (new_socket != -1) {
+        lg::error("[nREPL:{}] accept error, returned: {}, errono: {}", tcp_port, new_socket,
+                  strerror(errno));
+      }
     } else {
       lg::info("[nREPL:{}]: New socket connection: {}:{}:{}", tcp_port, address_to_string(addr),
                ntohs(addr.sin_port), new_socket);
-
       // Say hello
       ping_response(new_socket);
       // Track the new socket
       if ((int)client_sockets.size() < max_clients) {
         client_sockets.insert(new_socket);
       } else {
-        // TODO - Respond with NO
+        // Respond with NO and close the socket
+        lg::warn("[nREPL:{}]: Maximum clients reached. Rejecting connection.", tcp_port);
+        error_response(new_socket, "Maximum clients reached. Rejecting connection.");
+        close_socket(new_socket);
       }
     }
   }
 
-  // otherwise (and no matter what) check all the clients to see if they have sent us anything
-  // else its some IO operation on some other socket
-  //
-  // RACE - the first client wins
-
-  // TODO - there are ways to do this with iterators but, couldn't figure it out!
-  std::vector<int> sockets_to_scan(client_sockets.begin(), client_sockets.end());
-  for (const int& sock : sockets_to_scan) {
+  // Check all clients for activity
+  for (auto it = client_sockets.begin(); it != client_sockets.end();) {
+    int sock = *it;
     if (FD_ISSET(sock, &read_sockets)) {
       // Attempt to read a header
-      // TODO - should this be in a loop?
       auto req_bytes = read_from_socket(sock, header_buffer.data(), header_buffer.size());
-      if (req_bytes == 0) {
-        // Socket disconnected
+      if (req_bytes <= 0) {
         // TODO - add a queue of messages in the REPL::Wrapper so we can print _BEFORE_ the prompt
         // is output
-        lg::warn("[nREPL:{}] Client Disconnected: {}", tcp_port, address_to_string(addr),
-                 ntohs(addr.sin_port), sock);
-
+        if (req_bytes == 0) {
+          lg::warn("[nREPL:{}] Client Disconnected: {}", tcp_port, address_to_string(addr));
+        } else {
+          lg::warn("[nREPL:{}] Error reading from socket on {}: {}", tcp_port,
+                   address_to_string(addr), strerror(errno));
+        }
         // Cleanup the socket and remove it from our set
         close_socket(sock);
-        client_sockets.erase(sock);
+        it = client_sockets.erase(it);  // Erase and move to the next element
+        continue;
       } else {
         // Otherwise, process the message
         auto* header = (ReplServerHeader*)(header_buffer.data());
@@ -123,7 +130,12 @@ std::optional<std::string> ReplServer::get_msg() {
         int expected_size = header->length;
         int got = 0;
         int tries = 0;
+        bool skip_to_next_socket = false;
         while (got < expected_size) {
+          if (want_exit_callback()) {
+            lg::warn("[nREPL:{}] Terminating nREPL early", tcp_port);
+            return std::nullopt;
+          }
           tries++;
           if (tries > 100) {
             break;
@@ -135,11 +147,25 @@ std::optional<std::string> ReplServer::get_msg() {
                 tcp_port, got, expected_size, buffer.size());
             return std::nullopt;
           }
-          auto x = read_from_socket(sock, buffer.data() + got, expected_size - got);
-          if (want_exit_callback()) {
-            return std::nullopt;
+          auto bytes_read = read_from_socket(sock, buffer.data() + got, expected_size - got);
+          if (bytes_read <= 0) {
+            if (bytes_read == 0) {
+              lg::warn("[nREPL:{}] Client Disconnected: {}", tcp_port, address_to_string(addr));
+            } else {
+              lg::warn("[nREPL:{}] Error reading from socket on {}: {}", tcp_port,
+                       address_to_string(addr), strerror(errno));
+            }
+            close_socket(sock);
+            it = client_sockets.erase(it);  // Erase and move to the next element
+            skip_to_next_socket = true;
+            break;
           }
-          got += x > 0 ? x : 0;
+          got += bytes_read;
+          std::this_thread::sleep_for(std::chrono::milliseconds(1));
+        }
+
+        if (skip_to_next_socket) {
+          continue;
         }
 
         switch (header->type) {
@@ -153,6 +179,7 @@ std::optional<std::string> ReplServer::get_msg() {
         }
       }
     }
+    ++it;
   }
   return std::nullopt;
 }

common/repl/nrepl/ReplServer.h

@@ -27,5 +27,6 @@ class ReplServer : public XSocketServer {
   fd_set read_sockets;
   std::set<int> client_sockets = {};
 
+  void error_response(int socket, const std::string& error);
   void ping_response(int socket);
 };

common/repl/repl_wrapper.cpp

@@ -1,43 +1,73 @@
-#include "util.h"
+#include "repl_wrapper.h"
 
 #include "common/util/FileUtil.h"
 #include "common/util/json_util.h"
 #include "common/util/string_util.h"
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/color.h"
-#include "third-party/fmt/core.h"
+#include "fmt/color.h"
+#include "fmt/core.h"
 #include "third-party/replxx/include/replxx.hxx"
-// TODO - expand a list of hints (ie. a hint for defun to show at a glance how to write a function,
-// or perhaps, show the docstring for the current function being used?)
 
 namespace REPL {
 void Wrapper::clear_screen() {
   repl.clear_screen();
 }
 
-void Wrapper::print_welcome_message() {
-  // TODO - dont print on std-out
-  // Welcome message / brief intro for documentation
-  std::string ascii;
-  ascii += " _____             _____ _____ _____ __    \n";
-  ascii += "|     |___ ___ ___|   __|     |  _  |  |   \n";
-  ascii += "|  |  | . | -_|   |  |  |  |  |     |  |__ \n";
-  ascii += "|_____|  _|___|_|_|_____|_____|__|__|_____|\n";
-  ascii += "      |_|                                  \n";
-  fmt::print(fmt::emphasis::bold | fg(fmt::color::orange), ascii);
-
-  fmt::print("Welcome to OpenGOAL {}.{}!\n", versions::GOAL_VERSION_MAJOR,
-             versions::GOAL_VERSION_MINOR);
-  fmt::print("Run {} or {} for help with common commands and REPL usage.\n",
-             fmt::styled("(repl-help)", fmt::emphasis::bold | fg(fmt::color::cyan)),
-             fmt::styled("(repl-keybinds)", fmt::emphasis::bold | fg(fmt::color::cyan)));
-  fmt::print("Run ");
-  fmt::print(fmt::emphasis::bold | fg(fmt::color::cyan), "(lt)");
-  fmt::print(" to connect to the local target.\n");
-  fmt::print("Run ");
-  fmt::print(fmt::emphasis::bold | fg(fmt::color::cyan), "(mi)");
-  fmt::print(" to rebuild the entire game.\n\n");
+void Wrapper::print_welcome_message(const std::vector<std::string>& loaded_projects) {
+  std::string message;
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "        ..:::::..\n");
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "     .:-----------:.\n");
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "         .-----.");
+  message += fmt::format(fmt::emphasis::bold, "         Welcome to OpenGOAL {}.{} [{}]",
+                         versions::GOAL_VERSION_MAJOR, versions::GOAL_VERSION_MINOR,
+                         fmt::format(fg(fmt::color::gray), "{}", build_revision()));
+  if (!username.empty() && username != "#f" && username != "unknown") {
+    message += fmt::format(fg(fmt::color::light_green), " {}", username);
+  }
+  message += "!\n";
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "          .---.");
+  if (repl_config.game_version == GameVersion::Jak1) {
+    message += fmt::format("          [{}]: ", fmt::format(fg(fmt::color::orange), "jak1"));
+  } else if (repl_config.game_version == GameVersion::Jak2) {
+    message += fmt::format("          [{}]: ", fmt::format(fg(fmt::color::purple), "jak2"));
+  } else if (repl_config.game_version == GameVersion::Jak3) {
+    message += fmt::format("          [{}]: ", fmt::format(fg(fmt::color::gold), "jak3"));
+  } else {
+    message += fmt::format("          [{}]: ", fmt::format(fg(fmt::color::magenta), "jakx"));
+  }
+  const auto loaded_projects_str = fmt::format("{}", fmt::join(loaded_projects, ","));
+  message += fmt::format(fg(fmt::color::gray), "{}\n", loaded_projects_str);
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "  .        ---        .");
+  message +=
+      fmt::format("  Project Path: {}\n",
+                  fmt::format(fg(fmt::color::gray), file_util::get_jak_project_dir().string()));
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "  -       :===:       -");
+  message += "  nREPL:";
+  if (!nrepl_alive) {
+    message += fmt::format(fg(fmt::color::red), "DISABLED\n");
+  } else {
+    message += fmt::format(fg(fmt::color::light_green), " Listening on {}\n",
+                           repl_config.get_nrepl_port());
+  }
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "  --.   .--: :--.   .--");
+  message += "  Source File Search Dirs: ";
+  const auto search_dir_string =
+      fmt::format("{}", fmt::join(repl_config.asm_file_search_dirs, ","));
+  message += fmt::format("[{}]\n", fmt::format(fg(fmt::color::gray), search_dir_string));
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "  .=======.    =======.");
+  message += fmt::format("  {} or {} for basic help and usage\n",
+                         fmt::format(fg(fmt::color::cyan), "(repl-help)"),
+                         fmt::format(fg(fmt::color::cyan), "(repl-keybinds)"));
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "   .-=====-. .-=====-");
+  message +=
+      fmt::format("    {} to connect to the game\n", fmt::format(fg(fmt::color::cyan), "(lt)"));
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "     .-===========-.");
+  message += fmt::format("     {} to recompile the active project.\n",
+                         fmt::format(fg(fmt::color::cyan), "(mi)"));
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "         .-===-.\n");
+  message += fmt::format(fmt::emphasis::bold | fg(fmt::color::orange), "            .\n");
+  fmt::print("{}", message);
 }
 
 void Wrapper::print_to_repl(const std::string& str) {
@@ -207,7 +237,7 @@ std::string find_repl_username() {
     }
   }
 
-  return "#f";
+  return "unknown";
 }
 
 fs::path get_startup_file_path(const std::string& username, const GameVersion game_version) {
@@ -242,20 +272,24 @@ StartupFile load_user_startup_file(const std::string& username, const GameVersio
   return startup_file;
 }
 
-REPL::Config load_repl_config(const std::string& username, const GameVersion game_version) {
+REPL::Config load_repl_config(const std::string& username,
+                              const GameVersion game_version,
+                              const int nrepl_port) {
   auto repl_config_path =
       file_util::get_jak_project_dir() / "goal_src" / "user" / username / "repl-config.json";
+  REPL::Config loaded_config(game_version);
   if (file_util::file_exists(repl_config_path.string())) {
     try {
-      REPL::Config config(game_version);
       auto repl_config_data =
           parse_commented_json(file_util::read_text_file(repl_config_path), "repl-config.json");
-      from_json(repl_config_data, config);
-      return config;
+      from_json(repl_config_data, loaded_config);
+      loaded_config.temp_nrepl_port = nrepl_port;
+      return loaded_config;
     } catch (std::exception& e) {
-      REPL::Config config(game_version);
+      // do nothing
     }
   }
-  return REPL::Config(game_version);
+  loaded_config.temp_nrepl_port = nrepl_port;
+  return loaded_config;
 }
 }  // namespace REPL

common/repl/repl_wrapper.h

@@ -17,18 +17,23 @@ struct StartupFile {
 };
 
 class Wrapper {
-  replxx::Replxx repl;
-
  public:
   std::string username;
   Config repl_config;
   StartupFile startup_file;
+  bool nrepl_alive = false;
   std::vector<std::string> examples{};
   std::vector<std::pair<std::string, replxx::Replxx::Color>> regex_colors{};
 
   Wrapper(GameVersion version) : repl_config(version) {}
-  Wrapper(const std::string& _username, const Config& config, const StartupFile& startup)
-      : username(_username), repl_config(config), startup_file(startup) {}
+  Wrapper(const std::string& _username,
+          const Config& config,
+          const StartupFile& startup,
+          bool _nrepl_alive)
+      : username(_username),
+        repl_config(config),
+        startup_file(startup),
+        nrepl_alive(_nrepl_alive) {}
   replxx::Replxx& get_repl() { return repl; }
   void init_settings();
   void reload_startup_file();
@@ -36,7 +41,7 @@ class Wrapper {
   // Functionality / Commands
   void clear_screen();
   void print_to_repl(const std::string& str);
-  void print_welcome_message();
+  void print_welcome_message(const std::vector<std::string>& loaded_projects);
   void set_history_max_size(size_t len);
   const char* readline(const std::string& prompt);
   void add_to_history(const std::string& line);
@@ -47,11 +52,14 @@ class Wrapper {
   std::pair<std::string, bool> get_current_repl_token(std::string const& context);
 
  private:
+  replxx::Replxx repl;
   replxx::Replxx::key_press_handler_t commit_text_action(std::string text_to_commit);
   std::vector<REPL::KeyBind> keybindings = {};
 };
 
 std::string find_repl_username();
 StartupFile load_user_startup_file(const std::string& username, const GameVersion game_version);
-REPL::Config load_repl_config(const std::string& username, const GameVersion game_version);
+REPL::Config load_repl_config(const std::string& username,
+                              const GameVersion game_version,
+                              const int nrepl_port);
 }  // namespace REPL

common/serialization/subtitles/subtitles.cpp

@@ -5,7 +5,7 @@
 #include "common/util/FileUtil.h"
 #include "common/util/json_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 void open_subtitle_project(const std::string& project_kind,
                            const std::string& file_path,
@@ -13,20 +13,20 @@ void open_subtitle_project(const std::string& project_kind,
   goos::Reader reader;
   auto& proj = reader.read_from_file({file_path}).as_pair()->cdr.as_pair()->car;
   if (!proj.is_pair() || !proj.as_pair()->car.is_symbol() ||
-      proj.as_pair()->car.as_symbol()->name != project_kind) {
+      proj.as_pair()->car.as_symbol() != project_kind) {
     throw std::runtime_error(fmt::format("invalid project '{}'", project_kind));
   }
 
   goos::for_each_in_list(proj.as_pair()->cdr, [&](const goos::Object& o) {
     if (o.is_pair() && o.as_pair()->cdr.is_pair()) {
       auto args = o.as_pair();
-      auto& action = args->car.as_symbol()->name;
+      auto& action = args->car.as_symbol();
       args = args->cdr.as_pair();
 
       if (action == "file-json") {
         auto new_file = GameSubtitleDefinitionFile();
         while (true) {
-          const auto& kwarg = args->car.as_symbol()->name;
+          const auto& kwarg = args->car.as_symbol();
           args = args->cdr.as_pair();
           if (kwarg == ":language-id") {
             new_file.language_id = args->car.as_int();
@@ -49,7 +49,7 @@ void open_subtitle_project(const std::string& project_kind,
         subtitle_files.push_back(new_file);
       } else {
         throw std::runtime_error(
-            fmt::format("unknown action {} in {} project", action, project_kind));
+            fmt::format("unknown action {} in {} project", action.name_ptr, project_kind));
       }
     } else {
       throw std::runtime_error(fmt::format("invalid entry in {} project", project_kind));

common/serialization/subtitles/subtitles_v1.cpp

@@ -5,7 +5,7 @@
 
 #include "subtitles_v2.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 void to_json(json& j, const SubtitleCutsceneLineMetadataV1& obj) {
   json_serialize(frame_start);

common/serialization/subtitles/subtitles_v2.cpp

@@ -5,7 +5,7 @@
 
 #include "subtitles_v1.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 void to_json(json& j, const SubtitleLineMetadata& obj) {
   json_serialize(frame_start);
@@ -89,7 +89,9 @@ const std::unordered_map<std::string, u16> jak2_speaker_name_to_enum_val = {
     {"citizen-male", 31},
     {"citizen-female", 32},
     {"oracle", 33},
-    {"precursor", 34}};
+    {"precursor", 34},
+    {"metalkor-before-consite", 35},
+    {"metalkor-intro", 36}};
 
 GameSubtitlePackage read_json_files_v2(const GameSubtitleDefinitionFile& file_info) {
   GameSubtitlePackage package;
@@ -373,7 +375,7 @@ bool GameSubtitleDB::write_subtitle_db_to_files(const GameVersion game_version)
       file_util::write_text_file(dump_path, lines_file.dump(2));
     }
   } catch (std::exception& ex) {
-    lg::error(ex.what());
+    lg::error("{}", ex.what());
     return false;
   }
   return true;

common/serialization/text/text_ser.cpp

@@ -147,7 +147,7 @@ void parse_text_goal(const goos::Object& data,
           throw std::runtime_error("invalid text version entry");
         }
 
-        font = get_font_bank(ver_name.as_symbol()->name);
+        font = get_font_bank(ver_name.as_symbol().name_ptr);
       }
 
       else if (head.is_int()) {
@@ -278,7 +278,7 @@ GameTextVersion parse_text_only_version(const goos::Object& data) {
           throw std::runtime_error("invalid text version entry");
         }
 
-        font = get_font_bank(ver_name.as_symbol()->name);
+        font = get_font_bank(ver_name.as_symbol().name_ptr);
       }
     }
   });
@@ -294,14 +294,14 @@ void open_text_project(const std::string& kind,
   goos::Reader reader;
   auto& proj = reader.read_from_file({filename}).as_pair()->cdr.as_pair()->car;
   if (!proj.is_pair() || !proj.as_pair()->car.is_symbol() ||
-      proj.as_pair()->car.as_symbol()->name != kind) {
+      proj.as_pair()->car.as_symbol() != kind) {
     throw std::runtime_error(fmt::format("invalid {} project", kind));
   }
 
   goos::for_each_in_list(proj.as_pair()->cdr, [&](const goos::Object& o) {
     if (o.is_pair() && o.as_pair()->cdr.is_pair()) {
       auto args = o.as_pair();
-      auto& action = args->car.as_symbol()->name;
+      auto& action = args->car.as_symbol();
       args = args->cdr.as_pair();
 
       if (action == "file") {
@@ -313,17 +313,18 @@ void open_text_project(const std::string& kind,
       } else if (action == "file-json") {
         auto& language_id = args->car.as_int();
         args = args->cdr.as_pair();
-        auto& text_version = args->car.as_symbol()->name;
+        auto& text_version = args->car.as_symbol();
         args = args->cdr.as_pair();
         std::optional<std::string> group_name = std::nullopt;
         group_name = args->car.as_string()->data;
         args = args->cdr.as_pair()->car.as_pair();
         goos::for_each_in_list(args->cdr.as_pair()->car, [&](const goos::Object& o) {
           text_files.push_back({GameTextDefinitionFile::Format::JSON, o.as_string()->data,
-                                (int)language_id, text_version, group_name});
+                                (int)language_id, text_version.name_ptr, group_name});
         });
       } else {
-        throw std::runtime_error(fmt::format("unknown action {} in {} project", action, kind));
+        throw std::runtime_error(
+            fmt::format("unknown action {} in {} project", action.name_ptr, kind));
       }
     } else {
       throw std::runtime_error(fmt::format("invalid entry in {} project", kind));

common/symbols.h

@@ -150,6 +150,180 @@ constexpr int FIX_FIXED_SYM_END_OFFSET = 0xec;
 
 }  // namespace jak2_symbols
 
+namespace jak3_symbols {
+
+constexpr int FIX_SYM_EMPTY_CAR = -0x8;
+constexpr int S7_OFF_FIX_SYM_EMPTY_PAIR = -0x6 - 1;
+constexpr int FIX_SYM_EMPTY_CDR = -0x4;
+constexpr int FIX_SYM_FALSE = 0x0;  // GOAL boolean #f (note that this is equal to the $s7 register)
+constexpr int FIX_SYM_TRUE = 0x4;   // GOAL boolean #t
+constexpr int FIX_SYM_FUNCTION_TYPE = 0x8;
+constexpr int FIX_SYM_BASIC = 0xc;
+constexpr int FIX_SYM_STRING_TYPE = 0x10;
+constexpr int FIX_SYM_SYMBOL_TYPE = 0x14;
+constexpr int FIX_SYM_TYPE_TYPE = 0x18;
+constexpr int FIX_SYM_OBJECT_TYPE = 0x1c;
+constexpr int FIX_SYM_LINK_BLOCK = 0x20;
+constexpr int FIX_SYM_INTEGER = 0x24;
+constexpr int FIX_SYM_SINTEGER = 0x28;
+constexpr int FIX_SYM_UINTEGER = 0x2c;
+constexpr int FIX_SYM_BINTEGER = 0x30;
+constexpr int FIX_SYM_INT8 = 0x34;
+constexpr int FIX_SYM_INT16 = 0x38;
+constexpr int FIX_SYM_INT32 = 0x3c;
+constexpr int FIX_SYM_INT64 = 0x40;
+constexpr int FIX_SYM_INT128 = 0x44;
+constexpr int FIX_SYM_UINT8 = 0x48;
+constexpr int FIX_SYM_UINT16 = 0x4c;
+constexpr int FIX_SYM_UINT32 = 0x50;
+constexpr int FIX_SYM_UINT64 = 0x54;
+constexpr int FIX_SYM_UINT128 = 0x58;
+constexpr int FIX_SYM_FLOAT = 0x5c;
+constexpr int FIX_SYM_PROCESS_TREE = 0x60;
+constexpr int FIX_SYM_PROCESS_TYPE = 0x64;
+constexpr int FIX_SYM_THREAD = 0x68;
+constexpr int FIX_SYM_STRUCTURE = 0x6c;
+constexpr int FIX_SYM_PAIR_TYPE = 0x70;
+constexpr int FIX_SYM_POINTER = 0x74;
+constexpr int FIX_SYM_NUMBER = 0x78;
+constexpr int FIX_SYM_ARRAY = 0x7c;
+constexpr int FIX_SYM_VU_FUNCTION = 0x80;
+constexpr int FIX_SYM_CONNECTABLE = 0x84;
+constexpr int FIX_SYM_STACK_FRAME = 0x88;
+constexpr int FIX_SYM_FILE_STREAM = 0x8c;
+constexpr int FIX_SYM_HEAP = 0x90;
+constexpr int FIX_SYM_NOTHING_FUNC = 0x94;
+constexpr int FIX_SYM_DELETE_BASIC = 0x98;
+constexpr int FIX_SYM_STATIC = 0x9c;
+constexpr int FIX_SYM_GLOBAL_HEAP = 0xa0;
+constexpr int FIX_SYM_DEBUG = 0xa4;
+constexpr int FIX_SYM_LOADING_LEVEL = 0xa8;
+constexpr int FIX_SYM_LOADING_PACKAGE = 0xac;
+constexpr int FIX_SYM_PROCESS_LEVEL_HEAP = 0xb0;
+constexpr int FIX_SYM_STACK = 0xb4;
+constexpr int FIX_SYM_SCRATCH = 0xb8;
+constexpr int FIX_SYM_SCRATCH_TOP = 0xbc;
+constexpr int FIX_SYM_ZERO_FUNC = 0xc0;
+constexpr int FIX_SYM_ASIZE_OF_BASIC_FUNC = 0xc4;
+constexpr int FIX_SYM_COPY_BASIC_FUNC = 0xc8;  // bugged name
+constexpr int FIX_SYM_LEVEL = 0xcc;
+constexpr int FIX_SYM_ART_GROUP = 0xd0;
+constexpr int FIX_SYM_TEXTURE_PAGE_DIR = 0xd4;
+constexpr int FIX_SYM_TEXTURE_PAGE = 0xd8;
+constexpr int FIX_SYM_SOUND = 0xdc;
+constexpr int FIX_SYM_DGO = 0xe0;
+constexpr int FIX_SYM_TOP_LEVEL = 0xe4;
+constexpr int FIX_SYM_QUOTE = 0xe8;
+constexpr int FIX_SYM_LISTENER_LINK_BLOCK = 0xec;
+constexpr int FIX_SYM_LISTENER_FUNCTION = 0xf0;
+constexpr int FIX_SYM_STACK_TOP = 0xf4;
+constexpr int FIX_SYM_STACK_BASE = 0xf8;
+constexpr int FIX_SYM_STACK_SIZE = 0xfc;
+constexpr int FIX_SYM_KERNEL_FUNCTION = 0x100;
+constexpr int FIX_SYM_KERNEL_PACKAGES = 0x104;
+constexpr int FIX_SYM_KERNEL_BOOT_MESSAGE = 0x108;
+constexpr int FIX_SYM_KERNEL_BOOT_MODE = 0x10c;
+constexpr int FIX_SYM_KERNEL_BOOT_LEVEL = 0x110;
+constexpr int FIX_SYM_KERNEL_BOOT_ART_GROUP = 0x114;
+constexpr int FIX_SYM_KERNEL_DEBUG = 0x118;
+constexpr int FIX_SYM_KERNEL_VERSION = 0x11c;
+constexpr int FIX_SYM_KERNEL_DISPATCHER = 0x120;
+constexpr int FIX_SYM_SYNC_DISPATCHER = 0x124;
+constexpr int FIX_SYM_PRINT_COLLUMN = 0x128;
+constexpr int FIX_SYM_DEBUG_SEGMENT = 300;
+constexpr int FIX_SYM_ENABLE_METHOD_SET = 0x130;
+constexpr int FIX_SYM_SQL_RESULT = 0x134;
+constexpr int FIX_SYM_COLLAPSE_QUOTE = 0x138;
+constexpr int FIX_SYM_LEVEL_TYPE_LIST = 0x13C;
+constexpr int FIX_SYM_DECI_COUNT = 0x140;
+constexpr int FIX_SYM_USER = 0x144;
+constexpr int FIX_SYM_VIDEO_MODE = 0x148;
+constexpr int FIX_SYM_BOOT_VIDEO_MODE = 0x14C;
+constexpr int FIX_SYM_BOOT = 0x150;
+constexpr int FIX_SYM_DEMO = 0x154;
+constexpr int FIX_SYM_DEMO_SHARED = 0x158;
+constexpr int FIX_SYM_PREVIEW = 0x15C;
+constexpr int FIX_SYM_KIOSK = 0x160;
+constexpr int FIX_SYM_PLAY_BOOT = 0x164;
+constexpr int FIX_SYM_SIN = 0x168;
+constexpr int FIX_SYM_COS = 0x16C;
+constexpr int FIX_SYM_PUT_DISPLAY_ENV = 0x170;
+constexpr int FIX_SYM_SYNCV = 0x174;
+constexpr int FIX_SYM_SYNC_PATH = 0x178;
+constexpr int FIX_SYM_RESET_PATH = 0x17C;
+constexpr int FIX_SYM_RESET_GRAPH = 0x180;
+constexpr int FIX_SYM_DMA_SYNC = 0x184;
+constexpr int FIX_SYM_GS_PUT_IMR = 0x188;
+constexpr int FIX_SYM_GS_GET_IMR = 0x18C;
+constexpr int FIX_SYM_GS_STORE_IMAGE = 400;
+constexpr int FIX_SYM_FLUSH_CACHE = 0x194;
+constexpr int FIX_SYM_CPAD_OPEN = 0x198;
+constexpr int FIX_SYM_CPAD_GET_DATA = 0x19C;
+constexpr int FIX_SYM_MOUSE_GET_DATA = 0x1A0;
+constexpr int FIX_SYM_KEYBD_GET_DATA = 0x1A4;
+constexpr int FIX_SYM_INSTALL_HANDLER = 0x1A8;
+constexpr int FIX_SYM_INSTALL_DEBUG_HANDLER = 0x1AC;
+constexpr int FIX_SYM_FILE_STREAM_OPEN = 0x1B0;
+constexpr int FIX_SYM_FILE_STREAM_CLOSE = 0x1B4;
+constexpr int FIX_SYM_FILE_STREAM_LENGTH = 0x1B8;
+constexpr int FIX_SYM_FILE_STREAM_SEEK = 0x1BC;
+constexpr int FIX_SYM_FILE_STREAM_READ = 0x1C0;
+constexpr int FIX_SYM_FILE_STREAM_WRITE = 0x1C4;
+constexpr int FIX_SYM_SCF_GET_LANGUAGE = 0x1C8;
+constexpr int FIX_SYM_SCF_GET_TIME = 0x1CC;
+constexpr int FIX_SYM_SCF_GET_ASPECT = 0x1D0;
+constexpr int FIX_SYM_SCF_GET_VOLUME = 0x1D4;
+constexpr int FIX_SYM_SCF_GET_TERRITORY = 0x1D8;
+constexpr int FIX_SYM_SCF_GET_TIMEOUT = 0x1DC;
+constexpr int FIX_SYM_SCF_GET_INACTIVE_TIMEOUT = 0x1E0;
+constexpr int FIX_SYM_DMA_TO_IOP = 0x1E4;
+constexpr int FIX_SYM_KERNEL_SHUTDOWN = 0x1E8;
+constexpr int FIX_SYM_AYBABTU = 0x1EC;
+constexpr int FIX_SYM_STRING_TO_SYMBOL = 0x1F0;
+constexpr int FIX_SYM_SYMBOL_TO_STRING = 500;
+constexpr int FIX_SYM_PRINT = 0x1F8;
+constexpr int FIX_SYM_INSPECT = 0x1FC;
+constexpr int FIX_SYM_LOAD = 0x200;
+constexpr int FIX_SYM_LOADB = 0x204;
+constexpr int FIX_SYM_LOADO = 0x208;
+constexpr int FIX_SYM_UNLOAD = 0x20C;
+constexpr int FIX_SYM_FORMAT = 0x210;
+constexpr int FIX_SYM_MALLOC = 0x214;
+constexpr int FIX_SYM_KMALLOC = 0x218;
+constexpr int FIX_SYM_KMEMOPEN = 0x21C;
+constexpr int FIX_SYM_KMEMCLOSE = 0x220;
+constexpr int FIX_SYM_NEW_DYNAMIC_STRUCTURE = 0x224;
+constexpr int FIX_SYM_METHOD_SET = 0x228;
+constexpr int FIX_SYM_LINK = 0x22C;
+constexpr int FIX_SYM_LINK_BUSY = 0x230;
+constexpr int FIX_SYM_LINK_RESET = 0x234;
+constexpr int FIX_SYM_LINK_BEGIN = 0x238;
+constexpr int FIX_SYM_LINK_RESUME = 0x23C;
+constexpr int FIX_SYM_DGO_LOAD = 0x240;
+constexpr int FIX_SYM_SQL_QUERY = 0x244;
+constexpr int FIX_SYM_MC_RUN = 0x248;
+constexpr int FIX_SYM_MC_FORMAT = 0x24C;
+constexpr int FIX_SYM_MC_UNFORMAT = 0x250;
+constexpr int FIX_SYM_MC_CREATE_FILE = 0x254;
+constexpr int FIX_SYM_MC_SAVE = 600;
+constexpr int FIX_SYM_MC_LOAD = 0x25C;
+constexpr int FIX_SYM_MC_CHECK_RESULT = 0x260;
+constexpr int FIX_SYM_MC_GET_SLOT_INFO = 0x264;
+constexpr int FIX_SYM_MC_MAKEFILE = 0x268;
+constexpr int FIX_SYM_KSET_LANGUAGE = 0x26C;
+constexpr int FIX_SYM_RPC_CALL = 0x270;
+constexpr int FIX_SYM_RPC_BUSY = 0x274;
+constexpr int FIX_SYM_TEST_LOAD_DGO_C = 0x278;
+constexpr int FIX_SYM_SYMLINK2 = 0x27c;
+constexpr int FIX_SYM_SYMLINK3 = 0x280;
+constexpr int FIX_SYM_ULTIMATE_MEMCPY = 0x284;
+constexpr int FIX_SYM_PLAY = 0x288;
+constexpr int FIX_SYM_SYMBOL_STRING = 0x28c;
+constexpr int FIX_SYM_KERNEL_SYMBOL_WARNINGS = 0x290;
+constexpr int FIX_SYM_NETWORK_BOOTSTRAP = 0x294;
+
+}  // namespace jak3_symbols
+
 constexpr int false_symbol_offset() {
   return jak1_symbols::FIX_SYM_FALSE;
 }
@@ -159,6 +333,7 @@ constexpr int true_symbol_offset(GameVersion version) {
     case GameVersion::Jak1:
       return jak1_symbols::FIX_SYM_TRUE;
     case GameVersion::Jak2:
+    case GameVersion::Jak3:
       return jak2_symbols::FIX_SYM_TRUE;
   }
 }
@@ -168,6 +343,7 @@ constexpr int empty_pair_offset_from_s7(GameVersion version) {
     case GameVersion::Jak1:
       return jak1_symbols::FIX_SYM_EMPTY_PAIR;
     case GameVersion::Jak2:
+    case GameVersion::Jak3:
       // minus 1 for the symbol table pointer's offset.
       return jak2_symbols::S7_OFF_FIX_SYM_EMPTY_PAIR;
   }

common/texture/texture_slots.cpp

@@ -42,8 +42,160 @@ std::vector<std::string> jak2_slots = {
     "cas-toxic-slime-scroll-dest",
 };
 
-}
+std::vector<std::string> jak3_slots = {
+    "skull-gem-dest",
+    "jakc-arm",
+    "jakc-eyebrow",
+    "jakc-face",
+    "jakc-finger",
+    "jakc-hair",
+    "jakchires-arm",
+    "jakchires-eye",
+    "jakchires-eyebrow",
+    "jakchires-eyelid",
+    "jakchires-facelft",
+    "jakchires-facert",
+    "jakchires-hair",
+    // default-water
+    "bomb-gradient",
+    "blue-beam-dest",
+    "lightjak-wings",
+    "mushroom-dest",
+    // default-warp
+    "shield-env-rim-dest",
+    // templea-water
+    "templea-waterfall-dest",
+    // templea-warp
+    "holograph-env-rim-dest",
+    // hanga-sprite
+    "glider-ring-dest2",
+    "glider-ring-dest",
+    // foresta-water
+    "fora-water-dest",
+    "fora-waterfall-01-dest",
+    "fora-water-wave-01-dest",
+    // forestb-water
+    "forb-water-dest",
+    "forb-waterfall-01-dest",
+    "forb-water-wave-01-dest",
+    // lforplnt-pris
+    "mh-gem-dest",
+    // lmhcitya-tfrag
+    "mhcitya-base-goo-01-dest",
+    // lmhcityb-tfrag
+    "mhcityb-base-goo-01-dest",
+    // mhcitya-pris
+    // "mhcity-de-door-skin-01-dest",
+    // templec-water
+    "tplc-water-dest",
+    // sewc-water
+    "sewer-water-01-c-dest",
+    "sewer-waterfall-01-c-dest",
+    "sewer-waterfall-02-c-dest",
+    "sewer-water-wave-01-c-dest",
+    "sewer-water-highlight-01-c-dest",
+    // sewd-water
+    "sewer-water-01-d-dest",
+    "sewer-waterfall-02-d-dest",
+    "sewer-water-highlight-01-d-dest",
+    "sewer-water-wave-01-d-dest",
+    "sewer-water-wave-02-d-dest",
+    "sewer-water-still-01-d-dest",
+    // sewe-water
+    "sewer-water-01-e-dest",
+    "sewer-waterfall-01-e-dest",
+    "sewer-waterfall-02-e-dest",
+    "sewer-water-highlight-01-e-dest",
+    // sewg-water
+    "sewer-water-01-g-dest",
+    "sewer-waterfall-02-g-dest",
+    "sewer-water-wave-01-g-dest",
+    // sewh-water
+    "sewer-water-01-h-dest",
+    "sewer-waterfall-02-h-dest",
+    "sewer-water-wave-02-h-dest",
+    "sewer-watefall-froth-01-h-dest",
+    // sewi-water
+    "sewer-water-still-01-i-dest",
+    "sewer-waterfall-02-i-dest",
+    "sewer-water-wave-01-i-dest",
+    // sewj-water
+    "sewer-waterfall-02-j-dest",
+    "sewer-watefall-froth-01-j-dest",
+    // sewl-water
+    "sewer-waterfall-02-l-dest",
+    "sewer-watefall-froth-01-l-dest",
+    // sewm-water
+    "sewer-water-01-m-dest",
+    "sewer-waterfall-01-m-dest",
+    "sewer-waterfall-02-m-dest",
+    "sewer-water-highlight-01-m-dest",
+    "sewer-water-wave-01-m-dest",
+    "sewer-water-still-01-m-dest",
+    "sewer-watefall-froth-01-m-dest",
+    // sewn-water
+    "sewer-waterfall-01-n-dest",
+    "sewer-waterfall-02-n-dest",
+    // "sewer-water-highlight-01-n-dest",
+    "sewer-water-wave-01-n-dest",
+    "sewer-water-still-01-n-dest",
+    // hanga-water
+    // "des-thermal-01-dest",
+    // desresc-warp
+    "sat-shield-dest",
+    // security
+    "security-env-dest",
+    "security-dot-dest",
+    // lgunnorm-water
+    "kg-target-c-forcefield-01-dest",
+    // templex-water
+    "temple-waterfall-dest",
+    // desertd-water
+    "des-waterfall-dest",
+    // towerb-water
+    "tow-energy-bridge-dest",
+    // factoryb-water
+    "hemi-gradient-dest",
+    "hemi-gradient-flames-dest",
+    // factoryc-alpha
+    "facc-convey-dest",
+    "facc-convey-02-dest",
+    // waspal-water
+    "waspala-water-dest",
+    "waspala-waterfall-dest",
+    // rubblea-water
+    "rub-water-dest",
+    // rubblea2-water
+    "rub-water-desta2",
+    // rubbleb-water
+    "rub-water-destb",
+    // rubblec-water
+    "rub-water-destc",
+    // nstb-quicksand
+    "nstb-quicksand-dest",
+    // ctyslumb-water
+    "ctyslumb-water-dest",
+    "ctyslumb-fountain-fall-dest",
+    // ctyslumc-water
+    "ctyslumc-water-dest",
+    "ctyslumc-fountain-fall-dest",
+    // mined-tfrag
+    "mined-pillar-side-dest",
+    "mined-pillar-top2side-dest",
+    "mined-pillar-top-dest",
+    // volcanoa-alpha
+    "vola-lava-01-dest",
+    "vola-lava-fall-dest",
+    // wasstada-alpha
+    "wstd-lava-base-dest",
+};
+
+}  // namespace
 
 const std::vector<std::string>& jak2_animated_texture_slots() {
   return jak2_slots;
 }
+
+const std::vector<std::string>& jak3_animated_texture_slots() {
+  return jak3_slots;
+}

common/texture/texture_slots.h

@@ -4,3 +4,4 @@
 #include <vector>
 
 const std::vector<std::string>& jak2_animated_texture_slots();
+const std::vector<std::string>& jak3_animated_texture_slots();

common/type_system/Type.cpp

@@ -10,7 +10,7 @@
 #include "common/log/log.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace {
 std::string reg_kind_to_string(RegClass kind) {
@@ -421,10 +421,12 @@ int Type::get_num_methods() const {
  * Add a method defined specifically for this type.
  */
 const MethodInfo& Type::add_method(const MethodInfo& info) {
-  for (auto it = m_methods.rbegin(); it != m_methods.rend(); it++) {
-    if (!it->overrides_parent && !it->only_overrides_docstring) {
-      ASSERT(it->id + 1 == info.id);
-      break;
+  if (!info.overrides_parent) {
+    for (auto it = m_methods.rbegin(); it != m_methods.rend(); it++) {
+      if (!it->overrides_parent && !it->only_overrides_docstring) {
+        ASSERT(it->id + 1 == info.id);
+        break;
+      }
     }
   }
 

common/type_system/Type.h

@@ -28,10 +28,15 @@ struct MethodInfo {
   std::string name;
   TypeSpec type;
   std::string defined_in_type;
+  // TODO - this might be redundant, but preventing an unknown breaking change, clean it up later
+  std::string type_name;
   bool no_virtual = false;
+  // TODO - this is no longer used anymore...but was not removed from th4e struct?
   bool overrides_parent = false;
+  // TODO - this is no longer used anymore...but was not removed from th4e struct?
   bool only_overrides_docstring = false;
   std::optional<std::string> docstring;
+  std::optional<std::string> overlay_name;
 
   bool operator==(const MethodInfo& other) const;
   bool operator!=(const MethodInfo& other) const { return !((*this) == other); }
@@ -238,6 +243,9 @@ class Field {
   int offset() const { return m_offset; }
   bool skip_in_decomp() const { return m_skip_in_static_decomp; }
   bool user_placed() const { return m_placed_by_user; }
+  void set_comment(const std::string& comment) { m_comment = comment; }
+  const std::string& comment() const { return m_comment; }
+  bool has_comment() const { return !m_comment.empty(); }
   bool operator==(const Field& other) const;
   bool operator!=(const Field& other) const { return !((*this) == other); }
   std::string diff(const Field& other) const;
@@ -274,6 +282,7 @@ class Field {
   int m_alignment = -1;
   bool m_skip_in_static_decomp = false;
   bool m_placed_by_user = false;  // was this field placed manually by the user?
+  std::string m_comment;          // optional comment placed next to the field
 
   double m_field_score = 0.;
 

common/type_system/TypeFieldLookup.cpp

@@ -9,7 +9,7 @@
 
 #include "common/log/log.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace {
 // debug prints for the reverse lookup

common/type_system/TypeSpec.cpp

@@ -7,7 +7,7 @@
 
 #include <stdexcept>
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 bool TypeTag::operator==(const TypeTag& other) const {
   return name == other.name && value == other.value;

common/type_system/TypeSpec.h

@@ -124,10 +124,12 @@ class TypeSpec {
     ASSERT(m_arguments);
     return m_arguments->at(idx);
   }
+
   TypeSpec& get_arg(int idx) {
     ASSERT(m_arguments);
     return m_arguments->at(idx);
   }
+
   const TypeSpec& last_arg() const {
     ASSERT(m_arguments);
     ASSERT(!m_arguments->empty());

common/type_system/TypeSystem.cpp

@@ -14,8 +14,8 @@
 #include "common/util/Assert.h"
 #include "common/util/math_util.h"
 
-#include "third-party/fmt/color.h"
-#include "third-party/fmt/core.h"
+#include "fmt/color.h"
+#include "fmt/core.h"
 
 namespace {
 template <typename... Args>
@@ -28,7 +28,7 @@ template <typename... Args>
   }
 
   throw std::runtime_error(
-      fmt::format("Type Error: {}", fmt::format(str, std::forward<Args>(args)...)));
+      fmt::format("Type Error: {}", fmt::format(fmt::runtime(str), std::forward<Args>(args)...)));
 }
 }  // namespace
 
@@ -451,6 +451,25 @@ Type* TypeSystem::lookup_type(const TypeSpec& ts) const {
   return lookup_type(ts.base_type());
 }
 
+/*!
+ * Same as lookup_type, but returns null instead of throwing.
+ */
+Type* TypeSystem::lookup_type_no_throw(const std::string& name) const {
+  auto kv = m_types.find(name);
+  if (kv != m_types.end()) {
+    return kv->second.get();
+  }
+
+  return nullptr;
+}
+
+/*!
+ * Same as lookup_type, but returns null instead of throwing.
+ */
+Type* TypeSystem::lookup_type_no_throw(const TypeSpec& ts) const {
+  return lookup_type_no_throw(ts.base_type());
+}
+
 /*!
  * Get type info. If the type is not fully defined (ie, we are parsing its deftype now) and its
  * forward defined as a basic or structure, just get basic/structure.
@@ -516,18 +535,25 @@ int TypeSystem::get_load_size_allow_partial_def(const TypeSpec& ts) const {
 }
 
 MethodInfo TypeSystem::override_method(Type* type,
-                                       const std::string& /*type_name*/,
-                                       const int method_id,
+                                       const std::string& method_name,
                                        const std::optional<std::string>& docstring) {
   // Lookup the method from the parent type
   MethodInfo existing_info;
-  bool exists = try_lookup_method(type->get_parent(), method_id, &existing_info);
+  bool exists = try_lookup_method(type->get_parent(), method_name, &existing_info);
   if (!exists) {
-    throw_typesystem_error("Trying to use override a method that has no parent declaration");
+    throw_typesystem_error("Trying to override a method that has no parent declaration");
   }
   // use the existing ID.
-  return type->add_method({existing_info.id, existing_info.name, existing_info.type,
-                           type->get_name(), existing_info.no_virtual, false, true, docstring});
+  return type->add_method({existing_info.id,
+                           existing_info.name,
+                           existing_info.type,
+                           type->get_name(),
+                           type->get_name(),
+                           existing_info.no_virtual,
+                           false,
+                           true,
+                           docstring,
+                           {}});
 }
 
 MethodInfo TypeSystem::declare_method(const std::string& type_name,
@@ -554,8 +580,7 @@ MethodInfo TypeSystem::declare_method(Type* type,
                                       const std::optional<std::string>& docstring,
                                       bool no_virtual,
                                       const TypeSpec& ts,
-                                      bool override_type,
-                                      int id) {
+                                      bool override_type) {
   if (method_name == "new") {
     if (override_type) {
       throw_typesystem_error("Cannot use :replace option with a new method.");
@@ -569,7 +594,7 @@ MethodInfo TypeSystem::declare_method(Type* type,
 
   if (override_type) {
     if (!got_existing) {
-      if (id != -1 && try_lookup_method(type->get_parent(), id, &existing_info)) {
+      if (try_lookup_method(type->get_parent(), method_name, &existing_info)) {
       } else {
         throw_typesystem_error(
             "Cannot use :replace on method {} of {} because this method was not previously "
@@ -579,8 +604,16 @@ MethodInfo TypeSystem::declare_method(Type* type,
     }
 
     // use the existing ID.
-    return type->add_method(
-        {existing_info.id, method_name, ts, type->get_name(), no_virtual, true, false, docstring});
+    return type->add_method({existing_info.id,
+                             method_name,
+                             ts,
+                             type->get_name(),
+                             type->get_name(),
+                             no_virtual,
+                             true,
+                             false,
+                             docstring,
+                             {}});
   } else {
     if (got_existing) {
       // make sure we aren't changing anything.
@@ -610,12 +643,48 @@ MethodInfo TypeSystem::declare_method(Type* type,
       return existing_info;
     } else {
       // add a new method!
-      return type->add_method({get_next_method_id(type), method_name, ts, type->get_name(),
-                               no_virtual, false, false, docstring});
+      return type->add_method({get_next_method_id(type),
+                               method_name,
+                               ts,
+                               type->get_name(),
+                               type->get_name(),
+                               no_virtual,
+                               false,
+                               false,
+                               docstring,
+                               {}});
     }
   }
 }
 
+/*!
+ * Adds a new method that is overlayed on top of a different, existing method.
+ * This should be used basically never (happens once in Jak 1).
+ */
+MethodInfo TypeSystem::overlay_method(Type* type,
+                                      const std::string& method_name,
+                                      const std::string& method_overlay_name,
+                                      const std::optional<std::string>& docstring,
+                                      const TypeSpec& ts) {
+  // look up the method
+  MethodInfo existing_info;
+  bool got_existing = try_lookup_method(type, method_overlay_name, &existing_info);
+
+  if (!got_existing) {
+    if (try_lookup_method(type->get_parent(), method_overlay_name, &existing_info)) {
+    } else {
+      throw_typesystem_error(
+          "Cannot use :overlay-at on method {} of {} because this method was not previously "
+          "declared in a parent.",
+          method_overlay_name, type->get_name());
+    }
+  }
+
+  // use the existing ID.
+  return type->add_method({existing_info.id, method_name, ts, type->get_name(), type->get_name(),
+                           false, true, false, docstring, std::make_optional(method_overlay_name)});
+}
+
 MethodInfo TypeSystem::define_method(const std::string& type_name,
                                      const std::string& method_name,
                                      const TypeSpec& ts,
@@ -643,8 +712,10 @@ MethodInfo TypeSystem::define_method(Type* type,
   // look up the method
   MethodInfo existing_info;
   bool got_existing = try_lookup_method(type, method_name, &existing_info);
-
   if (got_existing) {
+    // The lookup will return a parents method, but the type should be equal
+    // to the type in question (it's a child)
+    existing_info.type_name = type->get_name();
     // Update the docstring
     existing_info.docstring = docstring;
     int bad_arg_idx = -99;
@@ -691,7 +762,8 @@ MethodInfo TypeSystem::add_new_method(Type* type,
 
     return existing;
   } else {
-    return type->add_new_method({0, "new", ts, type->get_name(), false, false, false, docstring});
+    return type->add_new_method(
+        {0, "new", ts, type->get_name(), type->get_name(), false, false, false, docstring, {}});
   }
 }
 
@@ -1027,6 +1099,7 @@ void TypeSystem::add_builtin_types(GameVersion version) {
       symbol_type = add_builtin_basic("basic", "symbol");
       break;
     case GameVersion::Jak2:
+    case GameVersion::Jak3:
       symbol_type = add_builtin_structure("object", "symbol", true);
       symbol_type->override_offset(1);
       break;
@@ -1374,6 +1447,21 @@ std::vector<std::string> TypeSystem::search_types_by_parent_type(
   return results;
 }
 
+std::vector<std::string> TypeSystem::search_types_by_parent_type_strict(
+    const std::string& parent_type) {
+  std::vector<std::string> results = {};
+  for (const auto& [type_name, type_info] : m_types) {
+    // Only NullType's have no parent
+    if (!type_info->has_parent()) {
+      continue;
+    }
+    if (type_info->get_parent() == parent_type) {
+      results.push_back(type_name);
+    }
+  }
+  return results;
+}
+
 std::vector<std::string> TypeSystem::search_types_by_minimum_method_id(
     const int minimum_method_id,
     const std::optional<std::vector<std::string>>& existing_matches) {
@@ -1414,8 +1502,7 @@ std::vector<std::string> TypeSystem::search_types_by_size(
     }
   } else {
     for (const auto& [type_name, type_info] : m_types) {
-      // Only NullType's have no parent
-      if (!type_info->has_parent()) {
+      if (dynamic_cast<NullType*>(type_info.get())) {
         continue;
       }
       const auto size_of_type = m_types[type_name]->get_size_in_memory();
@@ -1884,30 +1971,36 @@ std::string TypeSystem::generate_deftype_footer(const Type* type) const {
     }
   }
 
-  if (type->heap_base()) {
+  if (type->heap_base() &&
+      type->heap_base() !=
+          ((type->get_size_in_memory() - get_type_of_type<BasicType>("process")->size() + 0xf) &
+           ~0xf)) {
+    // don't print if auto heap-base does the job
     result.append(fmt::format("  :heap-base #x{:x}\n", type->heap_base()));
   }
 
   auto method_count = get_next_method_id(type);
-  result.append(fmt::format("  :method-count-assert {}\n", get_next_method_id(type)));
-  result.append(fmt::format("  :size-assert         #x{:x}\n", type->get_size_in_memory()));
+  // result.append(fmt::format("  :method-count-assert {}\n", get_next_method_id(type)));
+  // result.append(fmt::format("  :size-assert         #x{:x}\n", type->get_size_in_memory()));
   TypeFlags flags;
   flags.heap_base = type->heap_base();
   flags.size = type->get_size_in_memory();
   flags.pad = 0;
   flags.methods = method_count;
 
-  result.append(fmt::format("  :flag-assert         #x{:x}\n  ", flags.flag));
+  // result.append(fmt::format("  :flag-assert         #x{:x}\n", flags.flag));
   if (!type->gen_inspect()) {
-    result.append(":no-inspect\n  ");
+    result.append("  :no-inspect\n  ");
   }
 
   std::string methods_string;
+  std::string state_methods_string;
+  std::string states_string;
 
   // New Method
   auto new_info = type->get_new_method_defined_for_type();
   if (new_info) {
-    methods_string.append("(new (");
+    methods_string.append("    (new (");
     for (size_t i = 0; i < new_info->type.arg_count() - 1; i++) {
       methods_string.append(new_info->type.get_arg(i).print());
       if (i != new_info->type.arg_count() - 2) {
@@ -1915,24 +2008,41 @@ std::string TypeSystem::generate_deftype_footer(const Type* type) const {
       }
     }
     methods_string.append(
-        fmt::format(") {} ", new_info->type.get_arg(new_info->type.arg_count() - 1).print(), 0));
+        fmt::format(") {}", new_info->type.get_arg(new_info->type.arg_count() - 1).print(), 0));
 
     auto behavior = new_info->type.try_get_tag("behavior");
     if (behavior) {
-      methods_string.append(fmt::format(":behavior {} ", *behavior));
+      methods_string.append(fmt::format(" :behavior {}", *behavior));
     }
 
-    methods_string.append("0)\n    ");
+    methods_string.append(")\n");
   }
 
   // Rest of methods
+  bool done_with_state_methods = false;  // TODO fix this... this depends on the order of m_methods
   for (auto& info : type->get_methods_defined_for_type()) {
+    if (!done_with_state_methods && info.type.base_type() == "state" && !info.overrides_parent) {
+      if (info.type.arg_count() > 1) {
+        state_methods_string.append(fmt::format("    ({}", info.name));
+        for (size_t i = 0; i < info.type.arg_count() - 1; ++i) {
+          state_methods_string.push_back(' ');
+          state_methods_string.append(info.type.get_arg(i).print());
+        }
+        state_methods_string.append(")\n");
+      } else {
+        state_methods_string.append(fmt::format("    {}\n", info.name));
+      }
+      continue;
+    } else {
+      done_with_state_methods = true;
+    }
+
     // check if we only override the docstring
     if (info.only_overrides_docstring) {
       continue;
     }
 
-    methods_string.append(fmt::format("({} (", info.name));
+    methods_string.append(fmt::format("    ({} (", info.name));
     for (size_t i = 0; i < info.type.arg_count() - 1; i++) {
       methods_string.append(info.type.get_arg(i).print());
       if (i != info.type.arg_count() - 2) {
@@ -1940,35 +2050,32 @@ std::string TypeSystem::generate_deftype_footer(const Type* type) const {
       }
     }
     methods_string.append(
-        fmt::format(") {} ", info.type.get_arg(info.type.arg_count() - 1).print()));
-
-    if (info.no_virtual) {
-      methods_string.append(":no-virtual ");
-    }
-
-    if (info.overrides_parent) {
-      methods_string.append(":replace ");
-    }
+        fmt::format(") {}", info.type.get_arg(info.type.arg_count() - 1).print()));
 
     auto behavior = info.type.try_get_tag("behavior");
     if (behavior) {
-      methods_string.append(fmt::format(":behavior {} ", *behavior));
+      methods_string.append(fmt::format(" :behavior {}", *behavior));
     }
 
     if (info.type.base_type() == "state") {
-      methods_string.append(":state ");
+      methods_string.append(" :state");
     }
 
-    methods_string.append(fmt::format("{})\n    ", info.id));
+    if (info.no_virtual) {
+      methods_string.append(" :no-virtual");
+    }
+
+    if (info.overrides_parent) {
+      if (info.overlay_name.has_value()) {
+        methods_string.append(fmt::format(" :overlay-at {}", *info.overlay_name));
+      } else {
+        methods_string.append(" :replace");
+      }
+    }
+
+    methods_string.append(fmt::format(")\n", info.id));
   }
 
-  if (!methods_string.empty()) {
-    result.append("(:methods\n    ");
-    result.append(methods_string);
-    result.append(")\n  ");
-  }
-
-  std::string states_string;
   for (auto& info : type->get_states_declared_for_type()) {
     if (info.second.arg_count() > 1) {
       states_string.append(fmt::format("    ({}", info.first));
@@ -1982,16 +2089,163 @@ std::string TypeSystem::generate_deftype_footer(const Type* type) const {
     }
   }
 
-  if (!states_string.empty()) {
-    result.append("(:states\n");
-    result.append(states_string);
-    result.append("    )\n  ");
+  if (!state_methods_string.empty()) {
+    result.append("  (:state-methods\n");
+    result.append(state_methods_string);
+    result.append("    )\n");
   }
 
-  result.append(")\n");
+  if (!methods_string.empty()) {
+    result.append("  (:methods\n");
+    result.append(methods_string);
+    result.append("    )\n");
+  }
+
+  if (!states_string.empty()) {
+    result.append("  (:states\n");
+    result.append(states_string);
+    result.append("    )\n");
+  }
+
+  result.append("  )\n");
   return result;
 }
 
+std::optional<std::string> find_best_field_in_structure(const TypeSystem& ts,
+                                                        const StructureType* st,
+                                                        int offset,
+                                                        const Field& requesting_field,
+                                                        bool want_fixed,
+                                                        int start_field,
+                                                        int end_field = -1) {
+  // performs best field lookup within a structure, at an offset.
+  const Field* best_val = nullptr;
+  const Field* best_exact = nullptr;
+  const Field* best_struct = nullptr;
+  std::pair<const Field*, int> best_val_arr = {nullptr, -1};
+  std::pair<const Field*, int> best_exact_arr = {nullptr, -1};
+  std::pair<const Field*, int> best_struct_arr = {nullptr, -1};
+  std::optional<std::string> best_struct_field_deref;
+  const Field* best = nullptr;
+  if (end_field == -1) {
+    end_field = st->fields().size();
+  }
+  for (size_t i = start_field; i < (size_t)end_field; ++i) {
+    const auto& field = st->fields().at(i);
+    auto type = ts.lookup_type_allow_partial_def(field.type());
+    if (field.is_dynamic() || field.offset() > offset || field.user_placed() != want_fixed) {
+      continue;
+    }
+    if (!field.is_array()) {
+      if (!field.is_inline() && field.offset() + type->get_load_size() > offset) {
+        if (field.offset() == offset) {
+          // not array, not inline - can fit in register, only check exact offset.
+          if (!best_val ||
+              type->get_load_size() == ts.lookup_type(requesting_field.type())->get_load_size()) {
+            best_val = &field;
+          }
+        }
+      } else if (field.is_inline() && field.offset() + type->get_size_in_memory() > offset) {
+        if (field.type() == requesting_field.type() && field.offset() == offset) {
+          // not array, inlined and exact same as this field, just overlay directly on top
+          best_exact = &field;
+        } else {
+          auto f_type = dynamic_cast<StructureType*>(type);
+          if (f_type) {
+            // struct that encompasses this field
+            // simply search that structure for the field we want, offset by the field's offset
+            auto best_field_in_struct = find_best_field_in_structure(
+                ts, f_type, offset - field.offset(), requesting_field, want_fixed, 0);
+            if (best_field_in_struct) {
+              best_struct_field_deref = best_field_in_struct;
+              best_struct = &field;
+            }
+          }
+        }
+      }
+    } else {
+      int rel_offset = offset - field.offset();
+      // array case (and array encompasses what we want)
+      ASSERT_MSG(
+          type->get_size_in_memory() > 0,
+          fmt::format(
+              "In type {}, type size was 0 for array field {} of type {}, check that the type "
+              "is fully defined.",
+              st->get_name(), field.name(), field.type().print()));
+      int array_idx = rel_offset / type->get_size_in_memory();
+      if (!field.is_inline() &&
+          field.offset() + field.array_size() * type->get_load_size() > offset) {
+        if (rel_offset % type->get_load_size() == 0) {
+          // found exact match for array index
+          if (!best_val_arr.first ||
+              type->get_load_size() == ts.lookup_type(requesting_field.type())->get_load_size()) {
+            best_val_arr.first = &field;
+            best_val_arr.second = rel_offset / type->get_load_size();
+          }
+        }
+      } else if (field.is_inline() &&
+                 field.offset() + field.array_size() * type->get_size_in_memory() > offset) {
+        if (field.type() == requesting_field.type() &&
+            rel_offset % type->get_size_in_memory() == 0) {
+          // same type
+          best_exact_arr.first = &field;
+          best_exact_arr.second = array_idx;
+        } else if (requesting_field.is_array() && rel_offset % type->get_size_in_memory() == 0 &&
+                   array_idx == 0) {
+          // starts at the same offset as another array. just use the field with nothing extra
+          best_exact = &field;
+        } else {
+          auto f_type = dynamic_cast<StructureType*>(type);
+          if (f_type && field.offset() + f_type->get_size_in_memory() > offset) {
+            // struct that encompasses this field
+            // simply search that structure for the field we want, offset by the field's offset
+            auto best_field_in_struct =
+                find_best_field_in_structure(ts, f_type, rel_offset % type->get_size_in_memory(),
+                                             requesting_field, want_fixed, 0);
+            if (best_field_in_struct) {
+              best_struct_field_deref = best_field_in_struct;
+              best_struct_arr.first = &field;
+              best_struct_arr.second = array_idx;
+            }
+          }
+        }
+      }
+    }
+  }
+
+  int best_array_idx = -1;
+  if (best_exact) {
+    best = best_exact;
+  } else if (best_exact_arr.first) {
+    best = best_exact_arr.first;
+    best_array_idx = best_exact_arr.second;
+  } else if (best_val) {
+    best = best_val;
+  } else if (best_val_arr.first) {
+    best = best_val_arr.first;
+    best_array_idx = best_val_arr.second;
+  } else if (best_struct) {
+    best = best_struct;
+  } else if (best_struct_arr.first) {
+    best = best_struct_arr.first;
+    best_array_idx = best_struct_arr.second;
+  }
+  if (best) {
+    auto ret =
+        best_array_idx == -1 ? best->name() : fmt::format("{} {}", best->name(), best_array_idx);
+    if (best == best_struct || best == best_struct_arr.first) {
+      return ret + " " + *best_struct_field_deref;
+    } else {
+      return ret;
+    }
+  } else if (!want_fixed) {
+    // try again but with a user-placed offset
+    return find_best_field_in_structure(ts, st, offset, requesting_field, true, start_field,
+                                        end_field);
+  }
+  return {};
+}
+
 std::string TypeSystem::generate_deftype_for_structure(const StructureType* st) const {
   std::string result;
   result += fmt::format("(deftype {} ({})\n", st->get_name(), st->get_parent());
@@ -2003,10 +2257,10 @@ std::string TypeSystem::generate_deftype_for_structure(const StructureType* st)
   int longest_field_name = 0;
   int longest_type_name = 0;
   int longest_mods = 0;
+  int longest_mods_with_user_placed = 0;
 
   const std::string inline_string = ":inline";
   const std::string dynamic_string = ":dynamic";
-  bool has_offset_assert = false;
 
   // calculate longest strings needed, for basic linting
 
@@ -2020,33 +2274,33 @@ std::string TypeSystem::generate_deftype_for_structure(const StructureType* st)
   // normal fields
   for (size_t i = st->first_unique_field_idx(); i < st->fields().size(); i++) {
     const auto& field = st->fields().at(i);
-    longest_field_name = std::max(longest_field_name, int(field.name().size()));
-    longest_type_name = std::max(longest_type_name, int(field.type().print().size()));
 
     int mods = 0;
     // mods are array size, :inline, :dynamic
     if (field.is_array() && !field.is_dynamic()) {
+      mods++;
       mods += std::to_string(field.array_size()).size();
     }
 
     if (field.is_inline()) {
-      if (mods) {
-        mods++;  // space
-      }
+      mods++;  // space
       mods += inline_string.size();
     }
 
     if (field.is_dynamic()) {
-      if (mods) {
-        mods++;  // space
-      }
+      mods++;  // space
       mods += dynamic_string.size();
     }
 
-    if (!field.user_placed()) {
-      has_offset_assert = true;
+    longest_field_name = std::max(longest_field_name, int(field.name().size()));
+    if (mods > 0 || field.user_placed()) {
+      // this is only relevant for fields that have mods
+      longest_type_name = std::max(longest_type_name, int(field.type().print().size()));
     }
     longest_mods = std::max(longest_mods, mods);
+    if (field.user_placed()) {
+      longest_mods_with_user_placed = std::max(longest_mods_with_user_placed, mods);
+    }
   }
 
   // now actually write out the fields
@@ -2056,10 +2310,9 @@ std::string TypeSystem::generate_deftype_for_structure(const StructureType* st)
     const auto& field = st->fields().at(i);
     result += "(";
     result += field.name();
-    result.append(1 + (longest_field_name - int(field.name().size())), ' ');
+    result.append(2 + (longest_field_name - int(field.name().size())), ' ');
     result += field.type().print();
     result.append(1 + (longest_type_name - int(field.type().print().size())), ' ');
-    result.append(1 + longest_mods, ' ');
     result.append(":override)\n   ");
   }
 
@@ -2068,40 +2321,51 @@ std::string TypeSystem::generate_deftype_for_structure(const StructureType* st)
     const auto& field = st->fields().at(i);
     result += "(";
     result += field.name();
-    result.append(1 + (longest_field_name - int(field.name().size())), ' ');
+    result.append(2 + (longest_field_name - int(field.name().size())), ' ');
     result += field.type().print();
-    result.append(1 + (longest_type_name - int(field.type().print().size())), ' ');
 
     std::string mods;
     if (field.is_array() && !field.is_dynamic()) {
-      mods += std::to_string(field.array_size());
       mods += " ";
+      mods += std::to_string(field.array_size());
     }
 
     if (field.is_inline()) {
-      mods += inline_string;
       mods += " ";
+      mods += inline_string;
     }
 
     if (field.is_dynamic()) {
-      mods += dynamic_string;
       mods += " ";
+      mods += dynamic_string;
     }
 
+    if (!mods.empty()) {
+      result.append(1 + longest_type_name - int(field.type().print().size()), ' ');
+    }
     result.append(mods);
-    result.append(longest_mods - int(mods.size() - 1), ' ');
 
-    if (!field.user_placed()) {
-      result.append(":offset-assert ");
-    } else {
-      if (has_offset_assert) {
-        result.append(":offset        ");
+    if (field.user_placed()) {
+      result.append(longest_mods_with_user_placed - int(mods.size()), ' ');
+      if (mods.empty()) {
+        result.append(1 + longest_type_name - int(field.type().print().size()), ' ');
+      }
+      // find best field for :overlay-at
+      // we find the first field that does not come after the current one
+      // and either use it, or check if one of its fields (recursively) is appropriate
+      // we also check for array offsets. we ALSO do bounds-checking!
+      // non-fixed offset fields get priority! dynamic fields are IGNORED.
+      // if all else fails, print as fixed offset.
+      auto best_match = find_best_field_in_structure(*this, st, field.offset(), field, false, 0, i);
+      if (!best_match) {
+        result.append(fmt::format(" :offset {:3d}", field.offset()));
+      } else if (best_match->find(' ') == std::string::npos) {
+        result.append(fmt::format(" :overlay-at {}", *best_match));
       } else {
-        result.append(":offset ");
+        result.append(fmt::format(" :overlay-at (-> {})", *best_match));
       }
     }
 
-    result.append(fmt::format("{:3d}", field.offset()));
     result.append(")\n   ");
   }
 

common/type_system/TypeSystem.h

@@ -155,14 +155,16 @@ class TypeSystem {
   Type* lookup_type(const TypeSpec& ts) const;
   Type* lookup_type(const std::string& name) const;
 
+  Type* lookup_type_no_throw(const TypeSpec& ts) const;
+  Type* lookup_type_no_throw(const std::string& name) const;
+
   Type* lookup_type_allow_partial_def(const TypeSpec& ts) const;
   Type* lookup_type_allow_partial_def(const std::string& name) const;
 
   int get_load_size_allow_partial_def(const TypeSpec& ts) const;
 
   MethodInfo override_method(Type* type,
-                             const std::string& type_name,
-                             const int method_id,
+                             const std::string& method_name,
                              const std::optional<std::string>& docstring);
   MethodInfo declare_method(const std::string& type_name,
                             const std::string& method_name,
@@ -175,8 +177,12 @@ class TypeSystem {
                             const std::optional<std::string>& docstring,
                             bool no_virtual,
                             const TypeSpec& ts,
-                            bool override_type,
-                            int id = -1);
+                            bool override_type);
+  MethodInfo overlay_method(Type* type,
+                            const std::string& method_name,
+                            const std::string& method_overlay_name,
+                            const std::optional<std::string>& docstring,
+                            const TypeSpec& ts);
   MethodInfo define_method(const std::string& type_name,
                            const std::string& method_name,
                            const TypeSpec& ts,
@@ -272,6 +278,7 @@ class TypeSystem {
   std::vector<std::string> search_types_by_parent_type(
       const std::string& parent_type,
       const std::optional<std::vector<std::string>>& existing_matches = {});
+  std::vector<std::string> search_types_by_parent_type_strict(const std::string& parent_type);
 
   std::vector<std::string> search_types_by_minimum_method_id(
       const int minimum_method_id,

common/type_system/defenum.cpp

@@ -13,7 +13,7 @@
 #include "common/util/BitUtils.h"
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace {
 const goos::Object& car(const goos::Object* x) {
@@ -38,7 +38,7 @@ bool is_type(const std::string& expected, const TypeSpec& actual, const TypeSyst
 
 std::string symbol_string(const goos::Object& obj) {
   if (obj.is_symbol()) {
-    return obj.as_symbol()->name;
+    return obj.as_symbol().name_ptr;
   }
   throw std::runtime_error(obj.print() + " was supposed to be a symbol, but isn't");
 }
@@ -58,10 +58,11 @@ EnumType* parse_defenum(const goos::Object& defenum,
   auto& enum_name_obj = car(iter);
   iter = cdr(iter);
   // check for docstring
+  std::optional<std::string> maybe_docstring;
   if (iter->is_pair() && car(iter).is_string()) {
-    // TODO - docstring - store and use docstring if coming from the compiler
     if (symbol_metadata) {
-      symbol_metadata->docstring = str_util::trim_newline_indents(car(iter).as_string()->data);
+      maybe_docstring = str_util::trim_newline_indents(car(iter).as_string()->data);
+      symbol_metadata->docstring = maybe_docstring.value();
     }
     iter = cdr(iter);
   }
@@ -69,7 +70,7 @@ EnumType* parse_defenum(const goos::Object& defenum,
   if (!enum_name_obj.is_symbol()) {
     throw std::runtime_error("defenum must be given a symbol as its name");
   }
-  std::string name = enum_name_obj.as_symbol()->name;
+  std::string name = enum_name_obj.as_symbol().name_ptr;
 
   auto current = car(iter);
   while (current.is_symbol() && symbol_string(current).at(0) == ':') {
@@ -157,6 +158,7 @@ EnumType* parse_defenum(const goos::Object& defenum,
   if (is_type("integer", base_type, ts)) {
     auto parent = ts->get_type_of_type<ValueType>(base_type.base_type());
     auto new_type = std::make_unique<EnumType>(parent, name, is_bitfield, entries);
+    new_type->m_metadata.docstring = maybe_docstring;
     new_type->set_runtime_type(parent->get_runtime_name());
     return dynamic_cast<EnumType*>(ts->add_type(name, std::move(new_type)));
   } else {

common/type_system/deftype.cpp

@@ -13,7 +13,7 @@
 #include "common/type_system/state.h"
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 /*!
  * Missing Features
@@ -54,7 +54,7 @@ std::string deftype_parent_list(const goos::Object& list) {
     throw std::runtime_error("invalid parent in deftype parent list");
   }
 
-  return parent.as_symbol()->name;
+  return parent.as_symbol().name_ptr;
 }
 
 bool is_type(const std::string& expected, const TypeSpec& actual, const TypeSystem* ts) {
@@ -63,7 +63,7 @@ bool is_type(const std::string& expected, const TypeSpec& actual, const TypeSyst
 
 std::string symbol_string(const goos::Object& obj) {
   if (obj.is_symbol()) {
-    return obj.as_symbol()->name;
+    return obj.as_symbol().name_ptr;
   }
   throw std::runtime_error(obj.print() + " was supposed to be a symbol, but isn't");
 }
@@ -87,7 +87,7 @@ double get_float(const goos::Object& obj) {
 void add_field(StructureType* structure,
                TypeSystem* ts,
                const goos::Object& def,
-               std::unordered_map<goos::HeapObject*, goos::Object>& constants) {
+               goos::EnvironmentMap* constants) {
   auto rest = &def;
 
   auto name = symbol_string(car(rest));
@@ -111,9 +111,8 @@ void add_field(StructureType* structure,
     if (car(rest).is_int()) {
       array_size = car(rest).integer_obj.value;
       rest = cdr(rest);
-    } else if (car(rest).is_symbol() &&
-               constants.find((car(rest)).as_symbol()) != constants.end()) {
-      array_size = get_int(constants[(car(rest)).as_symbol()]);
+    } else if (car(rest).is_symbol() && constants && constants->lookup((car(rest)).as_symbol())) {
+      array_size = get_int(*constants->lookup((car(rest)).as_symbol()));
       rest = cdr(rest);
     }
 
@@ -134,14 +133,66 @@ void add_field(StructureType* structure,
           throw std::runtime_error(fmt::format("Field {} not found to override", name));
         }
       } else if (opt_name == ":overlay-at") {
-        auto field_name = symbol_string(car(rest));
-        Field overlay_field;
-        if (!structure->lookup_field(field_name, &overlay_field)) {
-          throw std::runtime_error(
-              fmt::format("Field {} not found to overlay for {}", field_name, name));
-        }
-        offset_override = overlay_field.offset();
+        const auto& param = car(rest);
         rest = cdr(rest);
+        Field overlay_field;
+        if (param.is_symbol()) {
+          auto field_name = symbol_string(param);
+          if (!structure->lookup_field(field_name, &overlay_field)) {
+            throw std::runtime_error(
+                fmt::format("Field {} not found to overlay for {}", field_name, name));
+          }
+          offset_override = overlay_field.offset();
+        } else if (param.is_pair() && car(&param).is_symbol("->")) {
+          auto name_it = cdr(&param);
+          if (name_it->is_empty_list()) {
+            throw std::runtime_error(
+                fmt::format("Field list for overlay-at in {} was empty", name));
+          }
+          auto type_to_use = structure;
+          offset_override = 0;
+          while (!name_it->is_empty_list()) {
+            const auto& deref_field = car(name_it);
+            if (deref_field.is_int()) {
+              auto ref_array_field = !type_to_use && !overlay_field.is_inline()
+                                         ? ts->lookup_type_allow_partial_def(overlay_field.type())
+                                         : nullptr;
+              if (ref_array_field) {
+                // we can have an array of references (non-inline) to a forward-declared type
+                offset_override += ref_array_field->get_load_size() * deref_field.as_int();
+              } else {
+                auto type_to_deref = type_to_use && overlay_field.is_inline()
+                                         ? TypeSpec("inline-array")
+                                         : TypeSpec("pointer");
+                type_to_deref.add_arg(overlay_field.type());
+                auto deref_info = ts->get_deref_info(type_to_deref);
+                if (!deref_info.can_deref) {
+                  throw std::runtime_error(
+                      fmt::format("Array could not be dereferenced for overlay-at in {}", name));
+                }
+                // overlay_field.type() = deref_info.result_type;
+                offset_override += deref_info.stride * deref_field.as_int();
+              }
+            } else {
+              if (!type_to_use) {
+                throw std::runtime_error(
+                    fmt::format("Field {} not inside a structure for overlay-at in {}",
+                                overlay_field.name(), name));
+              }
+              auto field_name = symbol_string(car(name_it));
+              if (!type_to_use->lookup_field(field_name, &overlay_field)) {
+                throw std::runtime_error(
+                    fmt::format("Field {} not found to overlay for {}", field_name, name));
+              }
+              type_to_use =
+                  dynamic_cast<StructureType*>(ts->lookup_type_no_throw(overlay_field.type()));
+              offset_override += overlay_field.offset();
+            }
+            name_it = cdr(name_it);
+          }
+        } else {
+          throw std::runtime_error(fmt::format("Unknown parameter for overlay-at in {}", name));
+        }
       } else if (opt_name == ":score") {
         score = get_float(car(rest));
         rest = cdr(rest);
@@ -274,14 +325,20 @@ void declare_method(Type* type,
     // - this effectively does a :replace without having to re-define the name and signature and
     // keep that in-sync
     std::string method_name;
+    std::string method_overlay_name;
     TypeSpec function_typespec("function");
     std::optional<std::string> docstring;
     goos::Object args;
     goos::Object return_type;
     bool no_virtual = false;
     bool replace_method = false;
+    bool overlay_method = false;
     bool overriding_doc = false;
 
+    // name
+    method_name = symbol_string(car(obj));
+    obj = cdr(obj);
+
     if (!obj->is_empty_list() && car(obj).is_symbol(":override-doc")) {
       obj = cdr(obj);
       if (car(obj).is_string()) {
@@ -294,10 +351,6 @@ void declare_method(Type* type,
     }
 
     if (!overriding_doc) {
-      // name
-      method_name = symbol_string(car(obj));
-      obj = cdr(obj);
-
       // docstring
       if (obj->is_pair() && car(obj).is_string()) {
         docstring = str_util::trim_newline_indents(car(obj).as_string()->data);
@@ -317,25 +370,21 @@ void declare_method(Type* type,
       //
       // Doing it like this makes the ordering not critical
       while (!obj->is_empty_list() && car(obj).is_symbol()) {
-        const auto& keyword = car(obj).as_symbol()->name;
+        const auto& keyword = car(obj).as_symbol();
         if (keyword == ":no-virtual") {
           no_virtual = true;
         } else if (keyword == ":replace") {
           replace_method = true;
         } else if (keyword == ":state") {
-          auto behavior_tag = function_typespec.try_get_tag("behavior");
           function_typespec = TypeSpec("state");
-          if (behavior_tag) {
-            function_typespec.add_new_tag("behavior", behavior_tag.value());
-          }
           // parse state docstrings if available
-          if (car(cdr(obj)).is_list()) {
+          if (!cdr(obj)->is_empty_list() && car(cdr(obj)).is_list()) {
             obj = cdr(obj);
             auto docstring_list = &car(obj);
             auto elem = docstring_list;
             while (!elem->is_empty_list() && car(elem).is_symbol()) {
-              const auto& handler = car(elem).as_symbol()->name;
-              const auto handler_kind = handler_keyword_to_kind(handler);
+              const auto& handler = car(elem).as_symbol();
+              const auto handler_kind = handler_keyword_to_kind(handler.name_ptr);
 
               // Get the docstring
               elem = cdr(elem);
@@ -360,6 +409,13 @@ void declare_method(Type* type,
             throw std::runtime_error("Bad usage of :behavior in a method declaration");
           }
           function_typespec.add_new_tag("behavior", symbol_string(obj->as_pair()->car));
+        } else if (keyword == ":overlay-at") {
+          obj = cdr(obj);
+          if (!car(obj).is_symbol()) {
+            throw std::runtime_error("Invalid parameter to method overlay-at");
+          }
+          method_overlay_name = symbol_string(car(obj));
+          overlay_method = true;
         }
         obj = cdr(obj);
       }
@@ -371,36 +427,62 @@ void declare_method(Type* type,
       function_typespec.add_arg(parse_typespec(type_system, return_type));
     }
 
-    // determine the method id, it should be the last in the list
-    int id = -1;
-    if (!obj->is_empty_list() && car(obj).is_int()) {
-      auto& id_obj = car(obj);
-      id = get_int(id_obj);
-      obj = cdr(obj);
+    if (!obj->is_empty_list()) {
+      throw std::runtime_error(fmt::format("found unknown data in a method declaration:\n{}\n\n{}",
+                                           obj->print(), _obj.print()));
     }
 
-    if (!obj->is_empty_list()) {
-      throw std::runtime_error("found symbols after the `id` in a method defintion: " +
-                               def.print());
+    if (overlay_method && (no_virtual || replace_method)) {
+      throw std::runtime_error(
+          fmt::format("method {} in type {} has invalid combination of keywords", method_name,
+                      type->get_name()));
     }
 
     MethodInfo info;
     if (overriding_doc) {
-      info = type_system->override_method(type, method_name, id, docstring);
+      info = type_system->override_method(type, method_name, docstring);
+    } else if (overlay_method) {
+      info = type_system->overlay_method(type, method_name, method_overlay_name, docstring,
+                                         function_typespec);
     } else {
       info = type_system->declare_method(type, method_name, docstring, no_virtual,
-                                         function_typespec, replace_method, id);
+                                         function_typespec, replace_method);
     }
+  });
+}
 
-    // check the method assert
-    if (id != -1) {
-      // method id assert!
-      if (id != info.id) {
-        lg::print("WARNING - ID assert failed on method {} of type {} (wanted {} got {})\n",
-                  method_name.c_str(), type->get_name().c_str(), id, info.id);
-        throw std::runtime_error("Method ID assert failed");
+void declare_state_methods(Type* type,
+                           TypeSystem* type_system,
+                           const goos::Object& def,
+                           StructureDefResult& /*struct_def*/) {
+  for_each_in_list(def, [&](const goos::Object& _obj) {
+    auto obj = &_obj;
+    // either state-name or (state-name args...) or (state-name "docstring" args...)
+    std::string method_name;
+    TypeSpec function_typespec("state");
+    std::optional<std::string> docstring;
+
+    if (obj->is_symbol()) {
+      method_name = obj->as_symbol().name_ptr;
+    } else if (obj->is_list()) {
+      if (!car(obj).is_symbol()) {
+        throw std::runtime_error(
+            fmt::format("{} is not a valid name for a state-method", obj->print()));
       }
+      method_name = car(obj).as_symbol().name_ptr;
+      auto& args = *cdr(obj);
+      if (car(obj).is_string()) {
+        // docstring first
+        docstring = car(obj).as_string()->data;
+        obj = cdr(obj);
+      }
+      for_each_in_list(args, [&](const goos::Object& o) {
+        function_typespec.add_arg(parse_typespec(type_system, o));
+      });
     }
+    function_typespec.add_arg(TypeSpec("_type_"));
+
+    type_system->declare_method(type, method_name, docstring, false, function_typespec, false);
   });
 }
 
@@ -419,8 +501,8 @@ void declare_state(Type* type,
         auto docstring_list = &car(obj);
         auto elem = docstring_list;
         while (!elem->is_empty_list() && car(elem).is_symbol()) {
-          const auto& handler = car(elem).as_symbol()->name;
-          const auto handler_kind = handler_keyword_to_kind(handler);
+          const auto& handler = car(elem).as_symbol();
+          const auto handler_kind = handler_keyword_to_kind(handler.name_ptr);
 
           // Get the docstring
           elem = cdr(elem);
@@ -458,12 +540,11 @@ void declare_state(Type* type,
   });
 }
 
-StructureDefResult parse_structure_def(
-    StructureType* type,
-    TypeSystem* ts,
-    const goos::Object& fields,
-    const goos::Object& options,
-    std::unordered_map<goos::HeapObject*, goos::Object>& constants) {
+StructureDefResult parse_structure_def(StructureType* type,
+                                       TypeSystem* ts,
+                                       const goos::Object& fields,
+                                       const goos::Object& options,
+                                       goos::EnvironmentMap* constants) {
   StructureDefResult result;
   for_each_in_list(fields, [&](const goos::Object& o) { add_field(type, ts, o, constants); });
   TypeFlags flags;
@@ -489,6 +570,8 @@ StructureDefResult parse_structure_def(
         declare_method(type, ts, *opt_list, result);
       } else if (list_name == ":states") {
         declare_state(type, ts, *opt_list, result);
+      } else if (list_name == ":state-methods") {
+        declare_state_methods(type, ts, *opt_list, result);
       } else {
         throw std::runtime_error("Invalid option list in field specification: " +
                                  car(rest).print());
@@ -686,8 +769,8 @@ TypeSpec parse_typespec(const TypeSystem* type_system, const goos::Object& src)
     while (rest->is_pair()) {
       auto& it = rest->as_pair()->car;
 
-      if (it.is_symbol() && it.as_symbol()->name.at(0) == ':') {
-        auto tag_name = it.as_symbol()->name.substr(1);
+      if (it.is_symbol() && it.as_symbol().name_ptr[0] == ':') {
+        auto tag_name = it.as_symbol().name_ptr + 1;
         rest = &rest->as_pair()->cdr;
 
         if (!rest->is_pair()) {
@@ -696,13 +779,13 @@ TypeSpec parse_typespec(const TypeSystem* type_system, const goos::Object& src)
 
         auto& tag_val = rest->as_pair()->car;
 
-        if (tag_name == "behavior") {
-          if (!type_system->fully_defined_type_exists(tag_val.as_symbol()->name) &&
-              !type_system->partially_defined_type_exists(tag_val.as_symbol()->name)) {
+        if (std::string_view(tag_name) == "behavior") {
+          if (!type_system->fully_defined_type_exists(tag_val.as_symbol().name_ptr) &&
+              !type_system->partially_defined_type_exists(tag_val.as_symbol().name_ptr)) {
             throw std::runtime_error(
-                fmt::format("Behavior tag uses an unknown type {}", tag_val.as_symbol()->name));
+                fmt::format("Behavior tag uses an unknown type {}", tag_val.as_symbol().name_ptr));
           }
-          ts.add_new_tag(tag_name, tag_val.as_symbol()->name);
+          ts.add_new_tag(tag_name, tag_val.as_symbol().name_ptr);
         } else {
           throw std::runtime_error(fmt::format("Type tag {} is unknown", tag_name));
         }
@@ -725,13 +808,9 @@ TypeSpec parse_typespec(const TypeSystem* type_system, const goos::Object& src)
 
 DeftypeResult parse_deftype(const goos::Object& deftype,
                             TypeSystem* ts,
-                            std::unordered_map<goos::HeapObject*, goos::Object>* constants) {
+                            goos::EnvironmentMap* constants) {
   DefinitionMetadata symbol_metadata;
-  std::unordered_map<goos::HeapObject*, goos::Object> no_consts;
-  auto& constants_to_use = no_consts;
-  if (constants != nullptr) {
-    constants_to_use = *constants;
-  }
+  goos::EnvironmentMap no_consts;
 
   auto iter = &deftype;
 
@@ -752,7 +831,7 @@ DeftypeResult parse_deftype(const goos::Object& deftype,
     throw std::runtime_error("deftype must be given a symbol as the type name");
   }
 
-  auto& name = type_name_obj.as_symbol()->name;
+  auto& name = type_name_obj.as_symbol().name_ptr;
   auto parent_type_name = deftype_parent_list(parent_list_obj);
   auto parent_type = ts->make_typespec(parent_type_name);
   DeftypeResult result;
@@ -770,8 +849,7 @@ DeftypeResult parse_deftype(const goos::Object& deftype,
     }
     new_type->inherit(pto);
     ts->forward_declare_type_as(name, pto->get_name());
-    auto sr =
-        parse_structure_def(new_type.get(), ts, field_list_obj, options_obj, constants_to_use);
+    auto sr = parse_structure_def(new_type.get(), ts, field_list_obj, options_obj, constants);
     result.flags = sr.flags;
     result.create_runtime_type = sr.generate_runtime_type;
     structure_result = sr;
@@ -800,8 +878,7 @@ DeftypeResult parse_deftype(const goos::Object& deftype,
     ASSERT(pto);
     new_type->inherit(pto);
     ts->forward_declare_type_as(name, pto->get_name());
-    auto sr =
-        parse_structure_def(new_type.get(), ts, field_list_obj, options_obj, constants_to_use);
+    auto sr = parse_structure_def(new_type.get(), ts, field_list_obj, options_obj, constants);
     result.flags = sr.flags;
     result.create_runtime_type = sr.generate_runtime_type;
     structure_result = sr;

common/type_system/deftype.h

@@ -17,8 +17,7 @@ struct DeftypeResult {
   bool create_runtime_type = true;
 };
 
-DeftypeResult parse_deftype(
-    const goos::Object& deftype,
-    TypeSystem* ts,
-    std::unordered_map<goos::HeapObject*, goos::Object>* constants = nullptr);
+DeftypeResult parse_deftype(const goos::Object& deftype,
+                            TypeSystem* ts,
+                            goos::EnvironmentMap* constants = nullptr);
 TypeSpec parse_typespec(const TypeSystem* type_system, const goos::Object& src);

common/util/Assert.cpp

@@ -16,12 +16,12 @@ void private_assert_failed(const char* expr,
   if (!msg || msg[0] == '\0') {
     std::string log = fmt::format("Assertion failed: '{}'\n\tSource: {}:{}\n\tFunction: {}\n", expr,
                                   file, line, function);
-    lg::die(log);
+    lg::die("{}", log);
   } else {
     std::string log =
         fmt::format("Assertion failed: '{}'\n\tMessage: {}\n\tSource: {}:{}\n\tFunction: {}\n",
                     expr, msg, file, line, function);
-    lg::die(log);
+    lg::die("{}", log);
   }
   abort();
 }

common/util/Assert.h

@@ -36,6 +36,9 @@
 
 #define ASSERT_NOT_REACHED_MSG(STR) \
   (void)((private_assert_failed("not reached", __FILE__, __LINE__, __PRETTY_FUNCTION__, STR), 0))
+
+#define ASSERT_EQ_IMM(EXPR, EXPECTED) \
+  ASSERT_MSG((EXPR) == (EXPECTED), fmt::format("result was {}, expected {}", (EXPR), (EXPECTED)))
 #else
 
 #define ASSERT(EX) ((void)0)

common/util/BinaryReader.h

@@ -7,6 +7,7 @@
 
 #include <cstdint>
 #include <cstring>
+#include <span>
 #include <vector>
 
 #include "common/common_types.h"
@@ -14,28 +15,29 @@
 
 class BinaryReader {
  public:
-  explicit BinaryReader(const std::vector<uint8_t>& _buffer) : m_buffer(_buffer) {}
+  explicit BinaryReader(std::span<const uint8_t> _span) : m_span(_span) {}
 
   template <typename T>
   T read() {
-    ASSERT(m_seek + sizeof(T) <= m_buffer.size());
+    ASSERT(m_seek + sizeof(T) <= m_span.size());
     T obj;
-    memcpy(&obj, m_buffer.data() + m_seek, sizeof(T));
+    memcpy(&obj, m_span.data() + m_seek, sizeof(T));
     m_seek += sizeof(T);
     return obj;
   }
 
   void ffwd(int amount) {
     m_seek += amount;
-    ASSERT(m_seek <= m_buffer.size());
+    ASSERT(m_seek <= m_span.size());
   }
 
-  uint32_t bytes_left() const { return m_buffer.size() - m_seek; }
-  uint8_t* here() { return m_buffer.data() + m_seek; }
+  uint32_t bytes_left() const { return m_span.size() - m_seek; }
+  const uint8_t* here() { return m_span.data() + m_seek; }
+  BinaryReader at(u32 pos) { return BinaryReader(m_span.subspan(pos)); }
   uint32_t get_seek() const { return m_seek; }
   void set_seek(u32 seek) { m_seek = seek; }
 
  private:
-  std::vector<u8> m_buffer;
+  std::span<const u8> m_span;
   uint32_t m_seek = 0;
 };

common/util/DgoReader.cpp

@@ -9,6 +9,7 @@
 #include "dgo_util.h"
 
 #include "common/link_types.h"
+#include "common/log/log.h"
 
 #include "third-party/json.hpp"
 
@@ -25,9 +26,9 @@ DgoReader::DgoReader(std::string file_name, const std::vector<u8>& data)
 
     if (reader.bytes_left() < obj_header.size && i == header.object_count - 1 &&
         obj_header.size - reader.bytes_left() <= 48) {
-      printf(
-          "Warning: final file %s in DGO %s has a size missing %d bytes.  It will be adjusted from "
-          "%d to %d bytes.\n",
+      lg::print(
+          "Warning: final file {} in DGO {} has a size missing {} bytes.  It will be adjusted from "
+          "{} to {} bytes.\n",
           obj_header.name, header.name, obj_header.size - reader.bytes_left(), obj_header.size,
           (int)reader.bytes_left());
       obj_header.size = reader.bytes_left();
@@ -40,7 +41,7 @@ DgoReader::DgoReader(std::string file_name, const std::vector<u8>& data)
 
     entry.unique_name = get_object_file_name(entry.internal_name, reader.here(), obj_header.size);
     if (all_unique_names.find(entry.unique_name) != all_unique_names.end()) {
-      printf("Warning: there are multiple files named %s\n", entry.unique_name.c_str());
+      lg::print("Warning: there are multiple files named {}\n", entry.unique_name.c_str());
       entry.unique_name += '-';
       entry.unique_name += std::to_string(obj_header.size);
     }

common/util/FileUtil.cpp

@@ -22,7 +22,7 @@
 // This disables the use of PCLMULQDQ which is probably ok, but let's just be safe and disable it
 // because nobody will care if png compression is 10% slower.
 #define FPNG_NO_SSE 1
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 #include "third-party/fpng/fpng.cpp"
 #include "third-party/fpng/fpng.h"
 #include "third-party/lzokay/lzokay.hpp"
@@ -57,6 +57,14 @@ fs::path get_user_home_dir() {
 #endif
 }
 
+struct {
+  bool initialized = false;
+  fs::path path_to_data_folder;
+  fs::path user_config_dir_override = "";
+  // by default - if the config dir is overridden, we don't use the default save location
+  bool use_overridden_config_dir_for_saves = true;
+} g_file_path_info;
+
 fs::path get_user_config_dir() {
   fs::path config_base_path;
 #ifdef _WIN32
@@ -80,23 +88,53 @@ fs::path get_user_config_dir() {
 
 fs::path get_user_settings_dir(GameVersion game_version) {
   auto game_version_name = game_version_names[game_version];
-  return get_user_config_dir() / game_version_name / "settings";
+  auto config_dir = get_user_config_dir();
+  if (!g_file_path_info.user_config_dir_override.empty()) {
+    config_dir = g_file_path_info.user_config_dir_override / "OpenGOAL";
+  }
+  return config_dir / game_version_name / "settings";
 }
 
 fs::path get_user_memcard_dir(GameVersion game_version) {
   auto game_version_name = game_version_names[game_version];
-  return get_user_config_dir() / game_version_name / "saves";
+  auto config_dir = get_user_config_dir();
+  if (!g_file_path_info.user_config_dir_override.empty() &&
+      g_file_path_info.use_overridden_config_dir_for_saves) {
+    config_dir = g_file_path_info.user_config_dir_override / "OpenGOAL";
+  }
+  return config_dir / game_version_name / "saves";
+}
+
+fs::path get_user_screenshots_dir(GameVersion game_version) {
+  auto game_version_name = game_version_names[game_version];
+  auto config_dir = get_user_config_dir();
+  if (!g_file_path_info.user_config_dir_override.empty()) {
+    config_dir = g_file_path_info.user_config_dir_override / "OpenGOAL";
+  }
+  return config_dir / game_version_name / "screenshots";
 }
 
 fs::path get_user_misc_dir(GameVersion game_version) {
   auto game_version_name = game_version_names[game_version];
-  return get_user_config_dir() / game_version_name / "misc";
+  auto config_dir = get_user_config_dir();
+  if (!g_file_path_info.user_config_dir_override.empty()) {
+    config_dir = g_file_path_info.user_config_dir_override / "OpenGOAL";
+  }
+  return config_dir / game_version_name / "misc";
 }
 
-struct {
-  bool initialized = false;
-  fs::path path_to_data;
-} gFilePathInfo;
+fs::path get_user_features_dir(GameVersion game_version) {
+  auto game_version_name = game_version_names[game_version];
+  auto config_dir = get_user_config_dir();
+  if (!g_file_path_info.user_config_dir_override.empty()) {
+    config_dir = g_file_path_info.user_config_dir_override / "OpenGOAL";
+  }
+  auto path = config_dir / game_version_name / "features";
+  file_util::create_dir_if_needed(path);
+  return path;
+}
+
+fs::path g_iso_data_directory = "";
 
 /*!
  * Get the path to the current executable.
@@ -147,9 +185,11 @@ std::optional<std::string> try_get_jak_project_path() {
   return try_get_project_path_from_path(get_current_executable_path());
 }
 
-std::optional<fs::path> try_get_data_dir() {
+std::optional<fs::path> try_get_data_dir(bool skip_logs) {
   fs::path my_path = get_current_executable_path();
-  lg::info("Current executable directory - {}", my_path.string());
+  if (!skip_logs) {
+    lg::debug("Current executable directory - {}", my_path.string());
+  }
   auto data_dir = my_path.parent_path() / "data";
   if (fs::exists(data_dir) && fs::is_directory(data_dir)) {
     return std::make_optional(data_dir);
@@ -158,31 +198,38 @@ std::optional<fs::path> try_get_data_dir() {
   }
 }
 
-bool setup_project_path(std::optional<fs::path> project_path_override) {
-  if (gFilePathInfo.initialized) {
+bool setup_project_path(std::optional<fs::path> project_path_override, bool skip_logs) {
+  if (g_file_path_info.initialized) {
     return true;
   }
 
   if (project_path_override) {
-    gFilePathInfo.path_to_data = fs::absolute(project_path_override.value());
-    gFilePathInfo.initialized = true;
-    lg::info("Using explicitly set project path: {}", gFilePathInfo.path_to_data.string());
+    g_file_path_info.path_to_data_folder = fs::absolute(project_path_override.value());
+    g_file_path_info.initialized = true;
+    if (!skip_logs) {
+      lg::debug("Using explicitly set project path: {}",
+                g_file_path_info.path_to_data_folder.string());
+    }
     return true;
   }
 
-  auto data_path = try_get_data_dir();
+  auto data_path = try_get_data_dir(skip_logs);
   if (data_path) {
-    gFilePathInfo.path_to_data = *data_path;
-    gFilePathInfo.initialized = true;
-    lg::info("Using data path: {}", data_path->string());
+    g_file_path_info.path_to_data_folder = *data_path;
+    g_file_path_info.initialized = true;
+    if (!skip_logs) {
+      lg::debug("Using data path: {}", data_path->string());
+    }
     return true;
   }
 
   auto development_repo_path = try_get_jak_project_path();
   if (development_repo_path) {
-    gFilePathInfo.path_to_data = *development_repo_path;
-    gFilePathInfo.initialized = true;
-    lg::info("Using development repo path: {}", *development_repo_path);
+    g_file_path_info.path_to_data_folder = *development_repo_path;
+    g_file_path_info.initialized = true;
+    if (!skip_logs) {
+      lg::debug("Using development repo path: {}", *development_repo_path);
+    }
     return true;
   }
 
@@ -190,9 +237,37 @@ bool setup_project_path(std::optional<fs::path> project_path_override) {
   return false;
 }
 
+void override_user_config_dir(fs::path user_config_dir_override,
+                              bool use_overridden_config_dir_for_saves) {
+  g_file_path_info.user_config_dir_override = user_config_dir_override;
+  g_file_path_info.use_overridden_config_dir_for_saves = use_overridden_config_dir_for_saves;
+}
+
 fs::path get_jak_project_dir() {
-  ASSERT(gFilePathInfo.initialized);
-  return gFilePathInfo.path_to_data;
+  ASSERT(g_file_path_info.initialized);
+  return g_file_path_info.path_to_data_folder;
+}
+
+fs::path get_iso_dir_for_game(GameVersion game_version) {
+  if (!g_iso_data_directory.empty()) {
+    return g_iso_data_directory;
+  }
+  // Find the location based on the game version
+  std::string expected_subdir = "jak1";
+  if (game_version == GameVersion::Jak2) {
+    expected_subdir = "jak2";
+  } else if (game_version == GameVersion::Jak3) {
+    expected_subdir = "jak3";
+  }
+  const auto temp_dir = get_jak_project_dir() / "iso_data" / expected_subdir;
+  if (fs::exists(temp_dir)) {
+    g_iso_data_directory = temp_dir;
+  }
+  return g_iso_data_directory;
+}
+
+void set_iso_data_dir(const fs::path& directory) {
+  g_iso_data_directory = directory;
 }
 
 std::string get_file_path(const std::vector<std::string>& input) {
@@ -381,7 +456,7 @@ std::string split_path_at(const fs::path& path, const std::vector<std::string>&
     split_str += folder + "/";
 #endif
   }
-  const auto& path_str = path.u8string();
+  const auto& path_str = path.string();
   return path_str.substr(path_str.find(split_str) + split_str.length());
 }
 
@@ -636,6 +711,9 @@ std::vector<fs::path> find_files_in_dir(const fs::path& dir, const std::regex& p
 
 std::vector<fs::path> find_files_recursively(const fs::path& base_dir, const std::regex& pattern) {
   std::vector<fs::path> files = {};
+  if (!fs::exists(base_dir)) {
+    return files;
+  }
   for (auto& p : fs::recursive_directory_iterator(base_dir)) {
     if (p.is_regular_file()) {
       if (std::regex_match(p.path().filename().string(), pattern)) {
@@ -692,15 +770,59 @@ void copy_file(const fs::path& src, const fs::path& dst) {
 std::string make_screenshot_filepath(const GameVersion game_version, const std::string& name) {
   std::string file_name;
   if (name.empty()) {
-    file_name = fmt::format("{}_{}.png", version_to_game_name(game_version),
-                            str_util::current_local_timestamp_no_colons());
+    file_name = fmt::format("{}.png", str_util::current_local_timestamp_no_colons());
   } else {
-    file_name = fmt::format("{}_{}_{}.png", version_to_game_name(game_version), name,
-                            str_util::current_local_timestamp_no_colons());
+    file_name = fmt::format("{}.png", name);
   }
-  const auto file_path = file_util::get_file_path({"screenshots", file_name});
+  const auto file_path = get_user_screenshots_dir(game_version) / file_name;
   file_util::create_dir_if_needed_for_file(file_path);
-  return file_path;
+  return file_path.string();
+}
+
+std::string get_majority_file_line_endings(const std::string& file_contents) {
+  size_t lf_count = 0;
+  size_t crlf_count = 0;
+
+  for (size_t i = 0; i < file_contents.size(); ++i) {
+    if (file_contents[i] == '\n') {
+      if (i > 0 && file_contents[i - 1] == '\r') {
+        crlf_count++;
+      } else {
+        lf_count++;
+      }
+    }
+  }
+
+  if (crlf_count > lf_count) {
+    return "\r\n";
+  }
+  return "\n";
+}
+
+std::pair<int, std::string> get_majority_file_line_endings_and_count(
+    const std::string& file_contents) {
+  size_t lf_count = 0;
+  size_t crlf_count = 0;
+
+  for (size_t i = 0; i < file_contents.size(); ++i) {
+    if (file_contents[i] == '\n') {
+      if (i > 0 && file_contents[i - 1] == '\r') {
+        crlf_count++;
+      } else {
+        lf_count++;
+      }
+    }
+  }
+
+  if (crlf_count > lf_count) {
+    return {lf_count + crlf_count, "\r\n"};
+  }
+  return {lf_count + crlf_count, "\n"};
+}
+
+bool is_dir_in_dir(const fs::path& parent, const fs::path& child) {
+  // Check if the parent path is a prefix of the child path
+  return child.has_parent_path() && child.parent_path().lexically_relative(parent) == fs::path(".");
 }
 
 }  // namespace file_util

common/util/FileUtil.h

@@ -31,14 +31,21 @@ fs::path get_user_home_dir();
 fs::path get_user_config_dir();
 fs::path get_user_settings_dir(GameVersion game_version);
 fs::path get_user_memcard_dir(GameVersion game_version);
+fs::path get_user_screenshots_dir(GameVersion game_version);
 fs::path get_user_misc_dir(GameVersion game_version);
+fs::path get_user_features_dir(GameVersion game_version);
 fs::path get_jak_project_dir();
+fs::path get_iso_dir_for_game(GameVersion game_version);
+void set_iso_data_dir(const fs::path& directory);
 
 bool create_dir_if_needed(const fs::path& path);
 bool create_dir_if_needed_for_file(const std::string& path);
 bool create_dir_if_needed_for_file(const fs::path& path);
+std::string get_current_executable_path();
 std::optional<std::string> try_get_project_path_from_path(const std::string& path);
-bool setup_project_path(std::optional<fs::path> project_path_override);
+bool setup_project_path(std::optional<fs::path> project_path_override, bool skip_logs = false);
+void override_user_config_dir(fs::path user_config_dir_override,
+                              bool use_overridden_config_dir_for_saves);
 std::string get_file_path(const std::vector<std::string>& path);
 void write_binary_file(const std::string& name, const void* data, size_t size);
 void write_binary_file(const fs::path& name, const void* data, size_t size);
@@ -69,4 +76,8 @@ std::vector<fs::path> sort_filepaths(const std::vector<fs::path>& paths, const b
 /// Will overwrite the destination if it exists
 void copy_file(const fs::path& src, const fs::path& dst);
 std::string make_screenshot_filepath(const GameVersion game_version, const std::string& name = "");
+std::string get_majority_file_line_endings(const std::string& file_contents);
+std::pair<int, std::string> get_majority_file_line_endings_and_count(
+    const std::string& file_contents);
+bool is_dir_in_dir(const fs::path& parent, const fs::path& child);
 }  // namespace file_util

common/util/FontUtils.cpp

@@ -16,13 +16,14 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
-#include "third-party/fmt/format.h"
+#include "fmt/core.h"
+#include "fmt/format.h"
 
 const std::unordered_map<std::string, GameTextVersion> sTextVerEnumMap = {
     {"jak1-v1", GameTextVersion::JAK1_V1},
     {"jak1-v2", GameTextVersion::JAK1_V2},
-    {"jak2", GameTextVersion::JAK2}};
+    {"jak2", GameTextVersion::JAK2},
+    {"jak3", GameTextVersion::JAK3}};
 
 const std::string& get_text_version_name(GameTextVersion version) {
   for (auto& [name, ver] : sTextVerEnumMap) {
@@ -261,7 +262,8 @@ bool GameTextFontBank::valid_char_range(const char in) const {
     return ((in >= '0' && in <= '9') || (in >= 'A' && in <= 'Z') ||
             m_passthrus->find(in) != m_passthrus->end()) &&
            in != '\\';
-  } else if (m_version == GameTextVersion::JAK2) {
+  } else if (m_version == GameTextVersion::JAK2 || m_version == GameTextVersion::JAK3 ||
+             m_version == GameTextVersion::JAKX) {
     return ((in >= '0' && in <= '9') || (in >= 'A' && in <= 'Z') || (in >= 'a' && in <= 'z') ||
             m_passthrus->find(in) != m_passthrus->end()) &&
            in != '\\';
@@ -550,6 +552,13 @@ static std::vector<ReplaceInfo> s_replace_info_jak1 = {
     {"E~Y~-6H~+2V,~Z", "Ę"},
     {"L~Y~-16H~+0V/~Z", "Ł"},
     {"Z~Y~-21H~-5Vº~Z", "Ż"},
+    {"E~Y~-20H~-5Vº~Z", "Ė"},
+    {"C~Y~-20H~-4Vˇ~Z", "Č"},
+    {"S~Y~-22H~-4Vˇ~Z", "Š"},
+    {"Z~Y~-22H~-4Vˇ~Z", "Ž"},
+    {"U~Y~-13H~+2V,~Z", "Ų"},
+    {"U~Y~-18H~-10V-~Z", "Ū"},
+    {"I~Y~-8H~+1V,~Z", "Į"},
 
     // tildes
     {"N~Y~-22H~-4V<TIL>~Z", "Ñ"},
@@ -906,7 +915,9 @@ GameTextFontBank g_font_bank_jak1_v2(GameTextVersion::JAK1_V2,
  * GAME TEXT FONT BANK - JAK 2
  * ================================
  * This font is used in:
- * - Jak 2 - NTSC - v1
+ * - Jak II
+ * - Jak II: Renegade
+ * - ジャックＸダクスター2
  */
 
 static std::unordered_set<char> s_passthrus_jak2 = {'~', ' ', ',', '.', '-', '+', '(', ')',
@@ -919,6 +930,43 @@ static std::vector<ReplaceInfo> s_replace_info_jak2 = {
     {"N~Y~-6Hº~Z~+10H", "Nº"},
     {"~+4Vç~-4V", ",c"},
 
+    // added for translations TODO - check these for jak 2
+    {"O~Y~-25H~-1V/~Z", "Ø"},
+    {"o~Y~-23H~+4V/~Z", "ø"},
+    {"A~Y~-13H~+8V,~Z", "Ą"},
+    {"a~Y~-8H~+6V,~Z", "ą"},
+    {"E~Y~-6H~+8V,~Z", "Ę"},
+    {"e~Y~-10H~+7V,~Z", "ę"},
+    {"L~Y~-21H~+0V/~Z", "Ł"},
+    {"l~Y~-16H~+0V/~Z", "ł"},  // TODO - this one is ugly, font character addition (small slash)
+    {"Z~Y~-25H~-11Vº~Z", "Ż"},
+    {"z~Y~-23H~-5Vº~Z", "ż"},
+    {"a~Y~-25H~-5Vº~Z", "å"},
+    {"S~Y~-21H~-5V'~Z", "Ś"},
+    {"s~Y~-25H~-5V'~Z", "ś"},
+    {"n~Y~-25H~-5V'~Z", "ń"},
+    {"c~Y~-25H~-5V'~Z", "ć"},
+    {"o~Y~-24H~-4V<TIL>~Z", "õ"},
+    {"a~Y~-24H~-4V<TIL>~Z", "ã"},
+    {"O~Y~-28H~-4V'~-9H'~Z", "Ő"},
+    {"U~Y~-27H~-4V'~-12H'~Z", "Ű"},
+    {"o~Y~-28H~-4V'~-9H'~Z", "ő"},
+    {"u~Y~-28H~-4V'~-9H'~Z", "ű"},
+    {"E~Y~-22H~-11Vº~Z", "Ė"},
+    {"e~Y~-25H~-5Vº~Z", "ė"},
+    {"C~Y~-27H~-10Vˇ~Z", "Č"},
+    {"c~Y~-25H~-5Vˇ~Z", "č"},
+    {"S~Y~-24H~-10Vˇ~Z", "Š"},
+    {"s~Y~-22H~-4Vˇ~Z", "š"},
+    {"Z~Y~-25H~-10Vˇ~Z", "Ž"},
+    {"z~Y~-23H~-4Vˇ~Z", "ž"},
+    {"U~Y~-15H~+5V,~Z", "Ų"},
+    {"u~Y~-15H~+5V,~Z", "ų"},
+    {"U~Y~-20H~-18V-~Z", "Ū"},
+    {"u~Y~-18H~-15V-~Z", "ū"},
+    {"I~Y~-8H~+4V,~Z", "Į"},
+    {"i~Y~-8H~+4V,~Z", "į"},
+
     // tildes
     {"N~Y~-22H~-4V<TIL>~Z", "Ñ"},
     {"n~Y~-24H~-4V<TIL>~Z", "ñ"},
@@ -937,12 +985,14 @@ static std::vector<ReplaceInfo> s_replace_info_jak2 = {
     {"o~Y~-26H~-4V'~Z", "ó"},
     {"U~Y~-24H~-3V'~Z", "Ú"},
     {"u~Y~-24H~-3V'~Z", "ú"},
+    {"Z~Y~-24H~-3V'~Z", "Ź"},
+    {"z~Y~-24H~-3V'~Z", "ź"},
 
     // circumflex
     {"A~Y~-20H~-4V^~Z", "Â"},
     {"a~Y~-24H~-5V^~Z", "â"},
     {"E~Y~-20H~-5V^~Z", "Ê"},
-    {"e~Y~-25H~-4V^~Zt", "ê"},
+    {"e~Y~-25H~-4V^~Z", "ê"},
     {"I~Y~-19H~-5V^~Z", "Î"},
     {"i~Y~-19H~-8V^~Z", "î"},
     {"O~Y~-20H~-4V^~Z", "Ô"},
@@ -967,6 +1017,7 @@ static std::vector<ReplaceInfo> s_replace_info_jak2 = {
     {"a~Y~-25H~-5V¨~Z", "ä"},
     {"E~Y~-20H~-5V¨~Z", "Ë"},
     {"I~Y~-19H~-5V¨~Z", "Ï"},
+    {"i~Y~-26H~-4V¨~Z", "ï"},
     {"O~Y~-26H~-8V¨~Z", "Ö"},
     {"o~Y~-26H~-4V¨~Z", "ö"},
     {"U~Y~-25H~-8V¨~Z", "Ü"},
@@ -1860,6 +1911,21 @@ GameTextFontBank g_font_bank_jak2(GameTextVersion::JAK2,
                                   &s_replace_info_jak2,
                                   &s_passthrus_jak2);
 
+/*!
+ * ================================
+ * GAME TEXT FONT BANK - JAK 3
+ * ================================
+ * This font is used in:
+ * - Jak 3
+ */
+
+// TODO cyrillic
+
+GameTextFontBank g_font_bank_jak3(GameTextVersion::JAK3,
+                                  &s_encode_info_jak2,
+                                  &s_replace_info_jak2,
+                                  &s_passthrus_jak2);
+
 /*!
  * ========================
  * GAME TEXT FONT BANK LIST
@@ -1870,7 +1936,8 @@ GameTextFontBank g_font_bank_jak2(GameTextVersion::JAK2,
 std::map<GameTextVersion, GameTextFontBank*> g_font_banks = {
     {GameTextVersion::JAK1_V1, &g_font_bank_jak1_v1},
     {GameTextVersion::JAK1_V2, &g_font_bank_jak1_v2},
-    {GameTextVersion::JAK2, &g_font_bank_jak2}};
+    {GameTextVersion::JAK2, &g_font_bank_jak2},
+    {GameTextVersion::JAK3, &g_font_bank_jak3}};
 
 const GameTextFontBank* get_font_bank(GameTextVersion version) {
   return g_font_banks.at(version);
@@ -1883,6 +1950,8 @@ const GameTextFontBank* get_font_bank_from_game_version(GameVersion version) {
       return get_font_bank(GameTextVersion::JAK1_V2);
     case GameVersion::Jak2:
       return get_font_bank(GameTextVersion::JAK2);
+    case GameVersion::Jak3:
+      return get_font_bank(GameTextVersion::JAK3);
     default:
       ASSERT_MSG(false, "Unsupported game for get_font_bank_from_game_version");
   }