goalc: add some nullptr checks around symbol map lookups (#3447)

Bumps
[mathieudutour/github-tag-action](https://github.com/mathieudutour/github-tag-action)
from 6.1 to 6.2.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/a22cf08638b34d5badda920f9daf6e72c477b07b"><code>a22cf08</code></a>
check in prod dependencies</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/f36074fe49d589315ac97253c23ce83ccbd6fbb9"><code>f36074f</code></a>
version 6.2</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/f40d7eefd8aa8e0422116f3fb945bbd96dfb7691"><code>f40d7ee</code></a>
match latest tag regex (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/171">#171</a>)</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/1bf7977a3da165de50510f793132286574222465"><code>1bf7977</code></a>
fix: add previous_version to action.yml (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/183">#183</a>)</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/5a16d1443262c7f1a1addb0de22dea8a784acdbe"><code>5a16d14</code></a>
Bump tough-cookie from 4.0.0 to 4.1.3 (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/184">#184</a>)</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/71c368965901c31849458bb74c4ffa22b48172c7"><code>71c3689</code></a>
Update README.md (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/205">#205</a>)</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/a079ba461db5f184f53dcd8d6031a82478b788f5"><code>a079ba4</code></a>
Update Node version to 20 (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/202">#202</a>)</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/86301c823dac64d427711dd6b99d7ffdd894d2dd"><code>86301c8</code></a>
Revert &quot;Use <code>${{github.token}}</code> as default&quot; (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/167">#167</a>)</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/5bd9038fd1cc379dfe97904dda3a4b5b3af5d606"><code>5bd9038</code></a>
Bump minimatch from 3.0.4 to 3.1.2 (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/162">#162</a>)</li>
<li><a
href="https://github.com/mathieudutour/github-tag-action/commit/ac8cb388384294d1215aed7911d7b7e57b3d0f0a"><code>ac8cb38</code></a>
getValidTags() should exclude tags that do not match the prefixRegex (<a
href="https://redirect.github.com/mathieudutour/github-tag-action/issues/158">#158</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/mathieudutour/github-tag-action/compare/v6.1...v6.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=mathieudutour/github-tag-action&package-manager=github_actions&previous-version=6.1&new-version=6.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.clang-format

@@ -10,7 +10,7 @@ IncludeCategories:
     Priority: 2
   - Regex: '^(<|")common.*'
     Priority: 3
-  - Regex: '^"third-party.*'
+  - Regex: '^"(third-party|fmt|curl).*'
     Priority: 4
     SortPriority: 5
   - Regex: '.*'

.github/workflows/draft-new-release.yaml

@@ -23,7 +23,7 @@ jobs:
       - name: Bump Version and Push Tag
         if: github.repository == 'open-goal/jak-project'
         id: tag_version
-        uses: mathieudutour/github-tag-action@v6.1
+        uses: mathieudutour/github-tag-action@v6.2
         with:
           github_token: ${{ secrets.BOT_PAT }}
           tag_prefix: v

.github/workflows/inform-pages-repo.yaml

@@ -58,12 +58,16 @@ jobs:
         run: |
           ./build/goalc/goalc --cmd "(begin (make-group \"all-code\") (gen-docs \"${PWD}\"))" --game jak2
 
+      - name: Generate Docs For Jak 3
+        run: |
+          ./build/goalc/goalc --cmd "(begin (make-group \"all-code\") (gen-docs \"${PWD}\"))" --game jak3
+
       - name: Append File Docstrings
         run: |
-          python ./scripts/docs/append-file-docs.py
+          python ./scripts/ci/docs/append-file-docs.py
 
       - name: Upload Docs Artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: opengoal-docs
           if-no-files-found: error
@@ -75,7 +79,7 @@ jobs:
         run: echo ${{ github.run_id }}
 
       - name: Send Dispatch
-        uses: peter-evans/repository-dispatch@v2
+        uses: peter-evans/repository-dispatch@v3
         with:
           token: ${{ secrets.BOT_PAT }}
           repository: 'open-goal/open-goal.github.io'

.github/workflows/linting.yaml

@@ -41,6 +41,9 @@ jobs:
       - name: Checkout Repository
         uses: actions/checkout@v4
 
+      - name: Check for Trailing Whitespace in goal_src
+        run: python ./scripts/ci/lint-trailing-whitespace.py
+
       - name: Check for Unresolved Conflicts
         run: python ./scripts/gsrc/check-for-conflicts.py
 

.github/workflows/linux-build-clang.yaml

@@ -20,13 +20,6 @@ jobs:
     runs-on: ubuntu-20.04
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -39,11 +32,13 @@ jobs:
             libxinerama-dev libxcursor-dev libpulse-dev \
             libxi-dev zip ninja-build libgl1-mesa-dev libssl-dev
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.12
         with:
-          cache_key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -51,8 +46,8 @@ jobs:
           CXX: clang++
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`nproc`))
@@ -71,7 +66,7 @@ jobs:
           strip ./build/lsp/lsp
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         if: ${{ inputs.uploadArtifacts }}
         with:
           name: opengoal-linux-${{ inputs.cachePrefix }}

.github/workflows/linux-build-gcc.yaml

@@ -16,13 +16,6 @@ jobs:
     runs-on: ubuntu-20.04
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -31,15 +24,22 @@ jobs:
         run: |
           sudo apt update
           sudo apt install build-essential cmake \
-            clang gcc g++ lcov make nasm libxrandr-dev \
+            clang gcc-10 g++-10 lcov make nasm libxrandr-dev \
             libxinerama-dev libxcursor-dev libpulse-dev \
             libxi-dev zip ninja-build libgl1-mesa-dev libssl-dev
 
-      - name: Setup Buildcache
-        uses: mikehardy/[email protected]
+            sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 100
+            sudo update-alternatives --set gcc /usr/bin/gcc-10
+            sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 100
+            sudo update-alternatives --set g++ /usr/bin/g++-10
+
+      - name: Setup sccache
+        uses: hendrikmuhs/[email protected]
         with:
-          cache_key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: linux-ubuntu-20.04-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -48,18 +48,11 @@ jobs:
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
             -DCODE_COVERAGE=OFF \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`nproc`)) -- -w dupbuild=warn
 
       - name: Run Tests
         run: ./test.sh
-
-      # - name: Submit Coverage Report to Codacy
-      #   uses: codacy/codacy-coverage-reporter-action@v1
-      #   continue-on-error: true
-      #   with:
-      #     project-token: ${{ secrets.CODACY_PROJECT_KEY }}
-      #     coverage-reports: ./build/goalc-test_coverage.info

.github/workflows/macos-build-arm.yaml

@@ -13,16 +13,9 @@ on:
 jobs:
   build:
     name: ARM
-    runs-on: macos-latest
+    runs-on: macos-12
     timeout-minutes: 120
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -33,11 +26,13 @@ jobs:
       - name: Install Package Dependencies
         run: arch -arm64 brew install cmake ninja
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.12
         with:
-          cache_key: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -45,8 +40,8 @@ jobs:
           CXX: clang++
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`sysctl -n hw.logicalcpu`))
@@ -55,7 +50,7 @@ jobs:
         run: ./test.sh
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: opengoal-macos-${{ inputs.cachePrefix }}
           if-no-files-found: error

.github/workflows/macos-build.yaml

@@ -17,16 +17,9 @@ on:
 jobs:
   build:
     name: Intel
-    runs-on: macos-11
+    runs-on: macos-12
     timeout-minutes: 120
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -34,11 +27,13 @@ jobs:
       - name: Install Package Dependencies
         run: brew install cmake nasm ninja
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.12
         with:
-          cache_key: macos-11-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
-          buildcache_tag: v0.28.1
+          variant: sccache
+          key: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: macos-12-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - name: CMake Generation
         env:
@@ -46,8 +41,8 @@ jobs:
           CXX: clang++
         run: |
           cmake -B build --preset=${{ inputs.cmakePreset }} \
-            -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache
+            -DCMAKE_C_COMPILER_LAUNCHER=sccache \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
 
       - name: Build Project
         run: cmake --build build --parallel $((`sysctl -n hw.logicalcpu`))
@@ -64,7 +59,7 @@ jobs:
           strip ./build/lsp/lsp
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         if: ${{ inputs.uploadArtifacts }}
         with:
           name: opengoal-macos-${{ inputs.cachePrefix }}

.github/workflows/release-pipeline.yaml

@@ -51,7 +51,7 @@ jobs:
       - name: Prepare Artifact Folder
         run: mkdir -p ./ci-artifacts/final
 
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         name: Download all Artifacts
         with:
           path: ./ci-artifacts/

.github/workflows/update-controller-db.yaml

@@ -19,7 +19,7 @@ jobs:
           wget -O ./game/assets/sdl_controller_db.txt https://raw.githubusercontent.com/gabomdq/SDL_GameControllerDB/master/gamecontrollerdb.txt
 
       - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v5
+        uses: peter-evans/create-pull-request@v6
         with:
           token: ${{ secrets.BOT_PAT }}
           author: 'OpenGOALBot <[email protected]>'

.github/workflows/windows-build-clang.yaml

@@ -21,13 +21,6 @@ jobs:
     # Set some sort of timeout in the event of run-away builds.  We are limited on concurrent jobs so, get rid of them.
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -42,10 +35,13 @@ jobs:
             choco install ${{ github.workspace }}/third-party/nasm/old/nasm.2.15.05.nupkg
           }
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.12
         with:
-          cache_key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          variant: sccache
+          key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - uses: ilammy/msvc-dev-cmd@v1
 
@@ -58,20 +54,19 @@ jobs:
 
       - name: CMake Generation
         shell: cmd
-        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe .
+        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache .
 
       - name: Build Project
         shell: cmd
         run: cmake --build build --parallel %NUMBER_OF_PROCESSORS%
 
       - name: Run Tests
-        timeout-minutes: 10
         env:
           GTEST_OUTPUT: "xml:opengoal-test-report.xml"
         run: ./build/bin/goalc-test.exe --gtest_color=yes --gtest_brief=0 --gtest_filter="-*MANUAL_TEST*"
 
       - name: Upload artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         if: ${{ inputs.uploadArtifacts }}
         with:
           name: opengoal-windows-${{ inputs.cachePrefix }}

.github/workflows/windows-build-msvc.yaml

@@ -16,13 +16,6 @@ jobs:
     runs-on: windows-2022
     timeout-minutes: 60
 
-    env: # overrides: https://github.com/mbitsnbites/buildcache/blob/master/doc/configuration.md
-      BUILDCACHE_MAX_CACHE_SIZE: 1000000000 # 1gb
-      BUILDCACHE_COMPRESS_FORMAT: ZSTD
-      BUILDCACHE_COMPRESS_LEVEL: 19
-      BUILDCACHE_DIRECT_MODE: true
-      BUILDCACHE_LOG_FILE: ${{ github.workspace }}/buildcache.log
-
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v4
@@ -37,10 +30,13 @@ jobs:
             choco install ${{ github.workspace }}/third-party/nasm/old/nasm.2.15.05.nupkg
           }
 
-      - name: Setup Buildcache
-        uses: mikehardy/build[email protected].0
+      - name: Setup sccache
+        uses: hendrikmuhs/ccache-action@v1.2.12
         with:
-          cache_key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          variant: sccache
+          key: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}-${{ github.sha }}
+          restore-keys: windows-2022-${{ inputs.cachePrefix }}-${{ inputs.cmakePreset }}
+          max-size: 1000M
 
       - uses: ilammy/msvc-dev-cmd@v1
 
@@ -52,7 +48,7 @@ jobs:
 
       - name: CMake Generation - MSVC
         shell: cmd
-        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe -DCMAKE_CXX_COMPILER_LAUNCHER=${{ github.workspace }}/buildcache/bin/buildcache.exe .
+        run: cmake -B build --preset=${{ inputs.cmakePreset }} -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache .
 
       - name: Build Project
         shell: cmd

.prettierrc

@@ -1 +0,0 @@
-endOfLine: "auto"

.prettierrc.json

@@ -0,0 +1,11 @@
+{
+  "endOfLine": "auto",
+  "overrides": [
+    {
+      "files": ["**/*.jsonc"],
+      "options": {
+        "parser": "json"
+      }
+    }
+  ]
+}

.vs/launch.vs.json

@@ -68,6 +68,18 @@
         "jak2"
       ]
     },
+    {
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "offline-test.exe (bin\\offline-test.exe)",
+      "name": "Tests - Offline Tests - Jak 3",
+      "args": [
+        "--iso_data_path",
+        "${workspaceRoot}/iso_data/jak3",
+        "--game",
+        "jak3"
+      ]
+    },
     {
       "type": "default",
       "project": "CMakeLists.txt",
@@ -131,13 +143,6 @@
       "name": "Game - Jak 2 - Runtime (release)",
       "args": ["-v", "--game", "jak2", "--", "-boot", "-fakeiso"]
     },
-    {
-      "type": "default",
-      "project": "CMakeLists.txt",
-      "projectTarget": "goalc.exe (bin\\goalc.exe)",
-      "name": "REPL",
-      "args": ["--user-auto"]
-    },
     {
       "type": "default",
       "project": "CMakeLists.txt",
@@ -218,6 +223,19 @@
         "--config-override \"{\\\"decompile_code\\\": false, \\\"levels_extract\\\": true, \\\"allowed_objects\\\": []}\""
       ]
     },
+    {
+      "type": "default",
+      "project": "CMakeLists.txt",
+      "projectTarget": "decompiler.exe (bin\\decompiler.exe)",
+      "name": "Decompiler - Jak 3",
+      "args": [
+        "${workspaceRoot}/decompiler/config/jak3/jak3_config.jsonc",
+        "${workspaceRoot}/iso_data",
+        "${workspaceRoot}/decompiler_out",
+        "--version",
+        "ntsc_v1"
+      ]
+    },
     {
       "type": "default",
       "project": "CMakeLists.txt",
@@ -268,7 +286,7 @@
       "project" : "CMakeLists.txt",
       "projectTarget" : "formatter.exe (bin\\formatter.exe)",
       "name" : "Tools - Formatter - Inplace",
-      "args" : ["--write", "--file", "C:\\Users\\xtvas\\Repos\\opengoal\\jak-project\\goal_src\\jak2\\pc\\util\\popup-menu-h.gc"]
+      "args" : ["--write", "--file", "C:\\Users\\xtvas\\Repos\\opengoal\\jak-project\\decompiler_out\\jak3\\mood-h_disasm.gc"]
     },
     {
       "type": "default",

.vscode/launch.json

@@ -5,13 +5,13 @@
   "version": "0.2.0",
   "configurations": [
     {
-      "name": "Append File Docs",
+      "name": "run python script",
       "type": "python",
       "request": "launch",
-      "program": "${workspaceFolder}/scripts/ci/lint-characters.py",
+      "program": "${workspaceFolder}/scripts/gsrc/copy-common-naming.py",
       "console": "integratedTerminal",
       "cwd": "${workspaceFolder}",
-      "args": ["--fix"]
+      "args": ["--update-names-from-refs", "--decompiler", "./out/build/Release/bin/decompiler"]
     },
   ]
 }

.vscode/settings.json

@@ -5,101 +5,8 @@
       "comments": true,
       "strings": true
     },
-    "editor.wordBasedSuggestions": true,
+    "editor.wordBasedSuggestions": "matchingDocuments",
     "editor.snippetSuggestions": "top"
   },
-  "python.formatting.provider": "black",
-  "files.associations": {
-    "optional": "cpp",
-    "algorithm": "cpp",
-    "any": "cpp",
-    "array": "cpp",
-    "atomic": "cpp",
-    "bit": "cpp",
-    "bitset": "cpp",
-    "cctype": "cpp",
-    "charconv": "cpp",
-    "chrono": "cpp",
-    "cinttypes": "cpp",
-    "clocale": "cpp",
-    "cmath": "cpp",
-    "codecvt": "cpp",
-    "compare": "cpp",
-    "concepts": "cpp",
-    "condition_variable": "cpp",
-    "cstdarg": "cpp",
-    "cstddef": "cpp",
-    "cstdint": "cpp",
-    "cstdio": "cpp",
-    "cstdlib": "cpp",
-    "cstring": "cpp",
-    "ctime": "cpp",
-    "cwchar": "cpp",
-    "cwctype": "cpp",
-    "deque": "cpp",
-    "exception": "cpp",
-    "filesystem": "cpp",
-    "format": "cpp",
-    "forward_list": "cpp",
-    "fstream": "cpp",
-    "functional": "cpp",
-    "future": "cpp",
-    "initializer_list": "cpp",
-    "iomanip": "cpp",
-    "ios": "cpp",
-    "iosfwd": "cpp",
-    "iostream": "cpp",
-    "istream": "cpp",
-    "iterator": "cpp",
-    "limits": "cpp",
-    "list": "cpp",
-    "locale": "cpp",
-    "map": "cpp",
-    "memory": "cpp",
-    "mutex": "cpp",
-    "new": "cpp",
-    "numeric": "cpp",
-    "ostream": "cpp",
-    "queue": "cpp",
-    "random": "cpp",
-    "ranges": "cpp",
-    "ratio": "cpp",
-    "regex": "cpp",
-    "set": "cpp",
-    "shared_mutex": "cpp",
-    "span": "cpp",
-    "sstream": "cpp",
-    "stack": "cpp",
-    "stdexcept": "cpp",
-    "stop_token": "cpp",
-    "streambuf": "cpp",
-    "string": "cpp",
-    "system_error": "cpp",
-    "thread": "cpp",
-    "tuple": "cpp",
-    "type_traits": "cpp",
-    "typeinfo": "cpp",
-    "unordered_map": "cpp",
-    "unordered_set": "cpp",
-    "utility": "cpp",
-    "valarray": "cpp",
-    "variant": "cpp",
-    "vector": "cpp",
-    "xfacet": "cpp",
-    "xhash": "cpp",
-    "xiosbase": "cpp",
-    "xlocale": "cpp",
-    "xlocbuf": "cpp",
-    "xlocinfo": "cpp",
-    "xlocmes": "cpp",
-    "xlocmon": "cpp",
-    "xlocnum": "cpp",
-    "xloctime": "cpp",
-    "xmemory": "cpp",
-    "xstddef": "cpp",
-    "xstring": "cpp",
-    "xtr1common": "cpp",
-    "xtree": "cpp",
-    "xutility": "cpp"
-  }
+  "cmake.configureOnOpen": false
 }

CMakeLists.txt

@@ -1,7 +1,7 @@
 # Top Level CMakeLists.txt
 cmake_minimum_required(VERSION 3.10)
 
-set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD 20)
 project(jak)
 include(CTest)
 
@@ -39,7 +39,6 @@ if(CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
         -fcxx-exceptions \
         -fexceptions \
         -fdiagnostics-color=always \
-        -std=c++17 \
         -mavx \
         -Wall \
         -Wno-c++11-narrowing \
@@ -86,7 +85,6 @@ elseif(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
         -Wdisabled-optimization \
         -Wformat \
         -Wextra \
-        -Wmissing-include-dirs \
         -Woverloaded-virtual \
         -Wredundant-decls \
         -Wshadow \
@@ -113,7 +111,6 @@ elseif(CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
         -Wdisabled-optimization \
         -Wformat \
         -Wextra \
-        -Wmissing-include-dirs \
         -Woverloaded-virtual \
         -Wredundant-decls \
         -Wshadow \
@@ -219,12 +216,15 @@ build_third_party_lib(sqlite3 sqlite3)
 
 # build tree-sitter parser
 include_directories(third-party/tree-sitter/tree-sitter/lib/include)
-include_directories(third-party/tree-sitter/tree-sitter-opengoal/include)
 build_third_party_lib(tree-sitter tree-sitter)
 
 # native OS dialogs for error messages
 build_third_party_lib(libtinyfiledialogs libtinyfiledialogs)
 
+# build format library
+include_directories(third-party/fmt/include)
+build_third_party_lib(fmt fmt)
+
 # build common library
 add_subdirectory(common)
 
@@ -277,9 +277,6 @@ include(test/CMakeLists.txt)
 # build lzokay library
 build_third_party_lib(lzokay lzokay)
 
-# build format library
-build_third_party_lib(fmt fmt)
-
 build_third_party_lib(stb_image stb_image)
 build_third_party_lib(tiny_gltf tiny_gltf)
 build_third_party_lib(xdelta3 xdelta3)

CMakePresets.json

@@ -108,6 +108,16 @@
       "description": "Build with Clang with Debug Symbols",
       "inherits": ["base-windows-debug", "base-clang"]
     },
+    {
+      "name": "Debug-windows-clang-static",
+      "displayName": "Windows Static Debug (clang)",
+      "description": "Build with Clang as Debug but statically linked",
+      "inherits": ["base-windows-debug", "base-clang"],
+      "cacheVariables": {
+        "STATICALLY_LINK": "true",
+        "ZYDIS_BUILD_SHARED_LIB": "OFF"
+      }
+    },
     {
       "name": "RelWithDebInfo-windows-clang",
       "displayName": "Windows RelWithDebInfo (clang)",

Taskfile.yml

@@ -74,24 +74,26 @@ tasks:
         msg: "Couldn't locate compiler executable in '{{.GOALC_BIN_RELEASE_DIR}}/goalc'"
     cmds:
       - "{{.GOALC_BIN_RELEASE_DIR}}/goalc --user-auto --game {{.GAME}}"
-  format:
-    desc: "Format code"
-    cmds:
-      - cmd: python ./scripts/cpp/format-includes.py
-      - cmd: python ./third-party/run-clang-format/run-clang-format.py -r common decompiler game goalc test tools lsp -i
-      # npm install -g prettier
-      - cmd: npx prettier --write ./decompiler/config/jak1/**/*.jsonc
-        ignore_error: true
-      - cmd: npx prettier --write ./decompiler/config/jak2/**/*.jsonc
-        ignore_error: true
   format-json:
     desc: "Format code"
     cmds:
       # npm install -g prettier
-      - cmd: npx prettier --write ./decompiler/config/jak1/**/*.jsonc
-        ignore_error: true
-      - cmd: npx prettier --write ./decompiler/config/jak2/**/*.jsonc
+      - cmd: npx prettier --write ./decompiler/config/jak*/**/*.jsonc
         ignore_error: true
+  format:
+    desc: "Format code"
+    cmds:
+      - cmd: python ./scripts/cpp/format-includes.py
+      - cmd: python ./scripts/ci/lint-trailing-whitespace.py --fix
+      - cmd: python ./third-party/run-clang-format/run-clang-format.py -r common decompiler game goalc test tools lsp -i
+      - task: format-json
+  format-gsrc:
+    desc: "Run formatter on gsrc file"
+    preconditions:
+      - sh: test -f {{.FORMATTER_BIN_RELEASE_DIR}}/formatter{{.EXE_FILE_EXTENSION}}
+        msg: "Couldn't locate formatter executable in '{{.FORMATTER_BIN_RELEASE_DIR}}/formatter'"
+    cmds:
+      - "{{.FORMATTER_BIN_RELEASE_DIR}}/formatter --write --file '{{.FILE}}'"
   # DECOMPILING
   decomp:
     cmds:
@@ -104,7 +106,7 @@ tasks:
       - python ./scripts/tasks/clean-decomp.py --game "{{.GAME}}"
   lint-gsrc-file:
     cmds:
-      - python ./scripts/gsrc/lint-gsrc-file.py --game {{.GAME}} --file {{.FILE}}
+      - python ./scripts/gsrc/lint-gsrc-file.py --game {{.GAME}} --file "{{.FILE}}"
   update-gsrc:
     cmds:
       - python ./scripts/gsrc/update-gsrc-via-refs.py --game "{{.GAME}}" --decompiler "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler" --decompiler_config {{.DECOMP_CONFIG}} --version {{.DECOMP_CONFIG_VERSION}}
@@ -114,8 +116,26 @@ tasks:
   update-gsrc-file:
     cmds:
       - task: decomp-file
-      - python ./scripts/gsrc/update-from-decomp.py --game "{{.GAME}}" --file {{.FILE}}
+      - python ./scripts/gsrc/update-from-decomp.py --game "{{.GAME}}" --file "{{.FILE}}"
       - task: lint-gsrc-file
+  copy-common-naming:
+    cmds:
+      - python ./scripts/gsrc/copy-common-naming.py --file "{{.FILE}}" --decompiler "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler"
+      - task: format-json
+  copy-common-naming-from-refs:
+    cmds:
+      - python ./scripts/gsrc/copy-common-naming.py --update-names-from-refs --decompiler "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler"
+      - task: format-json
+  # ASSET RIPPING
+  rip-textures:
+    cmds:
+      - "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler \"./decompiler/config/{{.DECOMP_CONFIG}}\" \"./iso_data\" \"./decompiler_out\" --version \"{{.DECOMP_CONFIG_VERSION}}\" --config-override '{\"decompile_code\": false, \"levels_extract\": true, \"save_texture_pngs\": true}'"
+  rip-levels:
+    cmds:
+      - "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler \"./decompiler/config/{{.DECOMP_CONFIG}}\" \"./iso_data\" \"./decompiler_out\" --version \"{{.DECOMP_CONFIG_VERSION}}\" --config-override '{\"decompile_code\": false, \"levels_extract\": true, \"rip_levels\": true}'"
+  rip-collision:
+    cmds:
+      - "{{.DECOMP_BIN_RELEASE_DIR}}/decompiler \"./decompiler/config/{{.DECOMP_CONFIG}}\" \"./iso_data\" \"./decompiler_out\" --version \"{{.DECOMP_CONFIG_VERSION}}\" --config-override '{\"decompile_code\": false, \"levels_extract\": true, \"extract_collision\": true, \"rip_collision\": true}'"
   # TOOLS
   analyze-ee-memory:
     cmds:
@@ -141,6 +161,9 @@ tasks:
     desc: Finds and tries to fix invalid translation Characters
     cmds:
       - python ./scripts/ci/lint-characters.py --fix
+  lint:
+    cmds:
+      - python ./scripts/ci/lint-trailing-whitespace.py
   # TESTS
   offline-tests: # ran by jenkins
     cmds:
@@ -166,9 +189,6 @@ tasks:
         ignore_error: true
       - python ./scripts/update_decomp_reference.py ./failures ./test/decompiler/reference/ --game {{.GAME}}
       - task: offline-test-file
-  type-test-jak1:
-    cmds:
-      - cmd: '{{.GOALCTEST_BIN_RELEASE_DIR}}/goalc-test --gtest_brief=0 --gtest_filter="*Jak1TypeConsistency*" --gtest_break_on_failure'
   type-test:
     cmds:
-      - cmd: '{{.GOALCTEST_BIN_RELEASE_DIR}}/goalc-test --gtest_brief=0 --gtest_filter="*Jak2TypeConsistency*" --gtest_break_on_failure'
+      - cmd: '{{.GOALCTEST_BIN_RELEASE_DIR}}/goalc-test --gtest_brief=0 --gtest_filter="*{{.TYPE_CONSISTENCY_TEST_FILTER}}*" --gtest_break_on_failure'

common/CMakeLists.txt

@@ -66,6 +66,7 @@ add_library(common
         type_system/TypeSpec.cpp
         type_system/TypeSystem.cpp
         util/Assert.cpp
+        util/ast_util.cpp
         util/BitUtils.cpp
         util/compress.cpp
         util/crc32.cpp
@@ -86,8 +87,7 @@ add_library(common
         util/term_util.cpp
         util/Timer.cpp
         util/unicode_util.cpp
-        versions/versions.cpp
-        )
+        versions/versions.cpp)
 
 target_link_libraries(common fmt lzokay replxx libzstd_static tree-sitter sqlite3 libtinyfiledialogs)
 

common/audio/audio_formats.cpp

@@ -3,7 +3,7 @@
 #include "common/log/log.h"
 #include "common/util/BinaryWriter.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 /*!
  * Write a wave file from a vector of samples.
@@ -56,11 +56,14 @@ void write_wave_file(const std::vector<s16>& left_samples,
   writer.write_to_file(name);
 }
 
-std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader, const bool mono) {
+std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader,
+                                                           const bool stereo) {
   std::vector<s16> left_samples;
   std::vector<s16> right_samples;
   s32 left_sample_prev[2] = {0, 0};
   s32 right_sample_prev[2] = {0, 0};
+  bool first_left = true;
+  bool first_right = true;
   constexpr s32 f1[5] = {0, 60, 115, 98, 122};
   constexpr s32 f2[5] = {0, 0, -52, -55, -60};
 
@@ -71,15 +74,29 @@ std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader,
   // instead they are partitioned into contiguous 8kb (thats 8192 bytes) chunks
   // alternating left/right
   bool processing_left_chunk = true;
+  // We need to skip the vag header for each channel
+  if (first_left) {
+    reader.ffwd(48);
+    bytes_read += 48;
+    first_left = false;
+  }
   while (true) {
     if (!reader.bytes_left()) {
       break;
     }
 
-    if (bytes_read == 0x2000) {
+    if (stereo && bytes_read == 0x2000) {
       // switch streams
       processing_left_chunk = !processing_left_chunk;
       bytes_read = 0;
+
+      // We need to skip the vag header for each channel
+      if (first_right) {
+        // skip the header
+        reader.ffwd(48);
+        bytes_read += 48;
+        first_right = false;
+      }
     }
 
     u8 shift_filter = reader.read<u8>();
@@ -107,7 +124,7 @@ std::pair<std::vector<s16>, std::vector<s16>> decode_adpcm(BinaryReader& reader,
       s32 sample = (s32)(s16)(nibble << 12);
       sample >>= shift;
 
-      if (mono || processing_left_chunk) {
+      if (!stereo || processing_left_chunk) {
         sample += (left_sample_prev[0] * f1[filter] + left_sample_prev[1] * f2[filter] + 32) / 64;
 
         if (sample > 0x7fff) {

common/cross_os_debug/xdbg.cpp

@@ -11,7 +11,7 @@
 #include "common/goal_constants.h"
 #include "common/util/Timer.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef __linux
 #include <fcntl.h>

common/cross_sockets/XSocket.cpp

@@ -19,7 +19,7 @@
 #include <stdio.h>
 #include <string.h>
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 #include "common/log/log.h"
 // clang-format on
 

common/cross_sockets/XSocketClient.cpp

@@ -15,7 +15,7 @@
 #endif
 #include "common/repl/nrepl/ReplServer.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 // clang-format on
 
 XSocketClient::XSocketClient(int _tcp_port) {

common/cross_sockets/XSocketServer.cpp

@@ -4,7 +4,7 @@
 #include "common/cross_sockets/XSocket.h"
 #include "common/common_types.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef _WIN32
 #define NOMINMAX

common/custom_data/TFrag3Data.cpp

@@ -42,6 +42,7 @@ void ShrubDraw::serialize(Serializer& ser) {
   ser.from_ptr(&num_triangles);
   ser.from_ptr(&first_index_index);
   ser.from_ptr(&num_indices);
+  ser.from_ptr(&proto_idx);
 }
 
 void InstancedStripDraw::serialize(Serializer& ser) {
@@ -416,6 +417,9 @@ void ShrubTree::serialize(Serializer& ser) {
   for (auto& draw : static_draws) {
     draw.serialize(ser);
   }
+
+  ser.from_ptr(&has_per_proto_visibility_toggle);
+  ser.from_string_vector(&proto_names);
 }
 
 void BVH::serialize(Serializer& ser) {

common/custom_data/Tfrag3Data.h

@@ -18,7 +18,7 @@ namespace tfrag3 {
 // - if changing any large things (vertices, vis, bvh, colors, textures) update get_memory_usage
 // - if adding a new category to the memory usage, update extract_level to print it.
 
-constexpr int TFRAG3_VERSION = 38;
+constexpr int TFRAG3_VERSION = 39;
 
 enum MemoryUsageCategory {
   TEXTURE,
@@ -219,6 +219,8 @@ struct ShrubDraw {
 
   // for debug counting.
   u32 num_triangles = 0;
+
+  u16 proto_idx = 0;
   void serialize(Serializer& ser);
 };
 
@@ -430,6 +432,10 @@ struct ShrubTree {
     std::vector<ShrubGpuVertex> vertices;  // mesh vertices
   } unpacked;
 
+  // jak 2 and later can toggle on and off visibility per proto by name
+  bool has_per_proto_visibility_toggle = false;
+  std::vector<std::string> proto_names;
+
   void serialize(Serializer& ser);
   void memory_usage(MemoryUsageTracker* tracker) const;
   void unpack();

common/dma/dma.cpp

@@ -2,7 +2,7 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 std::string DmaTag::print() {
   std::string result;

common/dma/dma.h

@@ -11,7 +11,7 @@
 #include "common/common_types.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 struct DmaStats {
   double sync_time_ms = 0;

common/dma/dma_copy.cpp

@@ -6,7 +6,7 @@
 #include "common/log/log.h"
 #include "common/util/Timer.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 /*!
  * Convert a DMA chain to an array of bytes that can be directly fed to VIF.

common/dma/gs.cpp

@@ -2,8 +2,8 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
-#include "third-party/fmt/format.h"
+#include "fmt/core.h"
+#include "fmt/format.h"
 
 std::string reg_descriptor_name(GifTag::RegisterDescriptor reg) {
   switch (reg) {

common/formatter/formatter.cpp

@@ -2,13 +2,16 @@
 
 #include "formatter_tree.h"
 
+#include "common/formatter/rules/formatting_rules.h"
+#include "common/formatter/rules/rule_config.h"
 #include "common/log/log.h"
 #include "common/util/FileUtil.h"
+#include "common/util/ast_util.h"
 #include "common/util/string_util.h"
 
 #include "tree_sitter/api.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 // Declare the `tree_sitter_opengoal` function, which is
 // implemented by the `tree-sitter-opengoal` library.
@@ -30,7 +33,19 @@ int hang_indentation_width(const FormatterTreeNode& curr_node) {
   return 1 + hang_indentation_width(first_elt);
 }
 
-// TODO - compute length of each node and store it
+// TODO - this doesn't account for paren's width contribution!
+int get_total_form_inlined_width(const FormatterTreeNode& curr_node) {
+  if (curr_node.token) {
+    return curr_node.token->length();
+  }
+  int width = 1;
+  for (const auto& ref : curr_node.refs) {
+    width += get_total_form_inlined_width(ref);
+  }
+  return width + 1;
+}
+
+// TODO - compute length of each node and store it AOT
 void apply_formatting_config(
     FormatterTreeNode& curr_node,
     std::optional<std::shared_ptr<formatter_rules::config::FormFormattingConfig>>
@@ -51,6 +66,7 @@ void apply_formatting_config(
       curr_node.formatting_config = predefined_config.value();
     }
   } else if (config_from_parent) {
+    // TODO - doesn't merge just replaces, a bit inflexible
     predefined_config = *config_from_parent.value();
     curr_node.formatting_config = predefined_config.value();
   }
@@ -67,10 +83,10 @@ void apply_formatting_config(
   //
   // Otherwise, we always default to a hang.
   //
-  // NOTE - any modifications here to child elements could be superseeded later in the recursion
-  // in order to maintain your sanity, only modify things here that _arent_ touched by default
+  // NOTE - any modifications here to child elements could be superseeded later in the recursion!
+  // In order to maintain your sanity, only modify things here that _arent_ touched by default
   // configurations.  These are explicitly prepended with `parent_mutable_`
-  if (!predefined_config) {
+  if (!predefined_config && !curr_node.formatting_config.config_set) {
     if (curr_node.metadata.is_top_level) {
       curr_node.formatting_config.indentation_width = 0;
       curr_node.formatting_config.hang_forms = false;
@@ -104,6 +120,8 @@ void apply_formatting_config(
   }
   // If we are hanging, lets determine the indentation width since it is based on the form itself
   if (curr_node.formatting_config.hang_forms) {
+    // TODO - this isn't being calculated for a pre-defined config
+    // TODO - another idea is to do this during consolidation
     curr_node.formatting_config.indentation_width = hang_indentation_width(curr_node);
   }
   // iterate through the refs
@@ -121,17 +139,37 @@ void apply_formatting_config(
       }
     }
   }
-}
-
-int get_total_form_inlined_width(const FormatterTreeNode& curr_node) {
-  if (curr_node.token) {
-    return curr_node.token->length();
+  // Precompute the column widths for things like deftype fields
+  if (curr_node.formatting_config.determine_column_widths_for_list_elements) {
+    // iterate through each ref and find the max length of each index (may be a token, may not be!)
+    // then store that info in each list element's `list_element_column_widths` to be used when
+    // printing out the tokens (pad width)
+    // Find the maximum number of columns
+    int max_columns = 0;
+    for (const auto& field : curr_node.refs) {
+      if (field.refs.size() > max_columns) {
+        max_columns = field.refs.size();
+      }
+    }
+    // Now find the column max widths
+    std::vector<int> column_max_widths = {};
+    for (int col = 0; col < max_columns; col++) {
+      column_max_widths.push_back(0);
+      for (const auto& field : curr_node.refs) {
+        if (field.refs.size() > col) {
+          const auto width = get_total_form_inlined_width(field.refs.at(col));
+          if (width > column_max_widths.at(col)) {
+            column_max_widths[col] = width;
+          }
+        }
+      }
+    }
+    // Apply column info to every list
+    for (auto& field : curr_node.refs) {
+      field.formatting_config.list_element_column_widths = column_max_widths;
+      field.formatting_config.config_set = true;
+    }
   }
-  int width = 1;
-  for (const auto& ref : curr_node.refs) {
-    width += get_total_form_inlined_width(ref);
-  }
-  return width + 1;
 }
 
 bool form_contains_comment(const FormatterTreeNode& curr_node) {
@@ -163,6 +201,9 @@ bool form_contains_node_that_prevents_inlining(const FormatterTreeNode& curr_nod
 
 bool can_node_be_inlined(const FormatterTreeNode& curr_node, int cursor_pos) {
   using namespace formatter_rules;
+  if (curr_node.formatting_config.force_inline) {
+    return true;
+  }
   // First off, we cannot inline the top level
   if (curr_node.metadata.is_top_level) {
     return false;
@@ -191,17 +232,20 @@ bool can_node_be_inlined(const FormatterTreeNode& curr_node, int cursor_pos) {
 }
 
 std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
-                                          std::vector<std::string> output = {},
+                                          std::vector<std::string> /*output*/ = {},
                                           int cursor_pos = 0) {
   using namespace formatter_rules;
   if (!curr_node.token && curr_node.refs.empty()) {
     // special case to handle an empty list
+    if (curr_node.node_prefix) {
+      return {fmt::format("{}()", curr_node.node_prefix.value())};
+    }
     return {"()"};
   }
 
   // If its a token, just print the token and move on
   if (curr_node.token) {
-    return {curr_node.token.value()};
+    return {curr_node.token_str()};
   }
 
   bool inline_form = can_node_be_inlined(curr_node, cursor_pos);
@@ -220,13 +264,19 @@ std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
     // Add new line entry
     if (ref.token) {
       // Cleanup block-comments
-      std::string val = ref.token.value();
+      std::string val = ref.token_str();
       if (ref.metadata.node_type == "block_comment") {
         // TODO - change this sanitization to return a list of lines instead of a single new-lined
         // line
-        val = comments::format_block_comment(ref.token.value());
+        val = comments::format_block_comment(ref.token_str());
       }
       form_lines.push_back(val);
+      if (!curr_node.metadata.is_top_level && i == curr_node.refs.size() - 1 &&
+          (ref.metadata.is_comment)) {
+        // if there's an inline comment at the end of a form, we have to force the paren to the next
+        // line and do a new-line paren this is ugly, but we have no choice!
+        form_lines.push_back("");
+      }
     } else {
       // If it's not a token, we have to recursively build up the form
       // TODO - add the cursor_pos here
@@ -250,6 +300,9 @@ std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
       if ((next_ref.metadata.node_type == "comment" && next_ref.metadata.is_inline) ||
           (curr_node.formatting_config.has_constant_pairs &&
            constant_pairs::is_element_second_in_constant_pair(curr_node, next_ref, i + 1))) {
+        // TODO
+        // has issues with not consolidating first lines, this should probably just be moved to
+        // outside this loop for simplicity, do it later
         if (next_ref.token) {
           form_lines.at(form_lines.size() - 1) += fmt::format(" {}", next_ref.token.value());
           i++;
@@ -260,6 +313,10 @@ std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
           }
           i++;
         }
+        if (!curr_node.metadata.is_top_level && next_ref.metadata.node_type == "comment" &&
+            (i + 1) == (int)curr_node.refs.size()) {
+          form_lines.push_back("");
+        }
       }
     }
     // If we are at the top level, potential separate with a new line
@@ -269,8 +326,12 @@ std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
   }
 
   // Consolidate any lines if the configuration requires it
+  // TODO there is a hack here so that multi-line forms that are consolidated still line up properly
+  // i have to make consolidate a more first-class feature of the config
   if (curr_node.formatting_config.inline_until_index(form_lines)) {
     std::vector<std::string> new_form_lines = {};
+    const auto original_form_head_width = str_util::split(form_lines.at(0), '\n').at(0).length();
+    bool consolidating_lines = true;
     for (int i = 0; i < (int)form_lines.size(); i++) {
       if (i < curr_node.formatting_config.inline_until_index(form_lines)) {
         if (new_form_lines.empty()) {
@@ -279,15 +340,35 @@ std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
           new_form_lines.at(0) += fmt::format(" {}", form_lines.at(i));
         }
       } else {
-        new_form_lines.push_back(form_lines.at(i));
+        if (str_util::starts_with(form_lines.at(i), " ") && consolidating_lines) {
+          new_form_lines.push_back(fmt::format(
+              "{}{}", str_util::repeat(original_form_head_width, " "), form_lines.at(i)));
+        } else {
+          consolidating_lines = false;
+          new_form_lines.push_back(form_lines.at(i));
+        }
       }
     }
     form_lines = new_form_lines;
   }
 
+  // Add any column padding
+  if (!curr_node.formatting_config.list_element_column_widths.empty()) {
+    for (int i = 0; i < form_lines.size(); i++) {
+      const auto& token = form_lines.at(i);
+      if (i < form_lines.size() - 1) {
+        form_lines[i] = str_util::pad_right(
+            token, curr_node.formatting_config.list_element_column_widths.at(i), ' ');
+      }
+    }
+  }
+
   // Apply necessary indentation to each line and add parens
   if (!curr_node.metadata.is_top_level) {
     std::string form_surround_start = "(";
+    if (curr_node.node_prefix) {
+      form_surround_start = fmt::format("{}(", curr_node.node_prefix.value());
+    }
     std::string form_surround_end = ")";
     form_lines[0] = fmt::format("{}{}", form_surround_start, form_lines[0]);
     form_lines[form_lines.size() - 1] =
@@ -314,16 +395,14 @@ std::vector<std::string> apply_formatting(const FormatterTreeNode& curr_node,
   return form_lines;
 }
 
-std::string join_formatted_lines(const std::vector<std::string> lines) {
-  // TODO - respect original file line endings
-  return fmt::format("{}", fmt::join(lines, "\n"));
+std::string join_formatted_lines(const std::vector<std::string>& lines,
+                                 const std::string& line_ending) {
+  return fmt::format("{}", fmt::join(lines, line_ending));
 }
 
 std::optional<std::string> formatter::format_code(const std::string& source) {
   // Create a parser.
   std::shared_ptr<TSParser> parser(ts_parser_new(), TreeSitterParserDeleter());
-
-  // Set the parser's language (JSON in this case).
   ts_parser_set_language(parser.get(), tree_sitter_opengoal());
 
   // Build a syntax tree based on source code stored in a string.
@@ -333,7 +412,11 @@ std::optional<std::string> formatter::format_code(const std::string& source) {
 
   // Get the root node of the syntax tree.
   TSNode root_node = ts_tree_root_node(tree.get());
-  if (ts_node_is_null(root_node) || ts_node_has_error(root_node)) {
+  if (ts_node_is_null(root_node)) {
+    lg::error("null root node");
+    return std::nullopt;
+  } else if (ts_node_has_error(root_node)) {
+    lg::error("grammar parsing error, go figure it out!");
     return std::nullopt;
   }
 
@@ -352,7 +435,8 @@ std::optional<std::string> formatter::format_code(const std::string& source) {
     const auto formatted_lines = apply_formatting(formatting_tree.root);
     // 4. Now we joint he lines together, it's easier when formatting to leave all lines independent
     // so adding indentation is easier
-    const auto formatted_source = join_formatted_lines(formatted_lines);
+    const auto formatted_source =
+        join_formatted_lines(formatted_lines, file_util::get_majority_file_line_endings(source));
     return formatted_source;
   } catch (std::exception& e) {
     lg::error("Unable to format code - {}", e.what());

common/formatter/formatter.h

@@ -3,13 +3,8 @@
 #include <optional>
 #include <string>
 
-#include "common/formatter/rules/formatting_rules.h"
-#include "common/formatter/rules/rule_config.h"
-
 #include "tree_sitter/api.h"
 
-// TODO:
-// - Considering _eventually_ adding line-length heuristics
 namespace formatter {
 
 struct TreeSitterParserDeleter {

common/formatter/formatter_tree.cpp

@@ -2,7 +2,7 @@
 
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 std::string get_source_code(const std::string& source, const TSNode& node) {
   uint32_t start = ts_node_start_byte(node);
@@ -78,16 +78,24 @@ FormatterTree::FormatterTree(const std::string& source, const TSNode& root_node)
   construct_formatter_tree_recursive(source, root_node, root);
 }
 
+const std::unordered_map<std::string, std::vector<std::string>> node_type_ignorable_contents = {
+    {"list_lit", {"(", ")"}},
+    {"quoting_lit", {"'"}},
+    {"unquoting_lit", {","}},
+    {"quasi_quoting_lit", {"`"}}};
+
 // TODO make an imperative version eventually
 void FormatterTree::construct_formatter_tree_recursive(const std::string& source,
                                                        TSNode curr_node,
-                                                       FormatterTreeNode& tree_node) {
+                                                       FormatterTreeNode& tree_node,
+                                                       std::optional<std::string> node_prefix) {
   if (ts_node_child_count(curr_node) == 0) {
     tree_node.refs.push_back(FormatterTreeNode(source, curr_node));
     return;
   }
   const std::string curr_node_type = ts_node_type(curr_node);
   FormatterTreeNode list_node;
+  std::optional<std::string> next_node_prefix;
   if (curr_node_type == "list_lit") {
     list_node = FormatterTreeNode();
   } else if (curr_node_type == "str_lit") {
@@ -97,22 +105,40 @@ void FormatterTree::construct_formatter_tree_recursive(const std::string& source
     tree_node.refs.push_back(FormatterTreeNode(source, curr_node));
     return;
   } else if (curr_node_type == "quoting_lit") {
-    // same story for quoted symbols
-    // TODO - expect to have to add more here
-    tree_node.refs.push_back(FormatterTreeNode(source, curr_node));
-    return;
+    next_node_prefix = "'";
+  } else if (curr_node_type == "unquoting_lit") {
+    next_node_prefix = ",";
+  } else if (curr_node_type == "quasi_quoting_lit") {
+    next_node_prefix = "`";
+  }
+  std::vector<std::string> skippable_nodes = {};
+  if (node_type_ignorable_contents.find(curr_node_type) != node_type_ignorable_contents.end()) {
+    skippable_nodes = node_type_ignorable_contents.at(curr_node_type);
   }
   for (size_t i = 0; i < ts_node_child_count(curr_node); i++) {
     const auto child_node = ts_node_child(curr_node, i);
-    // We skip parens
     const auto contents = get_source_code(source, child_node);
-    if (contents == "(" || contents == ")") {
+    bool skip_node = false;
+    for (const auto& skippable_content : skippable_nodes) {
+      if (skippable_content == contents) {
+        skip_node = true;
+        break;
+      }
+    }
+    if (skip_node) {
       continue;
     }
     if (curr_node_type == "list_lit") {
-      construct_formatter_tree_recursive(source, child_node, list_node);
+      construct_formatter_tree_recursive(source, child_node, list_node, next_node_prefix);
+      if (node_prefix) {
+        list_node.node_prefix = node_prefix;
+      }
     } else {
-      construct_formatter_tree_recursive(source, child_node, tree_node);
+      construct_formatter_tree_recursive(source, child_node, tree_node, next_node_prefix);
+      // TODO - im not sure if this is correct
+      if (node_prefix && !tree_node.refs.empty()) {
+        tree_node.refs.at(tree_node.refs.size() - 1).node_prefix = node_prefix;
+      }
     }
   }
   if (curr_node_type == "list_lit") {

common/formatter/formatter_tree.h

@@ -18,8 +18,9 @@
 // we really care about is:
 // - getting all the text tokens for the source code
 // - having them in a proper, nested format
-// The treesitter format is complicated and highly nested, leading to some very hard to understand
-// code. So my solution is a 2-pass format.
+//
+// TLDR - The treesitter format is complicated and highly nested, leading to some very hard to
+// understand code. So my solution is atleast a 2-pass format.
 //
 // Pass 1 - convert the AST into a simplified FormatterTree
 // Pass 2 - use the simplified tree to output the final code
@@ -37,8 +38,9 @@ class FormatterTreeNode {
   std::vector<FormatterTreeNode> refs;
   Metadata metadata;
   // The token is optional because list nodes do not contain a token, they just contain a bunch of
-  // eventually token node refs
+  // eventually-containing token node refs
   std::optional<std::string> token;
+  std::optional<std::string> node_prefix;
 
   formatter_rules::config::FormFormattingConfig formatting_config;
 
@@ -47,6 +49,15 @@ class FormatterTreeNode {
   FormatterTreeNode(const Metadata& _metadata) : metadata(_metadata){};
 
   bool is_list() const { return !token.has_value(); }
+  std::string token_str() const {
+    if (node_prefix && token) {
+      return node_prefix.value() + token.value();
+    }
+    if (token) {
+      return token.value();
+    }
+    return "";
+  }
 };
 
 // A FormatterTree has a very simple and crude tree structure where:
@@ -62,5 +73,6 @@ class FormatterTree {
  private:
   void construct_formatter_tree_recursive(const std::string& source,
                                           TSNode curr_node,
-                                          FormatterTreeNode& tree_node);
+                                          FormatterTreeNode& tree_node,
+                                          std::optional<std::string> node_prefix = {});
 };

common/formatter/rules/formatting_rules.cpp

@@ -2,11 +2,9 @@
 
 #include <set>
 
-#include "rule_config.h"
-
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace formatter_rules {
 

common/formatter/rules/rule_config.cpp

@@ -1,24 +1,25 @@
 #include "rule_config.h"
 
-#include "common/formatter/formatter_tree.h"
-
 namespace formatter_rules {
 namespace config {
 
-// TODO - this could be greatly simplified with C++20's designated initialization
-FormFormattingConfig new_flow_rule(int start_index) {
-  FormFormattingConfig cfg;
-  cfg.hang_forms = false;
-  cfg.inline_until_index = [start_index](const std::vector<std::string>& curr_lines) {
-    return start_index;
-  };
-  return cfg;
+static FormFormattingConfig new_permissive_flow_rule() {
+  return {.config_set = true, .hang_forms = false, .combine_first_two_lines = true};
 }
 
-FormFormattingConfig new_flow_rule_prevent_inlining_indexes(
+static FormFormattingConfig new_flow_rule(int start_index) {
+  return {.config_set = true,
+          .hang_forms = false,
+          .inline_until_index = [start_index](const std::vector<std::string>& /*curr_lines*/) {
+            return start_index;
+          }};
+}
+
+static FormFormattingConfig new_deftype_rule(
     int start_index,
     const std::vector<int>& inlining_preventation_indices) {
   FormFormattingConfig cfg;
+  cfg.config_set = true;
   cfg.hang_forms = false;
   cfg.inline_until_index = [start_index](std::vector<std::string> curr_lines) {
     if (curr_lines.size() >= 4 && curr_lines.at(3) == "()") {
@@ -28,26 +29,37 @@ FormFormattingConfig new_flow_rule_prevent_inlining_indexes(
   };
   for (const auto& index : inlining_preventation_indices) {
     auto temp_config = std::make_shared<FormFormattingConfig>();
+    temp_config->config_set = true;
     temp_config->prevent_inlining = true;
     temp_config->hang_forms = false;
     temp_config->indentation_width = 1;
+    auto temp_list_config = std::make_shared<FormFormattingConfig>();
+    temp_list_config->force_inline = true;
+    temp_list_config->hang_forms = false;
+    temp_config->default_index_config = temp_list_config;
+    if (index == 3) {
+      temp_config->determine_column_widths_for_list_elements = true;
+    }
     cfg.index_configs.emplace(index, temp_config);
   }
   return cfg;
 }
 
-FormFormattingConfig new_binding_rule() {
+static FormFormattingConfig new_binding_rule(int form_head_width) {
   FormFormattingConfig cfg;
+  cfg.config_set = true;
   cfg.hang_forms = false;
   cfg.combine_first_two_lines = true;
   auto binding_list_config = std::make_shared<FormFormattingConfig>();
+  binding_list_config->config_set = true;
   binding_list_config->hang_forms = false;
   binding_list_config->indentation_width = 1;
-  binding_list_config->indentation_width_for_index = [](FormFormattingConfig /*cfg*/, int index) {
+  binding_list_config->indentation_width_for_index = [form_head_width](FormFormattingConfig /*cfg*/,
+                                                                       int index) {
     if (index == 0) {
       return 0;
     }
-    return 4;
+    return form_head_width;
   };
   binding_list_config->should_prevent_inlining = [](FormFormattingConfig /*config*/, int num_refs) {
     // Only prevent inlining a binding list, if there are more than 1 bindings
@@ -62,12 +74,14 @@ FormFormattingConfig new_binding_rule() {
   return cfg;
 }
 
-FormFormattingConfig new_pair_rule(bool combine_first_two_expr) {
+static FormFormattingConfig new_pair_rule(bool combine_first_two_expr) {
   FormFormattingConfig cfg;
+  cfg.config_set = true;
   cfg.hang_forms = false;
   cfg.prevent_inlining = true;
   cfg.combine_first_two_lines = combine_first_two_expr;
   auto pair_config = std::make_shared<FormFormattingConfig>();
+  pair_config->config_set = true;
   pair_config->hang_forms = false;
   pair_config->indentation_width = 1;
   cfg.default_index_config = pair_config;
@@ -78,11 +92,19 @@ const std::unordered_map<std::string, FormFormattingConfig> opengoal_form_config
     {"case", new_pair_rule(true)},
     {"cond", new_pair_rule(false)},
     {"defmethod", new_flow_rule(3)},
-    {"deftype", new_flow_rule_prevent_inlining_indexes(3, {3, 4, 5})},
+    {"deftype", new_deftype_rule(3, {3, 4, 5, 6})},
     {"defun", new_flow_rule(3)},
+    {"defun-debug", new_flow_rule(3)},
+    {"defbehavior", new_flow_rule(4)},
+    {"if", new_permissive_flow_rule()},
+    {"define", new_permissive_flow_rule()},
+    {"define-extern", new_permissive_flow_rule()},
+    {"defmacro", new_flow_rule(3)},
     {"dotimes", new_flow_rule(2)},
-    {"let", new_binding_rule()},
+    {"let", new_binding_rule(4)},
+    {"rlet", new_binding_rule(5)},
     {"when", new_flow_rule(2)},
+    {"begin", new_flow_rule(0)},
     {"with-dma-buffer-add-bucket", new_flow_rule(2)}};
 }  // namespace config
 }  // namespace formatter_rules

common/formatter/rules/rule_config.h

@@ -7,12 +7,15 @@
 #include <unordered_map>
 #include <vector>
 
+// TODO - some way to apply a config to all list elements (index configs with -1?)
 namespace formatter_rules {
 namespace config {
 struct FormFormattingConfig {
-  // new
-  bool hang_forms = true;  // TODO - remove this eventually, it's only involved in setting the
-                           // indentation width, which we can do via the indentation_width function
+  bool config_set = false;
+  bool force_inline = false;
+  bool hang_forms =
+      true;  // TODO - remove this eventually, it's only involved in setting the
+             // indentation width, which we can do via the new indentation_width function
   int indentation_width =
       2;  // 2 for a flow // TODO - also remove this, prefer storing the first node's width in the
           // metadata on the first pass, that's basically all this does
@@ -21,8 +24,8 @@ struct FormFormattingConfig {
   bool combine_first_two_lines =
       false;  // NOTE - basically hang, but will probably stick around after hang is gone, may be
               // redundant (inline_until_index!)
-  std::function<std::optional<int>(const std::vector<std::string>& curr_lines)> inline_until_index =
-      [](std::vector<std::string> curr_lines) { return std::nullopt; };
+  std::function<std::optional<int>(const std::vector<std::string>& /*curr_lines*/)>
+      inline_until_index = [](std::vector<std::string> /*curr_lines*/) { return std::nullopt; };
   bool has_constant_pairs = false;
   bool prevent_inlining = false;  // TODO - duplicate of below
   std::function<bool(FormFormattingConfig, int num_refs)> should_prevent_inlining =
@@ -30,6 +33,9 @@ struct FormFormattingConfig {
   int parent_mutable_extra_indent = 0;
   std::optional<std::shared_ptr<FormFormattingConfig>> default_index_config;
   std::unordered_map<int, std::shared_ptr<FormFormattingConfig>> index_configs = {};
+
+  bool determine_column_widths_for_list_elements = false;
+  std::vector<int> list_element_column_widths = {};
 };
 
 extern const std::unordered_map<std::string, FormFormattingConfig> opengoal_form_config;

common/global_profiler/GlobalProfiler.cpp

@@ -9,7 +9,7 @@
 #include "common/util/FileUtil.h"
 #include "common/common_types.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 #include "third-party/json.hpp"
 
 #ifdef OS_POSIX
@@ -42,8 +42,17 @@ void GlobalProfiler::set_max_events(size_t event_count) {
   m_nodes.resize(event_count);
 }
 
+void GlobalProfiler::set_waiting_for_event(const std::string& event_name) {
+  if (!event_name.empty()) {
+    m_waiting_for_event = event_name;
+  }
+}
+
 void GlobalProfiler::event(const char* name, ProfNode::Kind kind) {
-  if (!m_enabled) {
+  if (m_waiting_for_event && m_waiting_for_event.value() == name) {
+    m_ignore_events = true;
+  }
+  if (!m_enabled || m_ignore_events) {
     return;
   }
   size_t my_idx = (m_next_idx++ % m_nodes.size());
@@ -68,7 +77,7 @@ void GlobalProfiler::begin_event(const char* name) {
 }
 
 void GlobalProfiler::end_event() {
-  if (!m_enabled) {
+  if (!m_enabled || m_ignore_events) {
     return;
   }
   size_t my_idx = (m_next_idx++ % m_nodes.size());
@@ -166,7 +175,7 @@ void GlobalProfiler::dump_to_json(const std::string& path) {
       lg::debug("out of order: {} {} {} ms", event.ts / 1000.f, info.debug / 1000.f,
                 (info.debug - event.ts) / 1000000.f);
       lg::debug("  idx: {}, range {} {}", event_idx, info.lowest_at_target, info.highest_at_target);
-      lg::debug("  now: {}", m_next_idx);
+      lg::debug("  now: {}", m_next_idx.load());
     }
     info.debug = event.ts;
   }

common/global_profiler/GlobalProfiler.h

@@ -1,6 +1,7 @@
 #pragma once
 
 #include <atomic>
+#include <optional>
 #include <string>
 #include <vector>
 
@@ -17,6 +18,7 @@ class GlobalProfiler {
  public:
   GlobalProfiler();
   void set_max_events(size_t event_count);
+  void set_waiting_for_event(const std::string& event_name);
   void instant_event(const char* name);
   void begin_event(const char* name);
   void event(const char* name, ProfNode::Kind kind);
@@ -31,9 +33,15 @@ class GlobalProfiler {
   u64 m_t0 = 0;
   std::atomic_size_t m_next_idx = 0;
   std::vector<ProfNode> m_nodes;
+  // this is very niche, but sometimes you want to capture up to a given event (ie. long startup)
+  // instead of having to make the user quit and record as fast as possible, we can instead just
+  // stop capturing events once we have received what we are looking for
+  std::optional<std::string> m_waiting_for_event = {};
+  bool m_ignore_events = false;
 };
 
 struct ScopedEvent {
+  ScopedEvent(GlobalProfiler* _prof) : prof(_prof){};
   ScopedEvent(const ScopedEvent&) = delete;
   ScopedEvent& operator=(const ScopedEvent&) = delete;
   GlobalProfiler* prof = nullptr;

common/goal_constants.h

@@ -62,15 +62,9 @@ constexpr int LEVEL_MAX = 6;
 constexpr int LEVEL_TOTAL = LEVEL_MAX + 1;
 }  // namespace jak2
 
-// TODO copypaste from jak 2 for now
 namespace jak3 {
 // for now, we don't have the ability to extend the size of the symbol table
 constexpr s32 GOAL_MAX_SYMBOLS = 0x4000;
-constexpr s32 SYM_TABLE_MEM_SIZE = 0x30000;
-// from the "off-by-one" symbol pointer
-constexpr int SYM_TO_STRING_OFFSET = 0xff37;
-constexpr int SYM_TO_HASH_OFFSET = 0x1fe6f;
-
 // amount of levels in level heap
 constexpr int LEVEL_MAX = 10;
 // total amount of levels, including ones outside level heap (default-level)
@@ -84,7 +78,7 @@ constexpr s32 max_symbols(GameVersion version) {
     case GameVersion::Jak2:
       return jak2::GOAL_MAX_SYMBOLS;
     case GameVersion::Jak3:
-      return jak2::GOAL_MAX_SYMBOLS;
+      return jak3::GOAL_MAX_SYMBOLS;
   }
 }
 

common/goos/Interpreter.cpp

@@ -16,7 +16,7 @@
 #include "common/util/string_util.h"
 #include "common/util/unicode_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 Interpreter::Interpreter(const std::string& username) {
@@ -1888,7 +1888,7 @@ Object Interpreter::eval_format(const Object& form,
                    fmt::format_args(args2.data(), static_cast<unsigned>(args2.size())));
 
   if (truthy(dest)) {
-    lg::print(formatted.c_str());
+    lg::print("{}", formatted.c_str());
   }
 
   return StringObject::make_new(formatted);

common/goos/Object.cpp

@@ -47,7 +47,7 @@
 #include "common/util/crc32.h"
 #include "common/util/print_float.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 

common/goos/ParseHelpers.cpp

@@ -1,6 +1,6 @@
 #include "ParseHelpers.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 

common/goos/PrettyPrinter.cpp

@@ -15,7 +15,7 @@
 #include "common/goos/PrettyPrinter2.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace pretty_print {
 

common/goos/PrettyPrinter2.cpp

@@ -3,8 +3,8 @@
 #include "common/common_types.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
-#include "third-party/fmt/format.h"
+#include "fmt/core.h"
+#include "fmt/format.h"
 
 namespace pretty_print {
 

common/goos/Printer.cpp

@@ -6,7 +6,7 @@
 #include "common/goos/Object.h"
 #include "common/util/print_float.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace pretty_print {
 namespace {

common/goos/Reader.cpp

@@ -16,7 +16,7 @@
 #include "common/util/FileUtil.h"
 #include "common/util/FontUtils.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 
@@ -238,9 +238,14 @@ Object Reader::read_from_string(const std::string& str,
  * Read a file
  */
 Object Reader::read_from_file(const std::vector<std::string>& file_path, bool check_encoding) {
-  std::string joined_path = fmt::format("{}", fmt::join(file_path, "/"));
+  std::string file_descriptor = fmt::format("{}", fmt::join(file_path, "/"));
+  const auto joined_file_path = file_util::get_file_path(file_path);
 
-  auto textFrag = std::make_shared<FileText>(file_util::get_file_path(file_path), joined_path);
+  if (!fs::exists(joined_file_path)) {
+    throw std::runtime_error(fmt::format("Cannot read {}, file doesn't exist", joined_file_path));
+  }
+
+  auto textFrag = std::make_shared<FileText>(joined_file_path, file_descriptor);
   db.insert(textFrag);
 
   auto result = internal_read(textFrag, check_encoding);

common/goos/TextDB.cpp

@@ -15,7 +15,7 @@
 
 #include "common/util/FileUtil.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace goos {
 
@@ -85,9 +85,9 @@ std::pair<int, int> SourceText::get_containing_line(int offset) {
 /*!
  * Read text from a file.
  */
-FileText::FileText(const std::string& filename, const std::string& description_name)
-    : m_filename(filename), m_desc_name(description_name) {
-  m_text = file_util::read_text_file(m_filename);
+FileText::FileText(const std::string& file_path, const std::string& description_name)
+    : m_filepath(file_path), m_desc_name(description_name) {
+  m_text = file_util::read_text_file(m_filepath);
   build_offsets();
 }
 

common/goos/TextDB.h

@@ -80,13 +80,13 @@ class ProgramString : public SourceText {
  */
 class FileText : public SourceText {
  public:
-  FileText(const std::string& filename, const std::string& description_name);
+  FileText(const std::string& file_path, const std::string& description_name);
 
   std::string get_description() { return m_desc_name; }
   ~FileText() = default;
 
  private:
-  std::string m_filename;
+  std::string m_filepath;
   std::string m_desc_name;
 };
 

common/link_types.h

@@ -54,6 +54,21 @@ struct LinkHeaderV4 {
   uint32_t code_size;  // length of object data before link data starts
 };
 
+struct LinkHeaderV5Core {
+  uint32_t length_to_get_to_code;  // 4 length.. of link data?
+  uint16_t version;                // 8
+  uint16_t unknown;                // 10
+  uint32_t length_to_get_to_link;  // 12
+  uint32_t link_length;            // 16
+  uint8_t n_segments;              // 20
+  char name[59];                   // 21 (really??)
+};
+
+struct LinkHeaderV5 {
+  uint32_t type_tag;  // 0 always 0?
+  LinkHeaderV5Core core;
+};
+
 // when a u32/s32 symbol link contains this value, (s7 + <val>) should be a 4-byte aligned address,
 // not including the 1 byte symbol offset. (no effect in jak 1).
 constexpr u32 LINK_SYM_NO_OFFSET_FLAG = 0xbadbeef;

common/log/log.cpp

@@ -1,10 +1,11 @@
 #include "log.h"
 
+#include <cstdarg>
 #include <cstdio>
 #include <cstdlib>
 #include <mutex>
 
-#include "third-party/fmt/color.h"
+#include "fmt/color.h"
 #ifdef _WIN32  // see lg::initialize
 #define NOMINMAX
 #define WIN32_LEAN_AND_MEAN
@@ -106,7 +107,28 @@ void log_print(const char* message) {
     }
 
     if (gLogger.stdout_log_level < lg::level::off_unless_die) {
-      fmt::print(message);
+      fmt::print("{}", message);
+      fflush(stdout);
+      fflush(stderr);
+    }
+  }
+}
+
+void log_vprintf(const char* format, va_list arg_list) {
+  {
+    // We always immediately flush prints because since it has no associated level
+    // it could be anything from a fatal error to a useless debug log.
+    std::lock_guard<std::mutex> lock(gLogger.mutex);
+    va_list arg_list_2;
+    va_copy(arg_list_2, arg_list);
+    if (gLogger.fp) {
+      // Log to File
+      vfprintf(gLogger.fp, format, arg_list);
+      fflush(gLogger.fp);
+    }
+
+    if (gLogger.stdout_log_level < lg::level::off_unless_die) {
+      vprintf(format, arg_list_2);
       fflush(stdout);
       fflush(stderr);
     }
@@ -114,6 +136,10 @@ void log_print(const char* message) {
 }
 }  // namespace internal
 
+void printstd(const char* format, va_list arg_list) {
+  internal::log_vprintf(format, arg_list);
+}
+
 // how many extra log files for a single program should be kept?
 constexpr int LOG_ROTATE_MAX = 10;
 

common/log/log.h

@@ -7,8 +7,8 @@
 #endif
 #include <string>
 
-#include "third-party/fmt/color.h"
-#include "third-party/fmt/core.h"
+#include "fmt/color.h"
+#include "fmt/core.h"
 
 namespace lg {
 
@@ -38,6 +38,7 @@ namespace internal {
 // log implementation stuff, not to be called by the user
 void log_message(level log_level, LogTime& now, const char* message);
 void log_print(const char* message);
+void log_vprintf(const char* format, va_list arg_list);
 }  // namespace internal
 
 void set_file(const std::string& filename,
@@ -60,13 +61,13 @@ void log(level log_level, const std::string& format, Args&&... args) {
 #else
   now.tim = time(nullptr);
 #endif
-  std::string formatted_message = fmt::format(format, std::forward<Args>(args)...);
+  std::string formatted_message = fmt::format(fmt::runtime(format), std::forward<Args>(args)...);
   internal::log_message(log_level, now, formatted_message.c_str());
 }
 
 template <typename... Args>
 void print(const std::string& format, Args&&... args) {
-  std::string formatted_message = fmt::format(format, std::forward<Args>(args)...);
+  std::string formatted_message = fmt::format(fmt::runtime(format), std::forward<Args>(args)...);
   internal::log_print(formatted_message.c_str());
 }
 template <typename... Args>
@@ -75,6 +76,9 @@ void print(const fmt::text_style& ts, const std::string& format, Args&&... args)
   internal::log_print(formatted_message.c_str());
 }
 
+// same as print but uses the C printf instead of fmt
+void printstd(const char* format, va_list arg_list);
+
 template <typename... Args>
 void trace(const std::string& format, Args&&... args) {
   log(level::trace, format, std::forward<Args>(args)...);

common/math/Vector.h

@@ -2,7 +2,7 @@
 
 #include <cmath>
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace math {
 

common/repl/config.cpp

@@ -2,7 +2,7 @@
 
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace REPL {
 void to_json(json& j, const Config& obj) {

common/repl/nrepl/ReplClient.cpp

@@ -4,7 +4,7 @@
 #include "common/cross_sockets/XSocket.h"
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef _WIN32
 #define NOMINMAX

common/repl/nrepl/ReplServer.cpp

@@ -4,7 +4,7 @@
 #include "common/cross_sockets/XSocket.h"
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 #ifdef _WIN32
 #define NOMINMAX

common/repl/util.cpp

@@ -5,8 +5,8 @@
 #include "common/util/string_util.h"
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/color.h"
-#include "third-party/fmt/core.h"
+#include "fmt/color.h"
+#include "fmt/core.h"
 #include "third-party/replxx/include/replxx.hxx"
 // TODO - expand a list of hints (ie. a hint for defun to show at a glance how to write a function,
 // or perhaps, show the docstring for the current function being used?)

common/serialization/subtitles/subtitles.cpp

@@ -5,7 +5,7 @@
 #include "common/util/FileUtil.h"
 #include "common/util/json_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 void open_subtitle_project(const std::string& project_kind,
                            const std::string& file_path,

common/serialization/subtitles/subtitles_v1.cpp

@@ -5,7 +5,7 @@
 
 #include "subtitles_v2.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 void to_json(json& j, const SubtitleCutsceneLineMetadataV1& obj) {
   json_serialize(frame_start);

common/serialization/subtitles/subtitles_v2.cpp

@@ -5,7 +5,7 @@
 
 #include "subtitles_v1.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 void to_json(json& j, const SubtitleLineMetadata& obj) {
   json_serialize(frame_start);
@@ -373,7 +373,7 @@ bool GameSubtitleDB::write_subtitle_db_to_files(const GameVersion game_version)
       file_util::write_text_file(dump_path, lines_file.dump(2));
     }
   } catch (std::exception& ex) {
-    lg::error(ex.what());
+    lg::error("{}", ex.what());
     return false;
   }
   return true;

common/symbols.h

@@ -150,6 +150,180 @@ constexpr int FIX_FIXED_SYM_END_OFFSET = 0xec;
 
 }  // namespace jak2_symbols
 
+namespace jak3_symbols {
+
+constexpr int FIX_SYM_EMPTY_CAR = -0x8;
+constexpr int S7_OFF_FIX_SYM_EMPTY_PAIR = -0x6 - 1;
+constexpr int FIX_SYM_EMPTY_CDR = -0x4;
+constexpr int FIX_SYM_FALSE = 0x0;  // GOAL boolean #f (note that this is equal to the $s7 register)
+constexpr int FIX_SYM_TRUE = 0x4;   // GOAL boolean #t
+constexpr int FIX_SYM_FUNCTION_TYPE = 0x8;
+constexpr int FIX_SYM_BASIC = 0xc;
+constexpr int FIX_SYM_STRING_TYPE = 0x10;
+constexpr int FIX_SYM_SYMBOL_TYPE = 0x14;
+constexpr int FIX_SYM_TYPE_TYPE = 0x18;
+constexpr int FIX_SYM_OBJECT_TYPE = 0x1c;
+constexpr int FIX_SYM_LINK_BLOCK = 0x20;
+constexpr int FIX_SYM_INTEGER = 0x24;
+constexpr int FIX_SYM_SINTEGER = 0x28;
+constexpr int FIX_SYM_UINTEGER = 0x2c;
+constexpr int FIX_SYM_BINTEGER = 0x30;
+constexpr int FIX_SYM_INT8 = 0x34;
+constexpr int FIX_SYM_INT16 = 0x38;
+constexpr int FIX_SYM_INT32 = 0x3c;
+constexpr int FIX_SYM_INT64 = 0x40;
+constexpr int FIX_SYM_INT128 = 0x44;
+constexpr int FIX_SYM_UINT8 = 0x48;
+constexpr int FIX_SYM_UINT16 = 0x4c;
+constexpr int FIX_SYM_UINT32 = 0x50;
+constexpr int FIX_SYM_UINT64 = 0x54;
+constexpr int FIX_SYM_UINT128 = 0x58;
+constexpr int FIX_SYM_FLOAT = 0x5c;
+constexpr int FIX_SYM_PROCESS_TREE = 0x60;
+constexpr int FIX_SYM_PROCESS_TYPE = 0x64;
+constexpr int FIX_SYM_THREAD = 0x68;
+constexpr int FIX_SYM_STRUCTURE = 0x6c;
+constexpr int FIX_SYM_PAIR_TYPE = 0x70;
+constexpr int FIX_SYM_POINTER = 0x74;
+constexpr int FIX_SYM_NUMBER = 0x78;
+constexpr int FIX_SYM_ARRAY = 0x7c;
+constexpr int FIX_SYM_VU_FUNCTION = 0x80;
+constexpr int FIX_SYM_CONNECTABLE = 0x84;
+constexpr int FIX_SYM_STACK_FRAME = 0x88;
+constexpr int FIX_SYM_FILE_STREAM = 0x8c;
+constexpr int FIX_SYM_HEAP = 0x90;
+constexpr int FIX_SYM_NOTHING_FUNC = 0x94;
+constexpr int FIX_SYM_DELETE_BASIC = 0x98;
+constexpr int FIX_SYM_STATIC = 0x9c;
+constexpr int FIX_SYM_GLOBAL_HEAP = 0xa0;
+constexpr int FIX_SYM_DEBUG = 0xa4;
+constexpr int FIX_SYM_LOADING_LEVEL = 0xa8;
+constexpr int FIX_SYM_LOADING_PACKAGE = 0xac;
+constexpr int FIX_SYM_PROCESS_LEVEL_HEAP = 0xb0;
+constexpr int FIX_SYM_STACK = 0xb4;
+constexpr int FIX_SYM_SCRATCH = 0xb8;
+constexpr int FIX_SYM_SCRATCH_TOP = 0xbc;
+constexpr int FIX_SYM_ZERO_FUNC = 0xc0;
+constexpr int FIX_SYM_ASIZE_OF_BASIC_FUNC = 0xc4;
+constexpr int FIX_SYM_COPY_BASIC_FUNC = 0xc8;  // bugged name
+constexpr int FIX_SYM_LEVEL = 0xcc;
+constexpr int FIX_SYM_ART_GROUP = 0xd0;
+constexpr int FIX_SYM_TEXTURE_PAGE_DIR = 0xd4;
+constexpr int FIX_SYM_TEXTURE_PAGE = 0xd8;
+constexpr int FIX_SYM_SOUND = 0xdc;
+constexpr int FIX_SYM_DGO = 0xe0;
+constexpr int FIX_SYM_TOP_LEVEL = 0xe4;
+constexpr int FIX_SYM_QUOTE = 0xe8;
+constexpr int FIX_SYM_LISTENER_LINK_BLOCK = 0xec;
+constexpr int FIX_SYM_LISTENER_FUNCTION = 0xf0;
+constexpr int FIX_SYM_STACK_TOP = 0xf4;
+constexpr int FIX_SYM_STACK_BASE = 0xf8;
+constexpr int FIX_SYM_STACK_SIZE = 0xfc;
+constexpr int FIX_SYM_KERNEL_FUNCTION = 0x100;
+constexpr int FIX_SYM_KERNEL_PACKAGES = 0x104;
+constexpr int FIX_SYM_KERNEL_BOOT_MESSAGE = 0x108;
+constexpr int FIX_SYM_KERNEL_BOOT_MODE = 0x10c;
+constexpr int FIX_SYM_KERNEL_BOOT_LEVEL = 0x110;
+constexpr int FIX_SYM_KERNEL_BOOT_ART_GROUP = 0x114;
+constexpr int FIX_SYM_KERNEL_DEBUG = 0x118;
+constexpr int FIX_SYM_KERNEL_VERSION = 0x11c;
+constexpr int FIX_SYM_KERNEL_DISPATCHER = 0x120;
+constexpr int FIX_SYM_SYNC_DISPATCHER = 0x124;
+constexpr int FIX_SYM_PRINT_COLLUMN = 0x128;
+constexpr int FIX_SYM_DEBUG_SEGMENT = 300;
+constexpr int FIX_SYM_ENABLE_METHOD_SET = 0x130;
+constexpr int FIX_SYM_SQL_RESULT = 0x134;
+constexpr int FIX_SYM_COLLAPSE_QUOTE = 0x138;
+constexpr int FIX_SYM_LEVEL_TYPE_LIST = 0x13C;
+constexpr int FIX_SYM_DECI_COUNT = 0x140;
+constexpr int FIX_SYM_USER = 0x144;
+constexpr int FIX_SYM_VIDEO_MODE = 0x148;
+constexpr int FIX_SYM_BOOT_VIDEO_MODE = 0x14C;
+constexpr int FIX_SYM_BOOT = 0x150;
+constexpr int FIX_SYM_DEMO = 0x154;
+constexpr int FIX_SYM_DEMO_SHARED = 0x158;
+constexpr int FIX_SYM_PREVIEW = 0x15C;
+constexpr int FIX_SYM_KIOSK = 0x160;
+constexpr int FIX_SYM_PLAY_BOOT = 0x164;
+constexpr int FIX_SYM_SIN = 0x168;
+constexpr int FIX_SYM_COS = 0x16C;
+constexpr int FIX_SYM_PUT_DISPLAY_ENV = 0x170;
+constexpr int FIX_SYM_SYNCV = 0x174;
+constexpr int FIX_SYM_SYNC_PATH = 0x178;
+constexpr int FIX_SYM_RESET_PATH = 0x17C;
+constexpr int FIX_SYM_RESET_GRAPH = 0x180;
+constexpr int FIX_SYM_DMA_SYNC = 0x184;
+constexpr int FIX_SYM_GS_PUT_IMR = 0x188;
+constexpr int FIX_SYM_GS_GET_IMR = 0x18C;
+constexpr int FIX_SYM_GS_STORE_IMAGE = 400;
+constexpr int FIX_SYM_FLUSH_CACHE = 0x194;
+constexpr int FIX_SYM_CPAD_OPEN = 0x198;
+constexpr int FIX_SYM_CPAD_GET_DATA = 0x19C;
+constexpr int FIX_SYM_MOUSE_GET_DATA = 0x1A0;
+constexpr int FIX_SYM_KEYBD_GET_DATA = 0x1A4;
+constexpr int FIX_SYM_INSTALL_HANDLER = 0x1A8;
+constexpr int FIX_SYM_INSTALL_DEBUG_HANDLER = 0x1AC;
+constexpr int FIX_SYM_FILE_STREAM_OPEN = 0x1B0;
+constexpr int FIX_SYM_FILE_STREAM_CLOSE = 0x1B4;
+constexpr int FIX_SYM_FILE_STREAM_LENGTH = 0x1B8;
+constexpr int FIX_SYM_FILE_STREAM_SEEK = 0x1BC;
+constexpr int FIX_SYM_FILE_STREAM_READ = 0x1C0;
+constexpr int FIX_SYM_FILE_STREAM_WRITE = 0x1C4;
+constexpr int FIX_SYM_SCF_GET_LANGUAGE = 0x1C8;
+constexpr int FIX_SYM_SCF_GET_TIME = 0x1CC;
+constexpr int FIX_SYM_SCF_GET_ASPECT = 0x1D0;
+constexpr int FIX_SYM_SCF_GET_VOLUME = 0x1D4;
+constexpr int FIX_SYM_SCF_GET_TERRITORY = 0x1D8;
+constexpr int FIX_SYM_SCF_GET_TIMEOUT = 0x1DC;
+constexpr int FIX_SYM_SCF_GET_INACTIVE_TIMEOUT = 0x1E0;
+constexpr int FIX_SYM_DMA_TO_IOP = 0x1E4;
+constexpr int FIX_SYM_KERNEL_SHUTDOWN = 0x1E8;
+constexpr int FIX_SYM_AYBABTU = 0x1EC;
+constexpr int FIX_SYM_STRING_TO_SYMBOL = 0x1F0;
+constexpr int FIX_SYM_SYMBOL_TO_STRING = 500;
+constexpr int FIX_SYM_PRINT = 0x1F8;
+constexpr int FIX_SYM_INSPECT = 0x1FC;
+constexpr int FIX_SYM_LOAD = 0x200;
+constexpr int FIX_SYM_LOADB = 0x204;
+constexpr int FIX_SYM_LOADO = 0x208;
+constexpr int FIX_SYM_UNLOAD = 0x20C;
+constexpr int FIX_SYM_FORMAT = 0x210;
+constexpr int FIX_SYM_MALLOC = 0x214;
+constexpr int FIX_SYM_KMALLOC = 0x218;
+constexpr int FIX_SYM_KMEMOPEN = 0x21C;
+constexpr int FIX_SYM_KMEMCLOSE = 0x220;
+constexpr int FIX_SYM_NEW_DYNAMIC_STRUCTURE = 0x224;
+constexpr int FIX_SYM_METHOD_SET = 0x228;
+constexpr int FIX_SYM_LINK = 0x22C;
+constexpr int FIX_SYM_LINK_BUSY = 0x230;
+constexpr int FIX_SYM_LINK_RESET = 0x234;
+constexpr int FIX_SYM_LINK_BEGIN = 0x238;
+constexpr int FIX_SYM_LINK_RESUME = 0x23C;
+constexpr int FIX_SYM_DGO_LOAD = 0x240;
+constexpr int FIX_SYM_SQL_QUERY = 0x244;
+constexpr int FIX_SYM_MC_RUN = 0x248;
+constexpr int FIX_SYM_MC_FORMAT = 0x24C;
+constexpr int FIX_SYM_MC_UNFORMAT = 0x250;
+constexpr int FIX_SYM_MC_CREATE_FILE = 0x254;
+constexpr int FIX_SYM_MC_SAVE = 600;
+constexpr int FIX_SYM_MC_LOAD = 0x25C;
+constexpr int FIX_SYM_MC_CHECK_RESULT = 0x260;
+constexpr int FIX_SYM_MC_GET_SLOT_INFO = 0x264;
+constexpr int FIX_SYM_MC_MAKEFILE = 0x268;
+constexpr int FIX_SYM_KSET_LANGUAGE = 0x26C;
+constexpr int FIX_SYM_RPC_CALL = 0x270;
+constexpr int FIX_SYM_RPC_BUSY = 0x274;
+constexpr int FIX_SYM_TEST_LOAD_DGO_C = 0x278;
+constexpr int FIX_SYM_SYMLINK2 = 0x27c;
+constexpr int FIX_SYM_SYMLINK3 = 0x280;
+constexpr int FIX_SYM_ULTIMATE_MEMCPY = 0x284;
+constexpr int FIX_SYM_PLAY = 0x288;
+constexpr int FIX_SYM_SYMBOL_STRING = 0x28c;
+constexpr int FIX_SYM_KERNEL_SYMBOL_WARNINGS = 0x290;
+constexpr int FIX_SYM_NETWORK_BOOTSTRAP = 0x294;
+
+}  // namespace jak3_symbols
+
 constexpr int false_symbol_offset() {
   return jak1_symbols::FIX_SYM_FALSE;
 }

common/type_system/Type.cpp

@@ -10,7 +10,7 @@
 #include "common/log/log.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace {
 std::string reg_kind_to_string(RegClass kind) {

common/type_system/TypeFieldLookup.cpp

@@ -9,7 +9,7 @@
 
 #include "common/log/log.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace {
 // debug prints for the reverse lookup

common/type_system/TypeSpec.cpp

@@ -7,7 +7,7 @@
 
 #include <stdexcept>
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 bool TypeTag::operator==(const TypeTag& other) const {
   return name == other.name && value == other.value;

common/type_system/TypeSystem.cpp

@@ -14,8 +14,8 @@
 #include "common/util/Assert.h"
 #include "common/util/math_util.h"
 
-#include "third-party/fmt/color.h"
-#include "third-party/fmt/core.h"
+#include "fmt/color.h"
+#include "fmt/core.h"
 
 namespace {
 template <typename... Args>
@@ -28,7 +28,7 @@ template <typename... Args>
   }
 
   throw std::runtime_error(
-      fmt::format("Type Error: {}", fmt::format(str, std::forward<Args>(args)...)));
+      fmt::format("Type Error: {}", fmt::format(fmt::runtime(str), std::forward<Args>(args)...)));
 }
 }  // namespace
 
@@ -1442,6 +1442,21 @@ std::vector<std::string> TypeSystem::search_types_by_parent_type(
   return results;
 }
 
+std::vector<std::string> TypeSystem::search_types_by_parent_type_strict(
+    const std::string& parent_type) {
+  std::vector<std::string> results = {};
+  for (const auto& [type_name, type_info] : m_types) {
+    // Only NullType's have no parent
+    if (!type_info->has_parent()) {
+      continue;
+    }
+    if (type_info->get_parent() == parent_type) {
+      results.push_back(type_name);
+    }
+  }
+  return results;
+}
+
 std::vector<std::string> TypeSystem::search_types_by_minimum_method_id(
     const int minimum_method_id,
     const std::optional<std::vector<std::string>>& existing_matches) {
@@ -1482,8 +1497,7 @@ std::vector<std::string> TypeSystem::search_types_by_size(
     }
   } else {
     for (const auto& [type_name, type_info] : m_types) {
-      // Only NullType's have no parent
-      if (!type_info->has_parent()) {
+      if (dynamic_cast<NullType*>(type_info.get())) {
         continue;
       }
       const auto size_of_type = m_types[type_name]->get_size_in_memory();
@@ -2113,7 +2127,7 @@ std::optional<std::string> find_best_field_in_structure(const TypeSystem& ts,
   }
   for (size_t i = start_field; i < (size_t)end_field; ++i) {
     const auto& field = st->fields().at(i);
-    auto type = ts.lookup_type(field.type());
+    auto type = ts.lookup_type_allow_partial_def(field.type());
     if (field.is_dynamic() || field.offset() > offset || field.user_placed() != want_fixed) {
       continue;
     }
@@ -2147,6 +2161,12 @@ std::optional<std::string> find_best_field_in_structure(const TypeSystem& ts,
     } else {
       int rel_offset = offset - field.offset();
       // array case (and array encompasses what we want)
+      ASSERT_MSG(
+          type->get_size_in_memory() > 0,
+          fmt::format(
+              "In type {}, type size was 0 for array field {} of type {}, check that the type "
+              "is fully defined.",
+              st->get_name(), field.name(), field.type().print()));
       int array_idx = rel_offset / type->get_size_in_memory();
       if (!field.is_inline() &&
           field.offset() + field.array_size() * type->get_load_size() > offset) {

common/type_system/TypeSystem.h

@@ -278,6 +278,7 @@ class TypeSystem {
   std::vector<std::string> search_types_by_parent_type(
       const std::string& parent_type,
       const std::optional<std::vector<std::string>>& existing_matches = {});
+  std::vector<std::string> search_types_by_parent_type_strict(const std::string& parent_type);
 
   std::vector<std::string> search_types_by_minimum_method_id(
       const int minimum_method_id,

common/type_system/defenum.cpp

@@ -13,7 +13,7 @@
 #include "common/util/BitUtils.h"
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace {
 const goos::Object& car(const goos::Object* x) {

common/type_system/deftype.cpp

@@ -13,7 +13,7 @@
 #include "common/type_system/state.h"
 #include "common/util/string_util.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 /*!
  * Missing Features

common/util/Assert.cpp

@@ -16,12 +16,12 @@ void private_assert_failed(const char* expr,
   if (!msg || msg[0] == '\0') {
     std::string log = fmt::format("Assertion failed: '{}'\n\tSource: {}:{}\n\tFunction: {}\n", expr,
                                   file, line, function);
-    lg::die(log);
+    lg::die("{}", log);
   } else {
     std::string log =
         fmt::format("Assertion failed: '{}'\n\tMessage: {}\n\tSource: {}:{}\n\tFunction: {}\n",
                     expr, msg, file, line, function);
-    lg::die(log);
+    lg::die("{}", log);
   }
   abort();
 }

common/util/BinaryReader.h

@@ -7,6 +7,7 @@
 
 #include <cstdint>
 #include <cstring>
+#include <span>
 #include <vector>
 
 #include "common/common_types.h"
@@ -14,28 +15,29 @@
 
 class BinaryReader {
  public:
-  explicit BinaryReader(const std::vector<uint8_t>& _buffer) : m_buffer(_buffer) {}
+  explicit BinaryReader(std::span<const uint8_t> _span) : m_span(_span) {}
 
   template <typename T>
   T read() {
-    ASSERT(m_seek + sizeof(T) <= m_buffer.size());
+    ASSERT(m_seek + sizeof(T) <= m_span.size());
     T obj;
-    memcpy(&obj, m_buffer.data() + m_seek, sizeof(T));
+    memcpy(&obj, m_span.data() + m_seek, sizeof(T));
     m_seek += sizeof(T);
     return obj;
   }
 
   void ffwd(int amount) {
     m_seek += amount;
-    ASSERT(m_seek <= m_buffer.size());
+    ASSERT(m_seek <= m_span.size());
   }
 
-  uint32_t bytes_left() const { return m_buffer.size() - m_seek; }
-  uint8_t* here() { return m_buffer.data() + m_seek; }
+  uint32_t bytes_left() const { return m_span.size() - m_seek; }
+  const uint8_t* here() { return m_span.data() + m_seek; }
+  BinaryReader at(u32 pos) { return BinaryReader(m_span.subspan(pos)); }
   uint32_t get_seek() const { return m_seek; }
   void set_seek(u32 seek) { m_seek = seek; }
 
  private:
-  std::vector<u8> m_buffer;
+  std::span<const u8> m_span;
   uint32_t m_seek = 0;
 };

common/util/DgoReader.cpp

@@ -9,6 +9,7 @@
 #include "dgo_util.h"
 
 #include "common/link_types.h"
+#include "common/log/log.h"
 
 #include "third-party/json.hpp"
 
@@ -25,9 +26,9 @@ DgoReader::DgoReader(std::string file_name, const std::vector<u8>& data)
 
     if (reader.bytes_left() < obj_header.size && i == header.object_count - 1 &&
         obj_header.size - reader.bytes_left() <= 48) {
-      printf(
-          "Warning: final file %s in DGO %s has a size missing %d bytes.  It will be adjusted from "
-          "%d to %d bytes.\n",
+      lg::print(
+          "Warning: final file {} in DGO {} has a size missing {} bytes.  It will be adjusted from "
+          "{} to {} bytes.\n",
           obj_header.name, header.name, obj_header.size - reader.bytes_left(), obj_header.size,
           (int)reader.bytes_left());
       obj_header.size = reader.bytes_left();
@@ -40,7 +41,7 @@ DgoReader::DgoReader(std::string file_name, const std::vector<u8>& data)
 
     entry.unique_name = get_object_file_name(entry.internal_name, reader.here(), obj_header.size);
     if (all_unique_names.find(entry.unique_name) != all_unique_names.end()) {
-      printf("Warning: there are multiple files named %s\n", entry.unique_name.c_str());
+      lg::print("Warning: there are multiple files named {}\n", entry.unique_name.c_str());
       entry.unique_name += '-';
       entry.unique_name += std::to_string(obj_header.size);
     }

common/util/FileUtil.cpp

@@ -22,7 +22,7 @@
 // This disables the use of PCLMULQDQ which is probably ok, but let's just be safe and disable it
 // because nobody will care if png compression is 10% slower.
 #define FPNG_NO_SSE 1
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 #include "third-party/fpng/fpng.cpp"
 #include "third-party/fpng/fpng.h"
 #include "third-party/lzokay/lzokay.hpp"
@@ -88,11 +88,23 @@ fs::path get_user_memcard_dir(GameVersion game_version) {
   return get_user_config_dir() / game_version_name / "saves";
 }
 
+fs::path get_user_screenshots_dir(GameVersion game_version) {
+  auto game_version_name = game_version_names[game_version];
+  return get_user_config_dir() / game_version_name / "screenshots";
+}
+
 fs::path get_user_misc_dir(GameVersion game_version) {
   auto game_version_name = game_version_names[game_version];
   return get_user_config_dir() / game_version_name / "misc";
 }
 
+fs::path get_user_features_dir(GameVersion game_version) {
+  auto game_version_name = game_version_names[game_version];
+  auto path = get_user_config_dir() / game_version_name / "features";
+  file_util::create_dir_if_needed(path);
+  return path;
+}
+
 struct {
   bool initialized = false;
   fs::path path_to_data;
@@ -381,7 +393,7 @@ std::string split_path_at(const fs::path& path, const std::vector<std::string>&
     split_str += folder + "/";
 #endif
   }
-  const auto& path_str = path.u8string();
+  const auto& path_str = path.string();
   return path_str.substr(path_str.find(split_str) + split_str.length());
 }
 
@@ -636,6 +648,9 @@ std::vector<fs::path> find_files_in_dir(const fs::path& dir, const std::regex& p
 
 std::vector<fs::path> find_files_recursively(const fs::path& base_dir, const std::regex& pattern) {
   std::vector<fs::path> files = {};
+  if (!fs::exists(base_dir)) {
+    return files;
+  }
   for (auto& p : fs::recursive_directory_iterator(base_dir)) {
     if (p.is_regular_file()) {
       if (std::regex_match(p.path().filename().string(), pattern)) {
@@ -692,15 +707,54 @@ void copy_file(const fs::path& src, const fs::path& dst) {
 std::string make_screenshot_filepath(const GameVersion game_version, const std::string& name) {
   std::string file_name;
   if (name.empty()) {
-    file_name = fmt::format("{}_{}.png", version_to_game_name(game_version),
-                            str_util::current_local_timestamp_no_colons());
+    file_name = fmt::format("{}.png", str_util::current_local_timestamp_no_colons());
   } else {
-    file_name = fmt::format("{}_{}_{}.png", version_to_game_name(game_version), name,
-                            str_util::current_local_timestamp_no_colons());
+    file_name = fmt::format("{}.png", name);
   }
-  const auto file_path = file_util::get_file_path({"screenshots", file_name});
+  const auto file_path = get_user_screenshots_dir(game_version) / file_name;
   file_util::create_dir_if_needed_for_file(file_path);
-  return file_path;
+  return file_path.string();
+}
+
+std::string get_majority_file_line_endings(const std::string& file_contents) {
+  size_t lf_count = 0;
+  size_t crlf_count = 0;
+
+  for (size_t i = 0; i < file_contents.size(); ++i) {
+    if (file_contents[i] == '\n') {
+      if (i > 0 && file_contents[i - 1] == '\r') {
+        crlf_count++;
+      } else {
+        lf_count++;
+      }
+    }
+  }
+
+  if (crlf_count > lf_count) {
+    return "\r\n";
+  }
+  return "\n";
+}
+
+std::pair<int, std::string> get_majority_file_line_endings_and_count(
+    const std::string& file_contents) {
+  size_t lf_count = 0;
+  size_t crlf_count = 0;
+
+  for (size_t i = 0; i < file_contents.size(); ++i) {
+    if (file_contents[i] == '\n') {
+      if (i > 0 && file_contents[i - 1] == '\r') {
+        crlf_count++;
+      } else {
+        lf_count++;
+      }
+    }
+  }
+
+  if (crlf_count > lf_count) {
+    return {lf_count + crlf_count, "\r\n"};
+  }
+  return {lf_count + crlf_count, "\n"};
 }
 
 }  // namespace file_util

common/util/FileUtil.h

@@ -31,7 +31,9 @@ fs::path get_user_home_dir();
 fs::path get_user_config_dir();
 fs::path get_user_settings_dir(GameVersion game_version);
 fs::path get_user_memcard_dir(GameVersion game_version);
+fs::path get_user_screenshots_dir(GameVersion game_version);
 fs::path get_user_misc_dir(GameVersion game_version);
+fs::path get_user_features_dir(GameVersion game_version);
 fs::path get_jak_project_dir();
 
 bool create_dir_if_needed(const fs::path& path);
@@ -69,4 +71,7 @@ std::vector<fs::path> sort_filepaths(const std::vector<fs::path>& paths, const b
 /// Will overwrite the destination if it exists
 void copy_file(const fs::path& src, const fs::path& dst);
 std::string make_screenshot_filepath(const GameVersion game_version, const std::string& name = "");
+std::string get_majority_file_line_endings(const std::string& file_contents);
+std::pair<int, std::string> get_majority_file_line_endings_and_count(
+    const std::string& file_contents);
 }  // namespace file_util

common/util/FontUtils.cpp

@@ -16,13 +16,14 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
-#include "third-party/fmt/format.h"
+#include "fmt/core.h"
+#include "fmt/format.h"
 
 const std::unordered_map<std::string, GameTextVersion> sTextVerEnumMap = {
     {"jak1-v1", GameTextVersion::JAK1_V1},
     {"jak1-v2", GameTextVersion::JAK1_V2},
-    {"jak2", GameTextVersion::JAK2}};
+    {"jak2", GameTextVersion::JAK2},
+    {"jak3", GameTextVersion::JAK3}};
 
 const std::string& get_text_version_name(GameTextVersion version) {
   for (auto& [name, ver] : sTextVerEnumMap) {
@@ -261,7 +262,8 @@ bool GameTextFontBank::valid_char_range(const char in) const {
     return ((in >= '0' && in <= '9') || (in >= 'A' && in <= 'Z') ||
             m_passthrus->find(in) != m_passthrus->end()) &&
            in != '\\';
-  } else if (m_version == GameTextVersion::JAK2) {
+  } else if (m_version == GameTextVersion::JAK2 || m_version == GameTextVersion::JAK3 ||
+             m_version == GameTextVersion::JAKX) {
     return ((in >= '0' && in <= '9') || (in >= 'A' && in <= 'Z') || (in >= 'a' && in <= 'z') ||
             m_passthrus->find(in) != m_passthrus->end()) &&
            in != '\\';
@@ -913,7 +915,9 @@ GameTextFontBank g_font_bank_jak1_v2(GameTextVersion::JAK1_V2,
  * GAME TEXT FONT BANK - JAK 2
  * ================================
  * This font is used in:
- * - Jak 2 - NTSC - v1
+ * - Jak II
+ * - Jak II: Renegade
+ * - ジャックＸダクスター2
  */
 
 static std::unordered_set<char> s_passthrus_jak2 = {'~', ' ', ',', '.', '-', '+', '(', ')',
@@ -1904,6 +1908,21 @@ GameTextFontBank g_font_bank_jak2(GameTextVersion::JAK2,
                                   &s_replace_info_jak2,
                                   &s_passthrus_jak2);
 
+/*!
+ * ================================
+ * GAME TEXT FONT BANK - JAK 3
+ * ================================
+ * This font is used in:
+ * - Jak 3
+ */
+
+// TODO cyrillic
+
+GameTextFontBank g_font_bank_jak3(GameTextVersion::JAK3,
+                                  &s_encode_info_jak2,
+                                  &s_replace_info_jak2,
+                                  &s_passthrus_jak2);
+
 /*!
  * ========================
  * GAME TEXT FONT BANK LIST
@@ -1914,7 +1933,8 @@ GameTextFontBank g_font_bank_jak2(GameTextVersion::JAK2,
 std::map<GameTextVersion, GameTextFontBank*> g_font_banks = {
     {GameTextVersion::JAK1_V1, &g_font_bank_jak1_v1},
     {GameTextVersion::JAK1_V2, &g_font_bank_jak1_v2},
-    {GameTextVersion::JAK2, &g_font_bank_jak2}};
+    {GameTextVersion::JAK2, &g_font_bank_jak2},
+    {GameTextVersion::JAK3, &g_font_bank_jak3}};
 
 const GameTextFontBank* get_font_bank(GameTextVersion version) {
   return g_font_banks.at(version);
@@ -1927,6 +1947,8 @@ const GameTextFontBank* get_font_bank_from_game_version(GameVersion version) {
       return get_font_bank(GameTextVersion::JAK1_V2);
     case GameVersion::Jak2:
       return get_font_bank(GameTextVersion::JAK2);
+    case GameVersion::Jak3:
+      return get_font_bank(GameTextVersion::JAK3);
     default:
       ASSERT_MSG(false, "Unsupported game for get_font_bank_from_game_version");
   }

common/util/Range.h

@@ -67,4 +67,4 @@ class Range {
  private:
   T m_start = {};
   T m_end = {};
-};
+};

common/util/Trie.h

@@ -12,7 +12,7 @@
  * It owns the memory for the objects it stores.
  * Doing an insert will create a copy of your object.
  *
- * Other that deleting the whole thing, there is no support for removing a node.
+ * Other than deleting the whole thing, there is no support for removing a node.
  */
 template <typename T>
 class Trie {
@@ -150,7 +150,7 @@ class Trie {
 };
 
 template <typename T>
-Trie<T>::~Trie<T>() {
+Trie<T>::~Trie() {
   m_root.delete_children();
   m_size = 0;
 }

common/util/ast_util.cpp

@@ -0,0 +1,31 @@
+#include "ast_util.h"
+
+namespace ast_util {
+std::string get_source_code(const std::string& source, const TSNode& node) {
+  uint32_t start = ts_node_start_byte(node);
+  uint32_t end = ts_node_end_byte(node);
+  return source.substr(start, end - start);
+}
+
+void search_for_forms_that_begin_with(const std::string& source,
+                                      const TSNode curr_node,
+                                      const std::vector<std::string>& prefix,
+                                      std::vector<TSNode>& results) {
+  if (ts_node_child_count(curr_node) == 0) {
+    return;
+  }
+  std::vector<std::string> node_elements;
+  bool added = false;
+  for (size_t i = 0; i < ts_node_child_count(curr_node); i++) {
+    const auto child_node = ts_node_child(curr_node, i);
+    const auto contents = get_source_code(source, child_node);
+    node_elements.push_back(contents);
+    // Check for a match
+    if (node_elements == prefix && !added) {
+      results.push_back(curr_node);
+      added = true;
+    }
+    search_for_forms_that_begin_with(source, child_node, prefix, results);
+  }
+}
+}  // namespace ast_util

common/util/ast_util.h

@@ -0,0 +1,15 @@
+#pragma once
+
+#include <string>
+#include <vector>
+
+#include "tree_sitter/api.h"
+
+namespace ast_util {
+std::string get_source_code(const std::string& source, const TSNode& node);
+void search_for_forms_that_begin_with(const std::string& source,
+                                      const TSNode curr_node,
+                                      const std::vector<std::string>& prefix,
+                                      std::vector<TSNode>& results);
+
+}  // namespace ast_util

common/util/compress.cpp

@@ -5,7 +5,7 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 #include "third-party/zstd/lib/zstd.h"
 
 namespace compression {

common/util/dgo_util.cpp

@@ -5,7 +5,7 @@
 #include "common/util/Assert.h"
 #include "common/versions/versions.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 /*!
  * Assert false if the char[] has non-null data after the null terminated string.
@@ -21,7 +21,7 @@ void assert_string_empty_after(const char* str, int size) {
   }
 }
 
-std::string get_object_file_name(const std::string& original_name, u8* data, int size) {
+std::string get_object_file_name(const std::string& original_name, const u8* data, int size) {
   const std::string art_group_text_strings[] = {
       fmt::format("/src/next/data/art-group{}/", versions::jak1::ART_FILE_VERSION),
       fmt::format("/src/jak2/final/art-group{}/", versions::jak2::ART_FILE_VERSION),

common/util/dgo_util.h

@@ -6,4 +6,4 @@
 #include "common/versions/versions.h"
 
 void assert_string_empty_after(const char* str, int size);
-std::string get_object_file_name(const std::string& original_name, u8* data, int size);
+std::string get_object_file_name(const std::string& original_name, const u8* data, int size);

common/util/print_float.cpp

@@ -5,8 +5,8 @@
 #include "common/goal_constants.h"
 #include "common/util/Assert.h"
 
+#include "fmt/core.h"
 #include "third-party/dragonbox.h"
-#include "third-party/fmt/core.h"
 
 /*!
  * Convert a float to a string. The string is _always_ in this format:
@@ -219,4 +219,4 @@ bool proper_float(float value) {
   } else {
     return true;
   }
-}
+}

common/util/string_util.cpp

@@ -102,6 +102,22 @@ std::vector<std::string> split(const ::std::string& str, char delimiter) {
   return google_diff::split_string(str, delimiter);
 }
 
+std::vector<std::string> split_string(const std::string& str, const std::string& delimiter) {
+  std::vector<std::string> parsed;
+  std::string::size_type pos = 0;
+  while (true) {
+    const std::string::size_type found = str.find(delimiter, pos);
+    if (found == std::string::npos) {
+      parsed.push_back(str.substr(pos));
+      break;
+    } else {
+      parsed.push_back(str.substr(pos, found - pos));
+      pos = found + delimiter.length();
+    }
+  }
+  return parsed;
+}
+
 std::vector<std::string> regex_get_capture_groups(const std::string& str,
                                                   const std::string& regex) {
   std::vector<std::string> groups;
@@ -229,4 +245,13 @@ std::string titlize(const std::string& str) {
   }
   return new_str;
 }
+
+std::string pad_right(const std::string& input, const int width, const char padding_char) {
+  if (input.length() >= width) {
+    return input;  // No need to pad if input length is already greater or equal to width
+  } else {
+    int padding_width = width - input.length();
+    return input + std::string(padding_width, padding_char);
+  }
+}
 }  // namespace str_util

common/util/string_util.h

@@ -18,7 +18,9 @@ int line_count(const std::string& str);
 bool valid_regex(const std::string& regex);
 std::string diff(const std::string& lhs, const std::string& rhs);
 /// Default splits on \n characters
+/// DEPRECATED - stop using it, limited as it only takes a char
 std::vector<std::string> split(const ::std::string& str, char delimiter = '\n');
+std::vector<std::string> split_string(const std::string& str, const std::string& delimiter = "\n");
 std::string join(const std::vector<std::string>& strs, const std::string& join_with);
 std::vector<std::string> regex_get_capture_groups(const std::string& str, const std::string& regex);
 bool replace(std::string& str, const std::string& from, const std::string& to);
@@ -33,4 +35,5 @@ std::string to_lower(const std::string& str);
 /// Is this a valid character for a hex number?
 bool hex_char(char c);
 std::string titlize(const std::string& str);
+std::string pad_right(const std::string& input, const int width, const char padding_char);
 }  // namespace str_util

common/util/trie_with_duplicates.h

@@ -0,0 +1,140 @@
+#pragma once
+
+#include <array>
+#include <memory>
+#include <string>
+#include <vector>
+
+// A normal Trie does not allow for duplicate keys, however this one does
+// It allows for insertion and removal
+//
+// Note, keys assume only basic ASCII, which is _fine_ as OpenGOAL itself has this
+// limitation as well.
+template <typename T>
+class TrieWithDuplicates {
+ private:
+  struct TrieNode {
+    std::array<std::unique_ptr<TrieNode>, 256> children;
+    std::vector<std::unique_ptr<T>> elements;
+  };
+
+  std::unique_ptr<TrieNode> root = std::make_unique<TrieNode>();
+
+ public:
+  TrieWithDuplicates() {}
+
+  T* insert(const std::string& key, const T& element) {
+    std::unique_ptr<T> new_element = std::make_unique<T>(element);
+    TrieNode* curr_node = root.get();
+    for (const char character : key) {
+      auto& child = curr_node->children[(uint8_t)character];
+      if (!child) {
+        child = std::make_unique<TrieNode>();
+      }
+      curr_node = child.get();
+    }
+    curr_node->elements.push_back(std::move(new_element));
+    return curr_node->elements.back().get();
+  }
+
+  std::vector<T*> retrieve_with_prefix(const std::string& prefix) const {
+    std::vector<T*> results;
+    TrieNode* curr_node = root.get();
+    for (const char character : prefix) {
+      const auto& child = curr_node->children.at((uint8_t)character);
+      if (child == nullptr) {
+        return results;  // tree ends, nothing found with that prefix
+      } else {
+        curr_node = child.get();
+      }
+    }
+    retrieve_elements(curr_node, results);
+    return results;
+  }
+
+  std::vector<T*> retrieve_with_exact(const std::string& key) const {
+    std::vector<T*> results;
+    TrieNode* curr_node = root.get();
+    for (const char character : key) {
+      const auto& child = curr_node->children.at((uint8_t)character);
+      if (child == nullptr) {
+        return results;  // tree ends, nothing found with that key
+      } else {
+        curr_node = child.get();
+      }
+    }
+    for (const auto& element : curr_node->elements) {
+      results.push_back(element.get());
+    }
+    return results;
+  }
+
+  bool remove(const std::string& key, const T* to_be_removed) {
+    TrieNode* curr_node = root.get();
+    for (const char character : key) {
+      const auto& child = curr_node->children.at((uint8_t)character);
+      if (child == nullptr) {
+        return false;  // tree ends, nothing found with that key
+      } else {
+        curr_node = child.get();
+      }
+    }
+    // Since the trie holds duplicates, we can't delete on the key alone
+    // now search to see which element is identical
+    auto it = curr_node->elements.begin();
+    while (it != curr_node->elements.end()) {
+      if (it->get() == to_be_removed) {
+        it = curr_node->elements.erase(it);
+        return true;  // we can assume that the same ptr isn't stored twice.
+      } else {
+        ++it;
+      }
+    }
+    return false;
+  }
+
+  // Return the total number of elements stored in the TrieMap
+  int size() const {
+    int count = 0;
+    count_elements(root.get(), count);
+    return count;
+  }
+
+  std::vector<T*> get_all_elements() const {
+    std::vector<T*> results;
+    get_all_elements_helper(root.get(), results);
+    return results;
+  }
+
+ private:
+  void retrieve_elements(const TrieNode* node, std::vector<T*>& results) const {
+    for (const auto& element : node->elements) {
+      results.push_back(element.get());
+    }
+    for (const auto& child : node->children) {
+      if (child.get() != nullptr) {
+        retrieve_elements(child.get(), results);
+      }
+    }
+  }
+
+  void count_elements(const TrieNode* node, int& count) const {
+    count += node->elements.size();
+    for (const auto& child : node->children) {
+      if (child.get() != nullptr) {
+        count_elements(child.get(), count);
+      }
+    }
+  }
+
+  void get_all_elements_helper(const TrieNode* node, std::vector<T*>& result) const {
+    for (const auto& element : node->elements) {
+      result.push_back(element.get());
+    }
+    for (const auto& child : node->children) {
+      if (child.get() != nullptr) {
+        get_all_elements_helper(child.get(), result);
+      }
+    }
+  }
+};

common/versions/versions.cpp

@@ -3,21 +3,23 @@
 #include "common/util/Assert.h"
 #include "common/versions/revision.h"
 
-#include "third-party/fmt/core.h"
-#include "third-party/fmt/format.h"
+#include "fmt/core.h"
+#include "fmt/format.h"
 
 GameVersion game_name_to_version(const std::string& name) {
   if (name == "jak1") {
     return GameVersion::Jak1;
   } else if (name == "jak2") {
     return GameVersion::Jak2;
+  } else if (name == "jak3") {
+    return GameVersion::Jak3;
   } else {
     ASSERT_MSG(false, fmt::format("invalid game name: {}", name));
   }
 }
 
 bool valid_game_version(const std::string& name) {
-  return name == "jak1" || name == "jak2";
+  return name == "jak1" || name == "jak2" || name == "jak3";
 }
 
 std::string version_to_game_name(GameVersion v) {
@@ -26,6 +28,8 @@ std::string version_to_game_name(GameVersion v) {
       return "jak1";
     case GameVersion::Jak2:
       return "jak2";
+    case GameVersion::Jak3:
+      return "jak3";
     default:
       ASSERT_MSG(false, fmt::format("no game_name for version: {} found", fmt::underlying(v)));
   }
@@ -37,13 +41,16 @@ std::string version_to_game_name_external(GameVersion v) {
       return "Jak 1";
     case GameVersion::Jak2:
       return "Jak 2";
+    case GameVersion::Jak3:
+      return "Jak 3";
     default:
       ASSERT_MSG(false, fmt::format("no game_name for version: {} found", fmt::underlying(v)));
   }
 }
 
 std::vector<std::string> valid_game_version_names() {
-  return {game_version_names[GameVersion::Jak1], game_version_names[GameVersion::Jak2]};
+  return {game_version_names[GameVersion::Jak1], game_version_names[GameVersion::Jak2],
+          game_version_names[GameVersion::Jak3]};
 }
 
 std::string build_revision() {

common/versions/versions.h

@@ -34,6 +34,7 @@ constexpr u32 TX_PAGE_VERSION = 8;
 namespace jak3 {
 constexpr u32 ART_FILE_VERSION = 8;
 constexpr u32 LEVEL_FILE_VERSION = 36;
+constexpr u32 DGO_FILE_VERSION = 1;
 constexpr u32 TX_PAGE_VERSION = 8;
 }  // namespace jak3
 

custom_levels/jak1/test-zone/test-zone.jsonc

@@ -20,16 +20,40 @@
   // this makes collision 2x slower and bigger, so only use if really needed
   "double_sided_collide": false,
 
-  // available res-lump tag data types:
-  // int32, float, meters, vector, vector4m (meters)
+  // available res-lump tag types:
+  // integer types: int32, uint32, enum-int32, enum-uint32
+  // float types: float, meters (1 meter = 4096.0 units), degrees (65536.0 = 360°)
+  // vector types: vector (normal floats), vector4m (meters), vector3m (meters with w set to 1.0), vector-vol (normal floats with w in meters), movie-pos (meters with w in degrees)
+  // special types: symbol, type, string, eco-info, cell-info, buzzer-info, water-height
   //
   // examples:
   //
   // adds a float tag 'spring-height' with value of 200 meters (1 meter = 4096.0 units):
   // "spring-height": ["meters", 200.0]
   //
-  // adds a vector tag 'movie-pos':
-  // "movie-pos": ["vector", [4096000.0, -176128.0, 1353973.76, 1.0]]
+  // adds a degrees tag 'rotoffset':
+  // "rotoffset": ["degrees", -45.0]
+  //
+  // adds a movie-pos tag:
+  // "movie-pos": ["movie-pos", [100.22, -25.3, 99.5, 180.0]]
+  //
+  // adds an enum tag 'options':
+  // "options": ["enum-int32", "(fact-options large)"]
+  //
+  // adds a water-height tag:
+  // "water-height": ["water-height", 25.0, 0.5, 2.0, "(water-flags wt08 wt03 wt01)"]
+  //
+  // adds an eco-info tag:
+  // "eco-info": ["eco-info", "(pickup-type health)", 2]
+  //
+  // adds a cell-info tag:
+  // "eco-info": ["cell-info", "(game-task training-gimmie)"]
+  //
+  // adds a buzzer-info tag:
+  // "eco-info": ["buzzer-info", "(game-task training-buzzer)", 5]
+  //
+  // adds a 'symbol' tag (using the 'type' and 'string' lump types works the same way):
+  // "symbol-list": ["symbol", "sym-1", "sym-2"]
 
    // The base actor id for your custom level. If you have multiple levels this should be unique!
    "base_id": 100,
@@ -65,7 +89,7 @@
       "lump": {
         "name": "test-ambient",
         "type": "'hint",
-        "text-id": ["int32", 557], // text id for the "POWER CELL" text
+        "text-id": ["enum-uint32", "(text-id fuel-cell)"], // text id for the "POWER CELL" text
         "play-mode": "'notice"
       }
     }
@@ -75,31 +99,32 @@
     {
       "trans": [-21.6238, 20.0496, 17.1191], // translation
       "etype": "fuel-cell",  // actor type
-      "game_task": 0, // associated game task (for powercells, etc)
+      "game_task": "(game-task none)", // associated game task (for powercells, etc)
       "quat": [0, 0, 0, 1], // quaternion
       "bsphere": [-21.6238, 19.3496, 17.1191, 10], // bounding sphere
       "lump": {
-        "name": "test-fuel-cell"
+        "name": "test-fuel-cell",
+        "eco-info": ["cell-info", "(game-task none)"]
       }
     },
 
     {
       "trans": [-15.2818, 15.2461, 17.1360], // translation
       "etype": "crate",  // actor type
-      "game_task": 0, // associated game task (for powercells, etc)
+      "game_task": "(game-task none)", // associated game task (for powercells, etc)
       "quat": [0, 0, 0, 1], // quaternion
       "bsphere": [-15.2818, 15.2461, 17.1360, 10], // bounding sphere
       "lump": {
         "name": "test-crate",
         "crate-type": "'steel",
-        "eco-info": ["int32", 5, 10]
+        "eco-info": ["eco-info", "(pickup-type money)", 10]
       }
     },
 
     {
       "trans": [-5.4630, 17.4553, 1.6169], // translation
       "etype": "eco-yellow",  // actor type
-      "game_task": 0, // associated game task (for powercells, etc)
+      "game_task": "(game-task none)", // associated game task (for powercells, etc)
       "quat": [0, 0, 0, 1], // quaternion
       "bsphere": [-5.4630, 17.4553, 1.6169, 10], // bounding sphere
       "lump": {
@@ -109,7 +134,7 @@
     // {
     //   "trans": [-7.41, 3.5, 28.42], // translation
     //   "etype": "plat",  // actor type
-    //   "game_task": 0, // associated game task (for powercells, etc)
+    //   "game_task": "(game-task none)", // associated game task (for powercells, etc)
     //   "quat": [0, 0, 0, 1], // quaternion
     //   "bsphere": [-7.41, 3.5, 28.42, 10], // bounding sphere
     //   "lump": {

custom_levels/jak2/test-zone/test-zone.jsonc

@@ -20,24 +20,41 @@
   // this makes collision 2x slower and bigger, so only use if really needed
   "double_sided_collide": false,
 
-  // available res-lump tag data types:
-  // int32, float, meters, vector, vector4m (meters)
+  // available res-lump tag types:
+  // integer types: int32, uint32, enum-int32, enum-uint32
+  // float types: float, meters (1 meter = 4096.0 units), degrees (65536.0 = 360°)
+  // vector types: vector (normal floats), vector4m (meters), vector3m (meters with w set to 1.0), vector-vol (normal floats with w in meters), movie-pos (meters with w in degrees)
+  // special types: symbol, type, string, eco-info, cell-info, buzzer-info, water-height
   //
   // examples:
   //
   // adds a float tag 'spring-height' with value of 200 meters (1 meter = 4096.0 units):
   // "spring-height": ["meters", 200.0]
   //
-  // adds a vector tag 'movie-pos':
-  // "movie-pos": ["vector", [4096000.0, -176128.0, 1353973.76, 1.0]]
+  // adds a degrees tag 'rotoffset':
+  // "rotoffset": ["degrees", -45.0]
+  //
+  // adds a movie-pos tag:
+  // "movie-pos": ["movie-pos", [100.22, -25.3, 99.5, 180.0]]
+  //
+  // adds an enum tag 'options':
+  // "options": ["enum-int32", "(fact-options large)"]
+  //
+  // adds a water-height tag:
+  // "water-height": ["water-height", 25.0, 0.5, 2.0, "(water-flags can-swim can-wade)"]
+  //
+  // adds an eco-info tag:
+  // "eco-info": ["eco-info", "(pickup-type health)", 2]
+  //
+  // adds a 'type' tag (using the "symbol" and "string" lump types works the same way):
+  // "spawn-types": ["type", "spyder", "juicer"]
 
    // The base actor id for your custom level. If you have multiple levels, this should be unique!
    "base_id": 100,
 
   // All art groups you want to use in your custom level. Will add their models and corresponding textures to the FR3 file.
   // Removed so that the release builds don't have to double-decompile the game
-  //"art_groups": ["prsn-torture-ag"],
-  "art_groups": [],
+  // "art_groups": ["prsn-torture-ag"],
 
    // Any textures you want to include in your custom level.
    // This is mainly useful for textures which are not in the common level files and have no art group associated with them.
@@ -55,14 +72,14 @@
       "bsphere": [-15.2818, 15.2461, 17.1360, 10], // bounding sphere
       "lump": {
         "name": "test-crate",
-        "eco-info": ["int32", 18, 2]
+        "eco-info": ["eco-info", "(pickup-type health)", 2]
       }
     },
 
     {
       "trans": [-5.4630, 17.4553, 1.6169], // translation
       "etype": "eco-yellow",  // actor type
-      "game_task": 0, // associated game task (for powercells, etc)
+      "game_task": "(game-task none)", // associated game task (for powercells, etc)
       "kill_mask": 0,
       "quat": [0, 0, 0, 1], // quaternion
       "bsphere": [-5.4630, 17.4553, 1.6169, 10], // bounding sphere
@@ -74,7 +91,7 @@
     {
       "trans": [-7.41, 13.5, 28.42], // translation
       "etype": "prsn-torture",  // actor type
-      "game_task": 0, // associated game task (for powercells, etc)
+      "game_task": "(game-task none)", // associated game task (for powercells, etc)
       "quat": [0, 0, 0, 1], // quaternion
       "bsphere": [-7.41, 13.5, 28.42, 10], // bounding sphere
       "lump": {

decompiler/Disasm/Instruction.cpp

@@ -10,7 +10,7 @@
 
 #include "decompiler/ObjectFile/LinkedObjectFile.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace decompiler {
 /*!

decompiler/Disasm/InstructionDecode.cpp

@@ -10,7 +10,7 @@
 
 #include "decompiler/ObjectFile/LinkedObjectFile.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace decompiler {
 // utility class to extract fields of an opcode.

decompiler/Disasm/InstructionParser.cpp

@@ -7,7 +7,7 @@
 #include "common/common_types.h"
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace decompiler {
 InstructionParser::InstructionParser() {

decompiler/Disasm/Register.cpp

@@ -9,7 +9,7 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/format.h"
+#include "fmt/format.h"
 
 namespace decompiler {
 namespace Reg {

decompiler/Function/Function.h

@@ -195,6 +195,8 @@ class Function {
 
   std::vector<std::string> types_defined;
 
+  int process_stack_size = 0;
+
  private:
   void check_epilogue(const LinkedObjectFile& file);
   void resize_first_block(int new_start, const LinkedObjectFile& file);

decompiler/Function/Warnings.h

@@ -7,7 +7,7 @@
 
 #include "common/util/Assert.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace decompiler {
 class DecompWarnings {
@@ -36,7 +36,7 @@ class DecompWarnings {
 
   template <typename... Args>
   void error_and_throw(const std::string& str, Args&&... args) {
-    auto text = fmt::format(str, std::forward<Args>(args)...);
+    auto text = fmt::format(fmt::runtime(str), std::forward<Args>(args)...);
     _warning(Warning::Kind::ERR, false, text);
     throw std::runtime_error(text);
   }
@@ -99,7 +99,7 @@ class DecompWarnings {
 
   template <typename... Args>
   void _warning(Warning::Kind kind, bool unique, const std::string& str, Args&&... args) {
-    std::string msg = fmt::format(str, std::forward<Args>(args)...);
+    std::string msg = fmt::format(fmt::runtime(str), std::forward<Args>(args)...);
     if (unique) {
       if (unique_warnings.find(msg) != unique_warnings.end()) {
         return;

decompiler/IR2/AtomicOp.cpp

@@ -12,7 +12,7 @@
 
 #include "decompiler/ObjectFile/LinkedObjectFile.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace decompiler {
 /////////////////////////////

decompiler/IR2/AtomicOpTypeAnalysis.cpp

@@ -10,7 +10,7 @@
 #include "decompiler/util/TP_Type.h"
 #include "decompiler/util/type_utils.h"
 
-#include "third-party/fmt/core.h"
+#include "fmt/core.h"
 
 namespace decompiler {
 
@@ -191,7 +191,7 @@ TP_Type SimpleExpression::get_type(const TypeState& input,
         return TP_Type::make_from_ts("float");
       }
       // new for jak 2:
-      if (env.version == GameVersion::Jak2 && in_type.is_integer_constant() &&
+      if (env.version >= GameVersion::Jak2 && in_type.is_integer_constant() &&
           (s64)((s32)in_type.get_integer_constant()) == (s64)in_type.get_integer_constant()) {
         return TP_Type::make_from_ts("float");
       }
@@ -406,7 +406,7 @@ TP_Type SimpleExpression::get_type_int2(const TypeState& input,
       }
 
       if (m_kind == Kind::RIGHT_SHIFT_ARITH) {
-        if (env.version == GameVersion::Jak2 && arg0_type.typespec().base_type() == "float") {
+        if (env.version >= GameVersion::Jak2 && arg0_type.typespec().base_type() == "float") {
           return TP_Type::make_from_ts(TypeSpec("float"));
         }
         return TP_Type::make_from_ts(TypeSpec("int"));

decompiler/IR2/Env.cpp

@@ -64,6 +64,44 @@ void Env::set_remap_for_new_method(const TypeSpec& ts) {
   }
 }
 
+void Env::set_remap_for_relocate_method(const TypeSpec& ts) {
+  int nargs = ts.arg_count() - 1;
+  m_var_remap["a0-0"] = "this";
+  if (ts.get_arg(1).base_type() == "kheap") {
+    m_var_remap["a1-0"] = "heap";
+  } else {
+    m_var_remap["a1-0"] = "offset";
+  }
+  if (nargs > 1 && ts.get_arg(2).base_type() == "pointer") {
+    m_var_remap["a2-0"] = "name";
+  }
+  for (int i = 3; i < nargs; i++) {
+    m_var_remap[get_reg_name(i)] = ("arg" + std::to_string(i - 3));
+  }
+  if (ts.try_get_tag("behavior")) {
+    m_var_remap["s6-0"] = "self";
+    m_pp_mapped_by_behavior = true;
+  } else {
+    m_var_remap["s6-0"] = "pp";
+  }
+}
+
+void Env::set_remap_for_memusage_method(const TypeSpec& ts) {
+  int nargs = ts.arg_count() - 1;
+  m_var_remap["a0-0"] = "this";
+  m_var_remap["a1-0"] = "usage";
+  m_var_remap["a2-0"] = "flags";
+  for (int i = 3; i < nargs; i++) {
+    m_var_remap[get_reg_name(i)] = ("arg" + std::to_string(i - 3));
+  }
+  if (ts.try_get_tag("behavior")) {
+    m_var_remap["s6-0"] = "self";
+    m_pp_mapped_by_behavior = true;
+  } else {
+    m_var_remap["s6-0"] = "pp";
+  }
+}
+
 void Env::set_remap_for_method(const TypeSpec& ts) {
   int nargs = ts.arg_count() - 1;
   m_var_remap["a0-0"] = "this";